resolution interpolation to reach higher iso

Started by GammaNu, March 22, 2017, 08:34:29 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

GammaNu

Hi everyone,

I've just read a Sony A7s II test and found stunning his noice management in high iso.
From what i know, what allow a DSLR to handle high iso is the number of photon by photosite / pixel it can capture. (same way with less iso and more exposure time)

Is it possible, with a custom firmware like MagicLantern, or in post processing (on raw, perhaps on jpg ?) to use interpolation (merge adjacent pixel or better interpolation) to simulate higher iso / longer exposure by lowering resolution ?

I'm thinking cumulate raw luminance in deep dark situation and rounding the color part.

Any information about feasibility ?
Is it already in use ?

Regards,
GammaNu

dmilligan

Quote from: GammaNu on March 22, 2017, 08:34:29 PM
in post processing (on raw, perhaps on jpg ?) to use interpolation (merge adjacent pixel or better interpolation) to simulate higher iso / longer exposure by lowering resolution ?
Yes. In post, simply apply a blur to your image. You will reduce noise at the expense of resolution (that's basically what pixel binning is).

Of course most people want to reduce noise without sacrificing resolution. There are various noise reduction algorithms out there that attempt this with varying degrees of usually very limited success.

SpcCb

Hello ΓΝ,

It's already done by some Canon cameras (like the 5D3) by the stock firmware (internal). See for << pixel binning >>.
You can also do it in post processing, several softwares can do it, even photoshop (menu filter>divers>other by using matrix convolution).
Best results will be with a RAW source image, of course (because of debayering).

Have fun ;)


Edit: Harf! dmilligan was first :D

SpcCb

Quote from: dmilligan on March 23, 2017, 01:51:57 AM
Yes. In post, simply apply a blur to your image. You will reduce noise at the expense of resolution (that's basically what pixel binning is).
(...)
I'm not so agree with that :)
Blurring pixels is far to binning pixels: for example you loose the level gain and so the SNR gain at same level.

a1ex

Binning is simply blurring followed by downsampling (reducing the resolution by dropping pixels).

Example on how 5D3 does 3x3 binning to get 1080p Bayer raw from a full-res Bayer matrix:

function im = bin_pixels_3x3(im)

f = 1/9 * ...
[
    1 0 1 0 1;
    0 0 0 0 0;
    1 0 1 0 1;
    0 0 0 0 0;
    1 0 1 0 1;
];

imf = imfilter(im, f);
im = imf(1:3:end, 1:3:end);


First step is a 3x3 blur (averaging) on each Bayer channel, second step is keeping the central pixel from each 3x3 block (after filtering).

Also FYI, on Canon DSLRs, pixel binning is done by hardware (analog electronics), not firmware.

Reference: http://magiclantern.fm/forum/index.php?topic=16516.0


SpcCb

Quote from: a1ex on March 23, 2017, 02:20:37 AM
Binning is simply blurring followed by downsampling (reducing the resolution by dropping pixels).

Example on how 5D3 does 3x3 binning to get 1080p Bayer raw from a full-res Bayer matrix:

function im = bin_pixels_3x3(im)

f = 1/9 * ...
[
    1 0 1 0 1;
    0 0 0 0 0;
    1 0 1 0 1;
    0 0 0 0 0;
    1 0 1 0 1;
];

imf = imfilter(im, f);
im = imf(1:3:end, 1:3:end);


First step is a 3x3 blur (averaging) on each Bayer channel, second step is keeping the central pixel from each 3x3 block (after filtering).

Also FYI, on Canon DSLRs, pixel binning is done by hardware (analog electronics), not firmware.

Reference: http://magiclantern.fm/forum/index.php?topic=16516.0
Sorry, I use to split both method in my work because averaging or binning, it's not the same thing for me.
Sort explanation -> http://harvestimaging.com/blog/?p=1560
As you spoke of binning in some camera I thought it was binning _I mean pixel signal addition_ (was there a square root f(px-nbr) noise reduction when you made tests about that last year?), but if you say it's averaging, it's not binning. Or this is not clear.
It makes me thinking what is doing the averaging.. ??? Maybe it will be interesting to see the source code of the firmware function? Or the source code of the ASIC||FPGA who does it?

a1ex

The hints I have are summarized here: https://www.magiclantern.fm/forum/index.php?topic=16608.msg174433#msg174433

I may not be familiar with the proper terminology, as I don't work in this field, sorry.

However, summing is (a+b+...) and averaging is (a+b+...)/n. When you do this in software on floating point, the SNR does not change (as the noise is scaled by the same amount).

When you do this on integer math, the latter has lower SNR from round-off (quantization) error.

When you do this on analog hardware (which is not my field), I'd expect additional noise from the electronics (whether it's with transistors, capacitors or whatever).

The hints from Chipworks suggest it's not digital electronics (not a FPGA), but analog electronics (an additional transistor on the CMOS chip).

What I can tell you from observations:




The signal coming from a 3x3 bin, that enters the ADC, appears to have roughly the same levels*) as the signal coming from a single sensel. In both cases, black level is 2048 and white level is above 15000. So, if the signals are summed somehow, they are normalized (therefore, averaged).




The black level appears to be clamped by an analog feedback loop, which I believe it's applied before the ADC.

Whether the signal normalization is done on the CMOS chip or a separate analog amplifier (before the gains we have identified), I don't know. However, the noise numbers are clearly higher than with software averaging, and the overall read noise level appears comparable to the level from regular photo mode / 5x zoom.

I see this as a hint that pixel binning happens before the main source of read noise in Canon sensors. This noise is rumoured to be on the way between the CMOS chip and the ADC, which are different chips on DIGIC 5 and earlier models. The design apparently changed with newer models.




Reducing the available gains (ADTG, registers 0xFE, registers 8/9/A/B) gives similar behavior in 3x3 and 1:1 modes (this was expected, since it all happens before these amplifiers, I think).




Binning on lines and columns are independent (see crop_rec, which can do 1x3 and 3x1 binning and works by overriding a small part of Canon's settings that are sent to CMOS and ADTG chips).

In particular, for 3x1 (bin 3 lines, read every column), obtained by modifying 1080p, the patches done are:


                cmos_new[2] = 0x10E;    /* read every column, centered crop */

                /* ADTG2/4[0x800C] = 2: vertical binning factor = 3 */
                /* ADTG2[0x8806] = 0x6088 (artifacts worse without it) */
                adtg_new[0] = (struct adtg_new) {6, 0x800C, 2};
                adtg_new[1] = (struct adtg_new) {2, 0x8806, 0x6088};


Register 0x800C is already 2 in 1080p; this change only has effect in 720p. Register 0x8806 appears to reduce artifacts (what's happening here is unknown, but it's not a process that alters overall brightness). Register CMOS[2] is used to enable 3x column binning (a single bit) and to position the scanned area horizontally.

That means, all the configuration that affects column binning is done on the CMOS chip, so it happens before the signal enters ADTG (and before ADC, since ADTG gain was found to be analog).

For 1x3 (read every line, bin 3 columns):


                cmos_new[1] = (is_720p())
                    ? PACK12(14,10)     /* 720p,  almost centered */
                    : PACK12(11,11);    /* 1080p, almost centered */

                cmos_new[6] = 0x170;    /* pink highlights without this */

                /* ADTG2/4[0x800C] = 0: read every line */
                adtg_new[0] = (struct adtg_new) {6, 0x800C, 0};


Here, we patch 0x800C to disable line binning (so, read every line). This register configures the number of skipped lines on cameras before 5D3. Here, 0 = read every line, 2 = bin every 3 lines, 4 = bin every 5 lines and so on. Intermediate settings work too, but I didn't run any noise tests on them.

I did run the noise tests (raw_diag.mo, SNR analysis from 2 images) on 1:1 crop_rec vs 1:1 zoom, and 720p 3x3 crop_rec vs 1080p 3x3 Canon default, and found them to be very similar.

So, the above note at *) was a little incorrect: when binning 3 lines, the signal level that enters in the ADC appears to be slightly higher (in any case, less than 0.5 EV, probably 0.1 or 0.2). So we have to patch an extra CMOS register to fix it (by increasing the level after disabling line binning).




You draw the conclusions.

GammaNu

Thank you all !
I think i have all what i was looking for, and more !  :)
Thanks !