CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.

I can try to implement the FPN fix in the camera too, but speed will be a problem.

[Marsu42]

Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.

Interesting to know, thanks for the information - btw keeping cr2 would improve the general acceptance of the ml dr improvement, still lots of folks out there who aren't comfortable with dng.

Last not least, another reason for cr2 is that except ~+4mb/shot it has no drawbacks in LR/ACR unless you want to use dng-only features (embedded color profile, lossy compresssion) - but saving metadata is much faster since it only writes the sidecar xmp.

[Audionut]

but another cr2hdr-like stop adds a lot of overhead

Only if you want the FPN correction.  Otherwise, it's just like any other Canon file, with reduced read noise.

[Marsu42]

Only if you want the FPN correction.  Otherwise, it's just like any other Canon file, with reduced read noise.

Btw this doesn't concern 6d users, I've never ever seen any fpn on this model (unlike my good ol' 60d).

[a1ex]

Yes, but sadly I can't target just the 6D

BTW, can you post the raw_diag dark frame graphs from 6D? I'm curious to see how the numbers compare with 5D3. Try a few ISOs that you consider relevant.

[Marsu42]

BTW, can you post the raw_diag dark frame graphs from 6D?

... here you go (how does the 6d do?): https://bitbucket.org/Marsu42/ml-pull/downloads/6d_fpn.zip

[Audionut]

6D - 5D3 (ISO 100)



http://www.clarkvision.com/articles/evaluation-canon-6d/index.html

But even more impressive than the high signal and low read noise, is the far better control of pattern noise. The 6D sensor performs well ahead of the 1DX at low ISO, similar to that for the 1DX at moderate to high ISOs, but the 1DX pulls slightly ahead at very high ISOs. The pattern noise of the 6D is overall the lowest I have measured for any Canon camera.

edit:  dual_iso images removed.

[Marsu42]

6D - 5D3 (ISO 100)

I don't quite know what to make of these greyscale images - does it mean that the 6d has nearly no vertical fpn, but some horizontal?

Edit: dual_iso images deleted because they're useless.

[Audionut]

Comparing the 6D/5D3 images above, you can see how the 5D3 has large amounts of vertical FPN.  The 6D is doing a remarkable job of removing the vertical FPN component, but there is still some horizontal FPN remaining.

I believe I read that Canon was doing some FPN correction on the 6D.  Looks like they concentrated on the vertical component.  It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras.  I'm not sure where to start looking though, other then poking registers.  Maybe a1ex has some better ideas, if it's possible, and if you or another 6D user has the time.

It's important to remember that the pattern noise is generally a fixed component (the noise level stays the same, for each individual camera).  And also, that human eyes/brain, are adapt at detecting patterns.  So when it looks like the pattern noise is getting stronger or weaker (on an individual camera), actually, the random noise is increasing, or decreasing.

[a1ex]

If you look at the DARKFPN graphs, the 6D shows:

- Vertical FPN: stdev 0.30 with ratio 0.06
- Horizontal FPN: stdev=0.22 with ratio 0.05
- Overall stdev: 4.59 (from DARKHIST)

At ISO 100, 5D3 has:

- Vertical FPN: stdev 0.85 with ratio 0.13
- Horizontal FPN: stdev=0.21 with ratio 0.03
- Overall stdev: 6.81

These ratios are the FPN stdev divided by overall stdev, that is, how much of the total noise is FPN. I believe it's measuring the subjective effect of the FPN (that is, a low ratio, let's say below 0.05, tells the FPN gets buried into noise, and a high ratio, say above 0.1, tells the FPN is becoming obvious).


Side note: on a 4000x5000 ideal Gaussian, the estimated vertical FPN ratio is 0.015.

octave:16> G = randn(4000,5000);
octave:17> std(mean(G)) / std(G( : ))
ans =  0.015700



Overall stdev:
From raw_diag data, 6D has around 0.57 stops of extra dynamic range at ISO 100, compared to 5D3. To compare these numbers, you need to normalize by white-black, but from the data I have, the white levels are 15331 and 15283, which does not really make a difference.

From Roger Clark, shadow noise levels at ISO 100 are 34.9 vs 29.0 electrons, so 6D is ahead 5D3 by only 0.26 stops. The extra difference in DR may be explained because 6D's ISO 100 captures 79600 electrons, compared to 68900 for 5D3. That's 0.2 stops of extra highlight detail for 6D (so 0.46 stops difference in dynamic range).

FPN:
The FPN ratios are explaining the first two screenshots posted by Audionut above. 5D3 has obvious vertical FPN (ratio=0.13); 6D improves it by one stop (ratio=0.6), so vertical FPN gets buried into noise. But the horizontal FPN is slightly better on 5D3 (ratio=0.03 vs 0.05).

So, considering the FPN improvement, I'd say that, in practice, out of the box, at ISO 100, the 6D is ahead of 5D3 by around one stop.

Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.

5D3 ISO 100 reimplemented from 200 (without black/white level tweaks)

- Vertical FPN: stdev 0.71 with ratios 0.20
- Horizontal FPN: stdev=0.14 with ratio 0.04
- Overall stdev: 3.70

So, the vertical FPN is reduced only a little (0.26 stops), but the Gaussian noise is reduced a little more (0.88 stops), which makes the FPN even more obvious (the ratio gets degraded by 0.62 stops).

And this is why we need a FPN correction tool: we have almost 0.9 stops of noise improvement on 5D3, but in practice it's not very useful because of the FPN. You have already seen the effect earlier in this thread: here on a test sample from Audionut, and here on a downsampled dark frame.

[a1ex]

And now here's a little experiment that reveals the feedback loop from the left OB area, and its time constant. Set ADTG 82F3 (the one that sets top optical bar size) to 0, then run the OB zones analysis, then average the data from the left OB area.



You may cross-check it with this graph and with g3gg0's theory: [1] and [2].

This info should explain the low frequency look of the horizontal FPN and should be very useful for correcting it.

[Audionut]

Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.

Which explains why I wasn't seeing the same FPN reduction that I do in real life images.  In practice, dual_iso does an excellent job of reducing FPN.

ML ISO / ML ISO + dual_iso

[Marsu42]

I believe I read that Canon was doing some FPN correction on the 6D.  Looks like they concentrated on the vertical component.  It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras.

Interesting, reminds me of test shots showing the 6d is slightly softer than the 5d3 - I though this would be due to some forced chroma noise reduction, but maybe fpn correction is (also) the reason. If it's really done in camera by software, this is indeed remarkable as the softening is minor.

Question is: If so, why did Canon concentrate on vertical fpn, maybe because their speed-tuned algorithm only works this way while the image is read from the sensor?

[Audionut]

Question is: If so, why did Canon concentrate on vertical fpn,

That was a poor choice of words on my behalf, sorry.
I have no true understanding why the vertical component is reduced, but the horizontal FPN remains similar to the 5D3.  I assume that there is some relationship to the column amplifiers.

[tjaja]

Offtopic:
Are you guys talking about this? Noticed it in h264 recording and on liveview: fixed noise. On monday Im going to the canon service site in the netherlands and on the phone they told me: It has to be re alligned ( :S :S) or something nothing to be concerned of.

[Marsu42]

Offtopic:

This.

[Audionut]

Pulled Canon ISO 200.



I suspect there may even be a little more then 0.1 EV of nonlinear highlights.  But I need to confirm.

8/9/A/B is defiantly biased towards the highlights, and a strong cause of the nonlinear roll off in the highlights.  This register bank is not really noisy, so if I have over estimated the non-linearity, adjusting these registers back towards default, won't have a large effect on noise (printed DR results).

Adjusting saturate offset on 5D3.123, blanks the rear display.  The camera appears to function correctly, except no display.

Code: [Select]

+    {0xC0F0,   0x819c, 0, "Saturate Offset (photo mode) (HIV_POST_SETUP)"},

[Audionut]

Looks like there is some room for improvement, by adjusting the dual 0xFE registers, individually.

[Greg]

So the gain is more than 1EV? 

[Audionut]

Well, there are issues with large individual differences to 0xFE.



There is a very faint pattern noise.  But here, I was using a difference of 3, between the register values.

Note:  This is a horizontal pattern noise, as opposed to the vertical pattern noise from other registers.

[naturalsound]

Hi,

could somebody maybe send upload me one darkframe and one or two pictures with severe vertical FPN (ISO100 pulled from Best would be a linear TIFF 16bit (maybe even including the OB zones)?

CR2 would be OK also, although I will not be able to get the optical blacks with my current software.

Thanks in advance.

[Luiz Roberto dos Santos]

@A1ex any news about the paper? The community can help you with something, like tests and so on?

[a1ex]

Not really; need to find the time and the mood to sit down and figure out stuff. Pretty math intensive...

But you can try to follow the thread and check our findings for other cameras; that would be nice. Raw_diag and adtg_gui are portable (they run on all cameras, first without modification, second may require minor tweaks) and should be a very good time killer during long train journeys, for example.

[Audionut]

Looks like Canon increased the ISO Digital Gain in 1.2.3.  The histogram has 1 pixel gaps with 1.2.3, and these are not present in 1.1.3.

[Audionut]

Losing the mentality of maximum DR, whites be damned.  I started looking more closely at the histogram with RawDigger.

@ISO 100, the maximum gain reductions I can do while still maintaining saturation in all color channels.
0xFE - 0x3
888x - 0x3ax

And with these tweaks.
CMOS4 - 0x308
ISO digital gain - 0x200

One thing became clear, the blue channel had less gain then the other channels.  On a fully saturated image, the stdevs.

Code: [Select]

    black   white
R   4.50    24.6
G   4.46    32.1
G   4.50    27.3
B   4.42    95.8
There is a lot more deviation at white in the blue channel, due to it only just hitting saturation.  From my research here, it's clear that ADTG2 is affecting G1 and B channels, and ADTG4 is affecting G2 and R channels.  So, reducing ADTG4(888x) to 0x38x resulted in these numbers.

Code: [Select]

    black   white
R   4.33    89.1
G   4.50    32.3
G   4.34    89.5
B   4.41    95.8
Dynamic range in the R and G2 channels increased by 0.05 EV.  All channels still saturated.