CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Out of curiosity, I tried two dual ISO shots to see whether the theory applies. Since the new trick only improves the highlights side, the theory says you will only benefit from the DR improvement at ISO 100 (because in shadows, the modified ISOs are just as noisy as before).

I had to update the white level detection routine in cr2hdr to consider two separate levels for bright and dark exposure. I'll only post the relevant lines from the conversion log. The subject was boring (a light bulb).

ISO 100/1600:

Code: [Select]

White levels    : 15094 15109
Noise levels    : 11.05 6.48 6.52 11.08 (14-bit)
ISO difference  : 3.97 EV (1572)
Semi-overexposed: 9.47%
Deep shadows    : 83.80%
ISO overlap     : 4.0 EV (approx)
Dynamic range   : 10.98 (+) 10.21 => 14.19 EV (in theory)
Dynamic range   : 14.35 EV (cooked)

ISO 77/1230:

Code: [Select]

White levels    : 15096, 15108
Noise levels    : 8.60 5.13 5.03 8.54 (14-bit)
ISO difference  : 3.96 EV (1559)
Semi-overexposed: 6.06%
Deep shadows    : 84.38%
ISO overlap     : 4.4 EV (approx)
Dynamic range   : 11.35 (+) 10.58 => 14.54 EV (in theory)
Dynamic range   : 14.68 EV (cooked)

Here, "in theory" means from analyzing noise level in input exposure, and "cooked" means right before saving the output file (might include some noise reduction, interpolation artifacts and whatever other tricks I'm using).

So, the DR improvement in this test was 0.33 stops (on the highlight side).

What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.

[Marsu42]

Audionut has sent me the corrected 5d3 stats, I changed them on my CR post (I didn't think this would make such a large splash, no ML development I have ever crossposted there has :-o)...

... one question that puzzles me: The DR for iso 800 is the same ML vs. Canon, what's this about? Also, did you do any recent changes that also invalidate the 6d calibration done by 1% this effect is also there?

[a1ex]

After changing the reference for the SNR graph to be able to compare the curves as in ejm's article, I hope to find out. Right now, I expect something like 1/100 ISO 100 might be as clean as 1/640 ISO 640 pulled from 800, but I did not run any tests to confirm this.

The calibration is still valid. You may want to fine-tune it for your camera, but I guess it will stay within +/- 0.05 stops from the defaults.

[Levas]

just for my curisoty, (notting else, I can wait a long time for this coming to the 6d, take your time  )
Since the 5d3 and the 6d are the same generation, sensor and digic wise.
How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?

[Marsu42]

How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?

Wrong thread, but short answer: They seem to be very similar, but concerning ML the 6d is unmaintained and in an earlier state than 5d3, so features might make it to the 6d or not.

[Audionut]

I correctly exposed @ ISO 100 and then boosted ISO to 1600.  Loaded all images with same ACR settings (WB from ISO 100 shot, no noise reduction etc & highlight slider -100) and brightness matched from the black patch top left.

My ADTG settings vs Canon vs Canon -1/3EV with shutter.

What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.

Indeed, on top of the 0.3-0.5EV less shot noise.  Because it would be pointless to pull the highlights back if you don't then expose for them. 

If you're DR limited then you gain 0.3-0.5EV of latitude.  You can either gain that advantage in the highlights or the shadows.

[Escaperoute]

I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect
.

[ayshih]

For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine.  I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).

Code: [Select]

    else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
    {
        ADTG_WRITE_FUNC = 0xFFA11FDC;
        CMOS_WRITE_FUNC = 0xFFA12190;
        ENGIO_WRITE_FUNC = 0xFF97D904;  // from stubs
        ENG_DRV_OUT_FUNC = 0xff97d794;
    }


[Audionut]

I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect

Not reading the last page of a thread and the OP, is just bad form.  Surprisingly enough, both of these places have the answers to your questions!

[Audionut]

Adobe doesn't have a problem with the black level, which is only offset from 0 to account for stdev.


edit:  This one doesn't scale well with ISO.

[a1ex]

Added another analysis type to raw_diag: fixed-pattern noise (banding) from a dark frame (image taken with lens cap on). FPN is estimated by averaging each line (or each column), and the graph shows a plot of this correction "signal" and its stdev. It may be useful for testing the custom configurations from Audionut (since one of them attempted to reduce exactly this FPN).



(I've showed earlier in this thread how to use this info to subtract the FPN a dark frame)

Source and binary in first post.

Some autocorrelation analysis would be nice, if anyone feels like doing the math.

[SpcCb]

Nice a1ex!

(...)
Some autocorrelation analysis would be nice, if anyone feels like doing the math.

I have some pretty nice FPN reduction algorithms, maybe it could help (maybe it is too gourmand for the DIGIC...).

[a1ex]

I'm interested. Especially if you can estimate the FPN from an arbitrary image without extra dark frames.

[SpcCb]

Indeed, without master offset.
It's too big to post the code here, I send you a PM.

[Audionut]

Canon vs My tweaks









FPN should be reducing with increasing ISO.

[ayshih]

Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).

For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624.

On my 50D, the white level at ISO 100 is only 13432, so I can adjust those same registers to expand the range from 1023..13432 to 31..16383, which gives an apparent DR increase of 0.4 EV.



I increased B/W Offset (0xc0f08034) from 0x1079 to 0x1c00 to max out the white level, and decreased Saturate Offset (0xc0f0819c) from 0xf87 to 0x20.  (I just now realized that these are essentially the same settings you cited for the 5D2.)  I find it curious that not only is this setting for B/W Offset "nice" as a multiple of 256, it also shifts the meaning of the Saturate Offset to correspond to the actual empirical black level (0x20 ~= 31).

[a1ex]

Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)

[a1ex]

Just noticed that on 5D2, at ISO 3200 vs 6400, raw_diag reports exactly the same values for noise and DR (and raw zebras agree). However, the CR2 at ISO 6400 contains clearly less highlight detail, is brighter, has larger file size, and its histogram has gaps: http://www.guillermoluijk.com/article/isos5dmkii/index.htm

Could it be a multiplication of the raw values directly at CR2 compression stage?! (in any case, after ML intercepts the raw buffer)

[Greg]

500D (default) :


500D (adtg 149 -> 100) :


500D (adtg 149 -> 0, bad white level) :

[a1ex]

5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

Results:

Code: [Select]

Canon ISO    ML ISO    Improvement
100          82        0.29 EV
200          165       0.27 EV
400          332       0.27 EV
800          692       0.21 EV
1600         1432      0.16 EV
3200         1432      1.16 EV

Forgot to mention that new ADTG GUI also has an option to group registers by task or program counter where the register was changed. With this trick, you should be able to identify the ADTG registers on EOS M (look for those in ShootCapture task or something similar, not for the LiveView ones).

[ayshih]

Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001.  Here are the CR2 files for default settings and tweaked settings.

I haven't looked too closely at the CR2 files, but here's a quick comparison of their histograms (with the channels lumped together).  What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.

(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)

At ISO 100, the default value of digital gain (0xc0f08030) is 0x1000, so I did not change it.

[Audionut]

On 5D3:

Digital Gain is white level control.  Does not effect black.  Does not effect stdev.  Sweet spot is 463 with pushed black/white levels.  Relax the contrast (black/white levels), and values down to 461 will still deliver a Gaussian bell curve.

ADTG gain.  Not sure.  Sweet spot (for ISO 100 and modified Saturate offset) is 1002.  You can lower this value much further before affecting white level, but values lower then 1002 effect the bell curve.  This one effects stdev.

Black Reference.  No Idea.  Agreed a1ex, Canon seems to have set this at sweet spot.  I haven't been able to find any other values that deliver a bell curve.

B/W offset.  Gain, this one shifts the entire signal.  Tied to Saturate offset.

Saturate offset is black level control.  Does not effect white.  Tied to B/W offset.

Both offsets can deliver a Gaussian bell curve at different levels.  Note here where I've changed black level with Saturate offset (fixed white level).



And here where I have shifted the signal with B/W offset.



There are sure to be other points for both controls where the histogram retains it's bell curve.
Note how stdev remains within shot to shot variation for all examples.
With Saturate offset at the minimum value (118) that still produces a bell curve (black level 574), I can push B/W offset to produce black levels 999 and 2048, matching the levels where saturate offset produced a bell curve with fixed white point, even with white point pushed way past 16383. 

CMOS/ADTG gains are clearly analog, they effect noise. 
Digital gain, B/W offset and Saturate offset seem to be digital level controls.  Probably only subject to quantisation errors.  They also clearly have defined points where they work well together (produce bell curve), and Digital Gain seems to only have 1 sweet spot (subject to contrast).


Suggested settings.



These produce maximum contrast and Dynamic range, with white level recorded at nearly 14bits.  White level does not change with ISO.  At higher white levels, high ISOs will push the WL even further.


Left to confirm. 
What programs, if any, respect high white levels.  They should, what's the point in having 14 bits when you don't use them all 
ADTG gain needs to be adjusted on a per ISO basis, find correct values.

[Audionut]

Adobe respects the white level.




Also, to high ADTG gain value.


And to low ADTG gain value.


Note the change in histogram.

[SpcCb]

5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

(...)

Code: [Select]

Canon ISO    ML ISO    Improvement
100          82        0.29 EV
200          165       0.27 EV
400          332       0.27 EV
800          692       0.21 EV
1600         1432      0.16 EV
3200         1432      1.16 EV

(...)

Well done!

By seeing this, with an optimized pedestal, max WL and a good FPN reduction I think we can reach 12 stops, maybe 12.5 stops DR on the 5D2 for daylight (short exposure) photography.

[a1ex]

(moved to next page because forum can't handle so many graphs)