CMOS/ADTG/Digic register investigation on ISO

[Audionut]

Blue is blue channel, red is red channel, green is green channel.

Yeah, that makes sense!

I assume the 2 banks of column gain registers (ADTG2/ADTG4) are for the dual amps that dual_iso uses, correct?  And ADTG6 looks to be the feedback amp?

I only ask because cr2hdr reports a low DR (about 0.2-0.3 EV to low) for the recovery ISO.  Looks like we are getting 0.6 EV of overlap too

[a1ex]

Dual ISO only changes the CMOS[0] register. But, of course, it benefits from the other tweaks too.

ADTG6... I don't know yet what it is, did you notice any effect by changing it?

[Audionut]

Looks like ADTG6[8880] 0x750 gives 0.02 EV.  Every little bit counts right?

Otherwise, I haven't found anything useful yet.

[a1ex]

Dynamic range series (ISO 100-12800 in Canon menu, ADTG gain at sweet spot, ADTG2/4[8,9,A,B] at 36 for ISO 100 and 0 for all other ISOs, CMOS patches enabled.

Code: [Select]

ISO    DR
66     11.589 + 0.1
115    11.772
230    11.649
460    11.427
912    10.968
1750   10.369
3333   9.610
6400   8.834

I've added 0.1 to the measurement at ISO 100 because of the nonlinear highlight rolloff (approximate). The other ISOs seem to clip harshly, the difference between median and clip ISO is smaller, so I think we can consider the response linear.

Comparing with my older measurements:



Note: the ISOs may be approximate. As I said in the dual ISO thread: I'm an engineer, for me, Pi is 3.

[Stedda]

Most of this is way over my head but from the looks of that chart you guys have hit on another incredible ML feature!!!!

[Audionut]

11.77 EV is remarkable.  I don't imagine it is going to be possible to modify the ADTG registers for dual_iso.

Based on a1ex's ISO data above, and the measured ISOs from DxO, the true ISOs should be.

Code: [Select]

Canon Menu ML
100 53
200 92
400 186
800 369
1600 729
3200 1377
6400 2697
12800 5042

[Audionut]

With new ADTG gain at 0x0, I haven't noticed any highlight banding.

Consider these histograms.

Canon
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/Canon.png

ML with ADTG gain at -0.37 EV, new gain at 36 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML36.png

ML with ADTG gain at -0.37 EV, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.png

We still gained highlight detail with new gain at 0.

[a1ex]

So, we have ISO even lower than 66?

Will take a closer look.

[Audionut]

It looks like it from here.  I trust your closer look more

If I am reading the histogram right, here is ISO 200.  Here we lose some highlight detail compared to ISO 100.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO200.png

Then we start to gain some more highlight detail through every higher ISO.  Here at ISO 6400.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO6400.png


Both shots, ML with calibrated ADTG gain, new gain at 0 and CMOS patched

Look at ISO 100 with calibrated ADTG gain, new gain at 36 and CMOS patched
vs
ISO 200 with calibrated ADTG gain, new gain at 0 and CMOS patched

We see a gain in highlight detail that matches ISO 200 through ISO 12800.
ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched seems to be applying a large amount of highlight roll off.  Need to check if there is actual detail right near saturation, or Rawdigger is struggling to accurately report details.

edit:  I'm probably just seeing increased stdev with increasing ISO, not extra highlight detail.

[a1ex]

Here's the banding I'm talking about:



(ISO 100, new gain = 25, white level forced to 15000 when developing)

So, it looks there is some more detail, but I think it would require some extra postprocessing (this signal is not exactly clean, has strong trend and banding noise). My settings were conservative, with clipped white being clean white.

By trial and error, I changed the white level for this shot from 15282 to 14682, and at that point the highlights were clean (well, just a tiny bit dirty). On the reference shot (the one underexposed by 1 stop), I changed the white level until the clipping area was the same (that was 12282). So, the input signal required to saturate the sensor at 1/25 was log2(15282-2048) - log2(12282-2048) = 0.371 stops weaker than the one required to saturate it at 1/50.

This means our ISO got lower by 0.629 stops, that is, ISO 64.7.

QED

[Audionut]

Whites are fine here, but there is a very faint pattern noise through the exposure, with new gain at 0x0 ISO 100.  White level - manual lens (15283).
The pattern is that faint, these images needed to be full resolution to show the effect.

ML ISO 100 with calibrated ADTG gain, new gain at default and CMOS patched
2.23 MB
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML_no_new_gain.jpg

1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg

Looks to me (my eyes), that there is actually less read noise with new gain 0.

[Greg]

1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg

It has vertical stripes.

[Audionut]

Nevermind.  One day I will learn maths.

[l_d_allan]

I enjoy just sort of watching this thread utterly without comprehension. Like a cat watching TV.

I'm in a similar state of non-comprehension. Over my head. But fascinating.

But ... is the following something of a "bottom line" and "rubber meets the road" ?

• If this ADTG tweaking works, ML equipped cameras would have the capability of using a "for real" lower ISO such as ISO 65 that would have lower noise and greater dynamic range than the "native" ISO 100 without ML.
• The increase in DR could be on the order of 0.25 to 0.50 EV?
• I'm not clear is this works with Dual-ISO
• I'm unclear if other ISO's are potentially improved, like something between 100 & 200?
• It wouldn't be like the existing ISO 50 that is kind of the equivalent of a neutral density filter, with reduced DR and no real improvement in IQ.
• Or am I confused and "unclear on the concept"
• My other less-than-informed impression is that there would be a conservative ADTG default value that would improve DR, and a in-camera tool that would allow tweaking of an individual camera to optimize DR (sort of like Auto-Dot-Tune)
• I'm fuzzy on whether ISO 100 improves, or remains the same?

[a1ex]

Look at the graph from this page, and at the FAQ from first post.

Generally speaking, by applying some of these tweaks you get some more highlight detail (which means a lower ISO, usually with some more DR).

With the latest findings, the improvement is nearly 0.8 stops on 5D3, ISO 115 pulled from 200. So it's not exactly ISO 100, because it will clip the highlights a little earlier, but I expect it to have 0.8 stops of less shadow noise. Didn't pixel-peep anything, only did the math with open source algorithms that you can try and review.

Dual ISO will be improved mostly on the highlight side (from ISO 66) and on the overlap amount (less aliasing - minor improvement). On the shadow side, at high ISOs most of the noise is photon shot noise rather than electronics noise, so there isn't much room to improve (we are close to the physics limit if I understand well, feel free to point me to a better explanation).

I've tried to exploit the nonlinearity by cleaning up the vertical banding and have barely reached ISO 60, so I don't think it's worth the hassle.

The registers on 6D are pretty much identical to 5D3, so you can fire up ADTG GUI and try these tweaks (you will need a custom compilation, it won't load on stock nightly).

[Marsu42]

The registers on 6D are pretty much identical to 5D3

Thank heaven for that :-p ... looking forward to a mini_iso with working white point casting to try :->

[a1ex]

Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

[Marsu42]

Ah, right, will do - but you've already got 6d iso.log from 1%, so you're only lacking av.log? Would you prefer having it with aperture f8 to f2.8 (my fastest lens) or with shutter that covers the ev range f8 to f1.2?

[a1ex]

It's OK (better than nothing). For white level, the aperture changes are important (not shutter).

Meanwhile I've got logs from 550D from LebedevRI (uploaded to https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW ) and also Nanomad tried to get them from 650D/1100D with only partial success (not yet sure why, probably wrong stubs, the birthday paradox issue from cache hacks, or simply Canon code does no longer calls what I expect it to call).

[Marsu42]

It's OK (better than nothing). For white level, the aperture changes are important (not shutter).

Well, if someone 6d-ish has to do it again in any case like 1% with his famed aperture gadget, I don't need to do half of it unnecessarily - or will the f8-f2.8 log be sufficient make the mini_mo usable for the 6d? As written before, the white point option (like acr 15k) doesn't work atm.

[a1ex]

I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts

[Marsu42]

I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts

Yes, but I'm still idling at work so writing is easy, when back at home and doing braindead test series it's my spare time I rather value and only invest if there's some roi (like a working 6d mini_iso) :-o

[Audionut]

I'm glad a1ex places ML as a high value of his spare time.  In fact, I am highly appreciative. 

[Marsu42]

I'm glad a1ex places ML as a high value of his spare time.

Obviously we all are as we all are using his free work, and he knows I cannot thank him enough for doing ML and not ending up as a complete nerd but very nice to communicate with.

Personally, I also appreciate a sound work-life balance, whatever that may be... I'm neither applauding self-exploitation (*not* directed towards alex) nor am I entering a contest for the most altruistic man alive, been there, done that, found it to be very unhealthy and being evicted from your flat for missing rent also doesn't help coding efforts.

[a1ex]

The new gain seems to be expressed in EV. At first sight, 1 unit seems to be around 0.006 EV.

Quick data set (only changed these gains, nothing else; started from ISO 100; the diffs are the median exposure difference printed by latest raw_diag, in EV):

Code: [Select]

gains = [0    10    20    30    40    50    60    70    80     90     100    150   200   256 ]
diffs = [0.66 0.70  0.79  0.84  0.89  0.94  1.01  1.06  1.13   1.20   1.25   1.56  1.84  2.17]

This means, in order to keep the factory calibration, simply translate the register values by some constant amount (just make sure you don't go below zero). This should fix the vertical banding (didn't try, it's just theory, let me know if it works).