CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Vertical scale is "to fit", obviously, sum(all these bars) = 100%

If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.

[Audionut]

Also, the standard (0x8882-0x8888) values seem to vary slightly on startup.  Haven't nailed down what causes the changes. 

Photo mode shows

Code: [Select]

ADTG2-8882  0x41c
ADTG2-8884  0x41a
ADTG2-8886  0x41a
ADTG2-8888  0x41c
ADTG4-8882  0x41a
ADTG4-8884  0x41a
ADTG4-8886  0x41c
ADTG4-8888  0x419
Exposure does not have any effect.  Register values always remain the same.  Whether the values change during exposure and reset after exposure is beyond my capabilities.  ISO does have an effect however, and I've plotted the values and put it into pastebin cause I suck at doing it with the forum code tag.

http://pastebin.com/jtND2EJd

In photo LV, there is a pattern, and the crux point seems to be EV 5 (ISO 100).  ie:  1/20s f/1.4 produces one set of register values and 1/15s f/1.4 (EV 5) produces another.  This is reproducible by changing aperture.  So above EV 5, the register values retain their values, below EV 5 the register values change in a 1 stop pattern.  ISO produces the same effect as a 1 stop change in shutter/aperture.

I've plotted that here.  ISO 100 results = ISO 100 - 1/30s - f/1.4

I've labelled the changes with ISO, but as mentioned 1 stop with shutter/aperture produces the same.  So for instance, you get the same stated ISO 100 results with ISO 100 - 4s - f/16.0 and with ISO 12800 - 1/3000s - f/1.4

This is with 1/2 stop exposure settings in Canon menu.  Interestingly, the crux point shifts with 1/3 stop exposure settings.  Here, 1/20s f/1.4 produces the change, and it sticks to a 1 stop pattern from there.

Notice how ISO 800 produces lower register values then ISO 400 in both photo mode and LV.  Wonder if this has something to do with unity gain? 

I've been of the opinion that the register values translate to gain.  ie:  lower values = lower gain.  And this makes sense when you consider the values rise with increasing ISO.  This wouldn't appear to be the case though considering the values also rise with increasing exposure.

ISO 12800 is the last ISO to make a change.  In the previous pages I should have labelled ISO 3200 as the last useful analog ISO.

So, I started to write my own module.

Nice work.  I can plot the same with RawDigger and output CSV files containing the values like so,


https://dl.dropboxusercontent.com/u/34113196/ML/Noise%20samples/_46A5329-rawdata.csv
Obviously, this way I have to remove the card, load it into Rawdigger, etc, etc.  Note:  That's my dodgy graph!

[a1ex]

http://pastebin.com/jtND2EJd

Same values here, +/- 3 units.

In photo LV, there is a pattern

This is called ExpSim. LiveView normally is limited to 1/30 exposure time.

Since the ADTG gain does not change the histogram pattern, and the histogram is mostly without gaps even with this gain at 3x the original value, I believe this gain is analog amplification. However, bumping the gain at 5x does not result in lower noise:

stdev at 1x: 6.71
stdev at 5x: 34.02

[Audionut]

This is called ExpSim.

Well that's disappointing.  For a moment there I thought I was smart!

[a1ex]

My current theory about 5D3 ISOs:

I believe these parameters are relevant (there may be others):
- ADTG gains (0x8882 ... 0x8888)
- ADTG black (0x8880)
- SaturateOffset (0xc0f0819c), related to ADTG gains
- 0xc0f37ae4, af0, afc, b08 (digital gain - this one changes the shape of the histogram)
- c0f37ae0/aec/af8/b04 (offset for black and white levels)

The last two registers can be used to configure the range of recorded data (Canon default is from 2048 to a little over 15000 for full-stop ISOs). Changing them will only influence the round-off errors. No big deal.

ADTG black is uncertain, I'll leave it unchanged for now.

ADTG gains and SaturateOffset can be used to recover some more highlight detail. They are tightly coupled (you can recover the same highlight detail from one, from another, or split between them). So, changing only ADTG gain is enough (because the other one runs out of range much quicker).

At some point, there's no more highlight detail to recover. When this happens, the white level will begin to decrease. Let's call this the sweet spot, and it can be found easily with binary search, for example.

Now, remember the ISO definition from DxO: ISO is all about the exposure necessary to saturate the sensor. Since we could change the clipping point to capture more highlights, we can effectively got some lower ISOs. They are just as noisy in shadows as their full-stop counterparts, but they include a little more highlight detail.

So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).

To confirm the formula, I've checked Canon ISOs (starting from ISO 50):
ISO 100, 100, 125, 185, 200, 250, 375, 400, 502, 750, 800, 1005, 1495, 1595, 2008, 2993, 3188, 4015, 5988, 6378, 8035, 12008, 12790, 16110, 20295.
After this, the digital gain register gets changed from 0x200 to 0x10200 and my formula shows 12790, so... 25580.

Take it with a grain of salt, I need to double-check the math and confirm the results with some test shots. I plan to post some code to play with, and some more details.

[Marsu42]

My current theory about 5D3 ISOs

It would be terrific if ML could result better dr/noise than Canon, this would be the ultimate killer feature - of course only if it makes its way to the 6d :->  ...

... I very much suspect that something can be done about this since the 1Dx has a nearly linear dr/iso curve, and Canon most likely doesn't use a different sensor tech on that model - instead the theory on CR is that they applied a high amount of fine-tuning to get this difference vs. non-1d cameras.

[Greg]

500D default values :

ISO             100    200    400    800    1600    3200    6400    12800
ADTG3[0]    149    149    148    148    142      142      142      142
ADTG3[3]    14a    14a    14a    14a    145      145      145      145

[1%]

of course only if it makes its way to the 6d

Its very similar to 5DIII so it should. Hopefully it makes it to all cameras.

[RenatoPhoto]

Very, very interesting.  Nice work guys!!  As I said before you guys should be paid by Canon R&D department!  Maybe in Bitcoins? 

[hjfilmspeed]

So is there any truth in this http://www.clarkvision.com/articles/iso/index.html

[Audionut]

Contrary to popular belief, ISO is post exposure gain.  Exposure is the light hitting the sensor which is a result of lens diameter, shutter and aperture.

So by having a lower gain from the electronics,

So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).

Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure).

So even though the new ISOs,

are just as noisy in shadows as their full-stop counterparts

We can ETTR a little more.  Which means we can increase the SNR of the shot noise.

[SpcCb]

(...)
If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.

This histograms show the RON.
It's why you don't see a gaussian random noise distribution.

I'm a bit busy right now but in a couple of hours I could publish some materials.

[Audionut]

This histograms show the RON.

Yes, but the interest is in the distribution.  Specifically, the effects by digital manipulation.

The histogram pattern does not change with either ADTG gains or SaturateOffset.

a1ex called it pattern.  The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain.  If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.

[a1ex]

Code: [Select]

octave:1> noise = round(randn(100000,1) * 7); # Gaussian noise of similar magnitude as Canon's ISO 100
octave:2> noise_positive_gain = round(noise * 1.1); # apply a small positive digital gain
octave:3> noise_negative_gain = round(noise * 0.9); # apply a small negative digital gain
octave:4> hist(noise_positive_gain, -30:30)
octave:5> hist(noise_negative_gain, -30:30)


[Audionut]

Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure)........................We can ETTR a little more.

a1ex showed me how to calculate exactly how far we can ETTR with these new ISOs vs Canon base ISOs.  Which is obviously better then random guesses from histograms.

Code: [Select]

100 0.377 EV
200 0.415 EV
400 0.439 EV
800 0.427 EV
1600 0.445 EV
3200 0.497 EV
6400 0.554 EV
12800 0.587 EV
Not only can we safely increase the exposure by 1/3, there's also some headroom.

edit:  This is over and above the DR gains from the tweak.

[ilguercio]

Half a stop at 12800? That's goooooood.

[Audionut]

Half a stop at 12800? That's goooooood.

It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.

[ilguercio]

It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.

I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should

[Marsu42]

I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should

I dunno, given the iq of the 6d using iso ~200 is very good esp. if ML adds a little more dynamic range than to iso 100, up to iso 400 the read noise halves which more or less drowns the dynamic range. With longer exposure time (larger dof for ~macro or bad lighting) in the seconds my observation you're getting another noise source, so raising the iso a little might not be a bad idea.

Half a stop at 12800? That's goooooood.

Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800? The dynamic range 6400->12800 drops by 1 ev :-p

[Audionut]

Shot noise, a significant contributor to total noise, is the square root of the number of photons hitting the sensor.

So noise actually increases with increased photons, but it's the SNR that's more important, and this also increases with increased photons hitting the sensor.

ISO doesn't increase the number of photons hitting the sensor.

ISO is useful in Canon cameras, because the ADC is noisy.  So where you can't saturate the pixel wells with exposure settings (shutter/aperture) and generate a large voltage from the sensor, you can use ISO to apply gain to the signal to boost it to the saturation point of the ADC, and hence, increase the SNR through the ADC.

[a1ex]

Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800?

It's analog push, not digital. On 5D3, from 1600 to 12800 you gain 0.22 EV in shadow noise and for this you pay 3 stops of highlight detail.

On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).

Some quick results from 60D, assumming full-stop ISOs are real:

Code: [Select]

Base ISO        With low ADTG gain       Highlights gained
100             88                       0.18 EV
200             158                      0.34 EV
400             311                      0.36 EV
800             633                      0.34 EV
1600            1260                     0.34 EV
3200            2515                     0.35 EV
6400            2515                     1.35 EV

(note that ISO 100 on 60D is implemented more like 160 than like 200 multiples; I didn't check saturation level yet)

[PressureFM]

Have anyone checked the 5D Mark II?

Or how would I do it?

[a1ex]

In theory, simply follow the discussion from the beginning (you need code compiling skils).

In practice, I could not find the equivalent registers on 5D2, but I've only tried for around 2 hours.

[dmilligan]

On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).

You're assuming that Canon is not doing something delibrate to hide the fact that it is actually a digital gain. I do not give Canon that much credit. If you scale by 4x from ISO1600, then dither (i.e. apply gaussian noise to) the two LSBs, your histogram will still look the same, it will not have gaps.

[a1ex]

At ISO 160 multiples, the histogram does have spikes (including ISO 1250). So if they were trying to hide it, I guess they would have done it at these ISOs too.