CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Do you have a tool that can plot a signal-noise graph like this?



I did this one a while ago by taking many pictures of a blank wall at different shutter speeds. I know darktable guys did such graphs from a single image, but they plotted it on linear axes, which is complete nonsense (they buried the midtones and shadows into a tiny corner and their graph covers well only the highlights).

If not, I can write a module that plots this kind of graphs directly in the camera, from a defocused test picture.

[SpcCb]

White level in LiveView is set to 15000 as a one-size-fits-all code. If the true white level is higher than that, it can't be wrong by more than 0.14 stops. If the true white is 15600, the error is roughly 0.07 stops.

1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.

You can always squeeze this extra DR with exiftool.

Thanks for the tips, I'll see how to do that.

Actually I'm trying to produce a 'very' RAW for scientific imaging, astro-photography or other special imaging by record a RAW with the minimal operations done on. It means the best bias/offset matching for the RON + FPN sigma at the optimal analogic gain etc. (no multiple ISO choice in this case so no way for regular photography). I don't fully understand how to use datas in ADTG datasheet yet, but it looks we are just at the beginning of what we can do with ML!

If you want I can do test on 5D2, just ask.

[a1ex]

1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.

1792 is Canon's choice.

[SpcCb]

I've been trying to figure out how increasing the black level via that register is keeping a similar/smaller Standard Deviation.
(...)

Over all sigma ADU values on a short exposure is ≈ RON + FPN + offset.
So if offset > sigma/2 ; sigma will always be the same. You can change as you want the pedestal, it will not affect sigma.
And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).
(I supposed a symmetric deviation for sigma here, to be simple)

[SpcCb]

1792 is Canon's choice.

I confirm not.
Canon offset is 1024 ADU on 5D2.
Here is a RON + FPN profil:

[a1ex]

And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).

I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.

[a1ex]

I confirm not.
Canon offset is 1024 ADU on 5D2.

LiveView is not the same as photo mode.

[SpcCb]

I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.

Indeed, I speak about analogic bias. We are OK.

LiveView is not the same as photo mode.

Ah, I did not know there's specific pedestal config in LiveView. Au temps pour moi.

[a1ex]

Okay, but the bias is around 1024...2048 and sigma is around 8.

I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.

[poromaa]

interesting!

[SpcCb]

(...)
I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.

Indeed again, as I said there's a relation between all these parameters.
The optimal config is easy to gess, on the paper, but hard to implement. I'm stuck at this point right now.

In a scientific view of all of this, the best choice will be:
_. the lower pedestal value where RON + FPN sigma is contained therein
[current original Canon bias is too high, because have to fit with all ISO; for example with 5D2 it could be N-times lower for ISO where gain <1 (in my graphic just up you can see it could be ~10x lower)]
_. the more closer analogic gain than 1
[but in real world and for usual photography, gain has to be <1 to equiv low ISO sensibility, so we should use a matrix for all ISO/gain/bias optimal values]
_. the lower digital gain
[in fact if we could have a matrix for all ISO up to gain <1 where DG is null it will be optimal, as it done for audio signal]

Please not that I ever do most of these optimizations in post for my work, I could demonstrate/detail/explain parts if need.

[Audionut]

Do you have a tool that can plot a signal-noise graph like this?

I've been manually entering the numbers in a spreadsheet and using it to generate the graphs.

[SpcCb]

gnuplot is useful for that.

[a1ex]

Made some progress in understanding how ISO 160x are implemented on 5D3 photo mode. This does not apply to LiveView / raw video.

I've looked at what DIGIC registers get changed between ISO 160 and 200 (you remember them from the good old digic poke, FPS override and all sorts of funky effects), and found this one: 0xc0f0819c = 0xC4C for full-stop ISOs and 250 multiples, and 0x8F0 for 160 multiples.

Code Select Expand


                                        black    stdev    white   DR
ADTG gains default:
ISO 160:                                2047     5.42     13307   11.02
ISO 200:                                2047     6.86     15283   10.91
ISO 200 with 0xc0f0819c = 0x8F0:        1187     6.80     15283   11.02
ISO 160 with 0xc0f0819c = 0xC4C:        2734     5.50     13307   10.91

ADTG gains changed to 0.85 x default (right above the point where white level begins to decrease):
ISO 160:                                2047     4.56     13307   11.27
ISO 200:                                2047     5.71     15283   11.18
ISO 200 with 0xc0f0819c = 0x8F0:        1187     5.71     15283   11.27
ISO 160 with 0xc0f0819c = 0xC4C:        2734     4.61     13307   11.16

Changed 8880 to 0xb04 to bring the black level back to 2047:
ISO 200 with 0xc0f0819c = 0x8F0:        2047     5.88     15283   11.14


You draw the conclusions.

[SpcCb]

Is it possible to set custom value for 0xc0f0819c ?
Ex.
ISO 200 with 0xc0f0819c = 0xAA4 => offset = 1024 ADU

[a1ex]

Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.

When changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes). Also, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.

[SpcCb]

Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.

Nice!

When changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes).

Agree, the overall brightness should not change.
In the same time pseudo-blacks should be less noisy because of lower RON levels and DR should be increased because of larger range between RON and max level.

Also, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.

Yes, but it's better to set that by analogic, at the source. Numerical post optimization with a single source frame is equivalent to stretch DR etc. Some informations are not record in the source so can't be restored after.

[Audionut]

Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.

Changing 8880 to 0x0 produces this.


Clearly, it's having a bigger effect through the exposure on RB then G.

Agree, the overall brightness should not change.

Most software has enough controls over exposure, I don't see it would matter.

[a1ex]

That's why I need a tool to plot the entire SNR curve, not just the black/noise/white levels.

[Audionut]

I couldn't find anything with a google search.  The only one I know of is imatest, but it isn't cheap.

The trial version might be enough for what you need.

[ilguercio]

I got my 6D and digic poke, if you need some data i can try and provide you some figures.
Just let me know the procedure.

[a1ex]

Yes, the last graph from their page is what I'm looking for. If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.

So, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):



The histogram pattern does not change with either ADTG gains or SaturateOffset.

@ilguercio: for the beginning, you can start here and see if you can follow the discussion and arrive at similar numbers. So far, we are just tweaking parameters kinda blindly.

"If we knew what it was we were doing, it would not be called research, would it?"

[ilguercio]

Damn, i haven't had a compiling environment since i bought this new laptop.
I'll try to get it running tomorrow and meanwhile i can try to understand what is exactly involved.
I need to learn all this stuff, so far what seems to be the trick?

[Audionut]

http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/

Because photon noise is an inherent property of CCD signal detection, which cannot be reduced by camera design factors, it essentially represents a "noise floor" that is the minimum achievable noise level, diminishing in relative effect as photon flux increases. Consequently, it is desirable to operate an imaging system under conditions that are limited by photon noise, with other noise components being reduced to relative insignificance. Under low illumination level conditions (assuming dark noise is essentially eliminated by CCD cooling), read noise is greater than photon noise and the image signal is said to be read-noise limited. The camera exposure time (integration time) can be increased to collect more photons and increase SNR, until a point is reached at which photon noise exceeds both read noise and dark noise. Above this exposure time, the image is said to be photon-noise limited.

[SpcCb]

(...) If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.

Yes, it's better to run tests on a plain surface without shape because all variations will be include in sigma.

So, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):
(...)

Useful tool! (I dream to make tools like that!)
We can well see how is the population around the 2048 ADU pedestal. Is the vertical scale is the same?
If yes, for ISO 160 it clearly shows less population. Some vergences too; RON harmonics. I'll see to publish a RON map, sometimes it's more clear with grfx.

http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/

It confirms what I explained posts ago.