CMOS/ADTG/Digic register investigation on ISO

Started by a1ex, January 10, 2014, 12:11:01 PM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

a1ex

Remapping the recorded range has nothing to do with DR, they are decoupled.

... except maybe for some round-off errors :P

Marsu42

Quote from: a1ex on January 27, 2014, 10:55:01 PM
Remapping the recorded range has nothing to do with DR, they are decoupled.

Ok, thanks for explaining, in this case why not stuff the data in the original range that is used w/o mini_iso to avoid any compatibility problems ... or am I still mis-understanding what is going on? If I want to try to debug any regressions with acr, I should at least have a vague idea :-p

a1ex

1) At larger apertures, Canon amplifies the raw data digitally (with the digital gain register), which works by changing the recorded range. That means, a raw processor that autodetects white level will not have any side effects (except for f1.4 and f1.2 where the white level overflows above 16383), but a raw processor that assummes a constant white level (most common ones I think) will effectively interpret this as a higher ISO (and throw away whatever gets above the "declared" or "assummed" white level). For Adobe DNG Converter (and probably other Adobe apps), this level seems to be 15000; for dcraw, it varies with camera model.

2) Right now I'm fixing the white level at the same level Canon uses with a manual lens (or at f8). However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.

So, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).

Marsu42

Quote from: a1ex on January 27, 2014, 11:08:45 PMSo, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).

Yes, that does sound like a good idea - maybe the oss converters might accept patches to have a more flexible white level, but acr most likely won't ... but it would be nice to be able to use DxO for their prime noise reduction then and again with an acr-cr2  if that's possible, maybe they happen to have the same clipping point.

If you've got an acr-calibrated mini_iso.mo, I'd certainly like to test drive that :->

Audionut

Quote from: a1ex on January 27, 2014, 11:08:45 PM
However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.

Quote from: Audionut on January 24, 2014, 04:48:51 PM
Adobe respects the white level.



Note the maximum pixel value of the sample patch in RawDigger. 

MAD does seem like a good idea.  It sounds like it will help narrow down DR increases to the signal itself without the precision errors having an effect.


Quote from: ayshih on January 27, 2014, 08:12:02 PM
It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR.  However, perhaps I am reading too much into your choice of words.

It was not my intent to imply that it wasn't useful.  However, I went on to describe that more gains can be made by not reducing saturate offset, and hence, leaving more room for ADTG gain reduction.

Quote from: Audionut on January 27, 2014, 04:43:47 AM
I can gain an extra hundredth or two of DR, by not adjusting the black level offset lower (as I was in my earlier tests), and leaving more room for ADTG gain reduction.

Lowering the offset does increase DR.  However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).

Quote from: ayshih on January 27, 2014, 08:12:02 PM
SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level..........................Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease.  Conversely, decreasing SaturateOffset must increase your DR.

The key here IMHO is your description (and my observation) of it just being a digitized value.  Did we really increase the DR of the signal (the photons that deliver our image)?  I would guess not, we just digitally cheated.  This black level that is being changed is so far down into the noise floor to be negligible.  Perhaps it does shift the entire signal (black weighted) down and increases the DR through the signal.  But IMO, by the time we move up to a signal level that is outside of the noise floor, the gains would be minimal at best.

With ADTG gain reduction, we do decrease the noise.  So IMHO, it's much more preferable (in increasing DR) then saturate offset.

Quote from: ayshih on January 27, 2014, 08:12:02 PM
In my mental picture of the electronics, there are two relevant DACs:

  • The first DAC produces the reference voltage for the black-level clamp, and this clamping occurs before the ADTG/DFE amplification stage.  The corresponding register is SaturateOffset (0xc0f0819c).
  • The second DAC produces an analog offset that is added to the signal after both amplification stages, before it is fed into the ADC.  The corresponding register is B/W offset (0xC0F08034).
Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space.  There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.

If this is true (and I am not saying otherwise), then it would appear that the analog reference voltage is only having an effect in the digital domain.  Noise (stdev) does not increase or decrease.  I think of it (reference voltage) as saying to the devices later in the chain, map black to this voltage.  Also, it seems clear that the reference voltage has a fixed level of noise.  The voltage per se hasn't changed the photons we are interested in, it's simply an analog instruction to a digital device.

B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth.  B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth.  Here, the gain is bit precision for the entire signal.

Take it with a grain of salt, I don't have a deep understanding of the electronics involved.  I have only got observational skills and common sense going for me  ;)

ayshih

Quote from: Audionut on January 28, 2014, 02:54:01 AM
Lowering the offset does increase DR.  However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).
...
With ADTG gain reduction, we do decrease the noise.  So IMHO, it's much more preferable (in increasing DR) then saturate offset.

If you take a look at my 50D numbers, you can see the following cases starting at Canon's ISO 100:

  • changing just DFE gains: 1023..13432, stdev = 4.79 => 11.34 DR
  • changing primarily offsets: 1024..15759, stdev = 5.76 => 11.32 DR
(Note that the 0.02 DR difference is below the uncertainty of the DR calculation and the reproducibility of the mini_iso calibration.)  It's true that changing the offsets doesn't decrease the noise as measured in ADUs, while the width in ADUs of the noise does decrease when reducing gain.  However, in both cases, the SNR at typical brightness levels hasn't actually changed.  Instead, what you get in both cases is increased range in the highlights.  Getting better SNR in the shadows requires you to ETTR to make use of this increased range.

Quote from: Audionut on January 28, 2014, 02:54:01 AM
B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth.  B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth.  Here, the gain is bit precision for the entire signal.

This is not correct: B/W offset has no effect on precision.  If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

Audionut

Quote from: ayshih on January 28, 2014, 03:33:33 AM
Getting better SNR in the shadows requires you to ETTR to make use of this increased range.

I make a point of ETTR in my tests for this reason.  Of course, the increased SNR is through the entire exposure.

Quote from: ayshih on January 28, 2014, 03:33:33 AM
This is not correct: B/W offset has no effect on precision.  If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.

Indeed, thanks.

dmilligan

Now that we have raw_diag working properly, here's more data for the 1100D

        Canon                   ADTG[888x] /= 2     
ISO     σ       White   DR      σ       White   DR
100     6.17    13583   10.869  3.1     9714    11.273
200     6.54    15303   10.985  3.25    10956   11.421
400     7.55    15303   10.778  3.81    10998   11.196
800     10.4    15303   10.315  5.26    11033   10.737
1600    16.7    15303   9.632   8.64    11092   10.031
3200    29.7    15303   8.802   14.97   11111   9.242
6400    59.89   15303   7.79    15.55   11110   9.187

Audionut

The white point looks a little low.  Are you applying to much ADTG gain reduction?

With no changes to digital gain, I can keep reducing ADTG gain and increase DR.  After tweaking the digital gain though, further reduction to ADTG gain (after the sweet spot) simply reduces WL.

ayshih

Once you've decreased the gain enough that your white level is dropping, you've already gone too far.  Note that even if the white level is unchanged as measured by raw_diag, it's still possible for the gain to be too low, because you may no longer have solid white highlights due to non-uniformity of saturation.  See these images to see what can happen in the highlights (there, from changing digital offsets rather than gain, but it's functionally equivalent).
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

dmilligan

I have no idea what I'm doing ;)

You're right, I can raise the gain a little bit and keep the white level the same without really impacting the DR. The sweet spot seems to be around 0x330. Here's some new numbers.


        Canon                   ADTG[888x] = 0x330     
ISO     σ       White   DR      σ       White   DR
100     6.17    13583   10.869  4.65    13583   11.277
200     6.54    15303   10.985  4.92    15303   11.397
400     7.55    15303   10.778  5.76    15303   11.168
800     10.4    15303   10.315  7.83    15303   10.726
1600    16.7    15303   9.632   12.6    15303   10.038
3200    29.7    15303   8.802   22.11   15303   9.227
6400    59.89   15303   7.79    22.10   15303   9.228

a1ex

Got some small updates to research tools:

raw_diag:
- JPEG curve analysis, see http://www.magiclantern.fm/forum/index.php?topic=10038.msg99049#msg99049
- cross-covariance analysis of FPN between two dark frames (it gives a hint about how much FPN you can fix from a dark frame)
- option to dump the raw buffer as DNG

adtg_gui:
- updated with 6D engio stubs

Levas

Trying to use the adtg_gui module on 6d.

But I got some linking errors with GDB.
I read in the first post the following:
"adtg_gui.mo (source: adtg_gui.c avl.c avl.h) - cross-platform, requires CONFIG_GDB=y:"

So it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?




a1ex

It's a safety check (this is how you tell it "hey, I know what I'm doing, now let me fry the sensor")

https://bitbucket.org/hudson/magic-lantern/src/tip/modules/adtg_gui/README.rst

Levas

Reading some stuff in the link you send, but I still don't get it.
Makefile.user ?
Does it mean I have to compile stuff myself ? (which I never did and is probably the reason why I don't understand how to set config_GDB = Y)




Levas

If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D

1%

QuoteSo it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?

You need an autoexec bin/symbols compiled with GDB=Y

Audionut

Quote from: Levas on February 01, 2014, 07:50:14 PM
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D

Unless you really have a good understanding on the effects and can help development, it's probably best just to wait until it has been polished and released.

Start learning how to compile.  This way you will be up to speed on the basics with later development.

Levas

The link to the "how to compile" stuff is handy, sure will read into it (Lot's to read and learn, probably will take a month or more before I can compile  :D).
But if I can, I will be one of the first people to use the next impressive magic lantern future!


Marsu42

Quote from: Levas on February 01, 2014, 07:50:14 PM
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D

Changing the code & compiling isn't the only way to help, and in all honesty it needs quite a lot of time to get started, it's only easy in hindsight - but I can remember that setting up the whole thing and getting the bearings on the core code isn't self-explanatory.

At least imho as valuable are suggestions / requests on how to improve usability (menus, options, button layout, ...), helping others with ML questions here on the forum & bug/enhancement reports because even with stable code, a couple of people cannot test it enough to iron out every quirk.

The very same mini_iso.mo module about to be released in this thread needs testing, too - does it work with your postprocessing workflow, is the calibration the same on your camera & lenses...

Marsu42

6d TL users see here to help out with mini_iso - I'm trying not to spam the ML thread with TL (and that's what I'm currently still using): http://www.magiclantern.fm/forum/index.php?topic=10292

a1ex


a1ex

Updated adtg_gui: I've added a routine that dumps all the registers into a log file after taking a picture (this is what I've used to make the funky graphs). I'd like to get these graphs for all cameras, so I need your help:

0) check if adtg_gui.c has engio stubs for your camera (if not, copy them from platform stubs, compile it and post the diff)
1) enable adtg_gui in debug menu
2) enable digic registers
3) take a picture at every single ISO, including intermediates, starting to lowest (50 or 100) until the highest, and back. After each picture, make sure it saves a large number of registers (I expect around 1000 or 2000).
4) send me iso.log from the root of card (on 5D3 this log file had 4 MB, so if you get only 10K or something, double-check the settings and the digic register stubs)

In addition, I'd also like a second data set at some fixed ISO, but with variable aperture.
0) rename or move iso.log from the previous experiment in order to start fresh
1) enable the same stuff as before
2) take pictures at iso 200, with variable aperture from f8 to f1.2 or whatever you have, and back
3) rename this iso.log to av.log and upload it

Optionally, you can do the same with shutter speed.

Troubleshooting: if you get overflow errors after loading adtg_gui, add a small quantity to extra_ram in tcc/tccrun.c (I added 1024 on 60D). If you get ERR70, change max_region heuristic to free_space / 4 in mem.c.

On 5D3, adtg_gui now now works on both 5D3 113 and 123. Also updated iso_regs but didn't try it.

edit: please repeat the tests by changing the tested parameter in both directions.

1%

WOW that module is BIG

Here is the first iso.log.. I went through all the ISOs in the menu, I think its the only module I can load. Even with bvram mirror in shoot mem.

6D:

http://ge.tt/6e5CtgI1/v/0?c

I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?