CMOS/ADTG/Digic register investigation on ISO

[IliasG]

Works for me on win vista32 with exiftool 9.51

[a1ex]

Thanks, I had 9.39, downloaded a few months ago.

Is there any software that uses this tag, given that exiftool didn't know about it until recently?

(ufraw doesn't, so I assume dcraw doesn't either, and Adobe DNG converter simply sets the white level to 15000 in the DNGs)

[Audionut]

ACR/LR ignores it.

[tjaja]

Digital Photo Professional is showing a difrence in the highlights when I put values between 0 and 900 in exiftool.

-edit: it changes the color of the highlight somehow

Hmm the lower values desaturate indeed. No headroom to change colors afterwards.

[a1ex]

Looks like it's changing the desaturate threshold. Cool find.

[Audionut]

DPP confirmed as respecting the tags.

[IliasG]

Yes, DPP should use this info but I don't have it to test.

The Normal white level was 14000 before I changed it to 4321 .. so the test with DPP should be between 14000 and a changed one ..

I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it. Although it does not always match the white clipping level as we can see it in raw histograms.

Although I think it's interesting that there are three tags about WL which DPP probably uses.
We see the linearity upper margin being 1/2 stop under normal WL . does this mean that the raw highlight data are non linear ? or were  they non linear at the start (ADC output) and then linearized before storage in *.CR2 ?.

WL values seem to be steady after gain manipulations, so I don't know  if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. .

BL values look to be measured in the optically black zones so they are reliable and could help for a faster BL identification ..

[Audionut]

It's not being desaturated per se, the white levels are being crushed.

You can create the same effect with luminance tone curves.

[tjaja]

@ IliasG what is that program showing all those metadata entries?

[a1ex]

I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it.

Ufraw doesn't care about spec(ta)cular white level either.

[Audionut]

Can you update your ufraw-mod to respect the tags?

[a1ex]

I think so (and dcraw too).

Now, is there a connection between these exif levels and the register named "white level?" from these graphs? https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW

(I tried to override it and did not notice any visible effect, but didn't check exif)

WL values seem to be steady after gain manipulations, so I don't know  if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. .

I can re-scale the data analogically so I match the actual white level with any value you want. Just tell me what value I should use.

So far I've included 15000 (Adobe), 16380 (full range), the dcraw value (camera-specific) and Canon's default setting with manual lens (that is, simply canceling the aperture gain).

[Audionut]

I just cycled through all of the WL options in mini_iso.  All exposures resulted in these values regardless of saturation.

NominalWhiteLevel: 16228
SpecularWhiteLevel: 16383
LinearityUpeerMargin: 11129

The RawMeasuredRGGB tag is weird.  With mini_iso on a batch of exposures, saturation in the center of the image cause tag values 0,0,0,0.  Without saturation, this tag contained large values.

Going back through a bunch of shots without mini_iso, this tag looks very random.

[IliasG]

@ IliasG what is that program showing all those metadata entries?

exiftool with exiftoolGUI as frontend

[IliasG]

Ufraw doesn't care about spec(ta)cular white level either.

I mean that Dcraw and derivatives only care for the topmost value of data. This is hardcoded in Dcraw and Ufraw (single value for all cases .. you know how good this is ..) or a lookup table.
Rawtherapee uses an external file named camconst.json (camera constants) where among others one can put per ISO WL and a multiplier for each Fnumber ..
DNG looks like also using a table for each ISO but has nothing to fine tune WL depending on apperture ..

In fact I came again in ML forums searching for data regarding the per ISO and per aperture WhiteLevels .. so if it's possible to use the already collected data I ask your permition to use them for RT .. and a bit help how to mine them from the log files .. or even better if the are readily available as numbers ..

[a1ex]

Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?

In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).

If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).

[IliasG]

Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?

In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).

If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).

I think raw_diag white levels is what I need .. basically the values that are same as if I would inspect  the raw histogram for white clipping. This for normal Canon cr2s not ML modified .. and I think ML should also target to the same WLevels as the native Canon cr2s.

I would strongly prefer to use the exif data (specularWL) so the best would be a DCraw patched to use the tags, but I have to recheck as I remember a not insignificant deviation from the histogram derived WL in some rare cases ..

floatDNG is not supported ... waiting for your patch ..

The WL info will be freely available in RT code just search for the "camconst.json" file. I don't know if you'll like the format but as I will use spreadsheet to get the data ready I can sent it when it's ready ..

[naturalsound]

Hi everybody,

I am following this thread from the beginning, being excited about the possible improvements of your amazing find.
Is my following short conclusion correct?

The DR improvement on 5D MKIII is confirmed and the registers are identified.
Nevertheless there is no tool that could automatically tweak all those registers for a specific camera (without the possibility of damage). This tweaking may be required because every sensor is different.

Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed. But ACR seems to work.

In case there already is a module for 5DMKIII that can be used for testing and does not involve setting all those registers manually, please tell me where to find it :-) I would love to help testing.

[a1ex]

This tool is not yet published.

Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed

Quite the opposite. It's exactly these raw converters that will benefit (because the CR2 can be adjusted to match what these converters expect). I've included presets for dcraw and Adobe.

[naturalsound]

Great!
Thanks for clarification :-)

[l_d_allan]

Are there more tests needed? Anything I can help with?
5d3 running the nighters

Are there useful tests that would be helpful from a 6d owned by a risk-adverse non-dev?

I notice there is a module that seems related to ADTG, but I am ignorant about what it does. Much of this thread is over my head.

[a1ex]

5D3, ADTG2 and ADTG4[0x8, 9, A, B]:

- at first sight, these seem to be some black biases: in LiveView, they will cause vertical lines in shadows, but there is a filter that will adapt to the changes fairly quickly (seems to be a moving average).

- at a closer look, changing this register also affects the gain, and we can lower the ISO a little bit more (noise gets lower and there is some more highlight detail captured). Initial guess: 0.15 ... 0.25 stops (maybe more if there is nonlinearity, I don't know yet).

- I believe these are the registers Greg was playing with on 500D.

Will try to update the research tools to the latest codebase today.

[Audionut]

Previously 11.44

A separate button to enter top of the ADTG menu would be handy.  This way when you are in the middle of a 600 (or more) item menu, you can get back to the top of the menu quickly.  Or listing the actual registers in a separate menu with the settings contained in a top menu?

Digic registers do not turn off. 
edit:  To be more precise, they do not get removed from the menu when turning the digic option off.

[ItsMeLenny]

Is this only for Raw? Or will it be able to be applied to in camera jpegs and h264s?

[a1ex]

We are doing the analysis only for RAW.

If you are interested in JPEG, you may be lucky, since the JPEG is created from RAW. But you'll have to do your own analysis

If you can find a digital gain that affects JPEG but does not get burned in the CR2 file, it may be interesting.