CMOS/ADTG/Digic register investigation on ISO

[a1ex]

Good idea, added a screenshot feature to raw_diag. Should be easier to get the screenshots in sync with raw files.

[Audionut]

Hopefully this solves this riddle.

* note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.

Boosting the WL of low aperture values?

That last histogram is getting close to Gaussian!

[a1ex]

Most raw processors use some hardcoded white level for CR2 files (dcraw, ufraw, Adobe DNG converter, so probably all Adobe apps). So, with this trick, Canon effectively added a small boost in image brightness at default settings for large apertures.

For example, Adobe DNG converter simply sets the white level to 15000 for 5D3 images taken at full-stop ISOs. DCRAW uses a hard-coded white level for each camera (which is plain wrong, and the cause of probably all pink highlight issues in dcraw-based raw processors).

With a program that reads the white level correctly (e.g. by looking at brightest white values or by figuring it out from EXIF), the effect of this ISO boost disappears.

Now, I don't know how Adobe handles the data above the white level. I know for sure that ufraw clips it (so here it's important to convert the files to DNG and set the correct white level manually). If Adobe doesn't clip the data above white level, and uses it for highlight recovery, all is fine (you did not lose any highlight detail because of Canon's tricks). But if Adobe behaves like ufraw, you'll have to use exiftool to fix the white level.

My conclusion is simple: Canon's aperture trick does not remove any highlight detail from CR2 files at f1.6 and smaller apertures. BUT raw processors that do not handle the white level properly may ignore the detail above what they consider to be white level, and clip it.

At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.

[hjfilmspeed]

Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)

[Audionut]

A lot of the information gained with these registers should help with cr2hdr and such correct?  With histograms for instance, there's no more guessing of WL.  Discard some hot pixels and you're good to go!

[Audionut]

Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic?

When I unsrew the lens on my 5D3 it uses the non-boosted white level.  Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.

[ItsMeLenny]

When I unsrew the lens on my 5D3 it uses the non-boosted white level.  Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.

Maybe I'm just suffering from a placebo effect, but when i do this, take one shot with the lens detected and one shot with it screwed half off, I see a considerable jump in brightness when the lens is properly detected at 1.8/f on a 50mm using the neutral picture style and shooting in jpeg.

[a1ex]

Yes, this confirms our theory.

[Levas]

Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)

The only reason I can think of canon is doing this (and probably all other camera manufacturers) is because most photographers expect exactly 1 stop extra light when they're going from f2.8 to f2.0.
If it's really true that the digital sensors don't get all the light with apertures bigger than 2.8, you must do this trick to get that extra stop of ligt on you're histogram.

But when you're after the best image quality, dynamic range etc. you're (a tiny bit) screwed.

[ItsMeLenny]

I just found something very interesting with my camera (which I think is the basis of this whole discussion).

So people can replicate;
550D -> Video mode -> Expo Menu -> Iso Menu.
I set the canon analog to 100, canon digital to 0, then ML digital to +7.0 EV (giving an equivalent of 12800).
Aperture set to something below 2.0/f
Now, when one scrolls through the canon digital 0 EV has about the same amount of noise than 0.5 EV
But when one sets the aperture to 1.8/f
Now, when one scrolls through the canon digital 0 EV has ALOT more noise than 0.5 EV



Wait a minute. That could be my own stupidity.
When it's set so low it has nothing to pull up but noise, hence why when the aperture is wider it has more image to pull up.
Or is it, I don't know now.

[a1ex]

Here's a research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect:

* CMOS gain, described in the Dual ISO PDF
* ADTG gains (0x8882 ... 0x8888) - same value for all of them
* Saturate Offset (0xc0f0819c)
* Digital Gain (0xc0f37ae4/af0/afc/b08)
* Black/White Offset (c0f37ae0/aec/af8/b04))

iso_regs.mo (source: iso_regs.c)



Notes:
- Do not mix photo mode settings with LiveView ones. Hit "disable overrides" before switching between photo and video.
- It requires a custom-compiled ML with CONFIG_GDB = y. As I said, it's not for end-users, it's a research tool.
- The parameters are explained in the help text. More info here and in this thread.
- This has a TON of parameters hardcoded for 5D3. Porting on other cameras is very hard. Assuming the registers behave in the same way, there are only 25 camera-specific parameters. On 60D for example, only ADTG gains are the same, so you will have to figure out all the other registers with ADTG_GUI and do the math from scratch. Don't even think about trying on 5D2/500D/50D.
- To disable Canon's aperture trick, simply fix digital gain to some value.

Be careful not to fry your sensor

[Levas]

there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?

[Audionut]

there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?

For the simple ADTG tweak, see the response below.

For the above.

Porting on other cameras is very hard.

a1ex opinion of easy, is suitably hard for most users.  Here he describes very hard with bold!

[a1ex]

Besides, for 6D somebody has to fix the raw backend in photo mode first: www.magiclantern.fm/forum/index.php?topic=10028

[hjfilmspeed]

I got the same results with the holding down the DOF button while unscrewing canon lens. I saw the jumps in the raws too in adobe LR 4.4. Didnt try recovering highlights but i shot 1.4 1.6 1.8 2.0 and 2.8 on canon 50mm with there unscrewed twins. It didnt really start to level off till about 2.8 5d3 I also compared it to 1.8 on an old 50mm nikon. Good news is canons 50mm 1.8 unscrewed with DOF button held is brighter then the 1.8 on nikon 50.  1.4 unscrewed was still brighter then all the apertures so other then it being soft its still good to use if you need that light. I notice a slight difference in color to even though it was all the same temp. So i guess that might have to do with the black and white levels

Will this jump occur with ML RAW video? Ill test that next, I assume yes since it seems like its in the RAW concoction duringr atdc.

When this is done this will be an amazing mod to have! I dought it but i hope this helps a lil.....il shut up now

[a1ex]

Can you compare the clipped highlights at f1.6 normal vs unscrewed?

(I want to see how Adobe handles the white level)

[Greg]

500D (default) :


ADTG3[0] = 0x149 (default) -> 0xb9 (trick)
ADTG3[3] = 0x14a (default) -> 0xba (trick)




500D - T1/200s, F1.4

ISO                100       160       200       320       400       640       800       1250       1600    3200    6400    12800
ADTG3[0]       149       149       149       148       148       148       148         142         142      142      142      142
ADTG3[3]       14a       14a       14a       14a       14a       14a       14a         145         145      145      145      145
CMOS[0]            0         90         90       1b0       1b0       1f8         1f8         168         168      168      168      168
C0F0[819c]     f87        f87        66f       66f        66f        66f         66f         66f          66f        66f       66f      700
C0F0[8030]  1570     1126    1570     1126     1570     1126      1570      1126       1570     1570    1570    1570
C0F0[8034]  1079     1079    1991     1991     1991     1991      1991      1991       1991     1991    1991    1900

[hjfilmspeed]

Can you compare the clipped highlights at f1.6 normal vs unscrewed?

(I want to see how Adobe handles the white level)

Sure. Sorry Just crashed for a while. My shots did include blown highlights. Il post them and there recovered twins in a bit

Sent a pm a1ex with raw files included

[Audionut]

At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.

 


0.003EV.  I did the f/4.0 shot first, so the camera was a little warmer in the other shots.

[a1ex]

Alright, so the side effects of Canon's ISO/aperture trick are:
- small overflow at f1.4 (0.03 stops), larger at f1.2 (I guess 0.25 stops from DxO chart, link below). This results in highlight loss in the CR2.
- histogram gaps indicating digital scaling (not quite good for denoising algorithms)
- possible highlight loss in some raw editors (ufraw for sure; I'll fix it in ufraw-mod). To check your favorite editor, take two test shots, f1.6 normal and with lens unscrewed by holding the DOF button, and compare the highlights.

http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml

The good news is that I can disable this trick fairly easy on all cameras.

[Audionut]

Good read.  I'll do a heap of tests later showing Canon vs ML with the WL capped and brightness fixed in post.

This will be a good test also to see just how much light loss there is at each aperture value below f/4.0 for my lenses.

[Greg]

500D - T1s, ISO 100

aperture    -    F                     1.4         1.6      1.8        2.0        2.2        2.5         2.8         4.0          8.0         22
digital gain -   C0F0[8030]  1570     130a    11eb     10f5     107a     107a      1000      1000       1000     1000


F1.4 lens unmount, the value is 1000

unmount lens (So that the contacts do not touch.):


mount lens :



No lens affects the colors:

unmount lens :


mount lens :


mount lens + trick (digital gain 1000):

[Greg]

Shutter speed.
Changes are 1s and longer.

1s :


10s :




I found something interesting. Turn on the camera. ADTG registers -> show set Modified since.
Take a picture.

ADTG3[0] = 0x149 (was 0x111)
ADTG3[3] = 0x14a (was 0x111)

[Audionut]

No lens affects the colors:

For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2

The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.

My light source was a led light, so it should have fairly consistent temperature output.

[Greg]

For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2

The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.

My light source was a led light, so it should have fairly consistent temperature output.

I set the same color temperature in ACR.