Author Topic: CMOS/ADTG/Digic register investigation on ISO  (Read 712430 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #350 on: February 04, 2014, 08:19:57 AM »

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #351 on: February 04, 2014, 08:28:41 AM »
http://optipng.sourceforge.net/


This is supposed to be better, but single threaded and slower:  https://en.wikipedia.org/wiki/PNGOUT

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #352 on: February 04, 2014, 11:20:20 AM »
Since there is a large difference between my logs and Audionut's logs, I'd like to ask you to repeat the tests by changing the tested parameter in both directions (incrementing and decrementing). This will help identifying which of these graphs are just noise and which of them are relevant.

The graphs now show all the data points from the log file, so if there are different values obtained in different experiments, that graphs may be just noise (or may be not).


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #354 on: February 04, 2014, 12:44:38 PM »
Updated the graphs and dropped some irrelevant ones. Interesting that after merging the logs I've got much fewer results (will double-check to make sure it's not a bug).

Also moved all the graphs here: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #356 on: February 04, 2014, 06:45:30 PM »
CMOS[1] 0x663   ISO 600
CMOS[1] 0xee3   ISO 1200

I couldn't find any other intermediate ISO steps.  Noise level changes for the better compared to next higher Canon ISOs (800/1600), but needs more consistent workflow for accurate results.

On my test samples, I changed ISO in Canon menu for Canon ISOs, so these have been tainted by other CMOS/ADTG/Digic register changes.  I'll attempt a more thorough workflow tomorrow for DR results.

I tested these CMOS[1] values.

Code: [Select]
3
113
223
333
443
553
663
773
883
993
aa3
bb3
cc3
dd3
ee3
ff3
1003
1113
1223
1333
1443
1553
1663
1773
1883
1993
1aa3
1bb3
1cc3
1dd3
1ee3
1ff3
2003

There is a defined pattern.  Here are the average raw values for a patch on the color checker.

Code: [Select]
101.8
195.9
392.1
777.6
1555.8
2984.6
567.4
5598
194.3
377.8
750.5
1501.4
3006.7
5814.9
1095.1
11196.7
100.6
194.8
387.2
772.5
1539.4
2969.5
561.4
5684.2
192.4
373.9
745.4
1489.3
2992.3
5792.2
1087.8
11130.1
99.6


Notice how the equivalent ISO steps have slightly lower raw values.

For instance, ISO 100 gives raw value 101.8 and other register values produce 100.6 and 99.6.
I can't test noise differences since I messed the workflow up.  I'll have another attempt tomorrow.

If you want to test yourself, the money registers for equivalent ISO 100

0x3
1003
2003
3003
4003....etc, etc.

It's the same pattern for other ISOs.

663
1663
2663......etc, etc.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #357 on: February 04, 2014, 07:44:24 PM »
I believe 0x773 has a little less noise than 0xDD3 (need to double-check). At first sight, this combined with CMOS[3] = 0x944 seems to give a 0.25 stop improvement in shadows at ISO 4370 (derived from 6400).

BTW, all research tools are now working on both 5D3 113 and 123.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #358 on: February 04, 2014, 08:09:50 PM »
0x223 - ISO 400


0x663 - ISO 600


0x333 - ISO 800


0xee3 - ISO 1200


0x443 - ISO 1600



Some other interests.

ISO 200 - 0x113 vs 0x883


ISO 400 - 0x223 vs 0x993


This trend continues for ISO 800 and ISO 1600. 

Only changing CMOS[1] with a starting point of Canon ISO 100, so not sure why the WL changes between 15283 and 15331.
I'd like to find ISOs 150 and 300.

I've made a dump here.  Up to 0xff3 for the moment.

ayshih

  • Contributor
  • Senior
  • *****
  • Posts: 266
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #359 on: February 04, 2014, 10:29:20 PM »
For the 50D, here's the register dump for the ISO sweep, from 100 to 12800 and back down:

https://www.dropbox.com/s/7ftxovt2oqu2njn/50D-ISO.log

I'm not going to do the aperture sweep on my lenses because they're not particularly fast, but I can borrow a lens next week if someone else hasn't provided that register dump before then.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #360 on: February 05, 2014, 07:56:07 AM »
Uploaded 50D graphs.

Also modified iso_regs to handle all the CMOS gains.

Also updated the first post with links to these graphs.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #361 on: February 05, 2014, 08:57:34 AM »
Hypothesis about CMOS gains:
- each ISO is described by a number from 0 to 15. That's 4 bits, from b3 (MSB) to b0 (LSB).
- these 4 bits might be toggling 4 amplifiers on and off
- if this is true, the individual gains of these amplifiers would be approx. b0=1.92, b1=3.85, b2=15.28 and b3=1.91 (based on Audionut's estimation).
- all gains except 6, 7, 14 and 15 follow this logic
- gains 6 and 14 seem to be attenuated by 10.5 (the attenuator is triggered by logical function b2 & b1 & !b0 )
- gains 7 and 15 seem to be attenuated by 2 (the attenuator is triggered by logical function b2 & b1 & b0 )

Code: [Select]
CMOS gain   Measured  Derived
       0     1.00     1.00     
       1     1.92     1.92     
       2     3.85     3.85     
       3     7.64     7.41     
       4    15.28    15.28     
       5    29.32    29.41     
       6     5.57    58.86    ?
       7    54.99   113.28    ?
       8     1.91     1.91     
       9     3.71     3.67     
      10     7.37     7.35     
      11    14.75    14.15     
      12    29.54    29.17     
      13    57.12    56.13     
      14    10.76   112.35    ?
      15   109.99   216.21    ?

Code: [Select]
from pylab import *

gains = array([101.8,195.9,392.1,777.6,1555.8,2984.6,567.4,5598,194.3,377.8,750.5,1501.4,3006.7,5814.9,1095.1,11196.7]);
gains = gains / gains[0];

print "CMOS gain   Measured  Derived"
for g in range(16):
    measured_gain = gains[g]
    derived_gain = 1
    for b in range(8):
        if g & (1 << b):
            derived_gain *= gains[1 << b];
    status = " " if abs((derived_gain - measured_gain) / measured_gain) < 0.1 else "?"
    print "%8d %8.2f %8.2f    %s" % (g, measured_gain, derived_gain, status)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #362 on: February 05, 2014, 09:53:18 PM »
5D3: CMOS[6] = 0xa00 results in an extremely low gain (very dark image). It gets a little better if you increase the CMOS gain. It has more highlight detail than ISO 100 with ADTG gain at -2 EV, and I've estimated the clipping point at around ISO 67 (by comparing with another image underexposed with shutter).

It's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #363 on: February 05, 2014, 10:03:56 PM »
It's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.

Your last discoveries sound great, I'm confident you'll squeeze every last bit of dynamic range out of Canon ... and as usual I hope all cameras will benefit from it. I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #364 on: February 05, 2014, 10:55:05 PM »
I forgot to mention that the raw values I listed above are black subtracted.  Not sure how that effects your gain calculations.

I'll take a look at CMOS[6] and [7] today.

I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?

At the very least, they could have had an almost real ISO 50.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #365 on: February 06, 2014, 02:43:18 AM »
CMOS[6] with saturated pixel wells

0x100 is the last register value to give White Level 15283 (0x1ff actually).
0x200-300 - WL 3100
0x400-500-600-700 - WL 2570
0x800 - WL 2081
0x900 - BL 15283 WL 15331 - Full white screen
0xa00-b00 - WL 2140
0xc00 and above WL 2100

This is a repeating pattern like CMOS[0].  At 0x1100 the pattern repeats.

While typing this, I thought I had better dig down further.

0x100  WL 15283
0x101  WL 8680
0x102  WL 12370
0x103-104  WL 15283
0x105  WL 10250
0x106  WL 12440
0x107  WL 12570
0x108-109-10a-10b  WL 15284
0x10c  WL 14610
0x10d-10e-10f  WL 15283

From here, something is happening in the blacks.  The histogram reminds me of high ISO.
0x110  WL 15283 - DR reduces 1 EV (vs no register change)
0x111  WL 8640
0x112  WL 12290
0x113-114  WL 15283 - DR reduced 1EV
0x115  WL 10200 - Histogram looks normal
0x116  WL 12410
0x117  WL 12540
0x118-119-11a-11b  WL 15283 - DR reduced 1EV
0x11c  WL 14520  BL 2046
0x11d-11e  WL 15282  BL 2044
0x11f  WL 15282  BL 2045

Now we are back to having normal looking histograms with a little more noise (deviation)
0x120  WL 15283 - DR reduced 0.2 EV
0x121  WL 8650
0x122  WL 12360
0x123-124  WL 15283 - DR reduced 0.2 EV
0x125  WL 10220
0x126  WL 12420
0x127  WL 12550
0x128-129-12a-12b  WL 15283 - DR reduced 0.2 EV
0x12c  WL 14540
0x12d-12e-12f  WL 15283  BL 2047

First thing to notice, repeating patterns.

Same thing happens at 0x13? except here there is extreme deviation and the black level also changes wildly.  I'm not testing each register as I have some respect for my shutter.

I'll test each 0x140 as the first few appear to mimick 0x10? (useful) with different WL's. 
0x140  WL 15283
0x141  WL 10480
0x142  WL 13800
0x143 -114  WL 15283
0x145  WL 9912
0x146  WL 11140
0x147  WL 9420
0x148-149-14a-14b  WL 15283
0x14c  WL 12530
0x14d  WL 14240
0x14e-14f  WL 15283

0x15? - 0x16? - 0x17? - 0x18? - 0x19? - 0x1a? - 0x1b? - 0x1c? - 0x1d? - 0x1f?  All appear to have the same pattern as 0x10?

A quick look at 0x2?? shows no sign of any difference.  ie all the 0x2?? values match 0x200

There's some funky stuff happening at 0x9??  Have a look at 0x909 for instance.

All of these are like ADTG gain.  Make sure you saturate the pixel wells!

edit1:  Noise level doesn't reduce with reducing WL in the batches where I haven't specifically mentioned changes.

edit2:  All of these tests where conducted at CMOS[0] 0x3.  At CMOS[4] 0x2?? and above where the WL drops drastically, you can bring everything back with higher CMOS[0] values (which a1ex mentioned above).  So for instance CMOS[0] 0x443 and CMOS[4] 0x200 brings back a normal image with WL 15283.  Initial observation suggests that increasing CMOS[0] simply negates the effect of CMOS[4].  ie:  No difference in produced image.  Needs further testing.

edit3:  CMOS[0] 0x10 through 0x18 produce WL 2060.  0x19 through x01f produce dual_iso pattern.  Same thing happens at 0x20 through 0x2f and again at 0x30 through 0x3f, so I'll assume this continues through the other 0x?? values.

What interesting though, the ISO difference reported by cr2hdr seems to match the register normal ISOs.  For instance, 0x1f = 0.96 EV ISO difference, 0x2f = 1.97 EV ISO difference etc.  Including those half stop ISOs.

a1ex, can you create an option to disable iso.log creation?  Also with the latest raw_diag the information overlay disappears after a short time.  This appears to be happening when the iso.log info gets cleared.

edit4:  The same pattern is repeating at CMOS[6] 0x? and 0x??

Have a look at CMOS[7] 0xc0

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #366 on: February 06, 2014, 06:15:53 AM »
I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?

See if the digital gain register is changing with aperture.  Can you set the aperture value lower then 1.2?  That could be interesting.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #367 on: February 06, 2014, 07:34:43 AM »
Will do; until then, just comment out the prop handler and recompile.

Didn't digest all of the stuff yet, just some small remarks:
- you mean CMOS[0] instead of CMOS[1]
- the CMOS register values are 12-bit, so the fourth hex digit does nothing
- ADTG values are 16-bit and ENGIO (DIGIC) registers are 32-bit

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #368 on: February 06, 2014, 08:25:36 AM »
Will do, thanks.  edit:  I might even be able to add the menu entry myself.

Yes I did.
The highlighted one  0x100?

ayshih

  • Contributor
  • Senior
  • *****
  • Posts: 266
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #369 on: February 12, 2014, 06:04:48 PM »
I finally was able to borrow that faster lens.  For the 50D, here's the register dump for the aperture sweep, from f/8 to f/1.4 and back:

https://www.dropbox.com/s/ihry3dq9drv2a05/50D-aperture.log

I can immediately see that the digital gain (0xC0F08030) is 0x1000 down to f/2.8, and then starts increasing, up to 0x147A at f/1.4.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #370 on: February 12, 2014, 10:48:02 PM »


So... 0.35 stops of digital gain? It matches the DxO value quite well.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #371 on: February 15, 2014, 10:19:40 AM »
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX

(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)

ItsMeLenny

  • Hero Member
  • *****
  • Posts: 928
  • 550D
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #372 on: February 15, 2014, 10:25:01 AM »
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX

(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)

I can do 550D, I am just unsure as to what "check if adtg_gui.c has engio stubs for your camera" means.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #373 on: February 15, 2014, 10:27:29 AM »
Open the source code for the modified adtg_gui (not the repo one, but the one from this thread) and look for "engio", you'll find them right away (some cameras have them, others don't).

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #374 on: February 15, 2014, 10:35:31 AM »
The ones that don't, they just need to be found?

The ones missing the stub,

500D, 550D, EOSM, 600D, 650D, 700D