Magic Lantern Forum

Developing Magic Lantern => Reverse Engineering => Topic started by: a1ex on January 10, 2014, 12:11:01 PM

Title: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 10, 2014, 12:11:01 PM
So, what's all this stuff about "sensor update"?


Just a small improvement in dynamic range in photo mode (around 0.3...0.5 0.8 stops). We were able to fine-tune the amplifier gains in order to squeeze a little more highlight detail.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-cmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d2-cmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/6d.png)

(http://i.imgur.com/kXrbKTq.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/5d3-movie.png)

Graphs for the other cameras can be found here: http://www.magiclantern.fm/forum/index.php?topic=10111.msg97780#msg97780

[February 23] 5D3 reaches nearly 0.8 stops of improvement: 11.77 EV at ISO 115 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103465#msg103465), and also a new ISO 66 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447).
[January 17] 5D2 reaches 11.92 EV of dynamic range at ISO 81 (http://www.magiclantern.fm/forum/index.php?topic=10111.msg98281#msg98281).

Only 0.5 EV? That's way too small!


Yes. You may take a look at Dual ISO (http://www.magiclantern.fm/forum/index.php?topic=7139) (which will get an additional 0.3...0.5 stop boost with this "sensor update"; also resolution issues were largely solved), or you may consider switching to Nikon.

Wait a minute, that means less noise, right?


Well, it means you get a little more detail in highlights. This doesn't mean less noise per se (the new ISOs will be just as noisy in shadows as the old ones), but it will let you shift the exposure to the right by 1/3 ... 2/3 EV and collect more photons. This will result in lower noise.

For example, on 5D Mark III I could lower the ISO by 0.37 stops from 100, resulting a new ISO 77. After some more tweaking, I've got 0.6 stops below ISO 100 => ISO 66.

Sample images?


No relevant samples yet, sorry.

How exactly are you getting more highlight detail compared to Canon firmware?


The signal from the sensor seems to be amplified in 2 stages (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p2.html#read_vs_iso): a CMOS amplifier (which operates in full stops - powers of 2 - and we have tweaked it when implementing Dual ISO (http://www.magiclantern.fm/forum/index.php?topic=7139)) and an ADTG (http://magiclantern.wikia.com/wiki/ADTG) amplifier which can be configured in finer increments. After these two stages, the signal is digitized (with an ADC), probably tweaked digitally, and saved to CR2. We have noticed the ADTG amplifier tends to run a little "hot" (that means, it gets saturated a little too early - nothing to do with temperature).

To get the extra highlight detail, one has to reduce the gain for the ADTG amplifier until the ADC will no longer be saturated. At this point, the white level (maximum recorded level in the raw file) will begin to decrease and no more detail will be recovered (since now the CMOS itself or the CMOS amplifiers will get saturated instead).

To play with these gains, scroll down to the research tools section.

Does this mean Canon did not fully optimize their sensor for low noise?


I'd say they simply left a safety margin in their code to make sure the ADC is always saturated (that is, to make sure white is always recorded as white).

How are intermediate ISOs implemented in Canon cameras?


I will try to answer this question in a detailed paper (including how exactly I've reached the conclusion).

For now, you may take a look at these graphs: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/1.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/5.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/7.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-iso-100-1600/8.png)

What's with that message about f1.2 lenses?


According to DxO, Canon applies a digital ISO boost at wide apertures (http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml) (about 0.5 stops on APS-C cameras, and 0.2 stops on FF from my findings (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96952#msg96952)) to compensate for light loss. Since the digital gain is burned into the CR2 files, at wide apertures you will lose a small amount of highlight details (under 0.1 stops on FF (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96952#msg96952), did not check on APS-C). Also you may get some extra noise from round-off errors, and if your raw editor does not handle white level properly, you may lose even more highlights (details here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96850#msg96850)).

We are going to fix this by turning off the digital gain.

Any side effects?


None yet.

What about Dual ISO?


You will be able to use both tweaks (it will just work).

What about video?


I've got only 0.1 stops of improvement on 5D3, but didn't try too hard.

What's the current state?


Research. We are trying to optimize the parameters that influence ISO, understand their effects (did we really gain 0.5 stops of DR or are we just daydreaming?) and port the results on other cameras. Scroll down for some research tools.

Will it work on my camera?


- If you have a 550D or newer camera, it will most likely work.
- If you have a 7D, no idea yet.
- If you have a 5D Mark II, 50D or 500D, don't get too excited. I've barely got 0.15 stops of improvement on 5D2. Mystery solved!
- Confirmed working on 5D3, 550D, 600D, 650D, 700D, 60D, 6D, 5D2, 50D and 500D.

How can I help?


- Play with the research tools below and report your findings.
- If you have access to laboratory equipment and you can measure the real ISO, you can help us check the validity of our ISO theory.

Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!


Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)

But if you have some basic coding/math skills - enough so you can follow the entire discussion without getting dizzy - I have some nifty research tools for you:

raw_diag.mo (https://builds.magiclantern.fm/modules.html#raw_diag) (source (https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/raw_diag/))- cross-platform, anyone can run it

This tool does technical analysis on the raw image data (black/white levels, noise, dynamic range, SNR curve):
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr-60d.png)

Alternative: RawDigger (proprietary, nonfree).

iso_regs.mo (https://builds.magiclantern.fm/modules.html#iso-research) - 5D3 only, requires either the crop_rec_4k build (http://www.magiclantern.fm/forum/index.php?topic=19300), or a custom ML build from the iso-research branch:

A research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect. Details here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96715#msg96715).
(http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.png)

adtg_gui.mo (https://builds.magiclantern.fm/modules.html#iso-research) - cross-platform, requires CONFIG_GDB=y a custom ML build from the iso-research branch (on 5D3, the crop_rec_4k build works fine as well):

The good old ADTG/CMOS tool (http://www.magiclantern.fm/forum/index.php?topic=6751.msg71720;topicseen#msg71720) updated to also intercept DIGIC (ENGIO) registers.

(http://s28.postimg.org/3p02r92st/VRAM0.png)

For sources, please check the branch iso-research on the main repository.

When it will be released?


When it's ready. I also want to summarize the findings in a small paper (like the Dual ISO PDF (http://acoutts.com/a1ex/dual_iso.pdf)), so I need a little time.

Any recommended reading?


http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html (main site: http://www.clarkvision.com/articles/ )
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity





Original message:
Quote
Now some fun stuff:

Compile ML with CONFIG_GDB=y and load the adtg_gui module. Override the column gain registers (0x8882-0x8888) to roughly half the original value.

My results (with 978bf96 and these changes (http://a1ex.magiclantern.fm/bleeding-edge/dr.patch); shutter 1/30):
Unaltered ISOs:

ISO    ADTG default   white level    noise stdev    DR
100    0x41b          15282          6.62           10.96
160    0x435          13306          5.53           10.99
200    0x54c          15282          6.83           10.92
250    0x435          15282          8.55           10.60
1600   0x454          15282          11.87          10.12


ISOs altered with ADTG gain overriden to 0x250:

ISO    ADTG overriden white level    noise stdev    DR
100    0x250          11911          3.68           11.38
160    0x250          9935           3.25           11.24
200    0x250          11924          3.88           11.31
250    0x250          11917          3.84           11.33
1600   0x250          11915          6.46           10.58


So... we can get almost half-stop of DR just by tweaking some registers?! Looking forward to see the RawDigger results.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 10, 2014, 12:46:22 PM
Keen to try the results when I get home.  I would be happy with an indicator to notify for true overexposure vs guessed.  That's probably a bit ugly though.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 10, 2014, 12:54:27 PM
Well, I think I found a possible bug. If the image is not overexposed, and max stops at say 13000, and there aren't 10 pixels of the same value to confirm that maximum, my algorithm will assume a white level of 12000 (or whatever the initial guess was).

If the heuristic will take the unconfirmed max if there's no confirmed one, it might be a little better. Also, the safety margin probably can be made smaller (100? need to experiment).

The main problem is that I don't know how to tell for sure whether the image is clipped or not (only through heuristics).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 10, 2014, 01:01:14 PM
Can you discard for hot pixels and average the next bunch?  Maybe some weighting towards the max values.

Or discard as hot pixels if their value is above 16000. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 10, 2014, 01:09:25 PM
Yes, something like averaging between 5th max and 50th max might work pretty well. Or maybe just the 5th max minus a small amount.

If you play with the algorithm, be sure to test it at 30-second exposure. This was the main reason for a big safety margin (if you over-estimate the white level by just 1 unit, auto ETTR will no longer work properly).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 10, 2014, 08:00:36 PM
Quick test before bed.

Default
ISO Saturation S.Dev SNR DR
100 16383 7.14 2294          11.21


1/4th shutter.  ML reported the same saturation level, or there abouts.  To tired to be testing properly.

ADTG tweak
ISO Saturation S.Dev SNR DR
100 13045 4.04 3329          11.73



I also noticed that if you only adjust 1 bank of registers, the saturation level differs (lower) for the G2 channel.  My first thought, highlight recovery.  Why blow both green channels and rely on post to pull detail from RB, when you can keep actual detail in one of the G channels.

edit:  Of course, that would only work if the ADC clips before the pixel wells saturate.

11.73 stops before dualISO.  You'll be spanking Nikon soon!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 10, 2014, 08:16:30 PM
I think there are 8 columns where you can tweak the gains.

I also think ADC does clip off a little earlier before the pixel wells saturate. With this tweak, notice that DR at 160/200/250 is now almost identical, with slight advantage towards 250 (differences being probably from quantization error in the noise).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 10, 2014, 11:51:33 PM
Quote from: a1ex on January 10, 2014, 08:16:30 PM
I think there are 8 columns where you can tweak the gains.

Yeah, they're in 2 batches of 4.

QuoteI also think ADC does clip off a little earlier before the pixel wells saturate.

Well, it actually looks to be around 0.5EV.  0.5EV increased exposure before saturation (tweaked vs normal). Needs further testing, but shows that not only do we gain 0.5EV extended range in the shadows, we also gain 0.5EV increased SNR through the entire exposure.  It's going to need your magic on the white level, there's pink highlights in saturated tweaked ISOs.  I should probably look at raw2dng, but I'm much to sleep deprived for that atm!


Looks like you nailed that register too.  Going lower continues to drop the noise, but the white level drops at a rate where DR remains equal (or there abouts).  Not to mention that you get to a point where even with full sensor saturation, there isn't enough gain to deliver any signal.  ::) I'd be interested to find out what the measured ISO is with 0x250.  DxO suggests measured ISO is 80 for camera ISO 100, which suggests Canon choose the register value as a compromise between noise and mapping the maximum value to 14bits.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 11, 2014, 02:10:33 AM
Quote from: a1ex on January 10, 2014, 08:16:30 PM
With this tweak, notice that DR at 160/200/250 is now almost identical, with slight advantage towards 250 (differences being probably from quantization error in the noise).

I need to brush up on quantization.

QuoteAs expected, intermediate ISOs like 160 or 250 do not cause any changes in ADTG/CMOS con-figuration. These ISOs are obtained by applying some digital gain to the raw data acquired at the nearest full-stop ISO, and this gain is configured from the DIGIC register 0xC0F08030 (SHAD_GAIN).

A different register again?  Digital and (0x8882-0x8888) being analog?

Makes you wonder what they were thinking with ISO 50.  They could have made that below 100 ISO useful ffs.  dmilligan seems to think decisions like this is par for the course (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95040#msg95040).   :(
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 11, 2014, 03:31:20 AM
For 500D:
ADTG3[0] = 0x149 (default) -> 0x0 (trick)
ADTG3[3] = 0x14a (default) -> 0x1 (trick)


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 11, 2014, 08:29:24 AM
Quote
A different register again?  Digital and (0x8882-0x8888) being analog?

Yes. I'm just not sure how to override it in photo mode on 5D3. On 5D2 it works (add raw_blinkies.o in the Makefile) but there's little or no change to DR numbers. It could be used to remap the values back to 14 bits, but we need to check the histogram for gaps.

So, the ADTG gain may be yet another amplifier stage?

In LiveView (raw video + silent pics in movie mode), the behavior is different. ISO 160/200/250 result in no difference at all in the raw data (so the raw histogram doesn't change at all). Instead, the JPEG is developed with different gains, and these gains are adjusted from 0xC0F08030. I read this register in photo mode to figure out how to adjust the white point for correct clipping warnings, and this register is also used for gradual exposure, for example.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 11, 2014, 09:56:37 AM
Looks like the ADC is clipping 0.7EV before sensor saturation...................

edit:  deleted graphs and conclusion since it was way off the mark.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 11, 2014, 04:21:45 PM
F1.4 T1/4s ISO 100 (-3EV in ACR) :
(http://s9.postimg.org/rucnfntpb/100.jpg)

F1.4 T1/4s ISO 200 HTP + ADTG trick (-3EV in ACR) :
(http://s9.postimg.org/p1jfvmtcv/200htp.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 05:57:14 AM
Signal on the left of the image is throwing the ML calculations out of whack.

Images such as this.
(http://i.imgur.com/6GTZLy4.jpg)
https://dl.dropboxusercontent.com/u/34113196/ML/Noise%20samples/_46A5167.CR2

ML reports,

Mean: 2048
Dev:   8.30
Max:   15463

RawDigger shows the Std.Dev as 6.52, same Mean and Max values.

If I measure right to the image edge in RawDigger, I still see a low S.D value.  If I measure from the very top left of the optical black, RawDigger reports S.D of 7.34.

If I make sure the left side of the image contains near black, ML doesn't have the issue.

edit:  I'm being lazy, I haven't even looked at your code.  I'll head off and do that now, and test some other means if I understand how it's being coded.

    if (raw_info.active_area.x1 > 10) /* use the left black bar for black calibration */
    {
        autodetect_black_level_calc(
            16, raw_info.active_area.x1 - 16,
            raw_info.active_area.y1 + 20, raw_info.active_area.y2 - 20,
            3, 16,
            &mean1, &stdev1
        );
        autodetect_black_level_calc(
            16, raw_info.active_area.x1 - 16,
            raw_info.active_area.y1 + 22, raw_info.active_area.y2 - 20,
            3, 16,
            &mean2, &stdev2
        );
    }


I don't understand what values are shifting the scanning area, and what values are setting the scanning size.  I'd like it to scan almost the entire optical black area leaving a few pixels off left and right, and more pixels top and bottom.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 06:35:21 AM
@a1es: Sorry for late reply; as dmilligan said, RON is Read On Noise.

I pointed you this because in conventional photography (I mean short exposure at 'temperate' temperature), when we speak about SNR, N (noise) is quasi resumed to RON. So where we get the smaller RON we get the best SNR. Of course I suppose S (signal) is the same to compare.

About DR, I'm surprised what you use an offset at 500 (?). Unit is not ADU?
Typical offset is 2048 ADU for 5D3, 1024 ADU for 5D2 and 128 ADU for 5D1.
I presume you know it, but in case; these values are to ensure sigma from RON + FPN will not produce clipped pixels because of low values. Of course, most the offset is high, most you can expect high RON + FPN levels, and less DR.

I find your studies very interesting because it's sure all of that could be optimised, it should be possible to get a DR gain (not 5EV of course) and a better RON, FPN, FWC and detectability.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 07:34:13 AM
Quote from: a1ex on January 10, 2014, 08:16:30 PM
I think there are 8 columns where you can tweak the gains.

ADTG2 adjusts blue and green channel 1
ADTG4 adjusts red and green channel 2

Would have been nice to dial the green sensitivity back, bring it more into line with the sensitivity of the RB channels.I'm playing with adjustments to each column, but it doesn't look like it will bring any benefit to the table.

Quote from: SpcCb on January 12, 2014, 06:35:21 AM
I pointed you this because in conventional photography (I mean short exposure at 'temperate' temperature), when we speak about SNR, N (noise) is quasi resumed to RON. So where we get the smaller RON we get the best SNR. Of course I suppose S (signal) is the same to compare.

Increased SNR from reduced noise assumes the signal stayed the same.  This ADTG tweak is reducing the noise more then the signal for instance.

Quote from: SpcCb on January 12, 2014, 06:35:21 AM
About DR, I'm surprised what you use an offset at 500 (?). Unit is not ADU?

That was for the white level that ML uses.  As a1ex mentioned, if ML uses a white level value above the actual peak white level in the image, things get broken.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 09:12:10 AM
edit:  fixed graphs.   All test images taken with 1/13s shutter and within 2 minutes, so temperature influence should be minimal.

(http://i.imgur.com/ZzvJNg0.png)
(http://i.imgur.com/eE7zlBT.png)
(http://i.imgur.com/oiRVFuH.png)

For dynamic range I averaged the white level of each ISO set.  With normal ISO 3200 and Canon pulled ISO 3200, the white level was a little higher then normal so these were not averaged and were calculated separately.  I'll paste the data in numbers a little later.

Quote from: Audionut on January 11, 2014, 09:56:37 AM
Looks like the ADC is clipping 0.7EV before sensor saturation.

I was a little overzealous.  0.3EV is closer to the truth.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 11:38:06 AM
Quote from: Audionut on January 12, 2014, 05:57:14 AM


        autodetect_black_level_calc(
            16, raw_info.active_area.x1 - 16,
            raw_info.active_area.y1 + 20, raw_info.active_area.y2 - 20,
            3, 16,
            &mean1, &stdev1
        );


I don't understand what values are shifting the scanning area, and what values are setting the scanning size.  I'd like it to scan almost the entire optical black area leaving a few pixels off left and right, and more pixels top and bottom.

Look at the definition of autodetect_black_level_calc:

static void autodetect_black_level_calc(int x1, int x2, int y1, int y2, int dx, int dy, int* out_mean, int* out_stdev)


and read it as "scan from x1 to x2, from y1 to y2, with downsampling factors dx and dy".

So, it has a safety margin on 16 pixels horizontally, on both sides.

However, the optical black size according to dcraw is 124, in ML it's assummed to be 126+20. This is where the bug comes from.

I changed skip_left and skip_right to 120+16 and it seems fine now (stdev the same with both dark frame and fully blown-out image, and no black bars on zebras). Need to run the tests again.

(just as with white level, the skip offsets should be overestimated a tiny bit, but not too much)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 03:15:17 PM
Some more findings:

1) 0x8880 is black level. Changing it doesn't seem to bring any improvement (DR decreases a little, not sure why).

2) The sweet spot seems to be where the white level is still the same as with the unmodified ISOs. Here it's around 0x320 => the real ISO under 100 might be ISO 80.

3) As you know from the dual ISO PDF, you can override the ISO gains from CMOS[0]. Now check the dynamic range of ISO 100 overriden to 1600, and ISO 1600 overriden to 100 (with tweak disabled). Here, the DR is roughy 0.1 EV higher whenever you select 100 in Canon menu.

This means dual ISO should work better as 100/1600 than as 1600/100. Anyone confirmed this experimentally? (though it's quite unlikely to notice a difference of 0.1 stops)

At ISO 1600, the difference vanishes when enabling the tweak. At ISO 100, the difference is still present, and it gets up to 1 stop when you select 25600 in Canon menu and 100 as override. So, there are some other settings used for these higher ISOs (and finding them may result in some more parameters that can be tweaked).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 05:13:02 PM
Quote from: a1ex on January 12, 2014, 03:15:17 PM
(...)
3) As you know from the dual ISO PDF, you can override the ISO gains from CMOS[0]. Now check the dynamic range of ISO 100 overriden to 1600, and ISO 1600 overriden to 100 (with tweak disabled). Here, the DR is roughy 0.1 EV higher whenever you select 100 in Canon menu.

This means dual ISO should work better as 100/1600 than as 1600/100. Anyone confirmed this experimentally? (though it's quite unlikely to notice a difference of 0.1 stops)

At ISO 1600, the difference vanishes when enabling the tweak. At ISO 100, the difference is still present, and it gets up to 1 stop when you select 25600 in Canon menu and 100 as override. So, there are some other settings used for these higher ISOs (and finding them may result in some more parameters that can be tweaked).
Some month ago (http://www.magiclantern.fm/forum/index.php?topic=5533.msg87217#msg87217) I noticed some strange things about DR, black & white levels; it was when using high ISO recovered by low ISO in Dual ISO. But it is with a 5D2, not a 5D3.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 06:24:18 PM
White level in LiveView is set to 15000 as a one-size-fits-all code. If the true white level is higher than that, it can't be wrong by more than 0.14 stops. If the true white is 15600, the error is roughly 0.07 stops.

You can always squeeze this extra DR with exiftool.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 06:34:15 PM
Quote from: a1ex on January 12, 2014, 03:15:17 PM
1) 0x8880 is black level. Changing it doesn't seem to bring any improvement (DR decreases a little, not sure why).

Raising it looks to improve things here.  Raising it with Canon ISO seems to clip the RGBG channels earlier, but you can offset it with (0x8882-0x8888)  ;)


All values below RGBG. 
Normal ISO
(0x8882-0x8888)
(0x8882-0x8888)+(0x8880)


Optical black average/SD
2048.0/7.27  2047.8/7.22  2048.0/7.10  2048.17.30
2047.9/5.63  2047.9/5.54  2048.1/5.49  2048.05.62
3932.5/5.59  3932.3/5.52  3923.8/20.4  3932.5/5.55    //Blue channel borked in OB


Here, I measured the levels/SD in the RGB patches in a colorchecker passport.  R value from R patch and so on.
2297/28.9  4374/45.0  1741/25.7  4344/44.6
1778/22.8  3339/33.7  1330/20.1  3354/34.8
1894/23.3  3454/31.0  1424/19.9  3469/32.0



Here I measured the RGBG levels/SD from the yellow colorchecker patch. (Brighter patch)
7962/50.1  10992/70.6  3724/24.5  10943/71.9
6633/39.3  8922/55.3   3336/19.0  8946/55.0
8624/39.1  10905/49.8  5313/18.8  10931/50.7


With the blue channel giving strange reading in the OB, I measured the black colorchecker patch.
2413/12.7  2683/20.8  2361/11.9  2885/20.0
2309/9.89  2610/15.9  2286/9.0   2548/15.8
4109/10.6  4541/14.5  4260/9.45  4585/15.4


Biggest difference looks to be in the green channel.  Wonder if there are other registers that perform a similar action?
This register is adjusting OB level, so would appear to be digital. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 07:32:24 PM
From what I've read in the ADTG datasheet (look in the dual ISO PDF at page 15), it's done by a feedback loop (there is a DAC somewhere inside). So I think black level is analog.

So... raising the black level could reduce the noise in midtones?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 08:29:39 PM
It looks to increase the SNR through the entire exposure.  I guess we can exploit it by reducing the gain (highlight biased?) with (0x8882-0x8888) and preventing saturation.

These maybe of interest:

http://documents.stsci.edu/hst/acs/documents/handbooks/DataHandbookv6/acs_Ch42.html
http://documents.stsci.edu/hst/acs/documents/handbooks/DataHandbookv6/acs_Ch43.html
http://www.stsci.edu/hst/wfc3/documents/handbooks/currentDHB/wfc3_Ch52.html

Rodger Clark has Gain e/DN values.  Here for the 5D3: http://www.clarkvision.com/articles/evaluation-canon-5diii/


Also, the standard (0x8882-0x8888) values seem to vary slightly on startup.  Haven't nailed down what causes the changes. 

When adjusting, I've been keeping the slight offsets in each column rather then setting all values the same.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 12, 2014, 10:34:32 PM
I've been trying to figure out how increasing the black level via that register is keeping a similar/smaller Standard Deviation.

It must be clamping the black value up to 2048, and hence by clamping it up further (stronger feedback), it reduces noise further.

Quote

Negative feedback loops
    When the fed-back output signal is out of phase with the input signal. This occurs when the fed-back signal is anywhere from 90° to 270° with respect to the input signal. Negative feedback is generally used to correct output errors or to lower device output gain to a pre-determined level. In feedback amplifiers, this correction is generally for waveform distortion reduction or to establish a specified gain level. A general expression for the gain of a negative feedback amplifier is the asymptotic gain model.

Positive feedback loops
    When the fed-back signal is in phase with the input signal. Under certain gain conditions, positive feedback reinforces the input signal to the point where the output of the device oscillates between its maximum and minimum possible states. Positive feedback may also introduce hysteresis into a circuit. This can cause the circuit to ignore small signals and respond only to large ones. It is sometimes used to eliminate noise from a digital signal. Under some circumstances, positive feedback may cause a device to latch, i.e., to reach a condition in which the output is locked to its maximum or minimum state. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 10:47:33 PM
Do you have a tool that can plot a signal-noise graph like this?

(http://a1ex.magiclantern.fm/bleeding-edge/profile-5D3-iso100.png)

I did this one a while ago by taking many pictures of a blank wall at different shutter speeds. I know darktable guys did such graphs from a single image (http://www.darktable.org/2012/12/profiling-sensor-and-photon-noise/), but they plotted it on linear axes, which is complete nonsense (they buried the midtones and shadows into a tiny corner and their graph covers well only the highlights).

If not, I can write a module that plots this kind of graphs directly in the camera, from a defocused test picture.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 10:50:07 PM
Quote from: a1ex on January 12, 2014, 06:24:18 PM
White level in LiveView is set to 15000 as a one-size-fits-all code. If the true white level is higher than that, it can't be wrong by more than 0.14 stops. If the true white is 15600, the error is roughly 0.07 stops.
1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.

Quote from: a1ex on January 12, 2014, 06:24:18 PMYou can always squeeze this extra DR with exiftool.
Thanks for the tips, I'll see how to do that.

Actually I'm trying to produce a 'very' RAW for scientific imaging, astro-photography or other special imaging by record a RAW with the minimal operations done on. It means the best bias/offset matching for the RON + FPN sigma at the optimal analogic gain etc. (no multiple ISO choice in this case so no way for regular photography). I don't fully understand how to use datas in ADTG datasheet yet, but it looks we are just at the beginning of what we can do with ML!

If you want I can do test on 5D2, just ask.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 10:52:46 PM
Quote from: SpcCb on January 12, 2014, 10:50:07 PM
1792 -> 15000 / 1024 -> 15600 => 1368 ADU reduction in the range; so more than 8% of reduction.

1792 is Canon's choice.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 11:07:18 PM
Quote from: Audionut on January 12, 2014, 10:34:32 PM
I've been trying to figure out how increasing the black level via that register is keeping a similar/smaller Standard Deviation.
(...)
Over all sigma ADU values on a short exposure is ≈ RON + FPN + offset.
So if offset > sigma/2 ; sigma will always be the same. You can change as you want the pedestal, it will not affect sigma.
And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).
(I supposed a symmetric deviation for sigma here, to be simple)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 11:09:09 PM
Quote from: a1ex on January 12, 2014, 10:52:46 PM
1792 is Canon's choice.
I confirm not.
Canon offset is 1024 ADU on 5D2.
Here is a RON + FPN profil:

(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_ron-profil-nominal-inv-gain.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 11:10:56 PM
Quote from: SpcCb on January 12, 2014, 11:07:18 PM
And if offset <= sigma/2 ; sigma could be better but with information lost (low values clipping).

I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 11:12:49 PM
Quote from: SpcCb on January 12, 2014, 11:09:09 PM
I confirm not.
Canon offset is 1024 ADU on 5D2.

LiveView is not the same as photo mode.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 12, 2014, 11:29:31 PM
Quote from: a1ex on January 12, 2014, 11:10:56 PM
I suppose you are talking about something else. We are changing the reference value for the black feedback loop (which is an analog bias), we are not changing black level in exiftool.
Indeed, I speak about analogic bias. We are OK.

Quote from: a1ex on January 12, 2014, 11:12:49 PM
LiveView is not the same as photo mode.
Ah, I did not know there's specific pedestal config in LiveView. Au temps pour moi.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 12, 2014, 11:38:06 PM
Okay, but the bias is around 1024...2048 and sigma is around 8.

I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: poromaa on January 12, 2014, 11:55:56 PM
interesting!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 13, 2014, 01:27:26 AM
Quote from: a1ex on January 12, 2014, 11:38:06 PM
(...)
I guess a larger bias could mean stronger signal for the amplifier that comes after the DAC that clamps black level. But if you raise the bias too much, you are forced to lower the ADTG gain, and this will introduce quantization noise. Probably there's a sweet spot somewhere.
Indeed again, as I said there's a relation between all these parameters.
The optimal config is easy to gess, on the paper, but hard to implement. I'm stuck at this point right now.

In a scientific view of all of this, the best choice will be:
_. the lower pedestal value where RON + FPN sigma is contained therein
[current original Canon bias is too high, because have to fit with all ISO; for example with 5D2 it could be N-times lower for ISO where gain <1 (in my graphic just up you can see it could be ~10x lower)]
_. the more closer analogic gain than 1
[but in real world and for usual photography, gain has to be <1 to equiv low ISO sensibility, so we should use a matrix for all ISO/gain/bias optimal values]
_. the lower digital gain
[in fact if we could have a matrix for all ISO up to gain <1 where DG is null it will be optimal, as it done for audio signal]

Please not that I ever do most of these optimizations in post for my work, I could demonstrate/detail/explain parts if need.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 13, 2014, 04:43:46 AM
Quote from: a1ex on January 12, 2014, 10:47:33 PM
Do you have a tool that can plot a signal-noise graph like this?

I've been manually entering the numbers in a spreadsheet and using it to generate the graphs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 13, 2014, 04:06:47 PM
gnuplot (http://www.gnuplot.info/) is useful for that.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 13, 2014, 04:35:00 PM
Made some progress in understanding how ISO 160x are implemented on 5D3 photo mode. This does not apply to LiveView / raw video.

I've looked at what DIGIC registers get changed between ISO 160 and 200 (you remember them from the good old digic poke, FPS override and all sorts of funky effects), and found this one: 0xc0f0819c = 0xC4C for full-stop ISOs and 250 multiples, and 0x8F0 for 160 multiples.


                                        black    stdev    white   DR
ADTG gains default:
ISO 160:                                2047     5.42     13307   11.02
ISO 200:                                2047     6.86     15283   10.91
ISO 200 with 0xc0f0819c = 0x8F0:        1187     6.80     15283   11.02
ISO 160 with 0xc0f0819c = 0xC4C:        2734     5.50     13307   10.91

ADTG gains changed to 0.85 x default (right above the point where white level begins to decrease):
ISO 160:                                2047     4.56     13307   11.27
ISO 200:                                2047     5.71     15283   11.18
ISO 200 with 0xc0f0819c = 0x8F0:        1187     5.71     15283   11.27
ISO 160 with 0xc0f0819c = 0xC4C:        2734     4.61     13307   11.16

Changed 8880 to 0xb04 to bring the black level back to 2047:
ISO 200 with 0xc0f0819c = 0x8F0:        2047     5.88     15283   11.14


You draw the conclusions.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 13, 2014, 10:10:03 PM
Is it possible to set custom value for 0xc0f0819c ?
Ex.
ISO 200 with 0xc0f0819c = 0xAA4 => offset = 1024 ADU
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 13, 2014, 10:40:11 PM
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.

When changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes). Also, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 14, 2014, 12:25:30 AM
Quote from: a1ex on January 13, 2014, 10:40:11 PM
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.
Nice!

Quote from: a1ex on January 13, 2014, 10:40:11 PMWhen changing this register, the image looks just as bright as the original ISO (unlike with ADTG gain, where overall brightness changes).
Agree, the overall brightness should not change.
In the same time pseudo-blacks should be less noisy because of lower RON levels and DR should be increased because of larger range between RON and max level.

Quote from: a1ex on January 13, 2014, 10:40:11 PMAlso, both dcraw/ufraw and Adobe DNG converter handle the modified black level properly.
Yes, but it's better to set that by analogic, at the source. Numerical post optimization with a single source frame is equivalent to stretch DR etc. Some informations are not record in the source so can't be restored after.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 14, 2014, 02:13:17 AM
Quote from: a1ex on January 13, 2014, 10:40:11 PM
Yes, I can offset the black level all the way to 0 and even below. This register is called SaturateOffset in Canon firmware.

Changing 8880 to 0x0 produces this.
(http://i.imgur.com/ZPc5RFZ.png)

Clearly (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95703#msg95703), it's having a bigger effect through the exposure on RB then G.

Quote from: SpcCb on January 14, 2014, 12:25:30 AM
Agree, the overall brightness should not change.

Most software has enough controls over exposure, I don't see it would matter.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 14, 2014, 07:29:01 AM
That's why I need a tool to plot the entire SNR curve, not just the black/noise/white levels.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 14, 2014, 08:24:00 AM
I couldn't find anything with a google search.  The only one I know of is imatest (http://www.imatest.com/docs/multicharts-noise/), but it isn't cheap.

The trial version might be enough for what you need.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilguercio on January 14, 2014, 10:29:01 AM
I got my 6D and digic poke, if you need some data i can try and provide you some figures.
Just let me know the procedure.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 14, 2014, 11:02:27 AM
Yes, the last graph from their page is what I'm looking for. If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.

So, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):

(http://a1ex.magiclantern.fm/bleeding-edge/ob-histo-200.png) (http://a1ex.magiclantern.fm/bleeding-edge/ob-histo-160.png)

The histogram pattern does not change with either ADTG gains or SaturateOffset.

@ilguercio: for the beginning, you can start here (http://www.magiclantern.fm/forum/index.php?topic=9867.msg95354#msg95354) and see if you can follow the discussion and arrive at similar numbers. So far, we are just tweaking parameters kinda blindly.

"If we knew what it was we were doing, it would not be called research, would it?"
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilguercio on January 14, 2014, 11:12:20 AM
Damn, i haven't had a compiling environment since i bought this new laptop.
I'll try to get it running tomorrow and meanwhile i can try to understand what is exactly involved.
I need to learn all this stuff, so far what seems to be the trick?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 14, 2014, 01:30:40 PM
http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/

QuoteBecause photon noise is an inherent property of CCD signal detection, which cannot be reduced by camera design factors, it essentially represents a "noise floor" that is the minimum achievable noise level, diminishing in relative effect as photon flux increases. Consequently, it is desirable to operate an imaging system under conditions that are limited by photon noise, with other noise components being reduced to relative insignificance. Under low illumination level conditions (assuming dark noise is essentially eliminated by CCD cooling), read noise is greater than photon noise and the image signal is said to be read-noise limited. The camera exposure time (integration time) can be increased to collect more photons and increase SNR, until a point is reached at which photon noise exceeds both read noise and dark noise. Above this exposure time, the image is said to be photon-noise limited.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 14, 2014, 04:29:28 PM
Quote from: a1ex on January 14, 2014, 11:02:27 AM
(...) If it runs on the camera from a defocused test image (without test charts or other special requirements), that's even better, since I can just tweak parameters and see the result right away.
Yes, it's better to run tests on a plain surface without shape because all variations will be include in sigma.

Quote from: a1ex on January 14, 2014, 11:02:27 AMSo, I started to write my own module. @SpcCb, I bet you will like these graphs a lot (ISO 200 vs 160):
(...)
Useful tool! (I dream to make tools like that!)
We can well see how is the population around the 2048 ADU pedestal. Is the vertical scale is the same?
If yes, for ISO 160 it clearly shows less population. Some vergences too; RON harmonics. I'll see to publish a RON map, sometimes it's more clear with grfx.

Quote from: Audionut on January 14, 2014, 01:30:40 PM
http://www.microscopyu.com/tutorials/java/digitalimaging/signaltonoise/
It confirms what I explained posts ago. ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 14, 2014, 04:51:40 PM
Vertical scale is "to fit", obviously, sum(all these bars) = 100% ;)

If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 15, 2014, 05:03:47 AM
Quote from: Audionut on January 12, 2014, 08:29:39 PM
Also, the standard (0x8882-0x8888) values seem to vary slightly on startup.  Haven't nailed down what causes the changes. 

Photo mode shows

ADTG2-8882  0x41c
ADTG2-8884  0x41a
ADTG2-8886  0x41a
ADTG2-8888  0x41c
ADTG4-8882  0x41a
ADTG4-8884  0x41a
ADTG4-8886  0x41c
ADTG4-8888  0x419


Exposure does not have any effect.  Register values always remain the same.  Whether the values change during exposure and reset after exposure is beyond my capabilities.  ISO does have an effect however, and I've plotted the values and put it into pastebin cause I suck at doing it with the forum code tag.

http://pastebin.com/jtND2EJd

In photo LV, there is a pattern, and the crux point seems to be EV 5 (http://www.fredparker.com/ultexp1.htm#EXPOSURE%20FACTOR%20RELATIONSHIP%20CHART%20B) (ISO 100).  ie:  1/20s f/1.4 produces one set of register values and 1/15s f/1.4 (EV 5) produces another.  This is reproducible by changing aperture.  So above EV 5, the register values retain their values, below EV 5 the register values change in a 1 stop pattern.  ISO produces the same effect as a 1 stop change in shutter/aperture.

I've plotted that here (http://pastebin.com/hJDzxZHb).  ISO 100 results = ISO 100 - 1/30s - f/1.4

I've labelled the changes with ISO, but as mentioned 1 stop with shutter/aperture produces the same.  So for instance, you get the same stated ISO 100 results with ISO 100 - 4s - f/16.0 and with ISO 12800 - 1/3000s - f/1.4

This is with 1/2 stop exposure settings in Canon menu.  Interestingly, the crux point shifts with 1/3 stop exposure settings.  Here, 1/20s f/1.4 produces the change, and it sticks to a 1 stop pattern from there.

Notice how ISO 800 produces lower register values then ISO 400 in both photo mode and LV.  Wonder if this has something to do with unity gain

I've been of the opinion that the register values translate to gain.  ie:  lower values = lower gain.  And this makes sense when you consider the values rise with increasing ISO.  This wouldn't appear to be the case though considering the values also rise with increasing exposure.

ISO 12800 is the last ISO to make a change.  In the previous pages I should have labelled ISO 3200 as the last useful analog ISO.

Quote from: a1ex on January 14, 2014, 11:02:27 AM
So, I started to write my own module.

Nice work.  I can plot the same with RawDigger and output CSV files containing the values like so,
(http://i.imgur.com/C43rNq3.png)

https://dl.dropboxusercontent.com/u/34113196/ML/Noise%20samples/_46A5329-rawdata.csv
Obviously, this way I have to remove the card, load it into Rawdigger, etc, etc.  Note:  That's my dodgy graph!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 15, 2014, 08:47:05 AM
Quote from: Audionut on January 15, 2014, 05:03:47 AM
http://pastebin.com/jtND2EJd

Same values here, +/- 3 units.

Quote
In photo LV, there is a pattern

This is called ExpSim. LiveView normally is limited to 1/30 exposure time.

Since the ADTG gain does not change the histogram pattern, and the histogram is mostly without gaps even with this gain at 3x the original value, I believe this gain is analog amplification. However, bumping the gain at 5x does not result in lower noise:

stdev at 1x: 6.71
stdev at 5x: 34.02
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 15, 2014, 08:57:48 AM
Quote from: a1ex on January 15, 2014, 08:47:05 AM
This is called ExpSim.

Well that's disappointing.  For a moment there I thought I was smart!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 15, 2014, 01:46:38 PM
My current theory about 5D3 ISOs:

I believe these parameters are relevant (there may be others):
- ADTG gains (0x8882 ... 0x8888)
- ADTG black (0x8880)
- SaturateOffset (0xc0f0819c), related to ADTG gains
- 0xc0f37ae4, af0, afc, b08 (digital gain - this one changes the shape of the histogram)
- c0f37ae0/aec/af8/b04 (offset for black and white levels)

The last two registers can be used to configure the range of recorded data (Canon default is from 2048 to a little over 15000 for full-stop ISOs). Changing them will only influence the round-off errors. No big deal.

ADTG black is uncertain, I'll leave it unchanged for now.

ADTG gains and SaturateOffset can be used to recover some more highlight detail. They are tightly coupled (you can recover the same highlight detail from one, from another, or split between them). So, changing only ADTG gain is enough (because the other one runs out of range much quicker).

At some point, there's no more highlight detail to recover. When this happens, the white level will begin to decrease. Let's call this the sweet spot, and it can be found easily with binary search, for example.

Now, remember the ISO definition from DxO (http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity): ISO is all about the exposure necessary to saturate the sensor. Since we could change the clipping point to capture more highlights, we can effectively got some lower ISOs. They are just as noisy in shadows as their full-stop counterparts, but they include a little more highlight detail.

So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).

To confirm the formula, I've checked Canon ISOs (starting from ISO 50):
ISO 100, 100, 125, 185, 200, 250, 375, 400, 502, 750, 800, 1005, 1495, 1595, 2008, 2993, 3188, 4015, 5988, 6378, 8035, 12008, 12790, 16110, 20295.
After this, the digital gain register gets changed from 0x200 to 0x10200 and my formula shows 12790, so... 25580.

Take it with a grain of salt, I need to double-check the math and confirm the results with some test shots. I plan to post some code to play with, and some more details.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 15, 2014, 05:00:46 PM
Quote from: a1ex on January 15, 2014, 01:46:38 PM
My current theory about 5D3 ISOs

It would be terrific if ML could result better dr/noise than Canon, this would be the ultimate killer feature - of course only if it makes its way to the 6d :->  ...

... I very much suspect that something can be done about this since the 1Dx has a nearly linear dr/iso curve, and Canon most likely doesn't use a different sensor tech on that model - instead the theory on CR is that they applied a high amount of fine-tuning to get this difference vs. non-1d cameras.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 15, 2014, 06:16:18 PM
500D default values :

ISO             100    200    400    800    1600    3200    6400    12800
ADTG3[0]    149    149    148    148    142      142      142      142
ADTG3[3]    14a    14a    14a    14a    145      145      145      145
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on January 15, 2014, 06:54:32 PM
Quoteof course only if it makes its way to the 6d

Its very similar to 5DIII so it should. Hopefully it makes it to all cameras.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: RenatoPhoto on January 15, 2014, 08:37:13 PM
Very, very interesting.  Nice work guys!!  As I said before you guys should be paid by Canon R&D department!  Maybe in Bitcoins?  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 16, 2014, 02:58:13 AM
So is there any truth in this http://www.clarkvision.com/articles/iso/index.html
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 03:07:18 AM
Quote from: Audionut on January 10, 2014, 03:36:04 AM
Contrary to popular belief, ISO is post exposure gain.  Exposure is the light hitting the sensor which is a result of lens diameter, shutter and aperture.

So by having a lower gain from the electronics,

Quote from: a1ex on January 15, 2014, 01:46:38 PM
So, assumming ISO 100 is really ISO 100, and setting ADTG gain to the sweet spot (found by binary search), I believe the new ISOs are, at least on my 5D3:
ISO 77, 150, 295, 595, 1175, 2267, 4360 and 8520 (based on ISO 100 ... 12800).

Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure).

So even though the new ISOs,

Quote from: a1ex on January 15, 2014, 01:46:38 PMare just as noisy in shadows as their full-stop counterparts

We can ETTR a little more.  Which means we can increase the SNR of the shot noise (http://www.magiclantern.fm/forum/index.php?topic=9285.msg93716#msg93716).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 16, 2014, 05:45:49 AM
Quote from: a1ex on January 14, 2014, 04:51:40 PM
(...)
If you take some gaussian random noise, round it, then scale with something like 0.9, its histogram will have the same kind of spikes. If you scale it by 1.1, it will have gaps. So, I believe these histograms show the results of some digital manipulation.
This histograms show the RON.
It's why you don't see a gaussian random noise distribution.

I'm a bit busy right now but in a couple of hours I could publish some materials.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 06:04:35 AM
Quote from: SpcCb on January 16, 2014, 05:45:49 AM
This histograms show the RON.

Yes, but the interest is in the distribution.  Specifically, the effects by digital manipulation.

Quote from: a1ex on January 14, 2014, 11:02:27 AM
The histogram pattern does not change with either ADTG gains or SaturateOffset.

a1ex called it pattern.  The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain.  If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 16, 2014, 08:08:20 AM

octave:1> noise = round(randn(100000,1) * 7); # Gaussian noise of similar magnitude as Canon's ISO 100
octave:2> noise_positive_gain = round(noise * 1.1); # apply a small positive digital gain
octave:3> noise_negative_gain = round(noise * 0.9); # apply a small negative digital gain
octave:4> hist(noise_positive_gain, -30:30)
octave:5> hist(noise_negative_gain, -30:30)


(http://a1ex.magiclantern.fm/bleeding-edge/noise-pos.png) (http://a1ex.magiclantern.fm/bleeding-edge/noise-neg.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 12:17:47 PM
Quote from: Audionut on January 16, 2014, 03:07:18 AM
Means that we can now increase the exposure (photons hitting the sensor) before saturation (overexposure)........................We can ETTR a little more.

a1ex showed me how to calculate exactly how far we can ETTR with these new ISOs vs Canon base ISOs.  Which is obviously better then random guesses from histograms.


100 0.377 EV
200 0.415 EV
400 0.439 EV
800 0.427 EV
1600 0.445 EV
3200 0.497 EV
6400 0.554 EV
12800 0.587 EV


Not only can we safely increase the exposure by 1/3, there's also some headroom.

edit:  This is over and above the DR gains from the tweak.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilguercio on January 16, 2014, 12:23:26 PM
Half a stop at 12800? That's goooooood.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 12:38:36 PM
Quote from: ilguercio on January 16, 2014, 12:23:26 PM
Half a stop at 12800? That's goooooood.

It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilguercio on January 16, 2014, 12:47:53 PM
Quote from: Audionut on January 16, 2014, 12:38:36 PM
It's bound to provide a little leeway, but if you're using ISOs above 100, you should (probably) only be doing so because you're already at the limits of exposure.
I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 16, 2014, 01:01:33 PM
Quote from: ilguercio on January 16, 2014, 12:47:53 PM
I guess everybody knows that if you have a pretty static scene and a tripod, iso 100 is the way to go.
At least, they should :)

I dunno, given the iq of the 6d using iso ~200 is very good esp. if ML adds a little more dynamic range than to iso 100, up to iso 400 the read noise halves which more or less drowns the dynamic range. With longer exposure time (larger dof for ~macro or bad lighting) in the seconds my observation you're getting another noise source, so raising the iso a little might not be a bad idea.

Quote from: ilguercio on January 16, 2014, 12:23:26 PM
Half a stop at 12800? That's goooooood.

Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800? The dynamic range 6400->12800 drops by 1 ev :-p
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 01:08:34 PM
Shot noise (http://en.wikipedia.org/wiki/Shot_noise), a significant contributor to total noise, is the square root of the number of photons hitting the sensor.

So noise actually increases with increased photons, but it's the SNR that's more important, and this also increases with increased photons hitting the sensor.

ISO doesn't increase the number of photons hitting the sensor.

ISO is useful in Canon cameras, because the ADC is noisy.  So where you can't saturate the pixel wells with exposure settings (shutter/aperture) and generate a large voltage from the sensor, you can use ISO to apply gain to the signal to boost it to the saturation point of the ADC, and hence, increase the SNR through the ADC.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 16, 2014, 01:17:58 PM
Quote from: Marsu42 on January 16, 2014, 01:01:33 PM
Well, yes, but that's digital push anyway - my understanding is for raw shooting you're equal or better off with iso 6400 and underexposing, or is there something to be gained with 12800?

It's analog push, not digital. On 5D3, from 1600 to 12800 you gain 0.22 EV in shadow noise (http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html) and for this you pay 3 stops of highlight detail.

On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).

Some quick results from 60D, assumming full-stop ISOs are real:

Base ISO        With low ADTG gain       Highlights gained
100             88                       0.18 EV
200             158                      0.34 EV
400             311                      0.36 EV
800             633                      0.34 EV
1600            1260                     0.34 EV
3200            2515                     0.35 EV
6400            2515                     1.35 EV


(note that ISO 100 on 60D is implemented more like 160 than like 200 multiples; I didn't check saturation level yet)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: PressureFM on January 16, 2014, 01:37:02 PM
Have anyone checked the 5D Mark II?

Or how would I do it?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 16, 2014, 01:43:05 PM
In theory, simply follow the discussion from the beginning (you need code compiling skils).

In practice, I could not find the equivalent registers on 5D2, but I've only tried for around 2 hours.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 16, 2014, 02:14:49 PM
Quote from: a1ex on January 16, 2014, 01:17:58 PM
On 60D, ISO 6400 is not digital push either. It's analog push from the ADTG gain (which we found to be analog from histogram analysis).

You're assuming that Canon is not doing something delibrate to hide the fact that it is actually a digital gain. I do not give Canon that much credit. If you scale by 4x from ISO1600, then dither (http://en.wikipedia.org/wiki/Dither) (i.e. apply gaussian noise to) the two LSBs, your histogram will still look the same, it will not have gaps.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 16, 2014, 02:24:23 PM
At ISO 160 multiples, the histogram does have spikes (including ISO 1250). So if they were trying to hide it, I guess they would have done it at these ISOs too.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 16, 2014, 02:56:59 PM
I thought 160 was pulled, not pushed?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 03:19:29 PM
In LV you should be able to auto adjust ADTG for maximum white point, correct?  So in manual mode you can choose your exposure settings, and ML applies ADTG gain as needed. 

An analog auto ISO only limited by the accuracy of the white point detection.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 16, 2014, 03:24:18 PM
LV ADTG settings are only relevant for raw video (and I didn't study them yet). In photo mode you get ExpSim, so the only interesting thing here is to match the clipping points for ETTR.

Quote from: dmilligan on January 16, 2014, 02:56:59 PM
I thought 160 was pulled, not pushed?

Yes, but I was talking about the attempts to hide digital trickery.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 06:37:19 PM
What would a development thread be without shots of boring stuff!

Canon ISO 100 (1,055 KB)
(http://i.imgur.com/Sm3cCHQ.jpg)

ADTG tweak (853 KB)
(http://i.imgur.com/tInNviT.jpg)

ADTG tweak + Dual_ISO 100/1600 (437 KB)
(http://i.imgur.com/NiuTETo.jpg)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 16, 2014, 06:44:32 PM
Quote from: Audionut on January 16, 2014, 06:37:19 PM
What would a development thread be without shots of boring stuff

Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 16, 2014, 07:05:10 PM
ML iso is much better!!!! Cant wait! I will always have ML iso on if it makes it to the nightly builds. (i sure it will be a little more complex then turn ml iso on but i hope its that simple) You guys rock!!!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Steve Kahn on January 16, 2014, 08:07:01 PM
Quote from: Marsu42 on January 16, 2014, 06:44:32 PM
Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.

I was going to ask the same question.  I have a 5D3 and have never seen that.  Also, it is not there on the dual ISO shot for some reason.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 16, 2014, 08:26:09 PM
Have you ever tried to shoot a light bulb outside at night, without clipping the highlights, then recover the shadows around it? If there are not a lot of details in the shadows you will see the sensor noise (not banding). I can assure you if you shot this exact situation with your 5d3 or 6d, recovering the shadows will give you this result. Its not in the dual iso image because thats what dual iso was designed to do. To give you better highlight and shadow detail.

These ML iso options will be a great advancement in the mysteries surrounding canon iso and cleaner, better iso.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 08:37:28 PM
That's deep into the shadows.

(http://i.imgur.com/rJ8jVQt.jpg)

http://graphics.stanford.edu/courses/cs448a-10/sensors-noise-14jan10-opt.pdf  <---  Page 43

http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
http://www.clarkvision.com/articles/evaluation-canon-6d/index.html
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on January 16, 2014, 08:40:17 PM
WOW.... that tweak and Dual ISO shot looks great!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 16, 2014, 08:43:16 PM
Especially considering where it came from, that is incredible.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 16, 2014, 09:52:37 PM
Quote from: hjfilmspeed on January 16, 2014, 08:26:09 PM
If there are not a lot of details in the shadows you will see the sensor noise (not banding).

I'd consider this noise pattern banding, I know that very well from the 60d where it appears very soon. At some point you might see it on any camera, that's why I was asking if this is usual or how much ev the shadows were raised to make it appear... and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 10:49:01 PM
Quote from: Marsu42 on January 16, 2014, 09:52:37 PM
and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Dual-ISO looks like it did fine  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Steve Kahn on January 16, 2014, 11:10:27 PM
Quote from: Audionut on January 16, 2014, 10:49:01 PM
Dual-ISO looks like it did fine  :P

At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: engardeknave on January 16, 2014, 11:21:34 PM
This is the problem with showing off any sort of impressive capture of a wide DR scene. Nobody really gets how bad it was before recovery. I may have to create a before/after section for my real estate photography.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 16, 2014, 11:23:15 PM
Quote from: Steve Kahn on January 16, 2014, 11:10:27 PM
At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   


Indeed there is.  I'll post some other comparisons and updated SNR graphs after sleep.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 16, 2014, 11:53:17 PM
Quote from: Audionut on January 16, 2014, 10:49:01 PM
Dual-ISO looks like it did fine  :P

No doubt, it's great and but this is really the type of shot made for it - but once you get details in the shadows and highlights it's a tough decision how much resolution to lose to gain a certain amount of dynamic range.

That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: arturochu on January 17, 2014, 12:49:32 AM
i don't understand 90% of this thread, but seeing so many "DR" talk makes me so excited about the progress of the topic haha.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on January 17, 2014, 01:14:28 AM
Yea that banding is on almost everything at higher ISOs, 6D, 7D, etc. After ACR or other post processing its mostly gone. Its hardest to remove it from LV dngs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 03:13:51 AM
Quote from: Marsu42 on January 16, 2014, 09:52:37 PM
Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.

On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.

I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 03:18:50 AM
Quote from: Audionut on January 16, 2014, 06:04:35 AM
Yes, but the interest is in the distribution.  Specifically, the effects by digital manipulation.

a1ex called it pattern.  The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain.  If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.

When you get this (5D2 distribution, but I think it's close to the 5D3):
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_sngl-frame_histo.jpg)

It is because of this (average of 300 frames with winsorized sigma clipping to reject random noises + bin2 + pix value x16):
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo_hi-ctrst.jpg)

With a 2D FFT we can better see how is the pattern:
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_sngl-frame_fft.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 04:41:23 AM
The banding pattern accounts for only a small part of these histogram distortions.

Proof:

ISO 100 dark frame (5D3), original and corrected
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso100.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso100-nobanding.png)

Histograms of horizontal and vertical banding correction (per-line and per-column)
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/hband.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/vband.png)

Image histogram before and after correction:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/hist0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/hist1.png)

How I've estimated the correction:

# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im);  fv = fv - mean(fv);

# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png

# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;

# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 05:32:56 AM
When I do:
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_sngl-frame_histo.jpg) * sub * (http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo_hi-ctrst.jpg)

I get this:
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo-sub_histo.jpg)

Of course it's not a perfect gaussian yet, but it's close. :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 05:42:53 AM
May I see a single dark frame minus the averaged one?

(I know, I could probably take 300 frames myself, get a PixInsight license and try it)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 05:57:00 AM
Ah, sorry, I don't have it right now. I just have the histograms of those frames (here in my last post).
But I could do it later. I'v planed to publish the same operation on a video RAW frame.


About Pixinsith, mh... You can use Iris (http://www.astrosurf.com/buil/iris/iris.htm) too, it's free.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 07:20:42 AM
Some new graphs with calibrated ML ISOs and fixed DR calculation.

(https://i.postimg.cc/CxHnY5LB/STD.png)
ISO Canon ML

100 7.18 5.48
200 7.58 5.58
400 8.25 6.01
800 9.45 6.98
1600 12.8 9.3
3200 19.4 13.9
6400 37.2 25.1
12800 63.4 42.6




(https://i.postimg.cc/FsSJWGMx/image.png)
ISO Canon ML

100 11.16 11.55
200 11.08 11.52
400 10.96 11.41
800 10.76 11.2
1600 10.32 10.78
3200 9.72 10.2
6400 8.78 9.35
12800 8.01 8.59



ML ISO 800 has greater DR then Canon ISO 100  8)  As a1ex points out on the next page though, this doesn't consider noise through the exposure (ie: your midtones).

Removed the inaccurate SNR graph.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 17, 2014, 07:48:53 AM
wow wow wow wow wow wow wow wow!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 07:56:48 AM
I would plot all these graphs on logarithmic scale (convert the units to EV). In linear plots, the differences appear much larger than they really are in practice.

How did you measure the SNR?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 08:47:13 AM
Quote from: a1ex on January 17, 2014, 07:56:48 AM
I would plot all these graphs on logarithmic scale (convert the units to EV). In linear plots, the differences appear much larger than they really are in practice.

Do you have an easy equation for doing that?

Quote from: a1ex on January 17, 2014, 07:56:48 AM
How did you measure the SNR?

saturation/std.dev
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 08:52:59 AM
1) log2
2) that's DR
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 08:56:33 AM
For DR I was

log2(saturation) - log2(std.dev)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 08:59:40 AM
...which is identical to log2(saturation/stdev)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 09:11:04 AM
So SNR would be the sqaure root of saturation/std.dev?

And I convert saturation and std.dev to EV?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 09:39:52 AM
SNR can be computed at any input (signal) level. Take a look at DxO's Full SNR graphs.

DR is about deep shadows, but it doesn't tell much about the midtones. The reason is that noise stdev also varies with signal level (it usually increases). Here's a SNR curve for 5D3 at ISO 100 (notice how SNR goes below the straight line, because noise gets higher):

(http://a1ex.magiclantern.fm/bleeding-edge/profile-5D3-iso100.png)

If you compare 60D and 5D3 on DxO for example, you will see the DR is identical at ISO 100, but the SNR curves are not. So, in shadows, both cameras are comparable, but in midtones you can see a difference.

In ETTR, I measure SNR at median signal value from the entire image (midtone SNR) and at 5th percentile (shadow SNR). This is not quite correct for bright images, but at low signal values (say under 7 stops, where ETTR normally works), the approximation is very good, as you can see on my graph.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 11:28:27 AM
I'll leave that for you (or someone else) to plot.  All this maths is making my head hurt!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 01:59:18 PM
Here's how to read the SNR from DxO:

1) Go to Full SNR graph, e.g. http://www.dxomark.com/Cameras/Canon/EOS-5D-Mark-III#measuretabs-6

2) Choose a reference level (say 18%) and read the SNR (in this example, 39.7 dB)

3) Remember the modified ISOs will let you expose a little more to the right, let's say by 0.4 stops. That means, your reference level on DxO graph will now be 18 * 2^0.4 = 23.75. Note that in your image, the reference level will be still 18% relative to your new white point (but this new white point just got higher than Canon default with 0.4 stops). New SNR: 40.9 dB.

4) SNR improvement at 18% level will be 1.2 dB = 0.2 stops if you shift the exposure to the right (so you keep the same amount of highlight detail). If you keep the same shutter speed and aperture, same scene, and develop at the same brightness in post, noise will not change (except for maybe some round-off errors).

So, the improvement in midtones is pretty small, but these midtones have little noise to begin with.

And here's a module that does a *rough* estimation of the SNR curve from a defocused image, and also computes the dynamic range:
raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))

The module does 3 types of analysis:
- optical black (it draws the noise histograms as the ones I've posted earlier in this thread)
- dark frame (same, but analyzes the entire image - very slow)
- SNR curve (take a defocused picture and it plots an approximation of SNR as a function of signal level)

I'd like you to run some tests with the raw_diag module and check the repeatability of the measurements (maybe compare it with DxO, RawDigger or whatever, or just review the math). For dynamic range, make sure you have something overexposed in the frame (e.g. point the camera to a light bulb). You may also post tables with dynamic range at each ISO from your camera (which should be comparable, since they will be computed with the same algorithm).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 04:21:08 PM
A quick play.

Canon (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6530.CR2) vs ML (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6533.CR2) - Last stop before Canon saturation.

(https://i.postimg.cc/8PZ7c9gw/VRAM0.png) (https://i.postimg.cc/nhQnxPFX/VRAM3.png)

Rawdigger results.
Canon ML
White 14907 11223  <---Highest recorded white pixel
Std.Dev 9.04 6.78  <---Worse case measuring entire left optical black area
Std.Dev 7.32 5.49  <---Worse case cropping pixels either side of measuring area
DR 10.687 10.693  <---Measuring entire left OB
DR 10.992 10.997  <---Measuring cropped OB






Canon saturated.
(https://i.postimg.cc/J02WVbS3/VRAM1.png)

Canon vs ML - Last stop before ML saturation.  I could have reduced ML negative gain here to push the white point.
(https://i.postimg.cc/QtNLF43k/VRAM2.png) (https://i.postimg.cc/k5Q96Rt9/VRAM4.png)

ML Saturated.
(https://i.postimg.cc/Bv94nBbM/VRAM5.png)

Canon vs ML.
(https://i.postimg.cc/9M1h13pn/VRAM6.png) (https://i.postimg.cc/j5RTzqzf/VRAM11.png)
(https://i.postimg.cc/65PBDXXs/VRAM7.png) (https://i.postimg.cc/9FsjLYdC/VRAM12.png)
(https://i.postimg.cc/7h6yWBtY/VRAM9.png) (https://i.postimg.cc/Kv127Hg0/VRAM13.png)
(https://i.postimg.cc/HWZmygjJ/VRAM10.png) (https://i.postimg.cc/rpNXF6vw/VRAM14.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: fiveservice on January 17, 2014, 04:28:37 PM
wow! when will this be released for 5D3?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 04:33:24 PM
To measure DR, you should have something overexposed in the image.

Also, white level of 16382 is likely a Canon bug (overflow in digital gain) and it will prevent a fair side-by-side comparison.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 04:52:22 PM
Quote from: a1ex on January 17, 2014, 05:42:53 AM
May I see a single dark frame minus the averaged one?
(...)
Here is the frames, with respective distributions and same screen range (960 - 1088 ADU):

Single frame @ 1250 ISO
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_sngl-frame.jpg) (http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_sngl-frame_histo.jpg)

Master offset @ 1250 ISO (average of 101 offset frames)
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo.jpg) (http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo_histo.jpg)

(Single frame @ 1250 ISO) - (Master offset @ 1250 ISO)
(http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo-sub.jpg) (http://www.bgimage.fr/publis_ext/ml/5d2_1250iso_mo-sub_histo.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 04:52:40 PM
The DR measurement was just to compare ML vs RawDigger.  Nothing serious.

Reducing negative gain (yes I was past calibrated result) fixed the WL, but here is a standard Canon ISO where ML reports a different WL to RawDigger.

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6566.CR2

(https://i.postimg.cc/Vk0V5ybc/canon.png)

Rawdigger
White 13956
Std.Dev 7.2
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 04:56:23 PM
This image is not overexposed, so I don't consider the white level relevant. As long as there's nothing overexposed in the frame, the DR reported is plain wrong.

ML doesn't just take the max value because there are hot pixels.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 05:06:21 PM
Quote from: a1ex on January 17, 2014, 04:56:23 PM
ML doesn't just take the max value because there are hot pixels.

Of course.  You explained that back on the first page  :(
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Steve Kahn on January 17, 2014, 06:21:09 PM
Quote from: Audionut on January 16, 2014, 11:23:15 PM

Indeed there is.  I'll post some other comparisons and updated SNR graphs after sleep.

When you do this test Audionut it would be beneficial if the camera were locked down on a tripod.  That way we can watch the noise in the blacks disappear as we go from Canon DR, to ML DR, to ML DR + Dual ISO.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 08:10:02 PM
ML vs Canon / 0EV settings
(https://i.postimg.cc/mZy6JN77/VRAM0.png) (https://i.postimg.cc/GmVMGM1n/VRAM1.png)


Imported into ACR, exposure levels for the 0EV frames were measured from the white patch, and all remaining frames were adjusted from these 0EV frames.  White balance set from 0EV ML frame.  dual ISO had it's white balance set separately and there was slight movement forward of the camera between dual ISO frames and all other frames.

ML spotmeter reads the grey patches at top of each image as being -4.7EV, -3.2EV, -2.2EV, -1.3EV, -0.6EV, 0EV, left to right.  Which puts the far left black patch in the -6EV shots @ -10.7EV below sensor saturation.

ML vs Canon / -1EV
(https://i.postimg.cc/m2SVpLfj/46_A6732-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/fRrc1kHc/46_A6739-as-_Smart-_Object-1.jpg)


ML vs Canon vs Dual_ISO 100/1600 / -2EV
(https://i.postimg.cc/bwWn3WJL/46_A6733-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/rwBdS4nP/46_A6740-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/HsWPgctR/46_A6748-as-_Smart-_Object-1.jpg)


-3EV
(https://i.postimg.cc/vTK8GRCN/46_A6734-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/5ysYvkTB/46_A6741-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/Mp2x5YwV/46_A6749-as-_Smart-_Object-1.jpg)


-4EV
(https://i.postimg.cc/44PhrmyY/46_A6735-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/SQgJg318/46_A6742-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/3xxMnf6y/46_A6750-as-_Smart-_Object-1.jpg)


-5EV
(https://i.postimg.cc/8zJJJ4Hv/46_A6736-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/XJp7m7zC/46_A6743-as-_Smart-_Object-1.jpg) (https://i.postimg.cc/sgNLKj78/46_A6751-as-_Smart-_Object-1.jpg)

-6EV  -   Images slightly underexposed due to lack of latitude in ACR.
(https://i.postimg.cc/FRQLQS0L/46_A6737-as-_Smart-_Object-1.jpg)(https://i.postimg.cc/gk6m3TCG/46_A6744-as-_Smart-_Object-1.jpg)(https://i.postimg.cc/FsPCsrWR/46_A6752-as-_Smart-_Object-1.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 17, 2014, 08:22:01 PM
Quote from: Audionut on January 17, 2014, 04:52:40 PM
The DR measurement was just to compare ML vs RawDigger.  Nothing serious.

Reducing negative gain (yes I was past calibrated result) fixed the WL, but here is a standard Canon ISO where ML reports a different WL to RawDigger.

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A6566.CR2

(http://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Canon.BMP)

Rawdigger
White 13956
Std.Dev 7.2


Found this thread after a lot of searching after the Tweet "Psst..." etc.
Looks promising, 1/3 or 1/2 stop of extra dynamic range (and less noise).

But this material raises a lot of questions to me ?

How is the dynamic range calculated with the black level, white level and st.dev (I know what St.dev means, but can it in this context literally be seen as a noise indicator ?)
In the example above the black level is 2049 and the white level is 13919 and a Stdev. of 9,3.
Does this mean that the sensor at that moment can roughly measure from values 2049=black and 13919= white which results in 11870 usable values (and are these number literally counted photons?)
What is the formula for the dynamic range with these numbers ?

Does anyone know already if this stuff is related to the temperature of the sensor ?





Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 17, 2014, 08:39:21 PM
Looking at the latest pictures of Audionut.

Looks to me  ??? that there's more chroma noise in the ML pictures, but there should be less noise in the ML pictures according to the numbers in this topic ?

Sorry, the first ones are the ML picture and the second ones are the canons  ::)
Clearly less chroma noise in the ML pictures  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 08:47:27 PM
Quote from: Levas on January 17, 2014, 08:22:01 PM
How is the dynamic range calculated with the black level, white level and st.dev (I know what St.dev means, but can it in this context literally be seen as a noise indicator ?)........................What is the formula for the dynamic range with these numbers ?

Quote from: Audionut on January 17, 2014, 08:56:33 AM
For DR I was

log2(saturation) - log2(std.dev)

Quote from: a1ex on January 17, 2014, 08:59:40 AM
...which is identical to log2(saturation/stdev)


Quote from: Levas on January 17, 2014, 08:22:01 PM
(and are these number literally counted photons?)

http://en.wikipedia.org/wiki/Color_depth


Quote from: Levas on January 17, 2014, 08:39:21 PM
Looks to me  ??? that there's more chroma noise in the ML pictures, but there should be less noise in the ML pictures according to the numbers in this topic ?

This tweak reduces read noise only.

http://www.cambridgeincolour.com/tutorials/image-noise.htm
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 08:53:58 PM
Quote from: a1ex on January 17, 2014, 04:33:24 PM
Also, white level of 16382 is likely a Canon bug (overflow in digital gain) and it will prevent a fair side-by-side comparison.

Why is this?  14bits should give 16384 with a black level set at 2048?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 17, 2014, 09:02:31 PM
http://en.wikipedia.org/wiki/Color_depth

Of course photons are measured through analoge signal... :-[ ::) stupid me, these are numbers after AD conversion, 14 bits so maximum of 16384.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 09:09:05 PM
Quote from: Audionut on January 17, 2014, 08:53:58 PM
Why is this?  14bits should give 16384 with a black level set at 2048?

The 14-bit limit is 16383. Since our white level is way too close to this limit, it's quite likely that the real white level may be above that.

I can't get this white level on my camera though (it varies with the aperture, but not that much).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 09:38:49 PM
Quote from: Audionut on January 17, 2014, 08:53:58 PM
Why is this?  14bits should give 16384 with a black level set at 2048?
16384 - 2048 does not give usable 14 bits.
And in fact the ADU range is (16383 - ?*) - (2048 - RON) with 5D3.

Like a1ex I never get 16383 ADU with 5D2 too. The most high value I can reach is 15600 ADU.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 09:41:58 PM
Quote from: SpcCb on January 17, 2014, 09:38:49 PM
The most high value I can reach is 15600 ADU.

Aperture effects white level.

f/1.4  16382
f/1.6  16316
f/1.8  16182
f/2.0  15950
f/2.2  15769
f/2.5  15655
f/2.8  15463
f/3.2  15440
f/3.5  15383
f/4.0  15330

After this point, WL remains the same.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 09:44:46 PM
???
Maybe a 5D3 special feature (?).
I'll try with f/0 to see that...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 09:48:48 PM
Quote from: SpcCb on January 17, 2014, 09:44:46 PM
Maybe a 5D3 special feature (?).

I can use larger negative gain values for ADTG then a1ex before WL changes.

Quote from: SpcCb on January 17, 2014, 09:44:46 PM
I'll try with f/0 to see that...

On my 5D3, detaching the lens didn't effect the register that adjusts the gain.  It needs an electronic link from the lens.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 09:50:17 PM
I think it's a trick to adjust the brightness in the jpeg previeew. It's done by adjusting the digital gain registers, and as a side effect it changes the recorded range and the roundoff errors.

If it's only at f1.4, and has a safety margin at 1.6 and below, it's probably not overflowing (or if it is, it's only a tiny amount).

So, it should not affect our highlight recovery trick in any way other than round-off errors.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 17, 2014, 09:55:46 PM
500D (raw_diag)
F1.4   16382
F1.6   15788
F1.8   14918
F2.0   14173
F22    13431
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 09:56:36 PM
Quote from: Audionut on January 17, 2014, 09:48:48 PM
(...)
On my 5D3, detaching the lens didn't effect the register that adjusts the gain.  It needs an electronic link from the lens.
Indeed, I use a custom chip where I can define f/whatIwant.

I can get on the 5D2:
f/5.6 -> 15764 ADU
f/1 -> 16383 ADU
f/0 -> 16383 ADU

Update: It depends of the ISO setting too, not only the f/stop.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 17, 2014, 10:03:06 PM
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 10:11:09 PM
Quote from: Greg on January 17, 2014, 09:55:46 PM
500D (raw_diag)

Make sure you are getting sensor saturation Greg.

Quote from: SpcCb on January 17, 2014, 09:56:36 PM
Update: It depends of the ISO setting too, not only the f/stop.

Not here.  If you are changing exposure settings with ISO, make sure you are getting sensor saturation.


Quote from: a1ex on January 17, 2014, 10:03:06 PM
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?

I'll check tomorrow.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 17, 2014, 10:27:27 PM
Quote from: Audionut on January 17, 2014, 10:11:09 PM
Make sure you are getting sensor saturation Greg.

Do you have it in mind?

500D (RawDigger) :
F1.4 (Black Level 1018, Max value 15365) 15365 + 1018 = 16383
F1.6 (Black Level 1019, Max value 14770) 14770 + 1019 = 15789

ISO :
F2.2 ISO100 (Black Level 1020, Max value 12782) 12782 + 1020 = 13802
F2.2 ISO200 (Black Level 1018, Max value 15181) 15181 + 1018 = 16199
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 10:29:29 PM
Quote from: Audionut on January 17, 2014, 10:11:09 PM
(...)
Not here.  If you are changing exposure settings with ISO, make sure you are getting sensor saturation.
Of course I'm sure to get sensor saturation. ;)
But with 5D2 and with just some quick tests I see it only depends of the ISO, f/stop doesn't affect RAW max value for a equivalent light intensity and same ISO.
I need to investigate about this to confirm relations..

Quote from: Audionut on January 17, 2014, 10:11:09 PMI'll check tomorrow.
+1
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 10:48:11 PM
Quote from: Greg on January 17, 2014, 10:27:27 PM
Do you have it in mind?

Point it at the sun  :o 

If you think you have saturation @whatever aperture, increase exposure by 1 stop with shutter to confirm.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 17, 2014, 10:51:58 PM
Quote from: Audionut on January 17, 2014, 10:48:11 PM
Point it at the sun  :o 

If you think you have saturation @whatever aperture, increase exposure by 1 stop with shutter to confirm.

Pictures are OvExp. In the background I put the flashlight, the exposure time 1s ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 11:43:56 PM
ML vs Canon

(https://i.postimg.cc/W3QT2j8R/ML100.png) (https://i.postimg.cc/8Cj2s9jT/VRAM0.png)

(https://i.postimg.cc/RCXLbFgb/ML200.png) (https://i.postimg.cc/Hsv3jx57/VRAM1.png)

(https://i.postimg.cc/MT0dGmbW/ML400.png) (https://i.postimg.cc/3JNFMT6r/VRAM2.png)

(https://i.postimg.cc/RVZXMf3B/ML800.png) (https://i.postimg.cc/Bn8pF0c7/VRAM3.png)

(https://i.postimg.cc/MHKbp2pD/ML1600.png) (https://i.postimg.cc/1tKrS5sX/VRAM4.png)

(https://i.postimg.cc/0yqRWcWf/ML3200.png) (https://i.postimg.cc/8CMbymYh/VRAM5.png)

(https://i.postimg.cc/hvq6yGMG/ML6400.png) (https://i.postimg.cc/PxHQ08G0/VRAM6.png)

(https://i.postimg.cc/tCdpHKVf/ML12800.png) (https://i.postimg.cc/j2g6WgWr/VRAM7.png)


edit:  I performed the Canon exposures first, so any temperature influence should be in Canons favour.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 17, 2014, 11:49:10 PM
Nice!

Do you see a stdev difference between 1/12s and 1/8000s @3200 ISO?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 17, 2014, 11:56:10 PM
(https://i.postimg.cc/0yqRWcWf/ML3200.png) (https://i.postimg.cc/CdTt2xN1/VRAM1.png) (https://i.postimg.cc/3rsxhjfy/VRAM2.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 08:05:15 AM
Quote from: a1ex on January 17, 2014, 10:03:06 PM
Now, if my theory is right, the sensor saturation point (where detail gets clipped) should be the same in all these cases. Can you confirm or infirm?

Well this was a lesson in frustration.  Between waiting for the light to warm up (60 mins) and trying to account for the inaccurate (not saturated) WL reading from ML (which I keep forgetting is not accurate when it's not saturated), I think I finally got some decent test images.

4 images @ f/1.4
(https://i.postimg.cc/tTW1FrYx/VRAM0.png) (https://i.postimg.cc/65yXS8fH/VRAM0.png)

(https://i.postimg.cc/j502qzZd/VRAM1.png) (https://i.postimg.cc/BvDZzj4W/VRAM1.png)

(https://i.postimg.cc/MTSZJzpn/VRAM2.png) (https://i.postimg.cc/3JJRwrCp/VRAM2.png)

(https://i.postimg.cc/fysVCx9g/VRAM3.png) (https://i.postimg.cc/pXGd5NqS/VRAM3.png)


4 images @ f/4.0
(https://i.postimg.cc/g06rPMyT/VRAM4.png) (https://i.postimg.cc/d14bNFkP/VRAM4.png)

(https://i.postimg.cc/HkXfDXct/VRAM5.png) (https://i.postimg.cc/tJcLyzNB/VRAM5.png)

(https://i.postimg.cc/zXLs79q6/VRAM6.png) (https://i.postimg.cc/NfScwQNQ/VRAM6.png)

(https://i.postimg.cc/4dL0b1qX/VRAM7.png) (https://i.postimg.cc/5yXRPCWL/VRAM7.png)


Taking a mean of all my previous saturated ML ISO 100 results, looks like DR is being reduced with increased aperture.  stdev is not being reduced enough to compensate for the lowered WL.  Also looks like f/4.0 is clipping more highlights.  Whether this falls into round off errors, I'll leave that for you to decide a1ex.

Of course, it's entirely possible that the aperture blades weren't closing a full 3 stops either.  Or they were closing slightly more then 3 stops.  I left aperture in electronic mode to increase deviation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 08:36:20 AM
I'd say it's because of different DOF, vignette at wide aperture, and the difference in how much light gets captured at f1.4 and f4.0 may not be exactly 3 stops.

You can try something like a gradient (e.g. point a lamp to a white wall) and see where the clipping point is (like you would read it on a ruler). Take the test shots exposed at -0.5, 0 (both f1.4 and f4.0) and +0.5, to see the scale of our clipping point ruler. I expect the difference between f4 and f1.4 to be around 0.1 stops.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 18, 2014, 08:38:50 AM
Quote from: a1ex on January 18, 2014, 08:36:20 AM
I'd say it's because of different DOF, vignette at wide aperture, and the difference in how much light gets captured at f1.4 and f4.0 may not be exactly 3 stops.

Starting with f2.8, Canon (and other manufacturers) secretly raise the sensor iso behind your back because unlike film, digital sensors cannot capture light from large angles - this effect gets very strong with fast lenses so that for example a f1.2 lens isn't able to gather much more light than f1.4. This effect might be one of the reason why Canon starts releasing slower primes with IS.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 08:44:23 AM
From looking at the registers, they only raise the ISO in the JPEG previews, not in RAW. This is what we are trying to confirm experimentally.

* note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 18, 2014, 09:01:14 AM
Quote from: a1ex on January 18, 2014, 08:44:23 AM
From looking at the registers, they only raise the ISO in the JPEG previews, not in RAW. This is what we are trying to confirm experimentally. * note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.

Ok, thanks for the explanation - it would be interesting to know how they actually do this, the current theory at least from all people I've read from on CR is that they raise iso, but then again few photogs have the in-depth tech knowledge to really tell.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 09:31:55 AM
(https://i.postimg.cc/vT0NhtKQ/1-30.png) (https://i.postimg.cc/L8Ybkf3p/1-4.png)

(https://i.postimg.cc/xTFZbgH5/1-20.png) (https://i.postimg.cc/G3jw9kqg/0.3.png)

(https://i.postimg.cc/QxMPtvBV/1-15.png) (https://i.imgur.com/HMDA9Do.png)

(https://i.postimg.cc/pLpg03x3/0.5-f1.4.png) (https://i.postimg.cc/HkRFF3kn/image.png)

Pretty sure I got it all right, been a while since I've slept.  I've got all the files uploading now if you want to take a better look.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 09:56:53 AM
Just fixed a typo in raw_diag (the sampling area for optical black was too small). Black level, noise stdev and dynamic range should be more accurate now.

Also added aperture on the graphs, since it affects the white level.

raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 10:52:16 AM
Yeah, that's better.  I'll post some updated graphs tomorrow if someone doesn't beat me to it.

Is there a specific reason why the screenshot is limited to 10s?  PITA when trying to pump out heaps of them quickly.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 11:10:53 AM
Good idea, added a screenshot feature to raw_diag. Should be easier to get the screenshots in sync with raw files.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 11:27:25 AM
Hopefully this solves this riddle.

(https://i.postimg.cc/tJnqG7HG/VRAM6.png) (https://i.postimg.cc/8CwPhqb9/VRAM7.png)

(https://i.postimg.cc/3rzp4vNp/VRAM2.png) (https://i.postimg.cc/G223zq72/VRAM3.png)

(https://i.postimg.cc/fRcbb3Sw/VRAM0.png) (https://i.postimg.cc/63mWPngh/VRAM1.png)

(https://i.postimg.cc/pyN2Wx2h/VRAM4.png) (https://i.postimg.cc/SQnCxrSk/VRAM5.png)


Quote from: a1ex on January 18, 2014, 08:44:23 AM
* note that by ISO I mean DxO definition, not overall brightness. They are trying to change overall brightness in RAW by faking the white level.

Boosting the WL of low aperture values?

That last histogram is getting close to Gaussian!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 11:41:04 AM
Most raw processors use some hardcoded white level for CR2 files (dcraw, ufraw, Adobe DNG converter, so probably all Adobe apps). So, with this trick, Canon effectively added a small boost in image brightness at default settings for large apertures.

For example, Adobe DNG converter simply sets the white level to 15000 for 5D3 images taken at full-stop ISOs. DCRAW uses a hard-coded white level for each camera (which is plain wrong, and the cause of probably all pink highlight issues in dcraw-based raw processors).

With a program that reads the white level correctly (e.g. by looking at brightest white values or by figuring it out from EXIF), the effect of this ISO boost disappears.

Now, I don't know how Adobe handles the data above the white level. I know for sure that ufraw clips it (so here it's important to convert the files to DNG and set the correct white level manually). If Adobe doesn't clip the data above white level, and uses it for highlight recovery, all is fine (you did not lose any highlight detail because of Canon's tricks). But if Adobe behaves like ufraw, you'll have to use exiftool to fix the white level.

My conclusion is simple: Canon's aperture trick does not remove any highlight detail from CR2 files at f1.6 and smaller apertures. BUT raw processors that do not handle the white level properly may ignore the detail above what they consider to be white level, and clip it.

At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 18, 2014, 12:10:47 PM
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 12:13:16 PM
A lot of the information gained with these registers should help with cr2hdr and such correct?  With histograms for instance, there's no more guessing of WL.  Discard some hot pixels and you're good to go!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 12:15:36 PM
Quote from: hjfilmspeed on January 18, 2014, 12:10:47 PM
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic?

When I unsrew the lens on my 5D3 it uses the non-boosted white level.  Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on January 18, 2014, 12:29:23 PM
Quote from: Audionut on January 18, 2014, 12:15:36 PM
When I unsrew the lens on my 5D3 it uses the non-boosted white level.  Now that I know what's happening, I'll post a comparison tomorrow of f/1.4 boosted vs non-boosted.

Maybe I'm just suffering from a placebo effect, but when i do this, take one shot with the lens detected and one shot with it screwed half off, I see a considerable jump in brightness when the lens is properly detected at 1.8/f on a 50mm using the neutral picture style and shooting in jpeg.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 12:40:56 PM
Yes, this confirms our theory.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 18, 2014, 12:47:10 PM
Quote from: hjfilmspeed on January 18, 2014, 12:10:47 PM
Still it seems that ML iso is superior to canon iso. And boosting the iso for low aperture is real silly on canons part. What happens if you use a low aperture lens thats not electronic? I still feel like 1.2 would get more light to the sensor even if its not an exact amount. right? Canon treats us like babies sometimes. Id rather know exactly how much light my sensor is getting rather then this fake boost thing. ehhh but who care cuz ML iso is so much better anyways =)

The only reason I can think of canon is doing this (and probably all other camera manufacturers) is because most photographers expect exactly 1 stop extra light when they're going from f2.8 to f2.0.
If it's really true that the digital sensors don't get all the light with apertures bigger than 2.8, you must do this trick to get that extra stop of ligt on you're histogram.

But when you're after the best image quality, dynamic range etc. you're (a tiny bit) screwed.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on January 18, 2014, 01:21:36 PM
I just found something very interesting with my camera (which I think is the basis of this whole discussion).

So people can replicate;
550D -> Video mode -> Expo Menu -> Iso Menu.
I set the canon analog to 100, canon digital to 0, then ML digital to +7.0 EV (giving an equivalent of 12800).
Aperture set to something below 2.0/f
Now, when one scrolls through the canon digital 0 EV has about the same amount of noise than 0.5 EV
But when one sets the aperture to 1.8/f
Now, when one scrolls through the canon digital 0 EV has ALOT more noise than 0.5 EV



Wait a minute. That could be my own stupidity.
When it's set so low it has nothing to pull up but noise, hence why when the aperture is wider it has more image to pull up.
Or is it, I don't know now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 02:12:01 PM
Here's a research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect:

* CMOS gain, described in the Dual ISO PDF
* ADTG gains (0x8882 ... 0x8888) - same value for all of them
* Saturate Offset (0xc0f0819c)
* Digital Gain (0xc0f37ae4/af0/afc/b08)
* Black/White Offset (c0f37ae0/aec/af8/b04))

iso_regs.mo (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.mo) (source: iso_regs.c (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.c))

(http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.png)

Notes:
- Do not mix photo mode settings with LiveView ones. Hit "disable overrides" before switching between photo and video.
- It requires a custom-compiled ML with CONFIG_GDB = y. As I said, it's not for end-users, it's a research tool.
- The parameters are explained in the help text. More info here (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96106;topicseen#msg96106) and in this thread.
- This has a TON of parameters hardcoded for 5D3. Porting on other cameras is very hard. Assuming the registers behave in the same way, there are only 25 camera-specific parameters. On 60D for example, only ADTG gains are the same, so you will have to figure out all the other registers with ADTG_GUI and do the math from scratch. Don't even think about trying on 5D2/500D/50D.
- To disable Canon's aperture trick, simply fix digital gain to some value.

Be careful not to fry your sensor :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 18, 2014, 02:22:57 PM
there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 18, 2014, 02:27:14 PM
Quote from: Levas on January 18, 2014, 02:22:57 PM
there are a few 6d users here, like 1%.
Is there anything for the 6d at this moment?

For the simple ADTG tweak, see the response below.

For the above.

Quote from: a1ex on January 18, 2014, 02:12:01 PM
Porting on other cameras is very hard.

a1ex opinion of easy, is suitably hard for most users.  Here he describes very hard with bold!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 02:30:56 PM
Besides, for 6D somebody has to fix the raw backend in photo mode first: www.magiclantern.fm/forum/index.php?topic=10028
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 18, 2014, 03:23:48 PM
I got the same results with the holding down the DOF button while unscrewing canon lens. I saw the jumps in the raws too in adobe LR 4.4. Didnt try recovering highlights but i shot 1.4 1.6 1.8 2.0 and 2.8 on canon 50mm with there unscrewed twins. It didnt really start to level off till about 2.8 5d3 I also compared it to 1.8 on an old 50mm nikon. Good news is canons 50mm 1.8 unscrewed with DOF button held is brighter then the 1.8 on nikon 50.  1.4 unscrewed was still brighter then all the apertures so other then it being soft its still good to use if you need that light. I notice a slight difference in color to even though it was all the same temp. So i guess that might have to do with the black and white levels

Will this jump occur with ML RAW video? Ill test that next, I assume yes since it seems like its in the RAW concoction duringr atdc.

When this is done this will be an amazing mod to have! I dought it but i hope this helps a lil.....il shut up now
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 18, 2014, 03:26:55 PM
Can you compare the clipped highlights at f1.6 normal vs unscrewed?

(I want to see how Adobe handles the white level)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 18, 2014, 04:18:38 PM
500D (default) :
(http://s28.postimg.org/frs15f6fx/VRAM0.png)

ADTG3[0] = 0x149 (default) -> 0xb9 (trick)
ADTG3[3] = 0x14a (default) -> 0xba (trick)
(http://s28.postimg.org/hkuxtqrml/VRAM5.png)



500D - T1/200s, F1.4

ISO                100       160       200       320       400       640       800       1250       1600    3200    6400    12800
ADTG3[0]       149       149       149       148       148       148       148         142         142      142      142      142
ADTG3[3]       14a       14a       14a       14a       14a       14a       14a         145         145      145      145      145
CMOS[0]            0         90         90       1b0       1b0       1f8         1f8         168         168      168      168      168
C0F0[819c]     f87        f87        66f       66f        66f        66f         66f         66f          66f        66f       66f      700
C0F0[8030]  1570     1126    1570     1126     1570     1126      1570      1126       1570     1570    1570    1570
C0F0[8034]  1079     1079    1991     1991     1991     1991      1991      1991       1991     1991    1991    1900
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 18, 2014, 10:17:33 PM
Quote from: a1ex on January 18, 2014, 03:26:55 PM
Can you compare the clipped highlights at f1.6 normal vs unscrewed?

(I want to see how Adobe handles the white level)

Sure. Sorry Just crashed for a while. My shots did include blown highlights. Il post them and there recovered twins in a bit

Sent a pm a1ex with raw files included
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 19, 2014, 03:58:49 AM
Quote from: a1ex on January 18, 2014, 11:41:04 AM
At f1.4 there seems to be indeed an overflow of 0.03 stops (which I would consider a bug in Canon code). From f1.6 there should be no differences in dynamic range.

(https://i.postimg.cc/ZK030LJJ/f1.4.png) (https://i.postimg.cc/9MWw2dx5/f1.6.png)
(https://i.postimg.cc/1tJn38Tg/f4.0.png)


0.003EV.  I did the f/4.0 shot first, so the camera was a little warmer in the other shots.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 19, 2014, 08:04:41 AM
Alright, so the side effects of Canon's ISO/aperture trick are:
- small overflow at f1.4 (0.03 stops), larger at f1.2 (I guess 0.25 stops from DxO chart, link below). This results in highlight loss in the CR2.
- histogram gaps indicating digital scaling (not quite good for denoising algorithms)
- possible highlight loss in some raw editors (ufraw for sure; I'll fix it in ufraw-mod). To check your favorite editor, take two test shots, f1.6 normal and with lens unscrewed by holding the DOF button, and compare the highlights.

http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml

The good news is that I can disable this trick fairly easy on all cameras.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 19, 2014, 08:49:29 AM
Good read.  I'll do a heap of tests later showing Canon vs ML with the WL capped and brightness fixed in post.

This will be a good test also to see just how much light loss there is at each aperture value below f/4.0 for my lenses.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 19, 2014, 04:13:13 PM
500D - T1s, ISO 100

aperture    -    F                     1.4         1.6      1.8        2.0        2.2        2.5         2.8         4.0          8.0         22
digital gain -   C0F0[8030]  1570     130a    11eb     10f5     107a     107a      1000      1000       1000     1000


F1.4 lens unmount, the value is 1000

unmount lens (So that the contacts do not touch.):
(http://s10.postimg.org/q8x1x57ex/VRAM2.png)

mount lens :
(http://s10.postimg.org/vydahgdl5/VRAM3.png)


No lens affects the colors:

unmount lens :
(http://s28.postimg.org/k4tbw3gj1/unmount.jpg)

mount lens :
(http://s28.postimg.org/q3756bzhp/mount.jpg)

mount lens + trick (digital gain 1000):
(http://s28.postimg.org/s91g0u2y5/mounttrick.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 19, 2014, 05:10:48 PM
Shutter speed.
Changes are 1s and longer.

1s :
(http://s28.postimg.org/3p02r92st/VRAM0.png)

10s :
(http://s28.postimg.org/cy291dbot/VRAM1.png)



I found something interesting. Turn on the camera. ADTG registers -> show set Modified since.
Take a picture.

ADTG3[0] = 0x149 (was 0x111)
ADTG3[3] = 0x14a (was 0x111)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 19, 2014, 06:05:05 PM
Quote from: Greg on January 19, 2014, 04:13:13 PM
No lens affects the colors:

For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2

The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.

My light source was a led light, so it should have fairly consistent temperature output.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 19, 2014, 06:09:49 PM
Quote from: Audionut on January 19, 2014, 06:05:05 PM
For me (5D3), with consistent white level, the as shot temperature doesn't vary, and the tint only varies by +/- 2

The RGB values, of the RGB patches, of an x-rite colorchecker only vary by +/-1.

My light source was a led light, so it should have fairly consistent temperature output.
I set the same color temperature in ACR.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 19, 2014, 06:29:12 PM
2 images, 1 with lens attached and 1 with lens unscrewed.  Both at f/4.0 to ensure no secret gain changes.
Both developed with the same settings in ACR.  Camera faithful profile, no noise reduction, no lens correction, as shot color temperature.
Layered in PS with a difference layer blend on the top layer.

(http://i.imgur.com/cojf1vu.png)

Only difference is the slight shot positioning error.

Quote from: Greg on January 19, 2014, 04:13:13 PM
unmount lens (So that the contacts do not touch.):
(http://s10.postimg.org/q8x1x57ex/VRAM2.png)

mount lens :
(http://s10.postimg.org/vydahgdl5/VRAM3.png)

Ouch. 

5D3.
(https://i.postimg.cc/QCQz7rQW/VRAM1.png)
(https://i.postimg.cc/6qJSTQvX/VRAM0.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 19, 2014, 07:22:40 PM
500D: log2(10.71)-log2(8.62) = 0.31 stops of ISO boost at f1.4, apparent ISO 100->124
DR 10.492-10.486 = 0.006 => real ISO 100->100.5

5D3: log2(7.24)-log2(6.57) = 0.14 stops, apparent ISO 100->110
DR 10.976-10.952 = 0.024 => real ISO 100->101.6

Apparent ISO: image brightness in JPEG preview and maybe at default settings in a raw editor.
Real ISO (by DxO definition, that is, from clipping point) assummes correct white level handling by the raw editing software.
If the raw editor assumes some fixed white level and clips everything above it, apparent ISO will be closer to reality.

According to DxO, the difference can be noticeable at f1.2. Any volunteer to check it?

You don't need ML to run the test; simply take a test picture at f1.2 with the lens normal and then with the lens unscrewed from the body,  compare the highlights and measure noise stdev from the left optical black area. Or just upload the two CR2's.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 19, 2014, 08:56:58 PM
Any particular settings? ISO etc as its night time here.
I have an 85mm 1.2 here, and also have a canon cinema lens T1.3 if that helps any.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 19, 2014, 09:05:01 PM
Yes, M mode, ISO 100, aperture wide open, and the image should include clipped highlights (e.g. a light bulb). Same settings in both (only change screwed vs unscrewed).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 19, 2014, 10:03:18 PM
4 images uploaded to dropbox as I don't seem to be able to upload to forum.
First 2 are from the 85mm 1.2 1st with data 2nd unscrewed
Second lot are using the canon cinema 50mm T1.3 not sure if they will help with any comparisons.
If these are not what you require let me know and I can try other shots.


https://dl.dropboxusercontent.com/u/49168208/85mmf1.2.CR2
https://dl.dropboxusercontent.com/u/49168208/85mmunlockedf1.2.CR2

https://dl.dropboxusercontent.com/u/49168208/50mm-t1.3.CR2
https://dl.dropboxusercontent.com/u/49168208/50mmunlockedt1.3.CR2
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 19, 2014, 10:18:54 PM
From the first two:

Noise 7.0886 vs 6.1815 => ISO was boosted by 0.2 stops

DR 10.98 vs 11.06 => you lost 0.08 stops of highlights because of Canon's trick. Not really noticeable in practice.

If you have an APS-C body, the difference may be a higher (DxO suggests 0.5 stops).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 19, 2014, 10:35:24 PM
Maybe the image wasn't the best, would you rather have something say black then a candle beside it to make sure the whole dr is covered?
DXO says that the real light for that particular lens is T1.4

Will be interesting to hear regarding the cinema lens, as there should not be any "trickery" needed by canon.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 19, 2014, 10:48:06 PM
For black, each CR2 has an optical black area (that's where I'm measuring noise). For a visible difference, some gradient pattern (gradual transition to clipped white) would be better.

The shots with the cinema lens were not taken at the same shutter speed.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 19, 2014, 11:47:06 PM
Here's a couple more from the cine lens T1.3 with same shutter speed.
Not the sharpest, as hand holding and trying to get it to clip at a low shutter speed, hope that will be OK, can take better ones during the day tomorrow with a tripod if needed.


https://dl.dropboxusercontent.com/u/49168208/50mmt1.3.CR2
https://dl.dropboxusercontent.com/u/49168208/50mmt1.3locked.CR2
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: engardeknave on January 19, 2014, 11:57:56 PM
I also have the 85 1.2 on a 5D2 if that would be useful.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 12:56:17 AM
50mm t1.3:

Noise 7.11 vs 6.16 => iso boost 0.2 stops
white 16383 vs 15283
DR 10.98 vs 11.07 => highlights lost 0.09 stops

Would be great if you could take this shot on tripod, so I could do a side-by-side comparison of the images.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: edwmotion on January 20, 2014, 02:48:22 AM
7D 50mm f1.2 pentax, can I help?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 20, 2014, 03:17:45 AM
Quote from: edwmotion on January 20, 2014, 02:48:22 AM
7D 50mm f1.2 pentax, can I help?

I don't think so. The camera needs to know that it has connected the lens and the current aperture.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 20, 2014, 02:04:05 PM
Quote from: a1ex on January 20, 2014, 12:56:17 AM
50mm t1.3:

Noise 7.11 vs 6.16 => iso boost 0.2 stops
white 16383 vs 15283
DR 10.98 vs 11.07 => highlights lost 0.09 stops

Would be great if you could take this shot on tripod, so I could do a side-by-side comparison of the images.
Not sure you are experimenting with the right things.
2 images below set all in manual and using the latest firmware 1.2.3
I can tell from the histogram before shooting that whenever you unscrew the lens slightly that its clipping sooner.
I had to move the clipped light from 1/40th to 1/20th to get it to clip in histogram when it was clipping at 1/40th when locked
Both shots taken exact same settings.
50mm T1.3

https://dl.dropboxusercontent.com/u/49168208/locked.CR2
https://dl.dropboxusercontent.com/u/49168208/unscrewed.CR2
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 02:19:23 PM
Quote from: Quinton on January 20, 2014, 02:04:05 PM
I can tell from the histogram before shooting that whenever you unscrew the lens slightly that its clipping sooner.

Histogram before shooting is not exact (even ML raw histogram in LV it's an approximation). Histogram *after* shooting (e.g. in RawDigger) is the one you should look at.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 02:30:49 PM
I have some good news for 5D2 (credits go to Greg, he found it on 500D):

- SaturateOffset register works and I have moved the black level from 1024 to 64.
- According to my 5D3 theory, this would be equivalent to reducing ADTG gain by -0.091 EV (ISO 100->94).
- However, the dynamic range in raw_diag is showing an improvement of over 0.7 stops at ISO 100 (!)

What's going on?!

edit: diagnosed. In some Canon cameras, bad pixels are marked as 0 in the raw buffer (and then interpolated). In raw_diag, I was not doing any checks of the raw pixel values (so, a bunch of zeros compared to 1024 were bumping stdev a lot more than the same bunch of zeros compared to 64).

The real improvement was close to 0.1 stops, so the theory about Saturate Offset is still valid.

Updated raw_diag to skip hot pixels:

raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c))

(the bad pixel modification should be backported to ML raw.c)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 03:47:33 PM
I have this sick obsession with tweaking things.  Hence the nut at the end of audio  ;)

(https://i.postimg.cc/rmKc5zsR/tweak.png)

vs Canon vs mini_ISO
(https://i.postimg.cc/qvxkT9Wq/canon.png) (https://i.postimg.cc/d0QJt5KB/mini_ISO.png)

Off now to take some sample images and see what the change is visually.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 04:41:03 PM
I have sighted bias, so I would like some opinion on these 2 test images before I give my own.

1
(https://i.postimg.cc/4NGNTyT3/image.jpg)


2
(https://i.postimg.cc/ZnSZZ72Q/image.jpg)


edit:  I can still see, what I think I can see, in these resized and JPG (minimum) compressed images.  I can upload sample CR2s at a later date if requested.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 20, 2014, 05:01:38 PM
I'm no expert and its hard to tell online but picture 2 for me seems noisier than 1 and the colours are flatter, or maybe I'm just seeing things because I'm expecting a difference.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: lasfede on January 20, 2014, 05:17:38 PM
I see a clear difference in colours but not in terms of noise, may be the contrast of picture 1 is weaker which leads to less noise. I prefer colours of shot 2, shot 1 seems to be more red.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 20, 2014, 05:36:26 PM
Nikon 500D :

(http://s17.postimg.org/9ncdyl2y7/VRAM21.png) (http://s17.postimg.org/i4bw9i7n3/VRAM20.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 05:59:13 PM
PM response is in favour of image 1.  In my opinion, this image has less banding noise.  Check the blurred background left and right of the colorchecker.  It's easier to see in full res CR2.

This image (1) is with these settings.
(https://i.postimg.cc/Fs0djwhY/settings.png)

Further testing favours a digital gain value of 462.

Note:  This is only with ISO 100, and only tested with a 5D3 (Greg would be interested to see results with 500D).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 20, 2014, 06:15:18 PM
Quote from: Audionut on January 20, 2014, 05:59:13 PM
PM response is in favour of image 1.  In my opinion, this image has less banding noise.  Check the blurred background left and right of the colorchecker.  It's easier to see in full res CR2.

TBH both images were pretty noisey, just no2 was more so.
I have only seen noise like that at much higher ISOs or where they have been considerably pushed in post processing.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 20, 2014, 06:18:31 PM
In practice, the difference is not so big.  :(

Canon 500D (-3EV ACR):
(http://s23.postimg.org/v5egtnyu3/default.jpg)

Nikon 500D (-3EV ACR):
(http://s23.postimg.org/3w33f5xqz/trick.jpg)

Crop 1:1 (+2,5EV ACR):
Canon 500D :
(http://s11.postimg.org/67wwbutzn/image.jpg)

Nikon 500D :
(http://s11.postimg.org/x4gvk6csz/image.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 06:32:16 PM
Quote from: Quinton on January 20, 2014, 06:15:18 PM
I have only seen noise like that at much higher ISOs or where they have been considerably pushed in post processing.

I expose so that the white patch (in the colorchecker) is exposed as high as possible before over exposure.  The images in the above test are exposed 5EV lower with shutter, and increased 5EV in post.  Note:  This should put the black patch around -9.7EV (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563).  I WB and brightness correct from a correctly exposed image.  This removes any error due to noise, and obviously, ensure lighting remains consistent.

To be clear, image 2 was an ISO calibrated mini_iso result.  And the difference was always going to be minor, because I've already shown (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563) the gains being made with just ADTG tweaks.

The quickest way to see any differences, is with images down near the noise floor IMHO. 

When I get time to sort out a 32bit (ACR limited to +/-5EV) raw based workflow, I'll test further into the noise floor  ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 06:37:34 PM
Quote from: Greg on January 20, 2014, 06:18:31 PM
In practice, the difference is not so big.  :(

The Nikon 500D result looks slightly better, but it's biased by a lower rendered brightness.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Quinton on January 20, 2014, 06:48:19 PM
Quote from: Audionut on January 20, 2014, 06:32:16 PM
I expose so that the white patch (in the colorchecker) is exposed as high as possible before over exposure.  The images in the above test are exposed 5EV lower with shutter, and increased 5EV in post.  Note:  This should put the black patch around -9.7EV (http://www.magiclantern.fm/forum/index.php?topic=9867.msg96563#msg96563).  I WB and brightness correct from a correctly exposed image.  This removes any error due to noise, and obviously, ensure lighting remains consistent.
Sorry new about here.
Thanks for the explanation, makes much more sense now ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 07:57:23 PM
Would it be possible to option a change of color of the SNR curve, or an option to only screenshot the curve without the background data.  This should make it easy to overlay data in post.

I can get this, but it misses the fine detail in one of the curves.
(https://i.postimg.cc/mkZYLrb9/overlay.png)


With my tweaks on the last page.  Changing only ISO in canon menu and CMOS gain in ML.

https://www.dropbox.com/sh/dwobqot98fdm8k3/AAC3WYZhGtwzuHtK_99_trKRa?dl=0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 08:34:54 PM
The SNR curve needs some offset tweaking first. Since we are tweaking the ISO (getting a different clipping), this results in shifting the curve horizontally (so, 100% in Canon ISO should not be the same as 100% in ML ISO, but right now on the graph they appear to be the same).

Instead, I'd like to consider the test scene as reference (that is, when you double the shutter speed and the shadow noise gets twice as small, you should see a 1-stop difference on the graph). That means we should shift the curve horizontally with the Bv value (from expo menu) and the ISO correction computed by the iso_regs module.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 20, 2014, 09:39:58 PM
The benefit oft the mini_iso module is automatic, but I still think adding it directly to auto_iso in M mode would be a good idea. In Av and Tv, you cannot do anything about it, the user has to add the +ec gained by ml (since we have no way of intercepting the shot to modify the ec, then reset it afterwards).

But if auto_iso & autoexpo set iso, aperture, shutter and ec in m anyway, wouldn't be a good idea to modify the ec in m value with the exact mini_iso.mo value to always gain optimum exposure? Should this be done in the modules or could mini_iso please deliver a weak function to return a modified Canon 8-system value according to the requested iso?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 10:29:24 PM
I assume,

static CONFIG_INT("black.ref", black_reference, 0);

is ADTG black (0x8880)?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 10:37:52 PM
Yes, not implemented yet.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 10:40:28 PM
Quote from: a1ex on January 20, 2014, 10:37:52 PM
Yes, not implemented yet.

I was going to have a go at doing it myself  :)

Quote from: a1ex on January 20, 2014, 08:34:54 PM
Instead, I'd like to consider the test scene as reference (that is, when you double the shutter speed and the shadow noise gets twice as small, you should see a 1-stop difference on the graph). That means we should shift the curve horizontally with the Bv value (from expo menu) and the ISO correction computed by the iso_regs module.

I'm not sure I understand.  A 1 stop shutter difference shows this.
(https://i.postimg.cc/L4JGR4bf/VRAM0.png) (https://i.postimg.cc/RVf86s9Z/VRAM1.png)

The entire curve has shifted 1EV.  1/3s shutter shot had clipped highlights which is why the WL didn't shift a full stop.  But the rest of the SNR curve looks like it did.


I know I don't understand what needs to happen for the true ISO adjustments.  But the above data looks accurate.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 20, 2014, 10:53:07 PM
Yes, but they overlap perfectly. Think at one single color patch from your test image: one of them will be rendered with twice as much signal, and roughly the same amount of noise (so the SNR should be roughly 1 stop higher on one of them). And it shows up correctly.

However, the two points on the curve corresponding to our patch are not at the same X position. I want them to be in the same position, so from two overlapped curves I should be able tell how big the improvement was for our patch (and all others). Now I can tell the exposure was shifted to the right, the curve did not really change with shutter speed, but I can't tell how better our patch was rendered.

So, there's nothing wrong with the curve, but to compare things it may be useful to change the X origin (link it to the test scene, not to the recorded data).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 20, 2014, 11:05:39 PM
I like how it shows where the data is rendered in the 14bits also.

(https://i.postimg.cc/66h4Ww4W/1.3.png) (https://i.postimg.cc/c1T6qmP6/1.6.png)

My highlight (red dot) got shifted up both scales (as it should IMO).  I should have gained a little extra SNR, but it got clipped at 13.5EV, but I did gain bit depth precision.

My shadow detail (green dot) got shifted down 1EV in both SNR and bitdepth.

edit:  I see your point now, it would be helpful to see how the SNR changed in the image without the bitdepth interference.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 22, 2014, 12:05:45 AM
Are there more tests needed? Anything I can help with? Not good at programming but in the middle of a snow storm so just sitting here refreshing this page ha ha JK.

5d3 running the nighters
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 22, 2014, 12:38:51 PM
Updated some research tools:

iso_regs.mo (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.mo) (source: iso_regs.c (http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.c)) - 5D3 only, requires CONFIG_GDB=y:

- added register 8880 (halfway done by Audionut)
- increased range of B/W offset

adtg_gui.mo (http://a1ex.magiclantern.fm/bleeding-edge/adtg_gui.mo) (source: adtg_gui.c (http://a1ex.magiclantern.fm/bleeding-edge/adtg_gui.c) avl.c (http://a1ex.magiclantern.fm/bleeding-edge/avl.c) avl.h (http://a1ex.magiclantern.fm/bleeding-edge/avl.h)) - cross-platform, requires CONFIG_GDB=y:

- also intercepts DIGIC registers (experimental, there are LOTS of them)
- it adds no less than 4096 menu entries (2048 were not enough on 5D3) => it will slow down boot and menu navigation.
- if you get ERR70 out of memory, change the max region heuristic in memory.c from free_space / 2 to free_space / 4 (or add a wrapper to GetSizeOfMaxRegion)
- it's not usable in LiveView, but if you really want to try, FPS override is your friend

raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (source: raw_diag.c (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.c)) - cross-platform, anyone can run it

- LiveView support (only relevant in movie mode, not in photo LiveView)
- updated README (including a small TODO list, if anyone feels like coding or doing some math)
- to read it, press Q in ML modules menu
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 22, 2014, 01:45:01 PM
I'm seeing some ADTG/CMOS registers being doubled/tripled in the menu.  I'm seeing ADTG6[8000] 4 times.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 22, 2014, 01:57:48 PM
SOLVED! thanks Nanomad for help
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 22, 2014, 03:37:37 PM
edit:  Of course you need to subtract the mean to calculate DR.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 22, 2014, 06:46:25 PM
Regarding raw_diag.mo, this is a very minor quibble, but you may want to tweak the definition of the optical-black region for the left black bar.  You start at y = 20 instead of y = raw_info.active_area.y1 + 20 as is done in raw.c.  The 50D, at least, has a structured pattern for the first ~thirty lines of the top black bar, so starting the optical-black region that early picks up that pattern.  It's so few pixels that it probably doesn't really affect any histograms, but you might as well play it safe.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 22, 2014, 06:50:14 PM
Solved.

Also updated the first post with a little FAQ and links to research tools. If you still have questions, just let me know.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: lionelp on January 22, 2014, 09:08:03 PM
Great work Alex and everyone! Appreciate all that you do!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 23, 2014, 03:23:58 AM
As discussed on a recent pull request (https://bitbucket.org/hudson/magic-lantern/pull-request/344/50d-fixed-starting-location-of-raw-photo/diff), a massively overexposed frame affects the left optical-black region for the 50D (but does not in the 5D Mark III).  This effect complicates determination of the black level and the read noise, and consequently can throw off the measurement of dynamic range.

The first plot shows the histogram for raw_diag.mo with only minor overexposure.  The measured read noise is comparable to that of a dark frame with the same exposure settings.  The second plot shows the histogram when the image is massively overexposed.  You can see that the read noise appears to be larger, which appears to reduce the dynamic range by ~0.2 EV.
(https://dl.dropboxusercontent.com/s/4pa2t3ft3hah2xs/raw_diag_0250.png)(https://dl.dropboxusercontent.com/s/q945sbuhw9d4bw3/raw_diag_0247.png)

Diving into the images themselves, here are the histograms for pairs of columns, starting from the left edge of the sensor (the bottom-most line) and going to just before the image itself (the top-most line).  The solid lines (as opposed to the dashed lines) are the columns in the inner optical-black region used by raw_diag.mo to determine the black level and the read noise.  The first plot, corresponding to the minor overexposure, shows that all of the columns in the inner optical-black region have about the same mean and standard deviation.  The second plot, corresponding to the massive overexposure, shows that all of the columns have about the same standard deviation, but there is significant drift in the mean, and averaging over this drift is what is producing the appearance of a larger read noise and a slight decrease in the black level.
(https://dl.dropboxusercontent.com/s/fld1glcryfcxkki/variation_0250.png)(https://dl.dropboxusercontent.com/s/s9y77vl6h1roluk/variation_0247.png)

Thus, the read noise isn't actually changing, which is good, but the apparent drift of the black level is not good.  I don't know whether this points to some form of sensor bleed or a deficiency of the black-level clamp.  Either way, on cameras with sensors like the 50D's, take care not to overexpose images by too much.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 04:49:48 AM
Have you tried making small adjustments to 0x8880?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 07:17:16 AM
My investigative work so far.

Old settings vs New
(https://i.postimg.cc/TY56ftk7/settings.png) (https://i.postimg.cc/XqTtr022/settings2.png)

mini_iso vs old settings vs new
(https://i.postimg.cc/k5PpvxJv/mini_iso.png) (https://i.postimg.cc/vTKPgn29/VRAM0.png) (https://i.postimg.cc/NMNcfSqV/ISO100.png)

With the new settings, I've brought the WL back to 15281 and cleaned the histogram further.  However, a close look at the histogram suggests there is probably still some room for (slight) improvement.

@a1ex:  Notice the Digital Gain default changes.  I have no idea what causes this.  I've tried retracing my steps but can't narrow it down.


@all:  No sample images as yet, but my last samples (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97073#msg97073) showed that the old settings vs simple ADTG gain adjustment delivered an improvement in image quality.

Digital Gain shifts the entire exposure.  This appears to be digital as it has a large impact on the histogram.  Try adjusting this one first.

Black/White Offset and Saturate Offset are closely aligned.  Adjust either one by N steps and you can adjust the other one by the same N step (in the opposite direction) to bring the histogram back to original.  Here, you can use the BW offset to compensate for digital gain reduction (bring the WL back) and Saturate offset to adjust the black mean. 
So for instance, you can adjust the WL higher then 15281 with BW offset (up to 16383), and use Saturate offset to bring the black mean back to 2048.  Note:  I have tried this and didn't see any improvement (slight negative effect actually), but I've been looking at optical black histograms and adjusting registers for the last 5 days or so, so YMMV.

After I adjust all these, I reduce the ADTG gain to the sweet spot (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96106#msg96106).

I haven't been able to find a pattern to black reference, but I also haven't spent much time with it.  This requires further investigation.


The effect of my new settings on all ISOs.  Note that stdev can vary from shot to shot.

(https://i.postimg.cc/NMNcfSqV/ISO100.png) (https://i.postimg.cc/GpdwPdbV/ISO200.png) (https://i.postimg.cc/qRDPdK2H/ISO400.png)
(https://i.postimg.cc/yd8M5jgT/ISO800.png) (https://i.postimg.cc/tJ1LRgzZ/ISO1600.png) (https://i.postimg.cc/SQ5BTWyW/ISO3200.png)
(https://i.postimg.cc/dVMK1Gpq/ISO6400.png) (https://i.postimg.cc/fypQYdbf/ISO12800.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 23, 2014, 07:59:33 AM
@ayshih: for these cameras I would point the camera to a light bulb in the middle of the image, and expose until the edge pixels become dark. If you have like 1% of the image overexposed, it's enough for DR calculation.

@Audionut: default digital gain changes with aperture (it shows Canon is trying to boost the ISO).

Tip: you don't really need to copy Canon settings; you can override only one or two settings at a time, and leave the others to Canon defaults. Might be useful when doing tests at more than one ISO.

I'm not yet sure about 8880 (at first sight it looks it was already tuned to sweet spot by Canon).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 23, 2014, 08:05:08 AM
Quote from: Audionut on January 23, 2014, 04:49:48 AM
Have you tried making small adjustments to 0x8880?

I'm still getting the hang of looking at ADTG registers, but I don't believe the 50D uses that particular register, so I don't know how to change the black level.  (As an aside, I also haven't figured out how to change the ADTG gains on the 50D.  Inspired by Greg's testing with the 500D, I tried overriding the ADTG3[0] and ADTG3[3] registers, but they don't appear to affect anything.)

Thinking more about the effect I've described, it looks like the black-level clamp just has a characteristic time for clamping to the reference voltage (presumably generated by the register value and a DAC), and actually oscillates a bit about the desired level as it gets there.  The amplitude of that oscillation then corresponds to how much the clamp needs to shift the signal in voltage.  If you consider the signal on a given line in an AC-coupled sense, a massive overexposure have a high average charge per pixel, which means that pixels in the optical-black region will be very far from the reference voltage, and the resulting amplitude of the oscillation to be large.  The 5D Mark III presumably has a different, more effective black-level clamp.

If that's the right description, my guess is that there'd be no benefit from small changes to the black level.  Changing the black level by a significant amount could help to stabilize the optical-black region for images that are significantly overexposed, but would compromise normally exposed images, so that doesn't really make sense as a change.

Edit: I got a little mixed up about the direction of the needed change.

Quote from: a1ex on January 23, 2014, 07:59:33 AM
@ayshih: for these cameras I would point the camera to a light bulb in the middle of the image, and expose until the edge pixels become dark. If you have like 1% of the image overexposed, it's enough for DR calculation.

Yup, agreed.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 08:23:52 AM
If the damage is being done in the CMOS stage, you could presumably fix it there with the digital registers and correct the black level with ADTG.  Just guessing though!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 23, 2014, 08:25:45 AM
Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).

For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624. Also fix the digital gain (0xc0f08030) to 0x1000.

The effect of SaturateOffset is that it expands the recorded range while bringing in more highlight detail. So, if your default range is say 1024...15760 and you increase this range to say 32-16383 without changing digital gain (these are the 5D2 values), this would be equivalent to scaling ADTG gain to (15760-1024) / (16383-32) = 0.9 (that is, -0.15 stops). The theory is explained in get_resulting_iso (iso_regs.c).

Basically, ADTG gains and SaturateOffset adjust the same thing internally. So, to get say ISO 77, you can:
a) simply scale factory ADTG gains by 0.77
b) increase the range with SaturateOffset (e.g. with equivalent gain of 0.9) and scale ADTG gains by 0.85 (so the total gain is 0.77). The total gain is limited by the sweet spot (under this point, white level begins to decrease at least in some columns => you get artifacts).

So, on 5D2, the mystery is what register is used for ADTG gain. I believe this gain is used to implement ISO 125 from 100, 250 from 200 and 3200 from 1600 (because of lack of gaps in histogram), but the ADTG tool doesn't report any ADTG register changes when switching between these ISOs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 23, 2014, 10:26:04 AM
Out of curiosity, I tried two dual ISO shots to see whether the theory applies. Since the new trick only improves the highlights side, the theory says you will only benefit from the DR improvement at ISO 100 (because in shadows, the modified ISOs are just as noisy as before).

I had to update the white level detection routine in cr2hdr to consider two separate levels for bright and dark exposure. I'll only post the relevant lines from the conversion log. The subject was boring (a light bulb).

ISO 100/1600:

White levels    : 15094 15109
Noise levels    : 11.05 6.48 6.52 11.08 (14-bit)
ISO difference  : 3.97 EV (1572)
Semi-overexposed: 9.47%
Deep shadows    : 83.80%
ISO overlap     : 4.0 EV (approx)
Dynamic range   : 10.98 (+) 10.21 => 14.19 EV (in theory)
Dynamic range   : 14.35 EV (cooked)


ISO 77/1230:

White levels    : 15096, 15108
Noise levels    : 8.60 5.13 5.03 8.54 (14-bit)
ISO difference  : 3.96 EV (1559)
Semi-overexposed: 6.06%
Deep shadows    : 84.38%
ISO overlap     : 4.4 EV (approx)
Dynamic range   : 11.35 (+) 10.58 => 14.54 EV (in theory)
Dynamic range   : 14.68 EV (cooked)


Here, "in theory" means from analyzing noise level in input exposure, and "cooked" means right before saving the output file (might include some noise reduction, interpolation artifacts and whatever other tricks I'm using).

So, the DR improvement in this test was 0.33 stops (on the highlight side).

What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 23, 2014, 11:25:59 AM
Audionut has sent me the corrected 5d3 stats, I changed them on my CR post (I didn't think this would make such a large splash, no ML development I have ever crossposted there has :-o)...

... one question that puzzles me: The DR for iso 800 is the same ML vs. Canon, what's this about? Also, did you do any recent changes that also invalidate the 6d calibration done by 1% this effect is also there?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 23, 2014, 11:35:02 AM
After changing the reference for the SNR graph to be able to compare the curves as in ejm's article, I hope to find out. Right now, I expect something like 1/100 ISO 100 might be as clean as 1/640 ISO 640 pulled from 800, but I did not run any tests to confirm this.

The calibration is still valid. You may want to fine-tune it for your camera, but I guess it will stay within +/- 0.05 stops from the defaults.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 23, 2014, 11:51:28 AM
just for my curisoty, (notting else, I can wait a long time for this coming to the 6d, take your time  ;D)
Since the 5d3 and the 6d are the same generation, sensor and digic wise.
How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 23, 2014, 12:04:29 PM
Quote from: Levas on January 23, 2014, 11:51:28 AM
How different are they for magic lantern, is it totally different meaning trial and error or do they share most of the stuff and can this relatively easy be ported to 6d ?

Wrong thread, but short answer: They seem to be very similar, but concerning ML the 6d is unmaintained and in an earlier state than 5d3, so features might make it to the 6d or not.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 02:40:00 PM
I correctly exposed @ ISO 100 and then boosted ISO to 1600.  Loaded all images with same ACR settings (WB from ISO 100 shot, no noise reduction etc & highlight slider -100) and brightness matched from the black patch top left.

My ADTG settings vs Canon vs Canon -1/3EV with shutter.
(https://s15.postimg.cc/cw615aj4b/mine.jpg) (https://s15.postimg.cc/ok030bdjf/Canon.jpg) (https://s15.postimg.cc/u9gbk4jkb/Canon-0.3.jpg)

Quote from: a1ex on January 23, 2014, 10:26:04 AM
What about the ~0.5-stop improvement at the ISO 1600? I believe it translates into larger overlap between the two ISOs => less aliasing.

Indeed, on top of the 0.3-0.5EV less shot noise.  Because it would be pointless to pull the highlights back if you don't then expose for them.  :D

If you're DR limited then you gain 0.3-0.5EV of latitude.  You can either gain that advantage in the highlights or the shadows.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Escaperoute on January 23, 2014, 02:44:08 PM
I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect
.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 23, 2014, 03:07:32 PM
For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine.  I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).

    else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
    {
        ADTG_WRITE_FUNC = 0xFFA11FDC;
        CMOS_WRITE_FUNC = 0xFFA12190;
        ENGIO_WRITE_FUNC = 0xFF97D904;  // from stubs
        ENG_DRV_OUT_FUNC = 0xff97d794;
    }
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 03:09:45 PM
Quote from: Escaperoute on January 23, 2014, 02:44:08 PM
I'm very much excited for this news, is it possible to port this module to 6D. And how much DR increase should i expect

Not reading the last page of a thread and the OP, is just bad form.  Surprisingly enough, both of these places have the answers to your questions!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 23, 2014, 05:40:12 PM
(https://s15.postimg.cc/mffpytr7v/Bell.png)
(https://s15.postimg.cc/tinleg4d7/Settings3.png)


Adobe doesn't have a problem with the black level, which is only offset from 0 to account for stdev (https://www.micro-manager.org/wiki/Measuring_camera_specifications#Measure_Offset_and_readout_noise_in_digital_numbers).


edit:  This one doesn't scale well with ISO.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 23, 2014, 09:38:20 PM
Added another analysis type to raw_diag: fixed-pattern noise (banding) from a dark frame (image taken with lens cap on). FPN is estimated by averaging each line (or each column), and the graph shows a plot of this correction "signal" and its stdev. It may be useful for testing the custom configurations from Audionut (since one of them attempted to reduce exactly this FPN).

(http://a1ex.magiclantern.fm/bleeding-edge/fpn.png)

(I've showed earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465;topicseen#msg96465) how to use this info to subtract the FPN a dark frame)

Source and binary in first post.

Some autocorrelation analysis would be nice, if anyone feels like doing the math.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 24, 2014, 12:10:37 AM
Nice a1ex!

Quote from: a1ex on January 23, 2014, 09:38:20 PM
(...)
Some autocorrelation analysis would be nice, if anyone feels like doing the math.
I have some pretty nice FPN reduction algorithms, maybe it could help (maybe it is too gourmand for the DIGIC...).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 12:15:57 AM
I'm interested. Especially if you can estimate the FPN from an arbitrary image without extra dark frames.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 24, 2014, 01:39:16 AM
Indeed, without master offset.
It's too big to post the code here, I send you a PM.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 24, 2014, 03:33:19 AM
Canon vs My tweaks (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/my_tweaks/settings2.BMP)
(https://s15.postimg.cc/rpkoqgja3/FPN_100_Canon.png) (https://s15.postimg.cc/75furzb8r/FPN_100_Mine.png)


(https://s15.postimg.cc/3ltx26g8r/FPN_1600_Canon.png) (https://s15.postimg.cc/xqidnjt1n/FPN_1600_Mine.png)


(https://s15.postimg.cc/9mrlz9iaj/FPN_6400_Canon.png) (https://s15.postimg.cc/ovhjd1jor/FPN_6400_Mine.png)


FPN should be reducing with increasing ISO.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 24, 2014, 08:02:53 AM
Quote from: a1ex on January 23, 2014, 08:25:45 AM
Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).

For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624.

On my 50D, the white level at ISO 100 is only 13432, so I can adjust those same registers to expand the range from 1023..13432 to 31..16383, which gives an apparent DR increase of 0.4 EV.

(https://dl.dropboxusercontent.com/s/i1cr5ff2fkawgkq/raw_diag_0279.png)(https://dl.dropboxusercontent.com/s/e8asngxvqsfspdt/raw_diag_0280.png)

I increased B/W Offset (0xc0f08034) from 0x1079 to 0x1c00 to max out the white level, and decreased Saturate Offset (0xc0f0819c) from 0xf87 to 0x20.  (I just now realized that these are essentially the same settings you cited for the 5D2.)  I find it curious that not only is this setting for B/W Offset "nice" as a multiple of 256, it also shifts the meaning of the Saturate Offset to correspond to the actual empirical black level (0x20 ~= 31).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 08:14:09 AM
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 01:06:34 PM
Just noticed that on 5D2, at ISO 3200 vs 6400, raw_diag reports exactly the same values for noise and DR (and raw zebras agree). However, the CR2 at ISO 6400 contains clearly less highlight detail, is brighter, has larger file size, and its histogram has gaps: http://www.guillermoluijk.com/article/isos5dmkii/index.htm

Could it be a multiplication of the raw values directly at CR2 compression stage?! (in any case, after ML intercepts the raw buffer)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on January 24, 2014, 02:22:38 PM
500D (default) :
(http://s22.postimg.org/jg91w114h/VRAM0.png)

500D (adtg 149 -> 100) :
(http://s22.postimg.org/yq8x37wmp/VRAM2.png)

500D (adtg 149 -> 0, bad white level) :
(http://s22.postimg.org/6eryjx241/VRAM3.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 02:26:00 PM
5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

Results:

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d2.png)


Canon ISO    ML ISO    Improvement
100          82        0.29 EV
200          165       0.27 EV
400          332       0.27 EV
800          692       0.21 EV
1600         1432      0.16 EV
3200         1432      1.16 EV


Forgot to mention that new ADTG GUI also has an option to group registers by task or program counter where the register was changed. With this trick, you should be able to identify the ADTG registers on EOS M (look for those in ShootCapture task or something similar, not for the LiveView ones).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 24, 2014, 03:04:24 PM
Quote from: a1ex on January 24, 2014, 08:14:09 AM
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.
The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001.  Here are the CR2 files for default settings (https://dl.dropboxusercontent.com/s/52au0i32lu62vrb/IMG_0279.CR2) and tweaked settings (https://dl.dropboxusercontent.com/s/z98wnby7c1bstbe/IMG_0280.CR2).

I haven't looked too closely at the CR2 files, but here's a quick comparison of their histograms (with the channels lumped together).  What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.
(https://dl.dropboxusercontent.com/s/4zyej10q9tar4v5/histogram_0279_0280.png)

Quote from: a1ex on January 24, 2014, 08:14:09 AM
(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)

At ISO 100, the default value of digital gain (0xc0f08030) is 0x1000, so I did not change it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 24, 2014, 03:58:34 PM
On 5D3:

Digital Gain is white level control.  Does not effect black.  Does not effect stdev.  Sweet spot is 463 with pushed black/white levels.  Relax the contrast (black/white levels), and values down to 461 will still deliver a Gaussian bell curve.

ADTG gain.  Not sure.  Sweet spot (for ISO 100 and modified Saturate offset) is 1002.  You can lower this value much further before affecting white level, but values lower then 1002 effect the bell curve.  This one effects stdev.

Black Reference.  No Idea.  Agreed a1ex, Canon seems to have set this at sweet spot.  I haven't been able to find any other values that deliver a bell curve.

B/W offset.  Gain, this one shifts the entire signal.  Tied to Saturate offset.

Saturate offset is black level control.  Does not effect white.  Tied to B/W offset.

Both offsets can deliver a Gaussian bell curve at different levels.  Note here where I've changed black level with Saturate offset (fixed white level).

(https://s15.postimg.cc/cfapovqij/2048.png) (https://s15.postimg.cc/ducadl4gb/999.png) (https://s15.postimg.cc/i3h0fqkkb/574.png)

And here where I have shifted the signal with B/W offset.

(https://s15.postimg.cc/o4epcup6z/15924.png) (https://s15.postimg.cc/pw7o7rg9n/16132.png) (https://s15.postimg.cc/fm598iy3v/16349.png)

There are sure to be other points for both controls where the histogram retains it's bell curve.
Note how stdev remains within shot to shot variation for all examples.
With Saturate offset at the minimum value (118) that still produces a bell curve (black level 574), I can push B/W offset to produce black levels 999 and 2048, matching the levels where saturate offset produced a bell curve with fixed white point, even with white point pushed way past 16383. 

CMOS/ADTG gains are clearly analog, they effect noise. 
Digital gain, B/W offset and Saturate offset seem to be digital level controls.  Probably only subject to quantisation errors.  They also clearly have defined points where they work well together (produce bell curve), and Digital Gain seems to only have 1 sweet spot (subject to contrast).


Suggested settings.
(https://s15.postimg.cc/qyhuqbm8b/Settings.png) (https://s15.postimg.cc/3k9vedwl7/Default.png)


These produce maximum contrast and Dynamic range, with white level recorded at nearly 14bits.  White level does not change with ISO.  At higher white levels, high ISOs will push the WL even further.


Left to confirm. 
What programs, if any, respect high white levels.  They should, what's the point in having 14 bits when you don't use them all  :o
ADTG gain needs to be adjusted on a per ISO basis, find correct values.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 24, 2014, 04:48:51 PM
Adobe respects the white level.

(https://s15.postimg.cc/idjxl7k1n/Adobe.png)


Also, to high ADTG gain value.
(https://s15.postimg.cc/w8i83wbyj/high_gain.png)

And to low ADTG gain value.
(https://s15.postimg.cc/ofrkbxt4r/low_gain.png)

Note the change in histogram.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on January 24, 2014, 05:07:14 PM
Quote from: a1ex on January 24, 2014, 02:26:00 PM
5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

(...)


Canon ISO    ML ISO    Improvement
100          82        0.29 EV
200          165       0.27 EV
400          332       0.27 EV
800          692       0.21 EV
1600         1432      0.16 EV
3200         1432      1.16 EV


(...)
Well done! :D

By seeing this, with an optimized pedestal, max WL and a good FPN reduction I think we can reach 12 stops, maybe 12.5 stops DR on the 5D2 for daylight (short exposure) photography.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 05:24:43 PM
(moved to next page because forum can't handle so many graphs)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 24, 2014, 05:39:00 PM
Quote from: a1ex on January 24, 2014, 08:14:09 AM
Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

Quote from: ayshih on January 24, 2014, 03:04:24 PM
The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001. ...  What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.

I take that back.  It looks like the overexposed regions are not solid white, and the pixel-to-pixel differences in saturation is what is producing that "rise" in the histogram.  Here are plots of where the pixels have values shy of the overall white level (in one of the green channels), for the default settings (13200..13400) and the tweaked settings (16100..16350).  I'd expect to see just the border of the overexposed region.
(https://dl.dropboxusercontent.com/s/r4ij8shlyq38gh5/pre-saturation_0279.png)(https://dl.dropboxusercontent.com/s/p59qhwveft4ojpi/pre-saturation_0280.png)

There does not appear to be any noticeable banding in the per-column black level when looking at the top black bar.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 05:43:19 PM
I choose the sweet spot as the minimum gain where a fairly large solid area is really solid white and stays at the original white level (except for isolated bad pixels). I don't go below this level (basically a binary search that looks at the output of this function and chooses the lowest gain that still results in the same white level as Canon):


/* calibration */
static int raw_center_white()
{
    int xc = (raw_info.active_area.x1 + raw_info.active_area.x2) / 2;
    int yc = (raw_info.active_area.y1 + raw_info.active_area.y2) / 2;
    int w = lv ? 100 : 500;
    int h = lv ? 100 : 500;
    int min = INT_MAX;
    for (int x = xc-w; x < xc+w; x++)
    {
        for (int y = yc-h; y < yc+h; y++)
        {
            /* a little bad pixel filter */
            int p1 = raw_get_pixel(x, y);
            int p2 = raw_get_pixel(x+1, y);
            int p3 = raw_get_pixel(x+1, y+1);
            int p4 = raw_get_pixel(x+1, y+1);
            int p = MAX(MAX(p1,p2), MAX(p3,p4));
            if (p > raw_info.black_level)
            {
                min = MIN(min, p);
            }
        }
    }
    bmp_draw_rect(COLOR_BLACK, RAW2BM_X(xc-w), RAW2BM_Y(yc-h), RAW2BM_DX(2*w), RAW2BM_DX(2*h));
    return min;
}
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 24, 2014, 07:09:50 PM
This is amazing!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 07:50:47 PM
Some new graphs:

5D3 and 5D2: I've measured them myself with latest raw_diag. Camera pointed to light bulb, left edge dark to avoid surprises, manual lens to exclude aperture trickery, shutter 1/30. I have simply read the DR number, with and without ADTG trick tuned to sweet-spot (didn't write down noise & other numbers).

Notice that ADTG trick reduces the actual ISO (by definition, it captures more highlights), so it makes sense to plot these new ISOs at the correct X position. This means there's no 0.5 stops improvement, but only 0.35 (what you get at ISO 100).

For 5D3, I've included 160 ISO multiples too. Notice these are not really ISO 160 according to clipping point, so I've estimated the real ones with iso_regs.mo (you can check the formula in the source code).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/6d.png)

For the others, I took the DR numbers from DxO (screen) and the calibration values from early testers. Take them with a grain of salt (I did not measure anything here).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/550d.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/60d.png)

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/650d.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/700d.png)

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/50d.png)

The script I used for these graphs, including test data (run it with Octave, might work in Matlab too): dr.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/dr.m). You are welcome to grab raw_diag and confirm (or infirm) the measurements.

Note the DR in these graphs is per-pixel (so they can't be compared directly if the cameras have different resolutions). If anyone would like me to normalize them (at say 8 megapixels or whatever), let me know.

I don't have calibration data for 600D (need a volunteer). Also I am 99% sure it will work on 1100D and EOS M (need volunteers who can read hexdumps). For 50D and 500D, I will have to take a closer look. 7D might require dual DIGIC trickery (didn't look at it yet).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 24, 2014, 08:00:41 PM
Quote from: a1ex on January 24, 2014, 07:50:47 PM
Notice that ADTG trick reduces the actual ISO (by definition, it captures more highlights), so it makes sense to plot these new ISOs at the correct X position. This means there's no 0.5 stops improvement, but only 0.35 (what you get at ISO 100).

Thanks for the clarification, actually I was already wondering about that, but felt too much intellectually challenged looking at all the data to ask :-o ... do you have a table with numbers from the grap with the actual dynamic range gained for each full stop iso?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 24, 2014, 08:09:30 PM
Yes, there's a link to the script right below the graphs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 24, 2014, 08:14:13 PM
Quote from: a1ex on January 24, 2014, 08:09:30 PM
Yes, there's a link to the script right below the graphs.

Thanks, I didn't realize I can just open this with notepad and copy/past the data.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 24, 2014, 09:51:22 PM
So, as far as iso is concerned, why did I buy a 5d3 over 5d2? ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 24, 2014, 09:59:56 PM
Quote from: hjfilmspeed on January 24, 2014, 09:51:22 PM
So, as far as iso is concerned, why did I buy a 5d3 over 5d2? ;)

The 5d3 (and more so: 6d) have less banding than 5d2, and this is what kills your dynamic range after lifting shadows much faster than just random noise. But of course as all reviews stated, the 5d3 does just about everything better than the 5d2 - but the sensor generation is the same. If you want high dr high res, get a d800.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on January 25, 2014, 01:07:34 AM
I know I know but its funny that the charts are showing better DR results with the 5d2. Believe me aside from native DR, I love my 5d3 + ML. I was about to sell and get a D800 but then ML saved me! 5D3 +ML is amazing. With these new ISOs, Im soooo excited.

So as this module evolves, are we going to need to input these settings and fine tune it for our specific cam or will it be more basic like "awesome ML ISO ON or OFF" ha ha either way Im happy!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on January 25, 2014, 02:24:46 AM
disregard
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on January 25, 2014, 03:14:07 AM
Quote from: Marsu42 on January 16, 2014, 11:53:17 PM
That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?

This is great stuff ... 0.3 to 0.5 DR improvement is nothing to sneeze at. This thread reminds me of the classic book from the early 80's, "Soul of a New Machine", except on Internet time at 3x to 10x to 100x faster pace.

Much of this thread is over my head, but I do have a concern that parameters are being optimally tweaked to specific cameras ... A1ex's 5d3 and AudioNut's 6d. Especially with analog, what works on AudioNut's 6d might be sub-optimal compared to the default on my 6d.

Or will the end result at release be tunable to a specific camera, like Auto-Dot-Tune? It might be like AudioNut using Auto-Dot-Tune with his 6d+50mm f1.8 and indicating the appropriate value was -7. Well, yes, on his camera, but unlikely to be optimal on my camera. And maybe even worse than the default of zero.

Or does that not apply?

FWIW: I think in an earlier post on this thread, someone mentioned that an incorrect value for some ATDG register (don't recall the specific) might mess up A-ETTR. Am I the only one concerned about such an issue?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on January 25, 2014, 03:21:16 AM
Quote from: Steve Kahn on January 16, 2014, 11:10:27 PM
At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.

I am also impressed by the image of the outdoor porch light. However, the left tube seems to be missing some of the yellow crud at the bottom. Or is that accounted for by the focus difference? The right side yellow crud seems mostly intact.

And would there be a difference between
* ATDG trick + Dual-ISO and
* "just" Dual-ISO?
That would seem to be a fourth combination.

Or do I have a flawed understanding of what is going on?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2014, 05:44:39 AM
Quote from: hjfilmspeed on January 25, 2014, 01:07:34 AM
So as this module evolves, are we going to need to input these settings and fine tune it for our specific cam or will it be more basic like "awesome ML ISO ON or OF" ha ha either way Im happy!

Right now you enter the low-level settings from iso_regs or adtg_gui, but I'm working on a user-friendly frontend.

Quote from: l_d_allan on January 25, 2014, 03:14:07 AM
Or will the end result at release be tunable to a specific camera, like Auto-Dot-Tune? It might be like AudioNut using Auto-Dot-Tune with his 6d+50mm f1.8 and indicating the appropriate value was -7. Well, yes, on his camera, but unlikely to be optimal on my camera.

It seems there's a small variation between cameras of the same model (I believe it's within +/- 0.1 stops). I'll include a calibration routines (so default settings would be fairly good - just turn it on - but you'll be able to find the sweet spot for your camera if you wish, or dial a custom ISO).

Also, if you shoot JPEG, you may be able to dial the ISO even lower, because with most picture styles, the white point seems to be a little lower than white level in raw.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 25, 2014, 08:53:30 AM
Quote from: a1ex on January 24, 2014, 02:26:00 PM
5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

For the 50D:
        SEND_DATA_TO_DFE_FUNC = 0xffa71598;


I confirmed that each of the four 1d0x registers changes the gain of a fourth of the columns, but I haven't yet done anything more (e.g., generating the analogue of your 5D2 graph).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2014, 09:05:28 AM
Nice. On 50D, from your older results, I believe you can recover all the highlights only via Saturate Offset (and you might even benefit from *increasing* the DFE gain, though it's probably not noticeable).

However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.

Can you tell me some default values for these DFE registers and send me a RAM dump via PM? (Debug -> Dump ROM and RAM -> RAM4.bin). You may also look them up yourself in the hexdump and only show me the relevant section (e.g. if register 1d02 is 0x456 and you search for 0x1d020456, you will find something that looks like an array of values for these registers; it should be fairly obvious where this array starts and stops).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 25, 2014, 09:20:32 AM
Quote from: a1ex on January 25, 2014, 09:05:28 AM
However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.

http://www.magiclantern.fm/forum/index.php?topic=7102.0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2014, 09:29:25 AM
@ayshih, can you write down a table with register values at all the ISOs? (photo mode, of course)

I'm interested especially in CMOS[0] (CMOS gain), C0F08030 (digital gain aka SHAD_GAIN), 0xC0F08034 (BW offset aka SHAD_PRESETUP), C0F0819C (Saturate Offset), and the DFE/ADTG gains. If you notice other relevant registers that are changed with ISO, you may include these too.

(actually I'm interested in a table with these values for all other cameras; simply write down the numbers from ADTG GUI)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 25, 2014, 09:55:00 AM
Quote from: a1ex on January 25, 2014, 09:05:28 AM
Nice. On 50D, from your older results, I believe you can recover all the highlights only via Saturate Offset (and you might even benefit from *increasing* the DFE gain, though it's probably not noticeable).

However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.

The low white level of 13432 is the case at only ISO 100.  At higher ISOs, the white level is 15760, so just like with the 5D2, decreasing the DFE gain could be useful.  (This does suggest that ISO 100 is handled differently in some way...)

Quote from: a1ex on January 25, 2014, 09:05:28 AM
Can you tell me some default values for these DFE registers and send me a RAM dump via PM? (Debug -> Dump ROM and RAM -> RAM4.bin). You may also look them up yourself in the hexdump and only show me the relevant section (e.g. if register 1d02 is 0x456 and you search for 0x1d020456, you will find something that looks like an array of values for these registers; it should be fairly obvious where this array starts and stops).

As an example, the default values for ISO 100 are:
1d02 = 0x419
1d04 = 0x41a
1d06 = 0x41e
1d08 = 0x41c

I'll send you the array by PM.

Quote from: a1ex on January 25, 2014, 09:29:25 AM
@ayshih, can you write down a table with register values at all the ISOs? (photo mode, of course)
Okay, I've already have some partial tables, but I might as well put together a comprehensive table tomorrow.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on January 25, 2014, 03:15:52 PM
hmm it looks like the DFE is the fpga in 5D2..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: PressureFM on January 25, 2014, 03:28:38 PM
Quote from: g3gg0 on January 25, 2014, 03:15:52 PM
hmm it looks like the DFE is the fpga in 5D2..

What would that imply?

Sorry if it comes off as being ignorant  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 25, 2014, 04:20:36 PM
trying to get adtg_gui to run on the 1100D, but I'm getting memory errors with CONFIG_GDB=y

underflow/overflow[31/0] malloc(0) at rbf_font.c:32, task ml_init

Allocated RAW   694kB, peak 716kB
malloc          157kB, 187kB used
AllocateMemory  946kB, 320 B used
shoot_malloc    30MB, 506kB used
stack_space     505kB


any ideas?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2014, 04:39:18 PM
The binary got bigger than 512K (MemSiz in the compile log). You can disable some debug stuff from features.h.

(there is a check for this, but only on cache-hacked boot, which I'm not quite convinced it's the way to go)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on January 25, 2014, 06:20:30 PM
Dunno what your total free is but you can try boosting the bin size to 640K or getting more free memory by putting the bvram mirror in another memory area. esp. just to get GDB working.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 25, 2014, 06:27:58 PM
Okay I got GDB working, but adtg_gui fails to load, 'Module Init Failed' is all I can tell you, the console text is so small it's unreadable. I already tried the free_space / 4 thing you mentioned. Free memory is 272K + 894K
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2014, 06:43:47 PM
You need the stub addresses for ADTG/CMOS functions for 1100D (in adtg_gui_init). If you trust my automatic tools, they should be:

    MakeName(0xFF2DBB28, "reg_Start_ADTG");
    MakeName(0xFF2DBD1C, "reg_Start_CMOS");

For the dropbox version (which requires some more RAM, but also catches more registers), you may also want the engio functions (from stubs).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: gravitatemediagroup on January 25, 2014, 11:42:22 PM
don't anybody kill me but is the latest nightly build the 15bit build or is that still a work in progress? 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 26, 2014, 12:10:59 AM
What does 15bit have to do with anything?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 08:29:38 AM
Added 50D graph, thanks to @ayshih for measurements. Looks kinda tricky, the only real improvement seems to be at ISO 100.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/50d.png)

From 5D2, I've combined the DFE gain with the previous tweak (SaturateOffset) and squeezed a tiny bit more DR (0.31 vs 0.29). Graph updated.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 26, 2014, 09:25:37 AM
Here's the table of all the data I've collected for the 50D.  The highlights:
Link to Google Docs (https://docs.google.com/spreadsheet/ccc?key=0Aqd_Zo2KQfZHdHBsVE5KZ3VsY3Vsd3FBVW1CeUtmSFE&usp=sharing)
(https://dl.dropboxusercontent.com/s/p38s6hra7ne2ilc/50D_ISO.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: budafilms on January 26, 2014, 09:41:30 AM
Great work!
Can we look two images taking with Canon Iso and ML Iso?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 26, 2014, 10:30:09 AM
http://www.magiclantern.fm/forum/index.php?topic=10111.msg96563#msg96563
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 26, 2014, 05:50:18 PM
Is it possible to make the camera use the actual iso values of mini_iso like ISO 77, or is Canon hardwired to the usual 1/3 ev stops?

The reason for the request is that in plain Av/Tv (and probably P) mode, flash users cannot easily benefit from the dr gain as a permanent +ec also modifies the ettl exposure - you also have to juggle with the flash ec, it's the same as with dual_iso but in the other direction.

Edit: In case you non-flash shooters don't know, the ettl flash ec isn't linear, at least on 60d/6d the effect from 0->1/3 is much stronger than the change from 1/3->2/3->... so to gain a "neutral" fill flash you really want to use 0 flash ec.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 05:55:17 PM
I can't imagine a clean solution, but I also don't understand why a +EC wouldn't do the trick.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 26, 2014, 06:28:49 PM
Quote from: a1ex on January 26, 2014, 05:55:17 PM
I can't imagine a clean solution, but I also don't understand why a +EC wouldn't do the trick.

I suspect so as you don't do the flash thing. In Av/Tv with flash on, ec doesn't change the exposure but the *ratio* between flash and ambient illumination - the general exposure stays always "neutral" and the same as metered by the camera unless you change *both* camera and flash ec.

Catch as written above is that flash ec for some reason isn't (always) linear, +-1/3 often has a very large effect while further flash ec changes are more or less like you'd expect them to be. That's why changing flash and camera ec to 1/3 to compensate for the gained dynamic range isn't as easy as it sounds.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: romainmenke on January 26, 2014, 07:55:36 PM
This is really amazing.
I work mostly on ads and editorial portraits. So I always use strobe light and I almost always end up setting up a strobe as a fill to take down the contrast a bit.
Really curious how these new iso's will compare to the fill light with the canon iso's

I know it isn't really the same, but instead of flattening my light and adding contrast in post I might be able to shoot more stuff the way I actually like it.
Reminds me of shooting on film. Slightly overexposing and under developing to get nice shadow detail.

I was also wondering about something.
The ISO 50 you can choose. Is it iso 100 on the sensor and then adjusted in the conversion to digital?
If so you probably also lose DR with ISO 50.

I didn't see any mention of it on the charts and they go down towards the higher ISO's put probably also lower towards lower Canon Iso's because the only good one is 100?

Curious because I often use 50 to get a wider aperture.
Thanks



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 07:57:20 PM
ISO 50 and ISO 100 are identical.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 08:14:09 PM
Quote from: Marsu42 on January 26, 2014, 06:28:49 PM
I suspect so as you don't do the flash thing. In Av/Tv with flash on, ec doesn't change the exposure but the *ratio* between flash and ambient illumination - the general exposure stays always "neutral" and the same as metered by the camera unless you change *both* camera and flash ec.

Catch as written above is that flash ec for some reason isn't (always) linear, +-1/3 often has a very large effect while further flash ec changes are more or less like you'd expect them to be. That's why changing flash and camera ec to 1/3 to compensate for the gained dynamic range isn't as easy as it sounds.

I still don't get it, can you show this with some images? (include the raw files too)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 26, 2014, 10:05:21 PM
Quote from: a1ex on January 25, 2014, 06:43:47 PM
    MakeName(0xFF2DBB28, "reg_Start_ADTG");
    MakeName(0xFF2DBD1C, "reg_Start_CMOS");
These appear to be correct.


ISO 100 200 400 800 1600 3200 6400
CMOS[0] 0x0 0x120 0x240 0x360 0x480 0x5a0 0x5a0

ADTG2[8882] 0x417 0x41c 0x421 0x430 0x427 0x428 0x850
ADTG2[8884] 0x424 0x428 0x426 0x437 0x429 0x42a 0x854
ADTG2[8886] 0x425 0x429 0x428 0x438 0x42b 0x42d 0x85a
ADTG2[8888] 0x419 0x41b 0x421 0x430 0x426 0x428 0x850

ADTG2[8839] 0x1 0x1 0x2 0x2 0x3 0x3 0x3

ADTG2[88bc] 0x0 0x0 0x0 0x0 0x0 0x0 0xff
ADTG2[88be] 0x1e 0x1e 0x1e 0x1e 0x1e 0x1e 0x708
ADTG2[88c0] 0x8 0x8 0x8 0x8 0x8 0x8 0x180
ADTG2[88eb] 0x850 0x850 0x850 0x850 0x850 0x850 0x900

Interestingly enough, ISO3200 appears to be analog on the 1100D. There are no intermediate ISOs. This was done at f/1.8, I'll compare that to a manual lens shortly...

edit: well thats dumb, the forum seems to be unaligning my columns, they're all lined up in preview mode
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 10:09:01 PM
Quote
There are no intermediate ISOs.
Not even from ML menu?

About the dump, you need to lookup 0x88820417 in the RAM dump (or just send it to me).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 26, 2014, 10:11:09 PM
Yes ML can, I meant for Canon, I was just testing the actual Canon ISOs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 10:14:03 PM
On 550D they were simply hidden from menu.

For 1100D I'm very curious about DR measurements, mostly because of your astro pics. I guess you use the 1100D for a reason.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 26, 2014, 10:25:36 PM
Quote from: a1ex on January 26, 2014, 10:14:03 PM
For 1100D I'm very curious about DR measurements, mostly because of your astro pics. I guess you use the 1100D for a reason.
Yeah, I'm working on that, I have to do it basically manually, 1100D doesn't have raw stuff working so raw_diag won't load.

It does have larger pixel pitch due to the smaller pixel count. That's also probably why ISO3200 is analog.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 26, 2014, 10:35:09 PM
Photo raw should have no trouble in running on 1100D (though somebody needs to find the hooks; most likely similar to 600D). LiveView is troublesome, I know, but it's not needed for these tests.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 27, 2014, 04:43:47 AM
Quote from: Audionut on January 24, 2014, 03:58:34 PM
B/W offset.  Gain, this one shifts the entire signal.  Tied to Saturate offset.

Saturate offset is black level control.  Does not effect white.  Tied to B/W offset.

Digital gain, B/W offset and Saturate offset seem to be digital level controls.  Probably only subject to quantisation errors.  They also clearly have defined points where they work well together (produce bell curve), and Digital Gain seems to only have 1 sweet spot (subject to contrast).

Saturate offset is not adjusting the black level per se, it's only adjusting the offset*.
B/W offset isn't any form of gain, it's digitally shifting the analog signal into the recorded bit depth.

When they produce a negative effect on histogram, simply they have not been optimised for the incoming analog signal, and the round off errors have become large.

Digital gain would seem to be the gain of the digital stage.  Judging by the effect on the histogram, this would appear to be gaining the digital signal.  ie:  1 in, 1.5 out.  Rather then using gain to boost the digital signal, it's preferable IMO to use B/W offset to map the bit depth of the recorded signal.

I can gain an extra hundredth or two of DR, by not adjusting the black level offset lower (as I was in my earlier tests), and leaving more room for ADTG gain reduction.  Such a shame that CMOS gain cannot be reduced further, Canon is probably (over)saturating that gain also.

Quote from: a1ex on January 10, 2014, 12:11:01 PM
Does this mean Canon did not fully optimize their sensor for low noise?


I'd say they simply left a safety margin in their code to make sure the ADC is always saturated (that is, to make sure white is always recorded as white).

*The offset is so far down into the noise floor that it does not effect black in the recorded image.  The offset can be adjusted higher then the default value, but there doesn't appear to be any gain to DR.

http://www.cool.id.au/astronomy/Neb_tut/Scientifically%20determining%20CCD%20Gain%20and%20Offset.pdf
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 09:02:59 AM
Quote from: Audionut on January 27, 2014, 04:43:47 AM
Such a shame that CMOS gain cannot be reduced further, Canon is probably (over)saturating that gain also.

Quote from: dmilligan on January 26, 2014, 10:05:21 PM


ADTG2[8839] 0x1 0x1 0x2 0x2 0x3 0x3 0x3

ADTG2[88bc] 0x0 0x0 0x0 0x0 0x0 0x0 0xff
ADTG2[88be] 0x1e 0x1e 0x1e 0x1e 0x1e 0x1e 0x708
ADTG2[88c0] 0x8 0x8 0x8 0x8 0x8 0x8 0x180
ADTG2[88eb] 0x850 0x850 0x850 0x850 0x850 0x850 0x900


I smell something here, though I can't tell yet exactly what. I did get an extremely low ISO (something like ISO 1) on 5D2, but very noisy.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 27, 2014, 11:26:22 AM
Quote from: a1ex on January 27, 2014, 09:02:59 AM
I smell something here, though I can't tell yet exactly what. I did get an extremely low ISO (something like ISO 1) on 5D2, but very noisy.

Really low iso, like 1, how was the dynamic range, normal like 11 stops ?
This way there's less need for ND filters...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 11:30:38 AM
More like 1-2 stops.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on January 27, 2014, 11:35:02 AM
Quote from: a1ex on January 27, 2014, 11:30:38 AM
More like 1-2 stops.

1 -2 stops  :'(  , can't see something useful for me in that, maybe for people creating modern art...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 27, 2014, 04:14:49 PM
Based on my calculations for the 1100D at ISO100, -1 EV of the ADTG gains resulted in DR increase from 10.7 -> 10.9, std dev went from 8.1 to 4.4

I'll do more analysis after work...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 04:22:37 PM
5D2:

CMOS[4] = 0x242 at iso 100...800 and 0x244 at iso 1600. Setting it to 0x[46ce]4[24] (e.g. 0x442 at low ISO or 0x444 at high ISO) seems to squeeze 0.2 more stops of DR. Not yet sure if in highlights or shadows (I believe it's in shadows, since clipping point didn't change); feel free to pixel-peep and tell me.

5D3:

CMOS[3] = 0x944 at lower ISO and 0x144 at higher ISO. Doesn't seem to make a difference.
CMOS[4] = 0x718. Set it to 0x300...0x37f to squeeze 0.1 stops of DR. Setting he 0x80 bit results in white image. Other values like 0x100, 0x200, 0x400 result in different noise/DR levels.
CMOS[6] and [7] seem to do something about white level clipping and darkening the image. No clear conclusion yet.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d2-cmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-cmos.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dspographer on January 27, 2014, 05:49:16 PM
I haven't read this entire thread, so I don't know if this has been discussed: but the raw converter should be updated to be able to handle the new type of saturation. I asked the libraw author what needs to be done and this was his answer:
http://www.dpreview.com/forums/post/52994258
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 06:05:26 PM
Short answer: current raw converters will work unmodified.

However, for best results, I believe raw converters should use an autodetected white level, instead of assumming a fixed value. This is especially important *without* this tweak.

With plain Canon firmware, white level varies wildly with intermediate ISOs and also with aperture. With the tweak, I'm disabling these variations.

There are also white level variations at long exposures. I can't do much about these.

There are no nonlinear issues in Canon sensors as far as I could tell. When it clips, it clips abruptly. There are, however, bad pixels with values above the clipping point, and one should be careful about this when autodetecting the white level.

I'm planning to submit a patch to dcraw and ufraw, with white level autodetection. Dcraw assummes fixed values around 13000...15600, depending on the camera model, which is plain wrong (sorry). I believe Dave simply took the white level from one of the test shots he received (because I could get his exact white level by adjusting the aperture). I believe this is the cause of most (if not all) pink highlight problems with dcraw-based converters.

Things might be different with Dual ISO, but that's another story (that's something I'm handling in the converter).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 27, 2014, 07:18:09 PM
(https://s15.postimg.cc/ki4amiyln/canon.png) (https://s15.postimg.cc/uffbfldx7/image.png)

I had a screenshot with DR @ 11.494 with a little less ADTG gain, but I lost it somewhere.  Still, this one is 0.13EV better then my previous best (so close to 11.5). 
I should be in bed, not tweaking registers  :o
I'll post some more information tomorrow.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 07:21:41 PM
With a little less ADTG gain than calibrated value, the DR reported by raw_diag may be false (overestimated), because you'll get variations in clipping point.

If in doubt, recalibrate the gain. This will make sure everything clips at the same point.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 27, 2014, 07:23:39 PM
Sorry, it's late.

A little less negative gain.  So a higher register value.  :)

The histogram wasn't as clean, but DR was higher.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 07:27:56 PM
I believe DR is also influenced by roundoff errors, so I'm not sure what to say yet.

Maybe using a more robust indicator instead of stdev (e.g. MAD (http://en.wikipedia.org/wiki/Median_absolute_deviation)) makes more sense?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 27, 2014, 07:33:46 PM
Quote from: a1ex on January 27, 2014, 06:05:26 PM
There are no nonlinear issues in Canon sensors as far as I could tell. When it clips, it clips abruptly.
This seems unlikely to me, electronics are by nature non-linear, esp. near saturation. What was your methodology?

http://photo.stackexchange.com/questions/33984/how-linear-are-dslr-sensors/33985#33985
Quote
However when a photosite is nearing saturation various electrical effects reduce the charge generated by incident photons leading to a roll-off of the response curve (i.e. it becomes shallower).

Rodger Clark describes a methodology for determining the linearity of the sensor reponse (among other things) in this article:
http://www.clarkvision.com/articles/evaluation-1d2/
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 07:35:24 PM
Quote from: dmilligan on January 27, 2014, 07:33:46 PM
What was your methodology?

cr2hdr --iso-curve

Quote
However when a photosite is nearing saturation various electrical effects reduce the charge generated by incident photons leading to a roll-off of the response curve (i.e. it becomes shallower).

I wish it was like that, but could not find any sign of it. A graph from last night, with ADTG gain lowered much under the sweet spot:

(http://a1ex.magiclantern.fm/bleeding-edge/nonlinear.png)

(I could not find any correlation between these variations and the actual signal data, so I believe they are just noise)

Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 08:07:11 PM
Quote from: a1ex on January 27, 2014, 06:05:26 PM
Short answer: current raw converters will work unmodified. However, for best results, I believe raw converters should use an autodetected white level, instead of assumming a fixed value. This is especially important *without* this tweak.

Is this also true for ACR, or only for the oss ufraw/dcraw, i.e. when using Adobe apps, will using mini_iso result in artifacts or other problems? From what I've tested for myself everything looks fine, but then again I didn't know what problems to look for.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 27, 2014, 08:12:02 PM
Quote from: Audionut on January 27, 2014, 04:43:47 AM
Saturate offset is not adjusting the black level per se, it's only adjusting the offset*.
B/W offset isn't any form of gain, it's digitally shifting the analog signal into the recorded bit depth.
...
*The offset is so far down into the noise floor that it does not effect black in the recorded image.  The offset can be adjusted higher then the default value, but there doesn't appear to be any gain to DR.

It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR.  However, perhaps I am reading too much into your choice of words.  SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level, which has DR consequences.

In my mental picture of the electronics, there are two relevant DACs:
Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space.  There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.

On the other hand, SaturateOffset is more than just an offset because it comes before at least one stage of amplification.  At the output of the amplifier, it becomes directly related to the separation between the black level and white level (which is the saturation at the rail of the amplifier).  Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease.  Conversely, decreasing SaturateOffset must increase your DR.  (The changes in DR may be negligible if the register change is small).

For example, at ISO 100 on the 50D, you can achieve most of the DR improvement by just changing these two registers, without touching any amplifier gains.  (While changing B/W offset itself doesn't change DR, increasing it provides more room to decrease SaturateOffset.)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 08:16:56 PM
Quote from: Marsu42 on January 27, 2014, 08:07:11 PM
Is this also true for ACR, or only for the oss ufraw/dcraw, i.e. when using Adobe apps, will using mini_iso result in artifacts or other problems? From what I've tested for myself everything looks fine, but then again I didn't know what problems to look for.

The only problems that may appear are either pink highlights or too many clipped highlights. Adobe DNG Converter seems to use a white level fixed to 15000, but it's important how the data above this level is actually handled (wheter it's used for highlight recovery or not).

You can check with "exiftool IMG_1234.dng -WhiteLevel=3000"; if you can recover the highlights, all is fine. With ufraw you can't.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dspographer on January 27, 2014, 08:22:18 PM
a1ex:
Have you looked at saturation vs which of the parallel read channels is being used? I am wondering if the 2 apparent clipping points in your plot are from different read channels.
Quote from: a1ex on January 27, 2014, 07:35:24 PM
cr2hdr --iso-curve

I wish it was like that, but could not find any sign of it. A graph from last night, with ADTG gain lowered much under the sweet spot:

(http://a1ex.magiclantern.fm/bleeding-edge/nonlinear.png)

(I could not find any correlation between these variations and the actual signal data, so I believe they are just noise)

Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 08:24:28 PM
Yes, they are. Each column has its own clipping point (but there are two groups, indeed). The pattern repeats roughly every 4 columns, but there are clipping point variations on top of that.

I've removed the constant offsets and looked in the clipped area, but could not find any image details. I will do a controlled experiment and post a detailed analysis if there are people interested; this one was just from troubleshooting a dual ISO shot from 6D.

But this is not what you are getting normally in a CR2 file. In a CR2, things are simply clipped sharply (at the level where the mouse cursor is in my graph). This graph could be relevant for recovering even more DR with some clever algorithm, but from my early tests it seems quite unlikely.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dspographer on January 27, 2014, 09:09:22 PM
Have you done any studies to see how stable the clipping point is? I am wondering if the fixed level chosen for .CR2 files might be too high for some circumstances. Some major things that come to mind that might change the clipping point are: battery voltage, temperature, and camera to camera variation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 09:26:37 PM
At short exposures I get the same white level every day (checked with raw_diag, with these tweaks enabled and ADTG gain at sweet spot, not below). On 5D2 I've configured it at 16380, but I can choose any other value.

At long exposures, white level may become lower, but this also happens with Canon firmware.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 27, 2014, 09:29:30 PM
Quote from: a1ex on January 27, 2014, 07:35:24 PM
Also, I could not see any nonlinearity on the graph from Roger Clark. So, for practical purposes, the data is linear and clips abruptly.
Right, but he was not using your ADTG trick, nor the same camera or even the same generation of camera. I simply posted that article as a demonstration of methodology, the idea being that you or someone else carry out that same methodology with the ADTG trick to determine if we are reaching into the non-linear part of the sensor response.

The basic idea is that you take pairs of exposures and subtract them, this residual is simply the square of the noise of one of the exposures, since photon noise is simply the sqrt of the number of photons, we can determine the true reponse in photons based on the std dev of the residual frame.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 09:39:21 PM
Quote from: dmilligan on January 27, 2014, 09:29:30 PM
The basic idea is that you take pairs of exposures and subtract them, this residual is simply the square of the noise of one of the exposures, since photon noise is simply the sqrt of the number of photons, we can determine the true reponse in photons based on the std dev of the residual frame.

I'm actually using this trick to get an estimation of the noise from bracketed shots on a sturdy tripod (download the samples from engardeknave from the CeroNoice thread), and I found the noise estimated in this way to be very good regarding Gaussian shape and lack of autocorrelation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jkaura on January 27, 2014, 10:11:08 PM
Quote from: a1ex on January 27, 2014, 08:16:56 PM
You can check with "exiftool IMG_1234.dng -WhiteLevel=3000"; if you can recover the highlights, all is fine. With ufraw you can't.

Tried this briefly with one DNG image file in Adobe Lightroom 5 and, indeed, the highlights were lost. After setting the white level to 3000 in the EXIF metadata, one cannot recover the lost highlights using the conventional Exposure, Highlights, White or Tone Curve adjustments.

In fact, my attempts to restore the original white level back by executing: exiftool IMG_1234.dng -WhiteLevel=15000

on the command line did not succeed completely even after reading the metadata back from the file. Changing DNG camera profile from Adobe Standard to Camera Standard on the Camera Calibration tab fixed the issue temporarily but after returning from the Library module back to Develop module, the highlights seemed to be lost again. I should better comprehend myself with the Lightroom's metadata management since I don't quite understand how reverting the white level setting (at EXIF level) should be done in Lightroom. Anyway, it seems to me that the most recent release of this software does not resolve the white level from the actual image data.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 10:20:12 PM
Quote from: jkaura on January 27, 2014, 10:11:08 PMI should better comprehend myself with the Lightroom's metadata management since I don't quite understand how reverting the white level setting (at EXIF level) should be done in Lightroom. Anyway, it seems to me that the most recent release of this software does not resolve the white level from the actual image data.

Thanks for trying this - so if I understand correctly, acr (or at least lr) cannot harvest the dynamic range gained from using mini_iso?

Btw I wouldn't have expected that simply changing one exif tag is enough for acr, I guess they're using a bunch of maker notes and dng tags for the technical stuff and the exif data is only there for a limited subset of image property descriptions.

Last not least, if saving/restoring metadata doesn't do it, maybe it's worth a try to remove the dng from the catalog and then re-add it, maybe not all "internal" tags Adobe doesn't expect to change are updated when reading an exif-modified dng (just a guess).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 10:22:31 PM
I'm asking you to check that, so I know what level I should choose in mini_iso (and whether it makes sense to make it configurable).

@jkaura: I think this explains why I've got better highlight recovery (compared to ACR) with a really dumb algorithm: http://www.magiclantern.fm/forum/index.php?topic=9458.msg91590#msg91590

and from your findings, I believe that - on top of the DR graphs I'm plotting - you will also get back some more highlights at wide apertures, that might be lost because of digital amplification of the raw data. Check the first post for details.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 10:29:55 PM
Quote from: a1ex on January 27, 2014, 10:22:31 PM
I'm asking you to check that, so I know what level I should choose in mini_iso (and whether it makes sense to make it configurable).

Make what exactly configurable - the cr2 tags aren't modified by mini_iso, or are they?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 10:33:52 PM
The CR2 doesn't have any tag for white level, but I can change the range of recorded data (black and white level).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jkaura on January 27, 2014, 10:44:01 PM
Quote from: Marsu42 on January 27, 2014, 10:20:12 PM
Last not least, if saving/restoring metadata doesn't do it, maybe it's worth a try to remove the dng from the catalog and then re-add it, maybe not all "internal" tags Adobe doesn't expect to change are updated when reading an exif-modified dng (just a guess).

Thanks for the tip, that worked for me! After removing the photo from the catalog and reimporting it, the white level seems to be again at a normal level.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 10:51:48 PM
Quote from: jkaura on January 27, 2014, 10:44:01 PM
Thanks for the tip, that worked for me! After removing the photo from the catalog and reimporting it, the white level seems to be again at a normal level.

Yippee, I'm contributing something useful again for a change :-p

Quote from: a1ex on January 27, 2014, 10:33:52 PM
The CR2 doesn't have any tag for white level, but I can change the range of recorded data (black and white level).

Meaning as in "keeping the increased dr, but just recording it with different 14bit (16bit?) values", or as in "dimming down the effect of mini_iso because acr currently cannot cope"?

Edit: In any case, an alpha mini_iso.mo with advanced configuration for this to debug probably is a good idea, esp. if the source code isn't available yet.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 10:55:01 PM
Remapping the recorded range has nothing to do with DR, they are decoupled.

... except maybe for some round-off errors :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 10:58:31 PM
Quote from: a1ex on January 27, 2014, 10:55:01 PM
Remapping the recorded range has nothing to do with DR, they are decoupled.

Ok, thanks for explaining, in this case why not stuff the data in the original range that is used w/o mini_iso to avoid any compatibility problems ... or am I still mis-understanding what is going on? If I want to try to debug any regressions with acr, I should at least have a vague idea :-p
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 27, 2014, 11:08:45 PM
1) At larger apertures, Canon amplifies the raw data digitally (with the digital gain register), which works by changing the recorded range. That means, a raw processor that autodetects white level will not have any side effects (except for f1.4 and f1.2 where the white level overflows above 16383), but a raw processor that assummes a constant white level (most common ones I think) will effectively interpret this as a higher ISO (and throw away whatever gets above the "declared" or "assummed" white level). For Adobe DNG Converter (and probably other Adobe apps), this level seems to be 15000; for dcraw, it varies with camera model.

2) Right now I'm fixing the white level at the same level Canon uses with a manual lens (or at f8). However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.

So, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on January 27, 2014, 11:18:21 PM
Quote from: a1ex on January 27, 2014, 11:08:45 PMSo, it looks like it's probably a good idea to choose the white level from menu at values used by common raw processors (15000, 15600, 16380 and so on).

Yes, that does sound like a good idea - maybe the oss converters might accept patches to have a more flexible white level, but acr most likely won't ... but it would be nice to be able to use DxO for their prime noise reduction then and again with an acr-cr2  if that's possible, maybe they happen to have the same clipping point.

If you've got an acr-calibrated mini_iso.mo, I'd certainly like to test drive that :->
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 28, 2014, 02:54:01 AM
Quote from: a1ex on January 27, 2014, 11:08:45 PM
However, if this level (camera-specific) is around 15600, and ACR clips at 15000, this may not be the best choice.

Quote from: Audionut on January 24, 2014, 04:48:51 PM
Adobe respects the white level.

(https://s15.postimg.cc/idjxl7k1n/Adobe.png)

Note the maximum pixel value of the sample patch in RawDigger. 

MAD does seem like a good idea.  It sounds like it will help narrow down DR increases to the signal itself without the precision errors having an effect.


Quote from: ayshih on January 27, 2014, 08:12:02 PM
It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR.  However, perhaps I am reading too much into your choice of words.

It was not my intent to imply that it wasn't useful.  However, I went on to describe that more gains can be made by not reducing saturate offset, and hence, leaving more room for ADTG gain reduction.

Quote from: Audionut on January 27, 2014, 04:43:47 AM
I can gain an extra hundredth or two of DR, by not adjusting the black level offset lower (as I was in my earlier tests), and leaving more room for ADTG gain reduction.

Lowering the offset does increase DR.  However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).

Quote from: ayshih on January 27, 2014, 08:12:02 PM
SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level..........................Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease.  Conversely, decreasing SaturateOffset must increase your DR.

The key here IMHO is your description (and my observation) of it just being a digitized value.  Did we really increase the DR of the signal (the photons that deliver our image)?  I would guess not, we just digitally cheated.  This black level that is being changed is so far down into the noise floor to be negligible.  Perhaps it does shift the entire signal (black weighted) down and increases the DR through the signal.  But IMO, by the time we move up to a signal level that is outside of the noise floor, the gains would be minimal at best.

With ADTG gain reduction, we do decrease the noise.  So IMHO, it's much more preferable (in increasing DR) then saturate offset.

Quote from: ayshih on January 27, 2014, 08:12:02 PM
In my mental picture of the electronics, there are two relevant DACs:

  • The first DAC produces the reference voltage for the black-level clamp, and this clamping occurs before the ADTG/DFE amplification stage.  The corresponding register is SaturateOffset (0xc0f0819c).
  • The second DAC produces an analog offset that is added to the signal after both amplification stages, before it is fed into the ADC.  The corresponding register is B/W offset (0xC0F08034).
Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space.  There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.

If this is true (and I am not saying otherwise), then it would appear that the analog reference voltage is only having an effect in the digital domain.  Noise (stdev) does not increase or decrease.  I think of it (reference voltage) as saying to the devices later in the chain, map black to this voltage.  Also, it seems clear that the reference voltage has a fixed level of noise.  The voltage per se hasn't changed the photons we are interested in, it's simply an analog instruction to a digital device.

B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth.  B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth.  Here, the gain is bit precision for the entire signal.

Take it with a grain of salt, I don't have a deep understanding of the electronics involved.  I have only got observational skills and common sense going for me  ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 28, 2014, 03:33:33 AM
Quote from: Audionut on January 28, 2014, 02:54:01 AM
Lowering the offset does increase DR.  However, it does not reduce noise (stdev) and the DR increase comes from the simple fact that DR = log2(saturation-offset/stdev).
...
With ADTG gain reduction, we do decrease the noise.  So IMHO, it's much more preferable (in increasing DR) then saturate offset.

If you take a look at my 50D numbers (http://www.magiclantern.fm/forum/index.php?topic=10111.msg98056#msg98056"), you can see the following cases starting at Canon's ISO 100:
(Note that the 0.02 DR difference is below the uncertainty of the DR calculation and the reproducibility of the mini_iso calibration.)  It's true that changing the offsets doesn't decrease the noise as measured in ADUs, while the width in ADUs of the noise does decrease when reducing gain.  However, in both cases, the SNR at typical brightness levels hasn't actually changed.  Instead, what you get in both cases is increased range in the highlights.  Getting better SNR in the shadows requires you to ETTR to make use of this increased range.

Quote from: Audionut on January 28, 2014, 02:54:01 AM
B/W offset (which acts in the same manner) is more useful IMHO, because of bit depth.  B/W offset doesn't increase or decrease the DR of the signal, simply that it pushes the signal into a higher recorded bitdepth.  Here, the gain is bit precision for the entire signal.

This is not correct: B/W offset has no effect on precision.  If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 28, 2014, 04:19:56 AM
Quote from: ayshih on January 28, 2014, 03:33:33 AM
Getting better SNR in the shadows requires you to ETTR to make use of this increased range.

I make a point of ETTR in my tests for this reason.  Of course, the increased SNR is through the entire exposure.

Quote from: ayshih on January 28, 2014, 03:33:33 AM
This is not correct: B/W offset has no effect on precision.  If you shift the range of digitization from 1024..15000 to 2048..16024 by increasing the offset by 1024, you are using larger numbers, but the width of each bin doesn't change.

Indeed, thanks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 28, 2014, 07:28:11 PM
Now that we have raw_diag working properly, here's more data for the 1100D

        Canon                   ADTG[888x] /= 2     
ISO     σ       White   DR      σ       White   DR
100     6.17    13583   10.869  3.1     9714    11.273
200     6.54    15303   10.985  3.25    10956   11.421
400     7.55    15303   10.778  3.81    10998   11.196
800     10.4    15303   10.315  5.26    11033   10.737
1600    16.7    15303   9.632   8.64    11092   10.031
3200    29.7    15303   8.802   14.97   11111   9.242
6400    59.89   15303   7.79    15.55   11110   9.187
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on January 29, 2014, 04:33:54 AM
The white point looks a little low.  Are you applying to much ADTG gain reduction?

With no changes to digital gain, I can keep reducing ADTG gain and increase DR.  After tweaking the digital gain though, further reduction to ADTG gain (after the sweet spot) simply reduces WL.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on January 29, 2014, 07:03:35 AM
Once you've decreased the gain enough that your white level is dropping, you've already gone too far.  Note that even if the white level is unchanged as measured by raw_diag, it's still possible for the gain to be too low, because you may no longer have solid white highlights due to non-uniformity of saturation.  See these images (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97761#msg97761) to see what can happen in the highlights (there, from changing digital offsets rather than gain, but it's functionally equivalent).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on January 29, 2014, 04:33:12 PM
I have no idea what I'm doing ;)

You're right, I can raise the gain a little bit and keep the white level the same without really impacting the DR. The sweet spot seems to be around 0x330. Here's some new numbers.


        Canon                   ADTG[888x] = 0x330     
ISO     σ       White   DR      σ       White   DR
100     6.17    13583   10.869  4.65    13583   11.277
200     6.54    15303   10.985  4.92    15303   11.397
400     7.55    15303   10.778  5.76    15303   11.168
800     10.4    15303   10.315  7.83    15303   10.726
1600    16.7    15303   9.632   12.6    15303   10.038
3200    29.7    15303   8.802   22.11   15303   9.227
6400    59.89   15303   7.79    22.10   15303   9.228
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 31, 2014, 09:13:19 PM
Got some small updates to research tools:

raw_diag:
- JPEG curve analysis, see http://www.magiclantern.fm/forum/index.php?topic=10038.msg99049#msg99049
- cross-covariance analysis of FPN between two dark frames (it gives a hint about how much FPN you can fix from a dark frame)
- option to dump the raw buffer as DNG

adtg_gui:
- updated with 6D engio stubs
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on February 01, 2014, 07:33:06 PM
Trying to use the adtg_gui module on 6d.

But I got some linking errors with GDB.
I read in the first post the following:
"adtg_gui.mo (source: adtg_gui.c avl.c avl.h) - cross-platform, requires CONFIG_GDB=y:"

So it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 01, 2014, 07:36:49 PM
It's a safety check (this is how you tell it "hey, I know what I'm doing, now let me fry the sensor")

https://bitbucket.org/hudson/magic-lantern/src/tip/modules/adtg_gui/README.rst
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on February 01, 2014, 07:48:14 PM
Reading some stuff in the link you send, but I still don't get it.
Makefile.user ?
Does it mean I have to compile stuff myself ? (which I never did and is probably the reason why I don't understand how to set config_GDB = Y)



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on February 01, 2014, 07:50:14 PM
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on February 02, 2014, 01:23:41 AM
QuoteSo it requires CONFIG_GDB=y, but what does that mean, what is that config_GDB ? Is it something I have to download or is it a setting in the ML menu ?

You need an autoexec bin/symbols compiled with GDB=Y
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 02, 2014, 02:35:05 AM
Quote from: Levas on February 01, 2014, 07:50:14 PM
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D

Unless you really have a good understanding on the effects and can help development, it's probably best just to wait until it has been polished and released.

Start learning how to compile (http://www.magiclantern.fm/forum/index.php?topic=9517.0).  This way you will be up to speed on the basics with later development.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on February 02, 2014, 01:52:19 PM
The link to the "how to compile" stuff is handy, sure will read into it (Lot's to read and learn, probably will take a month or more before I can compile  :D).
But if I can, I will be one of the first people to use the next impressive magic lantern future!

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 02, 2014, 03:24:38 PM
Quote from: Levas on February 01, 2014, 07:50:14 PM
If I do need to compile, it's better for me to wait until something, considered safe enough, is available for download  ;D

Changing the code & compiling isn't the only way to help, and in all honesty it needs quite a lot of time to get started, it's only easy in hindsight - but I can remember that setting up the whole thing and getting the bearings on the core code isn't self-explanatory.

At least imho as valuable are suggestions / requests on how to improve usability (menus, options, button layout, ...), helping others with ML questions here on the forum & bug/enhancement reports because even with stable code, a couple of people cannot test it enough to iron out every quirk.

The very same mini_iso.mo module about to be released in this thread needs testing, too - does it work with your postprocessing workflow, is the calibration the same on your camera & lenses...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 02, 2014, 09:44:04 PM
6d TL users see here to help out with mini_iso - I'm trying not to spam the ML thread with TL (and that's what I'm currently still using): http://www.magiclantern.fm/forum/index.php?topic=10292
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 03, 2014, 09:53:46 PM
Got some funky graphs (5D3). These cover all the ISO registers I could find.

https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 03, 2014, 10:45:10 PM
Updated adtg_gui: I've added a routine that dumps all the registers into a log file after taking a picture (this is what I've used to make the funky graphs). I'd like to get these graphs for all cameras, so I need your help:

0) check if adtg_gui.c has engio stubs for your camera (if not, copy them from platform stubs, compile it and post the diff)
1) enable adtg_gui in debug menu
2) enable digic registers
3) take a picture at every single ISO, including intermediates, starting to lowest (50 or 100) until the highest, and back. After each picture, make sure it saves a large number of registers (I expect around 1000 or 2000).
4) send me iso.log from the root of card (on 5D3 this log file had 4 MB, so if you get only 10K or something, double-check the settings and the digic register stubs)

In addition, I'd also like a second data set at some fixed ISO, but with variable aperture.
0) rename or move iso.log from the previous experiment in order to start fresh
1) enable the same stuff as before
2) take pictures at iso 200, with variable aperture from f8 to f1.2 or whatever you have, and back
3) rename this iso.log to av.log and upload it

Optionally, you can do the same with shutter speed.

Troubleshooting: if you get overflow errors after loading adtg_gui, add a small quantity to extra_ram in tcc/tccrun.c (I added 1024 on 60D). If you get ERR70, change max_region heuristic to free_space / 4 in mem.c.

On 5D3, adtg_gui now now works on both 5D3 113 and 123. Also updated iso_regs but didn't try it.

edit: please repeat the tests by changing the tested parameter in both directions.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on February 04, 2014, 01:40:28 AM
WOW that module is BIG

Here is the first iso.log.. I went through all the ISOs in the menu, I think its the only module I can load. Even with bvram mirror in shoot mem.

6D:

http://ge.tt/6e5CtgI1/v/0?c

I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 04:34:24 AM
5D3.123

f/1.4.....f/16
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture.log

1/8000s.......30seconds in full stops.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Shutter.log
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 04, 2014, 08:19:57 AM
Moved all the graphs here: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 08:28:41 AM
http://optipng.sourceforge.net/


This is supposed to be better, but single threaded and slower:  https://en.wikipedia.org/wiki/PNGOUT
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 04, 2014, 11:20:20 AM
Since there is a large difference between my logs and Audionut's logs, I'd like to ask you to repeat the tests by changing the tested parameter in both directions (incrementing and decrementing). This will help identifying which of these graphs are just noise and which of them are relevant.

The graphs now show all the data points from the log file, so if there are different values obtained in different experiments, that graphs may be just noise (or may be not).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 12:17:16 PM
Aperture reverse direction
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture_reverse.log

Shutter reverse direction
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Shutter_reverse.log

Shutter 30s....1/8000s in full stops with lens untwisted

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/30-8000_lens_disconnect.log

Aperture in half stops

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Aperture_0.5_stop.log
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 04, 2014, 12:44:38 PM
Updated the graphs and dropped some irrelevant ones. Interesting that after merging the logs I've got much fewer results (will double-check to make sure it's not a bug).

Also moved all the graphs here: https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 01:09:55 PM
ISO up and down.

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/iso.log
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 06:45:30 PM
CMOS[1] 0x663   ISO 600
CMOS[1] 0xee3   ISO 1200

I couldn't find any other intermediate ISO steps.  Noise level changes for the better compared to next higher Canon ISOs (800/1600), but needs more consistent workflow for accurate results.

On my test samples, I changed ISO in Canon menu for Canon ISOs, so these have been tainted by other CMOS/ADTG/Digic register changes.  I'll attempt a more thorough workflow tomorrow for DR results.

I tested these CMOS[1] values.

3
113
223
333
443
553
663
773
883
993
aa3
bb3
cc3
dd3
ee3
ff3
1003
1113
1223
1333
1443
1553
1663
1773
1883
1993
1aa3
1bb3
1cc3
1dd3
1ee3
1ff3
2003


There is a defined pattern.  Here are the average raw values for a patch on the color checker.

101.8
195.9
392.1
777.6
1555.8
2984.6
567.4
5598
194.3
377.8
750.5
1501.4
3006.7
5814.9
1095.1
11196.7
100.6
194.8
387.2
772.5
1539.4
2969.5
561.4
5684.2
192.4
373.9
745.4
1489.3
2992.3
5792.2
1087.8
11130.1
99.6



Notice how the equivalent ISO steps have slightly lower raw values.

For instance, ISO 100 gives raw value 101.8 and other register values produce 100.6 and 99.6.
I can't test noise differences since I messed the workflow up.  I'll have another attempt tomorrow.

If you want to test yourself, the money registers for equivalent ISO 100

0x3
1003
2003
3003
4003....etc, etc.

It's the same pattern for other ISOs.

663
1663
2663......etc, etc.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 04, 2014, 07:44:24 PM
I believe 0x773 has a little less noise than 0xDD3 (need to double-check). At first sight, this combined with CMOS[3] = 0x944 seems to give a 0.25 stop improvement in shadows at ISO 4370 (derived from 6400).

BTW, all research tools are now working on both 5D3 113 and 123.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 04, 2014, 08:09:50 PM
0x223 - ISO 400
(https://s15.postimg.cc/f45if33rf/0x223.png)

0x663 - ISO 600
(https://s15.postimg.cc/upmtz1v57/0x663.png)

0x333 - ISO 800
(https://s15.postimg.cc/sl2gxz18b/0x333.png)

0xee3 - ISO 1200
(https://s15.postimg.cc/gx8f2w1nf/0xee3.png)

0x443 - ISO 1600
(https://s15.postimg.cc/lulzoiymz/0x443.png)


Some other interests.

ISO 200 - 0x113 vs 0x883
(https://s15.postimg.cc/eemq2pvi3/0x113.png) (https://s15.postimg.cc/kfkezu04r/0x883.png)

ISO 400 - 0x223 vs 0x993
(https://s15.postimg.cc/f45if33rf/0x223.png) (https://s15.postimg.cc/mww673rqz/0x993.png)

This trend continues for ISO 800 and ISO 1600. 

Only changing CMOS[1] with a starting point of Canon ISO 100, so not sure why the WL changes between 15283 and 15331.
I'd like to find ISOs 150 and 300.

I've made a dump here (https://www.dropbox.com/sh/u8bgqownig8s3og/D1A5E8P5vy).  Up to 0xff3 for the moment.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on February 04, 2014, 10:29:20 PM
For the 50D, here's the register dump for the ISO sweep, from 100 to 12800 and back down:

https://www.dropbox.com/s/7ftxovt2oqu2njn/50D-ISO.log

I'm not going to do the aperture sweep on my lenses because they're not particularly fast, but I can borrow a lens next week if someone else hasn't provided that register dump before then.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 05, 2014, 07:56:07 AM
Uploaded 50D graphs.

Also modified iso_regs to handle all the CMOS gains.

Also updated the first post with links to these graphs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 05, 2014, 08:57:34 AM
Hypothesis about CMOS gains:
- each ISO is described by a number from 0 to 15. That's 4 bits, from b3 (MSB) to b0 (LSB).
- these 4 bits might be toggling 4 amplifiers on and off
- if this is true, the individual gains of these amplifiers would be approx. b0=1.92, b1=3.85, b2=15.28 and b3=1.91 (based on Audionut's estimation).
- all gains except 6, 7, 14 and 15 follow this logic
- gains 6 and 14 seem to be attenuated by 10.5 (the attenuator is triggered by logical function b2 & b1 & !b0 )
- gains 7 and 15 seem to be attenuated by 2 (the attenuator is triggered by logical function b2 & b1 & b0 )


CMOS gain   Measured  Derived
       0     1.00     1.00     
       1     1.92     1.92     
       2     3.85     3.85     
       3     7.64     7.41     
       4    15.28    15.28     
       5    29.32    29.41     
       6     5.57    58.86    ?
       7    54.99   113.28    ?
       8     1.91     1.91     
       9     3.71     3.67     
      10     7.37     7.35     
      11    14.75    14.15     
      12    29.54    29.17     
      13    57.12    56.13     
      14    10.76   112.35    ?
      15   109.99   216.21    ?



from pylab import *

gains = array([101.8,195.9,392.1,777.6,1555.8,2984.6,567.4,5598,194.3,377.8,750.5,1501.4,3006.7,5814.9,1095.1,11196.7]);
gains = gains / gains[0];

print "CMOS gain   Measured  Derived"
for g in range(16):
    measured_gain = gains[g]
    derived_gain = 1
    for b in range(8):
        if g & (1 << b):
            derived_gain *= gains[1 << b];
    status = " " if abs((derived_gain - measured_gain) / measured_gain) < 0.1 else "?"
    print "%8d %8.2f %8.2f    %s" % (g, measured_gain, derived_gain, status)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 05, 2014, 09:53:18 PM
5D3: CMOS[6] = 0xa00 results in an extremely low gain (very dark image). It gets a little better if you increase the CMOS gain. It has more highlight detail than ISO 100 with ADTG gain at -2 EV, and I've estimated the clipping point at around ISO 67 (by comparing with another image underexposed with shutter).

It's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 05, 2014, 10:03:56 PM
Quote from: a1ex on February 05, 2014, 09:53:18 PMIt's too noisy for real use, but at least it shows that our CMOS amplifier at ISO 100 is running too hot for my taste.

Your last discoveries sound great, I'm confident you'll squeeze every last bit of dynamic range out of Canon ... and as usual I hope all cameras will benefit from it. I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 05, 2014, 10:55:05 PM
I forgot to mention that the raw values I listed above are black subtracted.  Not sure how that effects your gain calculations.

I'll take a look at CMOS[6] and [7] today.

Quote from: Marsu42 on February 05, 2014, 10:03:56 PM
I didn't stop wondering why Canon didn't harvest these possibilities, probably they rather concentrate on (non-raw) video and jpeg?

At the very least, they could have had an almost real ISO 50.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 06, 2014, 02:43:18 AM
CMOS[6] with saturated pixel wells

0x100 is the last register value to give White Level 15283 (0x1ff actually).
0x200-300 - WL 3100
0x400-500-600-700 - WL 2570
0x800 - WL 2081
0x900 - BL 15283 WL 15331 - Full white screen
0xa00-b00 - WL 2140
0xc00 and above WL 2100

This is a repeating pattern like CMOS[0].  At 0x1100 the pattern repeats.

While typing this, I thought I had better dig down further.

0x100  WL 15283
0x101  WL 8680
0x102  WL 12370
0x103-104  WL 15283
0x105  WL 10250
0x106  WL 12440
0x107  WL 12570
0x108-109-10a-10b  WL 15284
0x10c  WL 14610
0x10d-10e-10f  WL 15283

From here, something is happening in the blacks.  The histogram reminds me of high ISO.
0x110  WL 15283 - DR reduces 1 EV (vs no register change)
0x111  WL 8640
0x112  WL 12290
0x113-114  WL 15283 - DR reduced 1EV
0x115  WL 10200 - Histogram looks normal
0x116  WL 12410
0x117  WL 12540
0x118-119-11a-11b  WL 15283 - DR reduced 1EV
0x11c  WL 14520  BL 2046
0x11d-11e  WL 15282  BL 2044
0x11f  WL 15282  BL 2045

Now we are back to having normal looking histograms with a little more noise (deviation)
0x120  WL 15283 - DR reduced 0.2 EV
0x121  WL 8650
0x122  WL 12360
0x123-124  WL 15283 - DR reduced 0.2 EV
0x125  WL 10220
0x126  WL 12420
0x127  WL 12550
0x128-129-12a-12b  WL 15283 - DR reduced 0.2 EV
0x12c  WL 14540
0x12d-12e-12f  WL 15283  BL 2047

First thing to notice, repeating patterns.

Same thing happens at 0x13? except here there is extreme deviation and the black level also changes wildly.  I'm not testing each register as I have some respect for my shutter.

I'll test each 0x140 as the first few appear to mimick 0x10? (useful) with different WL's. 
0x140  WL 15283
0x141  WL 10480
0x142  WL 13800
0x143 -114  WL 15283
0x145  WL 9912
0x146  WL 11140
0x147  WL 9420
0x148-149-14a-14b  WL 15283
0x14c  WL 12530
0x14d  WL 14240
0x14e-14f  WL 15283

0x15? - 0x16? - 0x17? - 0x18? - 0x19? - 0x1a? - 0x1b? - 0x1c? - 0x1d? - 0x1f?  All appear to have the same pattern as 0x10?

A quick look at 0x2?? shows no sign of any difference.  ie all the 0x2?? values match 0x200

There's some funky stuff happening at 0x9??  Have a look at 0x909 for instance.

All of these are like ADTG gain.  Make sure you saturate the pixel wells!

edit1:  Noise level doesn't reduce with reducing WL in the batches where I haven't specifically mentioned changes.

edit2:  All of these tests where conducted at CMOS[0] 0x3.  At CMOS[4] 0x2?? and above where the WL drops drastically, you can bring everything back with higher CMOS[0] values (which a1ex mentioned above).  So for instance CMOS[0] 0x443 and CMOS[4] 0x200 brings back a normal image with WL 15283.  Initial observation suggests that increasing CMOS[0] simply negates the effect of CMOS[4].  ie:  No difference in produced image.  Needs further testing.

edit3:  CMOS[0] 0x10 through 0x18 produce WL 2060.  0x19 through x01f produce dual_iso pattern.  Same thing happens at 0x20 through 0x2f and again at 0x30 through 0x3f, so I'll assume this continues through the other 0x?? values.

What interesting though, the ISO difference reported by cr2hdr seems to match the register normal ISOs.  For instance, 0x1f = 0.96 EV ISO difference, 0x2f = 1.97 EV ISO difference etc.  Including those half stop ISOs.

a1ex, can you create an option to disable iso.log creation?  Also with the latest raw_diag the information overlay disappears after a short time.  This appears to be happening when the iso.log info gets cleared.

edit4:  The same pattern is repeating at CMOS[6] 0x? and 0x??

Have a look at CMOS[7] 0xc0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 06, 2014, 06:15:53 AM
Quote from: 1% on February 04, 2014, 01:40:28 AM
I have a chipped ring that lets you change the aperture from 1.2 to whatever... would that be better than a real electronic lens?

See if the digital gain register is changing with aperture.  Can you set the aperture value lower then 1.2?  That could be interesting.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 06, 2014, 07:34:43 AM
Will do; until then, just comment out the prop handler and recompile.

Didn't digest all of the stuff yet, just some small remarks:
- you mean CMOS[0] instead of CMOS[1]
- the CMOS register values are 12-bit, so the fourth hex digit does nothing
- ADTG values are 16-bit and ENGIO (DIGIC) registers are 32-bit
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 06, 2014, 08:25:36 AM
Will do, thanks.  edit:  I might even be able to add the menu entry myself.

Yes I did.
The highlighted one  0x100?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on February 12, 2014, 06:04:48 PM
I finally was able to borrow that faster lens.  For the 50D, here's the register dump for the aperture sweep, from f/8 to f/1.4 and back:

https://www.dropbox.com/s/ihry3dq9drv2a05/50D-aperture.log

I can immediately see that the digital gain (0xC0F08030) is 0x1000 down to f/2.8, and then starts increasing, up to 0x147A at f/1.4.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 12, 2014, 10:48:02 PM
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/50d-digital-gain-aperture.png)

So... 0.35 stops of digital gain? It matches the DxO value quite well.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 15, 2014, 10:19:40 AM
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX

(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 10:25:01 AM
Quote from: a1ex on February 15, 2014, 10:19:40 AM
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

Also, while digging in history, I've found a log from 5D2 that shows all the debug messages from capturing and processing a LiveView frame: http://pastebin.com/J21HuvLX

(you may now recognize the registers mentioned in this thread, and that's one small step towards understanding this process)

I can do 550D, I am just unsure as to what "check if adtg_gui.c has engio stubs for your camera" means.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 15, 2014, 10:27:29 AM
Open the source code for the modified adtg_gui (not the repo one, but the one from this thread) and look for "engio", you'll find them right away (some cameras have them, others don't).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 15, 2014, 10:35:31 AM
The ones that don't, they just need to be found?

The ones missing the stub,

500D, 550D, EOSM, 600D, 650D, 700D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 10:36:20 AM
Quote from: a1ex on February 15, 2014, 10:27:29 AM
Open the source code for the modified adtg_gui (not the repo one, but the one from this thread) and look for "engio", you'll find them right away (some cameras have them, others don't).
Ah!

And maybe it's drop box or my internet, but I go to the link for "adtg_gui.c" and the page just hangs loading. So I might have to wait til tomorrow.

(Not an excuse, that actually happens. So I will do it tomorrow, as I would like to see this tstop issue dead.)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 15, 2014, 10:37:22 AM
Try right click, save as.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 10:37:58 AM
Quote from: Audionut on February 15, 2014, 10:35:31 AM
The ones that don't, they just need to be found?

The ones missing the stub,

500D, 550D, EOSM, 600D, 650D, 700D

Well I'm on 550D. But it says:
check if adtg_gui.c has engio stubs for your camera (if not, copy them from platform stubs, compile it and post the diff)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 10:39:58 AM
Quote from: Audionut on February 15, 2014, 10:37:22 AM
Try right click, save as.

Oh that worked.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 15, 2014, 10:41:05 AM
Here is the stub for 550D.

NSTUB(0xff1c15cc, _engio_write)

I'll sort the other cameras out and submit a patch to a1ex.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 10:44:41 AM
Quote from: Audionut on February 15, 2014, 10:41:05 AM
Here is the stub for 550D.

NSTUB(0xff1c15cc, _engio_write)

I concur.

Thanks for looking it up though.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 15, 2014, 10:45:05 AM
Just do like ayshih did here:

Quote from: ayshih on January 23, 2014, 03:07:32 PM
For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine.  I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).

    else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
    {
        ADTG_WRITE_FUNC = 0xFFA11FDC;
        CMOS_WRITE_FUNC = 0xFFA12190;
        ENGIO_WRITE_FUNC = 0xFF97D904;  // from stubs
        ENG_DRV_OUT_FUNC = 0xff97d794;
    }

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 15, 2014, 10:54:09 AM
Added missing stubs:

https://bitbucket.org/Audionut/zebras/commits/8779a5fa91ed4750268087d104bbf80b646a5a5c


Where do you find the CMOS/ADTG registers?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 11:15:01 AM
I have compiled ML before with no problems.
And I have just gone to compile it and I get this error.
mem.o: In function `GetMaxRegionForAllocateMemory':
mem.c:(.text+0x168): undefined reference to `GetSizeOfMaxRegion'
collect2: error: ld returned 1 exit status
make: *** [magiclantern] Error 1



But also!!!
what is: requires CONFIG_GDB=y
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 15, 2014, 11:20:37 AM
Need to find the stub for GetSizeOfMaxRegion and add it to stubs.S
I have no idea how to do that.


Enable CONFIG_GDB in the makefile (https://bitbucket.org/hudson/magic-lantern/src/a7b8f4853167bc8763e22fc14f5cab966a58bd9c/Makefile.user.default?at=unified).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 15, 2014, 11:23:56 AM
Oh and, compiling the module, (as I wasn't actually doing that), spits out this:

/hudson-magic-lantern-1d65c42c0448/modules/adtg_gui$ makeabort: no repository found in '/hudson-magic-lantern-1d65c42c0448/modules/adtg_gui' (.hg not found)!
REBUILDING
[ README   ]   module_strings.h
Traceback (most recent call last):
  File "../readme2modulestrings.py", line 111, in <module>
    last_change_date, last_changeset, author, commit_msg = last_change_info.split("\n")
ValueError: need more than 2 values to unpack
make: *** [module_strings.h] Error 1


PS. I think I give up
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 15, 2014, 11:40:30 AM
Can you compile other modules?

The module build system extracts some info from the mercurial repository.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: LebedevRI on February 15, 2014, 11:52:38 AM
5D3.113

ISO: https://www.dropbox.com/s/ltxk0117up6eb9h/iso.log
Aperture(f/2.8 ... f/8): https://www.dropbox.com/s/c59d874jibzw7bw/av.log
(Is shutter log needed?)

Will do same for 550D when stub for GetSizeOfMaxRegion is added.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 15, 2014, 12:04:16 PM
I already have the 5D3 graphs, but being able to cross-check the data never hurts.

Added 550D GetSizeOfMaxRegion from mk11174.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: LebedevRI on February 15, 2014, 01:25:21 PM
Quote from: a1ex on February 15, 2014, 12:04:16 PM
Added 550D GetSizeOfMaxRegion from mk11174.

550D.109

ISO: https://www.dropbox.com/s/sifq319w4533bpu/ISO.LOG
Aperture(f/2.8 ... f/32): https://www.dropbox.com/s/p5f508ds249l3cq/Aperture.LOG
Shutter(30" ... 1/4000): https://www.dropbox.com/s/5dtntvhsahv7wqr/Shutter.LOG
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 17, 2014, 07:07:46 PM
Dear Mr. Developer Sir, any chance of getting "re-cr2.exe" to change the [edit: white point] in a cr2 (from whatever it was) to a standard acr, or dcraw?

OSS raw converters might be modified to work with mini_iso unorthodox [edit: white point] values, but acr surely won't so not having to decide for one way or another while shooting but being able to convert would lift a huge burden from simply photogs' minds :-o

Next to that, forcing any Canon-changed [edit: white point] to the fixed acr value might be nice anyway, because since I know about this I better understand how acr deals with highlight recovery (or, more to the point, doesn't).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 17, 2014, 07:10:23 PM
What are you talking about?

None of these tweaks can be used to change WB.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 17, 2014, 07:21:41 PM
Quote from: a1ex on February 17, 2014, 07:10:23 PM
What are you talking about? None of these tweaks can be used to change WB.

Sorry, just came back from a day crawling around photographing wildlife :-p ... I'm of course talking of the white level, the thingy you can set to various values by newer versions of your mini_iso module.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 17, 2014, 07:26:03 PM
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.

I believe Canon autodetects it somewhat like raw_diag does.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 17, 2014, 07:29:24 PM
Quote from: a1ex on February 17, 2014, 07:26:03 PM
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.

Well, I was thinking of converting the *data* (that's what you do with mini_iso?) so the cr2 can be used with the new white point.  Is tagging the dng with the acr white point the same thing, what's the exact tag? Thanks for any info.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 17, 2014, 07:31:07 PM
exiftool foobar.dng -BlackLevel=64 -WhiteLevel=15000
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 04:35:32 AM
Quote from: a1ex on February 17, 2014, 07:26:03 PM
Convert them to DNG and use exiftool if needed. There are no white level tags on the CR2.

I believe Canon autodetects it somewhat like raw_diag does.

With exitool on CR2s you can try
-Canon:PerChannelBlackLevel  (needs four values)
-Canon:NormalWhiteLevel
-Canon:SpecularWhiteLevel
-Canon:LinearityUpperMargin
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 06:56:07 AM
Thanks, but...

exiftool foobar.CR2 -Canon:NormalWhiteLevel=1234
Warning: Tag 'NormalWhiteLevel' does not exist
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 12:18:56 PM
Does not exist ? for which model do you take this ?. Any sample available ?.

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#ColorData7
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 12:27:04 PM
5D2 and 5D3. There are many samples on the forum (e.g. in the dual ISO thread).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 03:01:00 PM
Works for me on win vista32 with exiftool 9.51

(http://filebin.net/xpt6c2c5wh/Canon50Dexif.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 03:13:43 PM
Thanks, I had 9.39, downloaded a few months ago.

Is there any software that uses this tag, given that exiftool didn't know about it until recently?

(ufraw doesn't, so I assume dcraw doesn't either, and Adobe DNG converter simply sets the white level to 15000 in the DNGs)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 19, 2014, 03:42:09 PM
ACR/LR ignores it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: tjaja on February 19, 2014, 04:21:12 PM
Digital Photo Professional is showing a difrence in the highlights when I put values between 0 and 900 in exiftool.

-edit: it changes the color of the highlight somehow

Hmm the lower values desaturate indeed. No headroom to change colors afterwards.

(http://www.alisultan.nl/magicbug/highlightcolor.JPG)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 04:41:53 PM
Looks like it's changing the desaturate threshold. Cool find.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 19, 2014, 04:44:13 PM
DPP confirmed as respecting the tags.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 04:45:26 PM
Yes, DPP should use this info but I don't have it to test.

The Normal white level was 14000 before I changed it to 4321 .. so the test with DPP should be between 14000 and a changed one ..

I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it. Although it does not always match the white clipping level as we can see it in raw histograms.

Although I think it's interesting that there are three tags about WL which DPP probably uses.
We see the linearity upper margin being 1/2 stop under normal WL . does this mean that the raw highlight data are non linear ? or were  they non linear at the start (ADC output) and then linearized before storage in *.CR2 ?.

WL values seem to be steady after gain manipulations, so I don't know  if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. :).

BL values look to be measured in the optically black zones so they are reliable and could help for a faster BL identification ..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 19, 2014, 04:52:14 PM
It's not being desaturated per se, the white levels are being crushed.

You can create the same effect with luminance tone curves.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: tjaja on February 19, 2014, 04:57:19 PM
@ IliasG what is that program showing all those metadata entries?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 05:05:08 PM
Quote from: IliasG on February 19, 2014, 04:45:26 PM
I don't know the exact meaning of these tags. Adobe DNG/Dcraw and derivatives only care about SpecularWhiteLevel as I understand it.

Ufraw doesn't care about spec(ta)cular white level either.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 19, 2014, 05:08:28 PM
Can you update your ufraw-mod to respect the tags?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 05:11:29 PM
I think so (and dcraw too).

Now, is there a connection between these exif levels and the register named "white level?" from these graphs? https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW

(I tried to override it and did not notice any visible effect, but didn't check exif)

Quote from: IliasG on February 19, 2014, 04:45:26 PM
WL values seem to be steady after gain manipulations, so I don't know  if they are reliable as of now. But I hope ML can push Digic to write the correct ones if needed .. :).

I can re-scale the data analogically so I match the actual white level with any value you want. Just tell me what value I should use.

So far I've included 15000 (Adobe), 16380 (full range), the dcraw value (camera-specific) and Canon's default setting with manual lens (that is, simply canceling the aperture gain).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 19, 2014, 06:01:15 PM
I just cycled through all of the WL options in mini_iso.  All exposures resulted in these values regardless of saturation.

NominalWhiteLevel: 16228
SpecularWhiteLevel: 16383
LinearityUpeerMargin: 11129

The RawMeasuredRGGB tag is weird.  With mini_iso on a batch of exposures, saturation in the center of the image cause tag values 0,0,0,0.  Without saturation, this tag contained large values.

Going back through a bunch of shots without mini_iso, this tag looks very random.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 08:50:44 PM
Quote from: tjaja on February 19, 2014, 04:57:19 PM
@ IliasG what is that program showing all those metadata entries?

exiftool with exiftoolGUI as frontend
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 09:05:01 PM
Quote from: a1ex on February 19, 2014, 05:05:08 PM
Ufraw doesn't care about spec(ta)cular white level either.

I mean that Dcraw and derivatives only care for the topmost value of data. This is hardcoded in Dcraw and Ufraw (single value for all cases .. you know how good this is ..) or a lookup table.
Rawtherapee uses an external file named camconst.json (camera constants) where among others one can put per ISO WL and a multiplier for each Fnumber ..
DNG looks like also using a table for each ISO but has nothing to fine tune WL depending on apperture ..

In fact I came again in ML forums searching for data regarding the per ISO and per aperture WhiteLevels .. so if it's possible to use the already collected data I ask your permition to use them for RT .. and a bit help how to mine them from the log files .. or even better if the are readily available as numbers ..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 19, 2014, 09:11:46 PM
Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?

In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).

If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on February 19, 2014, 10:07:22 PM
Quote from: a1ex on February 19, 2014, 09:11:46 PM
Are you referring to the register values tagged as WhiteLevel in my dropbox graphs, or to values detected by raw_diag?

In any case, feel free to use them for RawTherapee. I took the AMaZE algorithm from them, and I'm thinking to send them a patch for floating-point DNG support (my version doesn't open CeroNoice DNGs, but didn't check the bleeding edge).

If you collect a table with white level values in a nice format, I'd be interested too (just to have them organized).

I think raw_diag white levels is what I need .. basically the values that are same as if I would inspect  the raw histogram for white clipping. This for normal Canon cr2s not ML modified .. and I think ML should also target to the same WLevels as the native Canon cr2s.

I would strongly prefer to use the exif data (specularWL) so the best would be a DCraw patched to use the tags, but I have to recheck as I remember a not insignificant deviation from the histogram derived WL in some rare cases ..

floatDNG is not supported ... waiting for your patch .. :)

The WL info will be freely available in RT code just search for the "camconst.json" file. I don't know if you'll like the format but as I will use spreadsheet to get the data ready I can sent it when it's ready ..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 21, 2014, 10:29:36 AM
Hi everybody,

I am following this thread from the beginning, being excited about the possible improvements of your amazing find.
Is my following short conclusion correct?

The DR improvement on 5D MKIII is confirmed and the registers are identified.
Nevertheless there is no tool that could automatically tweak all those registers for a specific camera (without the possibility of damage). This tweaking may be required because every sensor is different.

Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed. But ACR seems to work.

In case there already is a module for 5DMKIII that can be used for testing and does not involve setting all those registers manually, please tell me where to find it :-) I would love to help testing.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 21, 2014, 10:59:40 AM
This tool is not yet published.

QuoteFurthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed

Quite the opposite. It's exactly these raw converters that will benefit (because the CR2 can be adjusted to match what these converters expect). I've included presets for dcraw and Adobe.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 21, 2014, 11:09:49 AM
Great!
Thanks for clarification :-)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on February 21, 2014, 04:18:43 PM
Quote from: hjfilmspeed on January 22, 2014, 12:05:45 AM
Are there more tests needed? Anything I can help with?
5d3 running the nighters

Are there useful tests that would be helpful from a 6d owned by a risk-adverse non-dev?

I notice there is a module that seems related to ADTG, but I am ignorant about what it does. Much of this thread is over my head.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 11:15:58 AM
5D3, ADTG2 and ADTG4[0x8, 9, A, B]:

- at first sight, these seem to be some black biases: in LiveView, they will cause vertical lines in shadows, but there is a filter that will adapt to the changes fairly quickly (seems to be a moving average).

- at a closer look, changing this register also affects the gain, and we can lower the ISO a little bit more (noise gets lower and there is some more highlight detail captured). Initial guess: 0.15 ... 0.25 stops (maybe more if there is nonlinearity, I don't know yet).

- I believe these are the registers Greg was playing with on 500D.

Will try to update the research tools to the latest codebase today.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 22, 2014, 11:53:15 AM
(https://s15.postimg.cc/mk4s0m48r/0x8.png)

Previously 11.44

A separate button to enter top of the ADTG menu would be handy.  This way when you are in the middle of a 600 (or more) item menu, you can get back to the top of the menu quickly.  Or listing the actual registers in a separate menu with the settings contained in a top menu?

Digic registers do not turn off. 
edit:  To be more precise, they do not get removed from the menu when turning the digic option off.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 22, 2014, 12:53:10 PM
Is this only for Raw? Or will it be able to be applied to in camera jpegs and h264s?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 01:03:52 PM
We are doing the analysis only for RAW.

If you are interested in JPEG, you may be lucky, since the JPEG is created from RAW. But you'll have to do your own analysis ;)

If you can find a digital gain that affects JPEG but does not get burned in the CR2 file, it may be interesting.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 01:46:43 PM
Updated research tools to the latest codebase (unified) and added them to a branch on the main repository.

https://bitbucket.org/hudson/magic-lantern/pull-request/412/iso-research-tools
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 22, 2014, 01:49:28 PM
Looks like the column gain registers need to be adjusted for each ISO when tweaking ADTG2 and ADTG4[0x8, 9, A, B].

Here is with the column gain registers at 0x32?

(https://s15.postimg.cc/5ibxyss17/100.png) (https://s15.postimg.cc/lgknoxtyz/200.png)
(https://s15.postimg.cc/8p6hifrwr/400.png) (https://s15.postimg.cc/imhibi78b/800.png)

Very close to 1 full stop improvement over Canon gain at ISOs 200, 400, 800  :D


And here with the column gain registers adjusted to bring the WL back to sweet spot.

(https://s15.postimg.cc/6xdinjnzf/1001.png)


Still have lots of observing to conduct, but so far I cannot raise ISO 100 any further.

Thanks for the menu tweaks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 22, 2014, 02:01:17 PM
Quote from: a1ex on February 22, 2014, 01:03:52 PM
We are doing the analysis only for RAW.

If you are interested in JPEG, you may be lucky, since the JPEG is created from RAW. But you'll have to do your own analysis ;)

If you can find a digital gain that affects JPEG but does not get burned in the CR2 file, it may be interesting.

Ah.

I didn't mean a separate thing, I mean the same thing that gets applied to Raw then renders the jpeg and video in camera, to get rid of the tstop at f/1.8.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 22, 2014, 03:14:44 PM
Hi again,

I am sorry if I am again posting something stupid, but Lenny reminded me of some fact I did read about the 5D III sensor.
There was a company that used an electron microscope to examine the sensor of Canons 5D MK III.
They revealed transistors that could be used for electronic pixel binning of three adjacent image pixels. The furthermore assumed this could be used in video mode to reduce aliasing. Unfortunately I could not find a proof of this speculation. Even worse is the fact that this paper was only accessible for some hours or days. I can not find it any more.

Why do I post it here?
Wouldn't binning of three pixels improve the resulting full well capacity by a factor of three? That COULD significantly improve DR in video mode.
On the other hand it would also increase the shot noise compared to one single channel. So if three binned pixels are read with the same settings (Gain & ADC) as one single this would result in WORSE DR compared to photo mode. (Which is what you seemed to see in the beginning?) If one could now somehow adjust the settings to respect the greater full well capacity it could finally increase the video DR, wouldn't it?

Again: I assume that somebody has already investigated / tested this and did not publish his results because it did not work or the information about those transistors is wrong. But since there is a tiny chance that this has not yet been tested I thought I should give it a try.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 03:20:17 PM
We didn't get this far (we focused mostly on photo mode), but the question does make sense.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: stevefal on February 22, 2014, 04:22:45 PM
Is the FPN filter mentioned earlier something that would run in-camera, or post? If in-camera, is it possible that it could be used in video mode, even if the ISO tweaks couldn't?

Making that deep shadow noise prettier would be big plus for video, especially given that the bands crawl, in my experience.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 04:35:24 PM
It's Canon's auto-calibration algorithm (you are already using it).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: stevefal on February 22, 2014, 05:12:13 PM
Maybe I've described it wrong. I'm talking about these mentions:

Quote from: a1ex on January 17, 2014, 03:13:51 AM
On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.

I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.

Quote from: a1ex on January 17, 2014, 04:41:23 AM
The banding pattern accounts for only a small part of these histogram distortions.

Proof:

ISO 100 dark frame (5D3), original and corrected
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso100.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso100-nobanding.png)

Histograms of horizontal and vertical banding correction (per-line and per-column)
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/hband.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/vband.png)

Image histogram before and after correction:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/hist0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/hist1.png)

How I've estimated the correction:

# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im);  fv = fv - mean(fv);

# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png

# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;

# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png


I took that as an idea for new software to de-pattern shadow noise.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 05:15:33 PM
Ah, you should have linked it from the beginning. I thought it's about today's findings.

It can run in camera, but expect around 10 seconds (maybe more) for one picture.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 22, 2014, 05:24:52 PM
An old personal goal of mine was 11.5  :)

(https://s15.postimg.cc/cljte6ctn/ISO200.png)


http://lifeinmegapixels.com/blog/2012/01/pixel-binning-does-it-work/

QuoteSo while you can do pixel binning in post if you want, it doesn't particularly gain much. So is it still worth doing? Yes, when done during capture.

The key issue is overcoming the noise floor, and once the noise is present, no amount of post work with remove it. Read noise is one of the problems, which is the noise generated when the pixels are read from the sensor. If pixel binning is done on sensor, i.e the sets of pixels are combined before being read, then the read noise will be greatly reduced (one portion of noise per final pixel instead of 4/9/etc). This means that additive binning really can provide some benefit in this case by reducing that source of noise. This is performed in some point and shoot cameras where high ISO images come out with reduced resolution, but also on high end cameras such as the Phase One medium format backs with the Sensor+ setting.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: stevefal on February 22, 2014, 05:27:21 PM
Quote from: a1ex on February 22, 2014, 05:15:33 PM
Ah, you should have linked it from the beginning. I thought it's about today's findings.

It can run in camera, but expect around 10 seconds (maybe more) for one picture.

Yeah sorry, I just read the whole thread for the first time today. I thought 'FPN' captured it.

So I assume your idea was for an post tool addition. I'm pitching the case for video. Banding is bad, but it's really bad when the bands crawl - the motion catches your eye, and you really see them.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 22, 2014, 05:32:08 PM
Quote from: stevefal on February 22, 2014, 05:27:21 PM
Banding is bad, but it's really bad when the bands crawl - the motion catches your eye, and you really see them.

Well FPN, is fixed by nature.

"Crawling" pattern noise, will be an entirely different beast to fix, afaik.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 22, 2014, 05:34:33 PM
Quote from: naturalsound on February 22, 2014, 03:14:44 PM
(...)
Wouldn't binning of three pixels improve the resulting full well capacity by a factor of three? That COULD significantly improve DR in video mode.
(...)
This is possible if the binning is done on-ship (accumulated charge from each pixel involved in the binning is brought together and summed before the process of ADC occurs) and if the output register pixels can hold the extend charge.
If not, binning reduces the detectable amount of light per pixel, hence the FWC.

Beside, the ADC quantification have to be calibrate for this special configuration.

But in the worse case binning significantly improves the SNR, witch is very interesting by the way.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 22, 2014, 05:39:52 PM
Quote from: SpcCb on February 22, 2014, 05:34:33 PM
If not, binning reduces the detectable amount of light per pixel, hence the FWC.

White level.  It would have to be a special post processor that reduces the capacity of the pixel wells  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 06:51:40 PM
Got some graphs.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newreg/canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newreg/adtg-2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newreg/new-reg.png)

On X, there is the raw data from ISO 100 1/200, manual lens, with no tweaks applied. Black subtracted and data converted to EV.

On Y, we have:
1) raw data from ISO 100 1/50, no tweaks applied. It's 2EV brighter (obvious).
2) ADTG gain at -2 EV, along with some other tweaks (black forced to 64, white forced to 15000, CMOS[4] patched to 0x318).
3) the tweaks from 2, plus the latest tweak discovered today (all those registers set to 0).

Source: https://bitbucket.org/hudson/magic-lantern/commits/32ef83a5ee41
Updated raw_diag binary on first post.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 22, 2014, 07:38:50 PM
for #3 > sigma looks higher, isn't? But transition before clipping looks smoother.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 22, 2014, 07:41:23 PM
Take it with a grain of salt; the white level detection algorithm doesn't quite agree with the graphs. Time to figure out why...

edit: graph autoscaling was broken, white level was OK, re-uploaded the graphs.

Here's another one, with what I believe to be the sweet spot for both gains. First I've tuned the ADTG gain to sweet spot (which can be done with 2 test pictures, since the response curve is linear), then I've reduced the new gain until the white level started to decrease (but this one is trickier because of the nonlinearity).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newreg/sweet-spot.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 09:11:55 AM
Disregard this one (poor repeatability, probably because of flicker from the light source).


Repeated the experiment with sweet spots only, no black/white level changes (so the graphs should be easier to compare) and no camera movement (the entire test sequence was ran from a script (https://bitbucket.org/hudson/magic-lantern/commits/efc750013a69)).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv2/canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv2/adtg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv2/adtg-sweetspot-cmos4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv2/new-reg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv2/new-reg-sweetspot-cmos4.png)

On X, there is the raw data from ISO 100 1/200, manual lens, with no tweaks applied (vanilla Canon settings).

On Y, we have:
1) raw data from ISO 100 1/50, no tweaks applied. It's 2EV brighter (obvious).
2) ADTG gain at sweet spot (-0.37 EV).
3) ADTG gain at sweet spot (-0.37 EV) and CMOS[4] patched to 0x318.
4) ADTG gain at sweet spot (-0.37 EV), new gain register set at 36 (default was 59, units are not yet known).
5) ADTG gain at sweet spot (-0.37 EV), new gain register set at 36, CMOS[4] patched to 0x318.

Ran the test again to check the repeatability of the results:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv3/canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv3/adtg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv3/adtg-sweetspot-cmos4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv3/new-reg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv3/new-reg-sweetspot-cmos4.png)

ISO differences, computed as median(real(log2(b)-log2(a))) from the second data set:
2.07, 1.71, 1.57, 1.51, 1.54 EV.

=> the modified ISOs (estimated from median brightness, not from clipping point) were, assumming Canon ISO 100 is really ISO 100:
ISO 100, 77.9, 70.7, 67.8 and 69.2.

The ISO 77.9 matches the theory (I've lowered ADTG gain by 0.36 stops, I've measured 0.36 stops of difference between the two pictures).

ISO differences, computed from clipping point: 2.07, 1.68, 1.44, 1.34, 1.49.

=> the modified ISOs (estimated from clipping point) were, assumming Canon ISO 100 is really ISO 100:
ISO 100, 76.3, 64.6, 60.2 and 66.8.

ISO estimation code: https://bitbucket.org/hudson/magic-lantern/commits/0c18a31891b6
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 10:11:27 AM
Quote from: a1ex on February 23, 2014, 09:11:55 AM
new gain register set at 36 (default was 59, units are not yet known).

My defaults.

(https://s15.postimg.cc/4rj7sscqj/register.png)

They don't appear to vary, at least not with the lower ISOs.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 10:25:30 AM
Yes, they don't vary with ISO. I've set all of them to 36 = 0x24 from iso_regs, and the first one (the one printed in iso_regs) was 59 = 0x3b.

Looks like the repeatability is much better at 1/50 vs 1/25 (probably because of artificial light flicker and maybe also mechanical flicker) => will repeat the experiment with these values.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 11:32:07 AM
Did the the experiment again with 1/25 (tested image) vs 1/50 (reference image) because this improves the repeatability. ISO 100 in Canon menu.

Repeatability series 1: Canon 1/25 vs 1/50, ISO tweaking modules not loaded, camera not moved during the entire experiment.

Expo difference (EV)
median  clip
1.01    1.03
1.01    1.01
1.01    1.00
1.01    1.00
1.01    1.01


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-canon-1.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-canon-2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-canon-3.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-canon-4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-canon-5.png)

Checking register settings:

Expo difference (EV)
median  clip
1.00    0.99          # Canon 1/25 vs 1/50
0.63    0.59          # 1/25 with ADTG gain at -0.37 EV vs Canon 1/50
0.64    0.60          # 1/25 with ADTG gain at -0.37 EV and CMOS patched vs Canon 1/50
0.49    0.40          # 1/25 with ADTG gain at -0.37 EV and new gain at 36 vs Canon 1/50
0.50    0.42          # 1/25 with ADTG gain at -0.37 EV, new gain at 36 and CMOS patched vs Canon 1/50


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/adtg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/adtg-sweetspot-cmos4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/new-reg-sweetspot-nocmos.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/new-reg-sweetspot-cmos4.png)

Repeatability series 2: settings from the third tweaked set (1/25, ADTG gain -0.37 EV, new gain 36, no CMOS patch).
Camera moved intentionally between each bracket, but not moved during a bracket.

Expo difference (EV)
median  clip
0.51    0.43
0.49    0.38
0.50    0.40
0.47    0.38
0.49    0.40


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-tweak-1.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-tweak-2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-tweak-3.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-tweak-4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/newregv4/repeat-tweak-5.png)

Clarifications:
- CMOS patched: changed CMOS[4] from 0x718 to 0x318.
- ADTG at -0.37 EV: all ADTG gains were scaled by this value from factory gains, and this value was found to be the sweet spot (lower this gain more => white level starts to decrease).
- new gain at 36: ADTG2/4[8,9,A,B] all set to 36 from iso_regs (one of them was 59 by default). Below this level, highlights are no longer clean (they start to get vertical banding).
- ISO estimation code: https://bitbucket.org/hudson/magic-lantern/commits/0c18a31891b6
- the script for running the experiment https://bitbucket.org/hudson/magic-lantern/commits/5df6093487a6 (note: you need mini_iso, which is not yet published, but I can send you a copy on request via PM).

Conclusions:
- on 5D3 we now have a really nice ISO 66 (-0.6 stops below ISO 100).
- we also have two methods of estimating the relative ISO in raw_diag + octave.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 12:45:49 PM
ISO 200 in Canon menu:

Expo difference (EV)
median  clip
1.00    1.03          # Canon 1/25 vs 1/50, ISO 200
0.58    0.59          # 1/25 with ADTG gain at -0.41 EV vs Canon 1/50
0.60    0.59          # 1/25 with ADTG gain at -0.41 EV and CMOS patched vs Canon 1/50
0.25    0.21          # 1/25 with ADTG gain at -0.41 EV and new gain at 0 vs Canon 1/50
0.24    0.20          # 1/25 with ADTG gain at -0.41 EV, new gain at 0 and CMOS patched vs Canon 1/50


=> the new ISO pulled from 200 was actually 115.

From ISO 400:

Expo difference (EV)
median  clip
1.00    1.04          # Canon 1/25 vs 1/50, ISO 400
0.57    0.58          # 1/25 with ADTG gain at -0.43 EV vs Canon 1/50
0.56    0.57          # 1/25 with ADTG gain at -0.43 EV and CMOS patched vs Canon 1/50
0.22    0.20          # 1/25 with ADTG gain at -0.43 EV and new gain at 0 vs Canon 1/50
0.22    0.21          # 1/25 with ADTG gain at -0.43 EV, new gain at 0 and CMOS patched vs Canon 1/50


=> the new ISO pulled from 400 was actually 230.

From ISO 800:

Expo difference (EV)
median  clip
0.97    1.02          # Canon 1/25 vs 1/50, ISO 800
0.55    0.56          # 1/25 with ADTG gain at -0.43 EV vs Canon 1/50
0.57    0.60          # 1/25 with ADTG gain at -0.43 EV and CMOS patched vs Canon 1/50
0.21    0.20          # 1/25 with ADTG gain at -0.43 EV and new gain at 0 vs Canon 1/50
0.20    0.21          # 1/25 with ADTG gain at -0.43 EV, new gain at 0 and CMOS patched vs Canon 1/50


=> the new ISO pulled from 800 was actually 460.

From ISO 1600:

Expo difference (EV)
median  clip
0.93    1.00          # Canon 1/25 vs 1/50, ISO 1600
0.53    0.58          # 1/25 with ADTG gain at -0.44 EV vs Canon 1/50
0.52    0.56          # 1/25 with ADTG gain at -0.44 EV and CMOS patched vs Canon 1/50
0.18    0.19          # 1/25 with ADTG gain at -0.44 EV and new gain at 0 vs Canon 1/50
0.18    0.19          # 1/25 with ADTG gain at -0.44 EV, new gain at 0 and CMOS patched vs Canon 1/50


=> the new ISO pulled from 1600 was actually 912.

From ISO 3200:

Expo difference (EV)
median  clip
0.97    0.99          # Canon 1/25 vs 1/50, ISO 3200
0.49    0.50          # 1/25 with ADTG gain at -0.49 EV vs Canon 1/50
0.50    0.51          # 1/25 with ADTG gain at -0.49 EV and CMOS patched vs Canon 1/50
0.14    0.14          # 1/25 with ADTG gain at -0.49 EV and new gain at 0 vs Canon 1/50
0.13    0.13          # 1/25 with ADTG gain at -0.49 EV, new gain at 0 and CMOS patched vs Canon 1/50

=> the new ISO pulled from 3200 was actually 1750.

From ISO 6400:

Expo difference (EV)
median  clip
0.95    1.00          # Canon 1/25 vs 1/50, ISO 6400
0.41    0.43          # 1/25 with ADTG gain at -0.55 EV vs Canon 1/50
0.43    0.49          # 1/25 with ADTG gain at -0.55 EV and CMOS patched* vs Canon 1/50
0.06    0.06          # 1/25 with ADTG gain at -0.55 EV and new gain at 0 vs Canon 1/50
0.10    0.11          # 1/25 with ADTG gain at -0.55 EV, new gain at 0 and CMOS patched* vs Canon 1/50

* CMOS[4] patched from 0x718 to 0x318 and CMOS[3] patched from 0x144 to 0x944.

=> the new ISO pulled from 6400 was actually 3333.

From 12800:

Expo difference (EV)
median  clip
0.91    1.05          # Canon 1/25 vs 1/50, ISO 12800
0.35    0.39          # 1/25 with ADTG gain at -0.58 EV vs Canon 1/50
0.36    0.40          # 1/25 with ADTG gain at -0.58 EV and CMOS patched* vs Canon 1/50
0.01    0.03          # 1/25 with ADTG gain at -0.58 EV and new gain at 0 vs Canon 1/50
0.00    0.00          # 1/25 with ADTG gain at -0.58 EV, new gain at 0 and CMOS patched* vs Canon 1/50

* CMOS[4] patched from 0x718 to 0x318 and CMOS[3] patched from 0x144 to 0x944.

=> the new ISO pulled from 12800 was actually 6400.

Graphs for these sets uploaded here: https://www.dropbox.com/sh/onppbwy44fqomxa/VlGTJBoSr4/5d3-iso-experiments

(note: the metadata from the graph itself may be incomplete; also look at the file name to know which is which)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 12:50:32 PM
I'd like to understand them first  :o
Haven't had a chance to play with updated raw_diag.

What do the blue/green/red lines represent?  I find it hard to tell what is what.  I think I know what I am looking at, but could you please explain it a little more.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 12:59:08 PM
Blue is blue channel, red is red channel, green is green channel. They are compared to the values from a reference shot taken with Canon settings (unmodified ISOs) and underexposed by 1 stop via shutter.

So, in the first graph (Canon 1/25 vs Canon 1/50), you see the graph exactly 1 stop above the diagonal. That's because the exposure difference between these two is exactly 1 stop.

In the subsequent graphs, you see the graph getting closer to the diagonal (that's because ISO is getting lower).

If you define ISO as "overall brightness", I estimate it from the median vertical distance between the graph and the diagonal.

If you define ISO as "clipping point" (DxO definition), I estimate the clipping point and then take the horizontal distance between the two clipping points. If that distance is 0.6 stops, it means our ISO is 1-0.6 = 0.4 stops lower. Note the overall brightness might difer a little because of nonlinearity.

When you want to know how much you can expose to the right with the new ISOs, you need to know the clipping point.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: engardeknave on February 23, 2014, 01:00:54 PM
I enjoy just sort of watching this thread utterly without comprehension. Like a cat watching TV.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 01:29:29 PM
Quote from: a1ex on February 23, 2014, 12:59:08 PM
Blue is blue channel, red is red channel, green is green channel.

Yeah, that makes sense!

I assume the 2 banks of column gain registers (ADTG2/ADTG4) are for the dual amps that dual_iso uses, correct?  And ADTG6 looks to be the feedback amp?

I only ask because cr2hdr reports a low DR (about 0.2-0.3 EV to low) for the recovery ISO.  Looks like we are getting 0.6 EV of overlap too :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 01:46:47 PM
Dual ISO only changes the CMOS[0] register. But, of course, it benefits from the other tweaks too.

ADTG6... I don't know yet what it is, did you notice any effect by changing it?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 02:07:43 PM
Looks like ADTG6[8880] 0x750 gives 0.02 EV.  Every little bit counts right?

Otherwise, I haven't found anything useful yet.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 02:10:23 PM
Dynamic range series (ISO 100-12800 in Canon menu, ADTG gain at sweet spot, ADTG2/4[8,9,A,B] at 36 for ISO 100 and 0 for all other ISOs, CMOS patches enabled.


ISO    DR
66     11.589 + 0.1
115    11.772
230    11.649
460    11.427
912    10.968
1750   10.369
3333   9.610
6400   8.834


I've added 0.1 to the measurement at ISO 100 because of the nonlinear highlight rolloff (approximate). The other ISOs seem to clip harshly, the difference between median and clip ISO is smaller, so I think we can consider the response linear.

Comparing with my older measurements:

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-cmos.png)

Note: the ISOs may be approximate. As I said in the dual ISO thread: I'm an engineer, for me, Pi is 3.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on February 23, 2014, 02:52:47 PM
Most of this is way over my head but from the looks of that chart you guys have hit on another incredible ML feature!!!!


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 03:27:30 PM
11.77 EV is remarkable.  I don't imagine it is going to be possible to modify the ADTG registers for dual_iso.

Based on a1ex's ISO data above, and the measured ISOs from DxO (http://www.dxomark.com/Cameras/Canon/EOS-5D-Mark-III---Measurements), the true ISOs should be.

Canon Menu ML
100 53
200 92
400 186
800 369
1600 729
3200 1377
6400 2697
12800 5042

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 04:12:26 PM
With new ADTG gain at 0x0, I haven't noticed any highlight banding.

Consider these histograms.

Canon
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/Canon.png

ML with ADTG gain at -0.37 EV, new gain at 36 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML36.png

ML with ADTG gain at -0.37 EV, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.png

We still gained highlight detail with new gain at 0.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 04:18:06 PM
So, we have ISO even lower than 66?

Will take a closer look.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 04:31:44 PM
It looks like it from here.  I trust your closer look more :)

If I am reading the histogram right, here is ISO 200.  Here we lose some highlight detail compared to ISO 100.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO200.png

Then we start to gain some more highlight detail through every higher ISO.  Here at ISO 6400.
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0ISO6400.png


Both shots, ML with calibrated ADTG gain, new gain at 0 and CMOS patched

Look at ISO 100 with calibrated ADTG gain, new gain at 36 and CMOS patched
vs
ISO 200 with calibrated ADTG gain, new gain at 0 and CMOS patched

We see a gain in highlight detail that matches ISO 200 through ISO 12800.
ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched seems to be applying a large amount of highlight roll off.  Need to check if there is actual detail right near saturation, or Rawdigger is struggling to accurately report details.

edit:  I'm probably just seeing increased stdev with increasing ISO, not extra highlight detail.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2014, 04:40:43 PM
Here's the banding I'm talking about:

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/highlight-banding.jpg)

(ISO 100, new gain = 25, white level forced to 15000 when developing)

So, it looks there is some more detail, but I think it would require some extra postprocessing (this signal is not exactly clean, has strong trend and banding noise). My settings were conservative, with clipped white being clean white.

By trial and error, I changed the white level for this shot from 15282 to 14682, and at that point the highlights were clean (well, just a tiny bit dirty). On the reference shot (the one underexposed by 1 stop), I changed the white level until the clipping area was the same (that was 12282). So, the input signal required to saturate the sensor at 1/25 was log2(15282-2048) - log2(12282-2048) = 0.371 stops weaker than the one required to saturate it at 1/50.

This means our ISO got lower by 0.629 stops, that is, ISO 64.7.

QED
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 05:03:58 PM
Whites are fine here, but there is a very faint pattern noise through the exposure, with new gain at 0x0 ISO 100.  White level - manual lens (15283).
The pattern is that faint, these images needed to be full resolution to show the effect.

ML ISO 100 with calibrated ADTG gain, new gain at default and CMOS patched
2.23 MB
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML_no_new_gain.jpg

1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg

Looks to me (my eyes), that there is actually less read noise with new gain 0.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 23, 2014, 05:20:40 PM
Quote from: Audionut on February 23, 2014, 05:03:58 PM
1.77MB
ML ISO 100 with calibrated ADTG gain, new gain at 0 and CMOS patched
https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/ADTG/ML0.jpg


It has vertical stripes.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 23, 2014, 05:25:55 PM
Nevermind.  One day I will learn maths.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on February 24, 2014, 03:24:03 PM
Quote from: engardeknave on February 23, 2014, 01:00:54 PM
I enjoy just sort of watching this thread utterly without comprehension. Like a cat watching TV.

I'm in a similar state of non-comprehension. Over my head. But fascinating.

But ... is the following something of a "bottom line" and "rubber meets the road" ????

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 24, 2014, 04:01:17 PM
Look at the graph from this page, and at the FAQ from first post.

Generally speaking, by applying some of these tweaks you get some more highlight detail (which means a lower ISO, usually with some more DR).

With the latest findings, the improvement is nearly 0.8 stops on 5D3, ISO 115 pulled from 200. So it's not exactly ISO 100, because it will clip the highlights a little earlier, but I expect it to have 0.8 stops of less shadow noise. Didn't pixel-peep anything, only did the math with open source algorithms that you can try and review.

Dual ISO will be improved mostly on the highlight side (from ISO 66) and on the overlap amount (less aliasing - minor improvement). On the shadow side, at high ISOs most of the noise is photon shot noise rather than electronics noise, so there isn't much room to improve (we are close to the physics limit if I understand well (http://www.dpreview.com/forums/post/53043775), feel free to point me to a better explanation).

I've tried to exploit the nonlinearity by cleaning up the vertical banding and have barely reached ISO 60, so I don't think it's worth the hassle.

The registers on 6D are pretty much identical to 5D3, so you can fire up ADTG GUI and try these tweaks (you will need a custom compilation, it won't load on stock nightly).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 24, 2014, 04:34:40 PM
Quote from: a1ex on February 24, 2014, 04:01:17 PM
The registers on 6D are pretty much identical to 5D3

Thank heaven for that :-p ... looking forward to a mini_iso with working white point casting to try :->
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 24, 2014, 04:48:13 PM
Quote from: a1ex on February 15, 2014, 10:19:40 AM
Now that we have a much more robust memory backend, I'd like to remind you that I'm still waiting for these logs:
http://www.magiclantern.fm/forum/index.php?topic=10111.msg99534#msg99534

(so far I've only got complete logs from 5D3 and 50D, partial logs from 6D, and I did my own logs on 5D2 and 60D)

;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 24, 2014, 04:51:40 PM
Quote from: a1ex on February 24, 2014, 04:48:13 PM
;)

Ah, right, will do - but you've already got 6d iso.log from 1%, so you're only lacking av.log? Would you prefer having it with aperture f8 to f2.8 (my fastest lens) or with shutter that covers the ev range f8 to f1.2?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 24, 2014, 04:59:07 PM
It's OK (better than nothing). For white level, the aperture changes are important (not shutter).

Meanwhile I've got logs from 550D from LebedevRI (uploaded to https://www.dropbox.com/sh/onppbwy44fqomxa/P75rs6pgTW ) and also Nanomad tried to get them from 650D/1100D with only partial success (not yet sure why, probably wrong stubs, the birthday paradox issue from cache hacks, or simply Canon code does no longer calls what I expect it to call).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 24, 2014, 05:18:08 PM
Quote from: a1ex on February 24, 2014, 04:59:07 PM
It's OK (better than nothing). For white level, the aperture changes are important (not shutter).

Well, if someone 6d-ish has to do it again in any case like 1% with his famed aperture gadget, I don't need to do half of it unnecessarily - or will the f8-f2.8 log be sufficient make the mini_mo usable for the 6d? As written before, the white point option (like acr 15k) doesn't work atm.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 24, 2014, 05:20:24 PM
I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 24, 2014, 05:29:48 PM
Quote from: a1ex on February 24, 2014, 05:20:24 PM
I hope it's sufficient. The time required to compile the iso-research branch and take the test pictures should be less than the time required to write all these forum posts :D

Yes, but I'm still idling at work so writing is easy, when back at home and doing braindead test series it's my spare time I rather value and only invest if there's some roi (like a working 6d mini_iso) :-o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 24, 2014, 06:27:56 PM
I'm glad a1ex places ML as a high value of his spare time.  In fact, I am highly appreciative. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 24, 2014, 07:52:40 PM
Quote from: Audionut on February 24, 2014, 06:27:56 PM
I'm glad a1ex places ML as a high value of his spare time.

Obviously we all are as we all are using his free work, and he knows I cannot thank him enough for doing ML and not ending up as a complete nerd but very nice to communicate with.

Personally, I also appreciate a sound work-life balance, whatever that may be... I'm neither applauding self-exploitation (*not* directed towards alex) nor am I entering a contest for the most altruistic man alive, been there, done that, found it to be very unhealthy and being evicted from your flat for missing rent also doesn't help coding efforts.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 24, 2014, 10:14:49 PM
The new gain seems to be expressed in EV. At first sight, 1 unit seems to be around 0.006 EV.

Quick data set (only changed these gains, nothing else; started from ISO 100; the diffs are the median exposure difference printed by latest raw_diag, in EV):

gains = [0    10    20    30    40    50    60    70    80     90     100    150   200   256 ]
diffs = [0.66 0.70  0.79  0.84  0.89  0.94  1.01  1.06  1.13   1.20   1.25   1.56  1.84  2.17]


This means, in order to keep the factory calibration, simply translate the register values by some constant amount (just make sure you don't go below zero). This should fix the vertical banding (didn't try, it's just theory, let me know if it works).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 25, 2014, 12:11:39 AM
Playback filter?

Image :
(http://s29.postimg.org/n56blno6v/image.jpg)

It looks like a dual iso   :D
(http://s29.postimg.org/4e4ebhtmf/image2.jpg)(http://s29.postimg.org/69r6mnign/image3.jpg)

(http://s29.postimg.org/w2r1j0gmv/image4.jpg)(http://s29.postimg.org/ylcqjp2d3/image5.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ItsMeLenny on February 25, 2014, 02:15:43 AM
Lol? Vertical dual Iso. :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 25, 2014, 02:31:28 AM
Quote from: ItsMeLenny on February 25, 2014, 02:15:43 AM
Lol? Vertical dual Iso. :P
Maybe it is a constant vertical banding.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 25, 2014, 08:20:36 AM
- removed -
I misunderstood Greg's post.



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 25, 2014, 11:12:12 AM
What about breaking the register lists up further?

List only these types,

ADTG2
ADTG4
ADTG6

COF0
COF1
COF2

etc.

When you are deep into a bunch of registers, its rather slow to navigate all the way back to the top of the menu to change editing step.

Menu caret would be extremely useful and remove the need for editing step.

I normally edit in steps of 16 first, and then if there is a register that tickles my fancy, I like to then edit by 1.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 25, 2014, 12:05:31 PM
Canon 5D3 - ISO 100 - 1/100s - Lens unscrewed.

ADTG pre-gain only (otherwise Canon defaults), from 0-200 in steps of 10.

Slightly blurred color checker, 100x100 pixel sample patch, and noting the average raw levels (black subtracted), in one of the green channels, together with patch stdev, and optical black stdev.

WL  stdev OB

2739 30.0 5.29
2852 30.5 5.50
2967 32.3 5.71
3092 33.4 5.96
3229 34.7 6.23
3361 35.5 6.46
3497 38.2 6.74
3644 39.2 7.00
3789 40.6 7.26
3958 43.0 7.61
4118 43.8 7.87
4281 46.1 8.28
4473 47.7 8.60
4658 50.0 8.96
4855 52.7 9.33
5063 53.7 9.74
5272 56.8 10.2
5497 57.7 10.5
5722 61.2 11.0
5969 63.6 11.4
6198 66.3 11.9
6468 69.2 12.4
6737 71.5 12.9
7028 75.5 13.5
7326 79.3 14.0
7611 82.6 14.7




ADTG gain, 0.0 EV through -1.00 EV in 0.05 EV steps.

WL  sdtev OB
10035 83.4 6.49
9925 80.2 6.29
9362 76.8 6.14
9032 75.9 5.92
8733 71.9 5.75
8427 68.9 5.49
8133 67.4 5.31
7850 63.8 5.18
7584 62.3 5.00
7319 60.2 4.83
7263 58.3 4.60
6828 55.2 4.46
6600 54.1 4.32
6365 51.8 4.15
6158 51.0 4.03
5942 48.0 3.87
5739 47.5 3.76
5543 45.0 3.57
5358 43.1 3.49
5174 42.6 3.33
4989 41.8 3.23



ADTG gain, -1.10 EV through -2.30 EV in 0.10 EV steps.

WL  sdtev OB
4655 38.2 3.02
4345 35.9 2.83
4047 33.2 2.57
3782 30.8 2.38
3519 28.6 2.20
3285 27.0 2.08
3065 25.3 1.95
2865 23.7 1.80
2666 21.8 1.68
2429 20.4 1.59
2321 19.2 1.49
2168 18.1 1.41
2022 16.9 1.31
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: vroem on February 25, 2014, 04:00:36 PM
Quote from: a1ex on February 23, 2014, 02:10:23 PM
Dynamic range series (ISO 100-12800 in Canon menu, ADTG gain at sweet spot, ADTG2/4[8,9,A,B] at 36 for ISO 100 and 0 for all other ISOs, CMOS patches enabled.


ISO    DR
66     11.589 + 0.1
115    11.772
230    11.649
460    11.427
912    10.968
1750   10.369
3333   9.610
6400   8.834

I made this DR chart. Is it correct?

(http://i.imgur.com/kXrbKTq.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 25, 2014, 04:35:58 PM
The values ​​saturate offset and black/white, such as ISO 200

(http://s8.postimg.org/ral6r7z39/VRAM1.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 25, 2014, 04:57:43 PM
Quote from: vroem on February 25, 2014, 04:00:36 PM
I made this DR chart. Is it correct?

Looks fine to me.

On the shadow end, one may argue that because of half resolution, you lose log2(sqrt(1/2)) = 0.5 stops (and it makes sense if you look at this test (http://www.magiclantern.fm/forum/index.php?topic=7139.msg99437#msg99437)). I didn't do the math to see if it's really 0.5, but it's probably close.

I should probably update the formulas from the menu => some settings like 800/1600 might result in worse shadows than either of them according to this theory. Didn't try to see if it's true or not, and the blending algorithm wasn't exactly optimized for situations like this.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 25, 2014, 08:21:42 PM
c0F0[819c] 0xf87 -> 0x20
c0F0[8034] 0x1079 -> 0x1e70 (less value to improve DR, but cause vertical stripes blue channel, wrong white point blue channel - http://www.magiclantern.fm/forum/index.php?topic=10111.msg103483#msg103483)

(http://s22.postimg.org/qgjv1vmep/VRAM1.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 25, 2014, 10:13:09 PM
Quote from: a1ex on February 25, 2014, 04:57:43 PM
(...)
On the shadow end, one may argue that because of half resolution, you lose log2(sqrt(1/2)) = 0.5 stops (and it makes sense if you look at this test (http://www.magiclantern.fm/forum/index.php?topic=7139.msg99437#msg99437)). I didn't do the math to see if it's really 0.5, but it's probably close.
(...)
I did the math for dual_iso* to compare with some tests; according to the sigma and Nyquist matching algorithm of the Aliasing Minimization and Zipper Elimination demosaicing, with multi-pass >4 and considering the matrix intrication at the source, the lose of DR should be less than 1/2 stops for all the spectral range in daylight photography.

*did for some special wavelengths, not relevant here.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 25, 2014, 10:27:20 PM
Quote from: Audionut on February 25, 2014, 05:28:48 PM
fe - default 0x4, lower values reduce stdev, higher values increase stdev

This one is interesting, try setting it to 0 in both ADTG2 and ADTG4. With raw_diag I can measure 11.83 stops, but my intuition says it might be a little more.

@SpcCb: can you share some more details about how you did the math?

(I tried to understand the AMaZE algorithm, but only figured out how to call it; if you can shed some light, I'm interested)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 03:07:44 AM
Quote from: a1ex on February 25, 2014, 10:27:20 PM
With raw_diag I can measure 11.83 stops,

Same here.

It seems to do this to the highlights.
(https://s15.postimg.cc/lqs681ny3/highlights.png)


I was only using ADTG pre-gain/gain, CMOS tweak and yet another gain.  Naturally, the WL was pretty low.  I bumped the lens back into electronic mode and hit 11.91 EV.

So then I unscrewed the lens and changed Digital Gain from 0x200 to 0x220, and again 11.91 EV.  This seemed to clean the highlight zebras also.

This configuration also looks to clean the highlights a little @ ISO 100 with pre-gain 0

@vroem
I really like that graph.  It visualises in a simple manner.  I like how it shows the linear loss in highlights, and the non-linear gain in the shadows.  Of course, using ML ISO, the loss in the shadows isn't as great as with Canon ISO.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 26, 2014, 04:48:41 AM
500D has the same problem when trying to gain more than 0.33EV.
Up to 0.4EV occur on the blue channel.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 05:31:13 AM
Quote from: Greg on February 25, 2014, 08:21:42 PM
c0F0[819c] 0xf87 -> 0x20

This one is saturate offset on 5D3.

Quote from: Greg on February 25, 2014, 08:21:42 PM
c0F0[8034] 0x1079 -> 0x1e70 (less value to improve DR, but cause vertical stripes blue channel, wrong white point blue channel - http://www.magiclantern.fm/forum/index.php?topic=10111.msg103483#msg103483)

I can't find this one on 5D3.  So many registers  :o
I did find c0F3[8034] 0x0

Looks like the faint banding is happening at other ISOs also, here is ISO 200

Canon
(https://s15.postimg.cc/3nz3gl7gb/Canon_ISO200.jpg)

ML
(https://s15.postimg.cc/77l16ehvv/ML_ISO200.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 26, 2014, 08:04:57 AM
When you get these artifacts in highlights, one of the gains is too low (below its sweet spot).

0xFE seems to control 3 amplifiers triggered by bits (valid values from 0 to 7):

gains = [0    1     2     3     4     5     6     7   ]
diffs = [0    0.08  0.11  0.17  0.54  0.65  0.72  0.86]


so changing it from 4 to 3 might be already too much (you may need to compensate by increasing some other gain).

@Audionut: can you do the noise tests (WL + stdev + OB) for this register too?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 09:50:05 AM
For clarity, this sample set was taken at 1/60s.  Same setup as previous test.  Also, my OB measurement is 10 pixels from left, 500 px from top, 90px wide and 2000px high.

ADTG[0xFE] - 0 through 7.

WL  stdev OB
1144 14.9 4.70
1205 16.3 4.91
1237 16.4 5.05
1314 17.2 5.29
1673 21.9 6.54
1809 24.0 6.96
1881 24.7 7.28
2087 27.4 7.99
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 26, 2014, 10:23:04 AM
Based on the data from Audionut, I've computed how much noise we gain (or lose) by increasing each of these 3 gains.


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/adtg-fe.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/adtg-888x.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/adtg-89ab.png)

Think at ISO 100 vs 200: you increase the gain and you get lower noise (cleaner shadows). There is a point in keeping this kind of gains at large values (you may clip highlight detail, but you get cleaner shadows in return). So, the 0xFE gain is one of these gains that help reducing noise.

Also think at ISO 3200 vs 6400: you increase the gain, but there's little or no improvement in noise. You want to keep this kind of gains at smaller values (because you may clip highlight detail and you get nothing in return). The other two gains have only a minor impact on the measured noise, so we want to keep them as low as possible.

Now, from these numbers, can we draw a conclusion regarding the position of each amp in the processing chain? Can we say the order is CMOS[0] -> 0xFE -> 888x + SaturateOffset -> 8/9/A/B -> ADC?

(note that I've renamed them back to their register address, because I'm no longer sure whether 8/9/A/B is before 888x or viceversa)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 10:28:14 AM
Would it be useful to measure saturateOffset?  I can do that when I get home later tonight.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 26, 2014, 10:32:43 AM
Yes, not a bad idea.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mk11174 on February 26, 2014, 10:33:07 AM
Hoping this is what was needed for 700D Iso log?

https://bitbucket.org/mk11174/t5i-backup/downloads/700D_ISO.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_ISO.LOG)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 26, 2014, 10:45:52 AM
That's it. Can you also do the aperture and shutter series?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mk11174 on February 26, 2014, 11:12:17 AM
Quote from: a1ex on February 26, 2014, 10:45:52 AM
That's it. Can you also do the aperture and shutter series?
will try and post if works.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mk11174 on February 26, 2014, 11:41:09 AM
700D PHOTO MODE ADTG Logs

Aperture from F2.8 to F22
https://bitbucket.org/mk11174/t5i-backup/downloads/700D_APERTURE_F2.8_TO_F22.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_APERTURE_F2.8_TO_F22.LOG)

Shutter from 30second to 1/4000
https://bitbucket.org/mk11174/t5i-backup/downloads/700D_SHUTTER_30sec_TO_4000.LOG (https://bitbucket.org/mk11174/t5i-backup/downloads/700D_SHUTTER_30sec_TO_4000.LOG)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 04:05:30 PM
edit:  I neglected to unscrew the lens

ADTG[0xFE] looks a little bumpy, here it is in reverse.

WL  stdev OB
1856 25.9 7.97
1676 23.9 7.21
1625 22.5 6.93
1499 20.9 6.47
1164 16.4 5.25
1111 15.4 5.00
1068 15.5 4.88
1012 14.4 4.66


SaturateOffset - 2000 through 100, in steps of 100.  BL is the reported black level in the file.
iso_regs reported default for this register 3148.  iso_regs only allowed adjustment up to 2000.

WL  stdev OB  BL
4633 41.8 6.53 883
4658 42.9 6.49 782
4619 41.4 6.80 681
4659 43.4 6.50 580
4696 42.5 6.49 478
4677 41.5 6.86 377
4646 41.9 6.54 275
4677 41.7 6.57 174
4668 42.2 6.63 73
4573 40.5 0.0
4508 41.0 0.0
4416 41.3 0.0
4330 41.8 0.0
4185 41.2 0.0
4156 42.2 0.0
3999 41.3 0.0
3897 41.8 0.0
3812 41.3 0.0
3729 42.6 0.0
3644 42.7 0.0


Here is the same test images as above, but measuring from a darker grey patch.

WL stdev
770 14.4
776 14.7
717 14.7
726 14.8
730 15.0
727 14.9
724 14.5
727 14.6
726 15.1
687 14.7
590 15.3
490 14.6
391 14.4
282 14.6
193 14.9
82 15.2
0.6 2.83
0.0 0.0
0.0 0.0
0.0 0.0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 26, 2014, 04:39:46 PM
My shutter unit says, give me a break.
Another 500 images...  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 04:40:51 PM
I forgot to unscrew the lens for the above results.

Patching iso_regs to allow adjustments up to 3148, just by checking that my code changes worked, I can see that SaturateOffset is not reducing the noise.  It fluctuates around the default Canon measured stdev.

The DR increase comes from lowering of the black level (crushing the blacks).

Quote from: Greg on February 26, 2014, 04:39:46 PM
My shutter unit says, give me a break.
Another 500 images...  :P

Heh.  Mine too :)

# HG changeset patch
# User Audionut
# Date 1393431415 -7200
# Branch unified
# Node ID 4ab57c1f286e8b7338feb162e9998ba37ede3431
# Parent  1ca7d872bdb2f18dc5813d69db474a7f25b74951
Increase SaturateOffset adjustment range

diff -r 1ca7d872bdb2 -r 4ab57c1f286e modules/iso_regs/iso_regs.c
--- a/modules/iso_regs/iso_regs.c Wed Feb 26 06:49:12 2014 +0200
+++ b/modules/iso_regs/iso_regs.c Wed Feb 26 18:16:55 2014 +0200
@@ -513,7 +513,7 @@
                 .priv = &saturate_offset,
                 .update = saturate_offset_update,
                 .min = 0,
-                .max = 2000,
+                .max = 3148,
                 .unit = UNIT_DEC,
                 .help  = "Alters black level and stretches the range, keeping white fixed.",
                 .help2 = "Decrease to get more highlight details, but watch out RAW zebras.",
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 26, 2014, 05:35:30 PM
500D ISO 200
ADTG[7] 0x4046 -> 0x4036
c0F0[819c] 0x66f -> 0x40
c0F0[8034] 0x1991 -> 0x1f11

(http://s10.postimg.org/bufirr7g9/VRAM2.png) (http://s10.postimg.org/brvn4x3sp/VRAM1.png)
(http://s10.postimg.org/mij9qlhfd/VRAM4.png)

Vertical banding in the shadows (default vs trick). ACR +5EV
(http://s2.postimg.org/4g28lp9g9/default.jpg) (http://s2.postimg.org/inrxacm55/trick.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 26, 2014, 05:47:37 PM
Here's a hint about where each amplifier might be in the processing chain. I've tried to estimate how much noise is at the input of each stage and how much is introduced after it.

Theory:

In complete darkness (zero input signal), let's consider this simplified noise behavior for any amplification stage:
output = K * input_noise + output_noise

Assumming the input and output noises are not correlated and Gaussian, assumming 100% linear behavior, and also assumming we can modify the linear gain K somehow (either by turning the amplifier on/off or by adjusting its gain with an integer parameter), we can find out the two noise components from the following linear system by measuring the output noise and the value of the gains (see this (http://www.statlect.com/normal_distribution_linear_combinations.htm)):
K1^2 * var(input_noise) + var(output_noise) = var(output1)
K2^2 * var(input_noise) + var(output_noise) = var(output2)

I will start from ISO 100, consider this setting the reference configuration (gain = 1.0, or 0.0 EV), and will change one amplifier at a time from iso_regs.

Measurements and results:

                      measured gain (EV)     measured OB noise (ADU)     input noise (ADU)     output noise (ADU)
                                                                         normalized to ISO 100 noise level       
Canon ISO 100          0.00                  6.54                         NaN                   NaN 
CMOS[0] = 0x773        5.66                 26.26                        0.50                  6.52 
CMOS[0] = 0xDD3        5.73                 28.00                        0.51                  6.52 
CMOS[0] = 0xFF3        6.47                 48.44                        0.54                  6.52 
CMOS[0] = 0xCC3        4.81                 16.93                        0.56                  6.52 
CMOS[0] = 0x553        4.79                 16.76                        0.56                  6.52 
CMOS[0] = 0x443        3.96                 11.15                        0.58                  6.51 
CMOS[0] = 0xEE3        3.52                  9.33                        0.58                  6.51 
CMOS[0] = 0xBB3        3.90                 10.90                        0.59                  6.51 
CMOS[0] = 0xAA3        2.95                  8.15                        0.63                  6.51 
CMOS[0] = 0x333        2.99                  8.33                        0.65                  6.51 
CMOS[0] = 0x663        2.52                  7.65                        0.70                  6.50 
CMOS[0] = 0x993        1.93                  7.04                        0.71                  6.50 
CMOS[0] = 0x883        0.97                  6.65                        0.72                  6.50 
CMOS[0] = 0x223        1.95                  7.14                        0.77                  6.49 
CMOS[0] = 0x113        0.99                  6.70                        0.85                  6.48 
ADTG 0xFE = 3         -0.35                  5.30                        6.18                  2.14 
ADTG 0xFE = 0         -0.55                  4.70                        6.23                  2.00 
ADTG 0xFE = 7          0.31                  8.03                        6.36                  1.53 
ADTG 8/9/A/B 0        -0.36                  5.17                        6.39                  1.39 
ADTG 888x +1EV         1.00                 13.04                        6.51                  0.59 
ADTG 8/9/A/B 30       -0.17                  5.81                        6.55                  0.00 
ADTG 888x -1EV        -1.00                  3.22                        6.57                  0.00 
ADTG 8/9/A/B 90        0.17                  7.41                        6.76                  0.00 



    k = 2 .^ gains(i);
    A = [1   1 ;
         k*k 1];
    b = [noise(1); noise(i)] .^ 2;
    noises = sqrt(A\b);
    input_noises(i) = noises(1);
    output_noises(i) = noises(2);


So, it seems clear that ADTG 0xFE is before the other two, but it's not that clear which one from the other two ADTG amps is the first in the image processing chain.

However, ADTG 8/9/A/B is able to pull more highlight detail than ADTG 888x. Since turning 888x all the way down did not bring the other one out of saturation, my conclusion is that ADTG 8/9/A/B is before 888x.

Something like this:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso-chain.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 06:06:05 PM
Applying that theory @ ISO 100

I can drop [0xFE] to 1 before WL drops giving 11.463 EV.

From here, I can drop ADTG [888x] to -0.04 EV before WL drop, giving 11.545 EV
Or, drop ADTG 8/9/A/B to 46 before WL drop, giving 11.514 EV.

In either case, once 1 of the gains has been optimised, the other can not be reduced without dropping WL.

Using ADTG[0xFE] 1, and ADTG [888x] -0.04, applying Digital Gain 540 increases DR to 11.593

edit:  ISO 200 should provide more useful results since here we can be sure of plenty of gain into the ADTG stage.

Here, I can reduce ADTG[0xFE] to 0 before WL drop, giving 11.460 EV

From here, I can drop ADTG [888x] -0.32 EV before WL drop, giving 11.776 EV
Or, drop ADTG 8/9/A/B to 0 before WL drop, giving 11.772 EV.

Again, in either case, I cannot adjust the other without reducing WL.

@ ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 EV.  Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + ADTG 8/9/A/B 0 = 11.641 + ADTG [888x] -0.03 EV = 11.673 EV
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG [888x] -0.34 EV = 11.671 EV.  Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG 8/9/A/B 29 = 11.662.  Cannot reduce ADTG [888x] without WL drop.


Looks to me, ADTG [888x] is before ADTG 8/9/A/B

A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 + digital gain 530 = 11.732 EV.


It would be nice to have more adjustment in ADTG[0xFE], since we can reduce this to 0 (at higher ISOs) and still have adjustment in the other gains.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 26, 2014, 06:15:05 PM
500D ISO 200

I lengthen exposure of 1/2EV with a trick.

Trick vs default :
(http://s3.postimg.org/5tw95mu6r/trick1.jpg) (http://s3.postimg.org/533izurtf/default1.jpg)
(http://s3.postimg.org/ca57vq2qb/trick2.jpg) (http://s3.postimg.org/swms4svo3/default2.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 26, 2014, 06:57:22 PM
Quote from: Audionut on February 26, 2014, 06:06:05 PM
A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.35 EV = 11.679 + digital gain 530 = 11.732 EV.

I have an idea about this. Could you estimate the gain by comparing both whitelevels?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 26, 2014, 07:13:51 PM
Quote from: naturalsound on February 26, 2014, 06:57:22 PM
I have an idea about this. Could you estimate the gain by comparing both whitelevels?

ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  WL 15283, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 520, WL 15490, stdev 4.09
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 525, WL 15619, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 530, WL 15748, stdev 4.06
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 540, WL 16007, stdev 4.11
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 26, 2014, 07:59:40 PM
To cut a long story short: This subtle amount of digital gain alters the shape of the noise probability distribution and thus alters the standard deviation.

Long version:
As the data is already quantized for any amount of digital gain there will be gaps (at value v_gap in the histogram (as seen in your shown histograms a few pages earlier) because there is no value v_old to fulfill

Round[gain*v_old]==v_gap

In the following I assume an histogram centered at 2024 at unity digital gain.

Depending on the gap position the retrieved standard deviation may rise. This occurs the number of gaps in the histogram increases. The initial histogram has no gaps (because of unity gain). Increasing the gain increases the number of gaps inside the data. But those gaps lie outside the noise distribution. So the will not affect the computed stdev. Nevertheless the white level rises, so the retrieved dynamic range rises.
For your gain ([0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 540, WL 16007, stdev 4.11) the first gap is produced within the distribution (at initial value of v_old=2023 -> v_gap=2090). Thus the histogram gets wider and the stdev rises. This bigger stdev is now divided by the white level which did only rise one incremental step, resulting in a decreased Dynamic range.

I think this explanation holds true also for initial histograms that already have gaps, because the additional gaps will appear anyway. I will try to add some graphics.


UPDATE: I am sorry, this only affects the standard deviation of the noise probability distribution. Depending on your calculation of the Dynamic range this may not change the dynamic range at all, if the calculation takes into account the median of the noise's probability distribution.
Maybe these finding at least help to figure out the exact amount of digital gain, as the positions of gaps in a full scale histogram are sharp indicators for digital gain. Maybe those "half gaps" seen in some histograms originate from digital smoothing algorithms or correction matrices?

UPDATE2: This is too obvious. I'm sure it is already stated somewhere in those 21 pages  :-[
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 27, 2014, 04:12:06 PM
Digital gain, from low values to higher values.

(https://s15.postimg.cc/pxd0nkaqz/VRAM0.png) (https://s15.postimg.cc/7ujxwcmm3/VRAM1.png)

(https://s15.postimg.cc/cgg24pfuz/VRAM2.png) (https://s15.postimg.cc/hruypf9nf/VRAM3.png)

(https://s15.postimg.cc/ttqcjkqln/VRAM4.png) (https://s15.postimg.cc/6fid7nge3/VRAM5.png)

(https://s15.postimg.cc/gcte0q3ff/VRAM6.png) (https://s15.postimg.cc/m0zorm023/VRAM7.png)

(https://s15.postimg.cc/6fid7o3jf/VRAM8.png)


When it comes to log2(white/stdev), there is a sweet spot.  This seems to be around value (530), for lower ISOs.

BTW, 14.5 EV with dual ISO and 11 EV @ ISO 1600 is remarkable.  And is better then all other consumer camera afaik.

Heck, ISO 3200 still gives 10.5 EV.  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on February 27, 2014, 04:55:56 PM
500D
CMOS[3] 0x810 -> 0xfff

ISO 100 0.015EV
ISO 200 0.062EV

Also cause vertical stripes...  :-\

So 500D can be changed safely only in black / white and saturate offset.
Gain 0.33 EV ISO 100
Other changes cause banding.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 27, 2014, 06:11:00 PM
ADTG 8/9/A/B defaults to 85 in movie mode.  Reducing it looks to pull back some highlight detail.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 27, 2014, 09:20:27 PM
Finally something that has effect in movie mode :D

My current settings for photo mode (based on the above theory, which basically says bump the gains with low input noise and lower the ones with high input noise):

- ISO 66 from 100: unchanged (see my previous posts).
- ISO 200 - 12800 in Canon menu: ADTG 0xFE = 3 instead of 4, ADTG 8/9/A/B as close to 0 as possible (keeping the deltas between them), ADTG gains -0.16, -0.18, -0.18, -0.19, -0.25, -0.31 and -0.35 EV. Resulting ISOs: 100, 200, 400, 800, 1530, 3000, 5700 (that is, I can effectively pull down all the important ISOs from the next full-stop ISO).

This time I did not measure the ISOs; I've simply used the theoretical model from iso_regs and rounded the ISOs. Feel free to compile raw_diag and iso_regs if you think you can get better measurements.


dr =        [ 10.989 10.928 10.799 10.594 10.153  9.521 8.584 7.778 ] ;     # measured with raw_diag
isos =      [ 100    200    400    800    1600   3200   6400  12800 ] ;     # ISOs from Canon menu

dr_tweak =  [ 11.589 11.782 11.687 11.463 11.028 10.447 9.704 8.930 ] + ... # measured with raw_diag
            [ 0.1    0.1    0.1    0.1    0.1    0      0     0     ] ;     # correction for nonlinearity
iso_tweak = [ 66     100    200    400    800    1530   3000  5700  ] ;     # estimated with iso_regs, adjusted for nonlinearity
                                                                            # and rounded (e.g. 218->203->200)


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-cmosfe.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/iso200from400.png)

Note: in the screenshot, the ISO equivalence model assumes linear response; however, the response near highlights is slightly nonlinear. I estimate this nonlinearity helps capturing another 0.1 EV of highlights. Therefore, with these tweaks applied, the equivalent ISO according to DxO definition (clipping point) is nearly 1 stop lower (so the screenshot shows ISO 200 pulled from 400). For the same reason, I've added 0.1 stops to DR measurements where the nonlinear response was noticeable.

This graph does not include the white level optimization for the raw converters or the digital gain correction from the aperture. On 5D3, I estimate this improvement from 0.03 EV (manual lens) to 0.183 EV (f1.4 lens), from the white levels and aperture logs:

log2(15282-2048) - log2(15000-2048) => 0.031
log2(569) - log2(512) => 0.152


Of course, each of these gains is tiny, but when all of them add up to almost 1 full stop, the total improvement should be noticeable.

P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 01:09:30 AM
Quote from: a1ex on February 25, 2014, 10:27:20 PM
(...)
@SpcCb: can you share some more details about how you did the math?

(I tried to understand the AMaZE algorithm, but only figured out how to call it; if you can shed some light, I'm interested)

I will try to be simple, I mean I hope I will be.. :)

We can refer to what we discussed in the dual_iso thread (http://www.magiclantern.fm/forum/index.php?topic=7139.msg102924#msg102924) about SVE images to explain the DR in this case:

We can see how is the gain with dual_iso in a perfect world (calculations done @10E-15 & results round @10E-3);
With a single frame:
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

-> Wooh, on the paper it is monstrous!

However I suppose you prefer a more physical approach to compare with samples, then from DRSVE we get:
DRdi = log2[(( WL - offset ) / (σSmaxx̄ σSmin) ) (smax / smin)]
Note: we should use the same WL and offset in both frames; because of course _for example_ with a different offset it will generate noise (!).

In my case I used a 5D2;
With a single frame, in the best case and with thermal considerations (short exposure time, up to CMOS @40°C):
DRsf = log2(( 15760 - 1024 ) / 6.47 )
DRsf = 11.153 stops

With dual_iso 100/800:
DRdi = log2[(( 15760 - 1024 ) / √{( 8² + 6.47² ) / 2 }) 64 ]
DRdi = 16.984 stops

Then, now we have basic maths to do projection of our SVE, but be careful it does not take consideration of the PSF + MTF, so the wavelength of the source, etc.
To do that we have to introduce light diffraction model in the equation.
I will not demonstrate the maths here, it's a bit complex and I think not relevant to make projections with samples we use: It requires calibrated sources, lens modelisation, etc.
BTW, in my study with a off-continuum mono-spectral source (λ = 0.656μm, Hα (http://en.wikipedia.org/wiki/H-alpha) source) I recorded an average difference of 3.9% and a sigma difference of 5.0% (both ±10E-1) between single frame image and dual_iso image by using AMaZE both for CR2->DNG & DNG demosaicing.
This mainly because I use 1 of 4px in the Bayer Matrix in this case and there's an intrication between the dual_iso SVE and the Bayer Matrix.
Beside, a similar effect should be observed in regular photography.

About AMaZE, what is very interesting is this algorithm do multiple analyses to proceed to the reconstruction.
There's a (spatial) vector analyse of structures (macro-blocks & special sophisticate diagonal), color/chromic analyse and _in the lasts versions_ a CA analyse, and there's a Nyquist texture (pattern) analyse to make better area interpolations.
IMHO, it is very smart from Emil Martinec _and certainly not foreign to the fact what Emil works in Astrophysics_ because this is well know in our domain what without taking consideration of the light nature it is hard to be close to the reality (even we never match her).

With this algorithm there's a significant SNR improvement face to other reconstruction algorithms; I ever measured +25% on hard cases face to VNG HAD HFFE etc. So compared to the simple mathematical projection where just an average is considered, we should see a gain, even if it's not +2 stops or -50% of sigma.
When we see SNR gain, we see DR gain. So, voilà. :)

Don't hesitate to point me what you don't find clear, I will see how to make it more simple or from a different point of view.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 28, 2014, 07:42:51 AM
Quote from: a1ex on February 27, 2014, 09:20:27 PM
P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.

Indeed.  The graph clearly shows how there is very little DR gain in extreme ISOs.  Simply shifting the ISO point.




Using the settings a1ex listed above, comparing the highlights @ ISO 200.

Images removed.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on February 28, 2014, 09:52:19 AM
Quote from: Audionut on February 28, 2014, 07:42:51 AM
Indeed.  The graph clearly shows how there is very little DR gain in extreme ISOs.

This probably just proves that Canon did this right by themselves, they already have superior dr vs. Nikon on higher iso ... and every little gain is a blessing in these situations. Plus since the base dr is lower, a small absolute gain here is a larger relative one :->
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 10:25:25 AM
Quote from: Audionut on February 28, 2014, 07:42:51 AM
Using the settings a1ex listed above, comparing the highlights @ ISO 200.

Can you share the EXIF info? Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 10:46:33 AM
Quote from: SpcCb on February 28, 2014, 01:09:30 AM
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

-> Wooh, on the paper it is monstrous!

I'm not familiar with these notations, can you define them first?

In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.

From here I'm lost, because the checksum doesn't match.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 28, 2014, 12:39:21 PM
Quote from: a1ex on February 28, 2014, 10:25:25 AM
Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?

I was still thinking in terms of relatively small ISO differences.  So Canon ISO 200 vs ML ISO 100, they were at the same exposure though.

In essence, the gain is all in the shadows now?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 12:51:42 PM
If you just turn down some amps starting from some base ISO from Canon menu, the gain will be in highlights.

Since the gain is almost 1 stop, you can now bump Canon ISO one stop and move some of this gain to the shadows (you know the story, you always lose 1 EV of highlights, but you gain back a little less than 1 EV in shadows, and this shadow improvement vanishes at high ISOs).

So, now that we can re-create the full-stop ISOs with different settings (e.g. pull them down from the next higher ISO), we can compare them to the unmodified Canon ISOs. There are two things you should look for:

1) clipping point (should be the same, because this is how ISO is defined - the input signal required to saturate the sensor)
2) noise (our modified ISOs should be cleaner: higher DR => higher SNR)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 28, 2014, 02:21:23 PM
Can you confirm recommended settings for ISO 100 please?

And is it ok to use the CMOS tweak and WL settings inside mini_iso?  With ADTG gain set to 0.0?

I have a set of data, but this has the above mentioned tweaks, and I need to redo the test with correct ISO 100 settings.

Canon shutter file WL    stdev
100 6    6880 13027 194.8
200 13  6896 13074 204.7
400 25  6909 13221 221.0
800 50  6922 13116 238.1
1600 100 6938 13115 271.1
3200 200 6951 13091 326.0
6400 400 6965 13318 433.8


ML  shutter file WL    stdev
100 6    6984 14016 180.9
200 13  6995 14106 193.4
400 25  7007 14060 203.5
800 50  7019 13964 223.5
1600 100 7033 13502 247.0
3200 200 7047 13180 203.5
6400 400 7062 12965 383.8


note: that ML ISO is reported as Canon ISO -1 EV.  Also, reported WL has not been black subtracted.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 02:27:47 PM
If you use both modules, there might be some conflicts over these settings; so, to be safe, just use iso_regs and a manual lens.

White level from raw_diag should be 15282 in all cases. There might be exceptions at high ISOs because of hot pixels, where WL might be estimated above 15282 (ignore them).

So:
- unmodified ISO 100
- ISO 200 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.16 EV (do the math) and ISO estimated in iso_regs around 109. The difference until 100 should be covered by nonlinearity.

- unmodified ISO 200
- ISO 400 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.18 EV (do the math) and ISO estimated in iso_regs around 218. The difference until 200 should be covered by nonlinearity.

and so on.

The CMOS and black/white tweaks are separate and do not have a noticeable impact over the measured ISOs. You can analyze these ones separately (to make sure their impact is indeed minor) or combined (to check the overall effect of all these tweaks).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 28, 2014, 05:19:11 PM
Lens unscrewed, last stop before full saturation of the green channel.
WL is the average WL measured in the sample patch of one of the green channels.  stdev is the standard deviation measured from the sample patch, in the green channel.

Canon shutter file WL    stdev
100 15  7316 13255 158.2
200 30  7323 13330 170.6
400 60  7330 13513 185.8
800 125 7337 13414 208.9
1600 250 7344 13471 247.3
3200 500 7351 13607 308.5
6400 1000 7358 13963 419.6
1280 2000 7365 14489 543.4*
1280 2500 7366 12010 523.9


* Max value in channel pushed to saturation.

ML  shutter file WL    stdev
100 10  7380 14305 158.9*
100 13  7381 12028 144.9
100 15  7382 10071 121.5
200 15  7386 14169 148.7
400 30  7393 14295 157.3
800 60  7400 14208 169.7
1600 125 7407 14176 195.9
3200 250 7414 13702 227.3
6400 500 7421 13627 285.8
1280 1000 7428 13366 370.1


* Last stop before total saturation in channel.
The ISO rating displayed is that in Canon menu, ie: the pulled ISO.  The gains used are as above, and for ISO 100 here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447).
CMOS tweak was applied.


If I have my maths right, The noise difference between ML ISO 100 and Canon ISO 100 in the highlights.

log2(14169/13255) - log2(158.2/148.7) = 0.0068 EV

And the exposure difference is log2(14169/13255) = 0.096 EV.  Which puts Canon ISO 200 @ ISO 107, subject to any nonlinear behaviour, not captured.

Canon ISO 100 vs Canon pulled ISO 100

log2(15/10) - log2(14305/13255) = 0.4750 EV

Which puts ISO 100 @ ISO 77, again, subject to any nonlinear behaviour, not captured.

Is the noise level difference log2(15/10) - log2(14305/13255) - (158.9/158.2) = 0.529 EV?




Having a quick look at the shadows.
Click thumbnails for full resolution JPGs.  These are around 10mb.

Canon ISO 100
(https://s15.postimg.cc/3lf7tobhn/Canon_thumb.jpg) (https://s15.postimg.cc/oibfyc1sr/Canon.jpg)

Canon pulled ISO 200
(https://s15.postimg.cc/mdr2x9fln/ML_thumb.jpg) (https://s15.postimg.cc/tgyyd2fy3/image.jpg)

We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 05:45:26 PM
Quote from: a1ex on February 28, 2014, 10:46:33 AM
Quote from: SpcCb on February 28, 2014, 01:09:30 AM
(...)
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

(...)
I'm not familiar with these notations, can you define them first?

In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.

From here I'm lost, because the checksum doesn't match.

OK, let's see this like that:
emax and emin are the maximum and minimum energy scale (not a correct naming, just to get a picture of this) present in the exposure pattern of the SVE.
We took as example to do maths ISO 100/800 in dual_iso, so we get:
ISO 100 - 200 - 400 - 800 range
Corresponding to:
e3 = 4e2 = 16e1 = 64e0
Remember we are in power of 2 world here, to be in accord with the ADU range (16384). Intercourse between ADU and ISO numbers, this is weird. :)
There's no standard deviation here as we are in the 'perfect world' example; it is just to give us the absolute viewing and to know what are the system limits. I always do it to be sure we will not go outside these limits after, when we add 'more physical' parameters, and to give the most 'simple' figure at the beginning (like an introduction).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 06:02:02 PM
Quote from: Audionut on February 28, 2014, 05:19:11 PM
Lens unscrewed, last stop before full saturation of the green channel.

WL: from your description, this is not white level (you have useful data above it).

Gains: you linked the settings before finding the 0xFE register. Once you have changed this one from 4 to 3, the sweet spot for ADTG gains is now the one from here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).

Shutter: please use SI units.

Noise: you are probably trying to compute the upper limit of the SNR curve (best-case SNR). Signal is that thing you named WL (assumming you subtracted the black level), noise is stdev. SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).

Exposure difference: yes, overall brightness is higher by 0.1 stops. If you measure patches at 1 stop below clipping point, you will not see the nonlinear behavior. If you compare the exact clipping point, you will see it.

You quoted Canon ISO in modified ISOs, which is irrelevant and makes it hard to choose what to compare with what. Moreover, since the equivalent ISOs above 800 are no longer full-stop ISOs, these lines can't be compared directly (you can either plot a graph showing the equivalent ISO vs measured SNR, or you can pull exactly one stop to get ISOs that can be compared directly).

@SpcCb: I'm lost. Where did you get these numbers from? Why ISO 800 has 64x more "energy" than 100? What kind of "energy" is this?

I use the engineering definition of DR: log2(full well capacity / noise RMS), which is log2(white - black) - log2(dark frame stdev), approximated to log2(white - black) - log2(optical black stdev). How does this "energy" fit here?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on February 28, 2014, 06:36:25 PM
Quote from: a1ex on February 28, 2014, 06:02:02 PM
WL: from your description, this is not white level (you have useful data above it).

Correct, it is the average ADU value.  This has not been black subtracted.

Quote from: a1ex on February 28, 2014, 06:02:02 PM
Gains: you linked the settings before finding the 0xFE register. Once you have changed this one from 4 to 3, the sweet spot for ADTG gains is now the one from here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).

The gains for Canon Pulled ISO 100 were the ones that I linked (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103447#msg103447).  The gains for Canon Pulled ISOs 200-12800 are the ones that you linked (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104170#msg104170).

Because you said,

Quote- ISO 66 from 100: unchanged (see my previous posts).

Quote from: a1ex on February 28, 2014, 06:02:02 PM
Noise: you are probably trying to compute the upper limit of the SNR curve (best-case SNR). Signal is that thing you named WL (assumming you subtracted the black level), noise is stdev. SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).

No.  I am reporting the standard deviation of the measured patch.  Simple.  If the data is not useful, well, it's better then leaving it out and finding out later, that it would have been useful.

In the images I linked, this is simple.  Same shutter, same claimed ISO rating, boost exposure +5 EV in post.  I can use my eyes to see this difference, I don't have to rely on my lack of maths skills.


Quote from: a1ex on February 28, 2014, 06:02:02 PM
Exposure difference: yes, overall brightness is higher by 0.1 stops. If you measure patches at 1 stop below clipping point, you will not see the nonlinear behavior. If you compare the exact clipping point, you will see it.

I specifically mentioned that any nonlinear behaviour probably wasn't captured.


Quote from: a1ex on February 28, 2014, 06:02:02 PM
You quoted Canon ISO in modified ISOs, which is irrelevant and makes it hard to choose what to compare with what. Moreover, since the equivalent ISOs above 800 are no longer full-stop ISOs, these lines can't be compared directly (you can either plot a graph showing the equivalent ISO vs measured SNR, or you can pull exactly one stop to get ISOs that can be compared directly).

Or I could report the Canon ISO that the data was obtained from.  Which is what I did, because at the time, it seemed like to most easiest way to describe the results.  Certainly, it left no room for error in the maths.

I enjoy helping out in anyway I can, you just need to remember that I am (maths) stupid.  I like to visualise.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 06:47:45 PM
Canon ISO is simply the amplifier configuration you are starting from. You could as well start from ISO 6400 in Canon menu, override CMOS[0] and ADTG gains to the values from ISO 200, an apply some other tweaks until you get the same highlight detail as with ISO 100.

The end result is ISO 100 and you should compare it with Canon ISO 100.

If you don't do this, you end up saying you get e.g. 0.5 stops more DR at ISO 6400, which is simply not true.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 06:59:16 PM
Quote from: a1ex on February 28, 2014, 06:02:02 PM
(...)
@SpcCb: I'm lost. Where did you get these numbers from? Why ISO 800 has 64x more "energy" than 100? What kind of "energy" is this?

Oh my.. I should not used 'energy' to get a picture.
Well, clear your mind. :)

Let's see this path;

emax / emin represent the delta between the maximum and the minimum.

We have taken as example ISO 100/800, so the range:
100 - 200 - 400 - 800
Witch means a delta between each:
1 - 2 - 4 - 6
-> ISO 200 is 2x ISO 100, ISO 400 is 4x ISO 100, etc.
As we are in a power of 2 world, we have the scaling:
21 - 22 - 24 - 26
Then:
1 - 4 - 16 - 64
Hence:
e3 = 4e2 = 16e1 = 64e0

I can't imagine you can't figure that, this is simple boolean maths. ;)

Quote from: a1ex on February 28, 2014, 06:02:02 PMI use the engineering definition of DR: log2(full well capacity / noise RMS), which is log2(white - black) - log2(dark frame stdev), approximated to log2(white - black) - log2(optical black stdev). How does this "energy" fit here?

Note that you can keep the condensate figure:
DR = log2((white - black) / optical black standard deviation)
We do log2 one time, it's faster.

This is close to the equation what I used here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104208#msg104208) to make calculation for a single frame:
DRsf = log2(( 15760 - 1024 ) / 6.47 )

In this equation, extend DR given by dual_iso appears nowhere.


EDIT: Maybe we should discuss of that in the dual_iso thread?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 07:03:43 PM
Quote from: SpcCb on February 28, 2014, 06:59:16 PM
As we are in a power of 2 world, we have the scaling:
21 - 22 - 24 - 28
Then:
1 - 4 - 16 - 64
Hence:
e3 = 4e2 = 16e1 = 64e0

Why do we have this scaling?

and 28 is not quite 64 ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 07:11:55 PM
Quote from: a1ex on February 28, 2014, 07:03:43 PM
Why do we have this scaling?

and 28 is not quite 64 ;)

Indeed! This is an error; I said I do sometimes... :D
I made the correction.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 07:16:06 PM
Okay. What's the reason you use these scalings?

Minor: where is that corrected 6 coming from?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 07:32:50 PM
Quote from: a1ex on February 28, 2014, 07:16:06 PM
Okay. What's the reason you use these scalings?

Minor: where is that corrected 6 coming from?

An imaging system with a spatially varying exposure (note that I replace energy by exposure, maybe a better word choice) pattern simultaneously measures local scene radiance I using different exposures. In our example, four exposures range are used such that the maximum exposure e3 measures low scene radiance with high fidelity, while the minimum exposure e0 can measure very high radiance values without saturation.
When information from e0 and e3 are used together, a non-linear quantization of scene radiance is obtained: 1 -> 6.
(my last error coming from here, I was thinking linear; sometimes it appends, I switch :) )
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 07:38:56 PM
Why 6 and why 64?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 07:50:11 PM
Quote from: a1ex on February 28, 2014, 07:38:56 PM
Why 6 and why 64?
100 - 200 - 400 - 800,
1 - 2 - 4 - 6,
21 - 22 - 24 - 26
Then:
1 - 4 - 16 - 64
???
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 07:52:16 PM
Why not 2^7 or not 2^3.14 or any other number? And then, why 2^x?

Remember you are talking about ISO 100/800. Where is the 6 coming from?

Arguments like "we are in a power of 2 world" are not accepted.

If anyone else understands this theory, please enlighten me.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on February 28, 2014, 09:05:11 PM
Quote from: Audionut on February 28, 2014, 05:19:11 PM
We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.

However the results are great. Maybe your subject forces the viewers attention towards the banding because the wood on the left side has a structure that is very similar to the banding period. I had a look at the cardbord (former battery package) in 1:1 and could clearly read the whole text - In contrast to the Canon standard photo, where I could barely read the bigger letters.
I think for efficient banding reduction you have to subtract a darkframe. (The question is, how often one has to take a darkframe to compensate for possible changes of the banding)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2014, 09:29:00 PM
The banding is not really correlated from one shot to another (which means a dark frame subtraction will not help much).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-fpn-iso100/fpn-canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-fpn-iso100/fpn-tweak.png)
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-fpn-iso100/fpn-xcov-canon.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-fpn-iso100/fpn-xcov-tweak.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/5d3-fpn-iso100/fpn-xcov-tweakvscanon.png)
- FPN at ISO 100 (from dark frame, averaged lines/columns)
- FPN at ISO 100 pulled from 200
- FPN cross-covariance between 2 dark frames at ISO 100
- FPN cross-covariance between 2 dark frames at ISO 100 pulled from 200
- FPN cross-covariance between 1 dark frame at ISO 100 (X) and 1 dark frame at ISO 100 pulled from 200 (Y)

So, the absolute amount of FPN is lower with the tweaks applied, but it doesn't decrease as much as the random noise => it becomes more noticeable.

The FPN has some autocorrelation at lags multiple of 16; this hint might help with correcting it in post.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on February 28, 2014, 11:57:24 PM
Quote from: a1ex on February 28, 2014, 07:52:16 PM
Why not 2^7 or not 2^3.14 or any other number? And then, why 2^x?

Remember you are talking about ISO 100/800. Where is the 6 coming from?
(...)

So, if I understand you suppose that there's no spatial cumulative energy in the Bayer Matrix from each ISO (100 & 800) so a linear quantization of scene radiance?
And that emax / emin = 8 for dual_iso 100/800?

If it is, we should get in this case (always with same 5D2 with dual_iso 100/800 to compare):
DRdi = log2[(( 15760 - 1024 ) / √{( 8² + 6.47² ) / 2 }) 8 ]
DRdi = 13.984 stops

Looks a low gain, no? ???
Face to:
DRsf = 11.153 stops

Can you make the maths for your 5D3 and run a physical test to confirm it?
If it is, it will be an interesting theory. BTW maybe it is, we have to stay open (I'm always open for that! :) ).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 01, 2014, 12:06:24 AM
I've asked you to define the terms you are using. What is "spatial cumulative energy" and how does it apply here?

(tried searching and found 2 pdf's, a paper that assumes I know what that thing is, and a thesis about kinetic energy spectrum in the upper ocean)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 01, 2014, 12:20:57 AM
Quote from: a1ex on March 01, 2014, 12:06:24 AM
I've asked you to define the terms you are using. What is "spatial cumulative energy" and how does it apply here?

(tried searching and found 2 pdf's, a paper that assumes I know what that thing is, and a thesis about kinetic energy spectrum in the upper ocean)

I mean that we have a SVE image composed by two frames 'interlaced' (I hope this word is correct) in the Bayer Matrix, like the matrix is half cut; in the RGGB pattern the first line RG will be @ISO 100 and the second GB @ISO 800.
So when the demosaicing algorithm will reconstruct the RGB image some of the second line values will be used by the first, first used by the second, etc.
In final, levels should be higher than taking lines discretely.

However, be careful; this is what I read in scientific papers (from Univs, Nikon & Canon) about SVE. I found this relevant but I don't find this myself.
Maybe with dual_iso it is quite different.


Edit: I tried to find papers on the web about this, without great success like you; only some bizarre PDF not in accord. Interesting papers I have was found with the search engine of an University.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 02:00:27 AM
Quote from: a1ex on February 28, 2014, 06:47:45 PM
Canon ISO is simply the amplifier configuration you are starting from. You could as well start from ISO 6400 in Canon menu, override CMOS[0] and ADTG gains to the values from ISO 200, an apply some other tweaks until you get the same highlight detail as with ISO 100.

The end result is ISO 100 and you should compare it with Canon ISO 100.

Personally, I think most people would read it as having been reduced from the Canon ISO.  Stock standard, everyday ISO, set via Canon menu.

Quote from: Audionut on February 28, 2014, 05:19:11 PM
The ISO rating displayed is that in Canon menu, ie: the pulled ISO.

Rather then some convoluted gain configuration that reaches the same value.

However, your point still stands.

Quote from: a1ex on February 28, 2014, 06:47:45 PM
If you don't do this, you end up saying you get e.g. 0.5 stops more DR at ISO 6400, which is simply not true.

I believe they call this, an honest mistake.

Quote from: a1ex on February 28, 2014, 06:02:02 PM
SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).

If I compare these 2 exposures.

ISO 200 - 1/60s - f/8.0 (lens unscrewed) - ADU 13519 - stdev 187.6
ISO 400 - 1/60s - f/8.0 (lens unscrewed) - ADU 7682 - stdev 92.7

We can see that the pure SNR (http://en.wikipedia.org/wiki/Signal-to-noise_ratio_%28imaging%29) is,

13519/187.6 = 72.063
7682/92.7 = 82.869

To me, this shows that we are not entirely shot noise limited.  The same exposure resulted in different noise levels.  Whether both exposures are considered clean or not is irrelevant.  Clearly, even at strong signal levels, there is a measurable difference.  And furthermore, since the 2 exposures were identical, the noise difference is entirely electronic induced.  ie:  Shot noise is not a contributor.

If we compare to the original example I described.

Quote from: Audionut on February 28, 2014, 05:19:11 PM
If I have my maths right, The noise difference between ML ISO 100 and Canon ISO 100 in the highlights.

log2(14169/13255) - log2(158.2/148.7) = 0.0068 EV

We can confirm that the exposures are very similar.  At least, if my maths is correct!

14169/148.7 = 95.286
13255/158.2 = 83.786

Even though the noise levels are different.

If I had a full blown lab setup where I could control light output precisely, to capture the nonlinear behaviour near saturation, then life would be all good.  In the real world, I'm trying to do the best I can, with what I have.  Both physical equipment and mental aptitude.

If my results are far removed from useful, then I am more then happy to stop clicking the shutter count higher and higher.




Quote from: a1ex on February 28, 2014, 09:29:00 PM
So, the absolute amount of FPN is lower with the tweaks applied, but it doesn't decrease as much as the random noise => it becomes more noticeable.


Indeed.  This is an important distinction.  The FPN is not being induced by these gain tweaks.  Simply, other sources of noise are being reduced sufficiently, to make this noise more apparent.

Using the 2 images on the previous page, we can conclude that the ADTG stage is a significant contributor to total noise.  CMOS amplification was increased, ADTG amplification was decreased, total apparent noise also decreased.  We can conclude that the noise is not (significantly) related to components further down the signal chain, because the signal level delivered to these components was very similar.

log2(14169/13255) = 0.096 EV

Here, the nonlinear behaviour in the highlights might effect the noise level in the highlights, but since the rest of the signal is linear, the signal difference between the point I measured, and the shadows, should also be linear.  In fact, if I compare a darker region of the 2 exposures.

log2(2722/2651) = 0.038 EV

This shows that as we move further towards the noise floor, the signal level between exposures becomes closer.  Further emphasising that the ADTG stage is a significant contributor to total noise.


If we compare the noise in the shadows from 2 identical exposures, 1 @ ISO 100 and 1 @ ISO 1600, we know from plenty of past examples that the apparent noise level reduces.

Now, if we compare the noise from 2 different exposures, ML ISO 100 - 1/15s and ML ISO 1600 - 1/250s, where we haven't used ISO to increase the rendered brightness, but instead, have used ISO to brightness match 2 different exposures, the result is this.

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Compare/ML1600.jpg

https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/Compare/ML.jpg

Here we can see 2 things. 

The apparent noise level increased.
The gain configuration for the ADTG was identical.  The signal level delivered to components further down the signal chain was very similar.

log2(14169/13702) = 0.048 EV

So we can conclude that the increased noise was from poisson statistics, and CMOS amplification.

We can also see that the FPN has not been (significantly) reduced.  The increased shot/CMOS noise has reduced the apparent effect of the FPN, but the FPN level is very similar.  I can attempt to use multiple exposures to average out the random noise, leaving the FPN component intact, however, if the FPN noise is not "really correlated from one shot to another", then the level of FPN will also be reduced.

Since the FPN is reduced in identical exposures with CMOS gain, where the signal through the ADTG stage is higher, and the level of FPN does not appear to differ (significantly), with reduced photons on sensor, or, directly with CMOS gain (ie: CMOS doesn't reduce the noise, it simply produces a knock on effect), it is my conclusion that the FPN noise is directly related to the ADTG stage.

This FPN noise may be a result of the scales of the individual ADTG gains.  Or it may simply be a fixed component of the ADTG stage at it's noise floor.
Since we reduced the total noise of the ADTG stage with register adjustments, the FPN component of the ADTG stage is now more apparent.

https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
QuoteThis gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.

If the FPN component can be reduced with register adjustments also, it may be beneficial to sacrifice total noise reduction (my DR is better then your DR), in order to reduce the FPN component.  In other words, we reduce (the engineering definition of) DR, but we increase the visual aspect of the output images.

This is what I was trying to explain in the dual_iso thread.  Engineering DR has it's uses, but it is useless for describing what is happening between (white) and (stdev).

My first guess (for reducing FPN), would be increasing the gain of a single component in the ADTG stage, ie: 888x or 0xFE or 8/9/A/B, and observing the effects.  Then expanding on that by changing the gain of the individual registers in each ADTG gain stage.

Quote from: a1ex on January 10, 2014, 12:11:01 PM
Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)

;) :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 02:07:31 AM
Quote from: SpcCb on March 01, 2014, 12:20:57 AM
I mean that we have a SVE image composed by two frames 'interlaced' (I hope this word is correct) in the Bayer Matrix, like the matrix is half cut; in the RGGB pattern the first line RG will be @ISO 100 and the second GB @ISO 800.

The RGGB pattern is scanned intact with dual ISO.  That is to say, 1 ISO scans an entire RGGB pattern, and the other ISO scans an entire RGGB pattern.

So the interlaced fields are not a single pixel width, but dual pixel width to capture the entire RGGB pattern.  a1ex makes specific mention of this in his paper on dual_iso.  Page 5 (http://acoutts.com/a1ex/dual_iso.pdf).

Quote from: naturalsound on February 28, 2014, 09:05:11 PM
I had a look at the cardbord (former battery package) in 1:1 and could clearly read the whole text - In contrast to the Canon standard photo, where I could barely read the bigger letters.

Yes, the reason why I choose low contrast text for the shadows.  :)

This area of the image averages -10 EV in the red channel, -9.5 EV in the green channels, and -11 EV in the blue channel, referenced to ISO 100 - 1/15s.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 01, 2014, 06:18:22 AM
Quote from: Audionut on March 01, 2014, 02:07:31 AM
The RGGB pattern is scanned intact with dual ISO.  That is to say, 1 ISO scans an entire RGGB pattern, and the other ISO scans an entire RGGB pattern.

So the interlaced fields are not a single pixel width, but dual pixel width to capture the entire RGGB pattern.  a1ex makes specific mention of this in his paper on dual_iso.  Page 5 (http://acoutts.com/a1ex/dual_iso.pdf).

(...)
Haa, OK. I thought it was interlaced every lines. Thank you Audionut to see that. ;)
So for the emax / emin gain, maths I used have to be adapted to this. As I said very first, I presumed that it will not be relevant here (I used 1/4px of the matrix).
Then maybe the emax / emin follow the ISOmax / ISOmin linearly with a minimum gap.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 01, 2014, 08:31:31 AM
QuoteIndeed.  This is an important distinction.  The FPN is not being induced by these gain tweaks.  Simply, other sources of noise are being reduced sufficiently, to make this noise more apparent.
Exactly.

And, since it's not correlated from picture to picture, stacking a few pictures (with averaging) will reduce it (because it is very close to a Gaussian random noise applied to each column), but a single dark frame will not reduce it (and might even increase it).

In LiveView, this FPN is corrected on the fly (you can see something like a moving average algorithm that recalibrates: toggle "ISO registers" on and off in movie mode, with LiveView visible behind ML menu, and you'll notice the effect in the first few seconds).

In photo mode, there is probably a similar auto-tuning. Might be worth trying some sensor cleaning routines?

(edit: tried, but didn't help; FPN stdev stays the same)

Quote
Personally, I think most people would read it as having been reduced from the Canon ISO.  Stock standard, everyday ISO, set via Canon menu.
We are not yet talking about the user interface of this thing (we may as well map ISO 66 -> ISO 50 in Canon menu, ISO 100 -> ISO 100 in Canon menu and so on, but we are not there yet). Right now, at this research stage, the ISO in Canon menu is simply a descriptor for the amplifier configuration (consider it on the same level as any other register value). That is, set Canon ISO to 200, set register X to A, set register Y to B, and the result is ISO 100.

To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.

QuoteCan you make the maths for your 5D3 and run a physical test to confirm it?
Any processing log from cr2hdr is such an experiment: it measures noise levels and DR for the two input exposures from OB area, estimates te output DR based on ISO difference, and at the end it measures the noise level in OB area after all the processing. However, this has two problems:
- the output image also uses some small noise reduction (well, chroma smoothing) => that's why I call it "cooked"
- the noise pattern after processing is no longer a plain old uncorrelated Gaussian

To consider the DR loss caused by the half-resolution in shadows, it may be interesting to downsample the image by 2x2 before comparing the noise levels.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 09:36:03 AM
Quote from: a1ex on March 01, 2014, 08:31:31 AM
Right now, at this research stage, the ISO in Canon menu is simply a descriptor for the amplifier configuration (consider it on the same level as any other register value). That is, set Canon ISO to 200, set register X to A, set register Y to B, and the result is ISO 100.

To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.

Agreed. 
I have a tenancy to call the same thing by multiple names also, which obviously doesn't help.  ::)

The next time I have some free time, I will attempt to remove the variable component of the pattern noise (post process), and see if I can use the registers to remove the fixed component.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 01, 2014, 09:47:26 AM
On raw_diag's FPN display it may be worth printing the OB or dark frame stdev too (on the same screen).

So, besides maximizing DR, you may want to minimize the ratio between FPN stdev and overall noise stdev. This ratio would show effectively how much of our noise is FPN.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 10:01:01 AM
For further analysis yes. 

Initially, I think it will be easiest to start at Canon settings, and concentrate on minimum FPN.

Once (if) I have a confirmed list of settings that minimise FPN, I'll leave it for you to correlate the data and draw the conclusions.  You're accuracy trumps mine!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 01, 2014, 04:58:02 PM
Quote from: a1ex on March 01, 2014, 08:31:31 AM
(...)
To avoid confusion, I think you can say "ISO 100 pulled from 200". From this, I understand ISO 200 in Canon menu and gains configured in such a way that you recover 1 stop of highlight detail.
Any processing log from cr2hdr is such an experiment: it measures noise levels and DR for the two input exposures from OB area, estimates te output DR based on ISO difference, and at the end it measures the noise level in OB area after all the processing. However, this has two problems:
- the output image also uses some small noise reduction (well, chroma smoothing) => that's why I call it "cooked"
- the noise pattern after processing is no longer a plain old uncorrelated Gaussian

To consider the DR loss caused by the half-resolution in shadows, it may be interesting to downsample the image by 2x2 before comparing the noise levels.
I figured that chroma smoothing can be deactivated by cr2hdr command line? Maybe you are speaking about an other chroma smoothing in the process?
I agree for the 2x2 down-sampling, it should help to compare signal levels.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 05:00:47 PM
These details are incorrect.  See correct analysis on the next page.

ADTG2[fe] - This affects RBG channels.  What Rawdigger calls Green 2.  This scales all channels equally, except for a smaller effect on one of the green channels.

ADTG4[fe] - This affects RGB channels.  This scales all channels equally, with a smaller effect on the other green channel.

ADTG2[8882]  - Blue channel - pixels 5,6,7
ADTG2[8884]  - Blue channel - pixels 1,2,3
ADTG2[8886]  - Blue channel - pixels 7,8,9
ADTG2[8888]  - Blue channel - pixels 3,4,5

This register predominately effects the blue channel and one of the green channels.

The pixel values listed, show the pixel count from OB on left side of image.  That is to say, ADTG2[8882]  - Blue channel - pixels 5,6,7, predominately effects the 6th pixel from OB, with a smaller effect on the 5th and 7th pixels.  This pattern repeats every 8th pixel.  So the same effect happens at the 14th, 15th and 16th pixels from OB.

Also, vertically, the strength varies every other pixel.

edit:  I am describing the effect in the RGB render.  Consider the raw composites on the next page.

Consider this image.
(https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/blue.PNG)

Each of the registers also seems to vary with strength.  Although I scaled all registers by the bolded digit 0x419.  So the varying apparent strength of the register, is probably related to the differences between the stock values.

The same pattern is repeated with registers ADTG2[8,9,A,B].  However, the gain reduction is less.

The same occurs at ADTG4[888?] and ADTG4[8,9,A,B].  However, these registers predominately effect the red channel, with a smaller effect on the green channels.

Also, the pixels effected, follow a slightly different pattern.

ADTG4[8880] - Red channel - 6, 7, 8
ADTG4[8884] - Red channel - 2, 3, 4
ADTG4[8886] - Red channel - 4, 5 ,6
ADTG4[8888] - Red channel - 1, 2

(https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/red.PNG)


raw composites on next page.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 01, 2014, 05:08:47 PM
wouldnt it be better visible in raw composite mode?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 01, 2014, 05:12:17 PM
Yes.

The raw composites of this image.
(https://s15.postimg.cc/fld8evocr/blue.png)

(https://s15.postimg.cc/8i5czadsb/red1.png) (https://s15.postimg.cc/yqghonq63/green1.png) (https://s15.postimg.cc/k7eejs36j/blue1.png) (https://s15.postimg.cc/97o5bmywb/green2.png) (https://s15.postimg.cc/c1wclmmnf/Composite.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on March 01, 2014, 06:38:11 PM
500D with ADTG on live view also has a constant vertical banding.
Maybe 500D need new fabric calibration  :-\

(http://s11.postimg.org/9q3h8udtf/500d.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 02, 2014, 03:33:19 AM
Some further analysis following the bayer pattern (http://acoutts.com/a1ex/dual_iso.pdf).

ab cd ef gh
0 RG RG RG RG
1 gb gb gb gb
2 rg rg rg rg
3 GB GB GB GB
4 RG RG RG RG
5 gb gb gb gb
6 rg rg rg rg
7 GB GB GB GB
8 RG RG RG RG


Column a - row 0 starts at pixel position 122 from left, and 80 from top.
Optical black finishes at 121 pixels from left, and 79 pixels from top.  However, pixel row 79 contains some amount of signal.

Column a is ADTG4[8888]
Column b is ADTG2[8884]
Column c is ADTG4[8884]
Column d is ADTG2[8888]
Column e is ADTG4[8886]
Column f is ADTG2[8882]
Column g is ADTG4[8882]
Column h is ADTG2[8886]

8882 looks to only effect a single column.  8884/8886/8888 look to have a predominate effect on the listed column, and a smaller effect on all other columns.

Further details to follow.

ADTG2/4[8,9,A,B] is also more complex.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 02, 2014, 04:54:36 AM
Maybe it can help: OB zone contains multiple Reference Areas.

Here is crops, top-left, at zoom 4x with color marks.
White mark corresponding to the standard output image.
Orange responding like the standard output image but does not get light.

5D2:
(http://www.bgimage.fr/publis_ext/ml/5d2_ob_ref-zones.png)

5D3:
(http://www.bgimage.fr/publis_ext/ml/5d3_ob_ref-zones.png)

Note: It needs to be confirmed, however OBRA of the 5D2 should be similar on the 50D and 550D (maybe other from the same generation). And OBRA of the 5D3 should be similar on the 6D.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 08:42:45 AM
These zones are definitely helpful (I need this data for cr2hdr too). Thinking to add a display mode in raw_diag to show exactly this cropped view, and some statistics for each zone.

Anyone can help me counting the pixels of each zone and write down their sizes?

888x: didn't check offline, but if you try to bump each one in LiveView, you get a brighter vertical line in the Magic Zoom window (and all of them behave in the same way). I believe the order is ADTG2[8882], ADTG4[8882], ADTG2[8884], ADTG4[8884], ADTG2[8886], ADTG4[8886], ADTG2[8888], ADTG4[8888].

At least, if you bump the first 2, you get a thicker vertical, and if you bump the first 4, you get 50% dark and 50% bright stripes.

When checking these gains, it can be useful to turn off demosaicing. For offline analysis, I prefer dcraw -4 -E folllowed by octave.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 02, 2014, 09:36:10 AM
Quote from: a1ex on March 02, 2014, 08:42:45 AM
Anyone can help me counting the pixels of each zone and write down their sizes?

I'm not entirely sure what you mean by "zone".

Would it be best to sample the first 8 columns of the image?

Some other useful things, WL for each color channel.
Also, histogram of the top stop, or top two stops of exposure.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 02, 2014, 09:40:44 AM
according to analog devices' manuals for CCD ADTGs, they have a more or less fixed use case.
the CCD readout happens in a serial way - you scan pixel for pixel in every line from top to bottom.

top optical black lines:
usually more than 10 lines to calibrate black level before sensor readout happens.
result is used for optical black clamping (= moving analog "zero" to a specific voltage)

left optical black pixels:
as they come right before the pixels in the current line, they are used to update current line black level for this line.

so now a question raises...
if we use the output of the AD converter which is in those OB areas the output of a closed loop filter,
can we be sure to get something useful from it?
i say "no, not without paying attention and knowing what we do"

why?
check the datasheet of e.-g. AD9992 (http://www.autex.spb.ru/ad/AD9992.pdf) from ADI, which is the manufacturer canon uses for their customized ADCs/ADTGs

there you see that the analog value from the CCD goes into a correlated double sampling path which will measure
sensel's "empty" and "exposed" values and gives out the delta for further processing.
this value then is added to a variable voltage to push "black" a bit higher that 0V. (=black clamp)
thats the black level we know from raw, just in analog domain.
this variable voltage is estimated by averaging OB areas and adjusting it to keep black level at some target value.

in OB digital data we get not just the charge of the sensels, but also the correction value that updates.
as closed loop filters usually oscillate and converge to the target value slowly, the first few DIGITAL OB lines would be trash and only reflect the filter's learning phase.
same for the first few OB pixels in every line.

conclusion:
throw away the first n lines/pixels of OB areas.
how much is n? i dont know, it depends on the loop parameters in the chip.
ADI recommends 20 OB pixels and 10 OB lines so i would say:
-> skip top 10 lines and the leftmost 40 pixels
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 02, 2014, 09:47:29 AM
additional note:
check the topmost lines in raw data of above images. it shows the filter learning its black clamp :)
randomly oscillating line brightness, so clamp is updated per-line.
seems to match my theories.

can someone average all pixels in the OB lines? (all pixels, no matter which color)
these values plotted should show a converging oscillation.

(i have to watch cartoons right atm....)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 01:05:19 PM
Quote from: g3gg0 on March 02, 2014, 09:47:29 AM
can someone average all pixels in the OB lines? (all pixels, no matter which color)
these values plotted should show a converging oscillation.
Dark frames at ISO 100 and 100/1600, 1/30, without any other tweaks applied (I had them saved from the first experiments on dual ISO).

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob/top_ob_100.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob/top_ob_100_1600.png)

Octave script:

function top_ob(im, name, outfile)
top = im(1:80,:);
top_ex = im(1:120,:);
figure
plot(mean(top_ex'), 'r'), hold on
plot(mean(top'))
legend('Active area', 'Top OB', 'location', 'southeast')
title(name)
xlabel('Line number')
ylabel('Raw average (per-line)')
set(gca, 'position', [0.18 0.18 0.7 0.7])
grid on
print("-dpng", "-r70", outfile)
end

octave:31> top_ob(im100_1600, "ISO 100/1600", "top_ob_100_1600.png")
octave:32> top_ob(im100, "ISO 100", "top_ob_100.png")


To load the raw data, you need to run "dcraw -4 -E foobar.cr2" first, then load the pgm in octave with imread.

Also note that with dual ISO, in the OB area, the higher ISO always ends up with the lower OB values (more graphs in the dual ISO PDF). I used this trick to autodetect which lines are dark and which are bright in the dual ISO preview experiment (not in cr2hdr); it worked very well, except for 100/200 where the difference was too weak and the algorithm got confused by noise. This behavior should give a hint about how the feedback loop works.

Quote
I'm not entirely sure what you mean by "zone".
Those rectangles highlighted by SpcCb. Inside one zone, the noise pattern is the same, but each zone has its own noise pattern (see g3gg0's theory about different clamping modes).

Quote
(i have to watch cartoons right atm....)
Good idea :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 01:57:28 PM
Some interesting autocorrelations regarding vertical FPN (5D3, ISO 100 pulled from 200):


function plot_xcov(x)
x = xcov(x, 49, 'coeff');
stem(x)
end

# average each column
octave:2> b = mean(im);

# look for autocorrelations
octave:11> plot_xcov(b), print -dpng -r70 0.png
octave:12> plot_xcov(b(1:8:end)), print -dpng -r70 1.png
octave:13> plot_xcov(b(2:8:end)), print -dpng -r70 2.png
octave:14> plot_xcov(b(3:8:end)), print -dpng -r70 3.png
octave:15> plot_xcov(b(4:8:end)), print -dpng -r70 4.png
octave:16> plot_xcov(b(5:8:end)), print -dpng -r70 5.png
octave:17> plot_xcov(b(6:8:end)), print -dpng -r70 6.png
octave:18> plot_xcov(b(7:8:end)), print -dpng -r70 7.png
octave:19> plot_xcov(b(8:8:end)), print -dpng -r70 8.png

# standard deviation of the vertical FPN
octave:22> std(b)
ans =  0.73383
octave:23> std(b(1:8:end))
ans =  0.75365
octave:24> std(b(2:8:end))
ans =  0.75451
octave:25> std(b(3:8:end))
ans =  0.63608
octave:26> std(b(4:8:end))
ans =  0.65589
octave:27> std(b(5:8:end))
ans =  0.73094
octave:28> std(b(6:8:end))
ans =  0.79068
octave:29> std(b(7:8:end))
ans =  0.75369
octave:30> std(b(8:8:end))
ans =  0.69111

# standard deviation of the entire dark frame
octave:35> std(im(:))
ans =  3.1371


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/1.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/3.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/5.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/6.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/7.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/8.png)

First graph: notice the autocorrelations at lags multiple of 16.
Subsequent graphs: notice the autocorrelations in columns 0, 4, 5 and 6 (modulo 8 ).
Also notice the lower stdev in columns 2, 3 and 7 (modulo 8 ). Not sure if relevant.

Why modulo 8? because there seem to be 8 parallel processing channels (there are 8 gains in the 888x set) and each of these gains affects one column in a group of 8.

You draw the conclusions.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 02, 2014, 03:20:11 PM
My first guess would be that 8882 are masters of some description, but that doesn't explain columns 5 and 6.

Can you compare with Canon default?  Or with one of mine (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7467.CR2)

Canon pulled ISO 200
ADTG gain -16
[fe] - 3
[8,9,A,B] - 2

Mine also contains the faint banding type noise.  Does iso_regs scale [8,9,A,B], keeping the deltas between them?  If not, then I guess that is the reason why I got the banding noise.  Else, a value of 2 is obviously to low for my 5D3.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on March 02, 2014, 04:51:02 PM
This register seems to associated with constant banding.
500D C0f0[4908]
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on March 02, 2014, 05:04:35 PM
Quote from: g3gg0 on March 02, 2014, 09:40:44 AM
left optical black pixels:
as they come right before the pixels in the current line, they are used to update current line black level for this line.
...
in OB digital data we get not just the charge of the sensels, but also the correction value that updates.
as closed loop filters usually oscillate and converge to the target value slowly, the first few DIGITAL OB lines would be trash and only reflect the filter's learning phase.
same for the first few OB pixels in every line.

Here are some plots that I posted much earlier in this thread here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97510#msg97510), with accompanying explanation here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97529#msg97529), that show the effectiveness of the per-line black-level clamp for a normal exposure and a massive overexposure in the 50D.  Each curve on the plot is the histogram of a pair of columns in the left optical-black region, with the top curve adjacent to the actual image region.

(https://dl.dropboxusercontent.com/s/fld1glcryfcxkki/variation_0250.png)(https://dl.dropboxusercontent.com/s/s9y77vl6h1roluk/variation_0247.png)

You can see both oscillation and convergence, even for the normally exposed image.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 02, 2014, 05:56:33 PM
Quote from: a1ex on March 02, 2014, 08:42:45 AM
These zones are definitely helpful (I need this data for cr2hdr too). Thinking to add a display mode in raw_diag to show exactly this cropped view, and some statistics for each zone.

Anyone can help me counting the pixels of each zone and write down their sizes?

(...)

For 5D2:
blue: 0;4 5792;26
green: 0;31 158;33
yellow: 158;31 5792;32
red: 0;34 158;37
cyan: 0;37 78;3804
orange: 78;37 158;3804


For 5D3:
blue: 0;5 5918;31
green: 0;32 122;41
yellow: 122;32 5918;41
red: 0;42 122;47
cyan: 0;47 122;56
orange: 0;56 122;3950


Edit: Add 5D3, but for other cameras I don't have RAW files.
Note: Coordinates origin is top-left(0;0).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 05:56:58 PM
Quote from: Audionut on March 02, 2014, 03:20:11 PM
Does iso_regs scale [8,9,A,B], keeping the deltas between them?

Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far.

However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

QuoteCan you compare with Canon default?
With my ISO 100 dark frame from early dual ISO experiments:

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/1.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/2.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/3.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/4.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/5.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/6.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/7.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn100/8.png)

Didn't expect this change; need to check a few more dark frames to be sure.

QuoteOr with one of mine (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7467.CR2)
The autocorrelation analysis was performed on a dark frame, not on a regular image.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 02, 2014, 06:05:30 PM
Quote from: a1ex on March 02, 2014, 05:56:58 PM
Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far.

However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

In this case, I should be safe with a value of 2, since the minimum value for the other registers, is within 2 of the first one.
Also, I was scaling 888x with mini-iso.

I will search for the cause of the problem.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 02, 2014, 06:28:20 PM
Quote from: a1ex on March 02, 2014, 05:56:58 PM
Didn't expect this change; need to check a few more dark frames to be sure.

Canon ISO 100 (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7706.CR2)

Canon pulled ISO 200 (https://dl.dropboxusercontent.com/u/34113196/ML/ADTG/_46A7709.CR2)
ADTG gain -16
[fe]  3
[8,9,A,B]  4 (to be safe)


Quote from: a1ex on March 02, 2014, 05:56:58 PM
The autocorrelation analysis was performed on a dark frame, not on a regular image.

Sorry, I thought OB would be sufficient.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on March 02, 2014, 08:28:05 PM
Quote from: a1ex on March 02, 2014, 08:42:45 AM
Anyone can help me counting the pixels of each zone and write down their sizes?

Quote from: SpcCb on March 02, 2014, 05:56:33 PM
For 5D2:
blue: 0;4 5792;26
green: 0;31 158;33
yellow: 158;31 5792;32
red: 0;34 158;37
cyan: 0;37 78;3804
orange: 78;37 158;3804


For 5D3:
blue: 0;5 5918;31
green: 0;32 122;41
yellow: 122;32 5918;41
red: 0;42 122;47
cyan: 0;47 122;56
orange: 0;56 122;3950


Edit: Add 5D3, but for other cameras I don't have RAW files.
Note: Coordinates origin is top-left(0;0).

Don't forget that there can be indexing differences between the CR2 files and the buffer that ML uses (that can be dumped with RAW_DEBUG_DUMP).  For example, ML intentionally skips the buffer ahead one line for both the 5D2 and the 50D to put the red pixel on even-parity lines, but there could be other discrepancies too.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 08:47:40 PM
The DNG buffer can be also dumped with raw_diag (no need to edit the source code; it's in the menu).

I'd say it's a good idea to align the internal DNG buffer to the output CR2 (+/- 1 line/column for forcing the active area starting with a red pixel); should make it easier if one wants to repeat some OB analysis with PC-based tools and compare it to raw_diag.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 02, 2014, 10:12:42 PM
Quote from: ayshih on March 02, 2014, 08:28:05 PM
Don't forget that there can be indexing differences between the CR2 files and the buffer that ML uses (that can be dumped with RAW_DEBUG_DUMP).  For example, ML intentionally skips the buffer ahead one line for both the 5D2 and the 50D to put the red pixel on even-parity lines, but there could be other discrepancies too.
Of course area coordinates I provided are from full original Canon CR2. ML features like dual_iso or mini_iso was not used.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 02, 2014, 10:33:37 PM
Updated raw_diag to show OB areas on all 4 sides, like this (no zones defined yet):

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob/ob-zones-5d3-6400.png)

Interesting thing: the in-camera raw buffer is 16 pixels larger than the CR2 one (as reported by dcraw, exiftool and Adobe DNG), so there seems to be a small part of OB which is not saved (or at least I couldn't find it in the CR2). Note the right margin has 18 pixels, and to get a tight fit I had to use skip_left = 122 (dcraw uses 124), so my best guess is that 2 pixels from the right OB will get shifted back to the left side before saving the CR2, and the remaining 16 are discarded.

Now I'd like to ask you to help with something really easy:
1) grab raw_diag.mo from first post, enable "Optical Black zones", take a picture and post the screenshot (you'll find it under raw_diag directory on your card).
2) if you have some coding skills (or, to be more precise, text editing and command-line skills), fine-tune the photo offsets for your camera in raw.c, like I did right now for 5D3 (https://bitbucket.org/hudson/magic-lantern/commits/aa362087a397); the active area in raw_diag should look (more or less) like in my screenshot.

Exact matching with CR2 is hard because a struct raw_pixblock should be aligned in memory at 16 bits, which means the raw buffer can be shifted horizontally only in multiples of 16 pixels. So, don't strive for that (approximate match is OK).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 03, 2014, 12:49:52 AM
Quote from: a1ex on March 02, 2014, 10:33:37 PM
(...)

Now I'd like to ask you to help with something really easy:
1) grab raw_diag.mo from first post, enable "Optical Black zones", take a picture and post the screenshot (you'll find it under raw_diag directory on your card).
2) if you have some coding skills (or, to be more precise, text editing and command-line skills), fine-tune the photo offsets for your camera in raw.c, like I did right now for 5D3 (https://bitbucket.org/hudson/magic-lantern/commits/aa362087a397); the active area in raw_diag should look (more or less) like in my screenshot.

(...)

Here is for 5D2:
(http://www.bgimage.fr/publis_ext/ml/5d2_ob-zones_raw_diag.png)


Note: I had a problem with the last raw_diag.mo, get a error with previous ML build (2014-02-04_ad):
Linking..
tcc: error: undefined symbol 'camera_model_short'
[E] failed to link modules

But it works with the last nighty (2014-03-03).
Beside mini_iso (2014-01-31) no longer works with these last builds, I get the same error.
Looks like it's since 'camera_model_short' became private (?).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on March 03, 2014, 07:16:52 AM
Here's the optical-black screenshot for the 50D:
(https://dl.dropboxusercontent.com/s/n1psoa9p0i7nmz8/ob-zones_50D.png)

I've double-checked the skip offsets, and skip_top and skip_left are as small as they can be before a gap appears in the raw zebras.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 03, 2014, 08:37:28 AM
@ayshih: it's fine; there may be roundoff errors from raw_get_gray_pixel or the code that uses it. You probably tried to decrease skip_top by 2 and got a top bar in zebras => just leave it like this.

@SpcCb: that colorful palette is likely a bug in the screenshot code (another reason to rewrite it) or in the file copying code. Reproduced it.

Regarding linking errors:

camera_model_short became private because most modules were checking only the camera model name, but not the firmware version. This could cause trouble if somebody loads a module expecting a different firmware version (the upgrades in progress are 5D3 113->123, 7D 203->205 and 700D 111->113). So, to force all modules do proper checking, I made the old way obsolete.

mini_iso is undergoing major rewrite (one of the reasons it's not yet published, besides the research part being in progress and new findings being made). You can apply all these tweaks via ADTG GUI (but with more clicks) and from iso_regs (but that one is 5D3-only) and I want the new module to just work rather than having to fiddle with it (it will be no longer a research tool). I'll refactor 70% of it as a reusable library for patching things around in Canon code/data areas, and publish this one first.

Once I'll publish mini_iso, I expect people trying to port it on the other cameras. I want this to happen after the theory is fully confirmed, documented and working at least on two cameras from different generations. Reasons:
- to make sure the implementation is indeed portable (unlike iso_regs which is hardcoded for 5D3)
- to have some instructions for porting (there are much more addresses and constants to be found, compared to say dual_iso, and some of them can't be found by pattern matching without understanding how things work)
- I don't want to throw away the porting work every time the theory gets changed.

The current research tools from first post should work on current nightly (binaries are up to date).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 03, 2014, 09:34:35 AM
Quote from: a1ex on March 03, 2014, 08:37:28 AM
and I want the new module to just work rather than having to fiddle with it (it will be no longer a research tool).

Depending on the complexity with different camera models, a simple on/off, sounds like a good idea.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 03, 2014, 09:40:04 AM
thanks for the plots.
it really looks like i expected it to be. values in the first OB line are either "overexposed" or "underexposed", the whole line too bright or dark.
thats because before reading out the sensor, the black clamp is at a not-perfect value.

then the first line is sampled. all values that go into the ADC get the black clamp voltage added.
as it is too high or low, the output will be too dark or bright.
the ADC will adjust the black clamp voltage (from an DAC) (*) to a value that makes the black region looking the best.
(we dont know the real target values yet)

this process happens in the first few lines, thus they are crap. totally white or black or just b/w noise.
lets call this the OB calibration.

for every line now the ADC uses the OB pixels in front of the real image to re-adjust the OB clamp to match the target value.
there should only be a small drift (i guess only V_adcref swings should need compensation)

so with dual_iso (if i understood alex correct) the ADC will have a lot of to compensate. it might or might not be finished with that adjustment when image data in this line starts.

@alex:
you wrote that with high iso difference the value of the OB areas to the left varies much more.
did you check the variation of the leftmost OB pixels compared to the rightmost OB pixels (on the left side of the image) in dual_iso images?
the rightmost OB pixels in front of the image data could give you a good hint how good the black level was compensated.
all lines should have about the same value there. (*) ...if the values are updated after every pixel though...

if these theories are correct then you might confuse the ADC with dual_iso as it doesnt have enough time to compensate OB.
this maybe just happens under extreme conditons, like a hot sensor or bad battery.
still i think we should know about this fact.

(*)=what i still dont know is if the black clamp is updated after every pixel or after the OB area ends (every line in top OB and on the left OB area when its over)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 03, 2014, 01:04:17 PM
Yes, so the top few lines are the raw (uncorrected) output from the amplifiers. Notice how it repeats every 8 columns.

This number being small, a possible hypothesis is that it's simply not enough to correct the banding because the noise could not be averaged well enough (so finding the register that increases the number of these lines should help). To check this hypothesis, I'd like to see such a screenshot from 6D (which doesn't suffer from FPN).

Regarding dual ISO, here's a screenshot that shows both OB zones (100/6400):
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob/ob-zones-5d3-100-6400.png)

A while ago you found a register named "line count that gets darker" (ADTG2[82F3]). For some reason I've tagged it as NRZI, though I didn't double-check it. So, now I've changed this register from 0x30 NRZI (0x20 decoded) to 0x50 NRZI (0x60 decoded) and the result is:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob/ob-zones-5d3-100-6400-82F3-0x60.png)

(that is, the OB area is now larger by ~70 pixels)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 03, 2014, 01:58:24 PM
From the pdf g3gg0 linked last page.

QuoteOptical Black Clamp

The optical black clamp loop is used to remove residual offsets in the signal chain and to track low frequency variations in the CCD's black level. During the optical black (shielded) pixel interval on each line, the ADC output is compared with a fixed black level reference, selected by the user in the clamp level register. The value can be programmed between 0 LSB and 255 LSB in 1023 steps. The resulting error signal is filtered to reduce noise; the correction value is applied to the ADC input through a DAC. Normally, the optical black clamp loop is turned on once per horizontal line, but this loop can be updated more slowly to suit a particular application. If external digital clamping is used during postprocessing, the AD9992 optical black clamping can be disabled using Bit D2 in the AFE Register Address 0x00. When the loop is disabled, the clamp level register can still be used to provide fixed offset adjustment.

The CLPOB pulse should be aligned with the CCD's optical black pixels. It is recommended that the CLPOB pulse duration be at least 20 pixels wide. Shorter pulse widths can be used, but the ability for the loop to track low frequency variations in the black level will be reduced. See the Horizontal Clamping and Blankingsection for timing examples. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 03, 2014, 05:28:22 PM
Quote from: a1ex on March 03, 2014, 08:37:28 AM
(...)
@SpcCb: that colorful palette is likely a bug in the screenshot code (another reason to rewrite it) or in the file copying code. Reproduced it.

When I saw that I retried a couple of times, but I thought it was not a problem because we want to check zone sizes.

Quote from: a1ex on March 03, 2014, 08:37:28 AM
Regarding linking errors:

camera_model_short became private because most modules were checking only the camera model name, but not the firmware version. This could cause trouble if somebody loads a module expecting a different firmware version (the upgrades in progress are 5D3 113->123, 7D 203->205 and 700D 111->113). So, to force all modules do proper checking, I made the old way obsolete.

(...)

I understood, it is not adequate to only check the camera model, firmware version also need to be check.

So to port mini_iso tool on last nightly builds I thought to update:
if (streq(camera_model_short, "target_model"))
with:
if (IS_CAMERA("target_model", "target_firmware"))
Should work like that isn't?

I don't use ATDG_GUI actually; I tried to compile ML with adequate config to run it but for the moment I only get wrong compilation and my camera crashes all times on load, I don't see why for the moment and it's a bit freaky to crash camera every times. :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 03, 2014, 05:38:42 PM
Again: do not rush, be patient.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on March 03, 2014, 05:49:26 PM
Quote from: a1ex on March 03, 2014, 08:37:28 AM
camera_model_short became private because most modules were checking only the camera model name, but not the firmware version.

To "fix" it and continue using the legacy binary, simply un-private the symbol in your local build - you can then be more patient to wait for the result of the current in-depth investigation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 03, 2014, 06:35:26 PM
Quote from: a1ex on March 03, 2014, 01:04:17 PM
Yes, so the top few lines are the raw (uncorrected) output from the amplifiers. Notice how it repeats every 8 columns.

hmm not sure. might even be corrected, but with the wrong black clamp value.
dont remember that the datasheets said to output dummy values during OB areas. have to doublecheck.
i thought these are the *corrected* values, but with a *wrong* black level untuil the internal calibration has no need to update black clamp anymore.
might be updated after every pixel, might also be updated after every line (when the OB state input is toggled) dont know.

the images you posted show in the top OB area how the b/w noise gets a gray noise smoothly. (3rd pixel line?)

Quote from: a1ex on March 03, 2014, 01:04:17 PM
(that is, the OB area is now larger by ~70 pixels)

hmmm 0x40 (=64) lines more?
after the original OB area, the brightness goes down a bit?

during OB area we see the same pattern that causes vertical banding on most models?
ever 8 pixels a bit brighter/darker? i see some connection between the two things...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 03, 2014, 06:40:04 PM
The top OB has some correlation with the vertical FPN, but it's fairly weak. I could use that info to reduce its stdev from ~0.72 to ~0.5, but didn't check with a validation data set.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 03, 2014, 06:41:21 PM
weak = not the same amount, but the same pattern?
then its an offset ;)

edit: i mean the same offset that was scaled in these pixels due to black level being wrong.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 03, 2014, 06:47:28 PM
weak =

b = mean(im(80:end,:));
top = mean(im(1:80,:));
t1 = mean(im(5:40,:));
t2 = mean(im(45:80,:));

xcov(top,b) => 0.31
xcov(t1,b) => 0.35
xcov(t2,b) => 0.64


Will add some graphs later.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 03, 2014, 06:53:18 PM
Quote from: a1ex on March 03, 2014, 05:38:42 PM
Again: do not rush, be patient.

Don't worry, I'm very.
Au contraire, I find that ML evolutes too fast! (many times I oar to assimilate new possibility :D )
But here it is to get a tool to help: Since last month I run a mod min_iso to observe impact of TV/ISO/ADTG/etc. settings on OBRA.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 03, 2014, 08:04:09 PM
what if:
we have two/four ADC converters with 8 channels each.
every ADC converter has 8 channels, just like the 4 channel one here (http://www.analog.com/en/audiovideo-products/cameracamcorder-analog-front-ends/addi7004/products/product.html)

4x ADTG with 8 channels each: explains the 8 pixel pattern.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 03, 2014, 09:42:37 PM
Do you think this architecture concerns 5D3 only or _some_ other cameras too?
Thought is if we could drive CDS it could help to reduce FPN.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 03, 2014, 10:22:12 PM
In addition to this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104514#msg104514), maybe it also could help:

(http://www.bgimage.fr/publis_ext/ml/cmos_typical-ob-zones.png)

Area delimitations are not accurate because it depend of the architecture, but this is the typical setup for CMOS, specially those made for high speed video applications.
The setup looks to what we get on our cameras, apparently.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on March 03, 2014, 10:43:07 PM
500D

(http://s24.postimg.org/b18agnscl/ob_zones.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 04, 2014, 09:31:50 AM
Quote from: SpcCb on March 03, 2014, 10:22:12 PM
In addition to this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104514#msg104514), maybe it also could help:

my current theory is that these top zones are no digital data.
its just the output of the ADC as it is in its black clamp learning phase.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 04, 2014, 04:37:07 PM
If you refer to the very first rows (4~5) I agree; the device maybe uses these rows to acquire the proper black level (apparently the first rows to be read out).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 04, 2014, 04:45:27 PM
ADTG6[8880] - Lower values
(https://s15.postimg.cc/96e7i426j/lower.jpg)

Higher values
(https://s15.postimg.cc/mnb60zk7v/higher.jpg)


ADTG6[886a] 0x10
(https://s15.postimg.cc/j3p8b74nf/886a.jpg)

Quote from: a1ex on March 03, 2014, 01:04:17 PM
A while ago you found a register named "line count that gets darker" (ADTG2[82F3]). For some reason I've tagged it as NRZI, though I didn't double-check it. So, now I've changed this register from 0x30 NRZI (0x20 decoded) to 0x50 NRZI (0x60 decoded) and the result is:

0x6f (decoded)
(https://s15.postimg.cc/6p2gauxpn/0x6f.jpg)

0x0
(https://s15.postimg.cc/5zjnyha0r/0x0.jpg)

As far as I know, the optical black area is labelled as such, because it doesn't receive photons.
So the image must be digitally shifted?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 04, 2014, 07:21:53 PM
Quote from: Audionut on March 04, 2014, 04:45:27 PM
ADTG6[8880] - Lower values

-> target black value the black clamp should be calibrated for?

Quote from: Audionut on March 04, 2014, 04:45:27 PM
ADTG6[886a] 0x10

-> number of top lines the ADTG has to interpret as OB area?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 05, 2014, 11:06:49 AM
Latest raw_diag has a tool that renders dark frames as grayscale, without demosaicing, downsampled by averaging from full size to 720x480, which reveals FPN. Gray levels from +/- ob_stdev/4, which maps to roughly +/- 2 averaged_stdev after downsampling.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/100a.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/100b.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/1600.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/6400.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/100from200.png)

Samples: ISO 100, 100 (second try), 1600, 6400, 100 pulled from 200, all 1/8000. Notice the lack of correlation between the two ISO100 dark frames, and notice how clean ISO 6400 sems to be. However, don't forget these are not absolute noise levels; they simply show the ratio between FPN and Gaussian noise.

Quick theory about why FPN gets revealed by downsampling:

Assume the total noise is a Gaussian noise applied to each pixel + Gaussian FPN applied to each column.

If you downsample the image by 0.125 x 0.125 (that is, 1/8 in each direction), the noise stdevs will be, compared to a 1:1 crop:

    sqrt(8x8) = 8 times smaller for the Gaussian noise
    sqrt( 8 ) = 2.8 times smaller for the Gaussian FPN

=> Gaussian noise gets averaged out faster than FPN.

Roger Clark has similar samples here: http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
but I believe they are downsampled by nearest neighbour (Audionut, since you already have contact with him, maybe you can ask?)

In this case (no averaging involved), the stdevs for noise and FPN would be similar to the ones from a 1:1 crop (which could explain why his 1600 sample is cleaner than mine).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 05, 2014, 05:06:54 PM
Quote from: a1ex on March 05, 2014, 11:06:49 AM
Roger Clark has similar samples here: http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
but I believe they are downsampled by nearest neighbour (Audionut, since you already have contact with him, maybe you can ask?)

I thought they were crops.

QuoteTable 2a. Apparent Read Noise, Central Image

Central 500 x 300 pixel statistics:
min= 132 electrons
max= 198 electrons
mean= 162 electrons
standard deviation= 2.99 electrons

But I'll ask.

The high frequency components of grain are generally visually pleasing, because they help to increase apparent detail.  Averaging removes the high frequency components of the image, leaving only the low to mid frequency components.  In this case it's even worse, because all those fine details (noise or otherwise), combine to become lower frequency components. 

Averaging increases apparent edge detail.  This is excellent on wanted (edge) detail, not so much on detail that isn't wanted.  Averaging also increases aliasing, but this isn't a problem with FPN, because the edges are straight.

The maths might say the noise level (of the FPN) was decreased 2.8 times when it was averaged, but it doesn't tell you exactly what happened with the remaining noise.  In this case, while the total noise dropped, the remaining noise became more visually destructive.

https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
QuoteThis gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.


There are a number of algorithms being designed to measure this subjectivity, but afaik, our eyes are still the best determinator.

http://en.wikipedia.org/wiki/Human_Visual_System_Model


For a visual representation of the FPN, the averaging works extremely well.  Because it emphasis the FPN (visually).

Since the FPN remains consistent (mainly vertical in this case), do you think you could measure the average brightness in a column (1 pixel wide, or maybe even 2 or 3 pixels wide will be sufficient), and then analyse the data in a row?

By measuring the data vertically, you should end up with a data set like so.

1, 3, 5, 1, 1, 5, 1, 9, 3, 6, 7

Since the Gaussian component of the noise should hover around a median (equal in each vertical measurement set), the data set should accurately represent the FPN component?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 05, 2014, 05:19:47 PM
Quote
The maths might say the noise level (of the FPN) was decreased 2.8 times when it was averaged, but it doesn't tell you exactly what happened with the remaining noise.  In this case, while the total noise dropped, the remaining noise became more visually destructive.

It does - the random component decreased 8 times. That's why the FPN became more destructive.

The ratio between FPN stdev and overall noise stdev should be a good indicator of how badly the FPN affects the image.

Quote
Since the FPN remains consistent (mainly vertical in this case), do you think you could measure the average brightness in a column (1 pixel wide, or maybe even 2 or 3 pixels wide will be sufficient), and then look at this data horizontally?

If I understand the question well, the FPN analysis from raw_diag does exactly this.

If you subtract the two FPN components, estimated by averaging, you end up with this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465#msg96465).

Problem: you can't do that on a dark frame and then use that data to correct a useful image, because the FPN is not correlated between images (well, the autocorrelation is extremely weak, as you can see from the "xcov" analysis in raw_diag).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 05, 2014, 06:53:32 PM
i believe your theory about the missing corellation of FPN between shots.the shots show that.

but. how do we get a FPN that is horizontally and vertically repeating in the single image?

if you average the top OB area, you will get the vertical FPN, right?
now subtract that from the whole image and do the same for horizontal FPN using the left OB area.
also subtract that noise frome very column and the low frequency part of the noise should be gone?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 05, 2014, 07:01:40 PM
Quote from: g3gg0 on March 05, 2014, 06:53:32 PM
if you average the top OB area, you will get the vertical FPN, right?

Nope, answered here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104758#msg104758).

You do get an improvement, but it's small.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 05, 2014, 07:39:48 PM
Quote from: a1ex on March 05, 2014, 11:06:49 AM
Latest raw_diag has a tool that renders dark frames as grayscale, without demosaicing, downsampled by averaging from full size to 720x480, which reveals FPN. Gray levels from +/- ob_stdev/4, which maps to roughly +/- 2 averaged_stdev after downsampling.

Samples: ISO 100, 100 (second try), 1600, 6400, 100 pulled from 200, all 1/8000. Notice the lack of correlation between the two ISO100 dark frames, and notice how clean ISO 6400 sems to be. However, don't forget these are not absolute noise levels; they simply show the ratio between FPN and Gaussian noise.

(...)

Wohh, you get a strong corners illumination @ISO 1600 & 6400!
'Hope this is not a electromagnetic field effect as we got on old camera models (350D etc.).

Quote from: Audionut on March 05, 2014, 05:06:54 PM
(...)
The high frequency components of grain are generally visually pleasing, because they help to increase apparent detail.
(...)

I agree. It is an interesting psycho visual effect.
Plus, it help to dither level banding effects in image with large gradients or pseudo flat areas.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on March 05, 2014, 08:51:27 PM
Quote from: a1ex on March 05, 2014, 07:01:40 PM
Nope, answered here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104758#msg104758).

You do get an improvement, but it's small.

A low covariance is nothing special if there is a lot of Gaussian noise evident. The fact that there is some covariance tempted me to try a FPN removal.
The improvement in terms of dynamic range my be small. Nevertheless the improvement in visual quality can be significant.
But judge by yourself:
I took your sample ISO 200 pulled to 100 and split it in the top 80 and the lower 400 rows. Below you see the lower 400 without processing:
(http://www.naturalsound.de/MagicLantern/CMOS/nothing_removed.png)
Then I did what g3gg0 just suggested. An vertical average (let's call it top) of the top 80. Below you see the result of a (floating point) subtraction of this median each line of the image pixels. After the subtraction I added Mean(top) to each pixel value to not shift the blacklevel and rounded the result to yield the new pixel values:
(http://www.naturalsound.de/MagicLantern/CMOS/just_horizontal_removed.png)
Now the same done in the vertical direction:
(http://www.naturalsound.de/MagicLantern/CMOS/vertical&horizontal_removed.png)

My subjective opinion is that removing the FPN can improve the visual quality significantly.

Please note that this test is just a quick&dirty one so there my be some errors induced during re-export of the data to .png format. Plus the input data is not optimal quantized as the histogram shows strong clipping at the borders:
(http://www.naturalsound.de/MagicLantern/CMOS/hist.png)

(some typos corrected)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 05, 2014, 08:57:32 PM
Quotetop 80 and the lower 400 rows

=> you have assumed a OB as large as 658 pixels ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on March 05, 2014, 10:22:28 PM
You're right, in my euphoria I missed the fact that your averaging provided me with lower Gaussian noise component / virtually increased OB.
I just took the values you used in your Reply #584.

I repeated my test with 10pixels (which should be about 80 pixels of OB), resulting in slight (but to my eye pleasing) vertical FPN reduction.
(http://www.naturalsound.de/MagicLantern/CMOS/just_horizontal_removed(OB=80).png)
The effect on horizontal FPN is negligible.

Below the reduction in mean horizontal pixel values distribution for different (virtual) OB sizes:
(http://www.naturalsound.de/MagicLantern/CMOS/hist(OB=40).png)(http://www.naturalsound.de/MagicLantern/CMOS/hist(OB=80).png)(http://www.naturalsound.de/MagicLantern/CMOS/hist(OB=160).png)
It is apparent that for OB=40 the FPN may even get worse.

I hope with this post my tests contribute at least a little for those, who are not such familiar with FPN...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 05, 2014, 10:47:58 PM

function ob_corr_plot(file)

    src_image=imread(file);

    %{
        average some top areas:
            1st: black clamp learning area, 4 through 28
            2nd: OB area that is just black, which is 42 to 79 on 5D3
            3rd: the top 500 pixels ofthe image data (dark frame) starting at 80
            4th: the whole dark frame
    %}
    vert_noise_1=avg_top(src_image, 4, 28);
    vert_noise_2=avg_top(src_image, 44, 38);
    vert_noise_3=avg_top(src_image, 80, 500);
    vert_noise_4=avg_top(src_image, 80, size(src_image)-80);

    %{
        now cut the left OB area and smooth over 256 pixels.
        should be gaussian, but for some tests it might be enough.
    %}
    smooth_1=smooth(vert_noise_1(122:1:end), 256);
    smooth_2=smooth(vert_noise_2(122:1:end), 256);
    smooth_3=smooth(vert_noise_3(122:1:end), 256);
    smooth_4=smooth(vert_noise_4(122:1:end), 256);
   
    % show correlation %
    corr(smooth_2, smooth_3)
   
    figure('name', 'Black clamp area');
        subplot(3,1,1);
        plot(smooth_1);
        title('Black clamp area profile');
        subplot(3,1,2);
        plot(smooth_2);
        title('Optical black area profile');
        subplot(3,1,3);
        plot(smooth_1, smooth_2);
        title('X/Y plot');
   
    figure('name', 'OB vs. Dark frame - top 500');
        subplot(3,1,1);
        plot(smooth_2);
        title('Optical black area profile');
        subplot(3,1,2);
        plot(smooth_3);
        title('Dark frame - top 500');
        subplot(3,1,3);
        plot(smooth_2, smooth_3);
        title('X/Y plot');
       
    figure('name', 'OB vs. Dark frame - full');
        subplot(3,1,1);
        plot(smooth_2);
        title('Optical black area profile');
        subplot(3,1,2);
        plot(smooth_4);
        title('Dark frame - top 500');
        subplot(3,1,3);
        plot(smooth_2, smooth_4);
        title('X/Y plot');
end

function ret = smooth(data, fact)
    for pos = 1:size(data)-fact
        ret(pos) = mean(data(pos : 1 : pos + fact));
    end
end

function ret = downsample(data, fact)
    for pos = 1:size(data)/fact
        ret(pos) = mean(data(((pos-1)*fact) + 1 : 1 : ((pos)*fact)));
    end
end

function ret = avg_top(src_image, start_idx, lines)
    part = src_image(start_idx:1:start_idx+lines, :);
    ret = sum(part) ./ size(part, 1);
    ret = transpose(ret);
end

% draw plots for a dark frame %
ob_corr_plot('D:\Users\g3gg0\Pictures\Kamera\2014_03_05\NO8A3083.pgm');

pause;


which produced this plot and a corr() of 0.95883 for the noise and some data from the top OB area.
dont use the topmost 44 lines. use line 44 to 79 and average them.

does it work better?
http://upload.g3gg0.de/pub_files/16261ae822934c9175945a5e34cba116/plot.png
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 1% on March 05, 2014, 11:08:06 PM
QuoteMy subjective opinion is that removing the FPN can improvement in visual quality significantly.

For raw video in less than lit conditions its the difference between usable shots and garbage...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 06, 2014, 03:52:17 AM
Quote from: a1ex on March 05, 2014, 05:19:47 PM
It does - the random component decreased 8 times. That's why the FPN became more destructive.

The ratio between FPN stdev and overall noise stdev should be a good indicator of how badly the FPN affects the image.

I was attempting to discuss the visual aspect of the FPN itself.  So we can say, the Gaussian noise reduced 8x, and the FPN reduced 2x, the destructive nature of the FPN became more apparent, because it's percentage in the image, increased as a result.
For large changes in the percentages, this should be an accurate subjective measurement.

But we don't know exactly what happened to the FPN.  Since stdev is basically an average, it doesn't (accurately) describe what happened on the edges of the FPN.  Did the edge contrast of the FPN increase?  See bottom.


Quote from: a1ex on March 05, 2014, 05:19:47 PM
If I understand the question well, the FPN analysis from raw_diag does exactly this.

Excellent.  I was misunderstanding how to determine its results earlier.

Quote from: a1ex on March 05, 2014, 05:19:47 PM
If you subtract the two FPN components, estimated by averaging, you end up with this (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96465#msg96465).

Problem: you can't do that on a dark frame and then use that data to correct a useful image, because the FPN is not correlated between images (well, the autocorrelation is extremely weak, as you can see from the "xcov" analysis in raw_diag).

For post processing, the correlation is important.  In terms of, adjust this register, did FPN reduce or increase?  It's not so important.

Quote from: naturalsound on March 05, 2014, 10:22:28 PM
I repeated my test with 10pixels (which should be about 80 pixels of OB), resulting in slight (but to my eye pleasing) vertical FPN reduction.
The effect on horizontal FPN is negligible.

It softens the edge detail of the FPN.  This is important, because contrast is a determining factor in subjectivity.

http://en.wikipedia.org/wiki/Contrast-to-noise_ratio

It is the contrast of this FPN, that makes it a noise component of images.

Let's consider the images at reduced size to emphasise the edge detail.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn/100from200.png) (http://www.naturalsound.de/MagicLantern/CMOS/just_horizontal_removed%28OB=80%29.png)

Because these images are made up of only 3 components, FPN, Gaussian, and black, the differences in the images might not be considered overly great.

However, with the forth component present in the images, wanted detail, the difference would be subjectively greater.  Where we reduce edge detail/contrast of the noise, the edge detail/contrast of the wanted detail becomes more predominant.

The contrast ratio of wanted detail becomes greater then the contrast ratio of the noise.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: AdamTheBoy on March 06, 2014, 03:20:29 PM
I've been following this thread and please forgive me if the answer is present somewhere but I was curious if this increase in dynamic range will be applicable to h264 videos.  If so, this development combined with the recent bitrate hack being investigated here http://www.magiclantern.fm/forum/index.php?topic=4124.0 is really exciting for 5d3 owners who want to shoot h264 for a project.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 06, 2014, 03:33:09 PM
I have had a quick play with it here (http://www.magiclantern.fm/forum/index.php?topic=4124.msg104972#msg104972), and I could see extra highlight detail when adjusting the registers.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on March 06, 2014, 05:33:42 PM
Just a question about calculating capability inside the camera (DIGIC I presume):

Will it be too much to imagine calculate the vertical 2D FFT harmonics of 'some' (dozen?) rows just top of the active area but in the OB, done the same process with OB area on left to find the horizontal 2D FFT harmonics, then do the invert FFT from this on the whole active area?

a1ex, you spoke about seconds for the FPN algorithm reduction; 2D FFT operations look faster to do, what do you think by doing this?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 06, 2014, 05:36:26 PM
Just take a look at how fast raw_diag is when processing the entire image (try the dark frame analyses).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 06, 2014, 11:36:07 PM
QuoteIt is apparent that for OB=40 the FPN may even get worse.

Even if you take a single line from top OB, as long as there is some weak correlation between that line and the FPN, you can still use it to reduce the FPN. The key is not to subtract it blindly, just give it a lower weight.

How much?

Let's look at the Kalman filter theory: http://robocup.mi.fu-berlin.de/buch/kalman.pdf

From page 3, if we know how noisy our estimations are, the optimal weights are inversely proportional with the noise variances. For averaging 2 random variables x1 and x2, the optimal weights can be simplified to var(x2) and var(x1):


x_optimal = (x1 * var(x2) + x2 * var(x1)) / (var(x1) + var(x2))


where var(x) = std(x)^2.

So, if we average a single line from the top OB, we need to know how noisy is that line and how bad our FPN is. Assumming Gaussian noise with variance vg = sg^2 + Gaussian FPN with variance vf = sf^2, our OB line estimates the FPN with a variance equal to vg. If we simply assume the FPN is zero, the error is a Gaussian with variance vf. Combining these two estimations optimally means simply downweighting the top OB line:


fpn_estimated_from_one_line = vf / (vf + vg) * that line + vg / (vf + vg) * 0


If we average n lines to estimate the FPN better, its error will have a variance equal to vg/n; that is, the stdev of our estimation will be sg/sqrt(n). To get minimal variance in the corrected FPN, the optimal weighting is now:


weight_avg = vf / (vf + vg/n)
weight_fpn = (vg/n) / (vf + vg/n).
fpn_estimated_from_n_lines = weight_avg * mean(n_lines) + weight_fpn * 0


What will be the variance of our corrected FPN?


var_combined = var_avg * weight_avg^2 + var_fpn * weight_fpn^2


where var_avg = vg/n and var_fpn = vf.


=> var_combined = vg/n * vf^2 / (vf + vg/n)^2 + vf * (vg/n)^2 / (vf + vg/n)^2
=> var_combined = (vg/n * vf^2 + vf * (vg/n)^2) / (vf + vg/n)^2
=> std_combined = sqrt(var_combined)


Now let's check the theory.

Let's say we have a 720x480 dark frame that contains a random noise of stdev sg=0.6 and a vertical FPN of stdev sf=0.25 (values typical for a downsampled ISO 100 pulled from 200).


sg = 0.6;                                                       # Noise stdev
sf = 0.25;                                                      # FPN stdev
vg = sg^2; vf = sf^2;                                           # Variances
G = randn(480,720) * sg;                                        # generate Gaussian noise
f = randn(1,720) * sf; F = ones(480,1) * f;                     # generate Gaussian FPN
GF = G + F;                                                     # combine both noises => ideal dark frame
imshow(GF, [])
print -dpng -r60 ideal-dark.png


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn-ideal/ideal-dark.png)


# try to average n lines
for n = 1:80
    fe = mean(GF(1:n,:)) * vf / (vf + vg/n);                    # FPN estimated and weighted optimally
    S(n) = std(f-fe);                                           # experimental stdev of corrected FPN
    SI(n) = sqrt((vg/n * vf^2 + vf * (vg/n)^2) / (vf+vg/n)^2);  # theoretical stdev of corrected FPN
    SR(n) = std(f - mean(GF(1:n,:) * (rand*2)));                # try some other random weights between 0 and 2
end

plot(SI/sf, 'r-o', 'markersize', 2), hold on
plot(S/sf, 'b-o', 'markersize', 2)
plot(SR/sf, 'kx', 'markersize', 2)
axis([0 80 0 1]), grid on
set(gca, 'ytick', 0:0.25:1);
set(gca, 'position', [0.2 0.15 0.7 0.7])
legend('Theoretical improvement', 'Simulated improvement', 'Random weighting', 'location', 'southwest')
title('FPN improvement in ideal conditions')
xlabel('Number of lines averaged')
ylabel('stdev(corrected FPN) / stdev(original FPN)')
print -dpng -r70 ideal-fpn-improvement.png


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn-ideal/ideal-fpn-improvement.png)

So, this graph shows:

1) this weighting is indeed optimal (all the random weights resulted in worse improvement)
2) we have an upper bound for the improvement that can be obtained by averaging some lines

=> to reduce the FPN component by 1 stop, we need to average 17 lines out of 480 for this numerical example.

How much is 1 stop of FPN improvement? (note the Gaussian noise is not touched)


fe = mean(GF(1:17,:)) * vf / (vf + vg/n);                    # FPN estimated from 17 lines and weighted optimally
std(f-fe) / std(f)                                           # experimental improvement for FPN
ans =  0.54244
FE = ones(480,1) * fe;                                       # extend the correction to the entire image
clf, imshow(GF-FE, [])                                       # subtract the estimated FPN
print -dpng -r60 ideal-corrected-1ev.png


(http://a1ex.magiclantern.fm/bleeding-edge/iso50/fpn-ideal/ideal-corrected-1ev.png)

To be continued; feel free to try to apply this theory to a practical example.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on March 07, 2014, 11:59:45 AM
@alex:

today i sat in front of matlab with a colleague and we tried to get some more information from the top OB area.
he had the idea to do a singular value decomposition for pattern recognition. we implemented it based on the example image generator and built a script for it.
the result was visually pleasing - could you check if it is improving things? thanks :)


sg = 0.6;
sf = 0.25;
vg = sg^2; vf = sf^2;
G = randn(480,720) * sg;
f = randn(1,720) * sf; F = ones(480,1) * f;
GF = G + F;

% the error seems to be with main energy in 1st order %
max_order = 1;
num_plots = 1+max_order;

% plot original image %
figure;
subplot(num_plots,1,1);
imagesc(GF);

% only use top 40 pixels for correction %
top = GF(1:40,:);
top_mean_free = top-ones(size(top))*mean(mean(top));

% run a singular value decomposition (major component analysis) %
[A,B,C] = svd(top_mean_free');

% start correction %
GF2=GF';
for order=1:max_order
   % get deviation pattern of n-th order %
   dev = A(:,order);
   % scale pattern with stddev of this pattern %
   corr_vec = sqrt(B(order,order)) * dev;
   % remove mean to make it mean-free %
   corr_vec_mean_free = corr_vec - ones(size(corr_vec)) * mean(corr_vec);
   
   % to detect the pattern polarity we do a projection of our pattern onto our top OB area and check its mean value %
   dir = sign(mean(corr_vec_mean_free' * top_mean_free'));
   
   % now we can add/subtract that pattern from image data %
   GF2 = bsxfun(@minus,GF2,corr_vec_mean_free * dir);
   
   % and show the result %
   subplot(num_plots,1,1+order);
   imagesc(GF2');
end
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 07, 2014, 12:21:05 PM
It simply finds the same optimal solution with a different method.

Result from your program:

std(mean(GF2')) / sf
ans =
    0.3525


My theoretical result for n=40:

n=40;
sqrt((vg/n * vf^2 + vf * (vg/n)^2) / (vf+vg/n)^2) / sf
ans =
    0.3548


In this ideal example, your principal component analysis summarized the top 40 lines into a single one (which is to be expected, because the signal we are looking for in these lines - the FPN - is the same in all these lines, and anything else besides this FPN is just Gaussian noise). What would be interesting is to try this one on a real OB. There I expect surprises (probably pleasant ones).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on March 16, 2014, 01:45:37 PM
Quote from: a1ex on February 03, 2014, 10:45:10 PM
Updated adtg_gui: I've added a routine that dumps all the registers into a log file after taking a picture (this is what I've used to make the funky graphs). I'd like to get these graphs for all cameras, so I need your help

Ok, here you go for 6d: https://bitbucket.org/Marsu42/ml-aiso/downloads/adtd-logs-6d.zip

On another note: The above research is absolutely impressive and all and no doubt this will result in an optimal usage of the Canon sensor, even though the complicated re may stall the work from time to time. But, speaking with my simple user's hat on...

... I would be most grateful if we'd get a working, but not optimal or final mini_iso.mo for latest core and with working white point adjustment for 6d. This is because every day w/o this, I'm losing some dr in my shots, and this is sad to know :-o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 16, 2014, 02:07:31 PM
Well, the next step towards mini_iso is a library for patching things around in Canon code. Around 70% of it is mini_iso code refactored to be generic, and the rest is a small GUI frontend to see what exactly got patched.

That's because mini_iso needs to patch a LOT of memory addresses, and I want to do it cleanly, with sanity checks before patching, integrity checks at runtime, and, very important to me, undo all patches when you disable the menu. This doesn't happen with iso_regs.

This could be reused for a cleaner implementation of things like bitrate hacks from Tragic Lantern, since this library would help with all the dirty stuff like how to undo the patches, when to enable the cache hacks, and will also let the user what hacks are enabled at any given time. When shooting something a little more serious, I want to know for sure what patches are applied, and that stuff turned off from menu is indeed turned off.

In my local copy I already refactored dual ISO and raw_rec/mlv_rec patches to be applied via this library.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on March 16, 2014, 02:29:13 PM
Quote from: a1ex on March 16, 2014, 02:07:31 PMIn my local copy I already refactored dual ISO and raw_rec/mlv_rec patches to be applied via this library.

Sounds great and I'm a big fan of safe and clean code (because even as a module coder, I sometimes have to look at it :-)).

I just wanted to add my 2ct not to let perfectionism stand in the way of pragmatism for a working mini_iso, just like Linus Torvalds always says about Linux I feel ML is not a research project, but made to be actually used :-o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 16, 2014, 02:30:52 PM
And exactly for this reason I want mini_iso to just work.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: hjfilmspeed on March 16, 2014, 09:42:37 PM
Im not a coder by any means but I liken the finely tuned modules to a song and artist is writing. He or she wont release it until all of it is perfect and flows just right. These modules/code are the coders babies, with all the work they put in to them Im sure they want them to be flawless before they are public or beta ect. But its hard for us non coders to be patient with such incredible features. Cant wait for this mod!!!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on March 23, 2014, 07:46:44 PM
Btw when this code goes public, it would be very nice to have an external function to get the optimal overexposure ec from the mini_iso calibration. This way autoexposers like autoexpo or auto_iso that work in m mode can automatically modify the exposure accordingly.


int get_mini_iso_calibration(int iso_canon8)
{
    int ec_canon8;
    ...
    return ec_canon8; // ec value in canon steps (0/3/4/5/8) 
}
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on April 04, 2014, 03:30:49 PM
ADTG[7] 0x4046 -> 0x4036

It is darken the image by 0.5 EV 500D.
White level and black level remains unchanged.

Other records ADTG change the white level or black level.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: kgv5 on April 08, 2014, 10:08:34 AM
I havent been here for a while, what is current status of the video mode on the 5d3? Is it still 0,1 EV (much less than in stills mode) like in the description on the first page or maybe there was some improvement in this matter?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 08, 2014, 12:00:14 PM
I'm pretty sure I was seeing some good changes with the latest iso_regs module.  But there didn't appear to be an easy way to see the OB area of raw video, and I don't shoot video, so I gave up very easily.  ;)

If someone points out how to see the OB area of raw video, I'll take another look.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 09, 2014, 03:15:16 PM
I was pondering something today.

Canon ISO 100/1600

Input file      : _46a0505.cr2
Camera          : Canon EOS 5D Mark III
White levels    : 15179 13789
Noise levels    : 11.05 6.48 6.54 10.98 (14-bit)
ISO difference  : 3.95 EV (1550)
Dynamic range   : 10.99 (+) 10.05 => 14.01 EV (in theory)
Semi-overexposed: 0.89%
Deep shadows    : 96.88%
ISO overlap     : 4.0 EV (approx)
Noise level     : 37.21 (20-bit), ideally 37.22
Dynamic range   : 14.46 EV (cooked)



ML ISO 100/800

Input file      : _46a0504.cr2
Camera          : Canon EOS 5D Mark III
White levels    : 15645 14250
Noise levels    : 6.49 3.95 3.96 6.40 (14-bit)
ISO difference  : 3.00 EV (799)
Dynamic range   : 11.75 (+) 10.88 => 13.87 EV (in theory)
Semi-overexposed: 0.82%
Deep shadows    : 95.91%
ISO overlap     : 5.8 EV (approx)
Noise level     : 42.41 (20-bit), ideally 42.41
Dynamic range   : 14.32 EV (cooked)


ML ISO 100, is actually Canon ISO 200, with the highlight saturation point adjusted.  In the midtones/shadows, it behaves exactly like Canon ISO 200, but the rated ISO gets shifted down to ISO 100, as per DxO's definition (http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity).  This is how the DR increases with ML ISO.  The midtones/shadows stay the same, but it increases highlight capturing ability.

ML ISO 800, is actually Canon ISO 1600, with the highlight saturation point adjusted.  So, the shadow point of ML ISO 800, is actually equal to Canon ISO 1600.

The DR of ML ISO 100, is 0.8 EV greater then Canon ISO 100.

So, we gain 0.8 EV of midtone overlap, from the cleaner ISO, and we gain another 1 EV of midtone overlap, because we are only 3 stops apart (Canon ISO 200/1600), instead of 4 stops apart (Canon ISO 100/1600).

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 22, 2014, 06:17:30 PM
Quote from: a1ex on March 16, 2014, 02:07:31 PM
Well, the next step towards mini_iso is a library for patching things around in Canon code.

The first step was made: https://bitbucket.org/hudson/magic-lantern/pull-request/476/experimental-library-for-managing-memory

As a second pre-requisite, I would also like a tool for pre-processing raw files - in this context, I need it to reduce the FPN. Without it, the noise improvement will have little practical value if the FPN will stay at the old levels. And since I wasn't able to find a way to fix the FPN in camera, the quickest solution would be a software tool similar to cr2hdr (which will most likely output a DNG file).

Advantage: the FPN fix is also needed for some regular ISOs, for example here (http://www.magiclantern.fm/forum/index.php?topic=9564), so the same tool could do the job in both cases.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 23, 2014, 02:48:33 PM
Quote from: a1ex on April 22, 2014, 06:17:30 PM
The first step was made: https://bitbucket.org/hudson/magic-lantern/pull-request/476/experimental-library-for-managing-memory

Great there's progress on this!

Quote from: a1ex on April 22, 2014, 06:17:30 PM
As a second pre-requisite, I would also like a tool for pre-processing raw files - in this context, I need it to reduce the FPN. Without it, the noise improvement will have little practical value if the FPN will stay at the old levels. And since I wasn't able to find a way to fix the FPN in camera, the quickest solution would be a software tool similar to cr2hdr (which will most likely output a DNG file).

Doh, I believe you this cannot be done in camera, but another cr2hdr-like stop adds a lot of overhead - it would be important that this at least a lossless conversion so the original cr2 can be deleted instead of dual_iso double storage space requirements. I'd also like to note that ending up with dng instead of cr2 makes use of dxo's prime noise reduction impossible, that's why I recently switched from dng to preserving cr2 again. If it's possible, an option to patch the original cr2 would be welcome.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 23, 2014, 03:11:14 PM
Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.

I can try to implement the FPN fix in the camera too, but speed will be a problem.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 23, 2014, 03:52:13 PM
Quote from: a1ex on April 23, 2014, 03:11:14 PM
Mars did some good progress with faking the CR2 data, so CR2 output is not excluded.

Interesting to know, thanks for the information - btw keeping cr2 would improve the general acceptance of the ml dr improvement, still lots of folks out there who aren't comfortable with dng.

Last not least, another reason for cr2 is that except ~+4mb/shot it has no drawbacks in LR/ACR unless you want to use dng-only features (embedded color profile, lossy compresssion) - but saving metadata is much faster since it only writes the sidecar xmp.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 25, 2014, 04:23:21 AM
Quote from: Marsu42 on April 23, 2014, 02:48:33 PM
but another cr2hdr-like stop adds a lot of overhead

Only if you want the FPN correction.  Otherwise, it's just like any other Canon file, with reduced read noise.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 25, 2014, 11:20:46 AM
Quote from: Audionut on April 25, 2014, 04:23:21 AM
Only if you want the FPN correction.  Otherwise, it's just like any other Canon file, with reduced read noise.

Btw this doesn't concern 6d users, I've never ever seen any fpn on this model (unlike my good ol' 60d).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 25, 2014, 12:35:40 PM
Yes, but sadly I can't target just the 6D :P

BTW, can you post the raw_diag dark frame graphs from 6D? I'm curious to see how the numbers compare with 5D3. Try a few ISOs that you consider relevant.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 26, 2014, 01:31:51 AM
Quote from: a1ex on April 25, 2014, 12:35:40 PM
BTW, can you post the raw_diag dark frame graphs from 6D?

... here you go (how does the 6d do?): https://bitbucket.org/Marsu42/ml-pull/downloads/6d_fpn.zip
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 26, 2014, 03:34:15 AM
6D - 5D3 (ISO 100)

(https://s15.postimg.cc/5ms9s6hff/DARKGRAY.jpg) (https://s15.postimg.cc/majruojwr/darkgray1.jpg)

http://www.clarkvision.com/articles/evaluation-canon-6d/index.html
QuoteBut even more impressive than the high signal and low read noise, is the far better control of pattern noise. The 6D sensor performs well ahead of the 1DX at low ISO, similar to that for the 1DX at moderate to high ISOs, but the 1DX pulls slightly ahead at very high ISOs. The pattern noise of the 6D is overall the lowest I have measured for any Canon camera.


edit:  dual_iso images removed.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 26, 2014, 08:58:40 AM
Quote from: Audionut on April 26, 2014, 03:34:15 AM
6D - 5D3 (ISO 100)

I don't quite know what to make of these greyscale images - does it mean that the 6d has nearly no vertical fpn, but some horizontal?

Edit: dual_iso images deleted because they're useless.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 26, 2014, 10:57:03 AM
Comparing the 6D/5D3 images above, you can see how the 5D3 has large amounts of vertical FPN.  The 6D is doing a remarkable job of removing the vertical FPN component, but there is still some horizontal FPN remaining.

I believe I read that Canon was doing some FPN correction on the 6D.  Looks like they concentrated on the vertical component.  It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras.  I'm not sure where to start looking though, other then poking registers.  Maybe a1ex has some better ideas, if it's possible, and if you or another 6D user has the time.

It's important to remember that the pattern noise is generally a fixed component (the noise level stays the same, for each individual camera).  And also, that human eyes/brain, are adapt at detecting patterns.  So when it looks like the pattern noise is getting stronger or weaker (on an individual camera), actually, the random noise is increasing, or decreasing.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 26, 2014, 02:48:10 PM
If you look at the DARKFPN graphs, the 6D shows:

- Vertical FPN: stdev 0.30 with ratio 0.06
- Horizontal FPN: stdev=0.22 with ratio 0.05
- Overall stdev: 4.59 (from DARKHIST)

At ISO 100, 5D3 has:

- Vertical FPN: stdev 0.85 with ratio 0.13
- Horizontal FPN: stdev=0.21 with ratio 0.03
- Overall stdev: 6.81

These ratios are the FPN stdev divided by overall stdev, that is, how much of the total noise is FPN. I believe it's measuring the subjective effect of the FPN (that is, a low ratio, let's say below 0.05, tells the FPN gets buried into noise, and a high ratio, say above 0.1, tells the FPN is becoming obvious).


Side note: on a 4000x5000 ideal Gaussian, the estimated vertical FPN ratio is 0.015.

octave:16> G = randn(4000,5000);
octave:17> std(mean(G)) / std(G( : ))
ans =  0.015700



Overall stdev:
From raw_diag data, 6D has around 0.57 stops of extra dynamic range at ISO 100, compared to 5D3. To compare these numbers, you need to normalize by white-black, but from the data I have, the white levels are 15331 and 15283, which does not really make a difference.

From Roger Clark, shadow noise levels at ISO 100 are 34.9 vs 29.0 electrons, so 6D is ahead 5D3 by only 0.26 stops. The extra difference in DR may be explained because 6D's ISO 100 captures 79600 electrons, compared to 68900 for 5D3. That's 0.2 stops of extra highlight detail for 6D (so 0.46 stops difference in dynamic range).

FPN:
The FPN ratios are explaining the first two screenshots posted by Audionut above (http://www.magiclantern.fm/forum/index.php?topic=10111.msg112945#msg112945). 5D3 has obvious vertical FPN (ratio=0.13); 6D improves it by one stop (ratio=0.6), so vertical FPN gets buried into noise. But the horizontal FPN is slightly better on 5D3 (ratio=0.03 vs 0.05).

So, considering the FPN improvement, I'd say that, in practice, out of the box, at ISO 100, the 6D is ahead of 5D3 by around one stop.

Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.

5D3 ISO 100 reimplemented from 200 (without black/white level tweaks)

- Vertical FPN: stdev 0.71 with ratios 0.20
- Horizontal FPN: stdev=0.14 with ratio 0.04
- Overall stdev: 3.70

So, the vertical FPN is reduced only a little (0.26 stops), but the Gaussian noise is reduced a little more (0.88 stops), which makes the FPN even more obvious (the ratio gets degraded by 0.62 stops).

And this is why we need a FPN correction tool: we have almost 0.9 stops of noise improvement on 5D3, but in practice it's not very useful because of the FPN. You have already seen the effect earlier in this thread: here on a test sample from Audionut (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104298#msg104298), and here on a downsampled dark frame (http://www.magiclantern.fm/forum/index.php?topic=10111.msg105001#msg105001).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 26, 2014, 02:58:21 PM
And now here's a little experiment that reveals the feedback loop from the left OB area, and its time constant. Set ADTG 82F3 (the one that sets top optical bar size) to 0, then run the OB zones analysis, then average the data from the left OB area.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/ob-zones-5d3-iso100-82F3-0.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/left-feedback.png)

You may cross-check it with this graph (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104546#msg104546) and with g3gg0's theory: [1] (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104531#msg104531) and [2] (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104682#msg104682).

This info should explain the low frequency look of the horizontal FPN and should be very useful for correcting it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 26, 2014, 06:40:25 PM
Quote from: a1ex on April 26, 2014, 02:48:10 PM
Side note: the raw_diag grayscale analysis on dual ISO images is not relevant, because it will simply average the two components. In deep shadows, the final output from dual iso only contains data from the higher ISO.

Which explains why I wasn't seeing the same FPN reduction that I do in real life images.  In practice, dual_iso does an excellent job of reducing FPN.

ML ISO / ML ISO + dual_iso
(https://s15.postimg.cc/yp6juuyij/46_A1297.jpg) (https://s15.postimg.cc/g9m2xgztn/46_A1301.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on April 26, 2014, 09:25:46 PM
Quote from: Audionut on April 26, 2014, 10:57:03 AM
I believe I read that Canon was doing some FPN correction on the 6D.  Looks like they concentrated on the vertical component.  It would be interesting to find out if the work Canon is doing in the 6D firmware, could be duplicated in other cameras.

Interesting, reminds me of test shots showing the 6d is slightly softer than the 5d3 - I though this would be due to some forced chroma noise reduction, but maybe fpn correction is (also) the reason. If it's really done in camera by software, this is indeed remarkable as the softening is minor.

Question is: If so, why did Canon concentrate on vertical fpn, maybe because their speed-tuned algorithm only works this way while the image is read from the sensor?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 27, 2014, 04:18:21 AM
Quote from: Marsu42 on April 26, 2014, 09:25:46 PM
Question is: If so, why did Canon concentrate on vertical fpn,

That was a poor choice of words on my behalf, sorry.
I have no true understanding why the vertical component is reduced, but the horizontal FPN remains similar to the 5D3.  I assume that there is some relationship to the column amplifiers.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: tjaja on May 02, 2014, 06:18:01 PM
Offtopic:
Are you guys talking about this? Noticed it in h264 recording and on liveview: fixed noise. On monday Im going to the canon service site in the netherlands and on the phone they told me: It has to be re alligned ( :S :S) or something nothing to be concerned of.
(http://www.alisultan.nl/magicbug/verticalbanding.JPG)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on May 02, 2014, 07:06:08 PM
Quote from: tjaja on May 02, 2014, 06:18:01 PM
Offtopic:
This.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on May 10, 2014, 05:04:14 PM
Pulled Canon ISO 200.

(https://s15.postimg.cc/3tpd3ne6j/ob-dr.png)

I suspect there may even be a little more then 0.1 EV of nonlinear highlights.  But I need to confirm.

8/9/A/B is defiantly biased towards the highlights, and a strong cause of the nonlinear roll off in the highlights.  This register bank is not really noisy (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103911#msg103911), so if I have over estimated the non-linearity, adjusting these registers back towards default, won't have a large effect on noise (printed DR results).

Adjusting saturate offset on 5D3.123, blanks the rear display.  The camera appears to function correctly, except no display.
+    {0xC0F0,   0x819c, 0, "Saturate Offset (photo mode) (HIV_POST_SETUP)"},
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on May 10, 2014, 06:06:14 PM
Looks like there is some room for improvement, by adjusting the dual 0xFE registers, individually.

(https://s15.postimg.cc/ku89cebt7/ob-dr1.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on May 10, 2014, 06:42:52 PM
So the gain is more than 1EV?  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on May 10, 2014, 06:54:19 PM
Well, there are issues with large individual differences to 0xFE.

(https://s15.postimg.cc/8scvicpqz/46_A3294.jpg)

There is a very faint pattern noise.  But here, I was using a difference of 3, between the register values.

Note:  This is a horizontal pattern noise, as opposed to the vertical pattern noise from other registers.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: naturalsound on May 25, 2014, 09:56:35 PM
Hi,

could somebody maybe send upload me one darkframe and one or two pictures with severe vertical FPN (ISO100 pulled from Best would be a linear TIFF 16bit (maybe even including the OB zones)?

CR2 would be OK also, although I will not be able to get the optical blacks with my current software.

Thanks in advance.


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Luiz Roberto dos Santos on June 01, 2014, 10:41:29 PM
@A1ex any news about the paper? The community can help you with something, like tests and so on?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 01, 2014, 11:08:14 PM
Not really; need to find the time and the mood to sit down and figure out stuff. Pretty math intensive...

But you can try to follow the thread and check our findings for other cameras; that would be nice. Raw_diag and adtg_gui are portable (they run on all cameras, first without modification, second may require minor tweaks) and should be a very good time killer during long train journeys, for example.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 02, 2014, 05:42:29 AM
Looks like Canon increased the ISO Digital Gain in 1.2.3.  The histogram has 1 pixel gaps with 1.2.3, and these are not present in 1.1.3.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 02, 2014, 08:16:19 PM
Losing the mentality of maximum DR, whites be damned.  I started looking more closely at the histogram with RawDigger.

@ISO 100, the maximum gain reductions I can do while still maintaining saturation in all color channels.
0xFE - 0x3
888x - 0x3ax

And with these tweaks.
CMOS4 - 0x308
ISO digital gain - 0x200

One thing became clear, the blue channel had less gain then the other channels.  On a fully saturated image, the stdevs.

    black   white
R   4.50    24.6
G   4.46    32.1
G   4.50    27.3
B   4.42    95.8


There is a lot more deviation at white in the blue channel, due to it only just hitting saturation.  From my research here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104511#msg104511), it's clear that ADTG2 is affecting G1 and B channels, and ADTG4 is affecting G2 and R channels.  So, reducing ADTG4(888x) to 0x38x resulted in these numbers.

    black   white
R   4.33    89.1
G   4.50    32.3
G   4.34    89.5
B   4.41    95.8


Dynamic range in the R and G2 channels increased by 0.05 EV.  All channels still saturated.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 04, 2014, 07:56:18 AM
Here are 2 histograms showing the description from my last post.

ADTG 2/4 equal values.
(https://s15.postimg.cc/z0o07fk1n/histogram1.png)


ADTG 4 reduced.
(https://s15.postimg.cc/9549o8ai3/histogram.png)


With ADTG 4 reduced, the histograms look remarkably similar.  For reference, without ADTG tweaks, the histogram is a single peak of pixels at saturation, with a small amount of pixels above this saturation point.

Looking closely at the data, it appears as though having only some amount of saturation, is not sufficient for correctly rendered images.  You'll notice on the histograms that there is some data above the saturation point, and reducing the gains sufficiently, this data is no longer there (presumably it has been pulled back into a useable range).  However this results in artifacts in the image.

This is the image of the second histogram above.
(https://s15.postimg.cc/coq7e1snf/imperfections.png)


It should be pure white.  Observing the color channels individually, the B, R and G2 channels show this pattern.  The G channel, which still has data above the saturation point, still retains an even appearance. 
I would say this data above the saturation point is simply noise, and where the gains are reduced to much, this noise gets pulled back into the rendered image.

I thinking about WL detection in ML.  Rather then guessing the hot pixels (based on some percentage), it may be useful to determine the saturation level based on the amount of pixels, at some level based on the histogram, with pixels above the saturation level being discarded.  I can't think of any situations in nature where this would occur, so it should be safe to assume that any pixels above the spike at white, are simply noise.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 04, 2014, 10:01:56 AM
Right, detecting a sharp peak makes sense.

The difficulties appear when the peak isn't very sharp (amplifier gains set way too low), or when the image doesn't have a peak (the image is not overexposed, but it's close - so, in this case, white level detection via percentile would cause useful detail to be marked as "clipped").
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 04, 2014, 11:30:46 AM
Personally, I wouldn't consider a situation where the gains have been set to low.  This is a user problem.

@ISO 200, ADTG 4 gains can only be reduced a little further to match the gain of ADTG 2.  For instance, ADTG2 - 888x - 0x37x, ADTG4 - 888x - 0x36x
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 01:23:21 AM
Changing the topic a bit.

I was trying to understand how Roger Clark figured out the full well capacity in electrons (http://www.clarkvision.com/articles/evaluation-1d2/), so I've implemented a fairly similar method in raw_diag.

Click for his analysis of 5D3 (http://www.clarkvision.com/photoinfo/evaluation-canon-5diii/index.html), 6D (http://www.clarkvision.com/photoinfo/evaluation-canon-6d/index.html), 5D2 (http://www.clarkvision.com/photoinfo/evaluation-canon-5dii/index.html) and 7D (http://www.clarkvision.com/photoinfo/evaluation-canon-7d/index.html).

Basic theory:

- SNR curve can be estimated from the difference of two test images, taken with identical settings, from a static scene.
- Roger recommends shooting a out-of-focus white wall and take a bunch of images. I went for something much easier (but less accurate): simply shoot a out-of-focus HDR scene (with both deep shadows and clipped highlights) and take two test pictures. I think it's enough for our purposes.
- The difference will contain noise multiplied by sqrt(2), because noise from two subsequent images is not correlated (well, mostly).
- If you take small patches of the same color, we can say the patch average (taken from one of the test images) is the input signal, and the patch stdev (taken from the difference image) is the noise level at that input signal. So, we have all we need to plot the SNR curve.

Curve fitting:

- Now, we can fit an ideal model for the SNR curve. I'll use one that combines read noise (assumed constant additive Gaussian) and shot noise (which follows Poisson statistics). Something like this:


dn = patch_average - black_level;
electrons = dn * gain;
shot_snr = sqrt(electrons);
shot_noise = dn / shot_snr;
combined_noise = sqrt(read_noise*read_noise + shot_noise*shot_noise);
model_snr = log2(dn) - log2(combined_noise);


- So, we have a model with two parameters (read_noise and gain), that gives an ideal SNR. We can compare that SNR with measured values (from the SNR curve) and minimize the difference between our model and our measurements.

- Robust statistics tricks / speedup: we have up to 20000 data points; a few of them are outliers, but most of them are OK. I've grouped them into bins, every 0.5 EV, and took the median from each bin. So, now I have up to 28 data points, without outliers. Clean and fast :)

- Since I don't know how to find a closed-form solution for the best fit (if you know how, please enlighten me), I went for a very simple minimization routine: at each step, choose a random point nearby and evaluate the function there; if it's better, keep it, if not, discard it. It does the job.

My results:

I've only tested ISO 100 and 1600 in photo mode.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr-fit-100.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr-fit-1600.png)



Wanted: your results



1. grab raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (no need to compile anything, just place it in your MODULES folder).
2. use 1/50 for PAL, 1/60 for NTSC, or any shutter speed if you don't use artificial lights.
3. point the camera to a out-of-focus HDR scene, and adjust ISO and aperture to get a good picture (both clipped highlights and fairly deep shadows).
4. enable Debug->RAW Diagnostics, and from there, select "SNR curve (2 shots)". Disable the other analyses types (unless you want them). Make sure "Auto screenshot" is enabled (it is by default).
5. take the two test pictures from Debug -> RAW Diagnostics -> Dummy bracket. Simply click it once and wait until it's finished.
6. post the screenshot (it's called SNR2.PPM and it's saved under the RAW_DIAG folder; convert it to PNG first to make it smaller, and give it a meaningful name).

I'm looking for results covering all ISOs, for all cameras, both photo and movie mode.

If you wish to review the code or compile yorself, look here (https://bitbucket.org/hudson/magic-lantern/commits/c4b81743843f).

Have fun!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 05, 2014, 04:43:16 AM
5D3 - Photo mode.  Canon ISOs @ f/1.4 and f/4.0.

ML ISO shots have ADTG 2/4 0xFE 0x3, CMOS4 0x308 and ISO digital gain 0x200 (Canon f/4.0 default) in common. 

ISO 100, ADTG2 888x 0x3cx, ADTG4 888x 0x3ax.
ISO 200-25600, ADTG2 888x 0x38x, ADTG4 888x 0x37x.

Making sure not to have gains reduced to much.

https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa

If you aren't aware, you can click the blue download button and save the results directly to your own dropbox. :)
It would be handy if the screenshot only captured useful data.  That is, it didn't capture the first image in this dual image sequence.

For those of you who fancy "unity gain", you may be interested in ML ISO 800.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dsManning on June 05, 2014, 05:49:35 AM
50D photo mode. ISO 100-3200

https://www.dropbox.com/sh/d333un8o8lwmtgy/AACHL0HAY2bjUNqzhxCbtsWga (https://www.dropbox.com/sh/d333un8o8lwmtgy/AACHL0HAY2bjUNqzhxCbtsWga)

50D movie mode. ISO 100-3200

https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva (https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 05, 2014, 08:46:13 AM
Care to share your thoughts on why the SNR reduces around 0.5 EV in the highlights, with an ISO bump?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 09:03:31 AM
Cool, so the tweaks are confirming the ISO changes with a totally different method (Poisson statistics, instead of linear matching of raw data).

ISO 100, f4->f1.8:
OB noise increase: 6.48 -> 7.14 (1.1x)
read noise increase: 6.71 -> 7.38 (1.1x)
gain decrease: 5.18 -> 4.74 (1.09x)

=> the digital gain applied at f1.4 is about 1.1x. From the digital gain register, previously we have estimated it at 569/512 = 1.11x. Not bad.

LiveView results are interesting, but need to figure out the math first. They might show how many pixels are binned.

@dsManning: can you retry the LiveView tests at ISO 400 and above? They have obvious outliers.

Quote
Care to share your thoughts on why the SNR reduces around 0.5 EV in the highlights, with an ISO bump?

Not sure I understand the question, but I assume it's the difference between ISO 100 f4.0 and ISO 200 f4.0, for example. ISO 200 captures half of the photons, compared to ISO 100, so the shot noise is 1.41x higher. In shadows, the read noise is dominant, and it changes very little from 100 to 200. In highlights, the Poisson noise is dominant, so the difference is roughly half-stop.

For exact figures, you can take the equation of the ideal SNR curve (pasted above), plug the two parameters (estimated read noise & gain) and do the math for any signal level.




This SNR model can be interesting for profiled denoising, since you can model the noise curve with only two parameters (cc @hanatos from Darktable).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 05, 2014, 10:56:49 AM
Quote from: a1ex on June 05, 2014, 09:03:31 AM
ISO 200 captures half of the photons, compared to ISO 100, so the shot noise is 1.41x higher. In shadows, the read noise is dominant, and it changes very little from 100 to 200. In highlights, the Poisson noise is dominant, so the difference is roughly half-stop.

Of course, now it makes sense, thanks.  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: vicnaum on June 05, 2014, 03:21:26 PM
Do I have to use a tripod? Found it - yes.

How long should it take to sample data? Mine's already working a couple of minutes - maybe it halted?
(600D)

Well, tried it for a couple of times. One time it always said "RAW Error", second time it halted again.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 03:36:23 PM
Well, it just has to stay fixed while it's taking the two exposures. In LiveView it's not a problem (no mechanical movement), but outside LV it might be (since the mirror vibrations can be strong).

However, if the image is defocused, minor vibrations are unlikely to affect the results.

Computations take a few seconds, maybe 1 minute... didn't time it. If in doubt, record a video of what happens and upload it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dsManning on June 05, 2014, 03:51:47 PM
Quote from: a1ex on June 05, 2014, 09:03:31 AM
@dsManning: can you retry the LiveView tests at ISO 400 and above? They have obvious outliers.

Fixed.  Same link as above.

https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva (https://www.dropbox.com/sh/spiepm7sbts0t7t/AACu3FU3iB6GfDKlkLvwTidva)


Quote from: vicnaum on June 05, 2014, 03:21:26 PM
Do I have to use a tripod? Found it - yes.

How long should it take to sample data? Mine's already working a couple of minutes - maybe it halted?
(600D)

Well, tried it for a couple of times. One time it always said "RAW Error", second time it halted again.

Make sure image review from the Canon menu is set to hold.  I had some issues with the different image layers (plot.mo grid, graphed data, curve line) all ended up in different .ppm files.  Seemed like it didn't have the time to process when it jumped back to liveview.  MAY be your trouble.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 04:00:31 PM
This image shows very good coverage (which is important, since this fitting problem is a bit ill-conditioned).

(https://dl.dropboxusercontent.com/sh/spiepm7sbts0t7t/AACfwIVfjTbgLD6k8-ZN2KZPa/50Dmovie100.png?token_hash=AAFKHjqcr1PqI_mdu4NTTs6Vz_wMzCAG7-nZqaUhsEhSuQ&expiry=1401980306)

Are there any conflicts with the plot module? I didn't use it yet, but this refactoring is on my todo list.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dsManning on June 05, 2014, 04:11:19 PM
Quote from: a1ex on June 05, 2014, 04:00:31 PM
Are there any conflicts with the plot module? I didn't use it yet, but this refactoring is on my todo list.

No it seemed to work fine once I got the settings correct.  Before I had image review set to hold in the Canon menu I was getting the data, but I was also getting separate .ppm files in different folders under RAW_DIAG folder (IMG_0001, IMG_0002, etc).  Shown below.

https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a (https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: vicnaum on June 05, 2014, 04:16:39 PM
Well, it still hangs. With Review Hold, without hold (Off), with 2sec hold. It just draws a dozen of dots and then becomes unresponsive. It hanged with LED on, hanged with LED off.
I used 600D may 08 nightly, then cleared the card and installed clean may 13 - same thing. Btw, 600D has all builds Failed starting from May 13.

Here's the vid:


And here's what's in PPMs:
(http://www.naumik.com/temp/scr4/20140605-u82-73kb.jpg)

Btw, I also get this SNR curve (noice profile) table overlay slightly blinking in every menu (including canons own) and LV - even after restart with battery out.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 05:17:51 PM
Can you try outside LiveView?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dsManning on June 05, 2014, 05:33:48 PM
One thing I noticed while in LV was that I had to hold the half shutter till a box popped up top left of screen saying raw_diag.  Dummy Bracket didn't work for the 50D in LV.  Uploading a quick video (got a terrible curve, not on a tripod, no audio, just showing how to make it work) to show that it works in LV, just have to not use Dummy Bracket.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on June 05, 2014, 06:00:29 PM
500D Photo Mode :

(http://s14.postimg.org/m5pl178qp/100.png) (http://s14.postimg.org/rslxyob9d/1600.png)

(http://s14.postimg.org/4gxu9kwzl/16002.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 07:24:48 PM
Quote from: dsManning on June 05, 2014, 05:33:48 PM
Dummy Bracket didn't work for the 50D in LV.

Were you in movie mode LiveView, or photo mode LiveView? It matters here.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dsManning on June 05, 2014, 07:46:48 PM
Quote from: a1ex on June 05, 2014, 07:24:48 PM
Were you in movie mode LiveView, or photo mode LiveView? It matters here.

Movie mode.  Get mixed results with LV on and Dummy Bracket.  Here is another one.  This one seemed to process, but the overlays are strange if I don't do halfshutter.

https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a (https://www.dropbox.com/sh/l4dlb32q75pqwkn/AAClqaIXy6vsCF6pGyFEcQt9a)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Marsu42 on June 05, 2014, 07:57:53 PM
Quote from: a1ex on June 05, 2014, 01:23:21 AM
Wanted: your results

You keep updating the module in the repo - should we wait some more, or is this the one you want the tests to be run with? If I take the time to cover the 6d, I really don't want to do it twice :-p
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 08:01:46 PM
The changes were not essential, so you can just grab a copy and run the tests. All the previous graphs are still valid, maybe with less fancy displays.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 09:53:30 PM
Reproduced the lockup in 5D3 at 50fps and hopefully fixed it.

I made some core changes for a clean fix, but I've applied some workarounds (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag_workarounds.patch) in the precompiled binary, to allow it to run on older nightlies out of the box. So, download the binary raw_diag.mo from this post (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) (or the first post, it's the same) and it should just work.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 05, 2014, 11:27:14 PM
Ran on 5D2 with 2014-05-30 NB, 2 shots pass but get the message: "You may need to solder some RAM chips :(".

edit:
Debug shows a memory error -> shoot_malloc(29MB) failed at raw_diag.c:445, raw_diag_task.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 05, 2014, 11:36:01 PM
Solved (well, workaround; will still show error in the Debug menu, but it will work).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 06, 2014, 12:19:29 AM
OK, works fine now -.^

Here is results for ISO 100-1600 with -1/3 steps (could be useful):
-> http://www.bgimage.fr/publis_ext/ml/2014-06-05_snr2_5d2.zip (bad samples)

Note: For me there's 'something somewhere'; in regular conditions 5D2 has a FWC of ~65700e- (around 5ke- more with some tweaks) and here I get ~55600e-.
Maybe an OB' stuff who false the analyse.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 12:33:22 AM
OB stdev is only used as an initial guess (but should have no effect on the final result).

However, in your ISO 100 sample, you don't have any data points above 8 EV. This is very important for a good fit.

Here's how it should look like (notice the red dots covering the entire range) : http://www.magiclantern.fm/forum/index.php?topic=10111.msg117999#msg117999

My quick test on 5D2 at ISO 100:
(http://a1ex.magiclantern.fm/bleeding-edge/snr-fit-5d2-100.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 06, 2014, 12:49:13 AM
Quote from: a1ex on June 06, 2014, 12:33:22 AM(...)
However, in your ISO 100 sample, you don't have any data points above 8 EV. This is very important for a good fit.
(...)
Ah, it is the why.
Though a part of the surface was overexposed (burned in fact) and another well dark. As usual..
I will see later to remake a run if needs, because you look to already have 5D2 data (?).

Beside, it will be interesting to run this kind of analyse with a special mini_iso conf (to see the FWC gain) and dual_iso feature (the FWC gain should be [...!] ). -.^

note: I don't know if it's a UI restriction, but the unit for FWC is e- (e is negative actually).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 01:05:09 AM
I only ran a test shot to check that memory error. I'd prefer not to sit down and take a bunch test pictures that other people can do very easily.

FWC gain with dual ISO... should be like ISO 100 for half of the rows, and like ISO 1600 for the other half (for example). The exact SNR curve will depend on the blending algorithm (so you will need to run this experiment offline) and also on the scene contents (where it tries to balance between noise and aliasing in shadows). Running it without --alias-map should give consistent results though (since the blending factors are chosen based on input signal and OB noise levels).

What would be interesting is to run it in LiveView and figure out how many pixels are averaged when downsampling. Can you help me figure out the math for this? (that is, how the noise should behave if N sensels are averaged on the CMOS sensor, before amplification - or wherever you think Canon might do this step)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 06, 2014, 02:28:54 AM
Quote from: a1ex on June 06, 2014, 01:05:09 AM
I only ran a test shot to check that memory error. I'd prefer not to sit down and take a bunch test pictures that other people can do very easily.
OK -.^

Quote from: a1ex on June 06, 2014, 01:05:09 AM(...)
What would be interesting is to run it in LiveView and figure out how many pixels are averaged when downsampling. Can you help me figure out the math for this? (that is, how the noise should behave if N sensels are averaged on the CMOS sensor, before amplification - or wherever you think Canon might do this step)
Of course, if I can help.

I thought resizing was done by pixel groups exclusion (?). Witch is causing, with VAF (working as a median filter), aliasing/moiré problems (because of Nyquist limit) except in x5 video mode (where CMOS surface is recorded at full resolution). So in this case, pixel informations (levels, etc.) should be the same before/after downsampling resizing.

Or are you speaking about the video feed directly on the camera back screen?

Beside, I well know about noise, FWC, etc. in case of binning downsampling (hardware or not BTW) but obviously this is not the case here (?). And I'v never seen hardware CMOS average capability (looks technically improbable), even it is _maybe_ possible.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 121nilsson on June 06, 2014, 01:16:53 PM
Hello!

thought i would try this on the Eos M. Got stuck with some message on the screen.
https://drive.google.com/file/d/0B04NFrlfFHNvUi15dy1Cb19ONUU/edit?usp=sharing
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 01:18:47 PM
I've translated the raw_diag code for SNR curves into octave, so you can run the same analysis offline (including on dual iso files).

snr_curve.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/snr_curve.m) (for two test images of the same scene, taken with identical settings)
run_tests.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/run_tests.m) (for all my test images, see below)

The curve fitting results might differ a little from raw_diag (nothing to worry about).

ISO 100 and 1600:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/100.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/1600.png)

Dual ISO 100/1600, developed with different options:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/100-1600-default.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/100-1600-no-alias-map.png)
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/100-1600-fast.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/100-1600-fast-fullres.png)

The graphs were scaled to show 15 EV below white level, and the SNR axis is from -2 to +10 EV. Therefore, you can overlap any two images and compare them directly (with alt+tab, for example).

edit: also added ISO 1600 darkened by 4 EV, to match the brightness of ISO 100. Ideally, the SNR of a dual ISO image should be roughly the maximum of the two single-iso SNRs (maybe better in overlapped areas) if you don't consider the resolution loss.
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/1600-darkened.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on June 06, 2014, 04:16:01 PM
(http://s29.postimg.org/d5c3yrulz/image.png)

Trick :
(http://s29.postimg.org/uh7v6gypj/snr2.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 06, 2014, 05:24:55 PM
Some more results, this time with the Read noise and Poisson noise plots, and trying to fill the entire graph with data, like so.

(https://s15.postimg.cc/gjtlh39dn/ML_ISO_100.png)

Canon ISO (https://www.dropbox.com/sh/suf1wchza37d9ig/AABVRktQl1usGQ9fuAgGZrYta)
ML ISO (https://www.dropbox.com/sh/vz309qys8zsv7t4/AAAEk48Y87S4abAd1K26vp6Ca)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 05:40:39 PM
Nice!

Can you use the same test scene (since it gives great coverage) for some movie mode tests? I'm interested in:
- 1080p (I expect identical results at all 3 FPS settings)
- 720p
- 5x zoom (tip: use raw video grayscale preview to frame the image)

The movie mode tests may run a little slower, but they don't waste any shutter actuations, so everybody should do them. All cameras, please!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 06, 2014, 07:44:16 PM
Well, I just completed a shit load of sample images, to find the screenshot function only captured the first image, and then went on strike.  Sure enough, the first image was setting up the testing procedure and doesn't contain useful information.  Anyway, I can share my thoughts.

Thoughts on the data:
At first it appeared like movie mode was entirely Read noise limited at lower ISOs.  I suspect that movie mode is somewhat rather Read noise limited (compared to photo mode), however, the curve was extending above the top of the graph, so I can't be certain.  It needs some automatic fitting of the curve, or at least, different scales on the graph for movie mode and photo mode.

The DR is greater in movie mode then photo mode.  Now, we know the ADTG registers don't change, so if we also consider that the the images are more Read noise dominated, then I believe that this shows pixel binning.

If memory serves me correctly, the stdev was similar to photo mode, but the FWC was less.  I didn't check the read noise measurement during the capture phase.  I know, this doesn't make sense.

Around ISO 800, the curve gets clipped at 0 EV on the vertical scale.  From here, it increases back towards -2 EV with increasing ISO.  Confirmed in both 1080p and 720p modes.  Speaking of which, I assume you meant Canon resolution modes, since cropping with ML resolution changes seems pointless?

In crop mode, somewhat expected, the figures were remarkably similar to photo mode.

ADTG tweaks work!  11.5 EV @ ISO 200 with some quick tweaking.  CMOS 5 is 0x501 and CMOS 6 is 0x370 in movie mode.  ADTG6[8000] is 0x6.

Swapping in and out of crop mode, adjusting ADTG registers, taking SNR curve measurements, eventually the camera had enough, and displayed really weird images like the ones I shared in the ISO research pull request.



Thoughts on functionality:
It says "sampling data" in the top right section, but at first I thought it wasn't doing anything, I'm pretty sure for whatever reason, the graph wasn't updating on the first try..  Maybe you could change that to, "Sit tight, I'm counting your photons"!  Or something else really obvious to us common folks.

Since the data is sampled in real time, the first image in the two shot sequence is rather redundant, right?

The framing of the image on the LCD screen changes, when the SNR curve is processing.  However, the curve still reflects the original framing.

Did I mention the screenshot function only works on the first image :P



And whatever else I forgot, since having a notepad and pen with me, makes to much sense  ::)
Off to bed.  I look forward to raw_diag_movie_mode version 2 when I wake up  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 08:19:25 PM
I think I know why the screenshot works only for one image. The PPM files are named after the last movie filename...

(so, it was the last screenshot that got saved, not the first one)

Solved.

"the curve was extending above the top of the graph" => there is a second view of the same thing, called "Noise curve". This one is a little harder to overflow. Still, the overflow is only on the display, and won't affect the fitting results.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 06, 2014, 11:12:16 PM
Quote from: Audionut on June 06, 2014, 07:44:16 PM
(...)
Thoughts on the data:
At first it appeared like movie mode was entirely Read noise limited at lower ISOs.  I suspect that movie mode is somewhat rather Read noise limited (compared to photo mode), however, the curve was extending above the top of the graph, so I can't be certain.  It needs some automatic fitting of the curve, or at least, different scales on the graph for movie mode and photo mode.

The DR is greater in movie mode then photo mode.  Now, we know the ADTG registers don't change, so if we also consider that the the images are more Read noise dominated, then I believe that this shows pixel binning.
(...)
To confirm if pixel binning is done it's quite simple: RON should be the same before/after binning (only the signal value change), or with a normalized signal value the SNR should be up in function of the number of pixel binned.

PS. I re-run some test with raw_diag but I don't find a way to get good samples with artificial lights. Audionut > Is your scene is by day?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 06, 2014, 11:30:47 PM
If you know the RON and the FWC values, one set from 1:1 crop mode (5x zoom), and the other from 1080p (or 720p), can you find out how many pixels were binned? How?

(with my "pi=3" math skills, I get around 10 pixels binned in 5D3 1080p and around 15 in 720p, which doesn't look quite right)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 06, 2014, 11:53:43 PM
Lets take SNR as reference, because overall values could be hard to compare, in a simple and not so realistic example:
If in crop mode (5x) you get 3 and in 1080p you get 12 it means that you should do a bin2: 4 pixels group, hence 4x3 and so 12.

SNR increasing by power of 2 of the pixel matrix size; bin2 -> 2x2 pixels -> 4 pixels -> 4x SNR, bin4 -> 4x4 pixels -> 16x SNR...

In the real world it could have some variances, but it should be close to that.

Note: Could you anticipate the binning value by comparing the full resolution to the 1080p? I was thinking about 5760px -> 1920px, so a bin3 BTW.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 07, 2014, 05:06:41 AM
Two 9 watt LED spot lights.  I believe that LED doesn't flicker, so this way I can use any shutter value.

One LED is around 7cm (3in) from a wall, pointed almost directly at it, the other LED is around 13cm (5in) from the wall, angled slightly away to give a nice gradient.  The angled light has a small hint of saturation, the point blank light is complete saturation (useful when ADTG tweaking).  Exposure time = 1/100s @ f/4.0.
The test setup is in a dimly lit room (I would say around 2 EV (http://www.fredparker.com/ultexp1.htm#Light%20Intensity%20Chart)).  I use objects in the scene to control light spill, and in the dark area, I have a window to outside, that for all intents and purposes, is pitch black (-6 EV).  So I estimate the test scene to have around 17 EV of DR, with reasonably good gradients throughout.

The part I found most difficult, is getting a nice gradient from the light, for very good coverage in the area around 0 EV through -4 EV from sensor saturation.  I have ideas for the next test batch :)  Daytime here so I cannot recreate the scene, but I will take a bracket later tonight when I run the tests again.


One day I will learn how to light a portrait with the same care I place onto test scenes.  ::)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 07, 2014, 01:37:39 PM
Ouch, looks like you didn't update the .mo link, and I didn't check functionality.
Third time is a charm right!  Good news is, I've become really proficient at changing ISO in movie mode, without triggering the SNR curve.

This one (https://bitbucket.org/hudson/magic-lantern/commits/1e1e18ba5acef8a7bf649229f1c770de3afd74df?at=unified) causes conflicts with raw_rec and cr2hdr.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 07, 2014, 02:48:44 PM
Whoops, updated now.

No conflicts here when merging unified and iso-research. I usually compile the module directly from the branch, without merging anything.

edit: ah, you mean compile error.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 07, 2014, 03:09:59 PM
Yes I did mean compile errors  :P

Finally finished stuffing around, and here is a good bunch of results.  Enjoy!
https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa

One with a little ADTG tweaking  :D
(https://s15.postimg.cc/n7ubxme23/5_D3-1080p-_ADTG-_FE-0x3_Pre-amp-0x1x.png)

Edit, I did this series as, 1080p, crop, 720p, ADTG tweaks.  By the end, the camera was getting a little warm, and probably affecting results, I guess.  They all had a resolution width of 1080p in ML.

Crop series doesn't have the coverage that I hoped, of course, I didn't remember that I could increase resolution until after I completed the captures.  ::)

Is there anyway to determine the CMOS/ADTG settings in crop mode?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 07, 2014, 06:53:42 PM
I expect temperature to cause differences in the read noise (in this model), but I'm interested mostly in the shot noise. So it should be fine.

CMOS/ADTG settings are printed by both iso_regs and adtg_gui (if you don't change any settings). Not all of them have effect in both modes though.

On 5D3, the ADTG preamp (8/9/A/B) is 91 in 1080p, 102 in 720p and 59 in 5x/photo. My previous tests showed the unit for this one is 0.006 EV, so the differences (compared to 5x/photo) are +0.19 EV for 1080p and +0.26 EV for 720p. I don't know if these differences result in different amounts of captured highlights.

Anyway, these differences should be OK if you can work with the "pi=3" hypothesis.




From the above data, I'll try to guess the pixel binning factors from LiveView.

Pixel binning discussion continued at http://magiclantern.fm/forum/index.php?topic=16516.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 07, 2014, 07:33:20 PM
Quote from: a1ex on June 07, 2014, 06:53:42 PM
On 5D3, the ADTG preamp (8/9/A/B) is 91 in 1080p, 102 in 720p and 59 in 5x/photo.

Here, they are 59, 63, 39.  Highest values of the set.

If I reduce ADTG preamp, the curve in the highlights has a slight bend towards lower EV values on the graph.  With ADTG 888x, I can create a huge bend in the curve at the highlights.  I seem to have deleted the graph, I'll post one tomorrow.

Notice with this capture, "5D3-1080p-ADTG-FE-0x3+888x-0x37x.png", ADTG 888x has been reduced to much.


What do you suspect is happening with the clipped shadows @ ISO 800?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 07, 2014, 08:17:59 PM
Clipped shadows? Not sure, can you take some silent pictures in that mode? (a good one and a bad one should be enough).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 07, 2014, 08:27:13 PM
I haven't inspected any images (yet).  Look at the curve data.

It only happens in LV.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 11, 2014, 12:18:25 AM
Dynamic range plot from Audionut's data (5D3, movie mode, no tweaks):

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/5d3-movie.png)

(noise graphs here: https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa )

Therefore:
- 5D3 video shooters: do not be afraid of ISO 6400 in non-crop modes (the improvement is a consequence of pixel binning, see my previous post (http://magiclantern.fm/forum/index.php?topic=16516))
- DR estimated as log2(white-black) - log2(DN where SNR = 1)
- the difference beween photo and LV crop ISO appears to be 0.4 EV (LV being darker), therefore I've shifted all the LV plots by 0.4 EV to the left (needs to be confirmed by checking the clipping point; I've got the difference from the number of electrons in the noise model)
- this graph is not directly comparable to the previous ones (I believe this method is more accurate, but need to double-check the theory to be sure)


isos =      [ 100    200    400    800    1600   3200   6400  12800 ];
dr_720 =    [ 11.02  10.96  10.93  10.85  10.67  10.36  9.89  8.89  ];
dr_1080 =   [ 11.10  11.05  11.02  10.93  10.73  10.39  9.88  8.88  ];
dr_crop =   [ 11.22  11.13  11.00  10.82  10.32   9.68  8.92  7.70  ];
dr_photo =  [ 10.92  10.83  10.69  10.46  10.02   9.40  8.49        ];

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 11, 2014, 05:23:10 PM
Quote from: a1ex on June 07, 2014, 06:53:42 PM
I expect temperature to cause differences in the read noise (in this model), but I'm interested mostly in the shot noise. So it should be fine.
Indeed, analysis I made show that temperature does not significantly affects RON. Values are slightly under 1% per a 50°C variation, in the range -50°C -> +60°C, temperature recorded directly on the back side of the CMOS with a special device (the internal thermometer is not relevant because placed on the mainboard).

Beside, the thermal signal well follow the CMOS temperature, causing major noise issues when we use our cameras with a short exposure time when the temperature rises (thermal signal is _mainly_ function of the temperature and the exposure duration).

So when we made noise analysis it is important to use a process where CMOS temperature is included.

Quote from: a1ex on June 07, 2014, 06:53:42 PM(...)
From the above data, I'll try to guess the binning factors from LiveView (and I'll ask SpcCb to double-check what follows):
(...)
Very interesting data.
But I don't figure yet why RON blow up @5x with high ISO face to photo mode. Maybe a level intercorrelation inside the logic pixel intrication area because of the .8n reduction factor.

BTW I did not know there was a reduction factor in 5x video mode; I cant make tests with 5D3 here, 'does not have Bayer matrix anymore, but I'm interested to make test to see if it's the same thing with a 5D2. What kind of resolution chart should we use?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 11, 2014, 07:47:32 PM
For temperature, as long as the noise does not depend on the image signal (if it's additive), the current model will include it in the read noise. So, the FWC should not be affected by temperature variations. An easy way to tell would be to retry the first test image at the end of the experiment (so you have a screenshot with camera hot, and another with the camera cold, from which you can check the repeatability).

Possible explanations for the RON difference in 5x vs photo:

- the ISO ends up higher in photo mode (why? the sensor registers are identical though, at least those identified in this thread)
- photo exposure uses mechanical shutter, while LV exposure uses electronic rolling shutter; as a result, the photo exposure (electronically) may be longer than the actual exposure time
- the readout speeds may be different (could this cause noise differences?)

For resolution charts: you can use any of them (even some fine print that gets progressively smaller). This one is well-known: http://www.techradar.com/news/photography-video-capture/cameras/camera-testing-resolution-charts-explained-1027585
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: SpcCb on June 11, 2014, 11:30:54 PM
Quote from: a1ex on June 11, 2014, 07:47:32 PM
(...)
- the readout speeds may be different (could this cause noise differences?)
(...)
Yes, RON is function of the read speed.
It could be an explanation _or a part of_ for the RON value differences.

For example, 2~5e- RMS corresponding to a read speed about some 10E+2 MHz and 6~9e- RMS corresponding to a read speed about a couple of 10E+1 MHz.
Note; There's some other hardware characteristics who defined RON, not only the read speed (photodiode current, etc.)

Beside we have to not forget that Canon CMOS are CDS (Correlated Double Sampling), so the extra low RON that we get is 'hardware optimized' and it could be adjusted to fit on demands (specific video operations etc.).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 14, 2014, 06:03:09 PM
Quote from: a1ex on June 05, 2014, 01:23:21 AM



Wanted: your results



1. grab raw_diag.mo (http://a1ex.magiclantern.fm/bleeding-edge/raw_diag.mo) (no need to compile anything, just place it in your MODULES folder).
2. use 1/50 for PAL, 1/60 for NTSC, or any shutter speed if you don't use artificial lights.
3. point the camera to a out-of-focus HDR scene, and adjust ISO and aperture to get a good picture (both clipped highlights and fairly deep shadows).
4. enable Debug->RAW Diagnostics, and from there, select "SNR curve (2 shots)". Disable the other analyses types (unless you want them). Make sure "Auto screenshot" is enabled (it is by default).
5. take the two test pictures from Debug -> RAW Diagnostics -> Dummy bracket. Simply click it once and wait until it's finished.
6. post the screenshot (it's called SNR2.PPM and it's saved under the RAW_DIAG folder; convert it to PNG first to make it smaller, and give it a meaningful name).

I'm looking for results covering all ISOs, for all cameras, both photo and movie mode.

If you wish to review the code or compile yorself, look here (https://bitbucket.org/hudson/magic-lantern/commits/c4b81743843f).

Have fun!

Canon 6d - all (native) iso's both photo and video mode(1080P), (filename of the png's are describing which mode and which iso):

https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing

:D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 15, 2014, 07:42:31 PM
See my results in the link posted before this post:

In the lower iso's the full well capacity is about 3,4 times larger in video mode(1080p) VS photo mode on the 6d
In the higher iso's the full well capacity is about 4 times larger in video mode(1080p) VS photo mode on the 6d

Don't know if my understanding about full well capacity is correct, but could that mean that the 6d in video mode uses a 4 pixel binning method ?

List of the results posted in the link before this post

                                            Canon 6D
         Full Well Capacity Dyn.Range             Full Well Capacity Dyn.Range
ISO                Photo mode                                      Video Mode(1080p)
100           79546e           11,48EV                   274301e             11,84EV
200           42220e           11,49EV                   142616e             11,82EV
400           20643e           11,28EV                     75858e             11,75EV
800           10482e           11,00EV                     39219e             11,62EV
1600           5215e           10,55EV                     19868e             11,38EV
3200           2568e             9,87EV                     10092e             10,95EV
6400           1271e             9,11EV                       4931e             10,47EV
12800           556e             8,04EV                       2408e               9,44EV
25600           371e             7,10EV                         X                      X

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on June 15, 2014, 08:20:37 PM
Quote from: Levas on June 15, 2014, 07:42:31 PM
Don't know if my understanding about full well capacity is correct, but could that mean that the 6d in video mode uses a 4 pixel binning method ?

5472 / 4 = 1368
5472 / 3 = 1824

3 is more likely
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 15, 2014, 08:28:49 PM
Quote from: Greg on June 15, 2014, 08:20:37 PM
5472 / 4 = 1368
5472 / 3 = 1824

3 is more likely

Ah, forgot about the real world logical numbers... ::) ML raw video live view feed is 1808 pixels wide.
3 pixel wide binning method (and lineskipping) is the most likely indeed.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 15, 2014, 09:56:33 PM
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).

Tip: turn off global draw to get clean screenshots in LiveView.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 15, 2014, 10:19:05 PM
Quote from: a1ex on June 15, 2014, 09:56:33 PM
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).

Tip: turn off global draw to get clean screenshots in LiveView.

Ah I already was wondering how I could get rid of the time and histogram... ::) on those screenshots, Will turn global draw off.
I think I can do the rest of the video modes tomorrow (It's 10 in the evening over here).

But just to be sure which modes you need:
In live view in 1080p mode I have 5x zoom  and 10x zoom.
In 720p mode there's also 5x zoom and 10 x zoom.
The 5x zoom turns automatic to ML gray scale preview, 10x zoom is in full color preview.

If I do the 720P (without zoom) and 1080p mode with 5x zoom, would those 2 complete the numbers you'd like to see ?

Oh, forget to mention, there's also a 640x480 25fps mode in the canon menu on the 6D, when I choose it, the framing doesn't change.
So or canon is writing/saving it to a 640x480 format on card or it uses another binning modes, maybe this video mode is interesting too perform the RAW_DIAG test on too...

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 15, 2014, 10:22:36 PM
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 15, 2014, 10:24:40 PM
Quote from: a1ex on June 15, 2014, 10:22:36 PM
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.

Oh, forget to mention, there's also a 640x480 25fps mode in the canon menu on the 6D, when I choose it, the framing doesn't change.
So or canon is writing/saving it to a 640x480 format on card or it uses another binning mode, maybe this video mode is interesting too, to perform the RAW_DIAG test on...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 16, 2014, 01:28:07 PM
Quote from: a1ex on June 15, 2014, 09:56:33 PM
Can you do the tests in all video modes? (I'm interested in 5x zoom, 1080p and 720p - does the 6D have other video modes?) I guess you should get a ratio closer to 3 if you compare 1080p with 5x zoom (which is also a 1:1 crop).

Tip: turn off global draw to get clean screenshots in LiveView.

Got the numbers for 5xzoom mode on canon 6d
will do 720 mode soon, couldn't find a good spot with HDR, very cloudy over here
(got not so nice RAW-DIAG graphs, dots in graphs where not nice linear, random dots in highlights)

         Full Well Capacity Dyn.Range             Full Well Capacity Dyn.Range         Full Well Capacity Dyn.Range
ISO                Photo mode                                      Video Mode(1080p)              Video Mode(5xzoom)
100           79546e           11,48EV                   274301e             11,84EV            97890e           11,78EV
200           42220e           11,49EV                   142616e             11,82EV            48335e           11,72EV
400           20643e           11,28EV                     75858e             11,75EV            25171e           11,59EV
800           10482e           11,00EV                     39219e             11,62EV            12958e           11,29EV
1600           5215e           10,55EV                     19868e             11,38EV              6508e           10,85EV
3200           2568e             9,87EV                     10092e             10,95EV              3354e           10,25EV
6400           1271e             9,11EV                       4931e             10,47EV              1663e             9,65EV
12800           556e             8,04EV                       2408e               9,44EV                831e             8,51EV
25600           371e             7,10EV                         X                      X                        X                     X

Full well capacity of 1080p divided by 5x zoom results in the following numbers (looks like 3 pixels are binned)
ISO - 1080p/5xzoom
100 - 2,8
200 - 3,0
400 - 3,0
800 - 3,0
1600 - 3,1
3200 - 3,0
6400 - 3,0
12800 - 2,9

Will try to find a good HDR spot for 720 mode
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 16, 2014, 01:52:23 PM
 :)
Updated this folderlink with the 5xzoom RAW-DIAG images.

https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 16, 2014, 01:57:58 PM
I noticed in the data above that the dynamic range in 5x zoom video mode is still 0.5 stop better then photo mode (almost equal to 1080p video mode).
There should be no pixelbinning in the 5x zoom mode, how come the dynamic range is still better compared to photo mode (Full well capacity is also higher) ?
Is this difference caused by mechanical shutter VS electronic shutter ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on June 16, 2014, 03:44:52 PM
Quote from: a1ex on June 15, 2014, 10:22:36 PM
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.

Hi Alex,

Tried various setups, but I can't get a decent blue line plotted with RAW_DIAG in 720p video mode.
The blue dots are scattered all over the place in the graph, does that mean that the images aren't alike ?
I use the dummy bracket function, but I use available light from outdoors on a cloudy windy day, maybe that doesn't work too well... :-\
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 16, 2014, 05:20:44 PM
Ideally the scene should be extremely static.  A cloudy windy day, probably doesn't satisfy those requirements.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 24, 2014, 06:13:28 PM
I don't have a printer, I might be able to access (a decent) one over the weekend, and printout a target to capture. 

Haven't been able to find anything around the house that seems suitable.   ::)
Title: Questions
Post by: astronomer on July 01, 2014, 11:24:34 PM
Hi all,

This is astronomer from NikonHacker. I had been investigating the Sony sensor for months. Could someone explain to me what's the difference between white offset and black level offset in this thread? I'd expect the black level and gain are the only 2 values to fix a straight line. Some tests done by others suggested Canon did offset to correct dark current and bring it back to 2048. I thought this is done digitally as the saturation point for dark frame is not 16383. But it seems contradict some of the posts here where you guys suggest an additional feedback loop during optical black pixel read out.

Sony sensor has all the analog chain integrated so it's much simpler. A register to set black level and 4 gain registers for each color channels. The sensor output spans from 0 to 16382. All the rest are done in digital domain, such as color channel scaling, black level clip (dark current correction), large aperture light loss scaling, etc.

Best,
astronomer
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 02, 2014, 03:00:38 AM
The black level is set to some value above 0, to allow the stdev to be easily captured.  This is controlled by a voltage feedback loop at the ADC, ADTG[8880].

The saturation point is controlled by a number of registers.  On a 5D3, I'll describe the ones I understand.
Analog domain.
ADTG[8,9,a,b] + ADTG[888x] - These control the (individual) column gains.  There are two banks of 4, ADTG2 + ADTG4.
ADTG[fe] - This is like some master gain control.

The native (Canon default) saturation level is 15283.  Increasing analog gains does not increase this value, only decreases dynamic range.  We can decrease these analog gains to recover highlight detail, without affecting the saturation level.

And the input to this ADTG is CMOS[0].

In the digital domain.
Saturate offset - I think a1ex mentioned this may be analog.  edit see here: http://www.magiclantern.fm/forum/index.php?topic=10111.msg103971#msg103971
Black/White offset
Digital gain - Canon uses this one to offset the light loss at wide apertures (http://www.luminous-landscape.com/essays/an_open_letter_to_the_major_camera_manufacturers.shtml).
Title: Re: Questions
Post by: a1ex on July 02, 2014, 07:45:30 AM
Quote from: astronomer on July 01, 2014, 11:24:34 PM
This is astronomer from NikonHacker.

Hi, I've been following your thread for a while. For the record, the link is https://nikonhacker.com/viewtopic.php?t=1840

Quotewhat's the difference between white offset and black level offset in this thread?

If you are asking about the registers that can be tweaked, you can find them documented in the iso-regs module:
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/iso_regs/iso_regs.c#cl-583

(we simply changed them, written down the observed effects, and then guessed what they might do)

If you are asking about how to interpret black level and white level, it's as simple as:

normalized = (raw - black_level) / (white_level - black_level)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 02, 2014, 11:56:51 AM
Quote from: a1ex on April 26, 2014, 02:58:21 PM
ADTG 82F3 (the one that sets top optical bar size)

For 5D2, the equivalent register is 0x1179.

(my theory is that increasing top bar size could help with correcting vertical FPN, but need to sit down and figure out the details)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 02, 2014, 02:54:21 PM
Quote from: a1ex on June 15, 2014, 10:22:36 PM
5x zoom is the same in all modes (at least on my cameras). So, adding 720p and 5x to the existing graphs will do the trick.

Hi Alex,

Do you still have interest in the graphs for 720p mode on 6D ?
Can give it another try if it's useful
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 02, 2014, 03:00:15 PM
Sure.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 02, 2014, 07:02:52 PM
Quote from: a1ex on July 02, 2014, 03:00:15 PM
Sure.

Nice bright sunny day over here, so here are the numbers, now also added 720P video mode:
all the graphs can be found on my google drive link:
https://drive.google.com/folderview?id=0B1BxGc3dfMDaUjlnaENVZ2drYUU&usp=sharing

CANON 6D
       Full Well Capacity Dyn.Range        Full Well Capacity Dyn.Range         Full Well Capacity Dyn.Range       Full Well Capacity Dyn.Range
ISO                Photo mode                             Video Mode(1080p)              Video Mode(5xzoom)                         Video Mode(720p)
100           79546e           11,48EV         274301e             11,84EV            97890e           11,78EV                  242833e        11,87EV
200           42220e           11,49EV         142616e             11,82EV            48335e           11,72EV                  110182e        11,85EV
400           20643e           11,28EV           75858e             11,75EV            25171e           11,59EV                    66885e        11,81EV
800           10482e           11,00EV           39219e             11,62EV            12958e           11,29EV                    36324e        11,67EV
1600           5215e           10,55EV           19868e             11,38EV              6508e           10,85EV                    18721e        11,40EV
3200           2568e             9,87EV           10092e             10,95EV              3354e           10,25EV                      9748e        10,96EV
6400           1271e             9,11EV             4931e             10,47EV              1663e             9,65EV                      4874e        10,48EV
12800           556e             8,04EV             2408e               9,44EV               831e             8,51EV                       2380e         9,35EV
25600           371e             7,10EV               X                      X                        X                     X                             X               X

The numbers of the 720p mode are almost identical as the ones in 1080P, so no extra binning, just extra row skipping I guess ?


Title: Re: Questions
Post by: astronomer on July 03, 2014, 12:40:17 AM
Quote from: a1ex on July 02, 2014, 07:45:30 AM
If you are asking about the registers that can be tweaked, you can find them documented in the iso-regs module:
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/iso_regs/iso_regs.c#cl-583

(we simply changed them, written down the observed effects, and then guessed what they might do)


OK, correct me if my understanding is wrong.
1. CMOS gain is the analog gain inside the sensor itself. So amplified signal in high iso will avoid noise contamination when the signal wire run across the PCB until it hits PGC/ADC chip?
2. All other ADTG gain are to fine tune the voltage range input to ADC?
3. Digital Gain is used when a larger aperture is selected?
4. From that image, it appear the saturation point is analog adjust but black level is offset in post processing?

This really contradicts what I understand in a Sony Exmor sensor.

In Sony sensor, the OFFSET 0x1F register sets the input number to bias voltage. It calibrated the ADC bias voltage so that the dummy pixels would be read out as your OFFSET value. During the first 50 rows, sensor will continue refine this bias level to within 1 ADU using dummy readout. Regardless of OFFSET and gain setting, the saturation point always reach 16382. It's like adjusting the intercept of a linear function.

Dark current will be registered on optical black pixels and all the active pixels as well. So it will show an elevated bias. This is where it differs from Canon. Somehow it clamped the dark current to bias level.

The stock Nikon firmware adjust the dark current to 0 using those optical black pixels in post processing. Thus saturation level will be pulled down. I'm curios to know if it's possible for Canon to disable dark normalization as well?

This is interesting since some Nikon DSLRs also uses external ADCs. D700/D3 uses AD9974.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on July 03, 2014, 08:57:40 PM
Testing ADTG module outdoors  :D

500D, the same exposure

(http://s28.postimg.org/6bh5yhfa5/image.jpg)

(http://s28.postimg.org/gmtir56zh/image.jpg)

(http://s28.postimg.org/n1x51k331/image.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 24, 2014, 11:08:18 PM
Trying to combine the results from 5D3 and 6D (data from Audionut (http://www.magiclantern.fm/forum/index.php?topic=10111.msg118220#msg118220) and Levas (http://www.magiclantern.fm/forum/index.php?topic=10111.msg120656#msg120656), no ISO tweaks applied).

Notice the 6D captures quite a bit more electrons than 5D3. You may say it's normal, because the 6D has lower resolution, so it captures more photons per pixel. However, the pixel count ratio is (5796*3870)/(5496*3670) = 1.1121, while the electron count ratio is about 1.1988. There are 0.11 EV not explained by the resolution difference. Let's find out what's going on.

Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above

Theory:
- photon to electron ratio = quantum efficiency (assumming it's a sensor constant)
- equal ISOs = equal number of photons per area (light required to saturate the sensor)
- ISO doubles => half as many photons (or electrons) captured
- N pixels binned => number of photons (or electrons) per pixel increase N times

Assumming Canon full-stop ISOs in photo mode match the ones measured by some other method (let's say DxO), we can find a scaling factor for converting our electron counts into ISO, while matching the reference values in the least squares sense. I'll minimize the ISO difference, in EV, for ISO 100..6400:

function err = crit_electrons(ref_electrons, measured_electrons, reference_isos)
    isos_estimated  = 100 ./ (measured_electrons / ref_electrons);
    err = norm(log2(isos_estimated) - log2(reference_isos));
end

function ref_electrons = fit_electrons(measured_electrons, isos_dxo)
    ref_electrons = fminsearch(@(e) crit_electrons(e, measured_electrons, isos_dxo), 50000);
    ref_electrons = round(ref_electrons);
end


Assuming Canon's full-stop ISOs are correct, the scaling factors are:

isos                = [ 100   200    400    800    1600   3200   6400  ];
electrons_photo_5d3 = [ 67684 34146  16835  8735   4288   2197   1085  ];
electrons_photo_6d  = [ 79546 42220  20643  10482  5215   2568   1271  ];
ref_electrons_5d3   = fit_electrons(electrons_photo_5d3, isos);
ref_electrons_6d    = fit_electrons(electrons_photo_6d, isos);


=>

ref_electrons_5d3 =  68787
ref_electrons_6d =  82465
ref_electrons_6d / ref_electrons_5d3 =  1.1988


Now I'm going to assume Canon's full-stop ISOs are pretty much correct, but I'll keep the relative differences measured by DxO between the two cameras. That difference is...

octave:1> isos_dxo_5d3        = [ 80    160    323    641    1280   2518   5179  ];
octave:2> isos_dxo_6d         = [ 80    153    311    616    1210   2400   4991  ];
octave:3> isos_dxo_5d3 ./ isos_dxo_6d
ans =
   1.0000   1.0458   1.0386   1.0406   1.0579   1.0492   1.0377
octave:4> log2(mean(isos_dxo_5d3 ./ isos_dxo_6d))
ans =  0.054522


=> 6D ISOs are 0.055 EV lower than 5D3's.

Let's distribute the ISO difference equally between the two cameras, and redo the fit:

iso_difference      = log2(mean(isos_dxo_5d3 ./ isos_dxo_6d));
ref_electrons_5d3   = fit_electrons(electrons_photo_5d3, isos * 2.^(iso_difference/2));
ref_electrons_6d    = fit_electrons(electrons_photo_6d, isos * 2.^(-iso_difference/2));


=>

ref_electrons_5d3 =  70099
ref_electrons_6d =  80927
ref_electrons_6d / ref_electrons_5d3 =  1.1545


Getting closer. The ratio is still a bit higher than what I expect from the resolution difference, and since I don't have any other hypothesis to explain it, I'll say the 6D's quantum efficiency is a little higher:


qe_ratio = log2(ref_electrons_6d / ref_electrons_5d3 * (5496*3670) / (5796*3870))
0.053998


So, 6D's quantum efficiency seems to be 0.054 EV better than 5D3's, and its ISOs seem to be calibrated at 0.055 EV lower.

With these numbers, the ISOs estimated from electron counts become:

5D3:

isos_720    = [ 100   163    326    644   1266   2519   5211   10277 ]
isos_1080   = [ 82    160    316    622   1243   2429   5059   10049 ]
isos_crop   = [ 80    157    318    630   1241   2495   5094   9777  ]
isos_photo  = [ 104   205    416    803   1635   3191   6461         ]


And 6D:

isos_720    = [ 100   220     363   668   1297   2491   4981   10201 ]
isos_1080   = [ 89    170     320   619   1222   2406   4924   10082 ]
isos_crop   = [ 83    167     322   625   1244   2413   4866   9739  ]
isos_photo  = [ 102   192     392   772   1552   3151   6367   14555 ]


Graphs:

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/5d3-movie.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/6d-movie.png)

To compare the two graphs, I think it's better to normalize the results to some reference output size. I'll normalize movie 1080p results to 1920x1080 and photo mode results to 8 megapixels.

(http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/5d3-vs-6d-movie.png) (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/5d3-vs-6d-photo-8mp.png)

Normaling the 6D results from 1792x1008 to 1920x1080 gives the 5D3 0.1 EV advantage.

Normalizing to 8 megapixels gives the 5D3 a 0.075 EV advantage (compare with DxO results):

norm_5d3 = log2(sqrt((5796*3870))) - log2(sqrt(8e6))
norm_6d  = log2(sqrt(5496*3670)) - log2(sqrt(8e6))
norm_5d3 =  0.74370
norm_6d  =  0.66708
5D3 normalized DR:   11.6637   11.5737   11.4337   11.2037   10.7637   10.1437    9.2337
6D normalized DR:    12.1471   12.1571   11.9471   11.6671   11.2171   10.5371    9.7771    8.7071


Complete octave script: dr.m (http://a1ex.magiclantern.fm/bleeding-edge/iso50/snr/dr.m)

Please double-check my math, as I might be a little beyond Ballmer peak :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on July 24, 2014, 11:32:44 PM
Quote from: a1ex on July 24, 2014, 11:08:18 PM
Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above

add:
- 6D's CFA lets more light pass (e.g. wider spectrum for the sensel filters)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 25, 2014, 12:17:31 AM
I think we can include this one in the quantum efficiency.

Updated the graphs to include the shot noise limit (maximum dynamic range, if the read noise would be zero). At high ISO, you can see that, while in photo mode, the maximum improvement that can be obtained by reducing the read noise is around 1.2 stops for 6D and 1.6 stops for 5D3, in movie mode, with 3x3 pixel binning, the maximum theoretical improvement would be over 4 stops.

So... who said we are limited by shot noise at high ISOs? :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dubzeebass on July 25, 2014, 02:29:05 AM
How far are we from implementation, even as a pre-alpha?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 25, 2014, 06:59:38 AM
I'll tell you after receiving the DR test results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) from at least 3 more cameras, and LV binning results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg118232#msg118232) from at least 2 cameras, one of them being 5D3.

None of the tests require you to compile anything; you only need to take some test pictures in controlled environments.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dubzeebass on July 25, 2014, 01:19:40 PM
I can help, have a 5Dmk3. What do I need to do?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 25, 2014, 01:39:28 PM
I can help for 5D III and have a High Resolution chart and color checkers if that helps. I'll pop them out today.

Are you looking for detail cause I can use a nice macro lens and get close or does it not matter?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 25, 2014, 02:02:45 PM
Doesn't matter; as long as you get the resolution chart details above the Nyquist frequency, it's fine.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 25, 2014, 02:18:20 PM
Quote from: a1ex on July 25, 2014, 02:02:45 PM
...Nyquist frequency...

Even after looking it up I'm still lost, so yeah ummm... I'll take some pics/video. I'll assume for 720/1080 I should use Canon .mov or RAW?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 25, 2014, 02:24:11 PM
Silent pictures, with "Silent zoom bracket" from raw_diag. That way, there will be no camera movement at all.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 25, 2014, 02:29:49 PM
After looking at it I get it now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 25, 2014, 06:20:49 PM
Hope they help...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 25, 2014, 06:34:40 PM
Quote from: a1ex on July 24, 2014, 11:08:18 PM

Possible causes:
- systematic errors in my measurement method
- 6D uses lower ISOs than 5D3 (even if they both print the same value)
- 6D's sensor has a higher quantum efficiency
- a combination of the above


Hi Alex,

Some of my data, shot on the 6d, is done with aperture f2.0
Later on I remembered reading somewhere on the forum here that canon automatic adds some gain by apertures below f2.8
Can this explain some of the differences measured between 6d and 5dIII
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 25, 2014, 06:44:22 PM
Also...

The 6d is known to have less banding noise then 5d3.
Is banding noise caused by "wiring" on the cmos ?
So does less banding noise mean, less "wiring" on the cmos chip, which results in more room to capture electrons...  ???
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 25, 2014, 06:53:38 PM
I don't expect that gain to influence the results, unless it's clipping useful data (and that happens beyond f1.4 IIRC). And in that case, the reported DR would be a little lower.

If in doubt, you can double-check a few of the numbers.

I don't know the cause of the FPN, but I'm making progress in understanding how Canon corrects it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 10:17:04 AM
5D3:  Pixel pitch= 6.25 microns.
6D:    Pixel pitch= 6.58 microns.

Bigger pixels are like bigger buckets, take longer to fill, but hold more water.  Data from Rodger Clark.

I wasn't ware of resolution playing a role, per se.  Since resolution is a count of the number of pixels, not how much data each pixel captures.  What am I missing?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 28, 2014, 10:19:40 AM
Yes, but still, the 6D holds even more water than you would expect from different bucket sizes.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 10:41:06 AM
Well if we assume that the reported pixel sizes are correct, and that Canon hasn't developed a way to squeeze photons, then it must be electrical efficiencies.
The only other possible causes I can think of. 

Shielding/copper efficiency: less electrons lost during transfer.
A/D improvements.  They both have the same 14bits of accuracy, but for all we know, Canon may be simply clipping some data in the 5D3.

Need to loan DxO' ISO rig. :D

Quote from: g3gg0 on July 24, 2014, 11:32:44 PM
add:
- 6D's CFA lets more light pass (e.g. wider spectrum for the sensel filters)

I'm not sure how this one can play a role, since this doesn't effect what happens after the photons are in the wells.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 28, 2014, 10:42:44 AM
In the wells you get electrons, not photons.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 10:46:41 AM
Where do the photons become electrons?

I always thought the wells captured photons, like buckets capture water, and then in simple terms, those photons are counted and converted into electrons.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 28, 2014, 10:56:31 AM
Somewhere before the electronic amplifiers. These amplifiers gets saturated, from too many electrons.

(it might be different at the lowest ISO, not 100% sure, but at higher ISOs, the highlights are clipped in by the amplifiers)

There is a ratio between the number of captured photons, and the number of electrons that reach the amplifiers; that one is called quantum efficiency, is usually less than 1:1, and I don't have a way to measure it. My method only counts the number of captured electrons.

http://www.clarkvision.com/articles/digital.sensor.performance.summary/
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 11:25:48 AM
You can determine accurate exposure time with the FPS registers now?
So you need a way to accurately measure the photons emitted per time period for your light?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 28, 2014, 11:36:24 AM
In photo mode, I can determine the time when shutter is "open" electronically, which is usually longer than the actual exposure time (limited by mechanical shutter). So, I can determine the exposure time with better accuracy by recording the sound of the shutter :P

Measuring photons... is way beyond my knowledge.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 01:29:48 PM
Counting photons may be something SpcCb can share advice on.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 28, 2014, 02:48:27 PM
5d3 has a CMOS chip which is capable of vertical pixelbinning, so it doesn't need vertical lineskipping like the 6d does in video mode.
I assume, that vertical pixelbinning is only possible by more copper-wiring/infrastructure within/on the cmos chip, which results in a lower full well capacity  ???

Quote from: Audionut on July 28, 2014, 10:17:04 AM
I wasn't ware of resolution playing a role, per se.  Since resolution is a count of the number of pixels, not how much data each pixel captures.  What am I missing?

Bigger pixels on the same area (full-frame), means bigger wells, higher full well capacity.
I think it's the same like the 3x3 pixelbinning that the 5d3 does, full well capacity raises in videomode by a factor 9.
So if you had a 2 megapixel full-frame chip from this generation, you're full well capacity would be enormous, like 900000.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 04:38:25 PM
Yeah I'm pretty sure I'm just being pedantic, since resolution doesn't care about well depth.

Quote from: a1ex on July 24, 2014, 11:08:18 PMNow I'm going to assume Canon's full-stop ISOs are pretty much correct, but I'll keep the relative differences measured by DxO between the two cameras. That difference is...

octave:1> isos_dxo_5d3        = [ 80    160    323    641    1280   2518   5179  ];
octave:2> isos_dxo_6d         = [ 80    153    311    616    1210   2400   4991  ];
octave:3> isos_dxo_5d3 ./ isos_dxo_6d
ans =
   1.0000   1.0458   1.0386   1.0406   1.0579   1.0492   1.0377
octave:4> log2(mean(isos_dxo_5d3 ./ isos_dxo_6d))
ans =  0.054522


=> 6D ISOs are 0.055 EV lower than 5D3's.

But only at gains higher then base?  Since the base gain is generating the same ISO on both cameras, this should mean that they capture the same amount of photons.  Or rather, that since the 6D has an higher rated well capacity, that it generates more electrons for the same number of photons.  Since ISO based on saturation, should be representing a fixed photon count, right?

edit:  Nevermind, I reread your post.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 28, 2014, 05:24:00 PM
Don't know if this is correct, but the way I think about it is:

A full frame CMOS chip has han area of 36mmx24mm= 864mm2 to catch light.
Since CMOS has circuitry inside the chip, this circuitry is blocking light (that's why backside illuminated CMOS sensors are slightly better in catching light, backside illumination prevents the circuitry from blocking light).
By normal CMOS, the pixel wells are below the circuitry.
So due to circuitry the active area of CMOS full-frame is a little lower then 864mm2

Now my assumption is, that the 5d3 CMOS chip has more/bigger circuitry, (it can do some more tricks than the 6D CMOS, like 3x3 pixelbinning).

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 06:43:19 PM
Yes, but a fixed ISO rating from DxO should be saying, for a constant N of photons, the 5D3 generates N electrons, and the 6D generates N electrons, at full well capacity.  By measuring the ISO at saturation, I believe this negates any influences by sensor filters.  With the sensor filter removed, this should lower the ISO rating, for the same quantum efficiency.

Since we know that even the base gains have some (ISO) headroom between the default gain, and actual well saturation (and seems to vary with camera), this should probably be taken into account.  And should drive the accuracy of the quantum efficiency results higher, since it's less affected by downstream influences. 

If you're looking at sensor efficiency, rather then camera efficiency, I guess.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 28, 2014, 07:20:16 PM
Ah I get what you mean, the circuitry has the same effect as any difference in the color filter array.
But since things are measured with saturated pixels, the filters/circuitry don't have effect on the end results.
So the difference in full well capacity between the 5d3 and 6d is not due to difference in color filter array or circuitry.

The way Alex measures the full well capacity, what does the result mean:
-Is the full well capacity result the best possible capacity a pixel can have.
or
-Is it an average amount, like all electrons divided by the number of pixels.
?

Since I don't believe every pixel has exact the same full well capacity, or has it  ::)
Now I think about it, maybe that's the reason why canon is staying on the save side with ISO/dynamic range.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 28, 2014, 07:55:31 PM
Quote from: Levas on July 28, 2014, 07:20:16 PM
So the difference in full well capacity between the 5d3 and 6d is not due to difference in color filter array or circuitry.

No, because full well capacity is a measure of the maximum number of electrons generated.  And the photon to electron witch craft happens after the filter array.


Quote from: Levas on July 28, 2014, 07:20:16 PM
The way Alex measures the full well capacity, what does the result mean:
-Is the full well capacity result the best possible capacity a pixel can have.
or
-Is it an average amount, like all electrons divided by the number of pixels.
?

Since the maths doesn't account for the number of pixels, I assume it's a measure of the entire sensor.
edit:  That doesn't make sense since there are significantly more pixels then the electron count.
http://www.clarkvision.com/articles/digital.sensor.performance.summary/#full_well

Quote from: Levas on July 28, 2014, 07:20:16 PM
Since I don't believe every pixel has exact the same full well capacity  ::)

I believe flat fields (http://en.wikipedia.org/wiki/Flat-field_correction) will correct variations in pixel output.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on July 29, 2014, 10:59:15 AM
Quote from: a1ex on July 24, 2014, 11:08:18 PM
....
- 6D's sensor has a higher quantum efficiency
...
Theory:
- photon to electron ratio = quantum efficiency (assumming it's a sensor constant)
- equal ISOs = equal number of photons per area (light required to saturate the sensor)
...

I think that equal ISOs = equal relative saturation (electrons counted/full saturation level ).

At least, Dxo's measures are using this 
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity
"... relates sensitivity to the exposure necessary to saturate the camera"

Higher QE gives higher "Dxo measured ISOs" for the same electron capacity ..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 11:24:48 AM
Exposure, in this context, is proportional to the number of captured photons per area (in the sensor's plane). On the same sensor, this translates, indeed, to equal relative saturation (electrons counted / full saturation level), but on different cameras, at the same ISO, both sensors would saturate at the same level of incoming light (photons), if the shutter speeds are equal.

A sensor with higher QE can be used with lower internal amplification, so the ISO value measured by DxO can be lower. If QE's were equal, I expect DxO ISOs to be even lower on 6D, because the difference in electron capacity is quite high (higher than what you would get from different pixel sizes).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on July 29, 2014, 02:03:18 PM
Alex, as I understand it, DxO does not measure the amplification but the output sensitivity. i.e. for an A level of illuminance how high in the 0-1 scale (0=black - 1=full saturation) is the output in raw levels and compares it to the nominal value.

Say for the same illumination
5D4 has QE=50% and saturation at 10000 electrons/square micron. Gives as output 0.5 in the 0-1 range (white clipping level - Black level) which corresponds to 5000 electrons
6D2 has QE=55% and saturation at 11000 electrons/ square micron. Gives as output 0.5 in the 0-1 range which corresponds to 5500 electrons

the two cameras will have the same dxo-measured ISO
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 02:06:46 PM
And that level of luminance is proportional to the number of photons.

I'm not sure where you are trying to get.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 29, 2014, 02:26:28 PM
Quote from: a1ex on July 25, 2014, 06:59:38 AM
I'll tell you after receiving the DR test results (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955) from at least 3 more cameras,

Finally sunny after 3 days of rain and overcast/100% humidity good day for HDR... if my last attempt to help with LV Binning wasn't a dud I'll give this one a shot today. Should I bother? :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 02:31:47 PM
Last attempt to help with LV binning? Where?

I don't see much else, other than...

Quote from: Stedda on July 25, 2014, 02:29:49 PM
After looking at it I get it now.

Quote from: Stedda on July 25, 2014, 06:20:49 PM
Hope they help...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 29, 2014, 02:35:07 PM
I PMd you a link with a Dropbox .zip 100 megs of shots that day...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 29, 2014, 02:36:59 PM
Dope.. sent it to alex not a1ex

Pm comming from dummy now...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 02:48:33 PM
Thanks, received now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 05:11:17 PM
Pixel binning test results moved here: http://magiclantern.fm/forum/index.php?topic=16516.msg160832#msg160832
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 29, 2014, 05:14:52 PM
Glad it worked out... I'll do the other test and get you results for tomorrow...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IliasG on July 29, 2014, 05:33:47 PM
Quote from: a1ex on July 29, 2014, 02:06:46 PM
And that level of luminance is proportional to the number of photons.

I'm not sure where you are trying to get.

Forgive me :) .. while my eyes were reading "photons" my brain translated it to "photoelectrons" ..

My point is that QE cannot affect the full saturation level .. 

I will reread the thread and come back if I have something useful to say ..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 29, 2014, 05:37:30 PM
Yes, QE alone can't affect the saturation level, but if you increase it and you want to keep the same ISO, you will have to turn down the amplifier gains - and this increases the full well capacity.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on July 29, 2014, 06:21:37 PM
500D, trying to increase the resolution of the video raw (crop mode).

Default :
http://s29.postimg.cc/ugjr8rxet/res1x.jpg

I changed (FPS timer A, C0F0[6084], C0F0[6088])
http://s29.postimg.cc/isuaryxnp/res2x.jpg

Add 512px width :
http://s4.postimg.cc/s4sbor8iz/res4x.jpg

Add 2048px and shift :
http://s16.postimg.cc/mpui9r4sz/res5x.jpg


Full width :
http://s27.postimg.cc/be7g4qaz5/full_width.jpg

(http://s29.postimg.cc/dtk387c7b/VRAM0.png) (http://s29.postimg.cc/mpzx99953/VRAM1.png)
(http://s28.postimg.cc/6uvo2lhrh/VRAM0.png)

Trying to change the height (photo mode):
(http://s18.postimg.cc/4tp8g2zyh/ob_zones.png) (http://s18.postimg.cc/ursum45ft/ob_zones2.png)
(http://s18.postimg.cc/gzefqheo9/VRAM0.png) (http://s18.postimg.cc/ibw4sdc3t/VRAM1.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: glubber on July 30, 2014, 08:08:26 PM
Well it took me forever to realise that i had to put "image review" to "HOLD"  :-\

But finally:

SNR for 550D ISO 100 Photomode:

(https://dl.dropboxusercontent.com/u/90483500/MAGICLANTERN/550D-ISO100-50-Photo.png)

SNR ISO 100 Moviemode

(https://dl.dropboxusercontent.com/u/90483500/MAGICLANTERN/550D-ISO100-50-Movie.png)


SNR ISO 1600 Photomode

(https://dl.dropboxusercontent.com/u/90483500/MAGICLANTERN/550D-ISO1600-50-Photo.png)


I hope those above make sense
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 30, 2014, 08:22:45 PM
The first image is almost perfect, the second is incorrect (very poor coverage in highlights), the third is fine, but could have better coverage in shadows.

If you can do a complete set (photo, movie 1080, movie 720, and 5x zoom), would be great.

Side note: I will also use this data for optimizing exposure choices in ETTR.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 30, 2014, 08:56:07 PM
A1ex, if I shot this out in the sun with a full set for 5D III would it suffice?


Spyder Cube
https://search.yahoo.com/search;_ylt=AwrBTzgKP9lTAhIA.x5XNyoA?ei=UTF-8&type=599486&fr=chr-greentree_ff&ilc=12&p=spyder+cube&SpellState=&fr2=sp-qrw-corr-top
Specular Highlights, White, Black Trap, 18% Grey
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 30, 2014, 08:59:55 PM
I'm looking for continuous coverage from deep shadows to clipped highlights, and static scene (lack of motion or light changes is very important, so natural light variations may cause issues).

Shooting a light bulb with something dark near it, and really out of focus, should do the trick. Also Audionut explained his setup a while ago (a picture would be nice).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on July 30, 2014, 09:05:55 PM
OK, I'll put that cube next to a light bulb and lock everything down like the last test.

Any time I've used that cube I've gotten blown highlights and pure black.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on August 01, 2014, 01:48:47 PM
I setup a nice little scene to shoot yesterday but when I powered up the camera I got an API warning for RAW_DIAG version 5 expected 6... is there one online I can use?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 01, 2014, 01:51:38 PM
Just download it again.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Stedda on August 01, 2014, 01:53:16 PM
Thank you Sir...  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: l_d_allan on August 04, 2014, 07:10:22 AM
Quote from: Stedda on July 30, 2014, 09:05:55 PM
OK, I'll put that cube next to a light bulb and lock everything down like the last test.

Any time I've used that cube I've gotten blown highlights and pure black.

The Datacolor SpyderCube is one of the neatest little devices I've come across. Overpriced, however.

BTW, it isn't difficult to make your own:

http://berean.zenfolio.com/dual_iso/h101028e1#h101028e1

http://berean.zenfolio.com/dual_iso/h101028e1#h3e0c323b
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 08, 2014, 05:59:35 AM
On the 6D I'm only seeing ADTG2, CMOS and Digic, no ADTG4/6.  Do you think this is 6D specific, or simply wrong/missing stub?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 08, 2014, 09:01:05 AM
Right, I see the same thing in the logs. Probably the 6D has only one chip, while most other cameras have two?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 09:56:48 AM
Did some testing with the "adtg_gui.mo" on 6d.
Changing the iso in canon menu gives the following numbers for CMOS[3]
Weird thing is, why doesn't use iso 6400 0x66 ? (checked it twice, it really uses 0x77)
And do you know what will happen if I override it to 0x88 or 0x99  ??? Will my precious cmos chip catch fire  :D
And how do you get the iso's lower than 100, it's not done in the CMOS register is it, how to approach ?

ISO     CMOS[3]
100      0x0
200      0x11
400      0x22
800      0x33
1600      0x44
3200      0x55
6400      0x77
12800    0x77
25600    0x77
    Hi1    0x77


EDIT: when I override it to 0x66 it shows ISO6400 in the exif, but I get exact same exposure results as ISO 800, weird  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 08, 2014, 10:43:53 AM
These are some "hidden" ISOs - 5D3 has them too (http://www.magiclantern.fm/forum/index.php?topic=10111.msg99665#msg99665), and a few of them seem cleaner then Canon's (http://www.magiclantern.fm/forum/index.php?topic=10111.msg103971;topicseen#msg103971).

Can you do a dynamic range test for all of them? (just like you did before). Set ISO 100 in Canon menu, use a manual lens, then set CMOS[3] to 0x00, 0x11 ... 0xFF => 16 ISOs (all based on Canon's ISO 100, and we'll only change one parameter).

I'd also like the same test on 5D3. Here you would set CMOS[0] to 0x003, 0x113 ... 0xFF3. Don't change any other settings, just ISO 100, manual lens, and this testing procedure (http://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955). If you can't compile ML yourself, ask for a build here (http://www.magiclantern.fm/forum/index.php?topic=12608).

These ISOs are applicable in both photo and movie; you may test any of these modes, but if you can do all of them, that's even better.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 12:26:57 PM
Did test some cmos[3] overrides, from 0x00 to 0xff (CMOS did not catch fire  :D).

CMOS[3] override on 6d   
Setting         ISO according to full well capacity numbers.
0x00                 100
0x11                 200
0x22                 400
0x33                 800
0x44                 1600
0x55                 3200
0x66                 640  :o  Weird one, full well capacity is between iso 800 and iso 400, looks like a "native" iso 640.
0x77                 6400
0x88                 100
0x99                 200
0xaa                 400
0xbb                 800
0xcc                 1600
0xdd                 3200
0xee                 1250   :o  Weird one, full well capacity is between iso 800 and iso 1600, looks like a "native" iso 1250.
0xff                   6400

The ISO settings that aren't used by Canon(the bold ones) don't seem much better/cleaner/different (But I did the test "Quick and dirty" so my measurement may not be accurate enough)
SNR curves are available on my google drive:
https://drive.google.com/folderview?id=0B1BxGc3dfMDabkZybDB6YldXR3M&usp=sharing
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 08, 2014, 12:55:06 PM
Ha, that was fast.

0xFF even seems to be a bit cleaner than 0x77, but it would be nice to have some more data points in the dark areas on the graph.

The dynamic range is from the intersection between SNR curve (red line) and the X axis at 0 EV (the bold one) - that is, from the point where SNR = 1, until the far right on the X axis (look for the last red pixel on the curve), where you have the white level (clipping point).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 02:32:14 PM
I remeasured the 0x77 and 0xff with more shadow points, now the difference is even smaller then before.
See the "remeasured" files on google drive:
https://drive.google.com/folderview?id=0B1BxGc3dfMDabkZybDB6YldXR3M&usp=sharing
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 03:03:20 PM
Some other stuff for 6d
CMOS[0] Default value is 0x593, changing it seems to do nothing.

CMOS[1] and CMOS[2] seems to be used for vertical and horizontal offset (didn't test it though)

CMOS[3] controls the analog iso (see posts above)

CMOS[4] on the 6d seems to give control over (digitally?) pulling back a few stops.
CMOS[4] default value = 0x4f0
CMOS[4] set to value 0x8f0, for the first digit, 4 and 8 can be used and it seems that 8 pulls back the whole histogram by 2 stops.  (Is this Canon's Auto lighting optimizer ?)
CMOS[4] set to value 0x800 seems to only pull back the highlights, so the second digit can give use some highlight pulling/crushing

CMOS[5] didn't test it yet, but probably does control vertical squeeze

CMOS[6] gives control over horizontal squeeze, not interesting.
Although it could be used to squeeze an 16:9 to a 4:3 ratio and record in 4:3 ratio and stretch it back to 16:9 in post, sort of fake anamorphic format(means lower bitrate for 16:9 aspect ratio and 1.33x stretch).
EDIT (The squeeze is probably used for smaller raw types, sRaw ?)

CMOS[7] and CMOS[8] gives weird dark frames/blackframes/readoutframes stuff ?
CMOS[8] default value 0xa
CMOS[8] value 0xf and 0x7 seems to give some blackframe or dark frame or something
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 03:10:42 PM
Some other interesting registers, don't know what normal values are for them and what values I should try, they do however change with iso setting, so maybe some fine-tuning for iso ?

EDIT: these four registers, seems to be some amplifiers and are per color channel, the first two are Red and Blue(or vice versa), the last two seems to be the two Green channels
EDIT 2 :Looking at the pictures now in lightroom, every 4th pixel is missing, so it's not used per color channel.
   
ISO               ADTG2[8882]   ADTG2[8884]   ADTG2[8886]   ADTG2[8888]
L1                    0x416                   0x416           0x416        0x417
100                  0x416                   0x416           0x416        0x417
200                  0x400                   0x400           0x400        0x400
400                  0x422                   0x422               0x423        0x423
800                  0x42d                   0x42d           0x42e        0x42e
1600                0x440                   0x440           0x441        0x442
3200                0x485                   0x484           0x484        0x484
6400                0x4a9                   0x4a8           0x4a7         0x4a7
12800              0x952                    0x950           0x94e           0x94e
25600              0x12a4                   0x12a0           0x129c        0x129c
Hi1                  0x12a4                   0x12a0           0x129c        0x129c
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on August 08, 2014, 04:35:14 PM
Those are the registers this entire discussion is about. They control some amplifier stage that is between CMOS and ADC. We can get about 0.5 EV more DR by turning them down. You might want to read the first couple of pages of this thread at least.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 04:43:27 PM
Read about the first ten pages of this thread, but didn't know it was about exact these registers.
Since the 5d3 has a nice module "iso_regs.mo" - 5D3 only
A lot of talk is only about one ADTG register, not 4 of them. So a bit confusing for me  :-[

But now I start to get the idea a little, the trick is to pull these numbers down.
They're in the 400 range, 250 seems to be a good starting point...
The sweet spot has to be found by trial and error with the raw-diag tool I Guess ?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on August 08, 2014, 05:12:18 PM
Quote from: Levas on August 08, 2014, 03:10:42 PM
EDIT: these four registers, seems to be some amplifiers and are per color channel, the first two are Red and Blue(or vice versa), the last two seems to be the two Green channels
EDIT 2 :Looking at the pictures now in lightroom, every 4th pixel is missing, so it's not used per color channel.
You might be interested in this description (http://www.magiclantern.fm/forum/index.php?topic=12789.msg123345#msg123345) that I wrote in another thread.  Because of how the colors are distributed between the vertical lines, you are somewhat correct in that the red pixels are affected by only 2 of the 4 registers (and the blue pixels are affected only by the other 2), but all 4 registers will affect green pixels.

For the purposes of the investigation in this thread, we've been setting all 4 registers to the same value, but Canon has some fine-tuning differences (as you can see in your table), which were presumably/hopefully determined from their calibration of the electronics.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 05:39:11 PM
Juicing up the battery now, so will try some more later on  :D
But theoreticaly , some very low iso's could be achieved by lowering the 400 number in these registers.
Has anyone tried using the number 50 in these registers ? Did it look like an ISO 12,5 ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on August 08, 2014, 05:48:45 PM
That won't accomplish what you're thinking because the ADTG/DFE amplification comes after the CMOS amplification.  The brightness at which you saturate in the first stage will still be saturated in the second stage, so you really can't reduce the gain of the second stage all that far before it becomes pointless (the effective ISO stops decreasing).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 05:53:41 PM
Hmmm.... :(, shall try later on, and see the results...

Other suggestions to get low ISO ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 08, 2014, 08:16:26 PM
Here are some curves from fake iso's below 100 (with lower ADTG gain).
https://drive.google.com/folderview?id=0B1BxGc3dfMDaU1NJd0dHUWNsNGc&usp=sharing

Used iso 100 as base iso and set registers to 300, 250, 200, 100 and 25.
Creating ISO's of about 75, 63, 50, 25 and 6.

Did get pink highlights, and it seems that highlights are cut off(by about using 250(ISO 63) or something, have to try some real photo's with it, but I think you could get away with ISO 25 if you want.

Seems like 275 is the number to go with if you care about dynamic range.
https://drive.google.com/folderview?id=0B1BxGc3dfMDaNGszU2g5aHBfTUE&usp=sharing
Did curves for ISO 200, 800 and 6400.
Dynamic range with ISO 200 -> 11,80 eV
Dynamic range with ISO 800 -> 11,36 eV
Dynamic range with ISO 6400 -> 9,60 eV

EDIT: Didn't alter any black or white levels, should I do that  :-\ and where do I do that ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 10, 2014, 12:53:59 PM
This "adtg_gui.mo" is very addicting.
Is there some overview page where known registers and the effects they have are reported ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 10, 2014, 01:07:37 PM
Hit page 1 of this thread and start reading  :P

My quick observations show that we will probably only get 0.4EV on the 6D.  ADTG [FE] is at 0 by default.
However, I'm seeing 11.4 EV of DR at stock, so 11.8 EV is decent enough I guess.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 10, 2014, 01:42:37 PM
Quote from: Audionut on August 10, 2014, 01:07:37 PM
Hit page 1 of this thread and start reading  :P
Was already afraid that this topic is the way all findings in these registers are organized...

The 0.4 eV extra is what I get too, fits perfectly with the theoretical values graph A1ex posted before:
(http://a1ex.magiclantern.fm/bleeding-edge/iso50/6d.png)

I'm also messing around a lot with the 8 CMOS registers.
CMOS[0], I have no idea what it does.
CMOS[6], is set on 400 in video mode (3x pixelbinning horizontal ?), setting it to 800, gives a stretched image and shows real pixels horizontal, if you make the image 3 times smaller horizontal in post, the aspect ratio is normal.
CMOS[7] seems to be used by switching from 1080p to 720p...

Have you done some other register testing ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: josepvm on August 10, 2014, 09:47:36 PM
After finding enough time to read most of this long (and quite intimidating ;)) thread I have done my first tests in my 500D using the iso-research branch and adtg_gui module.

I can play modifying C0F0[819c] (SaturateOffset) and also CMOS[0] (to change analog ISO), and everything seems to work.

But if I try to modify C0F0[8034] (BlackLevel/BW offset) my camera hangs.  As soon as I press "Set" button to modify it, the menu screen gets frozen, I cannot move with arrow keys, or exit with LV button. The shutter button works, but only to get a blank screen, and I cannot shutdown the camera, I need to remove the battery.

I am missing something? ( do I need a specific camera setup?). It happens equally with manual or EF lenses.

The procedure I have followed is:

1 - "hg clone" the iso-research branch.
2 - modify "Makefile_user" with CONFIG_GDB=y and Makefile.modules.default to include adtg_gui and raw_diag in the build.
3 - modify "adtg_gui.c" to include missing values for 500D, "ENGIO_WRITE_FUNC" and "ENG_DRV_OUT_FUNC"


else if (is_camera("500D", "1.1.1")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg70325#msg70325
    {
        ADTG_WRITE_FUNC = 0xFF22F8F4; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0xFF22F9DC; //"[REG] ############ Start CMOS"
ENGIO_WRITE_FUNC = 0xFF190CF4;
        ENG_DRV_OUT_FUNC = 0xFF190B84;
    }



I have copied these values from stubs.S, and double checked them, comparing the stubs with ROM dumps from other cameras, the stubs are Ok.



With these values adtg_gui is able to find DIGIC registers. Everything seems to be working fine ... but I cannot edit C0F0[8034]  :(

And according to Greg's posts (http://www.magiclantern.fm/forum/index.php?topic=10111.msg104136#msg104136) in this thread, modifying C0F0[819c] and C0F0[8034] is the only way so far to get some DR gain on 500D (0,33EV).

Any thoughts? I am doing something wrong?





Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 10, 2014, 10:36:37 PM
I know this problem. Number of known regs is too large.
The easiest solution to remove unnecessary in adtg_gui.c

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: josepvm on August 10, 2014, 10:54:18 PM
Quote from: Greg on August 10, 2014, 10:36:37 PM
The easiest solution to remove unnecessary in adtg_gui.c

Where? Are you refering to this struct list, at the beginning of "adtg_gui.c" ?


/* todo: check for duplicates */
static struct known_reg known_regs[] = {
    {DST_CMOS,      0, 0, "Analog ISO (most cameras)"},
    {DST_CMOS,      1, 0, "Vertical offset"},
    {DST_CMOS,      2, 0, "Horizontal offset / column skipping"},
    {DST_CMOS,      3, 0, "Analog ISO on 6D"},
    {DST_CMOS,      4, 0, "ISO-related?"},
    {DST_CMOS,      5, 0, "Fine vertical offset, black area maybe"},
    {DST_CMOS,      6, 0, "ISO 50 or timing related: FFF => darker image"},
    {DST_CMOS,      7, 0, "Looks like the cmos is dieing (g3gg0)"},
    {DST_ADTG, 0x8000, 0, "Causes interlacing (g3gg0)"},

...




Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 10, 2014, 10:58:34 PM
Yes, this list is too long.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: josepvm on August 10, 2014, 11:18:55 PM
Quote from: Greg on August 10, 2014, 10:58:34 PM
Yes, this list is too long.

Done, and it works now. Thanks, Greg !  ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 10, 2014, 11:33:52 PM
@Levas

I'm pretty sure a1ex' graph is only based on the ADTG [888x] bank of registers, which fits in with my quick test.  Based on my past experience, 0x34x seems about right for these registers @ ISO 100, but I haven't confirmed the results.  I've been looking for other registers that stand out as useful, but so far no luck.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 11, 2014, 08:54:02 PM
The preamp registers on 6d
ADTG2 8   ADTG2 9    ADTG2 a   ADTG2 b
Normal value about 0x4b (75 decimal)
If you set them to 0x190 (400decimal) you get 2 stops extra (noisier ofcourse, but nonetheless 2 stops extra)

Now for video, where the ISO limit is 25600, you can set these 4 preamps to 190, and you have ISO 102400 video mode  8)
And a lot of vertical lines across the frame(CMOS pattern), so blackframe extraction in post must be done to get usable results.
Will try it if it get's dark tonight  :D

EDIT:
If you use the other amplifier, ADTG[fe] (standardvalue=0, maxvalue=f)
And set it on max value f, gives another extra stop.
So ISO 204800 videomode  :D

Tried it out, video looks like crap, noise is all over the place  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 11, 2014, 10:25:48 PM
500D
1080p ADTG2[100c] = 0x2
720p   ADTG2[100c] = 0x4

vertical skipping lines (video mode)

0x2         0x4       0x5
RG+         RG+      RG+
GB           GB         GB
RG           RG         RG
GB+         GB         GB
RG           RG         RG
GB           GB+      GB
RG+         RG        RG+

So 0x1 and 0x5 bad colors (confirmed preview live view)

0x0 give an image of 1920x1080 (distorted) -> scaled post process 1920x360 without moire (less noise).
To work 0x0, need tune CMOS[5]
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 11, 2014, 10:41:00 PM
@Greg, what do you mean with "To work 0x0, need tune CMOS[5]"
What values are normal in CMOS[5] on 500d ?
And where do you set them too, to get a value of 0x0 to work in video mode ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 11, 2014, 10:55:11 PM
See in crop mode when change the height of the focus box (change the sensor area).
CMOS[5] value will be different/new.

ADTG2 [100c] = 0x0 // no skipping vertical lines, so we need to set which part of the sensor has to work
-----------------------------------------------------------------------------------------------------------------------------------


In this way we can get crop H264 video on all cameras :

ADTG2 [100c] = 0x0 // no skipping vertical lines
CMOS [4] = value of the crop mode (center) // horizontal lines?
CMOS [5] = value of the crop mode (center)

The image is not good, horizontal pixels (3) still averaged or something :

(http://s3.postimg.org/fnvaiv0ar/test_crop.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 11, 2014, 11:43:46 PM
For the 6d
ADTG2[800c] = vertical lineskipping, 0(crop), 2 (1080p) and 4(720p)
CMOS[6]
CMOS[7]

Overriding ADTG[800c] = 0
CMOS[6] = d0
CMOS[7] = 208

Gives me correct live view preview for crop mode on 6d, so it gives 3x zoom (only white border in the bottom visible... :-\)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 11, 2014, 11:56:40 PM
Quote from: Levas on August 11, 2014, 11:43:46 PM
Gives me correct live view preview for crop mode on 6d, so it gives 3x zoom (only white border in the bottom visible... :-\)
On the 500D also, maybe it is caused by raw size
1080p = 1590x1060
crop = about 2000x838?
So lack height.


ADTG2[1000] horizontal (column) pixels averaging

1080p ADTG2[1000] = 0x6
crop ADTG2[1000] = 0x5

Now look good:
(http://s28.postimg.org/qvaqvvvp9/crop_test2.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 12, 2014, 10:38:25 AM
Does anyone know where the horizontal pixelbinning is done ?
I'm curious if it's a baked in setting in the CMOS chips (then it's probably controlled via CMOS[6] or CMOS[7] on the 6d.)
Or could it be a setting which tells to average 3 pixels of the same color  ?

These registers are changed if you change the view from 10x zoom to standard view (1080p25)
                     1080p25       10x zoom(no pixelbinning, probably stretching pixels for live view)
ADTG2-805f       2b0         13c          Edit:         These values seems to vary within days, so not hard values.
ADTG2-8061      2b0         13c          Edit:         These values seems to vary within days, so not hard values.
CMOS[6]           400           d0
CMOS[7]            0             208
ADTG2-802c       0              110
ADTG2-8047      550         110
ADTG2-82f3           c         319
ADTG2-8820      303         301
ADTG2-8826      e04           6
ADTG2-8827        42           4
ADTG2-8828    3010         301
ADTG2-8860          f           e
ADTG2-8866          7           f
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 12, 2014, 12:40:47 PM
Canon 6d          Video mode   Video mode    Photomode               Photomode
                        1080p25    10xzoom      Live view off              Live view on
ADTG2-802c       0              110                   110                           0
ADTG2-8047      550*           110                   110                          550*           
ADTG2-8860      f                   e                      d                             f
ADTG2-82f3       c                 319                  211                           8 

Looks like ADTG2-802c and ADTG2-8047 do sort of control if every pixel horizontal is shown or that it does 3x pixelbinning

ADTG2-8860 has a unique settings for 1080p/10x zoom and live view-off, has anyone figured out what f, e and d setting does ?
Edit: Also ADTG2-82f3 has unique values for all modes

EDIT: *Could 550 mean, take 5 pixels wide for the first color, for example red(rgrgr) and 5 pixels wide for green (grgrg) and bin them together, this would give 3x pixelbinning... When my battery is recharged I'll try a value of 330 (Could mean pixelbinning of 2 pixels horizontal, gives a readout of 2736 pixels horizontal instead of 1824(binning of 3pixels)

Nope, doesn't work,gives funky live view colors and bottom half of the frame is placed at the top, nothing usuable
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 12, 2014, 02:25:01 PM
I add 0x60 to CMOS[5]
Now there is no white border in the bottom :

(http://s23.postimg.org/t9y9ccsgb/VRAM0.png)

Now we need to understand DIGIC scaling.


Raw (crop mode) vs H264 (1080p - ADTG/CMOS crop mode):
(http://s14.postimg.org/6oupf7oj1/rawvs.jpg)

H264 is upscaled.

Now I can record crop raw 1590x1060, height increase  :o
Default crop mode is 2000x838
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 12, 2014, 06:08:33 PM
Crop video with a good preview/live view? Possible!  ;D
Is not perfect, but near. We need tune DIGIC scaling regs.

(http://s28.postimg.org/40djk12wt/VRAM0.png)

mlv_rec frame - http://s28.postimg.org/w1rkxq871/testcropp.jpg
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on August 12, 2014, 06:31:12 PM
Ale zajebiscie :)!
Did you actually found out how to get a centered 3x zoom mode with a good preview?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: MA Visuals on August 12, 2014, 06:33:07 PM
Amazing find!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 12, 2014, 08:55:13 PM
The camera work in 1080p. Sensor work in crop mode.
So the preview is almost good, you can compare with my previous post.

This solved the problem of pink preview on old cameras.

Raw buffer is 1080p mode, so we have a different resolution, only 5D3 will give you 1920x1280.
I think it can be changed, but it requires a lot of research (when you change the size of the raw, preview Canon freeze).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: MA Visuals on August 12, 2014, 09:35:00 PM
I'm not sure if I understood correctly.  Are you saying that 5d3 crop mode will display realtime liveview at 1920x1280 but not 1920x1080? Is that right?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 12, 2014, 10:16:36 PM
@Greg, can you share the data of the values and registers you changed ?

ADTG2 [100c] = standard value = 2, override value = 0       (lineskipping register)
CMOS [4] = standard value = ?, Override value = ?             (value of the crop mode (center) // horizontal lines?)
CMOS [5] = Standard value = ? Override value = ?             (value of the crop mode (center))
ADTG2[1000] = Standard value = 6, Override value = 5       (horizontal (column) pixels averaging)

Can you share the values for the CMOS[4] and CMOS[5] ?

You said you'd also change some digic registers for scaling, can you share those registers and values ?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 12, 2014, 10:50:49 PM
@MA Visuals,
500D in 1080p mode has 1590x1060, crop trick 1590x1060 (same buffer). The camera still thinks that it is in 1080p, but the sensor give the crop area.
So this way only 5D3 can 1920x1280. I mean raw sizes.

Quote from: Levas on August 12, 2014, 10:16:36 PM
@Greg, can you share the data of the values and registers you changed ?

I discharged all batteries 500D, I'll try tomorrow.

Quote from: Levas on August 12, 2014, 10:16:36 PM
You said you'd also change some digic registers for scaling, can you share those registers and values ?
I'm still looking for those registers. This is only needed for more accurate preview or H264 video crop.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 03:16:45 PM
500D

                          1080p        crop-mode(5x)      1080p-trick-crop
ADTG2[105f]N    0x0             0x38f                       0x38f //shutter blanking for 5x/10x zoom
ADTG2[1061]     0x67f          0x0                          0x0    //shutter blanking for 1080/720p mode?
CMOS[4]            0xc0d          0xc85                      0xc85
CMOS[5]            0x400         0x609                      0x689
ADTG2[100c]     0x2              0x4                          0x0  //vertical skipping lines, not needed?
ADTG2[1000]     0x6             0x5                          0x5 //vertical skipping, horizontal binning?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 05:27:22 PM
Thanks Greg,

I can get 3xzoom/crop mode on 6d by overriding the following registers:
                      1080p     10xzoom     3xzoom(1:1 crop)

ADTG2-805f      2b0        13c             No change needed
ADTG2-8061     2b0        13c             No change needed

CMOS[6]          400         d0             d0       (=Changing the "d" moves horizontal position on the sensor)
CMOS[7]             0         208          206       (Lowering the 8 to 6 moves the white bar outside 16:9 view)
ADTG2-800c         2           0              0         (vertical lines to skip)
ADTG2-8000        6           5             5       (Don't know what it does, if it is 6 you get some vertical interlaced looking footage...)


With these settings I get 3xzoom/crop mode, correct preview in live view on the 6d
Raw recording works, gives correct framing.
I can even record canon "native" h.264 with it, and it works too!

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: nikfreak on August 13, 2014, 07:55:12 PM
wow hats off for taking so much time and efforts to help.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on August 13, 2014, 08:06:00 PM
Quote from: Levas on August 13, 2014, 05:27:22 PM
With these settings I get 3xzoom/crop mode, correct preview in live view on the 6d
Raw recording works, gives correct framing.
I can even record canon "native" h.264 with it, and it works too!

Curious. Do you get a centered 3x zoom mode or is it on the left or right side?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 08:09:28 PM
Still wonder how to increase edmac buffer or something to increase the size of the crop raw.
A few pages previously, I found how to get the full width of the sensor - http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909

I would get a crop raw 1500x1500px (so it will be less FPS). Here need to find something else than ADTG/CMOS. I do not know what.

Sensor say "I'm hot! Give me a break!"   :o
(http://s9.postimg.org/na44d5jj3/VRAM1.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 08:17:51 PM
Quote from: Danne on August 13, 2014, 08:06:00 PM
Curious. Do you get a centered 3x zoom mode or is it on the left or right side?

I can move it horizontal in 16 steps (0 to f hexadecimal) with the first "digit" value in CMOS[6]

CMOS[6]         d0         (=Changing the "d" moves horizontal position on the sensor)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 08:26:51 PM
Quote from: Greg on August 13, 2014, 08:09:28 PM
Still wonder how to increase edmac buffer or something to increase the size of the crop raw.

Have you tried setting ADTG2-100c on 0 in 1080p mode ?  (override it and switch between photo and videomode to "reset" video live view to do 0 line skipping.
On 6d I have to use CMOS[7] to center the image on the live view screen (white borders above and below it), on your 500d it probably is CMOS[4] or CMOS[5].

I can get my 6d to do zero vertical lineskipping in 1080p mode, only it does 1/3 of the sensor height, probably has to do with the edmac buffer size, like you say, increase the edmac to increase the frame height. So I get a stretched image with real vertical pixels and horizontal pixelbinning of 3 pixels.

For those of you with a 6d
Go to 1080p25 video mode
Override registers ADTG2-800c to value "0"
Override registers CMOS[7] to value "14"

Press magnifying button to go to 5x zoom and 10x zoom and back to normal view (this somehow resets live view to the new override settings)
Or
Switch to photo mode and switch back to video mode (this resets live view to the new override settings)

Now you have full sensor width view with 1/3 vertical sensor height(with no lineskipping)



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 08:42:15 PM
To refresh the parameters use (being in 5x not need to change to another mode) :

static void run_test()
{
   msleep(200);
   set_lv_zoom(5);
}

Yes, I tried it ADTG2-100c 0 in 1080p mode.
Quote from: Greg on August 11, 2014, 10:25:48 PM
0x0 give an image of 1920x1080 (distorted) -> scaled post process 1920x360 without moire (less noise).

This is just an example for the 500D to give it to 1590x1060 -> scaled post process 1590x353.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 09:12:06 PM
If the image buffer size can be made bigger, this may be the way to go for 4k, as a 1:1 center crop...   Edit: And we need some memory card and SD write speed upgrade to get 360Mb/s write speed  ;D

Or 1920(full sensor width) x 2160(2/3 sensor height) distorted -> Scaled post proces, 1920 x 720 moire free.

Couldn't find any mode which bins vertical, the 5d3 probably has such a mode in one of the CMOS[] registers...
Wouldn't surprise me if the 6d has such a mode too...same generation CMOS...

Does someone know if the 5d3 has the ADTG-x00c register to set the vertical lineskipping ?
It is using lineskipping in 50p and 60p modes... can someone confirm this register stays at zero in 1080p mode ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 09:35:57 PM
Quote from: Levas on August 13, 2014, 09:12:06 PM
If the image buffer size can be made bigger, this may be the way to go for 4k, as a 1:1 center crop...   
Yes, but it will be less FPS on the 5D3 should be 15-20FPS (4096x1714).

FPS timer A is width
FPS timer B is height
You can take a photo normal mode, lock timer A and B and go to live view. FPS will be continuous photo mode (500D 3.554)
I wonder if there is somewhere a clock sensor (50D sensor is faster than the 500D).

Maybe we can do ultra fast camera 320x240 crop mode, but it will not be easy. It will take a very long time. And on the way may be other limitations.

Quote from: Levas on August 13, 2014, 09:12:06 PM
Couldn't find any mode which bins vertical, the 5d3 probably has such a mode in one of the CMOS[] registers...
Wouldn't surprise me if the 6d has such a mode too...same generation CMOS...
Yes, this is interesting. I only have a 500D so I can not help.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 09:42:41 PM
Quote from: Greg on August 13, 2014, 09:35:57 PM
FPS timer A is width
FPS timer B is height

Can these FPS registers be found in the "adtg_gui.mo" tool ?
How are these registers named on you're 500d ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 09:48:24 PM
ADTG Registers -> Show -> FPS timres only
(requires DIGIC registers)

0xC0F0,   0x6008, 0, "FPS register A"
0xC0F0,   0x6014, 0, "FPS register B"
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 10:03:58 PM
Oh boy :D, can't wait to try some settings with those registers when the battery is recharged.

Do you know what values the 6008 and 6014 stand for ? Can't make any resolution out of it ?
And do you know if there's some place (bitbucket maybe  :-\)  where this information is documented ?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 10:39:54 PM
Photo mode :
(http://s30.postimg.org/8whjq8uap/VRAM0.png)

Movie mode (with values photo mode):
(http://s30.postimg.org/63oc67ty9/VRAM1.png)


This does not affect the image. But the sensor work in full resolution (this is confirmed by previous research the full width video).
(focus box should be on the left, I do not know whether the top or bottom)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 13, 2014, 10:55:49 PM
Got it, same registers on the 6d.
Locking photo mode values gives 4.7 FPS in video mode  :D

But how did you record the full image width ?
I can't get any higher resolutions in MLV_record then in standard video mode, still max of 1792pixels wide ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 13, 2014, 11:09:37 PM
It is complicated because the buffer is still the same.
We get the full width by reducing the height.

This requires a modification in raw.c and few registers.

When lock all registers ADTG/CMOS 1080p. Go to 720p, buffer edmac is 720p. So this is not it.


Maybe it will help?
static void run_test()
{
    msleep(2000);

    debug_intercept();
    call("lv_save_raw", 1);
    msleep(1000);
    call("lv_save_raw", 0);
    debug_intercept();
}
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 14, 2014, 12:13:39 AM
Can someone tell me the values of digic registers:

C0F0[6008]
C0F0[6014]

On the 5d3 in video mode ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 14, 2014, 12:32:59 AM
Quote from: Greg on August 13, 2014, 09:35:57 PM
FPS timer A is width
FPS timer B is height

How are those values used, I can't make any sense out of the number which is used in timer A
6d
                  Photomode           Video(1080)
Timer A       5970597               2210221
Timer B          e8d                          63f

Value for timer B e8d = 3725 in decimals, could make sense for height...
Value for timer B 63f = 1599 in decimals, could make sense for height...

But I can't make anything out of the numbers in timer A  :-\
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 14, 2014, 12:40:37 AM
Timer You can change in the FPS menu.

I received a the log, there are a lot of registers. We need someone experienced.
So we need wait for A1ex

And now we can go on vacation  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on August 14, 2014, 02:25:29 AM
Quote from: Levas on August 14, 2014, 12:32:59 AM
How are those values used, I can't make any sense out of the number which is used in timer A
6d
                  Photomode           Video(1080)
Timer A       5970597               2210221
Timer B          e8d                          63f
...
But I can't make anything out of the numbers in timer A  :-\
Timer A, which are 0x597 and 0x221 in your two examples because it is doubled, is the number of clock ticks to wait before each successive row is read out.  Thus, it should not be set shorter than the time it takes to read out a single row of the image (i.e., the width), but it can be set longer depending on the needs of a rolling shutter or an exact frame rate.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 14, 2014, 10:40:33 AM
Ah, that explains why the real low values didn't gave any live view at all...
Is it double because of 2 column, rows that are read out or something ?
Anyone tried combining them, for example:

Timer A value on 5970221 ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on August 14, 2014, 11:37:07 AM
Yeah, I imagine Timer A is doubled for the same reason that the CMOS gain is doubled: each row in a row pair will use one of the halves.  However, I can't think of a clear benefit from mismatching the two halves of Timer A, unlike with CMOS gain and dual ISO.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 14, 2014, 12:26:12 PM
@A1ex and @Nikfreak,

Would be nice if we can get the 3xzoom/1:1 crop preview in live view eventually in the nightly builds.
A1ex, any suggestions how to implement it, since it differs per camera model, would you put it as an option in the raw and mlv recording modules or is it better to place the code somewhere else ?
Maybe we could put an option in the raw/mlv recording menu, which put the camera in crop mode. For example with an option "Crop mode" on or off.


To get 1:1 crop / 3x zoom preview in live view on 6d, these registers need to be changed to the following values:
CMOS[6]                   d0       (=Changing the "d" moves horizontal position on the sensor)
CMOS[7]               206       (Vertical centering the image on the screen)
ADTG2-800c             0         (vertical lines to skip)
ADTG2-8000             5       (Don't know what it does, if it is 6(as in 1080p25mode) you get some vertical interlaced looking footage... so must be 5)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: nikfreak on August 15, 2014, 09:03:33 AM
Levas thx for all that awesome findings. We will need to wait until a1ex is back. Actually I think I am not capable of implementing what you requested. Don't want to reinvent the wheel so it's better to hope that a1ex will do that job...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 15, 2014, 04:29:11 PM
And thanks to Greg  :), he did the crop findings on his 500d, I translated it to the registers on the 6d.

A1ex probably has a clear view on how he would like to add options like these to the existing code.


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 15, 2014, 07:34:26 PM
I can read the values​​, but do not know how to overwrite.

(http://s1.postimg.org/mnuf4687z/VRAM0.png)

crop_rec  ;D
(http://s1.postimg.org/7pc09qt5r/VRAM1.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 15, 2014, 07:44:27 PM
Aha, a Crop_rec module!

Only programming language I know is Quickbasic ;D and some HTML... :P
But the answer in how to overwrite registers probably can be found in the source code of the adtg_gui module
https://bitbucket.org/hudson/magic-lantern/src/3288e152d8ae0dcb2cdaf1f9afc1a17611eea2ff/modules/adtg_gui/adtg_gui.c?at=unified
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 15, 2014, 08:01:14 PM
adtg_gui is not the solution, it requires 15FPS live view and GDB.
We need a module that will let 30FPS ;)

Not easy  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ayshih on August 15, 2014, 08:24:23 PM
adtg_gui is slow because it intercepts everything.  it's more appropriate to look at iso_regs for guidance, although you'll need the camera-specific addresses found in adtg_gui.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 15, 2014, 08:31:37 PM
I don't mean using the adtg_gui module to modify or rebuild.
I expect there to be some command/syntax which shows how to replace values in the registers... :o  <- my face looking at the source code  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 15, 2014, 09:33:19 PM
I modified ADTG_GUI (remove some REG_ENTRY). Now can I get more FPS.

500D in crop mode can get 24FPS (default is 22FPS).
The 1080p (crop trick) can get 21FPS when try more, the camera will freeze.

So maybe somewhere is to limit FPS for each mode.

Currently, it does not make sense for a 500D.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 15, 2014, 09:47:12 PM
ah, some unforseen problem... nog getting a minimum of 24fps...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 15, 2014, 09:52:08 PM
On the other cameras should be 30FPS, 500D has a limit in full HD.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 15, 2014, 09:54:57 PM
That reminds me, the 500d is limited to 20 FPS, as factory default, isn't it ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 15, 2014, 09:59:46 PM
Yes 500D without ML in 1080p has 20FPS. With ML can go to 21FPS.

1080p crop trick, the camera think that it is still 1080p (not crop mode). So you can not go over 21FPS.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on August 16, 2014, 05:16:37 PM
500D H264 default vs crop (crop mode is blinking :o)



Default - http://s22.postimg.org/mklz071wf/deafult.jpg
Crop - http://s22.postimg.org/sbc5dx9wf/crop.jpg
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on August 16, 2014, 08:41:09 PM
Maybe you should try some different values for ADTG2-1000 registers, maybe 5 isn't good either ?
I got weird vertical interlacing with the value 6, 5 seems good for me, but I didn't check that much, I shall record some video to see if a value of 5 for 6d is good or not.

And I don't really get where the value in the ADTG-1000 register stands for, have no idea what the value exactly does...

Edit: The value is set to zero in photo mode.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ronnie Chan on September 02, 2014, 02:34:59 PM
Does this work on actual image?or it is just playing with graph?
I tried to install the module on my 5dmark3 but it appears to be fail to load.
Some please advice,I am eager to finds out.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on September 02, 2014, 04:05:05 PM
Please read the OP

Quote
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!


Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)

Quote
requires a custom ML build with CONFIG_GDB=y
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: itsskin on September 23, 2014, 09:34:29 AM
CONFIG_GDB=y is put to makefile.user, right?
I'm compiling silentpics branch with this line, but loading iso_regs gives me an error. Please help to find out what I'm doing wrong.
Thx!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on September 23, 2014, 09:39:58 AM
Try merging the fullres-silent-pic branch with the iso-research branch.

If you don't want to merge, compile the fullres-silent-pics branch as usual, with CONFIG_GDB=y, then compile only the extra modules from iso-research.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: itsskin on September 23, 2014, 09:46:35 AM
Great! Thank you!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on October 31, 2014, 12:28:52 PM
My test scene looks like this.   Underexposure zebras are set to 0.
(https://i.postimg.cc/G2FwvQB0/Scene.png)

I ETTR the second brightest object (which is angled for a good gradient) and gives a good response near the highlights.  Which produces results like so.

(https://i.postimg.cc/prPNCF8m/Canon-6-D-Photo-ISO-100.png)

Some further detail about this test setup is back here (https://www.magiclantern.fm/forum/index.php?topic=10111.msg118194;topicseen#msg118194).


My results for 6D photo mode and no aperture gain.
https://www.dropbox.com/sh/rrn7chasohg36hk/AAD7fngMRzvGNfrz6IzOgX3pa?dl=0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on November 04, 2014, 10:39:47 PM
Moved my off-topic discussion regarding lenses over here (http://www.magiclantern.fm/forum/index.php?topic=13770).

Dynamic range and Equivalence discussion moved here (http://www.magiclantern.fm/forum/index.php?topic=13787.0).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 07, 2015, 06:13:29 AM
Is there a reason why the 7D never got included?

Found the easy stubs
ENG_DRV_OUT_FUNC = 0xFF1F675C;  // from stubs
ENGIO_WRITE_FUNC = 0xFF1F6B20;  // from stubs


But I guess I have to do some stub checking for CMOS_WRITE_FUNC and ADTG_WRITE_FUNC, unless someone wants to use his automatic stub tool (http://www.magiclantern.fm/forum/index.php?topic=10111.msg97953#msg97953).   :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: CITY-U1001 on April 10, 2015, 11:56:24 AM
work on 50D ? Who can make this module ? I think 66 ISO it's best)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on April 10, 2015, 01:04:32 PM
Quote from: Audionut on April 07, 2015, 06:13:29 AM
Is there a reason why the 7D never got included?
The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.

Quote from: CITY-U1001 on April 10, 2015, 11:56:24 AM
work on 50D ? Who can make this module ? I think 66 ISO it's best)
What module? There are several different modules associated with this thread and they are all for reverse engineering and research purposes. They also require compiling ML in 'debug mode' (Config_GDB=y). This is something you must do yourself.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 10, 2015, 08:01:03 PM
Quote from: dmilligan on April 10, 2015, 01:04:32 PM
The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.

Yes, that sounds familiar.  Thanks for the memory jog.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 14, 2015, 07:51:46 PM
Quote from: Greg on August 13, 2014, 03:16:45 PM
500D
                          1080p        crop-mode(5x)      1080p-trick-crop
ADTG2[105f]N    0x0             0x38f                       0x38f //shutter blanking for 5x/10x zoom
ADTG2[1061]     0x67f          0x0                          0x0    //shutter blanking for 1080/720p mode?
CMOS[4]            0xc0d          0xc85                      0xc85
CMOS[5]            0x400         0x609                      0x689
ADTG2[100c]     0x2              0x4                          0x0  //vertical skipping lines, not needed?
ADTG2[1000]     0x6             0x5                          0x5 //vertical skipping, horizontal binning?

Quote from: Levas on August 14, 2014, 12:26:12 PM
To get 1:1 crop / 3x zoom preview in live view on 6d, these registers need to be changed to the following values:
CMOS[6]                   d0       (=Changing the "d" moves horizontal position on the sensor)
CMOS[7]               206       (Vertical centering the image on the screen)
ADTG2-800c             0         (vertical lines to skip)
ADTG2-8000             5       (Don't know what it does, if it is 6(as in 1080p25mode) you get some vertical interlaced looking footage... so must be 5)

For 5D3, a configuration that appears to work is:

CMOS[1]                 0xB8B           vertical position and size
CMOS[2]                 0x10E           horizontal position and downsizing factor
CMOS[6]                 0x170           ISO related? pink highlights without it
ADTG2[0x8000]           0x5             it's 5 in zoom mode and 6 in 1080p
ADTG4[0x8000]           0x5             same
ADTG2[0x8806]           0x6088          without this, you get some weird artifacts
                                        note: ADTG4[0x8806] should not be changed (default 0x6048)


How to apply it:
- compile ML for adtg_gui (see first post)
- load adtg_gui and enable it in Debug menu
- go to movie mode 1080p, 24 or 25 fps
- go to play mode and back
- configure the registers in adtg_gui submenu (latest adtg_gui has a preset for crop mode registers)
- go to play mode and back (again)

Minor bug: it has a few black lines at the top.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 18, 2015, 12:08:04 AM
Updated adtg_gui: it now works even at 60fps*, and you no longer need CONFIG_GDB to run it (just compile ML from iso-research normally).

https://bitbucket.org/hudson/magic-lantern/commits/14776f860522

* if you also log DIGIC registers, those are not fast enough for 60fps.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Greg on April 20, 2015, 02:55:46 PM
So it should work on all cameras, just find only registers.
On old cameras we need read shutter table and overwrite ADTG2[105f]N (if it's possible).

At 5D3 probably image is ok.
On the other cameras will be extended. Native (raw) 1080p mode -> H264 (1920x1080).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on April 24, 2015, 04:07:19 PM
Hi! Is the changes to register settings reverted to default on restart or do I have to manually reset reg changes?
Got the adtg_gui loaded and planning on trying the 3d crop mode settings.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 25, 2015, 03:01:01 AM
They are all reverted to default on power off.  Also registers that have not been manually changed, will also change with camera settings.  So if you manually change some register, in essence you lock the register to that value.  If changing from photo mode to movie mode normally changes that register, this will not happen.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on April 25, 2015, 08:15:49 AM
Quote from: Danne on April 24, 2015, 04:07:19 PM
Hi! Is the changes to register settings reverted to default on restart or do I have to manually reset reg changes?
Got the adtg_gui loaded and planning on trying the 3d crop mode settings.
Great question as I was just thinking the same earlier...

Quote from: Audionut on April 25, 2015, 03:01:01 AM
They are all reverted to default on power off.  Also registers that have not been manually changed, will also change with camera settings.  So if you manually change some register, in essence you lock the register to that value.  If changing from photo mode to movie mode normally changes that register, this will not happen.
valuable info to know... Thanks @Audionut!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: kidfob on June 18, 2015, 09:35:59 PM
I'd like to use this in combination with dual ISO.

Is there a nightly build version of this so I dont have to attempt to try to build this myself?

Thanks
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on June 19, 2015, 02:09:22 AM
Nope. Read the OP.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ShootMeAlready on June 30, 2015, 03:24:51 AM
I have a 5DMKII to use for the summer, and if there is a build to check out, would love to check out the 3X screen.
It seems to me that this is one of the biggest hurdles to solve to make 2K shooting more practical.

BTW love the 5DMKII with ML.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mothaibaphoto on February 22, 2016, 02:12:00 PM
It seems that crop mode (http://www.magiclantern.fm/forum/index.php?topic=10111.msg145036#msg145036) has some practical value. I shot some videos without issues.
Advantages:
1. Centered (at lest at 24 FPS) - minimal distortion with wide lenses.
At 30 FPS slightly down center(CMOS[1] - 0xb8E), but centered horizontally.
Maybe it's possible to get it centered and with 30 FPS, I just not so skilled in brute forcing.
2. Works via HDMI.
3. Same as "Canon" preview - fast updating and color.
Disadvantages:
1. Recording times about 30%-50% less.
2. Cumbersome activating process. Of course, that adtg_gui tool was not intended for shoot.
3. No way to exit back to non-crop view without restarting - liveview gets corrupted.
4. Just 1920x1080.
5. Dual ISO doesn't work. Its's looks like it simply commented out.
Notes:
1. I don't have "a few black lines at the top" a1ex was told about.
2. I didn't notice any artifacts, it looks like "regular" crop-mode video.
So main question - is it possible to get this without such a great impact on performance?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on February 22, 2016, 06:32:08 PM
Thanks for the detailed explanation Motbaiphoto.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mothaibaphoto on February 23, 2016, 08:17:37 AM
Now I want to try that ISO 66 from first post.
First, its unclear, do I need to tweak with iso_regs, or I can do the same with adtg_gui, or I need both? I decided to start with iso_regs, as it seems more relevant.
I compiled with CONFIG_GDB=y, iso_regs from iso-research branch(as it looks newer version than posted in OP) start camera with module loaded, ISO 100, take picture - everything is OK. But, as soon as I activate ISO registers in Debug menu every image I take gets corrupted intill I deactivate ISO registers and restart camera. iso_regs from OP behaves the same :(
What am I doing wrong?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 23, 2016, 09:20:20 AM
On 5D3, you can just use iso_regs.

Will double-check the issue; I suspect a problem in the patching library.

Meanwhile, I suggest trying to compile 497b816 from the iso-research branch, instead of the latest one. Or, if it still doesn't work, I think just the unified branch with CONFIG_GDB=y should do the trick (so you can load iso_regs). It will probably not play nice in LiveView at higher FPS, but should work fine in photo mode.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mothaibaphoto on February 23, 2016, 07:30:45 PM
Quote from: a1ex on February 23, 2016, 09:20:20 AM
unified branch with CONFIG_GDB=y should do the trick (so you can load iso_regs)
Yes, this works, thanks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on February 23, 2016, 07:33:51 PM
Just curious ... How is ISO 66 different from ISO 50 that we can get from Canon's settings (better or worse)?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dmilligan on February 23, 2016, 07:59:20 PM
If you define ISO based on the clipping point, your camera's ISO 50 is actually just ISO 100. This ISO 66 is a true ISO 66. It allows capturing more photons without clipping, than does ISO 100. Canon ISO 50 and ISO 100 clip at the same number of photons captured.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on February 24, 2016, 12:13:11 AM
Brilliantly well said, David and Thanks for the clarifications!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2016, 08:23:04 PM
Quote from: mothaibaphoto on February 22, 2016, 02:12:00 PM
1. I don't have "a few black lines at the top" a1ex was told about.

I wasn't told about it, I've seen it myself, and I'm seeing it right now.

Quote
At 30 FPS slightly down center(CMOS[1] - 0xb8E), but centered horizontally.

Didn't really check how centered it is, but at 30 fps I see a big dark bar at the top (not black, but darker than the entire image), plus a lot of artifacts in the image (somewhat like vertical banding).

edit: 30fps works much better with 0xb8E, no more artifacts, nice find :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mothaibaphoto on February 28, 2016, 11:01:47 PM
Quote from: a1ex on February 28, 2016, 08:23:04 PM
I wasn't told about it, I've seen it myself, and I'm seeing it right now.
Maybe some camera settings affect this - i.e PAL/NTSC(i'm on NTSC)
Quote from: a1ex on February 28, 2016, 08:23:04 PM
Didn't really check how centered it is, but at 30 fps I see a big dark bar at the top (not black, but darker than the entire image), plus a lot of artifacts in the image (somewhat like vertical banding).
This probably means you didn't follow exactly your own manual regarding entering play mode twice or changed camera FPS after enabling registers hook. It works even with 60 FPS - 0xB95.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 28, 2016, 11:28:10 PM
You can't get 30fps in PAL.

I did follow the instructions (without pressing PLAY twice, you can't even see the last registers in adtg_gui). Didn't touch FPS (override is off).

At 60 FPS I get green image, even with 0xB95.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: mothaibaphoto on February 29, 2016, 12:02:43 AM
Quote from: a1ex on February 28, 2016, 11:28:10 PM
Didn't touch FPS (override is off).
I didn't mean override. I mean canon menu changes. Need restart camera, set "Movie rec size" in canon menu and just after that hooks and so on.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: kidfob on March 04, 2016, 06:16:38 AM
Is there a nightly build for the iso research functionality? I need the special branch in order to use ISO 66 correct?

Thanks
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 17, 2016, 12:26:45 AM
Quote from: a1ex on January 09, 2014, 09:04:50 PM
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!


Take it easy, the current state is research.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 02, 2016, 12:03:48 PM
Updated adtg_gui to allow non-destructive overriding of registers (currently only implemented for CMOS/ADTG/DFE registers, and only tested for CMOS ones). Trick borrowed from crop_rec.mo (http://www.magiclantern.fm/forum/index.php?topic=17021).

That means, as soon as you stop overriding a register, it gets back to its original value.

Previously, if a register value was not updated continuously by Canon code, disabling our overrides did not restore the old value.
Title: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on April 02, 2016, 06:27:20 PM
Good catch @a1ex ... I was fiddling with the adtg_gui last night but had forced myself to take a screenshot of defaults before making changes.

Now this eliminates that process ... Nice work once again! [emoji106]
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ibrahim on June 09, 2016, 05:43:45 PM
Hi,

Is anyone getting ISO 35, 70, 140 on my 600d and 5d3?
I set the ml digital iso to -1,5 EV and HTP off.

When I turn HTP on I only get ISO 70 140, ... (no longer 35)

Which one results in less noise?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: michael84 on June 28, 2016, 12:44:07 AM
I spent the last days learning how to compile ML to use these tweaks on my 5D MK III. On the weekend i decided to give the ISO REGS module a go and used ISO 860 (pulled from 1600) and ISO 3400 (pulled from 6400) for some scenes which are typically troublesome because of the extreme dynamic range involved like harsh stage lighting and a subject sitting in the sun with a dark wooden wall behind him.

Exposure was set at +2/3 EV from the correct value for unmodified ISO. Shadow/midtone noise is clearly better. I need to post some crops tomorrow, i also shot some test images of real world scenes.

Just one question - i do not really get what "CMOS patched" means? As far as i understand, this is some tweak which has to be applied with ADTG GUI, and cannot be done with ISO REGS, right? Can i use ADTG GUI to apply the CMOS patch, and then use ISO REGS to set the rest?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 28, 2016, 07:49:16 AM
Indeed, it's not implemented in iso_regs, but it's easy to do so (in cmos_log).

Unfortunately, you can't run both adtg_gui and iso_regs at the same time.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: michael84 on June 29, 2016, 04:15:15 PM
A quick test with a high DR scene that would require dual iso, or exposure bracketing HDR, or a flash. One Picture was taken with Canon ISO 100, 1/125 s shutter and f8. Second exposure was taken with ML ISO 100 (actually 119 reported by iso_regs), ADTG gain at 962 (-0.16 EV as reported by iso_regs), preamp at 1, 0xfe at 3, same exposure as with Canon ISO 100.

Both raws loaded in Lightroom 6.6, exposure pushed by almost 4 stops (+3.74 EV on ML ISO one, +3.84 on Canon ISO) to bring out the shadows, highlights at -100, WB as shot. ML ISO visibly reduces color noise and the purple shadow color shift i get when i push them too much from Canon ISO. Luma noise is also reduced and looks finer, which helps to preserve details. Only the banding noise seems to remain. Color noise reduction in Lightroom was left at standard (25), even bringing this one to 100 on the Canon ISO picture could not bring it to a comüparable level to the ML one.

Now if there is a fix for the banding noise....


Full Size JPEG, Canon ISO 100: https://www.dropbox.com/s/4jjnvt9y9h1p3m4/5D3_ISO%20100.jpg?dl=0 (https://www.dropbox.com/s/4jjnvt9y9h1p3m4/5D3_ISO%20100.jpg?dl=0)
Full Size JPEG, ML ISO: https://www.dropbox.com/s/kh16i53nb1zw4iz/5D3_ML_ISO-1.jpg?dl=0 (https://www.dropbox.com/s/kh16i53nb1zw4iz/5D3_ML_ISO-1.jpg?dl=0)

Crop 1, Canon ISO: https://www.dropbox.com/s/c779b25yn4sgali/5D3_ISO%20100_Crop%201.jpg?dl=0 (https://www.dropbox.com/s/c779b25yn4sgali/5D3_ISO%20100_Crop%201.jpg?dl=0)
Crop 1, ML ISO: https://www.dropbox.com/s/b3ahzvatyjf3fg4/5D3_ML%20ISO_Crop%201.jpg?dl=0 (https://www.dropbox.com/s/b3ahzvatyjf3fg4/5D3_ML%20ISO_Crop%201.jpg?dl=0)

Crop 2, Canon ISO: https://www.dropbox.com/s/f5e4fuh09cbz2uj/5D3_ISO%20100_Crop%202.jpg?dl=0 (https://www.dropbox.com/s/f5e4fuh09cbz2uj/5D3_ISO%20100_Crop%202.jpg?dl=0)
Crop 2, ML ISO https://www.dropbox.com/s/cukp18kvkdaiclh/5D3_ML%20ISO_Crop%202.jpg?dl=0 (https://www.dropbox.com/s/cukp18kvkdaiclh/5D3_ML%20ISO_Crop%202.jpg?dl=0)

Crop 3, Canon ISO: https://www.dropbox.com/s/0wnhr84g4am458b/5D3_ISO%20100_Crop%203.jpg?dl=0 (https://www.dropbox.com/s/0wnhr84g4am458b/5D3_ISO%20100_Crop%203.jpg?dl=0)
Crop 3, ML ISO: https://www.dropbox.com/s/97luk8kwiptdlhw/5D3_MIL%20ISO_Crop%203.jpg?dl=0 (https://www.dropbox.com/s/97luk8kwiptdlhw/5D3_MIL%20ISO_Crop%203.jpg?dl=0)

Crop 4, Canon ISO: https://www.dropbox.com/s/olers9xsnux0d2l/5D3_ISO%20100_Crop%204.jpg?dl=0 (https://www.dropbox.com/s/olers9xsnux0d2l/5D3_ISO%20100_Crop%204.jpg?dl=0)
Crop 4, ML ISO: https://www.dropbox.com/s/miauq2zk6xw11sc/5D3_ML%20ISO_Crop%204.jpg?dl=0 (https://www.dropbox.com/s/miauq2zk6xw11sc/5D3_ML%20ISO_Crop%204.jpg?dl=0)

Crop 5, Canon ISO: https://www.dropbox.com/s/l31k4xfjuuzhme3/5D3_ISO%20100_Crop%205.jpg?dl=0 (https://www.dropbox.com/s/l31k4xfjuuzhme3/5D3_ISO%20100_Crop%205.jpg?dl=0)
Crop 5, ML ISO: https://www.dropbox.com/s/ncpg1hzxcc7361h/5D3_ML%20ISO_Crop%205.jpg?dl=0 (https://www.dropbox.com/s/ncpg1hzxcc7361h/5D3_ML%20ISO_Crop%205.jpg?dl=0)


Now i need to learn how to use cmos_log to apply the CMOS patch.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 29, 2016, 04:22:58 PM
Looks better than I've expected. Was the ISO 119 pulled down from Canon ISO 200?

Can you share the screenshot with iso_regs settings?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: michael84 on June 29, 2016, 05:58:52 PM
Yes, this was pulled from Canon ISO 200.

Screenshot - had to reproduce settings, because i forgot to take one: https://www.dropbox.com/s/pw7epn5d9zw94h1/VRAM0.PPM?dl=0 (https://www.dropbox.com/s/pw7epn5d9zw94h1/VRAM0.PPM?dl=0)

raw_diag Canon ISO: https://www.dropbox.com/s/60h6a5b6la4aqyr/Canon%20ISO.ppm?dl=0 (https://www.dropbox.com/s/60h6a5b6la4aqyr/Canon%20ISO.ppm?dl=0)
raw_diag ML ISO: https://www.dropbox.com/s/hys867wmy136bll/ML%20ISO.ppm?dl=0 (https://www.dropbox.com/s/hys867wmy136bll/ML%20ISO.ppm?dl=0)

Next tests coming: ISO 200, 400, 800 and 1600 with flash illumination, long nighttime exposures, and high ISO (3200 and 6400) with typical exposure times needed to freeze motion, e.g. 1/400 to 1/1000.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on June 29, 2016, 07:24:27 PM
 :o Gnarly work done by @michael84 and Thanks for sharing the results. Definitely came out much better than expected.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on June 29, 2016, 09:10:45 PM
Tip: you may want to convert the PPM screenshots to PNG before uploading. That way, those who want to see the images will be able to do so with fewer mouse clicks ;)

PNG files are also usually a lot smaller.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on June 29, 2016, 10:45:23 PM
All color noise gone? Cannot believe my eyes.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on June 30, 2016, 07:27:09 AM
Here you go @michael84...

Canon ISO
(https://c7.staticflickr.com/8/7460/27713329230_f5cbc7326c_n.jpg) (https://flic.kr/p/JdW4gd)

ML ISO
(https://c1.staticflickr.com/8/7232/27713329120_424e851405_n.jpg) (https://flic.kr/p/JdW4ej)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: michael84 on July 04, 2016, 01:19:02 PM
ML ISOs tend to reduce what i recall i called "amplifier glow" (magenta-like glow in the shadows, visible when pushing high ISOs and/or in long exposures). Especially the preamp seems to be the major contributor here. I tried various settings starting from Canon ISO 200, from ADTG only (reduced until WL drop), to 0xFE gain 0 and preamp standard, to 0xFE 0 and pregain 1. Only lowering the pregain reduces the glow significantly.

Only ADTG: https://www.dropbox.com/s/qb09hezgo8crxml/5D3_ML%20ISO%20149_only%20ADTG%20crop.jpg?dl=0 (https://www.dropbox.com/s/qb09hezgo8crxml/5D3_ML%20ISO%20149_only%20ADTG%20crop.jpg?dl=0)

Only 0xFE: https://www.dropbox.com/s/o2a6ebavivq6o79/5D3_ML%20ISO%20137_only%200xFE%20crop.jpg?dl=0 (https://www.dropbox.com/s/o2a6ebavivq6o79/5D3_ML%20ISO%20137_only%200xFE%20crop.jpg?dl=0)

0xFE=0 and pregain 1: https://www.dropbox.com/s/jgls31kr5vfls4f/5D3_ML%20ISO%20110_0xFE%280%29_pregain%281%29_crop.jpg?dl=0 (https://www.dropbox.com/s/jgls31kr5vfls4f/5D3_ML%20ISO%20110_0xFE%280%29_pregain%281%29_crop.jpg?dl=0)


At high ISOs, ML ISO settings tend to clean up the colors a bit, and move the a bit toward green (presumably because the amp glow is reduced):

ISO 3200, Canon: https://www.dropbox.com/s/1ookn81oekq9pvb/5D3_C_ISO_3200%20Crop%202.jpg?dl=0

ISO 3200, ML (pulled from 6400): https://www.dropbox.com/s/afukgac8y90br1s/5D3_ML%20ISO%203200%28from6400%29%20Crop%202.jpg?dl=0 (https://www.dropbox.com/s/afukgac8y90br1s/5D3_ML%20ISO%203200%28from6400%29%20Crop%202.jpg?dl=0)


Now lets push the exposure a little bit:

Canon: https://www.dropbox.com/s/pdymz714lpov89u/5D3_C_ISO_3200%20Crop%202%20%28%2B1%20EV%29.jpg?dl=0 (https://www.dropbox.com/s/pdymz714lpov89u/5D3_C_ISO_3200%20Crop%202%20%28%2B1%20EV%29.jpg?dl=0)

ML: https://www.dropbox.com/s/yiq2qdeb4exkq95/5D3_ML_ISO_3200%20Crop%202%20%28%2B1%20EV%29.jpg?dl=0 (https://www.dropbox.com/s/yiq2qdeb4exkq95/5D3_ML_ISO_3200%20Crop%202%20%28%2B1%20EV%29.jpg?dl=0)


BTW, is there any way to convert a DNG to ML RAW video or MLV? So that i could try the vertical stripes fix from MLV Mystic to reduce the banding noise?
Title: Re: CMOS/ADTG/Digic register investigation on ISO ... Status update appreciated
Post by: l_d_allan on August 16, 2016, 01:29:59 AM
Not meaning to be impatient, but did the research on this get to the point of making it into a "Nightly Build"?

The info on the introductory OP from Jan, 2014 seems to indicate it is still experimental if someone is willing and able to do a compile from source. Or not?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: markanini on September 12, 2016, 01:39:44 AM
Do devs need help with testing on 600D at this time?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on September 12, 2016, 09:47:24 AM
Yes, any test you find on this thread, that was not done on 600D.

Same for any other camera ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: markanini on September 12, 2016, 05:39:41 PM
Quote from: a1ex on September 12, 2016, 09:47:24 AM
Yes, any test you find on this thread, that was not done on 600D.

Same for any other camera ;)

I'll start reading through the thread when I'm less busy. Though I don't mind  if I can be pointed to specific test procedures in the mean time.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DerMax on September 28, 2016, 02:17:27 PM
Lets say I'd like to play with iso_regs module..
where can I find it? :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Walter Schulz on September 28, 2016, 02:44:19 PM
You may want to read first post of this very thead.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on September 28, 2016, 03:02:36 PM
Updated first post - on 5D3, you may now use the crop_rec build to run these tools (they no longer require GDB (https://bitbucket.org/hudson/magic-lantern/branch/iso-research)).

FYI, the latest adtg_gui contains an experimental 3K crop (http://www.magiclantern.fm/forum/index.php?topic=17021.msg171809;topicseen#msg171809) mode for 5D3 as well.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DerMax on September 28, 2016, 05:24:51 PM
Thank You! ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jasonrwoods on January 25, 2017, 06:32:29 PM
Been lurking for a while, and have been running ML on my 6D for over a year and love it.  I finally am able to compile on my Mac after a week of trial and much error (I know nothing about coding).  I'm trying to run the ADTG module and edit the registers.  (yes, I've enabled CONFIG_GDB)

My problem is that when I try to run the adtg_gui module, I get these errors:

Quotetcc: error: undefined symbol 'strstr'
tcc: error: undefined symbol 'patch_hook_function'
[E]failed to link modules

Can anyone help a newbie out?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 25, 2017, 07:11:45 PM
Try compiling ML core as well from the iso-research branch, not just the module.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jasonrwoods on January 25, 2017, 11:47:02 PM
Quote from: a1ex on January 25, 2017, 07:11:45 PM
Try compiling ML core as well from the iso-research branch, not just the module.

Ah, thank you!!!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: festr on May 01, 2017, 06:10:46 PM
adtg_gui.mo - is there new version which is compatible with current experimental branches crop_rec? the currnet link on 1st page does not work with it
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: bpv5P on July 14, 2017, 02:36:14 AM
No progress? Need any testers for 600D? I could compile and help with some data. I would really like to see this on nightly builds...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 14, 2017, 08:06:43 AM
Welcome - feel free to experiment and share your findings.

FYI - the raw_diag module is on the downloads page (http://builds.magiclantern.fm/modules.html) for quite some time.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: bpv5P on July 18, 2017, 04:33:04 PM
Quote from: a1ex on July 14, 2017, 08:06:43 AM
Welcome - feel free to experiment and share your findings.

FYI - the raw_diag module is on the downloads page (http://builds.magiclantern.fm/modules.html) for quite some time.


Cool, thanks a1ex. I'll do some test and send some data...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: nikfreak on July 30, 2017, 10:47:13 AM
Probably got a freaky request but care to share a working adtg_gui.mo for 7D or is it impossible?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Kharak on July 30, 2017, 10:28:17 PM
@a1ex

Do you think we will see ISO 66 in any of the Nightlies or Experimentals? Or will it always be a "thing" from adtg_gui.mo? Also someone posted examples with the negative ML ISO and how there was almost no color noise? I don't remember where I saw those examples.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 31, 2017, 12:20:13 AM
It's on the back burner, but cannot promise any time frame.

As usual, the priority will get higher once there will be interest in experimenting with this stuff, rather than just asking for it to be done.

BTW, updated the iso-research branch for latest codebase and solved this error (https://www.magiclantern.fm/forum/index.php?topic=19307.msg182220#msg182220), but it's completely untested (other than checking whether it compiles). Sorry - currently working on something else.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on July 31, 2017, 01:41:20 AM
Quote from: nikfreak on July 30, 2017, 10:47:13 AM
Probably got a freaky request but care to share a working adtg_gui.mo for 7D or is it impossible?

+1
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 01, 2017, 12:47:26 AM
Quote from: nikfreak on July 30, 2017, 10:47:13 AM
Probably got a freaky request but care to share a working adtg_gui.mo for 7D or is it impossible?

http://www.magiclantern.fm/forum/index.php?topic=10111.msg144472#msg144472

Quote from: a1ex on July 31, 2017, 12:20:13 AM
As usual, the priority will get higher once there will be interest in experimenting with this stuff, rather than just asking for it to be done.

Do you have some specific things that need doing for this branch (things I am capable of)?

Quote from: a1ex on July 31, 2017, 12:20:13 AM
BTW, updated the iso-research branch for latest codebase and solved this error (https://www.magiclantern.fm/forum/index.php?topic=19307.msg182220#msg182220), but it's completely untested (other than checking whether it compiles). Sorry - currently working on something else.

My 6D is still in one piece.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 01, 2017, 01:45:59 AM
Quote from: Audionut on August 01, 2017, 12:47:26 AM
Do you have some specific things that need doing for this branch (things I am capable of)?

Restoring the missing screenshots would be a good starting point. It will probably take a while to re-read the thread and refresh my memory on all this stuff.

Some issues with raw_diag mentioned here, didn't look into them yet: http://www.magiclantern.fm/forum/index.php?topic=16516.msg186527#msg186527

Try to reproduce the above findings on different camera models. Some may need different registers.

Simply play with adtg_gui and document register behavior from your camera.

Find out how low you can push the ISO without getting artifacts in highlights. Different cameras (of the same model) might have different results.

Figure out the registers required for the ISO tweaks in LiveView (some are the same, others are not).

Script various test scenarios in Lua (e.g. cycle through all ISOs, take test pictures, change settings, create tables with results).

Evaluate the repeatability of our measurements under various test scenes.

Propose a way to measure the sensor response curve without expensive lab equipment.

Try the pattern noise reduction algorithm from MLVFS or mlv_dump on steroids on the images taken with ISO tweaks (IIRC 5D3 has problems with this, 6D less so). Try to fine-tune the parameters.

Figure out how to reproduce the ETTR + Dual ISO crash on the patchmgr branch (doesn't happen on unified; happens with low probability on patchmgr and any other branches that use this backend).

Figure out why this dark frame (http://www.magiclantern.fm/forum/index.php?topic=18975.msg186759#msg186759) introduced additional pattern noise, instead of removing it.

Just some random ideas. Don't limit yourself to these :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 02, 2017, 07:08:04 AM
Quote from: a1ex on August 01, 2017, 01:45:59 AM
Figure out why this dark frame (http://www.magiclantern.fm/forum/index.php?topic=18975.msg186759#msg186759) introduced additional pattern noise, instead of removing it.

Are you sure pattern noise has been introduced?  All of the other noise sources help to hide pattern noise.

Pattern noise removal really needs an average of several dark frames to be effective. (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 02, 2017, 08:06:16 AM
Quote from: Audionut on August 02, 2017, 07:08:04 AM
Are you sure pattern noise has been introduced?  All of the other noise sources help to hide pattern noise.

Pattern noise removal really needs an average of several dark frames to be effective. (http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise)

Maybe my pixel peeping skills aren't the best, but all files (input and output) and commands used are on the Jenkins page linked in that thread. The dark frame is averaged from 31 frames.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: keepersdungeon on August 20, 2017, 12:09:25 PM
Quote from: Levas on August 13, 2014, 05:27:22 PM
Thanks Greg,

I can get 3xzoom/crop mode on 6d by overriding the following registers:
                      1080p     10xzoom     3xzoom(1:1 crop)

ADTG2-805f      2b0        13c             No change needed
ADTG2-8061     2b0        13c             No change needed

CMOS[6]          400         d0             d0       (=Changing the "d" moves horizontal position on the sensor)
CMOS[7]             0         208          206       (Lowering the 8 to 6 moves the white bar outside 16:9 view)
ADTG2-800c         2           0              0         (vertical lines to skip)
ADTG2-8000        6           5             5       (Don't know what it does, if it is 6 you get some vertical interlaced looking footage...)


With these settings I get 3xzoom/crop mode, correct preview in live view on the 6d
Raw recording works, gives correct framing.
I can even record canon "native" h.264 with it, and it works too!

Just saw this! didn't even know it was possible on 6D
Anws was trying it,  and when I hit the crop 1:1 in the raw settings I can see 1826*1092 x3.0 but the recording is identical as the footage without hitting zoom
I missed something again didn't I?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on September 25, 2017, 02:57:06 PM
Just posted new builds of the iso-research modules (adtg_gui, iso-regs and raw_diag), to be loaded on top of the current crop_rec_4k builds (as the older crop_rec is no longer on the download page).

http://builds.magiclantern.fm/modules.html#iso-research

If you decide to experiment with them, I'm looking forward to seeing your results.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: kichetof on September 26, 2017, 06:41:18 PM
@a1ex I would try to play with it, but I've an error with adtg module:

ML ASSERT:
new_entry->name
at ../../src/menu.c:1241 (menu_add_internal), task module_task
lv:0 mode:3

module_task stack: 1db7b8 [1dba50-1d7a50]
0xUNKNOWN  @ b68c8:1db878
0x00072678 @ a0f318:1db860
0x0006EA48 @ 726a4:1db848
0x0006EA48 @ 6ee30:1db818
0x00069FF0 @ 6ec90:1db7e8
0x00069868 @ 6a05c:1db7b8

Magic Lantern version : crop_rec_4k.2017Sep26.5D3113
Mercurial changeset   : 382dafb8a0bd (crop_rec_4k) tip
Built on 2017-09-25 23:32:14 UTC by jenkins@nightly.
Free Memory  : 134K + 3143K


It seems that there are a lot of changes in menu.c between crop_rec_4k and iso-research, I'll try to find how to solve it
https://bitbucket.org/hudson/magic-lantern/branches/compare/crop_rec_4k%0Diso-research#chg-src/menu.c
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on September 26, 2017, 07:03:36 PM
Right - it needs a valid name, such as .name = "(empty)". Did that to avoid a bunch of null pointer issues, but only tried iso_regs and raw_diag from the latest build.

Hopefully fixed, but unable to test right now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: kichetof on September 26, 2017, 08:59:00 PM
Quote from: a1ex on September 26, 2017, 07:03:36 PM
Hopefully fixed, but unable to test right now.

You're the king!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on October 06, 2017, 05:31:49 PM
Here's a quick example on how adtg_gui can be used to find interesting stuff.




In the crop_rec_4k branch I've used raw type 0x12 (DIGIC 5) (http://www.magiclantern.fm/forum/index.php?topic=19336.msg189750#msg189750) in order implement 8..12-bit lossless compression (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183106#msg183106). This works by keeping the 14-bit container (as I don't know how to reconfigure the encoder for other bit depths (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)) and darkening the image (applying a digital gain aka SHAD_GAIN) so it ends up with fewer levels (as many as a real 8..12-bit image (http://www.magiclantern.fm/forum/index.php?topic=19300.msg185746;topicseen#msg185746)).

This trick appears to work - it improves the compression ratio.

Problem: LiveView also gets darkened (as a side effect of bit depth reduction).

By default, ML uses raw type CCD (0x10 on D5, 0x5 on D4) (http://www.magiclantern.fm/forum/index.php?topic=18393.0), as this one gives consistent white levels. This raw type is not affected by ISO changes, therefore it doesn't react to digital gain (so there's no easy way to darken it digitally). I could darken it from analog controls, but didn't want to touch them for bit depth reduction.

Anyway - I'm currently trying to find a way to brighten the image of raw type 0x12 (DEFCORRE, apparently after SHAD in the pipeline, but before other image processing steps, such as the picture style), but *without* brightening the raw buffer contents. That means, I'm looking for some sort of digital gain later in the pipeline (after DEFCORRE).

Incidentally, I've noticed our low-light exposure simulation feature (Expo Override + ExpSim enabled), after a certain threshold (a few seconds of exposure at some high ISO), it brightens the LiveView image, but the raw histogram is no longer pushed to the right. In these (extreme) cases, on DIGIC 5, ML uses something like this:

call("lvae_setdispgain", some_gain); /* 1024 = 1.0 */


Problem: on 5D3, this call only works in photo mode...

On DIGIC 4, we've identified this register by brute force (http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force):

#define ISO_PUSH_REGISTER 0xc0f0e0f8


But that doesn't work on 5D3.

Looking in lvae_setdispgain (FF72C0B0 in 5D3 1.2.3), it stores the gain at 0x68C8C, prints a message and sets some flags. The actual configuration is likely done by some other code that reads this value. Unfortunately, that code was not identified by IDA's autoanalysis.

How to find it? This is likely some register used to configure LiveView, like those handled by adtg_gui. So, I've tried to find it using this tool.

How?

1) entered photo mode LiveView (where our function works), enabled ADTG Registers, DIGIC registers and selected "Modified from now on"
2) back to LiveView, then back to ML menu -> a bunch of registers changed. Selected "Modified from now on" again until things settled.
3) called the analyzed function (had it on "don't click me" with random gains) and noticed how the LiveView brightness changed
4) went back to adtg_gui menu and it printed these modified registers (5D3 1.1.3):

C0F42730: 0x1010202 (was 0x1)       - from AeWb:ff33ff94:6b84fc
C0F4273C: 0x46f076c (was 0x5640483) - from AeWb:ff33ff94:6b8514
C0F42740: 0x6900690 (was 0x3fd03fd) - from AeWb:ff33ff94:6b851c
C0F42744: 0x4040404 (was 0x0)       - from AeWb:ff33ff94:6b8524


5) Locked the above registers and confirmed the brightness could no longer be changed with lvae_setdispgain. That's a good sign - it means the register I'm looking for is one of these (or maybe all of them).

6) Changed the above registers to their old value (the "was" one). Result: LiveView back to original brightness (before calling lvae_setdispgain). Another good sign.

7) Now, to find out what they do - the "trial and error" part:

C0F42730 looks like a series of 4 8-bit values. Each of them appears to be a boost of the 4 Bayer channels. Valid values: 0...3 (4 is the same as 0). Making them equal will make the image brighter; otherwise, the colors will be changed.

C0F42744 is similar, but valid values go from 0 to 7. It also gives some Bayer artifacts.

The remaining two are 16-bit values (two values packed in one register). Probably some sort of WB gains.

C0F4273C: the lower half looks like red gain, the higher half looks like blue gain (range 0 - 7FF)
C0F42740: these look like green gains; if the two are different, some interesting artifacts appear - just like with C0F42744.

8.) With the above registers 4 locked, white balance controls (from Canon menu) are no longer working. This makes sense - these are all changed from the AeWb task.

Overriding just C0F4273C/40 does not completely lock WB controls - they still work in some rough increments.

9) Switched to Movie mode and tried to adjust the above registers. It works!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on October 06, 2017, 06:08:29 PM
Once again excellent progress with this troubleshoot of yours, @a1ex!  8)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: deix on October 29, 2017, 02:27:34 PM
Hi there,

will the "raw_diag.mo" be loadable also for the main branch, or is it only available for experimental crop_rec_4k build - I get the following version error while using the main build "magiclantern-Nightly.2017Oct04.60D111.zip"

"Wrong version(v7.0, expected v6.0)
http://ibb.co/fhzJrR (http://ibb.co/fhzJrR)

Best regards
Michal Powalko
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on October 29, 2017, 10:27:39 PM
Will be after merging lua_fix; meanwhile, you can use that experimental build to load it.

The other two modules also require the patch manager (not included in lua_fix, so they won't load on that build).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on December 09, 2017, 06:26:42 PM
ISO 109 tested for shadow noise improvement:

Three images, all settings identical except iso.  Exposed for highlights in a low light scene to really showcase shadow noise.  This time I adjusted register settings to compensate for highlights (as I should have before, but was too eager just to see results!)

No image processing except exposure compensation.  No noise reduction at all.

109 rocks.

5D3 1.13
(http://thumb.ibb.co/bsxgub/iso_100.jpg) (http://ibb.co/bsxgub)

(http://thumb.ibb.co/g7SEEb/iso_109.jpg) (http://ibb.co/g7SEEb)

(http://thumb.ibb.co/cgLefG/iso_200.jpg) (http://ibb.co/cgLefG)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: johannsebastianbach on February 09, 2018, 09:03:00 PM
Quote from: 70MM13 on December 09, 2017, 06:26:42 PM
ISO 109 tested for shadow noise improvement:

Three images, all settings identical except iso.  Exposed for highlights in a low light scene to really showcase shadow noise.  This time I adjusted register settings to compensate for highlights (as I should have before, but was too eager just to see results!)

No image processing except exposure compensation.  No noise reduction at all.

109 rocks.

5D3 1.13
(http://thumb.ibb.co/bsxgub/iso_100.jpg) (http://ibb.co/bsxgub)

(http://thumb.ibb.co/g7SEEb/iso_109.jpg) (http://ibb.co/g7SEEb)

(http://thumb.ibb.co/cgLefG/iso_200.jpg) (http://ibb.co/cgLefG)

Wow, this is crazy!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on February 12, 2018, 02:37:09 AM
Yes, it works extremely well!
I shared a music video I shot using this mode on the "share your videos" forum.  Check it out!
This experimental iso mode is long overdue for serious attention...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on February 12, 2018, 04:55:40 AM
Hi @70MM13
Really nice trick at ISO 109!
But how you got it? and how did you know you are at ISO 109?
I played with CMOS 0 register , the ISO was changed but the value stayed 100 on ML.
Can you explain your steps?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on February 12, 2018, 10:02:16 AM
Hi, help! receive error when enable modules adtg_gui.mo and raw_diag.mo on the camera 6d(1.1.6)

adtg_gui.mo
tcc: error: undefined symbol "unpatch_memory"
tcc: error: undefined symbol "patch_hook_function"
[E] failer to link modules

raw_diag.mo
Wrong version (v7.0, expected v6.0)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on February 12, 2018, 11:13:00 AM
I also have a question about this magic ISO 109.  Isn't is possible hard code it and add it to the ML ISO menu, along with other similar ISO numbers, so that users can easily select them when needed?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 12, 2018, 11:57:44 AM
It is, but first I need to know what to hardcode (adtg_gui). The tweaks (and how far they can be pushed) are model-specific, though they usually follow a similar pattern - certain analog gains can be reduced, revealing additional highlight detail.

There are LOTS of things that can be done by users with minimal coding skills - be sure to read the previous posts for more details. For example, a huge time saver (for me) would be to automate the process of evaluating the equivalent ISO (clipping point) and the resulting dynamic range under various conditions (photo mode, movie mode 1080p/720p/x5/crop/etc, iterating through all ISOs). That should be doable with Lua scripting around raw_diag and adtg_gui. For more ideas, be sure to read the previous posts.

A review of the methods I'm using for evaluating dynamic range and clipping point, maybe by cross-checking the results with other software/methods, would be also very welcome. Details can be found in previous posts as well. In particular, I'm interested in double-checking the clipping point (is that ISO really 109 or is it more like 150 or 75?), the repeatability of the results (are different users following the same test procedure getting similar results, or are the results very different?) and the sensor response curve (is it linear?). All without requiring special equipment for measurements (color chart, second camera or household lights are OK, but that's pretty much it).

@Ron100: try the experimental 6D builds with lossless compression (from the forum), or compile from source (iso-research branch). Modules compiled from that branch are also loadable on crop_rec_4k (all) and lua_fix (only modules without low-level hacks, such as raw_diag).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on February 12, 2018, 02:30:13 PM
a1ex, please give a link to experimental builds 6D with lossless compression (from the forum). I looked at the whole branch, but I couldn't find it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on February 12, 2018, 03:22:04 PM
Regarding the video, I didn't use "iso" 109, but more like 80-320, based on isos 160-640.

I don't have a clue as to what the resulting isos really are, I just go by what magic lantern tells me.

I found that using actual iso 320-400 as the base gives the best results for my video work, particularly for very low light conditions.

I have a very simple method based on my own experiments following a quick reading of a1ex's information contained within this thread:

1: clear the custom iso settings
2: set desired base iso (canon menu)
3: take a silent photo
4: "copy canon settings"
5: set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value

That's it!

I'm sure that better results are obtainable, but I'm very happy with what I'm producing using this simple method!

I've filmed a lot of scenes using this method, and it always works well for me.  I'd be overjoyed to have a simple menu option for this setting!

It's not a huge hassle to do manually, just a bit of a time waster.

Thanks a1ex and everyone for the magic of magic lantern!

I'm looking forward to seeing what's possible when we really nail down the best settings!

Edit: added missing step 4
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on February 12, 2018, 03:37:42 PM
@Ron100
Here is a link (https://bitbucket.org/daniel_fort/magic-lantern/downloads/) from @dfort's downloads page you will find experimental build for 6d.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on February 12, 2018, 04:42:53 PM
Thanks! @70MM13
Quote from: 70MM13 on February 12, 2018, 03:22:04 PM
4: set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value
So you are using iso_regs.mo to adjust these settings? which only compatible with 5D3, I am using 700d no luck :-[
But wait :D @a1ex How it's hard to make working it on 700d ? What should I change, the registers only?

Can you help with that and take me to the right place?
If I can use adtg_gui instead, please tell me that. And some useful links to that I will be appreciate it!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on February 12, 2018, 06:04:04 PM
Yes, you can use adtg_gui to perform all the tweaks discussed in this thread. Refer to previous posts - there were many example screenshots back then (unfortunately, some of them were eaten by Dropbox).

Some of the screenshots may be available on archive.org or similar websites; if you have the time to dig and rescue any of them (or maybe just remind the original authors to reupload them), I can re-host them on ML server and update the links.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on February 12, 2018, 06:19:24 PM
Quote from: a1ex on February 12, 2018, 11:57:44 AM

There are LOTS of things that can be done by users with minimal coding skills - be sure to read the previous posts for more details. For example, a huge time saver (for me) would be to automate the process of evaluating the equivalent ISO (clipping point) and the resulting dynamic range under various conditions (photo mode, movie mode 1080p/720p/x5/crop/etc, iterating through all ISOs). That should be doable with Lua scripting around raw_diag and adtg_gui. For more ideas, be sure to read the previous posts.

A review of the methods I'm using for evaluating dynamic range and clipping point, maybe by cross-checking the results with other software/methods, would be also very welcome. Details can be found in previous posts as well. In particular, I'm interested in double-checking the clipping point (is that ISO really 109 or is it more like 150 or 75?), the repeatability of the results (are different users following the same test procedure getting similar results, or are the results very different?) and the sensor response curve (is it linear?). All without requiring special equipment for measurements (color chart, second camera or household lights are OK, but that's pretty much it).

There is no doubt that this is another very exciting step in ML development that could vastly improve shooting experience and final results.  The problem is that most people, including me, don't understand how to do these tests, despite the detailed explanations in this thread.  In my opinion, it would be very helpful if someone would agree to make a video tutorial, explaining the test and automation (optimization) procedure step by step.  In this way, several people could give it a try and share their results with their camera models. 

A1ex, do I understand properly that the 5D3 with its magic ISO numbers from the first post: 66, 115, 230, 460, 912, etc. is ready for hardcoding these numbers in the ML ISO menu and such magic numbers are needed also for the other camera models?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on February 12, 2018, 06:36:32 PM
Thanks, I installed the experimental build 6d and enabled the module adtg_gui.mo

I am willing to carefully experiment,but maybe someone detailed the instructions for use of the module adtg_gui.mo?

I fully support IDA_ML, create a video manual would be great!!!

P.s. Can decrease the ISO to give a greater preserving highlight detail than Canon's tone priority (d+)?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on February 13, 2018, 08:59:34 PM

(https://thumb.ibb.co/nm4JJ7/rock_elm_ice.jpg) (https://ibb.co/nm4JJ7)


Here's a photo I snapped using the technique I listed above, base iso 320, resulting "iso" 160.

Extreme dynamic range in the scene, full sun and shade in snow, sunlight reflected in the ice...

No postprocessing except for a nice fujifilm lut for my taste.

This amplifier hacking is incredible.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on February 15, 2018, 03:11:12 PM
RAW_DIAG - video instruction
https://www.youtube.com/watch?v=kt8l7GKi6n0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on February 18, 2018, 08:51:28 AM
how to make iso 6D reduction in Cmos[3] iso100 to iso77?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 14, 2018, 04:18:55 PM
here's an excellent example of the benefits of amplifier gain optimisation:

the two images are frame grabs from MLV video.  no noise reduction applied.

one is using "straight" iso 200, while the other is using 200 dropped to 100 in the usual manner.

check out the difference in the chroma noise on the pant legs and the jacket sleeve.  there's none in the optimised shot!

this REALLY needs to become mainstream.

now.

please!

(https://thumb.ibb.co/cW1M2S/amplifier_adjusted.jpg) (https://ibb.co/cW1M2S)

(https://thumb.ibb.co/kbs5U7/default.jpg) (https://ibb.co/kbs5U7)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: michael84 on June 13, 2018, 12:02:06 PM
I have a problem with the current iso_regs module in the 4k crop_rec build. I can only adjust the ADTG gain in very coarse steps (100 or 200), an the ADTG preamp in steps of five until i am below 30 or so. Is this a bug, or do i do something wrong? I did not have these problems with older builds.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on June 15, 2018, 05:10:17 PM
Confirmed here. Same behaviour.
5dmkiii 1.13

I installed it to check, and immediately switched back to the December 2017 build.  It works correctly.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 10, 2018, 01:17:03 PM
It's a real curiosity that this amplifier optimization is being ignored.

Those using dual iso would benefit from this also, because even at canon iso 100, shadows are just plain ugly.  Not so after gain optimization!

AHH, the joys of bias and prejudice.  Why, I wonder, should it apply to this?

"Cornelius..."
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 10, 2018, 04:41:07 PM
@70MM13

Can you compare please 100 from 200 ISO (Your technique) vs Dual ISO 100/800 vs normal Canon ISO at 100. And upload DNG sample please for each one.

Thanks!

Edit: Can you even get 100 from 400 ISO,, its there better result ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 10, 2018, 05:18:58 PM
Will do.

Yes, you could drop 400 to 100, but 200 is the sweet spot for dynamic range...

Quote from: theBilalFakhouri on July 10, 2018, 04:41:07 PM
@70MM13

Can you compare please 100 from 200 ISO (Your technique) vs Dual ISO 100/800 vs normal Canon ISO at 100. And upload DNG sample please for each one.

Thanks!

Edit: Can you even get 100 from 400 ISO,, its there better result ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on July 10, 2018, 05:34:34 PM
Yes some comparison would be nice which shows tge clear benefits. No time atm to dig into anything but checking is always possible :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 10, 2018, 07:43:49 PM
ok, here it is!

these images were created by loading the DNGs into mlv app, and dropping the dark strength to zero.

note the aliasing on the dual iso shot.  no surprise there...

also note that the optimised "experiments" shot is beautiful.  no hassle, no slow post processing, no unexpected behaviours and side effects, etc.

the canon 100 iso... sucks.

CAN WE FINALLY DEVELOP THIS????????????????????????
what the.
i'm done pushing for it.  if this isn't enough proof, the world has gone mad.

Don't forget, these results are just from me arbitrarily reducing ISO 200 to 100.  Who knows how much better it can get!

No one will know unless this gets some serious attention.

here's the DNGs, zipped:  http://drive.google.com/open?id=145YfRJB9lbauqHGJpHTzbD__iBYj4WOt


(https://thumb.ibb.co/jU0Zbo/optimised_iso_100_M10_1206_frame_1.jpg) (https://ibb.co/jU0Zbo)

(https://thumb.ibb.co/e8s2U8/canon_iso_100_M10_1212_frame_1.jpg) (https://ibb.co/e8s2U8)

(https://thumb.ibb.co/m6JWNT/dual_iso_100_800_M10_1213_frame_1.jpg) (https://ibb.co/m6JWNT)

funny music poems (https://poetandpoem.com/music)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 10, 2018, 08:52:10 PM
Wow! It's Nikon quality in Canon cameras! Dual ISO without dual_iso.mo at least for clean shadows!
I still confusing how I can do this in 700D,, maybe I should read from the first post.

@a1ex.. COME ON :D very BIG DEAL.

How we can do this ? In a module or something like "Optimized ISO" in ISO submenu ON/OFF. Is the values should be hardcoded  for each camera?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilia3101 on July 10, 2018, 09:52:56 PM
Nice shadows. How's the highlights doing?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 10, 2018, 10:19:17 PM
Grab the DNGs and play, or better still, play with the iso experiments tools and help get the ball rolling!

Quote from: Ilia3101 on July 10, 2018, 09:52:56 PM
Nice shadows. How's the highlights doing?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ArcziPL on July 10, 2018, 10:34:51 PM
Quote from: 70MM13 on July 10, 2018, 07:43:49 PM
these images were created by loading the DNGs into mlv app, and dropping the dark strength to zero.

note the aliasing on the dual iso shot.  no surprise there...

also note that the optimised "experiments" shot is beautiful.  no hassle, no slow post processing, no unexpected behaviours and side effects, etc.

the canon 100 iso... sucks.
Absolutely stunning results!

EDIT: but... after closer look the "optimized" file seems to be amplified by about +0.85EV, i.e. the overall brightness is matched only after pushing the normal ISO100 by +0.85EV from RAW. One can also already notice loss of data in the highlights on the metal cover of the floppy disk on the left. So, unfortunately, it's not comparing apples with apples. I don't say it doesn't have a potential but a better comparison is needed to quantize the difference in DR.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 10, 2018, 10:53:19 PM
Exactly.  But as I've said all along, I'm not doing anything special or analytical.  I'm just happy with the results I'm getting, and want to see this taken seriously so we can all benefit from this!

Quote from: ArcziPL on July 10, 2018, 10:34:51 PM
Absolutely stunning results!

EDIT: but... after closer look the "optimized" file seems to be amplified by about +0.85EV, i.e. the overall brightness is matched only after pushing the normal ISO100 by +0.85EV from RAW. One can also already notice loss of data in the highlights on the metal cover of the floppy disk on the left. So, unfortunately, it's not comparing apples with apples. I don't say it doesn't have a potential but a better comparison is needed to quantize the difference in DR.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 11, 2018, 12:46:46 PM
Before this thread once again fades to oblivion, I'll just provide this final thought:

I'm not looking for reasons NOT to take advantage of this excellent functionality.  I'm not shooting pictures of charts or cats.  I'm making beautiful video using this experiment ISO, without knowing what I'm doing.  I'm just loving the clean shadows and actually using this.

Watch this video, and then you'll see for yourself.

See any aliasing?

I had to use canon ISO for the opening shot because it was filmed at 60fps, and I couldn't get the right results with the iso experiments build.  I had to apply extreme denoising to that shot to get a barely acceptable result, but you can still see how bad it is compared to the rest of the video.

Amplifier optimization is a great tool!

I'll just keep enjoying this incredible capability and scratching my head as to why no one else is.

http://drive.google.com/open?id=1RXZ5vrQggaX1EHi1Yeg0hA9gokJmnquc
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 11, 2018, 01:56:38 PM
A point has been made.
Further progress will result from development of the code, not beating a dead horse.

Regards,
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on July 20, 2018, 07:28:52 PM
Found yet another ISO-related gain: register ADTG[7]. Works for both stills and LiveView. There are 4 gains, 2-bit each, applied to columns. 5D3 appears to have two ADTG chips (ADTG2 and ADTG4); same values is written to both. Most other models have only one ADTG chip.

Values that give clean image:
- 0x00 (default, all gains set to 0b00)
- 0x55 (all gains set to 0b01)
- 0xAA (all gains set to 0b10)
- 0xFF (all gains set to 0b11)

Didn't evaluate gain values and noise performance yet. The column offsets have to be recomputed after changing this gain (takes a few seconds in LiveView).

And another one at ADTG[0x1B]. This one darkens the image a lot.

Also found a way to override registers on the fly, even if Canon firmware sets them only once during video mode switches. Previously, you had to get out of LiveView and back to see the effect of changing many registers. Now you can see the effect right away.

Didn't update builds yet; will do after looking into the issue reported earlier.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on July 20, 2018, 07:45:51 PM
Cool, on the fly
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 20, 2018, 07:49:55 PM
Alex, do you know if these new found gain registers are digital or analog ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 23, 2018, 07:39:41 PM
Porting iso_regs.mo to 700D in this commit (https://bitbucket.org/bilal_fakhouri/magic-lantern-crop_rec/commits/7e85c78efc711717a18563fba76eaa124ca28129) with some problems:

"There is problem with ADTG Preamp registers it's not giving the write value? Is there something specific for 5D3 I did not notice it in the code?
And also what the effect of these registers in 5D3 ? in 700D these registers doesn't change between the ISOs.

ADTG[FE] is missing in 700D how I can find the right register for that (for same effect of this register). Or how did you find it in 5D3?"

Okay but I can still make the trick only with ADTG 8882 to 8888 (ADTG Gain in iso_regs.mo) :

How the scene looks like without editing:
(https://preview.ibb.co/dKjEZJ/How_the_scene_looks_like_in_the_camera.png) (https://ibb.co/mmG6gy)

Normal ISO 100 (Pushed 4.00 ev in MLVApp) :
(https://preview.ibb.co/emfSod/Normal_100_ISO.png) (https://ibb.co/j22wgy)

ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp):
(https://preview.ibb.co/duMHod/400_to_100_ISO.png) (https://ibb.co/fafouJ)

Dual ISO 100/800 (Pushed 4.00 ev in MLVApp):
(https://preview.ibb.co/grs1EJ/Dual_ISO_800_100.png) (https://ibb.co/hWAXMy)

Very nice clean shadows with little small clipping of highlights in contrasty scenes only (there are no clipping highlights in the example above).

From 800 ISO to 100 (Using ADTG Gain) the shadows is more cleaner but with losing dynamic range in highlights of course. Now I am trying to restore that dynamic range (highlights) at ISO 800 with the registers that related to the gain and I am not really sure if that possible.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on July 24, 2018, 10:42:18 AM
I can confirm register ADTG Preamp not working and ADTG[FE] missing for eosm and 100D too same as Bilal:
https://bitbucket.org/Dannephoto/magic-lantern/commits/bcad4c9e1f86f5ce01bd9001dbd8f781a0cae457?at=iso-research_eosm_100D_and_more

On another note I´ve been corresponding with Bilal getting hints and reading conversations from Levas and a1ex realizing we can use analog gain registers to change 14bit lossless to lower bits. Working with higher resolutions:
https://bitbucket.org/Dannephoto/magic-lantern/src/ad649aff48a02c942a4ce6ecbe825af1da5097ea/modules/crop_rec/crop_rec.c

code excerpt:
    if (is_100D)
    {
        switch (crop_preset)
        {
            /* 3x3 binning in 720p (in 1080p it's already 3x3) */
            case CROP_PRESET_3x3_1X:
                /* ADTG2/4[0x800C] = 2: vertical binning factor = 3 */
                adtg_new[0] = (struct adtg_new) {6, 0x800C, 2};
                break;
        }

switch (crop_preset)
{
case CROP_PRESET_2K_100D:
case CROP_PRESET_3K_100D:
case CROP_PRESET_4K_100D:
{
/* assuming FPS timer B was overridden before this */
                int fps_timer_b = (shamem_read(0xC0F06014) & 0xFFFF) + 1;
                int readout_end = shamem_read(0xC0F06804) >> 16;    /* fixme: D5 only */

                /* PowerSaveTiming registers */
                /* after readout is finished, we can turn off the sensor until the next frame */
                /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
                /* to be tested to find out exactly how much */

/* will add 10bit through analog gain */
adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};

                adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
                adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
                adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */

                adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (6D/700D) */
                adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
                adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 1) }; /* PowerSaveTiming OFF (5D3) */

                adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */

                /* ReadOutTiming registers */
                /* these shouldn't be 0, as they affect the image */
                adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
                adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */
}
                break;
}
    }


.......

/* Values for 100D */
static inline uint32_t reg_override_2K_100d(uint32_t reg, uint32_t old_val)
{
    switch (reg)
    {
        /* raw resolution (end line/column) */
        /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
        case 0xC0F06804: return 0x53902a1; // 2520x1304  x5 Mode;
        case 0xC0F06014: return 0x71c;
        case 0xC0F0713c: return 0x535;

/* correct liveview brightness */
case 0xC0F42744: return 0x4040404;
    }

    return 0;
}



Will darken image lowering causing pinks(footage can easily be restored in Mlv App working the white level slider):
/* will add 10bit through analog gain */
adtg_new[0] = (struct adtg_new) {2, 0x8882, 0x46};
adtg_new[1] = (struct adtg_new) {2, 0x8884, 0x47};
adtg_new[2] = (struct adtg_new) {2, 0x8886, 0x46};
adtg_new[3] = (struct adtg_new) {2, 0x8888, 0x45};


Preserving the liveview brightness:
/* correct liveview brightness */
case 0xC0F42744: return 0x4040404;



Only caveat right now is that running "framing" preview(slow but correct preview) we still get the darkened image since white level preview is reported wrongly. Seems fixable? But where? raw.c, mlv_lite.c or by registry change in camera? Is it fixable? If so we could have working 10/12bit lossless for higher resolution probably without needing to fix white levels in post.

Major thanks to Bilal for handing me registers and what and where to look for answers. And Levas for pointing at the analog gain registers in conversations with a1ex.

builds for 100D available. Code runs in crop_rec settings:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Jul23.100D101_2K-3K-4K_10bit_patch_g3gg0.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Jun30.100D101_2K-3K-4K_10bit_patch.zip


Back to vacation obligations....
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: bouncyball on July 24, 2018, 03:08:31 PM
"ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp)" vs "Dual ISO 100/800 (Pushed 4.00 ev in MLVApp)" IS amazing!
It looks so organic and nice in comparison to dual iso aliased image!!! And low noise levels are the same.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on July 24, 2018, 05:17:16 PM
Quote from: bouncyball on July 24, 2018, 03:08:31 PM
"ISO 400 from Canon to 100 (Pushed 4.00 ev in MLVApp)" vs "Dual ISO 100/800 (Pushed 4.00 ev in MLVApp)" IS amazing!
It looks so organic and nice in comparison to dual iso aliased image!!! And low noise levels are the same.

Just checked on my computer. It looks amazingly clean.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on July 24, 2018, 05:27:59 PM
Yeah, hard coding a few of these magic numbers to each of our cameras would be God sent !  Sometimes, even a fraction of a stop improvement in dynamic range can make a HUGE difference in the signal to noise ratio in the Shadows after proper exposing to the right.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dfort on July 25, 2018, 12:59:53 AM
Got a note that I've been sitting on for a while and am not sure which branch I should be working on.

The EOSM (and some others) require ENGIO_WRITE_FUNC and ENG_DRV_OUT_FUNC in order to compile the adtg_gui module in the lua_fix, crop_rec_4k and other branches.

modules/adtg_gui/adtg_gui.c
    else if (is_camera("EOSM", "2.0.2")) // from 1%
    {
        ADTG_WRITE_FUNC = 0x2986C;
        CMOS_WRITE_FUNC = 0x2998C;
        ENGIO_WRITE_FUNC = 0xff2c19ac;
        ENG_DRV_OUT_FUNC = 0xff2c1694;
    }


It would make sense to put this into the iso-research branch but surprise, it isn't needed in that branch. Where should it go so we don't have to keep looking up these stubs when compiling the adtg_gui module these other branches?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 25, 2018, 01:39:23 AM
@dfort

It would be nice if you made a PR for these addresses in iso-research branch. Here it is for 700D:
else if (is_camera("700D", "1.1.5"))
    {
        ADTG_WRITE_FUNC = 0x178FC; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0x17A1C; //"[REG] ############ Start CMOS"
ENGIO_WRITE_FUNC = 0xFF2C2D00;  // from stubs
        ENG_DRV_OUT_FUNC = 0xFF2C29E8;
    }
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 25, 2018, 10:07:51 AM
Found some ADTG gain registers in 700D:

Negative gain     Original value     Overridden
ADTG2[21] =        0x4626              0x1036
ADTG2[22] =        0x4626              0x1036
ADTG2[23] =        0x4626              0x1036
ADTG2[24] =        0x4626              0x1036

It darken the image about 1.5 stop with some vertical artifacts and increased noise in the shadows and it maybe affects the satrution.

Positive gain     Original value     Overridden
ADTG2[0] =        0x2062              0x2762
ADTG2[2] =        0x2066              0x2766
ADTG2[4] =        0x2062              0x2762
ADTG2[6] =        0x206c              0x276c

It bright the image about 2 stops? with shifting highlights of course.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ghost0cnc on July 25, 2018, 05:25:37 PM
Quote from: theBilalFakhouri on July 23, 2018, 07:39:41 PM
Very nice clean shadows with little small clipping of highlights in contrasty scenes only (there are no clipping highlights in the example above).

From 800 ISO to 100 (Using ADTG Gain) the shadows is more cleaner but with losing dynamic range in highlights of course. Now I am trying to restore that dynamic range (highlights) at ISO 800 with the registers that related to the gain and I am not really sure if that possible.

The biggest increase in dynamic range I could get on my 5D3 was about one stop at ISO 200->100.

Let's assume you can also get about 1 EV of extra dynamic range by reducing the analog gain (like on the 5D3).

At ISO 400->100 or even 800->100 you will get about 1 EV of extra dynamic range (that was clipped before). The remaining 1 or 2 EV will just darken the photo.

Let's have a look at 400->100:

Basically you dropped the analog gain by 2 EV, which is 1 EV below the saturation point, and pushed that 1 EV back up in post-processing, effectively replacing 1 EV of analog gain with 1 EV of digital gain in post.

When using ISO 400->200, you can possibly get better noise levels [1] and more dynamic range than at 400->100 (at the same shutterspeed+aperture).

Just reducing the analog gain will not help reducing noise. After passing the sweet spot that allows you to capture more light and/or use higher "base"/CMOS ISOs (which will improve SNR), you just replace analog with digital amplification (which is a problem due to ADC noise and limited precision, as you only use a fraction of the 14 bit resolution, but can be better, depending on the analog amplifier noise).

Test on the 5D3:
Same subject and lighting, shutterspeed+aperture fixed. WB and all other settings except "Exposure" (in LR) are the same.
(Sharpening 33/1.0/25/25, Luminance NR disabled, Color NR 25/50/75)
a) ISO 800->400 (-1 EV, sweet spot)
b) ISO 800->200 (-2 EV, 1 EV below sweet spot)
Pushed both images in Lightroom to about the same brightness (a +4EV, b +5EV). I know the exposures are not 100% identical, but the difference should be small enough.
Noise levels are pretty similar (hard to compare as the color-rendering is a bit off), but photo "b" has much stronger banding.

(https://preview.ibb.co/dCovDo/5_D3_800_400_4_vs_800_200_5_crop.png) (https://ibb.co/jbsNto)

Fine-tuning the ADTG parameters is another story. There are many combinations that result in the same amplification but differ in image quality. Sadly I haven't had time to analyze which settings work best.

[1] Better ADC SNR, as you have a stronger analog signal; depends on the noise generated by the analog amplifier vs ADC noise
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on July 28, 2018, 02:30:19 PM
Inspired by this discussion, I have been testing ISO 320 pulled down to 100.

I tried a few other Isos as well and found that 320 is excellent for the cleanest shadows and only a very small loss of highlights in actual usage.

The highlights clip straight to pink, making it very easy to expose exactly at the threshold, and the beautiful shadows are a joy to work with. In the example frame, I exposed for the direct sunlight on the wall. The room was dark otherwise.
(https://thumb.ibb.co/fMTZeT/early_morning_sunlight_and_darkness.jpg) (https://ibb.co/fMTZeT)


I'm using 320 a lot now, except those times I need as much highlight detail as possible at the expense of slightly noisier shadows.  Then it's back to 200!

Excellent stuff.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: theBilalFakhouri on July 28, 2018, 03:05:44 PM
Quote from: ghost0cnc on July 25, 2018, 05:25:37 PM
Let's have a look at 400->100:

Basically you dropped the analog gain by 2 EV, which is 1 EV below the saturation point, and pushed that 1 EV back up in post-processing, effectively replacing 1 EV of analog gain with 1 EV of digital gain in post.

When using ISO 400->200, you can possibly get better noise levels [1] and more dynamic range than at 400->100 (at the same shutterspeed+aperture).

Just reducing the analog gain will not help reducing noise. After passing the sweet spot that allows you to capture more light and/or use higher "base"/CMOS ISOs (which will improve SNR), you just replace analog with digital amplification (which is a problem due to ADC noise and limited precision, as you only use a fraction of the 14 bit resolution, but can be better, depending on the analog amplifier noise).

Yes you are right I noticed that the whole story about setting ISO 400 as the base ISO from Canon which will help for reducing noise in the shadows (without tweaking anything) and with losing some of dynamic range (clipped highlights) and we can restoring already some of this highlights (some of that dynamic range)  by tweaking down ADTG 8882 to 8888 the analog gains registers .

I did 400 to 100 ISO only for matching up the two exposures in normal 100 ISO and with the trick but then it was not okay to do that because of pink highlights I couldn't fix it in some cases. So it's better to find the sweet spot as you mentioned and stop there :D .

But I am confusing how Preamps registers are working ? what it does in 5D3 for enhancing the shadows ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jpegmasterjesse on July 28, 2018, 06:32:54 PM
I've been enjoying keeping up with this thread, but I'll admit that changing registers is a bit above my skill level. I'd really appreciate a step-by-step to try some of this out on my 5d2.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 50mm1200s on July 29, 2018, 06:18:05 AM
Best settings from @Audionut and @a1ex, for 5D3:
https://www.magiclantern.fm/forum/index.php?topic=10111.msg97743#msg97743
https://www.magiclantern.fm/forum/index.php?topic=10111.msg104278#msg104278

"Rules":
QuoteADTG gains and SaturateOffset can be used to recover some more highlight detail.
changing only ADTG gain is enough (because the other one runs out of range much quicker).
At some point, there's no more highlight detail to recover. When this happens, the white level will begin to decrease. Let's call this the sweet spot, and it can be found easily with binary search, for example.
QuoteOnce you've decreased the gain enough that your white level is dropping, you've already gone too far.

For 5D2:
QuoteSaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624. Also fix the digital gain (0xc0f08030) to 0x1000.
The effect of SaturateOffset is that it expands the recorded range while bringing in more highlight detail. So, if your default range is say 1024...15760 and you increase this range to say 32-16383 without changing digital gain (these are the 5D2 values), this would be equivalent to scaling ADTG gain to (15760-1024) / (16383-32) = 0.9 (that is, -0.15 stops)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: jpegmasterjesse on July 31, 2018, 01:45:38 AM
Appreciate it! However, I am getting some errors when I load raw_diag.mo and adtg_gui.mo

https://drive.google.com/file/d/1k9bWM2Wz3gorjws2lC36iam9-ngTwFau/preview
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on July 31, 2018, 04:27:02 AM
Quote from: a1ex on July 20, 2018, 07:28:52 PM
Found more ISO-related gain

Couldn't seem to get anything useful here.  Another that reduces DR by 0.3 is ADTG2[6] 0x0 > 0x1
Back when, I found a few that darkened the image considerably, but I don't recall if I ever mentioned them here.


For those following along with a 6D.
(https://s15.postimg.cc/ejq0m0jsp/6_D-_Stock.png?dl=1) (https://s15.postimg.cc/z3uuki79l/6_D-_Tweaked.png?dl=1)

ADTG2[8]  0x56 > 0x36
ADTG2[9]  0x55 > 0x35
ADTG2[a]  0x57 > 0x37
ADTG2[b]  0x57 > 0x37

CMOS[6]  0x0 > 0x2

ADTG2[8882] 0x400 > 0x330
ADTG2[8884] 0x400 > 0x330
ADTG2[8886] 0x400 > 0x330
ADTG2[8888] 0x400 > 0x330

ISO digital gain 0x200 > 0x229


I seem to recall CMOS[0] netting about 0.1 DR, but I'll have to dig through this thread to find what value was the winner.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on July 31, 2018, 12:35:08 PM
@Audionut
Interesting, the values for improving DR on the 6d.
Fiddled a little with it, but results were all rather small,   :P
Are you sure you need to change CMOS 6 form 0x0 to 0x2 ?
CMOS 6 is used for vertical position in crop record...

The values for ADTG2 8,9,a & b and 8882 to 8888 are interesting, I guess this could be used on all iso until iso 6400, right ?
That is, if you want maximum DR and not exact proper corresponding ISO/exposure values.

EDIT,

Weird, my standard values for 8,9,a and b registers are much lower than yours  ???
I have values of about '4B' and not '56'.
Put an extra tab in the 6d sheet and write out standard values of the registers for all iso's.
https://docs.google.com/spreadsheets/d/1iapLI7UrgfCJGwPSFsyhYeKl8fSTY4RVXvDR7MiaOp4/edit#gid=837197681 (https://docs.google.com/spreadsheets/d/1iapLI7UrgfCJGwPSFsyhYeKl8fSTY4RVXvDR7MiaOp4/edit#gid=837197681)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Kharak on July 31, 2018, 09:09:48 PM
Anyone have experience with ISO 66 on 5d3. How is it?

I remember a1ex got a small increase of DR about 0.2 in video and I think 0.5 in photo on ISO 66. I'd also like it for going under ISO 100, to possibly avoid having to put ND on. I cant seem to find that post.

Just read this article from imaging-recource.com about how the biggest pride of the Nikon sensor development team was that they managed to get the D850 sensor to go to ISO 64 at its lowest. Its a really good read for anyone interested in Sensor development and production. https://www.imaging-resource.com/news/2018/07/17/pixels-for-geeks-a-peek-inside-nikons-super-secret-sensor-design-lab
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on August 01, 2018, 12:59:15 AM
ISO 74 is the lowest I get while keeping the full range without losing highlights.

It's a pleasure to work with when there's lots of light.

My preference is for darker scenes, but the more I'm tinkering with the experimental ISO, the more comfortable I'm getting with a wider variety of conditions...

This frame was saved from mlvapp, in standard mode, no lut, no filters, no noise reduction, etc.  Simple exposure adjustment and low strength to raise the shadows a touch.

Looks good to me!
(https://thumb.ibb.co/b30Ove/M31_1604_frame_51.jpg) (https://ibb.co/b30Ove)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 01, 2018, 02:38:31 AM
Quote from: Levas on July 31, 2018, 12:35:08 PM
Are you sure you need to change CMOS 6 form 0x0 to 0x2 ?

I seem to recall a1ex mentioned this one worked in the shadows, vs other registers that affect highlights.  But I can't find the post.
Here is CMOS[6] 0x0 > 0x2
(https://s22.postimg.cc/e37vzjwyn/CMOS-0.png?dl=1) (https://s22.postimg.cc/vt9kkli9b/CMOS-2.png?dl=1)


Quote from: Levas on July 31, 2018, 12:35:08 PM
The values for ADTG2 8,9,a & b and 8882 to 8888 are interesting, I guess this could be used on all iso until iso 6400, right ?
That is, if you want maximum DR and not exact proper corresponding ISO/exposure values.

From my quick experiments, you should be able to hit a sweet spot with 8,9,a,b that works at all ISO's.  The sweet spot for 888? needs adjusting for all ISO's if you want maximum results.  Generally higher ISO's require higher values.  With 888?, you can simply find a value that works at all ISO's, with slightly (emphasis on slightly) less then optimal results.

There's some initial work by a1ex to determine what amps are first in the processing chain, berried (follow the links) in the link below.

0xFE should already be at 0x0 on 6D.

Don't forget the hidden ISO's. (https://www.magiclantern.fm/forum/index.php?topic=10111.msg124967;topicseen#msg124967)

edit:  Corresponding ISO values are of no concern to me.  I like saving those photons and not letting them get clipped.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 50mm1200s on August 02, 2018, 07:24:28 PM
I need some help here (disclaimer: newbie here). After reading all this thread and making some notes, I flashed a @dfort's build for 50D with the iso_research branch. Basically, from what I understood:
- Take a picture of a light bulb with normal settings. Use raw_diag.
- Enter menu, expand the BL and WL
- Use ADTG gains and SaturateOffset until the WL starts to decrease
- Increase DFE gain a bit
- Take another picture with these settings. Use raw_diag. Probably able to gain >0.5 on DR.

It seems easy, but on the "ADTG Registers" menu, I don't see how to change BL, WL and SaturateOffset you guys talk on this thread. How can I do that? Also, you folks write using EV values (like "decrease adtg to -0.37EV"), but I can only see registers (e.g, 0x344). I have to calculate these values manually? Here is what's on the menu:



ADTG1[105f]N
ADTG1[1061]
ADTG1[1172]
ADTG1[1173]
ADTG1[1178]
ADTG1[1179]N
DFE[1d02]
DFE[1d04]
DFE[1d06]
DFE[1d08]
CMOS[0]
CMOS[1]
CMOS[2]
CMOS[3]
CMOS[4]
CMOS[5]
CMOS[6]
ADTG1[9]
ADTG2[9]
ADTG1[b]
ADTG2[b]
ADTG1[13]
ADTG2[13]




I'm sure many people already suggested that, but, wouldn't be possible to create a script to test all the possible values (brutal force)?

I see there's some pressure here about making this feature into mainstream ML, but it's kinda difficult to help the research on the current state. Even people willing to spend hours reading can't easily get into it.
Based on the practical examples from this thread, this feature would have a high impact on the final result.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 03, 2018, 05:09:35 AM
Quote from: 50mm1200s on August 02, 2018, 07:24:28 PM
I need some help here.........It seems easy,

Looks can be deceiving.

Quote from: a1ex on January 10, 2014, 12:11:01 PM
What's the current state?
Research. We are trying to optimize the parameters that influence ISO, understand their effects (did we really gain 0.5 stops of DR or are we just daydreaming?) and port the results on other cameras..........



Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!


Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)

But if you have some basic coding/math skills - enough so you can follow the entire discussion without getting dizzy - I have some nifty research tools for you:

Don't assume that "we" know what we are doing, just because "we" post about the subject.


Quote from: 50mm1200s on August 02, 2018, 07:24:28 PM
but on the "ADTG Registers" menu, I don't see how to change BL, WL and SaturateOffset you guys talk on this thread. How can I do that?

These registers should be labeled in the camera menu, as per this code snippet.
https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/adtg_gui/adtg_gui.c

{0xC0F0,   0x8034, 0, "Black level in LiveView / BW offset in photo mode (SHAD_PRESETUP)"},
    //~ {0xC0F0,   0x8038, 0, "ISO related?"},
    //~ {0xC0F0,   0x8050, 0, "ISO related?"},
    //~ {0xC0F0,   0x814c, 0, "ISO related?"},
    {0xC0F0,   0x819c, 0, "Saturate Offset (photo mode) (HIV_POST_SETUP)"},
    {0xC0F1,   0x2054, 0, "White level?"},



Quote from: 50mm1200s on August 02, 2018, 07:24:28 PM
Also, you folks write using EV values (like "decrease adtg to -0.37EV"), but I can only see registers (e.g, 0x344). I have to calculate these values manually?

At this stage, yes.  If you understand the maths really well (I do not), I think it can be calculated.  Otherwise, use raw_diag to monitor the DR as you change the registers.

Quote from: 50mm1200s on August 02, 2018, 07:24:28 PM
I'm sure many people already suggested that, but, wouldn't be possible to create a script to test all the possible values (brutal force)?

This would make life so much easier.  Sounds like a coding nightmare.


Quote from: 50mm1200s on August 02, 2018, 07:24:28 PM
I see there's some pressure here about making this feature into mainstream ML, but it's kinda difficult to help the research on the current state. Even people willing to spend hours reading can't easily get into it.
Based on the practical examples from this thread, this feature would have a high impact on the final result.

See the first quote at the top of this post from a1ex. 

"Pressure" to put this feature into mainstream, isn't the same as doing what is needed to put this into mainstream.  It is very frustrating seeing such a wonderful feature not reaching it's peak value to the community, and being unable to help.  We should be mindful of not putting pressure on others to perform functions that we are otherwise unable too.

I am quietly confident that people such as a1ex can accomplish this task within a reasonably short timeframe.  I am extremely confident that a1ex is his own person, with his own desires and expectations, who has a keen understanding on keeping this development project (ML) moving in the best possible direction.  ML is more then just this single feature.  So for me, personally, I sit patiently, helping where I can, and respect the choices made by others within this project, especially when those choices made by others do not seem to fit within my own expectations of where the project should be heading.

Everything about this feature needs reverse engineering.  It would be extremely easier if Canon sent us a copy of their documentation on these registers.  But..........such is life.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 50mm1200s on August 03, 2018, 05:58:47 AM
I know this is an ongoing research and I'm familiar with the open source model. However:

Quote from: Audionut on August 03, 2018, 05:09:35 AM
"Pressure" to put this feature into mainstream, isn't the same as doing what is needed to put this into mainstream.

I'm not putting pressure. I said "I see there's some pressure here".
My point being: if more collaboration from the community is needed to make things faster, then it should be somewhat more accessible for people without programming/math skills to do the tests and share them.

Quote
Otherwise, use raw_diag to monitor the DR as you change the registers.

So, currently, you need to change each register, press HalfShutter to activate raw_diag and just then see the changes? This would take too much time, even for a simple test.
Since the brutal force idea seems to be too complicated, can't this be printed inside the menu (e.g "Current WL is xxxx", and so on), using a automatic low resolution silent pic as base for raw_diag calculation everytime a register is changed? Like a daemon in a operating system. DIGIC is not that powerful, but if the pic is lowres enough this might be possible, no?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 03, 2018, 10:59:00 AM
Quote from: 50mm1200s on August 03, 2018, 05:58:47 AM
I'm not putting pressure. I said "I see there's some pressure here".

(https://www.ford-trucks.com/wp-content/uploads/2016/01/beating-a-dead-horse.gif)

Quote from: 50mm1200s on August 03, 2018, 05:58:47 AM
My point being: if more collaboration from the community is needed to make things faster, then it should be somewhat more accessible for people without programming/math skills to do the tests and share them.

I'm pretty sure you missed the message I was trying to hammer home with the below quote.

My bold for emphasis.
Quote from: Audionut on August 03, 2018, 05:09:35 AM
I am quietly confident that people such as a1ex can accomplish this task within a reasonably short timeframe.  I am extremely confident that a1ex is his own person, with his own desires and expectations, who has a keen understanding on keeping this development project (ML) moving in the best possible direction.  ML is more then just this single feature.  So for me, personally, I sit patiently, helping where I can, and respect the choices made by others within this project, especially when those choices made by others do not seem to fit within my own expectations of where the project should be heading.

The point isn't to ask for assistance, so that you can assist.  There are a bunch of other tasks that this entire development project has outstanding.  Almost everybody can assist with some outstanding project tasks, now, which will free spare time for specific tasks, such as this feature, by those people with the skills to make it happen.  There is plenty of information available, describing how people can help the Magic Lantern development project (my signature should get the reader started).

Programming and maths are subjects that people spend a significant amount of money (and time) on, in educating themselves, on a daily basis.  People should not expect a bunch of guys playing around in their spare time, on a hobby project, without financial assistance, to educate others in these subjects.  That takes time!  The time being spent on this project by it's collaborators is already spare, not plentiful.


Quote from: 50mm1200s on August 03, 2018, 05:58:47 AMThis would take too much time, even for a simple test.

Yes, we know it does.  I personally have a couple thousand shutter actuation's on my hardware, from testing this feature.

What does this register do?
Tweak it.
Hit shutter.
Quickly observe raw zebras before the overlay is removed.
Observe raw_diag feedback.
Observe image taken.
Tweak register again.
Hit shutter.
Quickly observe raw zebras before the overlay is removed.
Observe raw_diag feedback.
Observe image taken.

Oh shit, I think I found a register that does something cool!

Repeat the above process some countless number of times.
Turns out to be nothing.............or maybe it turns out to be something, document as much as possible, present the data to people with more knowledge.

Move to the next register address.
Rinse and repeat.

I don't even bother to track the amount of time spent, I would feel guilty for my family.

Quote from: 50mm1200s on August 03, 2018, 05:58:47 AM
Since the brutal force idea seems to be too complicated, can't this be.....

Be careful, if you make it sound to easy, it will start smelling like an "easy coding task" (https://www.magiclantern.fm/forum/index.php?action=search)*.

*otherwise known as low hanging fruit.  A fairly simple coding task that does not have a significant effect on the development project, and is thus left as a "low hanging fruit" to encourage those with less skills to get involved.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on August 03, 2018, 01:19:16 PM
I find this functionality essential to my work, even though I only use the parameters on the ISO registers GUI.  I'm getting great results from using those, even if it could be better (what couldn't?)

That being said, it's extremely tedious and sometimes very frustrating to change the base ISO and redo the whole process just to be able to get a shot under much different lighting conditions.

All I ask is for information on any possible way to store and retrieve settings for faster setup in use.

Is it possible?

The ideal under current (and seemingly for years to come) circumstances would be "ISO presets" from the GUI, perhaps with the ability to provide a name for each to avoid confusion in the field.

I'm just asking for information.

Please keep it friendly.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 03, 2018, 05:49:04 PM
Quote from: 70MM13 on August 03, 2018, 01:19:16 PM
That being said, it's extremely tedious and sometimes very frustrating to change the base ISO and redo the whole process just to be able to get a shot under much different lighting conditions.

All I ask is for information on any possible way to store and retrieve settings for faster setup in use.

Is it possible?

Take the iso_regs module....
Quote from: a1ex on January 10, 2014, 12:11:01 PM
iso_regs.mo (https://builds.magiclantern.fm/modules.html#iso-research) - 5D3 only, requires either the crop_rec_4k build (http://www.magiclantern.fm/forum/index.php?topic=19300), or a custom ML build from the iso-research branch:

A research tool (or hacker's tool if you prefer) that lets you change most ISO-related parameters on 5D Mark III only and study their effect. Details here (http://www.magiclantern.fm/forum/index.php?topic=10111.msg96715#msg96715).
(http://a1ex.magiclantern.fm/bleeding-edge/iso_regs.png)

Adapt it for your camera.  Add in preset values to the menu entries.
The LUA scripting module (https://www.magiclantern.fm/forum/index.php?topic=14828.0) may be able to be used also, but I'm not entirely sure.

I like answering these questions, because I can provide answers.  I can't solve lack of skill with a swift hand movement (because I lack skills also), but I can provide assistance.  Perhaps the assistance is a little to blunt and to the point, but we're all fallible..............so.........

I do have a problem with being able to ignore the not so smart questions.  See below.

Quote from: 70MM13 on August 03, 2018, 01:19:16 PM
The ideal under current (and seemingly for years to come) circumstances would be "ISO presets" from the GUI, perhaps with the ability to provide a name for each to avoid confusion in the field.

I'm just asking for information.

This sounds like a thinly veiled "when will it be released" question.

Quote from: a1ex on January 10, 2014, 12:11:01 PM
Where's the download link?!??!?!!??!!!!!!!!!!!!!!!!!!?!?!?!?!?!


Take it easy, the current state is research. As in, "If we knew what it was we were doing, it would not be called research, would it?" (http://quotationsbook.com/quote/34064/)




When it will be released?


When it's ready. I also want to summarize the findings in a small paper (like the Dual ISO PDF (https://bitbucket.org/hudson/magic-lantern/downloads/dual_iso.pdf)), so I need a little time.

https://wiki.magiclantern.fm/faq#any_progress_on_xyz
https://wiki.magiclantern.fm/faq#when_will_you_implement_xyz
https://wiki.magiclantern.fm/faq#when_will_you_release_the_next_version
http://www.catb.org/esr/faqs/smart-questions.html

Quote from: 70MM13 on August 03, 2018, 01:19:16 PM
Please keep it friendly.

http://www.catb.org/esr/faqs/smart-questions.html#not_losing
QuoteOdds are you'll screw up a few times on hacker community forums — in ways detailed in this article, or similar. And you'll be told exactly how you screwed up, possibly with colourful asides. In public.

When this happens, the worst thing you can do is whine about the experience, claim to have been verbally assaulted, demand apologies, scream, hold your breath, threaten lawsuits, complain to people's employers, leave the toilet seat up, etc. Instead, here's what you do:.............

Community standards do not maintain themselves: They're maintained by people actively applying them, visibly, in public. Don't whine that all criticism should have been conveyed via private e-mail: That's not how it works. Nor is it useful to insist you've been personally insulted when someone comments that one of your claims was wrong, or that his views differ. Those are loser attitudes..............

There have been hacker forums where, out of some misguided sense of hyper-courtesy, participants are banned from posting any fault-finding with another's posts, and told "Don't say anything if you're unwilling to help the user." The resulting departure of clueful participants to elsewhere causes them to descend into meaningless babble and become useless as technical forums.

Exaggeratedly "friendly" (in that fashion) or useful: Pick one.............

Remember: When that hacker tells you that you've screwed up, and (no matter how gruffly) tells you not to do it again, he's acting out of concern for (1) you and (2) his community. It would be much easier for him to ignore you and filter you out of his life. If you can't manage to be grateful, at least have a little dignity, don't whine, and don't expect to be treated like a fragile doll just because you're a newcomer with a theatrically hypersensitive soul and delusions of entitlement.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Ron100 on August 08, 2018, 12:11:03 PM
Dear friends,are there ADTG values (8, 9, a, b and 8882 - 8888)for maximum highlights saving in photo mode??
on 6D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ghost0cnc on August 21, 2018, 11:28:19 PM
Yesterday I encountered serious vertical fixed-pattern banding on some images. On other images there was no banding at all.

5D Mark III 1.2.3 + ML crop_rec_4k.2018Mar10.5D3123
Canon EF80-200mm f/2.8L (very old lens)

Same exposure settings and iso_regs configuration on all images, iso_regs modified to enable the "CMOS-Patch".
AI Servo, 1/1000s, f/2.8, Canon ISO 3200, tweaked with the following overrides:

CMOS[3] = 2372 (0x944)
CMOS[4] = 792 (0x318)

ADTG-0xFE = 4
ADTG-Preamp (8/9/A/B) = 2 (2 is the lowest value that does not cause banding due to an underflow of one of the other parameters)
ADTG-Gain (8888) = 950
SaturateOffset = 1200 (caused some green tint issues on a few older images)



Top 800px of each image at +2EV and the same white-balance applied, images 1 + 2 with banding, 3 without:
(https://milanstephan.de/ext/ml/5d3_banding_top_small.jpg) (https://milanstephan.de/ext/ml/5d3_banding_top.jpg)

Top "optical black" bars of the three images (https://milanstephan.de/ext/ml/5d3_banding_ob.png) (dcraw -T -E -W -b 1 -g 50 500)
The fixed-pattern noise is pretty visible. The last ob-image (without banding) is far brighter - if these dcraw settings are not useful for a comparison, please tell me better settings and I will update the image.

Download the RAWs (90 MB, zip) (https://milanstephan.de/ext/ml/5d3_banding_raws.zip)



Do you have an idea what could have caused the fixed-pattern banding?

Brainstorming:
- Bad iso_regs settings
  -> critical value for SaturateOffset (?), but this should change the tint
  -> variations of ADTG-Preamp, if the offsets change and I fix the first vaule to 2, it could possibly cause an underflow of another value

- Electromagnetic interferences from the lens
  -> would cause horizontal banding (?)
  -> no fixed pattern

- Mobile-phone interferences
  -> also horizontal (?), non-fixed-pattern noise

- Voltage fluctuations due to AF-motor power-consumption

- Aliens



In this case a normal dark-frame subtraction will not work, as the intensity of the FPN exceeds the amount of FPN I could get from a "normal" dark-frame.

As the approach of correcting banding using data from the optical-black bars was mentioned earlier in this thread (https://www.magiclantern.fm/forum/index.php?topic=10111.msg105075#msg105075) - is there a banding correction tool for CR2 files available yet?
If not, I will have a look at it. Just need to get into RAW de- and encoding.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on August 22, 2018, 07:47:22 AM
Is the FPN only in the shadows?  Or only most notable in the shadows?
One of the effects of reducing read noise, the fixed pattern noise becomes even more distinguishable (the percentage of FPN vs other noise sources increases).

http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html#patternnoise
QuoteThis gives an indication of how visually disruptive pattern noise can be -- even though the fixed pattern noise is only about 20% of the overall noise, it is quite apparent because our perception is adapted to picking out patterns, finding edges, etc.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on August 24, 2018, 07:45:18 PM
Here's a sample frame of the results I'm getting with my current settings, emphasis on highlights and maximum dynamic range.

Based on ISO 200.

It's working well in every scenario so far, from screaming sunlight to dark scenes.
(https://thumb.ibb.co/gGPvsp/mlvapp_screaming_sun_full_shadows.jpg) (https://ibb.co/gGPvsp)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 25, 2018, 07:55:30 PM
Started to write a generic register overriding interface. Goals:
- to be used for other features as well: FPS override, crop_rec (for now)
- allow any modules or core features to override any registers (ADTG, CMOS, ENGIO, DFE, whatever else we'll find in the future)
- allow more than one module to override the same registers without "fighting", e.g.:
  - if dual_iso changes CMOS[0], the ISO tweaks module should be able to override CMOS[7], for example [easy]
  - if dual_iso changes CMOS[0] and the ISO tweaks module decides to use some custom gains in CMOS[0], that should work as well
  - FPS override and crop_rec should work well together (for some definition of "well"... TBD)
- allow custom overriding logic
- do that fast (LiveView doesn't like slow overrides; searching in a big list for every single ADTG or ENGIO register is likely overkill)
- minimally invasive (install/uninstall the hooks only when actually used)
- compatible with old and new models (DIGIC 4 and 5 for now, keeping in mind DIGIC 2, 3, 6, 7 and 8 )

Initial draft, for review:


/* Register overrides */

/* Core APIs for overriding various registers:
* - ADTG
* - CMOS
* - DFE
* - ENGIO
* - ...
*/

/* Each register type has an "add" function, which can be used to set up certain overrides,
* and a "del" function, which "uninstalls" previous overrides (identified by what "add" returned).
*
* Optional features: check for expected values or call back some CBR.
*/

/* Each "add" function has the following parameters:
* - register "target" (optional, if there are more chips of the same type, see e.g. ADTG "destination")
* - register address (each register has some unique identifier)
* - register mask (which address bits should be ignored, e.g. to override more than one register)
* - new value (after overriding)
* - expected value (with a boolean flag to enable this check)
* - optional CBR for custom overriding logic (if not NULL, the "new value" is ignored)
* and returns a positive ID if successful, or negative error code if not.
*
* Each "del" function takes one parameter (the ID returned by "add")
* and returns 0 on success, or negative error code on failure.
*/

#ifndef _regs_h_
#define _regs_h_

typedef uint16_t (*adtg_override_func)(
    uint8_t dest,       /* ADTG chip "destination" (bit field, e.g. 6 will override both ADTG2 and ADTG4) */
    uint16_t reg,       /* ADTG register ID (0x800C, 0xFE etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint16_t (*cmos_override_func)(
    uint8_t reg,        /* CMOS register ID (0, 1, 2 etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint32_t (*engio_override_func)(
    uint32_t reg,       /* ENGIO register address (0xC0F06014 etc) */
    uint32_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

typedef uint32_t (*dfe_override_func)(
    uint16_t reg,       /* DFE register address (0x1D02 etc) */
    uint16_t old_val    /* value before overriding */
);                      /* returns: updated value (or old_val if nothing changes) */

int regs_ovr_adtg_add(uint8_t dest, uint16_t reg, uint16_t mask,
                      uint16_t new_val,
                      adtg_override_func cbr, void * opaque);

int regs_ovr_cmos_add(uint8_t reg, uint8_t mask,
                      uint16_t new_val, cmos_override_func cbr, void * opaque);

int regs_ovr_engio_add(uint32_t reg, uint32_t mask,
                       uint32_t new_val, engio_override_func cbr, void * opaque);

int regs_ovr_dfe_add(uint16_t reg, uint16_t mask,
                     uint16_t new_val, dfe_override_func cbr, void * opaque);

int regs_ovr_adtg_del(int id);
int regs_ovr_cmos_del(int id);
int regs_ovr_engio_del(int id);
int regs_ovr_dfe_del(int id);

/* TODO (if the above will work well enough):
* - TFT SIO registers?
* - HDMI chip registers?
* - Sound registers? (integrate with new-sound-system?)
* - Other peripheral registers?
*/

#endif /* _regs_h_ */

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on August 26, 2018, 01:43:20 PM
Nice!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: reddeercity on August 26, 2018, 11:56:51 PM
Have problems with ADTG_GUI in the "iso-research" branch , I can't modified/override
c0f06084
c0f06088

but if I use "digital poke" that enabled in "wasa57" "crop_rec_4k_5d2" branch
I can override them to increase the capture area a little bit (100 - 200 lines vertically)
but there's nothing in the ADTG_GUI , the only one's that seem to work are the cmos's
& some of the "adtg"[1] , [2] etc. ...

mainly I need those reg's to work in adtg_gui in combination with the cmos reg's .
I put the sensor I 1:1 mode etc. .... for 4k but I can't override 6084 & 6088 to increase
raw size .
@a1ex , is there hooks of stub that could be missing for those reg's ?
Where should I look ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on August 27, 2018, 07:25:39 AM
Hm, right, This is definitely working on DIGIC 5.

Figured it out (https://bitbucket.org/hudson/magic-lantern/commits/91e97004c7b087be9dd0852f1e43547f46034a52) - DIGIC 4 requires poking 0xC0F06000 after changing any other register from this group. See FPS_REGISTER_CONFIRM_CHANGES in fps-engio.c. This was not necessary on DIGIC 5.

The DIGIC poking code, being written when DIGIC 4 was the main focus, already had this workaround enabled.

Background:

These registers are only changed by Canon firmware when switching video modes (unlike CMOS registers, for example, which are poked once every LiveView frame). So, until recently, the only way to see the change to these registers was to get out of LiveView and back, or to manually trigger some video mode change (e.g. zoom in and back, flip to some other video mode and back).

A workaround for that was introduced in reply #967, commit 5314bde. That one mostly worked, except for two quirks: the one mentioned above, and another one that prevented ML from reading back the value of registers modified on the fly. So, on DIGIC 5, you could modify FPS timers and see the new frame rate on the screen, but the status bar was only updated after getting out of LiveView and back. Fixed that one, too.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: reddeercity on August 29, 2018, 02:41:45 AM
Thanks , now I can finally finish 4k/UHD for D4  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: reddeercity on August 29, 2018, 04:15:02 AM
Ok I can confirm the change that a1ex made are now working on 5d2/D4 for c0f06084 & c0f06088 in the "adtg_gui.c"
I posted a new build if any 5d2 user wishes to try it out/help with 4k  https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/
I also modified mlv_lite.mo & mlv_rec.mo with extended resolutions ,
I add :
3072
3840
4096
5632

the 5632 is the max res for the 5d2
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on August 29, 2018, 06:47:16 AM
Reddeercity,

Will your investigations and progress with the 5D2 work on the 7D too?  Could you possibly provide a build for that camera too?  After Sept. 17-th I will have access to my 7D and would love to test.

Thanks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: reddeercity on August 30, 2018, 06:01:18 AM
Quote from: IDA_ML on August 29, 2018, 06:47:16 AM
Reddeercity,
Will your investigations and progress with the 5D2 work on the 7D too? 
Yes ,  should be -- later when I have it close I'll write a tutorial , but in the mean time just follow my posts  :)

Quote from: IDA_ML on August 29, 2018, 06:47:16 AM

........ Could you possibly provide a build for that camera too .......
Sure , but I have to add 7d code for srm "raw_slurp" , you know the one , for 10-12bit raw video
That one of the reasons I'm getting so close to 4k I'm used my Super Bleeding Edge "SRM_Buffer"
It's the one I use for 10-12bit FHD on the 5D2 that works really well in 12bit 1856x1044 .
I did make some extra changes to the buffer I'm using a 32MB buffer
4096*4096
and as per my last quick test here (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205534#msg205534) , it seem ML need a 59 MB buffer
so the next test I'll increase the default raw buffer to

2048*16377 ---Not Good ER70---

which is ML is telling me that it needs 59MB , but I not too sure how that ends up to be 59MB -- Hex is not my strong point , but I get by .

If any one can point my to some math examples I would be very grateful  :D

Edit: ok that buffer produce a error 70 when mlv_lite raw video enabled so not a good buffer back to my 32MB 4096*4096
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on August 30, 2018, 03:00:48 PM
Thanks a lot, Reddeercity for your amazing progress.  I can't wait to test a new working build for the 7D.  This camera seems to be the only one with that dreamy filmic look and really deserves 4k-crop recording.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on September 27, 2018, 08:02:04 AM
Quote from: a1ex on August 01, 2017, 01:45:59 AM
Restoring the missing screenshots would be a good starting point.

Done.

Quote from: a1ex on August 01, 2017, 01:45:59 AMScript various test scenarios in Lua (e.g. cycle through all ISOs, take test pictures, change settings, create tables with results).

I've been avoiding LUA.  ::)  But, it looks like a few of your other requests can be knocked out reasonably quickly and easily with this one.


Quote from: a1ex on August 01, 2017, 01:45:59 AM
Find out how low you can push the ISO without getting artifacts in highlights. Different cameras (of the same model) might have different results.

It will probably take a while to re-read the thread and refresh my memory on all this stuff.

Some issues with raw_diag mentioned here, didn't look into them yet: http://www.magiclantern.fm/forum/index.php?topic=16516.msg186527#msg186527

Try to reproduce the above findings on different camera models. Some may need different registers.

Simply play with adtg_gui and document register behavior from your camera.

Figure out the registers required for the ISO tweaks in LiveView (some are the same, others are not).

Evaluate the repeatability of our measurements under various test scenes.

Propose a way to measure the sensor response curve without expensive lab equipment.

Try the pattern noise reduction algorithm from MLVFS or mlv_dump on steroids on the images taken with ISO tweaks (IIRC 5D3 has problems with this, 6D less so). Try to fine-tune the parameters.

Figure out how to reproduce the ETTR + Dual ISO crash on the patchmgr branch (doesn't happen on unified; happens with low probability on patchmgr and any other branches that use this backend).

Figure out why this dark frame (http://www.magiclantern.fm/forum/index.php?topic=18975.msg186759#msg186759) introduced additional pattern noise, instead of removing it.

Just some random ideas. Don't limit yourself to these :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on October 31, 2018, 07:22:48 PM
I want to mess around a little with analog gain registers, goal is not to get a clean image, goal is to get stupidity high iso value  :P
I'm curious if anyone knows more analog gain registers then these commonly known ?

analog gains
ADTG2 [8882]
ADTG2 [8884]
ADTG2 [8886]
ADTG2 [8888]

ADTG2 [FE]

analog preamps
ADTG2 [ 8 ]
ADTG2 [ 9 ]
ADTG2 [ a ]
ADTG2 [ b ]

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 04, 2018, 12:44:27 PM
Decided to play around with the analog gain registers thinking I could get a cleaner output. Here is one take on this.
First I lower analog gain registers severly causing it to act as 10bit?(maybe lower). What happens is highlights will take a hit and get clipped. However this move will clean shadows by a lot. I matched each file by modifying also cmos 0.
Maybe it could mean that when shooting 12 or 10 bit we would gain much better shadows if this was implemented? I didn´t mess with the preamps at all. How would this test benefit from those?

All files were analysed in Mlv App:
Shortened MLV files and coresponding PNG(brightened) here. Note that the modified mlv files could have been set to 3000 in white level but I skipped that parameter in this test. Only wanted to show shadows:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/ISO_gain_tests.zip

analog gains changed to(note there are eight of those registers on the 5D mark III)
ADTG2 [8882] 0x13b
ADTG2 [8884] 0x13b
ADTG2 [8886] 0x13b
ADTG2 [8888] 0x13b


Recommend to download the png files to really see the difference.

ISO 100
(https://i.postimg.cc/fTghxWx7/M04-1256iso100-frame-1-png-scaled.png)

ISO 100 CMOS[0] 0x222 gain to 0x13b
(https://i.postimg.cc/5y4Mm67g/M04-1257iso100gain-frame-1-png-scaled.png)

ISO 200
(https://i.postimg.cc/MKGSMfjK/M04-1259iso200-frame-1-png-scaled.png)

ISO 200 CMOS[0] 0x333 gain to 0x13b
(https://i.postimg.cc/sf4jZR6K/M04-1260iso200gain-frame-1-png-scaled.png)

ISO 400
(https://i.postimg.cc/rFhqQNLr/M04-1261iso400-frame-1-png-scaled.png)

ISO 400 CMOS[0] 0x443 gain to 0x13b
(https://i.postimg.cc/Rh4vbPFc/M04-1262iso400gain-frame-1-png-scaled.png)

ISO 800
(https://i.postimg.cc/52TbM9y9/M04-1300iso800-frame-1-png-scaled.png)

ISO 800 CMOS[0] 0x553 gain to 0x13b
(https://i.postimg.cc/d1mYd8BD/M04-1301iso800gain-frame-1-png-scaled.png)

ISO 1600
(https://i.postimg.cc/GmHdNTcS/M04-1302iso1600-frame-1-png-scaled.png)

ISO 1600 CMOS[0] 0xddd gain to 0x13b
(https://i.postimg.cc/QMKNB9wW/M04-1303iso1600gain-frame-1-png-scaled.png)

ISO 3200
(https://i.postimg.cc/fT5R1hYk/M04-1304iso3200-frame-1-png-scaled.png)

ISO 3200 CMOS[0] 0xfff gain to 0x13b
(https://i.postimg.cc/RC7FQtF7/M04-1305iso3200gain-frame-1-png-scaled.png)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on November 04, 2018, 01:01:12 PM
If this can be done to the top ISO in dual ISO, the results would be tremendous.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 04, 2018, 01:50:36 PM
Even a slighter change will have a major impact on iso 100. In this case highlights won´t suffer as much:

iso 100
(https://i.postimg.cc/HjDqZztQ/Screenshot-2018-11-04-at-13-46-39-png-800px.png)

iso 100 CMOS[0] 0x111 gain to 0x21a
(https://i.postimg.cc/rpVk2h2j/Screenshot-2018-11-04-at-13-46-46-png-800px.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 04, 2018, 02:24:08 PM
A few last frames:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/iso_100.zip

CMOS[0] is vital. Simply lowering the analog gain registers won´t do:

iso 100 original
(https://i.postimg.cc/m2zPqyQK/Screenshot-2018-11-04-at-14-16-01-png-800px.png)

iso 100 Lowering analog gain to 0x19a (white level to around 5000)
(https://i.postimg.cc/ZYD9VMyr/Screenshot-2018-11-04-at-14-16-25-png-800px.png)

iso 100 Lowering analog gain to 0x19a and CMOS[0) 0x222 (white level to around 5000)
(https://i.postimg.cc/sXTBFnHC/Screenshot-2018-11-04-at-14-17-08-png-800px.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 04, 2018, 06:21:40 PM
Here is a build for the 5D mark III which lets you set iso 100-3200 from within crop_rec menu:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec_4k_mlv_snd_isogain_1x3_5D3113_presets.zip

Commit/branch:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets
https://bitbucket.org/Dannephoto/magic-lantern/commits/9b8625caa3554d09003bd8a89dee7f43d66a634e

(https://i.postimg.cc/tgct2pn0/VRAM0-PPM-500px.png)

Have only tested this briefly but hopefully you get something like below. Compare the color noise and grain in both. Think of the crop_rec mv1080p iso settings as iso for mv1080p. It will override whatever iso you have set in camera. Liveview should be matched to original iso and analog gain registers are lowered to around 12bit. Might wanna change white level to around 5000 in post, who nows, needs more testing.

iso 100
(https://i.postimg.cc/x8DGV5Pt/Screenshot-2018-11-04-at-17-47-21-kopia.jpg)

iso 100
(https://i.postimg.cc/j2HHxfgK/Screenshot-2018-11-04-at-17-47-29-kopia.jpg)

iso 100
(https://i.postimg.cc/NGm7khgV/Screenshot-2018-11-04-at-18-10-35-kopia.jpg)

iso 100
(https://i.postimg.cc/gJ6y7Q3x/Screenshot-2018-11-04-at-18-10-42-kopia.jpg)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 05, 2018, 12:03:49 AM
Interesting, the noise looks quitte different in your examples  :o
Will check if this stuff also has the same effect on 6d
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ilia3101 on November 05, 2018, 12:20:01 AM
Wow a lot has happened since I last looked at this thread

@Danne looks like massive improvements to dynamic range! How 5D3 specific is the stuff being done here? I'm really struggling with the huge amount of noise my 5D2 has
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: squig on November 05, 2018, 01:31:08 AM
I'm seeing really severe highlight clipping with mv1080p ISO 100 and ISO 3200.

ISO 100
(https://i.imgur.com/SSzKjDm.png)

mv1080p ISO 100 (white level 5000)
(https://i.imgur.com/lWhMeYo.png)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 05, 2018, 08:20:53 AM
Quote from: Danne on November 04, 2018, 12:44:27 PM
First I lower analog gain registers severly causing it to act as 10bit?(maybe lower). What happens is highlights will take a hit and get clipped. However this move will clean shadows by a lot. I matched each file by modifying also cmos 0.

Lowering analog gain registers will improve the DR... until the clipping point starts to drop. This is the sweet spot for ADTG gains. Without any other tweaks, when starting from ISO 100 in photo mode on my 5D3, this method improves the DR by about 0.37 EV, reducing the ISO to 77 (i.e. you should scale the original ADTG gains by 77%). On other cameras (possibly other 5D3's too), the numbers will be different.

Further reducing the ADTG gain is not going to improve DR any more; it will start to decrease the clipping point instead (and that happens in way that tricks the white level autodetection routine from raw_diag, so take the measurements with a grain of salt). Once white level starts to drop, DR is going to remain roughly constant, then it will start to get lower and lower (causes: analog noise and quantization error).

Now, if you set the ADTG gains to some very low value, but you keep the old (i.e. close to 15000) white level, this will result in a dark DNG. You might think you've just got a lower ISO, so you may be tempted to increase the exposure time and compare the shadows. Yes, this increase in exposure time will give *much* cleaner shadows. Once you look at the highlights, you will realize it's a false alarm.

When comparing noise at different flavors of ISO 100 (or any other ISO), make sure you capture the same amount of light (i.e. same shutter speed, aperture and test scene). This is important, otherwise the difference in noise will be given by the different amounts of captured light (i.e. the test results will be... well... bogus).

Then, render the two images at the same brightness on the screen, but you already did that.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 05, 2018, 10:35:23 AM
Quote from: a1ex on November 05, 2018, 08:20:53 AM
Lowering analog gain registers will improve the DR... until the clipping point starts to drop. This is the sweet spot for ADTG gains. Without any other tweaks, when starting from ISO 100 in photo mode on my 5D3, this method improves the DR by about 0.37 EV, reducing the ISO to 77 (i.e. you should scale the original ADTG gains by 77%). On other cameras (possibly other 5D3's too), the numbers will be different.
Seems not that much but might do a difference in shadows and color noise. My lowering of the analog gains are in other words, over the top. How to correctly calculate scaling to this?" (i.e. you should scale the original ADTG gains by 77%)"?

Question. Recording in situations without cruicial highlights(dark lit scenes for instance), would benefit from the harsh lower gain values in my examples rather than scaling down to 77%?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 05, 2018, 10:38:59 AM
Like this: old_value * 77 / 100.

To check whether it's too much, use something strongly overexposed, and make sure it ends up completely white (i.e. not slightly pinkish, or with noise, or stripes, or whatever). If OK, try a lower value. If you get artifacts, increase the value.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 05, 2018, 10:49:35 AM
Quote from: a1ex on November 05, 2018, 10:38:59 AM
Like this: old_value * 77 / 100.
of course, thanks  8)

Quote from: a1ex on November 05, 2018, 10:38:59 AM
To check whether it's too much, use something strongly overexposed, and make sure it ends up completely white (i.e. not slightly pinkish, or with noise, or stripes, or whatever). If OK, try a lower value. If you get artifacts, increase the value.
How about lowering gains to match white level of 12bit and clip accordingly for the benefit of very clean shadows or should these changes stick with 14bit white level only?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 05, 2018, 11:01:11 AM
If you also want 12-bit lossless recording, then you can further reduce the gains by 2 stops. Total gain for 12-bit would be 2^-2 * 77% = 19.25%.

Though, for reducing bit depth, I'd prefer to look once again at the digital gains, and leave the analog side for ISO optimization.


Quote from: Danne on November 05, 2018, 10:35:23 AM
Question. Recording in situations without cruicial highlights(dark lit scenes for instance), would benefit from the harsh lower gain values in my examples rather than scaling down to 77%?

In this case, I'd recommend the following, in this order:
- increase amount of captured light (whether from shutter speed, aperture or scene light)
- increase gains from CMOS amplifiers, where electronic noise is minimal
- increase gains from other analog amplifiers (todo: I need to revisit the noise performance evaluation)
- increase digital gains if that doesn't create gaps in the raw histogram (also need to double-check this)

Reducing any kind of gain will make the shadows *worse* (it will remove information from there). If that reduction comes with additional highlights captured, it may be worth the effort (e.g. for increased DR), otherwise it does not bring any benefits for image quality.

Reducing to 12-bit levels results in better lossless compression, but the image quality suffers a little (possibly just a negligible amount). Overall, it might be a worthy tradeoff.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 05, 2018, 02:17:27 PM
Quote from: a1ex on November 05, 2018, 11:01:11 AM
In this case, I'd recommend the following, in this order:
- increase amount of captured light (whether from shutter speed, aperture or scene light)
- increase gains from CMOS amplifiers, where electronic noise is minimal
- increase gains from other analog amplifiers (todo: I need to revisit the noise performance evaluation)
- increase digital gains if that doesn't create gaps in the raw histogram (also need to double-check this)

@Alex, I'm interested in step 2 and 3 and curious if you know some more analog amplifiers I can modify ?

What I have
ISO setting:
                              CMOS[3]  For 6d that is
Analog amplifiers:
                              ADTG2 [8882]
                              ADTG2 [8884]
                              ADTG2 [8886]
                              ADTG2 [8888]
Analog preamps:
                              ADTG2 [ 8 ]
                              ADTG2 [ 9 ]
                              ADTG2 [ a ]
                              ADTG2 [ b ]

And found register 
                              ADTG2 [FE], which also seems to be some sort of gain, not sure if it's analog or digital.

Did I miss any other known amplifiers ?


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 05, 2018, 03:35:11 PM
My old notes (to be taken with a grain of salt):


* There are 4 classes of adjustments available. They are optional (one camera can use only one of them,
* or all of them, or anything in-between) and they are applied in this order:
*
* - ADTG preamps: fixed, no adjustable parameters, no menu options
*      - ADTG 0xFE         : only applied at Canon ISO >= 200; lowering it by one notch is too aggressive for ISO 100
*      - ADTG 8/9/A/B      : we lower it a little at Canon ISO 100, and all the way down at ISO >= 200
*
* - Saturate Offset & related:
*      - Saturate Offset   : shifts raw data by constant value, alters black level but keeps white level constant
*                            => can capture more highlights
*      - B/W offset        : optional, shifts raw data by constant value, alters both black level and white levels
*                            => digital offset (can be used to increase the useful range for Saturate Offset)
*      - Digital gain      : we simply fix this one at the value believed to be 1.0
*                            note: at large apertures like f1.4, Canon increases this gain, which clips useful highlights;
*                            we cancel this behavior
*
* - ADTG gain: the first adjustment discovered
*      - analog gain in fine steps (default value is usually around 1000-1500)
*      - this does not alter black and white levels, but lowering it brings more highlight detail
*      - at least one camera (500D) doesn't have it (we couldn't find it)
*      - ADTG gain and Saturate Offset behave somewhat like a waterbed (todo: explain in detail)
*      - if you can adjust ADTG gain, Saturate Offset, Digital Gain, and - highly recommended - B/W offset,
*        you get full control over black and white levels, without impact on the recorded dynamic range
*
* - CMOS trick:
*      - I don't know how this one works, but it seems to reduce the noise at least in the OB area
*      - this does not change the equivalent ISO
*      - to be double-checked with the SNR curve
*
* Movie mode support:
*      - at least 3 different modes need to be analyzed: 1080p, 720p, 5x crop
*      - ADTG gain OK, tiny effect
*      - ADTG preamps OK, noticeable effect
*      - Saturate Offset and B/W Offset not working
*      - Digital gain is not applied in raw video anyway (not needed)
*      - CMOS trick still investigated


Some more useful notes in iso_regs.c:

https://bitbucket.org/hudson/magic-lantern/src/iso-research/modules/iso_regs/iso_regs.c?fileviewer=file-view-default#iso_regs.c-345

Quote from: theBilalFakhouri on July 25, 2018, 10:07:51 AM
Found some ADTG gain registers in 700D:

Negative gain     Original value     Overridden
ADTG2[21] =        0x4626              0x1036
ADTG2[22] =        0x4626              0x1036
ADTG2[23] =        0x4626              0x1036
ADTG2[24] =        0x4626              0x1036

Could not confirm on 700D - these values give a mostly black image, with strong vertical lines.

IIRC these didn't work on 700D either:
https://www.magiclantern.fm/forum/index.php?topic=10111.msg204415#msg204415
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: IDA_ML on November 05, 2018, 04:21:10 PM
Quote from: a1ex on November 05, 2018, 11:01:11 AM
Reducing to 12-bit levels results in better lossless compression, but the image quality suffers a little (possibly just a negligible amount). Overall, it might be a worthy tradeoff.

Absolutely, A1ex!  This is a very worthy tradeoff.  In my own experience, going from 14 to 12 bit lossless results in a negligible quality degradation but if you could achieve better lossless compression at 12 bits, resulting in longer recording times of contrasty scenes, that would be very welcome.  I am sick and tired of early stops when I direct the camera towards a brighter and more contrasty scene during recording. 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 05, 2018, 09:38:16 PM
ADTG2[21] has interesting effects, but definately not gain (I can get horizontal type of dual exposure)
ADTG2[22, 23, 24] don't give any visible effects on 6d  ???

That leaves me with the current known gains/preamps.
Made a preset which I can load in crop_rec module.
According to raw histogram I get 6 extra stops by maxing out analog gain and preamps.
So with highest 'native' iso setting of iso 6400 on 6d, I can get iso 6400 + 6 stops with the preset,
giving an iso setting of 409600.
Only problem for now...is getting a usable picture out of this noisy mess  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 10:33:45 AM
Been sweating the registers a few times trying to get some decent results/understaning about what to do here.

Firstly:
QuoteWhen comparing noise at different flavors of ISO 100 (or any other ISO), make sure you capture the same amount of light (i.e. same shutter speed, aperture and test scene). This is important, otherwise the difference in noise will be given by the different amounts of captured light (i.e. the test results will be... well... bogus).
All isos compared were at same settings. i.e iso 100 to iso 100 with lowered analog gains compensated with pushed CMOS[0] value to match brightness by eye in liveview. Also note that if I don´t push CMOS[0] the ouput will still look shitty but only now it´s dark and shitty. Difference could occur if comparing let´s say iso 100 to iso 400 but that is not what we are looking at right now.

Two examples at exact same settings below:
Iso 400(untouched register)
(https://i.postimg.cc/Dw0WdNwS/Screenshot-2018-11-06-at-20-54-35-png-scaled.png)

Iso 400(reduced analog gain compensated CMOS[0] to match brightness iso 400)
(https://i.postimg.cc/9FJR92C3/Screenshot-2018-11-06-at-20-54-43-png-scaled.png)

My crude code:
https://bitbucket.org/Dannephoto/magic-lantern/raw/49f23e49f96129f0777d10e65fdf82942ef48522/modules/crop_rec/crop_rec.c
     if (is_5D3)
         {
    case CROP_PRESET_mv1080p_iso100:
    case CROP_PRESET_mv1080p_iso200:
    case CROP_PRESET_mv1080p_iso400:
    case CROP_PRESET_mv1080p_iso800:
    case CROP_PRESET_mv1080p_iso1600:
    case CROP_PRESET_mv1080p_iso3200:
adtg_new[13] = (struct adtg_new) {6, 0x8882, 250};
                adtg_new[14] = (struct adtg_new) {6, 0x8884, 250};
                adtg_new[15] = (struct adtg_new) {6, 0x8886, 250};
                adtg_new[16] = (struct adtg_new) {6, 0x8888, 250};

adtg_new[17] = (struct adtg_new) {6, 0x8882, 250};
                adtg_new[18] = (struct adtg_new) {6, 0x8884, 250};
                adtg_new[19] = (struct adtg_new) {6, 0x8886, 250};
                adtg_new[20] = (struct adtg_new) {6, 0x8888, 250};
                break;
         }



            /* clean iso 100 */

            case CROP_PRESET_mv1080p_iso100:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0xd4a;
                break;

            /* clean iso 200 */

            case CROP_PRESET_mv1080p_iso200:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0xd6d;
                break;

         /* clean iso 400 */

            case CROP_PRESET_mv1080p_iso400:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0x191;
                break;

         /* clean iso 800 */

            case CROP_PRESET_mv1080p_iso800:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0xadb;
                break;

         /* clean iso 1600 */

            case CROP_PRESET_mv1080p_iso1600:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0xddd;
                break;

            /* clean iso 3200 */

            case CROP_PRESET_mv1080p_iso3200:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0xfff;
                break;


QuoteFurther reducing the ADTG gain is not going to improve DR any more; it will start to decrease the clipping point instead (and that happens in way that tricks the white level autodetection routine from raw_diag, so take the measurements with a grain of salt). Once white level starts to drop, DR is going to remain roughly constant, then it will start to get lower and lower (causes: analog noise and quantization error).
With this statement shadows should not look that clean in the images above. I understand that white level is cut off but I can only see benefits in darks and shadows.

Quote
Now, if you set the ADTG gains to some very low value, but you keep the old (i.e. close to 15000) white level, this will result in a dark DNG. You might think you've just got a lower ISO, so you may be tempted to increase the exposure time and compare the shadows. Yes, this increase in exposure time will give *much* cleaner shadows. Once you look at the highlights, you will realize it's a false alarm.
In a personal test I compared files with only lowered analog gain and with analog gain compensated(pushed CMOS[0] value to match original iso and it made all the difference with CMOS[0] pushed compared with no compensation.
Didn´t test any longer shutter vs shorter shutter time etc.

Did a lot of checking the other registers around iso, pre gain, digital gain  etc but could not make much out of these. Nothing compared to reducing analog gain as a starting point.


Also have a noob qeustion. Do these match below?
14bit analog gain:
iso 100 41a=1050
iso 200 435=1077
iso 400 437=1079
iso 800 434=1076
iso 1600 445=1093
is0 3200 469=1129

So 12 bit for iso hundred would be:
1050/14x12=900=0x384


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 07, 2018, 10:42:59 AM
Quote from: Danne on November 07, 2018, 10:33:45 AM
Iso 400(reduced analog gain compensated CMOS[0] to match brightness iso 400)

Sounds like you've got a higher ISO (which usually gives cleaner shadows as long as other exposure parameters remain constant). Can you confirm by looking at the clipping point?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 10:47:28 AM
Yes, higher iso:

            case CROP_PRESET_mv1080p_iso400:
                /* start/stop scanning line, very large increments */
                cmos_new[0] = 0x191;
                break;


So a question:
In practice this would mean that if I set iso to 0x191(iso 401 ?) in camera with the same shutter etc as the test above the output would match the one in the test when lowering back exposure in post? Hm, seems I set it to iso 400 in CMOS[0] already. I´m getting mindf...d here  8)

Will see if I still have the MLV test file to check clipping point.



EDIT:
Couldn´t find the former files but here are two other files:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/short_M07-1147.MLV
https://bitbucket.org/Dannephoto/magic-lantern/downloads/short_M07-1146.MLV


Both shot at iso 400 at the same settings:
Before
(https://i.postimg.cc/kGBqvqMG/Screenshot-2018-11-07-at-11-00-02-png-800px.png)

After
(https://i.postimg.cc/8k01PJw6/Screenshot-2018-11-07-at-10-59-54-png-800px.png)

Now off for work...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 07, 2018, 01:56:21 PM
Quote from: Danne on November 07, 2018, 10:33:45 AM

Also have a noob qeustion. Do these match below?
14bit analog gain:
iso 100 41a=1050
iso 200 435=1077
iso 400 437=1079
iso 800 434=1076
iso 1600 445=1093
is0 3200 469=1129


So 12 bit for iso hundred would be:
1050/14x12=900=0x384

I don't think it works that way, an extra bit doubles the number of unique numbers.
8 bit -> 256 unique number possibilities
9 bit -> 512 unique number possibilities
10 bit -> 1024 unique number possibilities
etc..
14 bit -> 16384 unique number possibilities

The bits in raw image files can almost literally be seen as stops.
So going from 14 bit to 12 bit, you lose 2 stops, going from 16384 to 4096 unique gradations.
As far as I know, the analog gains work that way too.
6d has maximum native CMOS iso 6400, when using iso 12800 I can see a doubling of the analog gain registers from 1193(decimal) to 2386(decimal)
Going to iso 25600 is another doubling, from 2386(decimal) to 4772(decimal).

So if with 14 bits your analog gain value is around 1000 (as shown in your table)
You need to divide the value in half, for two times, since you want to lose two bits.
So going from 14 bit to 12 bit you need analog gain values of around 250.

If in doubt, you can always check the amount of stops you have in raw histogram, If you're doing it right, you can never have highlight values in the last two bars of the raw histogram
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 07, 2018, 02:05:57 PM
Quote from: Danne on November 07, 2018, 10:47:28 AM
Both shot at iso 400 at the same settings:
Before
(https://i.postimg.cc/kGBqvqMG/Screenshot-2018-11-07-at-11-00-02-png-800px.png)

After
(https://i.postimg.cc/8k01PJw6/Screenshot-2018-11-07-at-10-59-54-png-800px.png)

Now off for work...

The shadows look really good in your example.
But there probably is a small loss in the highlights, maybe try to create an example with both (clipped ?) shadows and highlights in one scene.
Shoot it at the same settings and see how much highlight information you can get back and how the shadows look.

For scenes where the dynamic range fits within these new settings, this certainly gives much cleaner shadows.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 02:21:51 PM
Thanks for feedback Levas, and a1ex. Yes, clipping in highlights are noticable. Will create files with more dynamic range when time.
Now the idea comes to mind if it would be possible to have variable bitrate depending on lighting.  :P
Or even if it could be selectable only by pointing camera on a scene right before recording. Maybe a lua script could help here...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 04:13:27 PM
Some more test files here:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/higlight_lowlight.zip

Clipped histogram(12bit)
(https://i.postimg.cc/MTxDZ9H5/VRAM1-PPM-scaled.png)

14bit
(https://i.postimg.cc/dVb6psKz/VRAM2-PPM-scaled.png)

A few examples:
12bit lossless through reduced gain
(https://i.postimg.cc/L55bkmfh/Screenshot-2018-11-07-at-15-55-17-png-scaled.png)

12bit lossless
(https://i.postimg.cc/L5W0NGBD/Screenshot-2018-11-07-at-15-55-24-png-scaled.png)

12bit lossless through reduced gain pushed
(https://i.postimg.cc/fWqFz0nd/Screenshot-2018-11-07-at-15-56-09-png-scaled.png)

12bit lossless pushed
(https://i.postimg.cc/rFvPWXpF/Screenshot-2018-11-07-at-15-56-15-png-scaled.png)




Last two files below have both been exposed differently. One was exposed to the right, the other exposed to the left to obtain maximum dynamic range in post. WHat you see below is the result after post processing in Mlv App.

12bit reduced gain lossless "ettl" exposed to the left, then post fixed for highlights and shadows
(https://i.postimg.cc/jdW1LdtF/Screenshot-2018-11-07-at-15-59-30-png-scaled.png)

12bit lossless "ettr" then post fixed for highlights and shadows(note that aperture is not the same for last two examples)
(https://i.postimg.cc/XJVhJfYf/Screenshot-2018-11-07-at-15-59-37-png-scaled.png)

Conclusion is that with lowered analog gain and pushed CMOS[0] register we can obtain more or less the same dynamic range as with 12bit lossless unaltered register files. The benefit kicks in when filming lowlight footage. By altering registers we can skip "exposing to the right" and even "expose to the left" keeping good, clean shadows.
If doing ettr and filming regularly lit scenes reglar 12bit lossless works just as good.

Now this is iso 400. The effect is probably even stronger with iso 100.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 09:29:03 PM
Hm, noticed something. Seems related to how registers are written in code:
      if (is_5D3)
         {
    case CROP_PRESET_mv1080p_iso100:
    case CROP_PRESET_mv1080p_iso200:
    case CROP_PRESET_mv1080p_iso400:
    case CROP_PRESET_mv1080p_iso800:
    case CROP_PRESET_mv1080p_iso1600:
    case CROP_PRESET_mv1080p_iso3200:
adtg_new[13] = (struct adtg_new) {6, 0x8882, 250};
                adtg_new[14] = (struct adtg_new) {6, 0x8884, 250};
                adtg_new[15] = (struct adtg_new) {6, 0x8886, 250};
                adtg_new[16] = (struct adtg_new) {6, 0x8888, 250};

adtg_new[17] = (struct adtg_new) {6, 0x8882, 250};
                adtg_new[18] = (struct adtg_new) {6, 0x8884, 250};
                adtg_new[19] = (struct adtg_new) {6, 0x8886, 250};
                adtg_new[20] = (struct adtg_new) {6, 0x8888, 250};
                break;
         }


Changed to:
      if (is_5D3)
         {
    case CROP_PRESET_mv1080p_iso100:
    case CROP_PRESET_mv1080p_iso200:
    case CROP_PRESET_mv1080p_iso400:
    case CROP_PRESET_mv1080p_iso800:
    case CROP_PRESET_mv1080p_iso1600:
    case CROP_PRESET_mv1080p_iso3200:
adtg_new[13] = (struct adtg_new) {6, 0x8882, 0x250};
                adtg_new[14] = (struct adtg_new) {6, 0x8884, 0x250};
                adtg_new[15] = (struct adtg_new) {6, 0x8886, 0x250};
                adtg_new[16] = (struct adtg_new) {6, 0x8888, 0x250};

adtg_new[17] = (struct adtg_new) {6, 0x8882, 0x250};
                adtg_new[18] = (struct adtg_new) {6, 0x8884, 0x250};
                adtg_new[19] = (struct adtg_new) {6, 0x8886, 0x250};
                adtg_new[20] = (struct adtg_new) {6, 0x8888, 0x250};
                break;
         }

Will yield a very different clipping result. The former tip from Levas(I think) might have been causing a lot of strangeness lately. Not sure wht is what here atm. Phew. Need a break...

No clipping in highlights like before:
iso 100 12bit, no fixing in registers
(https://i.postimg.cc/brDKDLjs/Screenshot-2018-11-07-at-21-25-19-png-500px.png)

iso 100 12bit
(https://i.postimg.cc/k4h0gzz0/Screenshot-2018-11-07-at-21-25-26-png-500px.png)



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on November 07, 2018, 10:08:10 PM
Wow!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on November 07, 2018, 10:16:05 PM
I don't know a lot about programming in c language.
But I think the first piece of code is right, the about 250 values I said in an earlier post are in decimal value. I'm not sure, but how it is written in your source I would say it works and it translates the value from decimal to hexadecimal input in the register.

In the second piece of code, you're using 0x prefix, which as far as I know means, here comes a hexadecimal value.
So in your second piece of code, 0x250 becomes a decimal value of 592. Which is about half the standar value of around 1000 so more like 13 bit or one stop lower then standard values.
True value of 250 in hexadecimal would be 0xFA ( there are handy calculator apps for that for your phone)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 07, 2018, 10:27:06 PM
hehe, ok, thanks Levas!
Checking the output 13bit doesn´t seem that bad. Keeping the highlights a lot better and still cleans the shadows by a lot...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: DeafEyeJedi on November 09, 2018, 08:48:17 PM
Daaaaamn @Danne... Once again you've literally brought me back from being a bit unmitigated. Life can sometimes bite us harder than we've ever imagined!

My apologies to everyone here on board for trying to get my $#!t together in the past year or so while being under the radar. No fair, I know.

I've been furtively following along in the shadows occasionally from time to time. Nevertheless this whole hypothesis in regards to this so called Sensor Upgrade is fucking REMARKABLE!

Quote from: a1ex on November 09, 2018, 05:33:45 PM
QED, initial hypothesis confirmed.
I had to look this up too @dfort -- haha and man this whole Sensor Upgrade feels like chasing a fools gold that actually existed. Much love to you guys and I'll be back!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 09, 2018, 08:55:17 PM
Well if this got you back @Deafeyejedi I´m a happy camper :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on November 11, 2018, 03:30:59 PM
Some minor updates:

raw_diag:
- real-time indicator for dynamic range & clipped area in LiveView (also enable raw video to see it); suggested by 50mm200s (https://www.magiclantern.fm/forum/index.php?topic=10111.msg204803#msg204803)
  (it's not very accurate, but can be handy when tweaking registers)
- testing yet another white level autodetection (https://bitbucket.org/hudson/magic-lantern/commits/0b362c5a49d48) heuristic, based on ideas from this thread (https://www.magiclantern.fm/forum/index.php?topic=20882.0)

adtg_gui:
- ENGIO register support for 700D, EOSM, 650D and 600D (first two confirmed by Bilal & Dfort; last two not tested)
- nicer filters for image capture/preview size registers

Precompiled modules available on the download page (https://builds.magiclantern.fm/modules.html#iso-research). They can be used on top of crop_rec_4k and derived builds, including the ones from Danne.

Thanks Danne for bumping the thread properly :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 22, 2018, 05:25:02 PM
Started off now working with raw_diag. What a great tool to check dr when trying out registers. Speaking registers I wanted to apply and switch some registers in crop_rec.c but the ones handling digital gain and shadow gain registers won´t apply. Any particular reason for this? Used elsewhere already? Trying to switch these registers but no luck:

"Digital gain for ISO (SHAD_GAIN)"}
    {0xC0F0,   0x8030, 0, "Digital gain for ISO (SHAD_GAIN)"},

    {0xC0F3,   0x7ae4, 0, "ISO digital gain (5D3 photo mode)"},
    {0xC0F3,   0x7af0, 0, "ISO digital gain (5D3 photo mode)"},
    {0xC0F3,   0x7afc, 0, "ISO digital gain (5D3 photo mode)"},
    {0xC0F3,   0x7b08, 0, "ISO digital gain (5D3 photo mode)"},


In a switch reg in crop:rec.c

    switch (reg)
    {
     
      case 0xC0F08030:
            return 0x1f6f;

        case 0xC0F37ae4:
            return 0x300;

        case 0xC0F37af0:
            return 0x300;

        case 0xC0F37afc:
            return 0x300;

    }

    return 0;


Nothing is added here, hm...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dfort on December 17, 2018, 07:40:57 AM
Found the pieces for the 7D:

modules/adtg_gui.c
    else if (is_camera("7D", "2.0.3"))
    {
        ADTG_WRITE_FUNC = 0xFF2C0944; //"[REG] @@@@@@@@@@@@ Start ADTG[CS:%lx]"
        CMOS_WRITE_FUNC = 0xFF2C0B3C; //"[REG] ############ Start CMOS"
        ENGIO_WRITE_FUNC = 0xFF1F6B20;  // from stubs
        ENG_DRV_OUT_FUNC = 0xFF1F675C;
        SEND_DATA_TO_DFE_FUNC = 0xFF32B800; //"[REG] DFE:[%#lx]"
    }


Unfortunately it doesn't work. Looks like we're bumping into this issue:

Quote from: dmilligan on April 10, 2015, 01:04:32 PM
The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.

Any way around this?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: afre on December 20, 2018, 08:40:56 AM
Quote from: Ron100 on February 12, 2018, 10:02:16 AM
Hi, help! receive error when enable modules adtg_gui.mo and raw_diag.mo on the camera 6d(1.1.6)

adtg_gui.mo
tcc: error: undefined symbol "unpatch_memory"
tcc: error: undefined symbol "patch_hook_function"
[E] failer to link modules

raw_diag.mo
Wrong version (v7.0, expected v6.0)

I have the same issue but for 550D (magiclantern-Nightly.2018Jul03.550D109.zip). A comment says that dfort has it bundled in his builds but I see magiclantern-Nightly.2018Jul04.550D110.zip and I am currently on firmware 1.0.9. Not all Canon websites contain firmware 1.1.0. Is it a regional thing, or is it okay to upgrade in order to use dfort's build? Please advise. Thanks!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Walter Schulz on December 20, 2018, 09:22:37 AM
Have you run sanity check on this issue? "Why and what for do I need those modules to run on my cam?".

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on December 20, 2018, 11:52:52 AM
Quote from: dfort on December 17, 2018, 07:40:57 AM
Unfortunately it doesn't work. Looks like we're bumping into this issue:
> The whole dual processor thing I think. All the interesting stuff happens on the master. ML can only debug on the slave.

Any way around this?

for experiments, i once made 7D_FIR target, which creates ML for 7D_MASTER and 7D and made a FIR for them.
so both digics could run ML code.
but ML on master was quite crude. no modules, just simple hooking stuff there.

not sure if that still works or died during a refactoring.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dfort on December 20, 2018, 06:57:16 PM
Quote from: afre on December 20, 2018, 08:40:56 AM
magiclantern-Nightly.2018Jul04.550D110.zip ... is it okay to upgrade in order to use dfort's build? Please advise. Thanks!

That firmware update was done without the camera (https://www.magiclantern.fm/forum/index.php?topic=19417.msg192667#msg192667). According to my firmware update notes (https://www.magiclantern.fm/forum/index.php?topic=19417.0) there were some issues with it though it wasn't thoroughly tested.

As far as those errors--are you using the modules from the modules download page (https://builds.magiclantern.fm/modules.html)? If so, it needs to be run on one of the experimental builds. The error, "Wrong version (v7.0, expected v6.0)" makes me suspect you are running it on a unified branch (Nightly) build.

Quote from: g3gg0 on December 20, 2018, 11:52:52 AM
for experiments, i once made 7D_FIR target, which creates ML for 7D_MASTER and 7D and made a FIR for them.

Intriguing. I built the 7D_MASTER.203 platform and tried running it but all I get is the screen saying that my camera doesn't look like a 7D_MASTER. Is there a certain configuration option that's needed to compile it? I thought creating FIR files were limited only to a few developers who hold the secret key.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: afre on December 20, 2018, 09:45:44 PM
Quote from: dfort on December 20, 2018, 06:57:16 PM
As far as those errors--are you using the modules from the modules download page (https://builds.magiclantern.fm/modules.html)? If so, it needs to be run on one of the experimental builds.

My mistake. Thanks! :)

Quote from: Walter Schulz on December 20, 2018, 09:22:37 AM
Have you run sanity check on this issue? "Why and what for do I need those modules to run on my cam?".

Curiosity more than anything else.




Honestly, I don't know what to do with the adtg_gui.mo module. As mentioned, I have a 550D (1.0.9). When I turn on "ENGIO Registers", I see there are 12 unidentified registers. Really, there are 6 because in "Advanced" every two rows are equal in key and value. I also see a status that says "12 uniq / [growing number]". Lastly, when the camera sleeps, "ADTG Registers" turns off.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: g3gg0 on December 26, 2018, 09:29:29 PM
Quote from: dfort on December 20, 2018, 06:57:16 PM
Intriguing. I built the 7D_MASTER.203 platform and tried running it but all I get is the screen saying that my camera doesn't look like a 7D_MASTER. Is there a certain configuration option that's needed to compile it? I thought creating FIR files were limited only to a few developers who hold the secret key.

yeah the FIR needs signing. had a web frontend, but not sure if it is still up.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dfort on December 27, 2018, 01:15:47 AM
Quote from: g3gg0 on December 26, 2018, 09:29:29 PM
yeah the FIR needs signing. had a web frontend, but not sure if it is still up.

For the information, thank you. What to do with it, I know not.

I'm also at a loss how to dump the 7D Master firmware then what to do with it once I get it.

In any case, I'll just keep this knowledge vacuum packed in my brain for now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on March 23, 2019, 02:39:41 PM
Two minor bug fixes for adtg_gui (actually in the menu backend):
- registers can now be edited while recording H.264
- no more crash when editing large values (such as 0x80000000)

HTH.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 12:42:40 PM
Quote from: 70MM13 on April 14, 2018, 04:18:55 PM
here's an excellent example of the benefits of amplifier gain optimisation:
...
this REALLY needs to become mainstream.now.please!

@70MM13 With results like these I totally agree!  8) But perhaps uptake is slow because people (like me  ;D ) are seriously baffled as to how to even begin testing this – and I've read through the entire thread **EDIT And I'm currently re-reading it** – and consider myself a seasoned ML user!

Could a user of these modules maybe make a tutorial post/video on: which modules to install, how to safely tweak the registers, (lol) etc. ? It would be tremendously helpful and might encourage more people to test, feedback, and develop these proof of concepts into a mainstream module.

Rock this!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 02, 2019, 03:13:45 PM
Started today to take a look at this on 600D, for now I'm looking for differeces in changing ISO and got this so far:

Changed Registers ISO100 ISO200 ISO400 ISO800 ISO1600 ISO3200 ISO6400
CMOS[0]        0       24    48    6c   90 B4 B4
CMOS[1]        423 423 423 423 423 443 443
ADTG2[8830] 1 1 1 1 1 3 3
ADTG2[8882] 417 413 427 41A 42E 453 8A6
ADTG2[8884] 417 413 428 41C 42F 454 8A8
ADTG2[8886] 41e 419 42a 41D 42D 44D 89A
ADTG2[8888] 41c 417 429 41B 42B 44A 894

Other notes:
CMOS[0] confirmed chaging ISO (will resut in horizontal stripes if set wrong)
CMOS[1] is changing exp, but can't understand what is, is set at 423 normally and 443 at highest ISO, if I set to 123 got a dark image like 4 less stop, at 223 or more than 200 i can't see any difference
CMOS[2] is set to 0x18 if I change it from 0x00 to 0xBF can't see any difference, then after 0xC0 I got totally black pics (with noise)
CMOS[5] (is at 140 by canon) set under 40 gives black image, but from 40 going up I have some funky color in shadows going green and lights going pinkish, so seems not related to vertical offset on 600D
CMOS[6] shifts images up if set to more than 800 (810 is already half pic ok, bottom half black) less than 800 is totally black image , so this could be vertical offset on 600D?
CMOS[7] not sure it starts at 1 and at 21 shows only a little lines of pixels captured on left side of image, other values gives black pic if it don't finish in 1
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 02, 2019, 03:48:55 PM
Quote from: timbytheriver on April 02, 2019, 12:42:40 PM
@70MM13 With results like these I totally agree!  8) But perhaps uptake is slow because people (like me  ;D ) are seriously baffled as to how to even begin testing this – and I've read through the entire thread **EDIT And I'm currently re-reading it** – and consider myself a seasoned ML user!

Could a user of these modules maybe make a tutorial post/video on: which modules to install, how to safely tweak the registers, (lol) etc. ? It would be tremendously helpful and might encourage more people to test, feedback, and develop these proof of concepts into a mainstream module.

Rock this!

I think I'm in your same situation, so far to find the code for my 600D I've put ML iso research branch on a card and started it up, then i did those steps:
- enabled adtg_gui and raw_diag in modules,
- set iso to 100 and restarted camera by opening and closing card door
- in Debug->ADTG Register set the show option to "Modified since ..."
- take a pic at 100 ISO
- change ISO to 200 and take another pic
- go in ML->Debug->ADTG Registers and you will see the registers changed from 100 to 200 iso (for example i got for CMOS[0]: 24 (was 0) so 24 is the actual value at 200ISO and 0 the value that was set at 100ISO
- note down the values and proced to the next: iso 400, set it take pic and return to ADTG registes to see the new values and go on like that for all the ISOs

Just to be more clear If you start camera with 100ISO, take a pic and look in ADTG Registers you will not see nothing changed (since values started from boot with 100 ISO, that's why at first you should take two consecutive pic at 100ISO and 200ISO to see the differences from those two...

Now I'm gonna try to change something to see if makes differences and maybe this wil hel me also in understanding what all this topic is talking about.
I'm thinking that a lot of stuff is more hard to understand by only reading the forum and instead by making some test while following the topic helps a lot in understand it better

Now to start play I've set 8882 to 0 take a pic and got like banding on the pic set also 8884 to 0 and the banding gone away, that explained me what it mean the help with: "ISO ADTG gain (per column, mod 4 or mod 8 )"
If I override also 8886 and 8888 to 0 I get black pics and if I set 8886 and 8888 to 0 and 8884 8882 leave untouched I got one pic pic with only red channel

so next step will be set up a scene with high DR take a couple of pic with original values and canonc values and try to see if the overrides gave better results or not
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 04:00:42 PM
Ok, so after another read I've made a start following @70MM13 hints and @a1ex's info and screenshots. So far I think I'm missing a piece of the puzzle:

I'm documenting this as a trail for others attempting to follow in my intrepid footsteps into the dark wood ahead...  :P

1) I've downloaded the ISO Research tools from here: https://builds.magiclantern.fm/modules.html

2) I've added the iso_regs.mo into my modules folder of @Danne's crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Apr02.5D3123.zip  build here: https://bitbucket.org/Dannephoto/magic-lantern/downloads/  (I understand I should load *either* iso_regs *or* adtg_gui.mo but not both! They provide similar functionality?)

3) I've enabled the iso_regs.mo

4) Gone into ML menu Debug > ISO Registers >

5) Now I've followed @70MM13's following instructions [brackets mine]:

"I have a very simple method based on my own experiments following a quick reading of a1ex's information contained within this thread:

1: clear the custom iso settings [I take this to mean if there are already changed values here from a previous shot?]
2: set desired base iso (canon menu) [Easy – I tried ISO200]
3: take a silent photo [Having trouble making this work, option is greyed out even when turned ON. I took a conventional 'shutter' pic instead]
4: "copy canon settings" [Pressed this and some values filled the page]
5: set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value [I set CMOS Gain (0xfe?) to 3, ADTG preamp to 2,  ADTG Gain to 100 (half of ISO200)]

That's it!"

6) I come out of that menu and shoot a short mv1080 clip of an HDR scene making sure to avoid clipped highlights but to include darkest shadows.

7) The clip records – but the liveview has smeary wide vertical green bands on it. These look very scary – but fade after a couple of shutter presses – and totally clear after I disable the Debug > ISO Registers > to OFF [BTW Do these values reset to default when I do this?]

8) I examine the [dark] clip in MLVApp1.6 and adjust the dark strength slide to recover the shadows. I adjust the RAW White value to tame the pink highlights. Overall result: Ugly coloured noise in the shadows!  :(

Am I on the right track here?

Thanks!

@scrax Just seen your post! Looks like you have loaded different ISO Research modules to mine?  ???






Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 04:05:34 PM
i'd be overjoyed to see more users play with this!

i don't really have anything new to offer in terms of tips, everything i'm doing is still pretty much the same as what i wrote before on this thread, but in terms of experience, i have another useful tidbit:

for still photography this is also quite useful.  i have been doing some long exposures for macro and night sky photographs, and the noise floor is so nice with the gain reduction that it is a joy to work with.

this same principle would be very good for dual iso as well, reducing the noise on the high iso portion.

plus, for extremely bright highlights, we could have iso 50 or even 25.  no ND filter required!

there's a lot that can be explored here, it just needs enthusiastic explorers!

these days i am using this mode alongside dual iso with the excellent binning.  between the two modes, everything i need to do is attainable!

this is the best video i have done so far using only this iso "109" mode, every single shot with only iso "109" on my 5d3 at 3k...

https://vimeo.com/314928371

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 04:23:12 PM
@timbytheriver:

if you turn it off, it no longer works.  you're looking at native canon iso.

the nastiness you are seeing fades away after 30 seconds and then you're ready to record.

BTW: lately i have been lazy with the settings here, and still getting great results.

here's what i do these days:

start with clear settings
set iso to 200
take a regular still photo
copy settings
set ADTG preamp to 2
set ADTG 0xFE to 0

done!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 04:42:29 PM
@70MM13 Hey!

Thanks for the response. Hope that green liveview isn't a fried sensor!  :P

The only module I have loaded is iso_regs.mo. Is this and my subsequent actions correct? 

Will try another test shortly with your updated settings.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 02, 2019, 04:43:06 PM
Quote from: timbytheriver on April 02, 2019, 04:00:42 PM
@scrax Just seen your post! Looks like you have loaded different ISO Research modules to mine?  ???

I've got the iso research branch 2019mar23 for 600D and I didn't added the module from Danne because iso-regs works only for 5D if I understand right, so I'm using adtg_gui and not iso_reg as module to edit adtg that's why you have different stuff in your module
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 04:49:58 PM
@timbytheriver
yes, for this simple "use it only" mode, iso_regs is all you need to load.

here's an example image i took using liveview silent capture using those settings, and i intentionally snapped it before the settling was finished, giving interesting stripes in the shadows as an effect!

fun stuff...

https://drive.google.com/open?id=1UVLHp0_qq1t_xfsCmkp47A0QfBL4H4uU
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 02, 2019, 05:02:42 PM
by the way... what is:
badcaffe
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 05:04:35 PM
@scrax I understand - thanks for the clarification! Good luck!

@70MM13 That smearing used as an effect looks edgy! :P So far with those settings starting out at ISO200 > 100 (ML) I get these 100% crops from MLVApp:

Spot before and after – not getting much improvement yet!  :-\ Is the 'take a silent picture' a necessary step, or can it be just a regular RAW/Jpeg shot in liveview mode?


(https://i.ibb.co/QrCCJks/Screen-Shot-2019-04-02-at-15-57-11.jpg) (https://ibb.co/QrCCJks)


(https://i.ibb.co/9hf5Tb1/Screen-Shot-2019-04-02-at-15-56-58.jpg) (https://ibb.co/9hf5Tb1)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 05:14:36 PM
any capture should work, i think...  i use it in stills mode also...

the difference shows up when you push the shadows.  play around and see what you can come up with.

i look forward to people making progress with this.  i am lazy and just do what works for me to get cleaner shadows.  there's probably a lot more just waiting to be discovered!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 05:22:59 PM
@70MM13 Thanks, but I'm wondering whether it's even working on my cam!  :o

When you take the 'reference' pic, are you ETTR or ETTL ? or in the middle?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 05:34:23 PM
you'll know it's working when you see the liveview get messed up and slowly clean up again...

if you look at the final shot of the video i linked above, you can see it happening.  i used it intentionally as an effect.  i added the "alien" writing scrolling up through part of the actual amplifier settling process!

as for the reference photo, i don't know if it matters?  i have never been careful about it.  maybe it matters, but i have no idea!

are you doing this all in video mode?  i do...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 05:47:54 PM
i don't know if this makes a difference, but i do this in 5x zoom mode for 3k recording...

part of this seems to involve stumbling around to get the right order of operations it seems.

also, i'm using a version from december 2017.  somewhere earlier in this thread i posted a copy of it for anyone to grab...  maybe it would help?

again, i have no idea!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 06:16:21 PM
@70MM13 Yes, video mode. Non-crop so far!

I love your suck-it-and-see style! Will keep playing...  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 06:17:23 PM
Quote from: scrax on April 02, 2019, 05:02:42 PM
by the way... what is:
badcaffe

I don't know! What is badcaffe?  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 06:37:14 PM
Quote from: 70MM13 on December 09, 2017, 06:26:42 PM
ISO 109 tested for shadow noise improvement:

Three images, all settings identical except iso.  Exposed for highlights in a low light scene to really showcase shadow noise.  This time I adjusted register settings to compensate for highlights (as I should have before, but was too eager just to see results!)

No image processing except exposure compensation.  No noise reduction at all.

109 rocks.

5D3 1.13
(http://thumb.ibb.co/bsxgub/iso_100.jpg) (http://ibb.co/bsxgub)

(http://thumb.ibb.co/g7SEEb/iso_109.jpg) (http://ibb.co/g7SEEb)

(http://thumb.ibb.co/cgLefG/iso_200.jpg) (http://ibb.co/cgLefG)


@70MM13 ISO 109? Are you setting this 109 as a ML ISO ? Not sure what your process was in this post – but it looks great!  :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 02, 2019, 06:47:09 PM
Quote from: timbytheriver on April 02, 2019, 06:17:23 PM
I don't know! What is badcaffe?  :o
I don't know either  :D

I got it set to all value after a fresh boot:


Canon EOS 600D 1.0.2
00f00000:       0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=4069464c Analog ISO (most cameras)
00f00001:     423 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=4069464e Vertical offset
00f00002:      18 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3138 addr=40694636 Horizontal offset / column skipping
00f00003:     1d0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3138 addr=40694638 Analog ISO on 6D
00f00007:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c3910 addr=40694650 5D3: image fading out; 6D, 700D: vertical offset
000200c7:    3201 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694288
00020002:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=4069428c
00020006:       1 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694290
00020007:       0 (was badcaffe) ISO=100 Tv=30 Av=45 lv=0 zoom=1 mv=0 res=-1 crop=-1 task=ShootCapture pc=ff1c2e9c addr=40694294 ADTG preamp (4 2-bit column gains; clean image at 00, 55, AA and FF)
...


I suppose is a magic code like deadbeef, waiting the answer of the experts

from wikipedia (https://en.wikipedia.org/wiki/Hexspeak):
"("bad cafe") is used by Libumem to indicate uninitialized memory area. "

so could be same here I suppose
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 02, 2019, 07:10:25 PM
Quote from: scrax on April 02, 2019, 06:47:09 PM
I don't know either  :D
...
so could be same here I suppose

New street word of the day: "You're soooo badcaffe!"  8)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on April 02, 2019, 09:05:56 PM
Hehe - badcaffe is a magic value for registers not yet discovered.

So, when comparing two video modes, "was badcaffe" actually means the register was discovered while switching to the second mode, but its value was unknown in the first mode.

Solution: just retry the procedure, as now adtg_gui will be aware of all registers. That's become most registers are configured only during a mode switch, not continuously on every LiveView frame.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 02, 2019, 09:41:41 PM
@timbytheriver

the process i used for 109 is the one i outlined back then for getting 109 from 200, pretty much exactly what a1ex taught us at the start of this thread! :)
just look at posts i made on here around that time and you'll see the exact values i used.

i think it will be similar to what i'm getting now from the simplified approach i posted earlier today, but that's just a guess...  who knows... not me!

the values i use now result in something like 111, i think.  i don't pay much attention, i just rock and roll

i'm the wrong guy to do serious analysis.  i just want to play, mainly in low light, and i know what i like!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 03, 2019, 10:04:52 AM
@70MM13 Thanks once again. You've given me confidence to play!

@a1ex badcaffe = so many possibilities...  :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 03, 2019, 12:35:58 PM
After further tests I have some more questions, as I'm still not seeing the dramatic improvements that others have demonstrated:

Could someone confirm that I'm not missing something important here as I hope this will be useful for those to come!

I'm using iso_regs.mo *only* on 5DMK3 shooting in 3K 1:1 mode, base ISO400.

Take the following shot of HDR shot Desk lamp and shadowy books.


(https://i.ibb.co/CVzbN83/b.jpg) (https://ibb.co/CVzbN83)


(https://i.ibb.co/5nT2VRz/a.jpg) (https://ibb.co/5nT2VRz)


1) I've exposed for the lampshade at the point where the raw histogram 'OVER' appears, and then backed off a hair. Is this correct procedure?

2) I take a still photo while in Liveview mode (CR2 and jpeg which save to my SD card). I don't shoot in 'silent picture' mode as this seems not be working here. Does this make a difference?

3) I take a reference movie clip at these settings.

4) I go into ISO registers and make sure default values are present.

5) I click 'Copy Canon settings'. Sometimes ALL the values populate – other times only some values (say CMOS, ADTG, Digi GAIN) Is this expected behaviour?

6) I tweak the registers as per @70MM13 instructions: I tried set 0xfe to 3, preamp to 2, and reduce adtg until the "iso" is half of the base value = I exit that menu. Is this correct?

7) I shoot the clip at these settings. Now, I'm confused about the instructions to 'match the liveview exposure' What does this mean?

8 ) I load the clips into MLVApp, adjust WB, Exp, Dark strength, and get disappointed!  :-[

Any thoughts / suggestions?

Many thanks.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ArcziPL on April 03, 2019, 12:53:09 PM
Quote from: timbytheriver on April 03, 2019, 12:35:58 PM
7) I shoot the clip at these settings. Now, I'm confused about the instructions to 'match the liveview exposure' What does this mean?
I'm not 100% sure but would understand it as follows: you are effectively reducing the sensitivity (ISO 400 -> xxx) so you should compensate it with with either increasing exposure time or aperture to match exposure between both photos, i.e. before and after adjustments.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 03, 2019, 01:11:11 PM
matching exposure in the live view is critical.  keep in mind you are reducing the ceiling when reducing gain.  it'll show in the liveview scopes.  when you have it working properly, you will find that you can't go above whatever the new ceiling is, for example 95%.

if i had to guess what isn't happening for you is the photo capture/copy settings.  sometimes it just doesn't want to work, and i have to repeat the procedure a few times until it finally works.  ALL of the parameters will be filled in when it worked...

maybe i should make a little example video/tutorial for what i'm doing, if it might help somehow.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 03, 2019, 01:21:25 PM
Quote from: 70MM13 on April 03, 2019, 01:11:11 PM
maybe i should make a little example video/tutorial for what i'm doing, if it might help somehow.

@70MM13 That would be *very* much appreciated!  :D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 04, 2019, 06:46:34 AM
video uploaded, started a thread to discuss...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on April 04, 2019, 07:09:20 AM
Hm, every time I see iso changing in these gain tests my own personal placebo journey into these regs comes to mind.
Not fully understanding how iso changing is relevant in this investigation I think the more correct way to work on these regs would be to don't let iso change at all. to check if there´s a way to actually get more dr from the sensor by pushing pregain and other related regs around and then check raw_diag.mo for proof. If iso ends up changing in these changes it's clearly something that affects the result much more than the actual pregain changes. So how much more I wonder?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 04, 2019, 12:33:33 PM
did you watch the video?

the results speak for themselves.

at the end of the day, i'm using it and getting indisputably good results, and it makes me happy.

just look at the last minute of the video if you refuse to watch the whole thing and ask yourself which result would you want on IDENTICAL images...

do results matter?  i'd call this the anti-placebo.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on April 04, 2019, 01:01:40 PM
I say I can´t hardly tell what´s going on in certain parts of that vid. If I understand correctly you have the camera set to iso 200 and then reduce gain from 52 to 2. You then record a file but prior to this you change exposure manually viewing histogram? Next you turn these settings off and lower the iso 200 exposure matching the first example by viewing histogram?

So, key is that you change shutter in first example to get back lost light since it now reflects iso 111. This would mean you capture much more light into this particular set up mimicing exposure to the right only now it will look like it´s normally exposed. The second example you do the contrary. Apply the "normal" looking exposure but it will give an unfair result since shutter speeds are not the same anymore.

Question of a retest:
1 - Use the same shutter for both recordings. The first will be the normal looking histogram image but with reduced gain. The second recording will be with the same shutter but look brighter, more exposed to the right since you now turned of the reduced gain regs. I suspect in real life these two recordings will look very much the same when fixed in post.

2 - In post match the viewable brightness and then compare the grain and other seemingly better results.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 05, 2019, 02:40:53 AM

(https://i.ibb.co/DQfRfB6/200.jpg) (https://ibb.co/DQfRfB6)

(https://i.ibb.co/k9bX1Gm/111.jpg) (https://ibb.co/k9bX1Gm)


i followed your procedure.  200 was overexposed by a half stop.  i pulled it back in post.  it's better, but 111 still wins even when cheating in 200's favour.

and this indisputable improvement is free, and who knows how much better it can get with better adjustments with so many other registers?

we will never find out without exploring...

everyone can play!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 05, 2019, 05:23:37 PM
Reporting some operational issues arising from testing iso_regs on latest 4K crop-rec build (Experiments page).

Using 'Disable overrides' prompts message to restart (Help text) but upon restart values are not set back to default. Requires more restarts to happen as expected.

Using 'Copy Canon settings' nearly always fills only some values. Requires [often several] restarts to fill all values correctly.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: lucoctober on April 05, 2019, 08:32:00 PM
I've tried it, and it works great. What I noticed, that the deep shadows now have green tint instead of magenta. Banding is still there, but overall nice improvement :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 06, 2019, 02:21:45 PM
Quote from: 70MM13 on July 10, 2018, 07:43:49 PM

these images were created by loading the DNGs into mlv app, and dropping the dark strength to zero.


How did you load DNGs into MLVApp? I can't find anyway to do this!  ???
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: histor on April 06, 2019, 11:45:59 PM
By the way, iso research build doesn't work on 5d2 (2019Mar23.5D2212.zip). It can't power off the camera – both leds stay on for long (not 2-3 sec, as described – long enough to panic) [I needed some time to get the card door trick].

Another finding, which makes me exited. After reading the nice article from RawDigger developers (https://www.rawdigger.com/howtouse/iso-is-seldom-just-digital-gain) I ran into checking my own files from 5D2. With such a histogram it takes just several minutes to know, that the most interesting ISOs were 160, 320 and 640 – with clear signs of data division. All others look like pulled up (with periodic gaps in the histogram). That just proved what everybody told (but I'm a hard believer).

But that's not the end of the story. I continued checking files from my archive and found a batch of files with ISO100 and ISO50 (which is supposed to be complete fake) – with absolutely smooth histogram – no gaps, no spikes. How could it be so? It was hard to reproduce them: finally I grasped it. With EMF chip attached (telling f1.4) or an AF lens I get gaps in the histogram. Without electrical connection to the lens – histogram is nice and smooth. Peripheral lighting correction is disabled, so I can't guess what kind of correction is applied here. Should we avoid it?
It needs much more testing in the terms of noise and impact on real images. Taking a break now. Maybe, I'm completely wrong...


Continued:
I've checked results with Canon 50mm lens:
At f/1.4 – each 13th value in the histogram has a gap. It becomes less noticeable at closed diaphragms.
f/1.4 - each 13th
1.8-25
2.8-31
2.2-43
2.5-55
2.8-84(85)
3.2-70
I can't imagine any fair reason to use pull up at open apertures. Just maybe their lenses lacked light at open apertures and canon gays are simply cheating to satisfy specs? Looks rather probable.
In any case it would be nice to disable this pull up if somebody finds it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 07, 2019, 04:03:22 AM
@timbytheriver
it must have been a typo...  i was clearly loading MLVs, not DNGs.

i think the reference might have been related to something i was outputting as DNG, therefore the easy mix-up.

sorry!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 07, 2019, 09:44:47 AM
@70MM13 No probs. I knew it was too good to be true!  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 08, 2019, 01:07:48 PM
Is there a knack/shortcut to turning the top wheel x hundred times to adjust each register in adtg_gui.mo?  ;)

E.g Change ADTG2 [8882]  to 0x13b (five minutes of turning!)



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: scrax on April 08, 2019, 01:16:52 PM
Quote from: timbytheriver on April 08, 2019, 01:07:48 PM
Is there a knack/shortcut to turning the top wheel x hundred times to adjust each register in adtg_gui.mo?  ;)

E.g Change ADTG2 [8882]  to 0x13b (five minutes of turning!)

can't you select each char in the number with arrows? On the module I use it works
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 08, 2019, 01:24:12 PM
@scrax Thanks! That works a treat. Doh!  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 08, 2019, 03:43:08 PM
Is the CMOS[0] register in adtg_gui.mo the same as CMOS Gain in iso_regs.mo ?

I read of an adjustment to CMOS[0] to say 0x333, but will the CMOS Gain take the same input setting/value?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 09, 2019, 10:52:14 AM
@70MM13 Am I correct in thinking the reduced gain hack can be used in conjunction with dual_iso – swapping the recovery isos if that makes sense! Any benefits here?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 09, 2019, 12:28:05 PM
@Danne

When I use these settings in adtg_gui.mo from your earlier experiments I get keep getting a liveview/recorded image which looks like 2-strip technicolor – just greens and pinks.

analog gains changed to:
ADTG2 [8882] 0x13b
ADTG2 [8884] 0x13b
ADTG2 [8886] 0x13b
ADTG2 [8888] 0x13b

ISO 100 CMOS[0] 0x222

Elsewhere you mention that there are 8 registers to change on the 5D3...  :-\  Any ideas?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 09, 2019, 12:37:00 PM
that's exactly what i've been asking about for quite a while now...

it would be interesting to see what can be done.  if there's nothing preventing it from working, it may extend the range into the darkest shadows!

boldly go, brother...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 09, 2019, 12:48:48 PM
Have you tried it already? Or am I taking a leap of faith...?  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on April 09, 2019, 01:03:34 PM
A1ex could advice here. My opinion atm is. Don't touch analog gains or cmos0 cause it will at best cause confusion. Instead enable raw_diag.mo and test pregain regs or other regs affecting highlights. Study raw_diag closely after changing and get used to the idea that only a slight refinement is likely to happen rather than "a new sensor" thing revolution.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 09, 2019, 01:06:02 PM
@Danne Do you mean for the dual_iso question – or the one previous (8 registers)?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on April 09, 2019, 03:54:33 PM
Have tried to get iso_regs and dual_iso working in tandem. Not working here. dual_iso menu shows only one iso... liveview shows no 'scanlines'.

Can they be made to work together for benefit? :)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Sganzerla on April 26, 2019, 11:05:37 PM
I'm interested in this technique to make my low light footage 'cleaner' so made 3 tests without any conclusion of the real benefits with my 5D MKIII, no crop mode.

The last test I made (ISO 3200) was puting ADTG Preamp to 2 (Default is 85 in my camera and not 52 as in the video) and ADTG 0xFE to 3 resulting ISO 1648. When exporting the .mlv and using ACR to make the adjustments I found no improvement after matching the brightness of both files - normal and with iso_reg.
If I turn camera off and on again, the resulting ISO changes to 1923 without touching anything!

My questions are:
- Is this the right way to test this module?
- Is the turn off/on 'resulting ISO' change expected?
- I haven't touched the shutter/aperture dials in my latest test, they were the same, was it the wrong approach?

PS: If there was a video showing how to use dual iso the proper way as @70MM13 video showed how to use this module, it would be very cool.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on April 27, 2019, 01:12:56 AM
the settings i use are strictly for iso 200.  it won't be optimal for any other iso.

i have a feeling that there will be no benefit for iso 3200, but you will have to experiment to find any setting that helps.

good luck!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Luther on April 27, 2019, 04:46:42 AM
Quote from: 70MM13 on April 27, 2019, 01:12:56 AM
i have a feeling that there will be no benefit for iso 3200, but you will have to experiment to find any setting that helps.

I think Canon's ISO goes from range 100 to 1600. Above that it uses digital push. @a1ex could correct me if am wrong here.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 27, 2019, 01:06:54 PM
Quote from: Sganzerla on April 26, 2019, 11:05:37 PM
- I haven't touched the shutter/aperture dials in my latest test, they were the same, was it the wrong approach?

These registers predominately affect highlights.  You gain highlight headroom.  If you don't change exposure settings, you simply increase the space between signal max and sensor saturation.

Quote from: Luther on April 27, 2019, 04:46:42 AM
I think Canon's ISO goes from range 100 to 1600. Above that it uses digital push.

Depends entirely on the camera.  5DIII is good to ISO 3200. (http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Luther on April 27, 2019, 01:48:00 PM
Quote from: Audionut on April 27, 2019, 01:06:54 PM
Depends entirely on the camera.  5DIII is good to ISO 3200. (http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html)

Couldn't understand the Table A2. Can you explain? From 1600 to 3200 it has a drop in DR and double more noise, from what I got from this table...


  noise           (electrons)  (e/DN)    ISO
16.86                2.655    0.157    3200
31.76                2.501    0.079    6400
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Sganzerla on April 28, 2019, 07:18:49 AM
Thanks for the insights guys, will take a more refined closer look at this topic again.
English is not my native language so sometimes the technical aspects are harder to understand.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on April 28, 2019, 11:00:54 AM
Quote from: Luther on April 27, 2019, 01:48:00 PM
Can you explain?

Table 1

  ISO   Apparent   Maximum   DR
         Read      Signal
         Noise

  1600   3.6        4050    10.1
  3200   2.7        2030     9.6
  6400   2.5        1000     8.6


Apparent Read Noise (http://www.clarkvision.com/articles/digital.sensor.performance.summary/#noise_sources).

From ISO 1600 to ISO 3200 (one full stop), the DR only decreases by 0.5EV.  Thus, you throw away the full stop of highlights from the ISO bump, but you gain 0.5EV in the shadows.  Backed up visually in table 2a and 2b.

From ISO 3200 to ISO 6400, you don't gain anything (except more accurate exposure in camera).  A full stop of highlights has been thrown away, but the DR has also reduced by one full stop.  You didn't gain any signal in the shadows (the entire point of bumping ISO).

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: histor on June 09, 2019, 09:00:15 PM
This month I was testing reduced analog gains on 5d2 while everyday shooting and results look great. In short: it looks like we can pull down iso100 to something about iso20-25 (upd: lets's say about 50) still having full dynamic range (up to 11.6).

Setup.
Reddeercity's builds for 5d2 + raw_diag.mo
(https://i.ibb.co/Xk74dtQ/VRAM8.png) (https://ibb.co/6JHZpFM)
As Alex once reported, it's reasonable to pull these DFE values from 48x down to 250 – until vertical banding becomes very noticeable. I've tried lower values and subtracted darkframes in post. Luckily vertical banding stays constant for the same ISO and DFE values. I don't know whether it is important or not, but I used CMOS[0]=0x200, 0x204, 0x208 to get the darkframes (for iso 100, 200 and 400 respectively). Either it's some physically stable bias (hard to believe that) or just the factory calibration playing against us. I keep just 3 master-darks, they work without a problem. Final images after subtraction show no FPN and very low random noise (reason 1 - analog amplifiers may generate a lot of noise themselves; reason 2 – longer exposure gives larger number of photons).
(https://i.ibb.co/12dkhcz/ob-dr.png) (https://ibb.co/kc9tkZm)
At DFE values about 100 raw_diag reports worse DR. We can see shadow clipping (posterised green shadows) in post. 111 was just my safe random choice. It turns selecting of iso 100-200-400 to something about iso 20-40-80. All look unbelievably clean. Iso800 is out of use here (it's produced with DFE values 0x8xx). Iso1600 is a digital push.
SHAD_GAIN (c0f08030) set to 0xfff disables digital push for wide apertures. Not a great change, but still useful.
(https://i.ibb.co/Cv0sZK2/IMG-19935.jpg) (https://ibb.co/y5sB1FS)  (https://i.ibb.co/rZKr5BJ/IMG-0234.jpg) (https://ibb.co/RCGwQXW)
Attaching 2  samples of processed images (shadows lifted a lot) and some raws to play with. "-df" in filenames means that I've already subtracted blacks with Pixelfixer (I like its CR2-IN > CR2-OUT approach).

Known problems.
The only way to see overexposure is looking for pink highlights. Auto-ETTR can't catch it but raw_diag.mo works perfectly.
Low analog gains may be used with Dual-ISO module (or simply patching CMOS[0]=0x343 for iso100/400). But final images are not always recognized by cr2hdr-20bit. Using –force gives processed images with striped lights. Attaching samples, didn't tried it much.
Works also with Reddeercity's video hacks (14bit raw, don't forget darkframes) but adtg_gui.mo is an overkill here – it takes a lot of resources (and power?) and a lot of time for setup. No way to save its settings. Not a run-and-gun tool. What we need to emploi low iso range is a tiny separate module, patching DFE values at start. Maybe just on/of and a choice of value for them. Unfortunately I'm not the one to write it.

Checked the same approach with 600d. It gives something, but nothing groundbreaking yet.

Test raw files - https://app.box.com/s/d7i3qj09pc300ffb73sgrtnj7totjc7i (https://app.box.com/s/d7i3qj09pc300ffb73sgrtnj7totjc7i)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on June 12, 2019, 01:20:39 AM
Quote from: histor on June 09, 2019, 09:00:15 PM
In short: it looks like we can pull down iso100 to something about iso20-25 still having full dynamic range (up to 11.6).

This can be thought as a digital reduction of ISO.  Yes, you have used an analog amplifier, but you haven't increased the cameras ability to capture more photons.

To determine if there is any positive effect to reducing the analog gain this much, you would need to measure the signal to noise ratio of the resulting images.  Did you reduce the noise in the ouput (increase SNR), or did you simply turn the volume knob down and create a bunch of known problems processing the resulting files?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 29, 2019, 04:03:44 PM
For those still interested in the controversial reduced gain/low-noise-shadows debate/saga!

I could have *sworn-blind* that when I did my initial (very positive) iso_regs / iso experiments some months ago that they all worked nicely with iso_regs.mo and crop_rec.mo ON at the extended resolutions (UHD, 3K, Anamorphic).

But then it just appeared to stop working on a recent build.

To put my mind (and memory) at ease I tested several different builds and discovered that on all of them – with the exception of the 3.5K 1:1 centered (x5) mode – the crop resolutions (UHD, 3K, Anam.) do not work with iso_regs ON. If you try this the resolution will simply jump back to native each time you record.

I tested:

Alex's builds from Jenkins:

crop_rec_4K Dec09_2017
crop_rec_4K Feb05_2018
crop_rec_4K Jul09_2018


Danne's builds from his repo:

crop_rec_4k_mlv_snd_isogain_1x3_presets Jun28_2018
crop_rec_4k_mlv_snd_isogain_1x3_presets Jul25_2019
crop_rec_4k_mlv_snd_isogain_1x3_presets Oct29_2019

All tested on 5D3_1.1.3

If anyone following the reduced gain/low-noise-shadows saga knows better, please feel free to put me right! Do you have a build version/crop_rec setting that works with iso_regs ON?

Can anyone techy explain why iso_regs.mo and crop_rec.mo won't play nicely together at extended resolutions? – it's way beyond my code skills to understand at the moment.

It would be great it could be made to work!

Thanks!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on October 29, 2019, 05:03:31 PM
Short answer: reply #999 (https://www.magiclantern.fm/forum/index.php?topic=10111.msg205429#msg205429) addresses exactly this issue. Both iso_regs and crop_rec need to adjust the same set of registers (ADTG and CMOS), but the current backend only allows one piece of code to work at a time. In the same group of conflicting items, you will find adtg_gui and the plain old FPS override.

Currently, the above is just a proposed specification, nothing more. I do not have any working code, not even on my PC.

It would probably take me about one week of full-time work to implement it, or maybe 1-2 months it I were to work on it for one day (8 hours) per week. I'm unable to make any meaningful progress on this particular topic in say 30-minute or even 1 hour slices, as it requires highly skilled and focused work. Back in 2014, while I was actively researching this topic, I was probably spending more than 8 hours per day on ML on average, as I had a very relaxed schedule at job, and very little else to do at home. Now, with a nearly full-time job + family, even one day per week fully dedicated to hobbies would already disrupt my schedule. Just in case anyone is wondering why the progress slowed down so much lately.

So, unless (until?) I'll manage to get my time back, all I can say is... good luck figuring it out!

Quotecontroversial reduced gain/low-noise-shadows debate/saga

FYI, I don't see this topic controversial at all; I only want to make sure these improvements are real and measurable. Other things were controversial to me in the past - such as mislabeling ISO 1500 to ISO 100 and claiming massive shadow improvements from that. Of course, ISO 1600 is going to be much cleaner in shadows, compared to ISO 100, all other variables being equal, and ISO 1500 is not going to be much different ;)

Of course, this requires a reasonably solid method for evaluating a sensor response, including measurements of the actual ISO, besides noise levels, and I'm no expert here. I tried to come up with a theory, apparently working reasonably well; the procedure still needs to be automated, the repeatability needs to be evaluated, and one important missing bit is the estimation of sensor response curve, i.e. how linear it actually is. Long answer in previous posts, such as this one (https://www.magiclantern.fm/forum/index.php?topic=10111.msg187900#msg187900).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 29, 2019, 07:16:33 PM
@a1ex Many thanks for the study material! I think I see where the challenge lies more clearly now – at least in abstract. Life will have a habit of getting in the way... :P

Quote from: a1ex on October 29, 2019, 05:03:31 PM

FYI, I don't see this topic controversial at all; I only want to make sure these improvements are real and measurable.

Aren't the eyes the final judge? *Ducks for cover*...  ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 10:00:15 AM
I've noted this behaviour – not sure whether it is useful:

With iso_regs and crop_rec ON. In iso_regs: If I go through the procedure of 'Take a picture first' > 'Copy Camera settings', almost always not all the list populates with settings. It often takes restarts/menu/trash can button pushes to fill out all the values below:

CMOS Gain
ADTG Gain
Saturation Offset
Digital Gain
Black/White Offset
Black Reference
ADTG Preamp
ADTG 0XFE
Top OB Size

The odd thing is that when only CMOS Gain, ADTG Preamp ADTG 0FXE, Top OB Size are populated , crop_rec extended resolutions (UHD, 3K et al) function correctly! It's only when all values are finally populated that crop_rec breaks.


Just saying...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 10:07:29 AM
I think adding the used iso_regs in crop_rec and work them in there with the other regs will solve the issue with conflicting modules. Would take a while and not a priority of mine but still...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 10:14:36 AM
@Danne  :o Where would I start with that? Needle in a haystack... Didn't you include your own iso_gain presets in a build at one time? Did they work in crop_rec rezs?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 10:21:25 AM
Well yes. And that was the placebo of the century. Line up the regs you want to include for the eosm and I will take a look, at least point where to start.
Quote from: a1ex on October 29, 2019, 05:03:31 PM
...- such as mislabeling ISO 1500 to ISO 100 and claiming massive shadow improvements from that. Of course, ISO 1600 is going to be much cleaner in shadows, compared to ISO 100, all other variables being equal, and ISO 1500 is not going to be much different ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 10:32:13 AM
Ah yes, Placebogate.  8) Bar the preset labels themselves – the results looked great to my eyes!  :)

Quote from: Danne on October 30, 2019, 10:21:25 AM
Line up the regs you want to include for the eosm and I will take a look, at least point where to start.

Not sure what this request means. Me, I'm 5D2 and 5D3 only (but I'm sure would be awesome for EOS-M also... !)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 10:35:00 AM
Ok, line up regs for 5d3 then.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 10:46:45 AM
Quote from: Danne on October 30, 2019, 10:35:00 AM
Ok, line up regs for 5d3 then.

Sorry, I'm being dense. ::) I don't know what 'line up' means in this context. ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 11:15:22 AM
I assume you are using certain registers that modifies recordings. You need to specify what registers. You can find them with adtg_gui.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 11:41:08 AM
Oh, I see.

In the iso_regs.mo the only ones I have ever touched are:

ADTG Preamp: 2
ADTG 0XFE: 0

I can't seem to make adtg_gui work without turning the image into two-strip technicolor!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 11:59:45 AM
Yeah, I can't work with that info so unless adtg_gui is fired up etc I'm out  8)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 12:21:12 PM
I'll have another look at adtg_gui and see if I can persuade you back in...  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 01:15:53 PM
@Danne Tried atdg_gui with your latest build 2019Otc15.

ISO 200

Modified:

CMOS[0] 0x333
CMOS[6] 0x170 /* seemed to help pink highlights? */

ADTG2 [8882] 0x19a 
ADTG2 [8884] 0x19a
ADTG2 [8886] 0x19a
ADTG2 [8888] 0x19a

ADTG4 [8882] 0x19a
ADTG4 [8884] 0x19a
ADTG4 [8886] 0x19a
ADTG4 [8888] 0x19a

Had to modify Raw White level in MLVApp to 5000 to control pink highlights.

Result was that crop_rec UHD 3072x1536 was [still] made non-functional e.g. actually recorded 1928x964 with atdg_gui enabled. Crop_rec resolutions return after atdg_gui is disabled & restart applied.

PS Is copy/paste available between reg values? My wrist hurts...  :o

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 01:52:42 PM
Getting closer :). The regs referred to can be reached already. Not sure about cmos0, it´s iso right? If so you can select your iso in cam instead and start modifying the other regs.

In crop mode sub menu look for reg_gain You then select either 12bit or 10bit in crop mode sub menu, NOT under RAW video menu, and your gain regs are active(not active with 14bit). See below:
12bit = {6, 0x8882, 250 + reg_gain};
10bit = {6, 0x8882, 60 + reg_gain};

As you can see these analog gain registers are already used to get into 10bit and 12bit.

            if (bitdepth == 0x3)
            {
                /* 10bit roundtrip only not applied here with following set ups */
                adtg_new[13] = (struct adtg_new) {6, 0x8882, 60 + reg_gain};
                adtg_new[14] = (struct adtg_new) {6, 0x8884, 60 + reg_gain};
                adtg_new[15] = (struct adtg_new) {6, 0x8886, 60 + reg_gain};
                adtg_new[16] = (struct adtg_new) {6, 0x8888, 60 + reg_gain};
            }
           
            if (bitdepth == 0x4)
            {
                /* 12bit roundtrip only not applied here with following set ups */
                adtg_new[13] = (struct adtg_new) {6, 0x8882, 250 + reg_gain};
                adtg_new[14] = (struct adtg_new) {6, 0x8884, 250 + reg_gain};
                adtg_new[15] = (struct adtg_new) {6, 0x8886, 250 + reg_gain};
                adtg_new[16] = (struct adtg_new) {6, 0x8888, 250 + reg_gain};
            }


cmos6 can be reached in crop mode sub menu already so test from there.
           
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 02:20:24 PM
Ok. Went into crop_rec submenu > Advanced > reg_gain = 250 (12bit) Is that the value format it's expecting?

Recorded results with this give correct crop_rec resolution 3072x1536 but is definitely 14bit lossless.

Should I leave atdg_gui OFF now and just be trying in this submenu? Bit lost now!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 02:25:41 PM
It will be 14bitlossless container but it's really 12bit. Open up a recorded file in mlv app.
You can work in adtg gui and develop some magic in there and then when ready publish your stuff here. But then just work outside crop_rec.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 02:33:34 PM
MLVApp > info says file is '14bit Lossless'

I changed reg_gain to 250 Is this correct?

Once this setting is in reg_gain, do I then enable adtg_gui and start altering regs also in there? Is this correct?

I haven't yet grasped how/whether I'm supposed to run these two modules together.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 02:48:26 PM
Check white level. It should say 5000 or 6000 in mlv app if 12bit.
If you change reg_gain you can increase and get closer to 14bit or reduce to go towards 10bit. Don't put in 250. Default when 12bit selected is 250. You simply increase or decrease this number.

You can't use adtg and crop rec together. Thought we established this already.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 03:04:17 PM
As an arbitrary start I've set reg_gain to 5

Disabled adtg_gui

Shot a test 200ISO, UHD 3072x1536

File in MLVApp is showing info > 14bit lossless,
White level = 16200

Either I haven't grasped something fundamental (likely) – or something's amiss.

My ultimate goal is to be able to adjust the ISO regs when in extended resolutions (UHD, 3K, Anam.)

Any pointers towards that goal please?  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 03:06:27 PM
And bitdepth in crop mode sub menu? Was it set to 12bit?
If reg_gain is set to 5 it will hardly be noticed. Try moving it in 20 or 30 increments. As I said. In this case you only add or reduce the 12bit analog gain already set to 250.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 03:20:37 PM
Ah.  ;) Missing bit of puzzle: Once 12bit is selected in crop mode sub menu (I thought setting it in reg_gain was enough).

File in MLVApp is now 12bit lossless, 3072x1536, white level 6000

Do I now play about with other registers in the crop mode sub menu to continue iso experiments? (CMOS, et al) Any I should target / avoid touching?

BTW The reg_gain setting seems not to hold after a restart. Is this expected?

Thanks for your help!  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 03:30:24 PM
Not kept after restart.
Good luck!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 30, 2019, 04:53:30 PM
@Danne

Is reg_gain adjustment the same as adjusting ADTG2 in atdg_gui.mo ? Just alters bit depth?

Where is the missing CMOS 0 in your crop_rec submenu?

Getting some results now...  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 30, 2019, 07:48:15 PM
You really need to read my posts again. Cmos0 is iso...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 31, 2019, 10:39:28 AM
Quote from: Danne on October 30, 2019, 07:48:15 PM
You really need to read my posts again. Cmos0 is iso...

Sorry, but you didn't seem definitive about CMOS 0 here:

Quote from: Danne on October 30, 2019, 01:52:42 PM
Not sure about cmos0, it´s iso right? ...

I have indeed read almost every post in this long thread – some many times! it's just that the information takes longer for some to digest more than others. :)

Quote from: timbytheriver on October 30, 2019, 04:53:30 PM
@Danne
Where is the missing CMOS 0 in your crop_rec submenu? ...

Again: Why are the:

CMOS 0
ADTG 0xFE
ADTG 8/9/A/B

registers not accessible to adjust in your crop_rec submenu > advanced? These registers seem to make important changes to iso behaviour!

Not a criticism – I just genuinely want to understand this puzzle and keep the investigation alive :)

Many thanks.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on October 31, 2019, 11:46:01 AM
Well, I have added a lot of regs to be able to modify and test directly in crop_rec. But not all. Only those I found useful and had tie to add. That´s why I asked for specific registers to put in there.
ADTG 0xFE
ADTG 8/9/A/B

Above are renamed regs. I have no idea what they are. But in adtg_gui you should be able to find them. There´s a category that grouped gain registers. So instead of adding all kinds of registers it would be better for you to just use adtg_gui, build some badass combination in adtg_gui, write your regs down and put them here in a post. Then I can take a look and point to where they can put into crop_rec.c.
Alternatively you can check into crop_rec.c and track down how I have added regs and then put in the regs you want yourself in there. It´s up to you.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on October 31, 2019, 12:21:07 PM
Very useful info. Lots of homework.  :D

Many thanks Danne.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 01, 2019, 11:15:29 AM
Did some initial experiments yesterday concentrating on modifying CMOS (0) in adtg_gui (I know! You've all been here before...) Anyway,...

5D3_1.1.3
ISO 200
1080p
14Bit lossless
Danne's crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Oct15.5D3113 build

CMOS (0) Default value is 0x113 (?) (275?)

Making small increases until 0x120 produces tiny upward brighter steps until 0x121 where any further brightness is coupled with 'scanlines' across the image (similar to dual-iso lines) and is unusable. Shot quite a few tests which don't show any improvements over default.

Tried Danne's earlier test settings at ISO 200 of:

CMOS(0) 0x333

ADTG2 (8882] 0x13b 
ADTG2 (8884] 0x13b
ADTG2 (8886] 0x13b
ADTG2 (8888] 0x13b

ADTG4 (8882) 0x13b
ADTG4 (8884) 0x13b
ADTG4 (8886) 0x13b
ADTG4 (8888) 0x13b


Images now show (by eye!) the improvement in less noisy shadows, even when compared with the untweaked ISO 400 – and the highlights don't appear to take a heavy hit.

PNG exports from MLVApp:

https://drive.google.com/open?id=1xfJXOPUoql6LwvL7ro7dZLySD8Eju8P_
https://drive.google.com/open?id=14k-ZN-WxelVEcSGZ9-NOvm4jXYhVZjIo
https://drive.google.com/open?id=16mKD39IGkJMPTKgM-w_btCM_41ryK58q


@Danne If we could access just these registers (CMOS(0), ADTG2 / 4) for starters via crop_rec submenu it would be great to test on crop resolutions. I doubt very much that I can find any 'badass combination in adtg_gui' that's better than you already have – but I'll have a go!

Quote from: Danne on October 31, 2019, 11:46:01 AM

ADTG 0xFE
ADTG 8/9/A/B

Above are renamed regs. I have no idea what they are. But in adtg_gui you should be able to find them.

@a1ex Do you know how these iso_regs values: ADTG Preamp, ADTG 0XFE, correspond to the same ones in atdg_gui so that they can be accessed by Danne's crop_rec version?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 01, 2019, 03:01:40 PM
If I understand this correctly you are standing on iso 100 and increase cmos 0 until it almost reaches iso 200? Then you pull back gain so it looks like you are back into iso 100? Or maybe you are going from iso 200 to iso 400 and then pull analog gan back into iso 200?
Anyway. A1ex can correct me if I´m wrong here. Both cmos 0 and analog gain registers are not going to achieve anything but very small refinements in dynamic range and when modified. By lowering analog ain regs you are loosing dr and by pushing cmos 0 you are only jumping to the next iso. So that leaves to look for what regs to modify to get any possible refinements? What if you exclude both cmos 0 and analog gain regs altogether and focus on the other registers? For instance, start on iso 200 and try to capture more dr with the other regs? What regs idk...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 01, 2019, 03:08:31 PM
I was always starting on Canon base ISO 200 and tweaking the cmos 0 from there. Reproducing your own experiments was a starting point for me so that I know I'm in the ballpark! Hence the cmos , adtg2 / 4 tweaks.

I will have another look and see what other regs might improve DR. Maybe @A1lex has some pointers?

Thanks!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 06, 2019, 02:10:07 PM
I have enabled raw_diag.mo and tried to get a profile using > Dummy bracket

The grid appears, but overlaid with a message 'You may need to solder some RAM chips'. No SNR curve is produced.

Sounds bad-news :( Can this be fixed?

(https://i.ibb.co/yPSGRB0/SNR2-error.png) (https://ibb.co/zxJj5Hs)

**EDIT**
Thanks! @a1ex informed me that this is an 'out of RAM' message. I temporarily disabled mlv_lite and mlv_snd modules while using the raw_diag tools and it now works as intended!  :D
**/EDIT**
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 28, 2019, 04:42:23 PM
@Danne Just doing some reading about lua scripting - I hadn't really looked at this before. Is it possible to call up [iso] register changes using lua? Or even the 'Don't click me' button... ? Even I could maybe manage a lua script ...  ;)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 28, 2019, 04:47:45 PM
Sorry, havn´t got around to help you with your regs in crop_rec yet.
Well, you can enable anything in ml menus with lua I think but there´s still issues running crop_rec and adtg_gui at the same time.
I think you´re better off now looking into the crop_rec.c code and try to understand where to put in your newly found reg settings.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 28, 2019, 05:06:52 PM
@Danne Ah, ok. Yes I've looked into your 'clean-iso' crop_rec.c and stared very hard at it  ??? – I'm looking at these lines:


adtg_new[13] = (struct adtg_new) {6, 0x8882, 250};
adtg_new[14] = (struct adtg_new) {6, 0x8884, 250};
adtg_new[15] = (struct adtg_new) {6, 0x8886, 250};
adtg_new[16] = (struct adtg_new) {6, 0x8888, 250};

adtg_new[17] = (struct adtg_new) {6, 0x8882, 250};
adtg_new[18] = (struct adtg_new) {6, 0x8884, 250};
adtg_new[19] = (struct adtg_new) {6, 0x8886, 250};
adtg_new[20] = (struct adtg_new) {6, 0x8888, 250};



and wondering how to translate ADTG2/4 8, 9, a, b, and ADTG4 (fe) into these lines. I especially don't understand the {6, 0x8882, 250}; lines. If I could crack these lines maybe it would mean I could relieve you of your help! :)



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 28, 2019, 06:36:38 PM
0x8882 is one of the analog iso regs. 250 its value. So I guess 0x2 and add the value you want after.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 28, 2019, 07:06:41 PM
Maybe I've misunderstood  ::)

I thought the reg identifier is say:


ADTG2 [8]

or

ADTG4 [fe]


and the value is say, 0x512 or whatever...

then what is the...

adtg_new [13]

and the 6 value?

bit?

Confused now... ???



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 28, 2019, 07:25:24 PM
So if I want to change:

ADTG2 reg 8 to value 0x2 (decimal 2)

I do:


adtg_new[?] = (struct adtg_new) {?, 8, 2};
adtg_new[?] = (struct adtg_new) {?, 9, 2};
adtg_new[?] = (struct adtg_new) {?, a, 2};
adtg_new[?] = (struct adtg_new) {?, b, 2};

?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on November 28, 2019, 08:37:04 PM
Sorry, hopefully someone else can help you here. You should be able to test on your own. Check for analog gain regs in adtg gui and you'll see where to put in stuff. No guarantees your reg works though.
Sorry but need my focus elsewhere now. Too much work...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on November 29, 2019, 04:14:27 AM
Tim - could you explain a bit more what it is you don't understand?  It's reading like maybe you don't recognise what the struct cast is doing?  Either that, or from my skim reading of Danne's code, the parts I list below?  I'll give my attempt to explain, my apologies if it's too low-level, or too high-level!

In answer to your specific question, which I will change to remove ambiguity of 2 vs 2:
Quote
[how to set] ADTG2 reg 8 to value 0x3 (decimal 3)

I believe you would create a new element in the adtg_new array, then set it like this:
adtg_new[22] = (struct adtg_new) {2, 8, 3};
However, literal 8 looks very suspicious to me given the other regs being in 0x8800 range.

IF YOU USE EXACTLY THAT CODE IT IS WRONG, MAY CRASH AND MAY BRICK YOUR CAM.  Now you have to read my long explanation below to understand why (need to extend adtg_new array).

Please bear in mind I have never looked at this code before and I haven't read all this thread.  I have good experience with C but none with this code, and I understand hardware register manipulation can be very risky.  You are taking the risk, not me, and I recommend you not trust my explanation.

I am working from a specific version of crop_rec.c, and it might be a stupid version!  It was simply the first version that a search found for me:
https://bitbucket.org/Dannephoto/magic-lantern/raw/49f23e49f96129f0777d10e65fdf82942ef48522/modules/crop_rec/crop_rec.c

All that said, it looks to me that dst is how you select which ADTGX is used.  dst is passed into the function and the lower 4 bits are masked in (int dst = cs & 0xF;).

It's important to note that I think adtg_new is an array of changes that we wish to make to registers (if I've got that part wrong my following reasoning will be quite wrong).  Here's the array declaration:
Quote
    /* expand this as required */
    struct adtg_new adtg_new[22] = {{0}};

Here's how a single element of the array is declared:
Quote
    struct adtg_new
    {
        int dst;
        int reg;
        int val;
    };

So, there are max 22 registers the code expects to change.  Here's where an example element of the array is initialised:
Quote
adtg_new[13] = (struct adtg_new) {6, 0x8882, 250};
This sets the 14th element (or 13th depending on how you like to think about 0 indexing...) to have dst == 6, reg == 0x8882, val == 250.  I will later call this "e", and "e.dst", "e.reg" etc.

Here's the code that I think is responsible for making changes to registers:
Quote
            if ((reg == adtg_new[i].reg) && (dst & adtg_new[i].dst))
            {
                int new_value = adtg_new[i].val;
                dbg_printf("ADTG%x[%x] = %x\n", dst, reg, new_value);
                *(uint16_t*)copy_ptr = new_value;
This only changes a reg if some conditions are met.  It's inside a loop that covers all 22 elements of the array.  We read from data_buf, which I haven't tried to understand but I assume holds some list of potential registers to change.  We check every reg struct in adtg_new against every element of data_buf.  If the buffer holds a reg value that matches our e.reg, we then check if passed in dst has at least the bits set that e.dst does.  Now, look at the dbg_printf!  If the passed in dst is 6, with our e, it would print:
ADTG6[8882] = 250

So we know that the passed in dst says which ADTGX to use, and the first field of the struct selects if your change should be applied.  If you want your change to apply to all, use 0xf, due to the logical "&" used earlier (you probably shouldn't do this!  I don't know what the ADTG registers are but applying to all is likely dangerous and wrong).  e.dst == 0x6 will cause the change to be applied if the passed in dst is 4, 2, or 6; i.e., if the passed in dst is ADTG2, ADTG4 or ADTG6.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 29, 2019, 10:32:41 AM
@names_are_hard That is sooo helpful. :D Thanks so much for taking the time to explain it in terms I think [even] I can understand. Reading your explanation has me musing on whether what I am trying to do is above my present skill level...

Basically what I'm trying to do is code some discovered register values (from the research in this thread) that impact iso response into a preset that can be applied in conjunction with an existing crop_rec setting. So:

Shooting in say, crop_rec 3K mode > Q menu > select new preset 'X' for 'adjusted iso registers' = on or off.

At the moment we have to either make the register adjustments in iso_regs.mo or adtg_gui.mo which is fiddly, doesn't work in crop modes, and needs to be reset on power off. I'm looking for a way to code discovered values into a preset 'X'.

*** EDIT ***
Thinking maybe these could be added to the existing crop mode > Q menu. Am I right in thinking these might be added to crop_rec.c at lines (appx) 166 and 1409 ?



static int32_t  target_yres = 0;
static int32_t  delta_adtg0 = 0;
static int32_t  delta_adtg1 = 0;
static int32_t  delta_head3 = 0;
static int32_t  delta_head4 = 0;
static uint32_t cmos1_lo = 0, cmos1_hi = 0;
static uint32_t cmos2 = 0;




and



static struct menu_entry crop_rec_menu[] =
{
    {
        .name       = "Crop mode",
        .priv       = &crop_preset_index,
        .update     = crop_update,
        .depends_on = DEP_LIVEVIEW,
        .children =  (struct menu_entry[]) {
            {




?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on November 29, 2019, 10:06:04 PM
I've never used crop_rec, or any modules with ML, so I don't know how their UI works.  It would likely be many hours work for me to understand the code.  I will say you can probably test those changes in QEMU.  The register manipulation probably doesn't work in QEMU (not checked), but the UI changes should.  Get the UI looking how you want (but the buttons do nothing because you don't need to make code that attempts any register changes, just make it do nothing).  When the UI looks right, attach the UI buttons to real code that changes registers.  Try that in QEMU (even though it probably won't work), and if it looks sensible (doesn't crash at least *before* you press the buttons), try it on a real cam.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on November 30, 2019, 12:45:41 PM
Thanks names_are_hard! QEMU seems a sensible option.  :)

Before I dive into an install, does anyone know whether changes to registers, running adtg_gui.mo or iso_regs.mo can be emulated in QEMU?

Thanks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 01, 2019, 05:51:30 AM
I think you are better off experimenting straight in crop_rec. Maybe your regs doesn't even work or need to be adjusted to work.
I can't say if it's safe or not but can speak from my own testing. Whatever adtg reg I fed in there has not affected anything permanently. Always restored on restart. The only time I soft bricked my eosm was when fooling around with movie crop mode registry and bypassing explicit warnings in code(property reg). @a1ex fixed the mess nevertheless).
I can never gurantee anything. Only tell what I've been doing and if I didn't try out a thousand things in crop_rec I would never learnt how to build the presets I did on my eosm.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 01, 2019, 11:30:26 AM
Ok. Thanks all for good advice. Will persevere!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 03, 2019, 11:22:54 AM
I'm working on translating regs in adtg_gui.mo to iso_regs.mo:

I notice that in iso_regs the following regs allow a single value, e.g.


ADTG Gain 100
ADTG Preamp 2
ADTG 0xFE 3 etc.


Whereas in adtg_gui the equivalent (I think) registers allow for a choice of four values x 2 which may vary by a few figures, e.g

ADTG Gain:

ADTG2 [8882] 1083 (dec equiv...)
ADTG2 [8884] 1080
ADTG2 [8886] 1081
ADTG2 [8888] 1082

ADTG4 [8882] 1079
ADTG4 [8884] 1079
ADTG4 [8886] 1079
ADTG4 [8888] 1082

etc...

Does this mean that adtg_gui allows for a finer (per column?) adjustment of the reg values than iso_regs? Is iso_regs averaging all these values into one value?
Since I'm working on hard coding some clean iso values into a preset I'm wondering whether I should be adjusting a different value for each – or using the single value from iso_regs.

Thanks!

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 03, 2019, 05:10:36 PM
I don't have that code so I can't check.  If you link to the lines in the repo you're using I could try and understand it.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 03, 2019, 05:23:08 PM
https://bitbucket.org/hudson/magic-lantern/src/75c1f3fe29273f1057ca932f71446d75be8937b8/modules/adtg_gui/adtg_gui.c?at=iso-research

adtg_gui.c Line 69-ish

https://bitbucket.org/hudson/magic-lantern/src/75c1f3fe29273f1057ca932f71446d75be8937b8/modules/iso_regs/iso_regs.c?at=iso-research

iso_regs.c Line 237-ish

Thanks @names_are_hard :)

PS The front-ends in ML look like this

(https://i.ibb.co/hm48fgn/IMG-3273.jpg) (https://ibb.co/BCQsNf7)

adtg_gui

(https://i.ibb.co/1QmqBj1/iso-200-from-400.png) (https://ibb.co/wgzW8jV)

iso_regs
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 03, 2019, 05:56:26 PM
So by looking in adtg_gui you should be able to just add the regs in crop_rec. What's the problem?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 03, 2019, 06:01:41 PM
Not a problem. I have already added in crop_rec.c so and they appear to be working.

My specific question was about why the adjustments in adtg_gui allow for four different values to be set per blue/green 1 (ADTG2) and red/green 2 (ADTG4) – whereas in iso_regs they seem summed into a single value. Just seeking to understand properly.  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 03, 2019, 06:25:15 PM
Oh, I see. Go on then :).
Could you share samples, comparisons?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 03, 2019, 06:52:05 PM
Tim, in the comparisons you give, one is an array of known_reg structs, the other is uint32_t type (without looking very hard I suspect this will be treated as an address later on).  They're not directly comparable so it's not clear to me what you mean.  In particular I don't know what you mean by "allowing" numbers of values.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 03, 2019, 07:05:25 PM
Ok, so. I guess you know this already but adtg_gui and iso_regs are separate modules that allow for manipulation of the registers.

There is overlap between them. iso_regs is 5D3 only; adtg_gui is other cameras also. For a new-to-coder like myself I am confused by the different way that what appear to be the same registers are called different names in either module.  :o

I started out tweaking iso_regs only and that works just fine. Then I tried adtg_gui and discovered that when in iso_regs the adjustment say for ADTG preamp is '1', in adtg_gui this single value seems to be 'split' among several wherein each has a slightly different value! e.g


ADTG2 [8882] = 0x43b
ADTG2 [8884] = 0x438
ADTG2 [8886] = 0x439
ADTG2 [8888] = 0x43a

ADTG4 [8882] = 0x43b
etc...


I'm just seeking to understand why, and what implications this has for hardcoding a value or set-of into a movie preset! :)

Maybe I've completely misunderstood something fundamental ... just seeking to do it right!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 03, 2019, 08:55:00 PM
That helps me understand what you're trying to do (I didn't know what the different modules were for), but it is still not clear without the code you're referring to.  You say a "1" is "split" in a different file, but only give one example of code, and it doesn't involve a "1".  The lines you quote aren't in the files you gave links to.  If you don't know, you can click on the line numbers in Bitbucket to get a link direct to that line.  Being able to see *both* things you're referring to would help.

I'm guessing you don't have much experience with C?  I think you might be getting confused with the different ways you deal with a struct and an array of structs?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 04, 2019, 10:14:20 AM
Quote from: names_are_hard on December 03, 2019, 08:55:00 PM
I'm guessing you don't have much experience with C?

How did you possibly guess that?  ;)

Yep, I'm a C-noob. I'm coming at this project from a 'visual' angle. Having seen visual benefits to this iso research I'm seeking to skill-up to be able to hard code the tweaks I'm able to do in iso_regs or adtg_gui and add them to a crop_rec (3K, UHD etc) preset.

At present if you're running iso_regs or adtg_gui, they won't allow for crop_rec extended resolutions to work at the same time.

Thanks again for your help! :) What camera are you running btw?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 04, 2019, 03:40:41 PM
Can someone clever help out with what I think is a maths problem please?  :o

I'm taking a single register value set in iso_regs.mo which is

ADTG Preamp = 2

and trying to duplicate it exactly in adtg_gui.mo. However, the corresponding preamp registers in adtg_gui are split up into several values that seem to be: blue and green channel 1, and red and green channel 2

Below are the default values:


/* ADTG2 adjusts blue and green channel 1 ? */
ADTG2[8] = 96
ADTG2[9] = 97
ADTG2[a] = 98
ADTG2[b] = 96

/* ADTG4 adjusts red and green channel 2 ? */
ADTG4[8] = 99
ADTG4[9] = 101
ADTG4[a] = 99
ADTG4[b] = 99


If I set all these values at an identical value e.g. = 2 to match the value in iso_regs I get the desired iso gain reduction but with strange yellow blotches / artefacts, and it does something weird to the demosaicing. I have spent many hours of setting/shooting/card pulls trying random combinations of values or ratios between figures – but without losing the artefacts. Maybe there's a quicker way to address this? :'(

Artefacts

(https://i.ibb.co/H2tvc7g/clean-200-but-yellow-blotches.png) (https://ibb.co/H2tvc7g)

The maths I'm looking for is the formula which reduces or sums all the values in the adtg_gui lines into the correct ratios to equal the '2' of iso_regs single value!

Does this makes sense?  :-\


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 04, 2019, 05:00:02 PM
Can you link to the exact lines of code in both files?  If so I'll take a look.  This question is much more clearly asked, you're explaining your assumptions better.

It's totally normal to be super confused if you're not coming from a coding background :)  I used to teach C in university, part time, to first years, so I've seen much worse.

Oh and I have a 200D, working (very slowly now, kind of stuck) on an ML port.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 04, 2019, 05:54:59 PM
Thanks @names_are_hard

The preset lives in crop_rec.mo. I'm working from one of Danne's earlier crop_rec.c as my template: https://bitbucket.org/Dannephoto/magic-lantern/src/9b8625caa3554d09003bd8a89dee7f43d66a634e/modules/crop_rec/crop_rec.c

Line 880 is where I started from...

My own version is local only at present. I have set up a bitbucket account today, once I get my head around it I can put the code in my own repo! :)

My own version I added to from about line 880 (with your earlier help! :) )to include the following code. I think this the main chunk.


switch (crop_preset)
        {
            /* all 1:1 modes (3x, 3K, 4K...) */
            case CROP_PRESET_3X:
            case CROP_PRESET_3X_TALL:
            case CROP_PRESET_3K:
            case CROP_PRESET_UHD:
            case CROP_PRESET_4K_HFPS:
            case CROP_PRESET_FULLRES_LV:
                /* ADTG2/4[0x8000] = 5 (set in one call) */
                /* ADTG2[0x8806] = 0x6088 on 5D3 (artifacts without it) */
                adtg_new[2] = (struct adtg_new) {6, 0x8000, 5};
                if (is_5D3) {
                    /* this register is model-specific */
                    adtg_new[3] = (struct adtg_new) {2, 0x8806, 0x6088};

                    /* timebytheriver edit below for testing cleaner iso on UHD preset. These relate to ADTG2/4 Preamps 8,9,a,b and ADTG4 0XFE gain */
                   
                    /* ADTG2 adjusts blue and green channel 1? */
                    adtg_new[13] = (struct adtg_new) {2, 0x8, 2}; /* was id 23 originally */
                    adtg_new[14] = (struct adtg_new) {2, 0x9, 2};
                    adtg_new[15] = (struct adtg_new) {2, 0xA, 2};
                    adtg_new[16] = (struct adtg_new) {2, 0xB, 2};
                     /* ADTG4 adjusts red and green channel 2? */
                    adtg_new[17] = (struct adtg_new) {4, 0x8, 2};
                    adtg_new[18] = (struct adtg_new) {4, 0x9, 2};
                    adtg_new[19] = (struct adtg_new) {4, 0xA, 2}; 
                    adtg_new[20] = (struct adtg_new) {4, 0xB, 2};

                    adtg_new[21] = (struct adtg_new) {2, 0xFE, 0};

                    /* end timebytheriver edit below for testing clean iso on UHD preset */
                }
                break;



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 04, 2019, 06:20:12 PM
Looking at your line up it seems you are using "maybe" already used slots. Didn´t look too carefully. For instance:
adtg_new[13] could be used somewhere else already. I would recommend to expand from this:
/* expand this as required */
    struct adtg_new adtg_new[30] = {{0}};

to this:
/* expand this as required */
    struct adtg_new adtg_new[40] = {{0}};

And then maybe start using slots from something like this:
                    adtg_new[30] = (struct adtg_new) {2, 0x8, 2}; /* was id 23 originally */
                    adtg_new[31] = (struct adtg_new) {2, 0x9, 2};



Edit:
Also note that you can write values like follows too:
Example:
adtg_new[13] = (struct adtg_new) {6, 0x8882, 0x438};
0x438

I am also a noob here, believe me ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 04, 2019, 06:43:37 PM
I still need to see where this part comes from:
Quote
ADTG Preamp = 2

I can't understand what you're comparing against without that.  You say it's iso_regs related but I can't find it in iso_regs.c.

Once again, a disclaimer that I know basically nothing about ADTG, I'm just reading the code.  Don't trust me very much :)  So, while I can comment on correctness of your code from a C perspective, I dunno what it's going to do on a real cam.

In your added code it looks like you always want to set ADTG2 and ADTG4 to the same values.  If so, just use 6 for the mask field and only have one set of the four similar lines:
Quote
                    /* ADTG2/4 adjusts blue and green channel 1? */
                    adtg_new[13] = (struct adtg_new) {6, 0x8, 2}; /* was id 23 originally */
                    adtg_new[14] = (struct adtg_new) {6, 0x9, 2};
                    adtg_new[15] = (struct adtg_new) {6, 0xA, 2};
                    adtg_new[16] = (struct adtg_new) {6, 0xB, 2};
To understand how that works, read my comment about masks and learn about bit operations in C.  Specifically you want to learn about "bitwise and".  You'd also want to learn about hexadecimal / binary / decimal conversions to understand how flipping single bits changes the number.

You have a discrepancy between one of your comments and one of your lines of code.  You say "ADTG4 0xFE gain", but your code has "adtg_new[21] = (struct adtg_new) {2, 0xFE, 0};", which is trying to modify ADTG2 (the mask, which is the first field in the struct, is 2).  I don't know which you intended.  Perhaps this should be 6 as well?

Your code is inside the is_5D3 guard.  This implies a) you have a 5D3 (if not, your changes shouldn't do anything for you) b) the changes you are making only work on 5D3.  Point b doesn't matter right now since it's just you using it, but if you expect this to work on multiple cams you'd want to move it outside of the guard at some point (after testing, etc).

By the way, I am guessing it's four lines because in bayer there's BGGR, four channels.  Here each line sets one register (presumably 0x8, 0x9, 0xa, 0xb are offsets into some set of registers), I assume the ADTG chip later checks each register for each colour channel setting.  Each register will control some different property of the hardware. I could easily be wrong; just guesses around how embedded stuff often works.  You could test by only setting one and seeing if it affects one colour only (or half of the green pixels).  Doesn't matter to me, might be interesting for you.

To illustrate what I mean about registers within ADTG chip, these lines suggest register 0x800c controls if ADTG should skip lines in the sensor when reading it:
Quote
                /* ADTG2/4[0x800C] = 0: read every line */
                adtg_new[2] = (struct adtg_new) {6, 0x800C, 0};
That only needs one line, because you either skip lines or you don't.  For colour related stuff, since ADTG (seems to) give the ability to affect each colour channel separately, you can set four offsets / registers if you want to affect all four colour channels.  Perhaps this is where you're getting confused and what your references to 1 vs 4 mean?

The switch / case code is quite hard to read due to the funky indenting :(  I guess the file has mixed tabs and spaces :(
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 04, 2019, 06:52:22 PM
Thanks plenty @Danne & @names_are_hard Will need to digest all of this good stuff. :)

@names_are_hard ADTG Preamp = 2 is a value I set in the iso_regs front-end gui.

Refs to it in the actual code are: https://bitbucket.org/hudson/magic-lantern/src/75c1f3fe29273f1057ca932f71446d75be8937b8/modules/iso_regs/iso_regs.c?at=iso-research

Lines, 27,36,243,375,513,582,599,675,765
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 04, 2019, 07:08:42 PM
Oh, adtg_preamp!  No wonder I couldn't find "ADTG Preamp"!

That fits my theory on your 1 vs 4 confusion being to do with colour channels.  The preamp presumably applies to the whole sensor readout.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 10:20:08 AM
Quote from: Danne on December 04, 2019, 06:20:12 PM
Edit:
Also note that you can write values like follows too:
Example:
adtg_new[13] = (struct adtg_new) {6, 0x8882, 0x438};


@Danne Thanks! Does using the Hex syntax make any operational difference? I see it used both ways in the same module!  :o
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 05, 2019, 10:55:55 AM
Quote from: timbytheriver on December 05, 2019, 10:20:08 AM
@Danne Thanks! Does using the Hex syntax make any operational difference? I see it used both ways in the same module!  :o
Do tests and you´ll see what happens. That´s how I do things.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 02:16:29 PM
Have done now – must be hundreds!  :o

Hex/Dec doesn't seem to make a difference here... :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 02:23:13 PM
Quote from: names_are_hard on December 04, 2019, 06:43:37 PM

You have a discrepancy between one of your comments and one of your lines of code.  You say "ADTG4 0xFE gain", but your code has "adtg_new[21] = (struct adtg_new) {2, 0xFE, 0};", which is trying to modify ADTG2 (the mask, which is the first field in the struct, is 2). 


@names_are_hard Great catch here, thank-you! :) I had meant to target ADTG4 as per my comment!

Quote
In your added code it looks like you always want to set ADTG2 and ADTG4 to the same values. 

Not necessarily: the values in adtg_gui differ by a few points over the 8 registers. That's why I'm still wondering how iso_regs sums these same values into just a single figure. I'm sure it's important!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 05, 2019, 02:47:55 PM
Quote
Not necessarily: the values in adtg_gui differ by a few points over the 8 registers
Not in the added code that you showed :)  If they differ elsewhere, I can't see it.

Again, I don't understand your description of "sum into a single figure".  I'm going to take another guess about your meaning, but if I'm wrong it would really help me if you could describe what you mean in more detail.  Do you mean that the UI / menu for some settings takes a single number, but when you look at the code you see the setting seems to be related to 4 lines of code that each set a value?  If so, I think that is because some menu settings are intuitively "one value", but the camera hardware requires 4.  The menus choose to only show one value, and then map to the equivalent 4 values.  The menu *could* allow the user to pick each value individually, but generally this wouldn't help the user, so the choice has been made to simplify.

As an example, guessing you're familiar with RGB colour, imagine a graphical colour picker.  You click on *one* colour.  But the code needs 3 values, R, G and B.  The code translates the chosen single value to three values.

For the specific code you're interested in I guessed the registers were RGGB related, hence 4 channels.  Now I've found comments in the code referencing columns, so maybe it's not colours and the sensor has 4 individually controllable amps.  Either way, the code assigns 4 values to control each individually - but the user wants all the ISO gains the same, not for every 4th column to be brighter than the other 3 (we'll ignore dual-ISO for now...).  So for ISO settings the menu takes one value, but behind the scenes still has to set 4, because that's how the hardware works.  Try setting only 1 of the 4 and you should see what actually changes.

Again, a fair chunk of guesswork here.  Hopefully Danne can set me right on mistakes, also hopefully I saved him some typing.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 02:56:08 PM
Quote from: names_are_hard on December 05, 2019, 02:47:55 PM
Do you mean that the UI / menu for some settings takes a single number, but when you look at the code you see the setting seems to be related to 4 lines of code that each set a value?  If so, I think that is because some menu settings are intuitively "one value", but the camera hardware requires 4.  The menus choose to only show one value, and then map to the equivalent 4 values.  The menu *could* allow the user to pick each value individually, but generally this wouldn't help the user, so the choice has been made to simplify.

This is exactly what I was trying to describe, yes! :) But the settings vary between module menus: One user set value in one module; A set of values for another module menu.!

The different values for each reg/column can be seen in this screenshot of adtg_gui:

(https://i.ibb.co/BCQsNf7/IMG-3273.jpg) (https://ibb.co/BCQsNf7)

I'd like to know how all those separate / different values end up being a single value in iso_regs – and whether it even matters at all!  :P

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 05, 2019, 03:10:09 PM
We found the answer!  Yay!

I'll wait for Danne to tell you where the code lives, but I can tell you the (very likely) *reason* why some menus have 4, some have 1 for the same value.  One of the menus is for fine control, one is for ease of use.  I mean, how often do you want to change the ISO gain for one quarter of the pixels only?  But, it's Magic Lantern, so it should let you if you really really want to.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 05, 2019, 04:03:58 PM
@timbytheriver
Study iso_reg code for answers but as @names_are_hard tells you a1ex must have connected the dots for a few regs here. I would recommend that you work the adtg_gui regs as a starting point also in crop_rec.c. Please test following. Look for this place:
    /* should probably be made generic */
    if (is_5D3 || is_100D || is_EOSM || is_700D || is_650D || is_6D)
    {

Then replace with this:
    /* should probably be made generic */
    if (is_5D3 || is_100D || is_EOSM || is_700D || is_650D || is_6D)
    {
                    adtg_new[20] = (struct adtg_new) {2, 0x8, 0x60 + reg_8};
                    adtg_new[21] = (struct adtg_new) {2, 0x9, 0x61 + reg_9};
                    adtg_new[22] = (struct adtg_new) {2, 0xA, 0x62 + reg_a};
                    adtg_new[23] = (struct adtg_new) {2, 0xB, 0x60 + reg_b};
                    adtg_new[24] = (struct adtg_new) {2, 0xFE, 04 + reg_fe};

Now also add following to be able to modify above adtg values from crop_rec.c:
Find this place:
            {
                .name   = "hdr iso B",
                .priv   = &HDR_iso_b,
                .max    = 6,
                .choices = CHOICES("OFF", "iso100", "iso200", "iso400", "iso800", "iso1600", "iso3200"),
                .help   =  "HDR workaround eosm",
                .advanced = 1,
            },

Replace with:
            {
                .name   = "hdr iso B",
                .priv   = &HDR_iso_b,
                .max    = 6,
                .choices = CHOICES("OFF", "iso100", "iso200", "iso400", "iso800", "iso1600", "iso3200"),
                .help   =  "HDR workaround eosm",
                .advanced = 1,
            },
            {
                .name   = "reg_8",
                .priv   = &reg_8,
                .min    = -2000,
                .max    = 2000,
                .unit   = UNIT_DEC,
                .help  = "reg 8",
                .advanced = 1,
            },
            {
                .name   = "reg_9",
                .priv   = &reg_9,
                .min    = -2000,
                .max    = 2000,
                .unit   = UNIT_DEC,
                .help  = "reg 9",
                .advanced = 1,
            },
            {
                .name   = "reg_a",
                .priv   = &reg_a,
                .min    = -2000,
                .max    = 2000,
                .unit   = UNIT_DEC,
                .help  = "reg a",
                .advanced = 1,
            },
            {
                .name   = "reg_b",
                .priv   = &reg_b,
                .min    = -2000,
                .max    = 2000,
                .unit   = UNIT_DEC,
                .help  = "reg b",
                .advanced = 1,
            },
            {
                .name   = "reg_fe",
                .priv   = &reg_fe,
                .min    = -2000,
                .max    = 2000,
                .unit   = UNIT_DEC,
                .help  = "reg fe",
                .advanced = 1,
            },

Lastly look for this:
static int isopatchoff = 1;
//static int preset1 = 1;
//static int preset2 = 1;
//static int preset3 = 1;

Replace with:
static int isopatchoff = 1;
//static int preset1 = 1;
//static int preset2 = 1;
//static int preset3 = 1;
static int32_t  reg_8 = 0;
static int32_t  reg_9 = 0;
static int32_t  reg_a = 0;
static int32_t  reg_b = 0;
static int32_t  reg_fe = 0;


Or just download my changes here and compile:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/crop_rec.c

Now you should be able to modify above regs from with crop mode sub menu. They´re called reg_8, reg_9 and so on. If you want to add more regs just try to follow my code and add more.
Don´t forget. If you hit this limit:
    /* expand this as required */
    struct adtg_new adtg_new[30] = {{0}};

Keep expand it.

EDIT:
Now the starting point will apply on all presets. So if numbers are ok as starting point I expect them to work otherwise not.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 05:01:57 PM
Wow @Danne! Christmas just came early. Many thanks!  :D

I am working from an old build of yours Nov2018 iso-experiments. Should I continue with that build – or update to your latest, and use that cro_rec.c ?

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 05, 2019, 05:04:54 PM
I think this is an old one. But do test it. Or manually add the regs.

I also note a few errors in my manual pastes so double check against crop_rec.c file.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 05:08:56 PM
Sure thing. Thanks again!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 06:40:00 PM
@Danne

Not working here yet!

I've cloned your most recent crop_rec_4k_mlv_snd_isogain_1x3_presets build and edited the lines you said. I've checked my edits with your crop_rec.c download. They are the same.

The Q menu appears with all the options settable, but the pre-amps are not being adjusted. I can tell this instantly because the Histogram allows for 'OVER'. With the regs working the histo should never be able to go into 'OVER'.

Also the _fe reg needs to allow for a value of 0 to be switched on. I can't select 0 with the present code.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 05, 2019, 06:50:17 PM
Yes, it usually takes a year or more to achieve something significant in here ;).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 06:54:13 PM
? :o

EDIT:
Can't see any cut/paste errors! Have compared line by line with your download crop_rec.c
No compile errors either! ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 05, 2019, 07:19:02 PM
Danne said:
Quote
Now you should be able to modify above regs from with crop mode sub menu.

Sounds like it does that?  Just not every thing you wanted?  I'm pretty sure it was intended as a starting point for you to debug and learn from.  It's good that it compiles and changes the menus, that's a strong start and means you can make changes knowing whether or not you broke things or improved things.

If you want some more general instructions in learning how to write and/or debug C, I'm happy to help, but we should move it out of this thread, too off-topic.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 05, 2019, 07:23:36 PM
Yes, starting point. That's how I ironed out most eosm presets by porting regs into crop_rec.c. Of course this only works because a1ex layed out the complete groundwork before.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 05, 2019, 07:31:43 PM
Thank-you gents for the ever-so-slight rap across the knuckles!  :) I shall persevere.

@names_are_hard C-advise would be welcome. Thanks! :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 06, 2019, 11:40:49 AM
Just about to ask a specific code question when I noticed that all Q menu > advanced, register changes are having no effect on the camera!

So maybe something more generic with the Q menu operation is at fault?  :o

I tried setting random values on:
8
9
a
b
fe

AND

bit depth
ratios
vertical offset
horiz offest
alter height
alter width
et al...

None of these changes are affecting the camera. Is there some special flag that needs to be set to allow for crop_rec Q menu > advanced to be made active? Maybe I'm doing some schoolboy error?

I'm running Danne's most recent crop_rec_4k_mlv_snd_isogain_1x3_presets on 5D3 1.1.3

Any help pls?

Many thanks.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 06, 2019, 12:30:36 PM
You need to enable a preset from crop rec of course.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 06, 2019, 12:36:36 PM
Err, yes! ;)

I have. did! UHD 1:1 > Q > Advanced.

BTW
If I enable the Terminal readout, I see no changes there when I try to alter Q menu settings. Should there be?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 06, 2019, 02:29:32 PM
Sorry. I'm out. No more time. I suggest you start a new thread about your progress and most important. Keep trying and then try harder to solve basic issues.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 06, 2019, 03:17:18 PM
Righto. I'm not sure my research is anything other than an extension of the work outlined in this thread, so unless I'm instructed otherwise by a grown-up, I'll keep my posts here.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: histor on December 06, 2019, 08:27:57 PM
I promised here this summer  (https://www.magiclantern.fm/forum/index.php?topic=10111.msg217380#msg217380) to provide better statistics on SNR while using reduced gains on 5D2.
A set of cloudy sky flats repeated twice. (Notice also, how much SNR is lost after 6 consequent shots, as the sensor becomes 7 degrees hotter). Numbers calculated with a fast PixInight script.


SNR = 26.42 db, K IMG_7671iso100_DFE111
SNR = 21.04 db, K IMG_7672iso200_DFE111
SNR = 16.76 db, K IMG_7673iso400_DFE111
SNR = 15.92 db, K IMG_7674iso100
SNR = 13.52 db, K IMG_7675iso200
SNR = 12.07 db, K IMG_7676iso400

SNR = 23.85 db, K IMG_7677iso100_DFE111
SNR = 19.55 db, K IMG_7678iso200_DFE111
SNR = 15.68 db, K IMG_7679iso400_DFE111
SNR = 15.27 db, K IMG_7680iso100
SNR = 12.92 db, K IMG_7681iso200
SNR = 11.63 db, K IMG_7682iso400


RawDigger produces a bit different results (and I cannot explain why), but the leader stays the same.
You can download the samples (https://app.box.com/s/3vv1ln1ugw67a36pvjm5yluu1fw8u99f) and check my math.

So it works, as expected. The only problem now: it's damn cold to tweak registers outdoors  ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 10, 2019, 11:06:00 AM
Thanks @histor for sharing the encouraging test results! Every little advancement helps. :D

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 10, 2019, 11:47:23 AM
Those users interested in shooting for the cleanest shadows in video might like my new test build for 5D3 1.1.3 only (at present) which includes a crop_rec 'UHD 1:1 Cleaner-ISO' preset. When selected, this makes changes to the iso registers, and yields cleaner, less-noisy shadows – whilst retaining dynamic range – in stock Canon ISOs of 160-800. As video shooters know: even the slightest clean-up in noise can mean the difference between a usable or unusable shot.

You can see this preset in use in @70MM13's 'DARK music video' here https://www.magiclantern.fm/forum/index.php?topic=24690.msg223329;topicseen#msg223329

These register tweaks are based on tools and data contributed to this thread by: @a1ex, @Danne, @Audionut, @michael84, and others.

My hope is that we can maybe move these presets into the Q sub-menu of the crop_rec preset (see a few posts back). But until that happens, here are the results so far.

Disclaimer: I am noob-developer! What I have picked up so far is from reading, and re-reading mainly this thread – and with help along the way from y'all. :) The usual risks apply if you want to test this code, but so far my own 5D3 is in fine shape!

Download is here: https://bitbucket.org/rivertim/magic-lantern/downloads/ (https://bitbucket.org/rivertim/magic-lantern/downloads/)(5D3 1.1.3 only)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 11, 2019, 12:21:30 AM
Congrats on getting it working, Tim!  Good persistance! :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 06:55:35 AM
Nice @timbytheriver  :D.
Trying to get into the source code but seems your repo is password protected. Could you include your changes in your code please.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 11, 2019, 09:44:19 AM
Thanks.  :)

Quoteseems your repo is password protected

Really? If I log out and try to access Source > Modules > foo.c I can look at the raw code. I thought I'd already committed my changes... will check.

EDIT
Sourcetree reports 'Nothing to Commit'. The bitbucket repo has my updates in crop_rec.c
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 10:32:33 AM
Ok, I think I found your code now.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 11:09:01 AM
I tested iso 200 with the new tweaks and then with a build at iso 200 without the tweaks. Note. Only very briefly and the test scene wasn´t even static but settings were exactly the same:

Iso 200 no modification(comes out brighter)
(https://i.postimg.cc/8chyJ75m/Screenshot-2019-12-11-at-11-00-14-png-300px.png)

Iso 200 modified(comes out darker)
(https://i.postimg.cc/1XxC3G9z/Screenshot-2019-12-11-at-11-00-09-png-300px.png)

iso 200 unmodified
(https://i.postimg.cc/wjFGQRkW/Screenshot-2019-12-11-at-11-00-51-png-300px.png)

iso 200 posterization in modified(more testing needs to be performed)?
(https://i.postimg.cc/ZKSskx77/Screenshot-2019-12-11-at-11-00-32-png-300px.png)

Ok, so I have no interest in proofing anything bad here. On the contrary I really want this to work. I would suggest someone to do following test.
1 - Record at iso 200 with the exact same settings with modified and unmodified version. Even if brightness will vary.
2 - Match brightness in post and compare shadows for refinements or not.
3 - Maybe try other isos as well and do above test

Recording scene should be chosen with shadows in mind.
If we could take the next step by doing these tests we will immidieately understand if the registry changed are the correct ones etc.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 11, 2019, 11:49:35 AM
Let's remind ourselves of @a1ex's recent apropos post here first! https://www.magiclantern.fm/forum/index.php?topic=24601.msg221994#msg221994  :)

Here are my comparisons. Black Strength lowered to 0 to bring up shadows in all. Strongly suggest looking at full-res (Load Full Resolution) on the upload site.

1) Stock ISO 200
(https://i.ibb.co/2dNQF83/200-stock.png) (https://ibb.co/0rtSnBy)

2) Mod ISO 200 – Brightness matched in MLVApp
(https://i.ibb.co/JQshGTV/200-mod-brightness-matched.png) (https://ibb.co/16mSPN1)

3) Mod ISO 200 – Exp raised in camera to compensate for reduced gain ceiling (spotmeter now maxes-out at 95% on the hole of the lampshade)
(https://i.ibb.co/HTqbv7R/200-mod-exp-tweaked.png) (https://ibb.co/chQZ5JR)

I know which combination of settings I'd prefer grading from. :)

PS @Danne I'm not seeing posterization [yet]!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 11:54:47 AM
Not at my computer atm but how are the first two files when compared? Identical? Will take a look in a while.

If you increase brightness like in third example we again are not comparing in a good way imo.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 11, 2019, 12:00:30 PM
I'd say that the mod changes the quality of the noise for the better in 2) (matched brightness)

If the exposure were simply brightened without the mod, the highlights would blow. So with the mod + recorrected exp ceiling we retain the complete dynamic range of stock ISO  – but with marked improvement in shadow noise.

Maybe we should be thinking of this as a way of exposing rather than just settings. ?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 12:11:01 PM
That's the other aspect. Modifying exposure behaviour as a user. Wouldn't it only act as in cam programmed ettr compensation? I mean. It's of course positive to be able and expose "normally" but let's keep the information real.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 11, 2019, 12:43:56 PM
I hear you, but I'm not necessarily suggesting modifying ML ISO behaviour across the board at a deep system level. I'd be happy to see this as a "let's turn on the 'clean-iso preset' and ETTR to the new exposure ceiling because this shot would really benefit from cleaner shadows..."

That's why I'd prefer to see it working as Q-sub-menu choice as opposed to a hard-coded preset like I have at present. Then at least those who wanted to use it could do so, and others could happily ignore it. :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on December 11, 2019, 12:48:20 PM
Sure. Just go ahead. You came much longer than I ever did in this subject.
Looking forward to the continuation.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 11, 2019, 01:34:30 PM
Now might be a good time to split my objective out into a new thread:

Cleaner ISO Preset https://www.magiclantern.fm/forum/index.php?topic=24702.msg223374;topicseen#msg223374 :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: 70MM13 on December 18, 2019, 08:38:56 PM
here's some graphs for "tnt" iso 120 (1/50s) vs canon iso 200(1/60s)...

i'm not certain that the SNR graph for 120 is correct because it was generated immediately after changing modes, so the amplifiers were possibly "unsettled".

a delay of 30 seconds may be needed in the diag module to ensure correct readings!

the black level dynamic range measurement is definitely correct, however.  i could see on the camera display that there were no issues with stripes or green colours during that phase of the task.

any insights would be appreciated!

(https://i.ibb.co/BGSZkK2/ob-dr-120.jpg) (https://ibb.co/BGSZkK2) (https://i.ibb.co/N2k1N4N/ob-dr-200.jpg) (https://ibb.co/N2k1N4N) (https://i.ibb.co/HqDjCpm/snr-120.jpg) (https://ibb.co/HqDjCpm) (https://i.ibb.co/GFG7J5j/snr-200.jpg) (https://ibb.co/GFG7J5j)

never mind, the two step graph worked...


(https://i.ibb.co/VqmMM6R/noise-200-tnt.png) (https://ibb.co/1m6RRc1)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 20, 2019, 11:14:53 AM
Hi

In several different 5D3 builds incl: my own, danne's, a1ex's (with no reg changes!) I'm having big problems with taking the second picture of the SNR 2 shots in raw_diag module. 9/10 I am getting 'BUSY' displayed, and the camera locks/needs batt pull. I get no second picture saved.

I notice also that when the camera is sitting there OFF, the red light pulses every few seconds... This doesn't happen consistently though. :( Something odd going on...


Could someone offer some help please?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 20, 2019, 05:19:49 PM
I have uninstalled ML completely. Restored Canon FW. Reinstalled ML.

Things are working again in basic crop_rec mode, however I can't get raw_diag.mo to show the small liveview reporting window (EV etc).

Is there a definitive version of raw_diag.mo I should be using?

Also, there is a message in Debug > (Red dot) Memory Patches - 1 ROM,2 RAM and a message in yellow text at the bottom: "Cache is locked down (not exactly clean).

Is this telling me something's wrong?


Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on December 21, 2019, 07:56:27 AM
Well... did anyone say these things are ready for the grand public? :D

It's certainly possible that some things were broken during "recent" commits; you may very well be the first person trying these things in several years. If in doubt, you could go back in history (older commits on the iso-research branch) and see if anything worked better in the past, then use "hg bisect" to identify where things went wrong. Caveat: there was no crop_rec back then.

Confirmed this particular bug on 5D3 1.1.3 with latest iso-research build. In LiveView it appears to work, as long as Image Review is "finite" in Canon menu (e.g. 2 seconds, rather than Hold).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 21, 2019, 09:03:10 AM
@a1ex Thanks!
Quoteready for the grand public? :D
I consider myself part of the great unwashed! ;)

Nice catch. The diag box does now appear! :) But to operate the 2-shot it has to be turned back to 'Hold' – whereupon my camera still goes 'Busy' and locks up. Will research "hg bisect".

Cheers!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on December 21, 2019, 11:15:21 AM
It seems to be some memory management issue; that's the usual cause for the BUSY message. For some reason, there is memory allocated from the SRM buffer, that has to be freed before the camera is able to take another picture (that's a limitation from Canon firmware).

Here's what I did to narrow it down:

1) some changes in mem.c to log the activity of "large" memory allocators (shoot_malloc and srm_malloc, or anything else other than the general-purpose ones):


diff -r 75c1f3fe2927 src/mem.c
--- a/src/mem.c
+++ b/src/mem.c
@@ -30,2 +30,3 @@
#endif
+#define exm_printf(fmt,...) { if (allocator_index > 2) printf(fmt, ## __VA_ARGS__); }

@@ -811,3 +812,4 @@

-        dbg_printf("using %s (%d blocks, %x free, %x max region)\n",
+        exm_printf("alloc(%s) from %s:%d task %s\n", format_memory_size_and_flags(size, flags), file, line, current_task->name);
+        exm_printf("using %s (%d blocks, %x free, %x max region)\n",
             allocators[allocator_index].name,
@@ -875,3 +877,3 @@

-    dbg_printf("free(%x %s) from task %s\n", buf, format_memory_size_and_flags(((struct memcheck_hdr *)ptr)->length, flags), current_task->name);
+    exm_printf("free(%x %s) from task %s\n", buf, format_memory_size_and_flags(((struct memcheck_hdr *)ptr)->length, flags), current_task->name);


You could also enable all the debug messages from there (define MEM_DEBUG in mem.c), but they are very verbose; the only way to see them is to save them to card (also enable CONSOLE_DEBUG in console.c).

2) this pointed to a 31MB block allocated from raw_diag.c at line 566 (where it allocates something named "second_buf" to copy one of the images there)

3) annotated raw_diag.c to see when this part of the code was changed, and why (hg blame modules/raw_diag/raw_diag.c | grep second_buf)

Result: this part of the code is the one from 2014. If anything, the memory backend is handling this large malloc request differently (in the past it used shoot_malloc, now it prefers SRM), so we need to either provide some hints to the memory allocator, or to tweak the heuristics in mem.c. This is not exactly well explained in comments, but the preference for shoot_malloc vs srm_malloc is controlled by MEM_TEMPORARY and MEM_SRM (which should be renamed to something more consistent, as the SRM buffer is something that has to be freed right away, but you can get away with using the "shoot" buffer for a while). The first "temporary" flag was created when we only knew about the existence of the "shoot" buffer (which shouldn't be kept allocated during the entire ML session, otherwise ERR70 with certain operations, so... that memory should be only "temporarily" allocated, i.e. freed when we are done with it), and then we discovered SRM buffers, which have even more usage restrictions (you can't keep any of these allocated if you want to take a picture with Canon routines, and you *have* to free them in exactly the order they were allocated).

So, using tmp_malloc in raw_diag.c for that large buffer, fixes the issue, but these wrappers should really be renamed to something that makes sense. Maybe @names_are_hard has a better idea here :)




Edit: new build on the experiments page (commit f7947b6 (http://bitbucket.org/hudson/magic-lantern/commits/f7947b627e33767708b6b59f303c8161e461a495)).

Now, the question is whether to trust these numbers; the biggest issues are, in my opinion:
1) white level should be manually checked for correctness
2) sensor response curve should be evaluated (there are tweaks that affect the response curve, from what I could tell, and the current code assumes perfect linearity)

A lower-hanging fruit might be automating the whole procedure (e.g. with Lua scripts) and checking its repeatability against various test scenes, settings etc, but definitely time-consuming. In theory, this should give correct results on any out-of-focus scene covering all tone ranges from deep blacks to overexposure (detailed requirements here (https://www.magiclantern.fm/forum/index.php?topic=10111.msg117955#msg117955)).

I'm tempted to build a dark box with software-controllable lighting, similar to the one at Apertus, but I doubt it will happen during this holiday.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 21, 2019, 12:53:39 PM
@a1ex Many thanks! Very thorough explanation. Have updated to latest raw_diag.mo

It's a whole lot better – but I'm still occasionally (2 out of 10?) getting the 'Busy' on second SNR shot. Sometimes I can settle the regs with a MENU button press. Other (now rare) times the camera locks (batt pull). But it seems certainly workable with now! :) Is this relevant: Debug > (Red dot) Memory Patches - 1 ROM,2 RAM and a message in yellow text at the bottom: "Cache is locked down (not exactly clean). A Ram management issue?

QuoteNow, the question is whether to trust these numbers;

Funny you should say this... ;)

I see a whole EV DR readout difference between the small readout window say at DR 11.62 and saved chart: EV 10.88

And the histogram often tells another story... Which to trust?  :-\

@anyone I think these have come to light as the testing of my Q-iso-regs build requires the clip point to be set exactly for the benefit to be seen at its best.

(https://i.ibb.co/zSHpQ3v/IMG-3307.png) (https://ibb.co/mHFkbnW)
(https://i.ibb.co/vV50hFv/IMG-3308.png) (https://ibb.co/tQfVzGP)

My SNR charts look a bit left biased. Shouldn't the curve be to the right more – given the scene DR?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on December 21, 2019, 01:29:46 PM
The small readout window uses a very crude approximation, but it's computed quickly: log2(white - black) - log2(noise_stdev); look at the comments in the source code. The other one measures from white level until the level where SNR = 1.0 = 0 EV (as defined in ISO 15739); details here (https://www.magiclantern.fm/forum/index.php?topic=13787.0). Both depend on whether the white level is guessed correctly, and on the linearity of the sensor response.

FYI:
- I cannot see your screenshots without looking at the source code of your post (JS issues? (https://www.magiclantern.fm/forum/index.php?topic=24608.msg222167#msg222167) edit: nope, ibb.co is now blocked by uBlock Origin)
- you've got screenshots saved on the card, no need to take pictures of the camera screen ;)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 21, 2019, 02:06:49 PM
@a1ex All good stuff to know. Many thanks for sharing.  :)

QuoteI cannot see your screenshots without looking at the source code of your post (JS issues?)

How odd. I'm seeing them on my browsers (Chrome & Safari on Mac)! I used https://imgbb.com/, which usually works ok.  :-\ Anyone else not seeing my images preview?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 27, 2019, 10:15:40 AM
I'm trying to pass a default value to some registers for which I can't find any [default] values listed in adtg_gui or iso_regs.

Is there some code that says: 'set to default value' even if that is not known?

Something like this:

adtg_new[25] = (struct adtg_new) {6, 0x14, default_value};


Thanks!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 27, 2019, 09:16:13 PM
Can you give an example of what you mean?  I can't understand your question.  Can you show the existing code that does similar to what you want?  What do you expect the default value to hold in your new case?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 28, 2019, 09:53:51 AM
Hey!
Certain registers such as ADTG Preamp et al are set at Canon default values at boot.

So the above example (ADTG Preamp) loads default values of: 0x60,0x61,0x62,0x60 and 0x63,0x65,0x63,0x63 for the various columns/colour channels. These values are discoverable through the documented work of others in adtg_gui.c and iso_regs.c

There are other register for which I can't find values that load at boot i.e. 'default' values, that I wish to address. I was asking whether there is some code that says "load default value if discoverable, or set a safe value as default" – rather than have me set some crazy inappropriate value of my own!  :o

Hence my crude attempt:

adtg_new[25] = (struct adtg_new) {6, 0x14, default_value};

Does that make sense?  :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 28, 2019, 11:04:40 AM
More sense, at least :)  I might make that code look like this:


set_adtg_defaults(adtg_new, adtg_new_size);  // does whatever the "discoverable" algorithm is for all registers
adtg_new[25] = (struct adtg_new) {6, 0x14, 0xdead};  // override the default with 0xdead


If you can set to a safe value, simply set it for all registers, then override after that.  You say it's possible to discover safe values, so make the set_adtg_defaults() function do that process for each register.  If it can't find one for a register, simply skip it.

I wonder, do you mean "initial", not "default"?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 28, 2019, 11:54:50 AM
Thanks! "initial" is certainly a better way of describing my "default" yes!  :)

Since I know certain initial values (which I already assign to certain regs) it looks like your code might override all adtg_new?  – whereas I want to set "initial" to only certain regs, on a line by line basis – and then override it.

Might that be something like?:

set_adtg_defaults(adtg_new[25], adtg_new_size);  // sets whatever the "discoverable" algorithm is for only adtg_new[25] register.
adtg_new[25] = (struct adtg_new) {6, 0x14, 0xdead};  // then override the initial of adtg_new[25] with 0xdead


PS Here's how I'm doing the existing overrides which is working:
https://bitbucket.org/rivertim/magic-lantern-danneclone/commits/7fce7f696aa1ae67c4aa16248c497a612bc10761
(Not sure why that file is marked 'DRAFT', as I committed and pushed changes to that repo days ago!?)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 28, 2019, 09:14:47 PM
Initial vs default changes it significantly.  Yes, I was intending to set all registers, that makes sense with a default.  There's still enough registers that I think a function to initialise them is reasonable, but now it would only set those where good values are known.  You'd still want to pass in adtg_new directly, so that you can access all members of the array.  The logic (and the name) of the function would change.  Note that I didn't provide any code for the body of the function!  I was only trying to provide a conceptual outline.  You don't have to do this, the existing code doesn't (would be cleaner in my opinion if it did).

Going back to your original question, "initial" makes me confused again:
Quote
Is there some code that says: 'set to (initial) value' even if that is not known?

How are you expecting that to work?  If there's no known value, what can the initial value be?  You have to assign a distinct value in C when you do assignments, C doesn't have the concept of abstract or ranged values, if that's what you mean (some languages do have this).

Looking at your code, I think you have an integer overflow bug when you're setting registers using 0x400 + reg_x pattern.  Because the addresses you're assigning to are later considered to be 2 bytes wide, and menu input + the constant could make this higher than uint16_t max.  This could lead to menu choices giving the user a radically different result than they expect.  You should probably clamp the value to between 0 and uint16_t max.  Also since your variable is signed, underflow will lead to sign extension and really mess your values up.

You should rename reg_8x to reg_8882 (or something reg_XXXX at least), the current name implies it's 1 byte wide which has real potential for confusion.

Then I looked at the code that does the register assigns and it's really wonky.  data_buf is uint32t*, but all assignments cast it to uint16_t*, and the values that are read from are plain int?  That's pretty confusing - it also feels like it should interact with memory alignment on ARM, but I don't know ARM well enough to be sure.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on December 29, 2019, 02:57:24 AM
It sounds like you are saying all of the register values (in adtg) have been found and hard coded (so that we know what they are and can call upon them).  That doesn't sound like something a1ex would do.  Most of the values are different for each camera and at different shooting modes for each camera.

In adtg after adjusting a value you can press set (or some other button I don't recall) to return the register to initial value (before modification).  Does that code help?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 29, 2019, 09:45:38 AM
@names_are_hard Thank you for your comprehensive answer. I will have to reflect awhile on the information to understand it fully. :)

@Audionut Thanks! I will explore the 'SET' button option. I am conscious that there are thousands of registers as yet unknown! So maybe the values I am trying to find are amongst those.

**UPDATE**
Used adtg_gui to spy registers: > 'Changed since now' and 'ISO related'
If anyone is looking for these values they report as follows:


ADTG24[7] = 0x0
ADTG24[8] = 0x5a
ADTG24[9] = 0x5a
ADTG24[a] = 0x5a
ADTG24[b] = 0x5a
ADTG24[1b] = 0x320


They've no doubt been discussed before – but possibly way back in the thread.

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on December 29, 2019, 01:59:40 PM
In adtg_gui.c there are no hardcoded values for registers.  adtg_gui.c is looking up the values (as currently set by Canon) and displaying them to the user.  From what I can gather, however adtg_gui does that is your answer.

iso_regs is an old attempt to bring register tweaking to a more user friendly level.  It's 5D3 only and contained hard coded values because it was useful for the specific task.  Those values were determined from looking at what adtg_gui displayed.

Quote from: timbytheriver on December 29, 2019, 09:45:38 AM
They've no doubt been discussed before – but possibly way back in the thread.

You haven't read the entire thread.....shame.  :P

Might find something useful here too: https://www.magiclantern.fm/forum/index.php?topic=6751.0
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 29, 2019, 03:16:18 PM
Thanks! More reading! lol :)

For the record: I have indeed read this entire thread through – twice. It just takes a long time to filter through me! :)

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on December 29, 2019, 03:49:08 PM
Quote from: names_are_hard on December 28, 2019, 09:14:47 PM

Looking at your code, I think you have an integer overflow bug when you're setting registers using 0x400 + reg_x pattern.  Because the addresses you're assigning to are later considered to be 2 bytes wide, and menu input + the constant could make this higher than uint16_t max.  This could lead to menu choices giving the user a radically different result than they expect.  You should probably clamp the value to between 0 and uint16_t max.  Also since your variable is signed, underflow will lead to sign extension and really mess your values up.


@names_are_hard So far, with the code as-is I'm seeing the kinds of changes I'd expect to the regs targeted. With the integer overflow bug, are we talking values that the user thinks they're inputing at 10 are actually [say] 100? Is that the sort of discrepancy you outline?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on December 29, 2019, 11:58:28 PM
Tim - read around "integer overflow" if you want to understand the problem, it's too off topic to go into here.  The short version is if you do user_value + 0x401 and the user puts in 0xfc00, the result is not 0x10001 as you'd expect, but 0x1.  I guess there's some risk when assigning unexpected values to control registers.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on January 04, 2020, 12:40:53 PM
QuoteAudionut on March 02, 2014, 03:20:11 PM
Does iso_regs scale [8,9,A,B], keeping the deltas between them?

Quotea1ex
Yes. It considers the first one as reference (that one will get the value from menu) and all the others as delta (so, when you set it to 0 or 1 or maybe even 2 or 3, it may underflow and cause strong banding). I preferred this underflow instead of clamping to 0 because in this way you see when you went too far. However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

I'm reviewing whether the decimal adjustments in my Q iso-regs menus are using the correct math. Line 1205 > https://bitbucket.org/rivertim/magic-lantern-danneclone/src/ddebb19831d7ce7472bb80db9926bc69b3bc2d6d/modules/crop_rec/crop_rec.c

Is the "deltas between them" described by @a1ex and @Audionut in this context meaning to keep the integer relationship constant? e.g.

Say for: ADTG2 8,9,a,b values = 0x60,0x61,0x62,0x60 meaning x, +1, +1, -2 Do the relationships scale as they change?

Sorry for math duh-ness!  ???

**EDIT**
Quotea1ex
However, ADTG gains 888x are not scaled in iso_regs (these are simply set all of them to the same value).

I notice in adtg_gui my values for these (888x) registers are different:


ADTG2[8882] = 0x436
ADTG2[8884] = 0x438
ADTG2[8886] = 0x439
ADTG2[8888] = 0x43a

ADTG4[8882] = 0x437
ADTG4[8884] = 0x437
ADTG4[8886] = 0x437
ADTG4[8888] = 0x43a



Should they in fact all be set to the same value?

Thanks!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on January 04, 2020, 09:17:01 PM
Short answer: everything from this thread needs to be taken with a grain of salt (as none of us is expert in sensor calibration), so it's probably best to verify each hypothesis from scratch.

The 5D3 reads out 8 columns at a time (that is, 8 analog circuits operating in parallel), and the 888x gains (4x2 = 8 values) are for each group of columns. So, one gain affects columns 1, 9, 17 etc, another gain affects columns 2, 10, 18 etc, and so on. The 8,9,a,b are IIRC similar. These values should be tweaked so all these 8 groups of columns have consistent output; otherwise, you get vertical stripes / banding (https://www.magiclantern.fm/forum/index.php?topic=17795.0). Other cameras may read out a different number of columns at a time (usually 4 or 2), so you'll see fewer registers there.

The 888x registers are linear gains (i.e. if you set them to 1000 vs 2000, that's a difference of 1 EV), so it's probably best to scale the factory values (camera-specific) by the same constant. The 8/9/A/B gains are adjusted in fractions of EV (about 0.0059 EV/step? (https://bitbucket.org/hudson/magic-lantern/src/f7947b627e33767708b6b59f303c8161e461a495/modules/iso_regs/iso_regs.c#lines-374)), so it's probably best to keep the deltas constant (https://www.magiclantern.fm/forum/index.php?topic=10111.msg103682#msg103682).

Now, since vertical stripes also appear at default configurations at least on 5D3, it's probably best to re-calibrate all these gains from scratch somehow. Related work (https://www.magiclantern.fm/forum/index.php?topic=23086).
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on January 05, 2020, 09:51:18 AM
@a1ex Many thanks for clarification and links.

Quote from: a1ex on January 04, 2020, 09:17:01 PM
These values should be tweaked so all these 8 groups of columns have consistent output; otherwise, you get vertical stripes / banding...

I've found this also true of the ADTG2/4 0xFE register. Uneven values between the banks give a similar result.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: timbytheriver on January 12, 2020, 05:43:31 PM
I'm trying to target Digital Gain regs with an adjustment. I find this in iso_regs:


if (reg == 0xc0f37ae4 || reg == 0xc0f37af0 || reg == 0xc0f37afc || reg == 0xc0f37b08)
        {
            default_digital_gain = val;
            if (digital_gain) *copy_ptr = digital_gain;
        }


I've tried:


adtg_new[40] = (struct adtg_new) {6, 0xc0f37ae4, 0x200 + gain_dgain};
adtg_new[41] = (struct adtg_new) {6, 0xc0f37af0, 0x200 + gain_dgain};
adtg_new[42] = (struct adtg_new) {6, 0xc0f37afc, 0x200 + gain_dgain};
adtg_new[43] = (struct adtg_new) {6, 0xc0f37b08, 0x200 + gain_dgain};


or

adtg_new[44] = (struct adtg_new) {6, 0x8030, 0x200 + gain_dgain};

Do any of these reg addresses look correct?

Cheers!

Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on May 04, 2020, 09:24:07 PM
Was messing with ADTG register override module and found some interesting registers for video mode (Not sure if these were mentioned before  ???).
2 registers bump the gain, without dynamic range penalty!
Edit: False alarm, the registers seemed to drop frames per second, from 25 to about 8 fps, so while ML was reporting 25fps based on timer registers, the recordings were in 8fps, so more light hitting the sensor...
combined they give about 1.66 stop of extra light, without sacrificing dynamic range, works on all iso's.
Not sure what the trickery is done in the background, but rolling shutter becomes huge with these 2 registers override.

(https://live.staticflickr.com/65535/49856218182_467ca1a7d2_o.png)

Now comes the weirdness, the registers only become visible in ADTG register override module, after a photo or video is taken.
To further investigate, I made a crop_rec preset with these two registers overriden, but they don't get activated the first time.
Crop_rec preset is loaded, nothing happens, record one video and after that the settings kick in, why doesn't it work the first time without having to record a video first ?

Also tried to take photos with these settings, but photos are messed up, purple or green color, sometimes even messed up frame, timing issues or something.

I don't know how values look on other cams, but I lowered the most left digit of these values until live view becomes brighter.

(https://live.staticflickr.com/65535/49856088831_972684b6aa_o.png)

Side by side comparison, same ISO, same Shuttertime and same Diaphragm Aperture  ;). Left with ADTG2 registers 23 and 24 overrides, on the right, basic values (with 1.66 exposure correction in post).

Here are the two dng frames on google drive:
https://drive.google.com/open?id=1mwtcX7FVyrJTth-Xjqpr0-cdIJxJj9XE (https://drive.google.com/open?id=1mwtcX7FVyrJTth-Xjqpr0-cdIJxJj9XE)
https://drive.google.com/open?id=11MEW5mOHFBXrBlrssz7uTj0FJx5-Xzj6 (https://drive.google.com/open?id=11MEW5mOHFBXrBlrssz7uTj0FJx5-Xzj6)




Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: a1ex on May 05, 2020, 04:12:20 PM
Quote from: Levas on May 04, 2020, 09:24:07 PM
2 registers bump the gain, without dynamic range penalty!
combined they give about 1.66 stop of extra light, without sacrificing dynamic range, works on all iso's.
Not sure what the trickery is done in the background, but rolling shutter becomes huge with these 2 registers override.

Confirmed on 5D3 - really interesting. It appears to reduce the frame rate as well.

ADTG[23]: default 0xF00, setting to 0x200 or some other magic values will cut the frame rate in half*, doubling the exposure time.
ADTG[24]: default 0x2d1e, setting to 0x1d1e will cut the frame rate in half*, doubling the exposure time.
* changing both registers together will result in 1/3 of the original frame rate (e.g. 24 -> 8 fps), and 3x exposure time.

This could explain the results, as the dynamic range does not change significantly with shutter speed.

In addition, changing any of those 2 registers in photo mode appears to have no effect (possibly because of the mechanical shutter), but changing both of them together results in garbled image (possibly timing issues because of slower readout). With FRSP, changing any or both of those 2 registers results in garbled image (?!)

Definitely worth a closer look, in particular at higher ISOs sorry, false alarm :)
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Danne on May 05, 2020, 04:15:19 PM
Tried some on eosm but couldn't get these results.
Reducing framerate, hmm...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on May 05, 2020, 05:51:39 PM
Quote from: a1ex on May 05, 2020, 04:12:20 PM
changing both registers together will result in 1/3 of the original frame rate (e.g. 24 -> 8 fps), and 3x exposure time.

You're right, only checked individual frames yesterday, didn't process them to a video file.
Now I did test some more and processed to a video file, and it's indeed about 3 times slower fps...
That explains why the exposure is so much better...damn was hoping to have found some magic register which could gave me ultimate low light performance on iso6400  :P

But it's interesting to know that there are other registers then A-timer and B-timer that influence fps...now if it only could speed up instead of slow down fps...
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on May 06, 2020, 02:18:25 AM
Quote from: a1ex on May 05, 2020, 04:12:20 PM
ADTG[24]: default 0x2d1e, setting to 0x1d1e will cut the frame rate in half*, doubling the exposure time.

The obvious first test for increasing frame rate would be ADTG[24] = 0x4d1e.  But don't blame me if it makes your cam explode :)  0x3d1e would be my second choice.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Levas on May 21, 2020, 02:49:24 PM
Obviously tried, but didn't work, couldn't increas fps speed with these two registers  :P
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Icaab on March 07, 2021, 09:11:19 PM
Hello. Has anyone tried this on camera 650d?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: ro xl on March 16, 2021, 05:58:05 PM
How can i find the readme for this module? I'm trying to use it on my 550D to inspect noise values at diffrent settings, and i get weird values like -2.11 eV read noise, 278051 e full well capacity, or 1006e read noise. How am i supposed to use it, there in no guide and in camera hints aren't very useful. I can't access the readme because bitbucket isn't working. Any ideas?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Audionut on March 17, 2021, 09:26:48 AM
The readme's are here (https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/iso-research/modules), but they won't give you what you want.

You're going to need to read the recommendations in the OP, and ask questions after.

Quote from: a1ex on January 10, 2014, 12:11:01 PM
Any recommended reading?


http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html (main site: http://www.clarkvision.com/articles/ )
http://www.dxomark.com/About/In-depth-measurements/Measurements/ISO-sensitivity

Happy reading.  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dream951 on July 01, 2021, 11:19:25 AM
Hi guys!
Is it possible in any build to take pictures from liveview with isoregs with an intervalometer.
This does not work for me, it turns out only a raw frame without processing by isoregs in build crop_rec_4k.2018Jul22 and I get only one frame at a resolution of 1920x1248 with isoregs, but then liveview resets the effect and all subsequent frames I get with the iso I chose for setting isoregs in build crop_rec_4k_mlv_snd_isogain_1x3_presets_2021Apr01.
It would be very cool for time lapses.
Thanks
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: dream951 on July 06, 2021, 06:21:03 PM
Quote from: ghost0cnc on August 21, 2018, 11:28:19 PM
Yesterday I encountered serious vertical fixed-pattern banding on some images. On other images there was no banding at all.

5D Mark III 1.2.3 + ML crop_rec_4k.2018Mar10.5D3123
Canon EF80-200mm f/2.8L (very old lens)

Same exposure settings and iso_regs configuration on all images, iso_regs modified to enable the "CMOS-Patch".
AI Servo, 1/1000s, f/2.8, Canon ISO 3200, tweaked with the following overrides:

CMOS[3] = 2372 (0x944)
CMOS[4] = 792 (0x318)

ADTG-0xFE = 4
ADTG-Preamp (8/9/A/B) = 2 (2 is the lowest value that does not cause banding due to an underflow of one of the other parameters)
ADTG-Gain (8888) = 950
SaturateOffset = 1200 (caused some green tint issues on a few older images)



Top 800px of each image at +2EV and the same white-balance applied, images 1 + 2 with banding, 3 without:
(https://milanstephan.de/ext/ml/5d3_banding_top_small.jpg) (https://milanstephan.de/ext/ml/5d3_banding_top.jpg)

Top "optical black" bars of the three images (https://milanstephan.de/ext/ml/5d3_banding_ob.png) (dcraw -T -E -W -b 1 -g 50 500)
The fixed-pattern noise is pretty visible. The last ob-image (without banding) is far brighter - if these dcraw settings are not useful for a comparison, please tell me better settings and I will update the image.

Download the RAWs (90 MB, zip) (https://milanstephan.de/ext/ml/5d3_banding_raws.zip)



Do you have an idea what could have caused the fixed-pattern banding?

Brainstorming:
- Bad iso_regs settings
  -> critical value for SaturateOffset (?), but this should change the tint
  -> variations of ADTG-Preamp, if the offsets change and I fix the first vaule to 2, it could possibly cause an underflow of another value

- Electromagnetic interferences from the lens
  -> would cause horizontal banding (?)
  -> no fixed pattern

- Mobile-phone interferences
  -> also horizontal (?), non-fixed-pattern noise

- Voltage fluctuations due to AF-motor power-consumption

- Aliens



In this case a normal dark-frame subtraction will not work, as the intensity of the FPN exceeds the amount of FPN I could get from a "normal" dark-frame.

As the approach of correcting banding using data from the optical-black bars was mentioned earlier in this thread (https://www.magiclantern.fm/forum/index.php?topic=10111.msg105075#msg105075) - is there a banding correction tool for CR2 files available yet?
If not, I will have a look at it. Just need to get into RAW de- and encoding.

Hi, how you modify isoregs to enable cmos patch? can you give me link for this module please
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: JogaDann on December 16, 2021, 05:51:55 PM
Hi! I have a 5D2 and the ISO experimental build have an annoying issue.;

It just stucks the camera at a blue + red LED on when I turn it off. When I delete ML files and put another version (eg. the lastest Lua Dec/2020) the problem is gone.

But I need the ISO experiments branch to run a module (hdmi_out.mo), so I really have to solve this. Is there a workarount do deal with it?

Thanks in advance  ;D
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: lea84 on April 12, 2022, 06:01:56 PM
I have the same problem with 5D II version 2.1.2 and the iso-research version. After I install "iso-research" Latest Build (2019-12-21 10:24) version on Canon 5D ii and restart the camera, the screen stays black only the red LED light is on. I have to disconnect power so that the camera goes off again. Does anyone have a solution for this?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Markelian on July 20, 2023, 06:43:41 PM
Can confirm as well, the 5D2 build is bricked, you get both leds on (red and blue) indefinitely when turning off, also it sometimes crashes during idle (?).
Tried to scrape the thread but red's old builds are down, and the only other build you can load these modules on is the 4k raw one, which 5d2 doesnt have, so basically this project doesnt exist for 5d2 on public internet.

Im not sure if it's that, but i think the both leds on thing might correspond to the state of the "ARM chip being locked in a loop, heating up very rapidly" that's on the wiki. Most i waited for 20s but there's another user on the forum who got the 2 leds on issue and left it a couple of minutes without taking the battery out, and now his camera works on and off (this thread https://www.magiclantern.fm/forum/index.php?topic=3358.0)
Anyway maybe delete it off the list since its not working anyway (and everyone moved on from the project i guess?) , so it doesnt destroy people's cameras in case it does represent the locked ARM processor state.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on July 21, 2023, 03:12:48 AM
I've never seen anyone call a build "bricked" before.  Hard to imagine what that would mean.

The post you reference is from 2012, with the user not using this build at all, so I don't think it's relevant.  No, having both LEDs lit doesn't mean the processor is locked up and heating rapidly.

I see no evidence that this is damaging cameras, do you have something specific?  It's explained that ML can crash and temporarily lock up your cam, and that you should remove the battery if this happens.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Walter Schulz on July 21, 2023, 03:47:18 AM
Basically it is just as Markelian wrote:
ISO_research build loads fine but fails to shutdown properly with both LEDs on indefinitely. Restart attempt requires battery pull and after that ML will tell you about shutdown issue.
Makes working with this build quite complicated.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on July 21, 2023, 07:17:19 AM
Okay, thanks for the correction!  So the symptoms are real, but the wild guesses at causes are just that :)

This may fix the problem:
https://drive.google.com/file/d/1RT2dI0VMUoN_QGSvLYhxu-wmAHpXShhk/view?usp=sharing
Works okay in qemu, I don't own 5D2 so I can't test further than that.  Try at your own risk.  It's a small change though so I'd expect it to work.

I applied this fix to iso-research branch and rebuilt:
https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/035648f634448331cc93af393ffbb70cc05daf34

Why are there so many branches? :(
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Walter Schulz on July 21, 2023, 06:56:21 PM
Looks good. Shuts down and restarts without issues.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on July 22, 2023, 12:54:11 AM
Great, thanks for testing.  Bug introduced only in lua_fix and iso-research in 2019, fixed in 2020 in lua_fix but not ported across.
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Markelian on July 23, 2023, 12:08:03 PM
Quote from: names_are_hard on July 21, 2023, 03:12:48 AM
I've never seen anyone call a build "bricked" before.  Hard to imagine what that would mean.

The post you reference is from 2012, with the user not using this build at all, so I don't think it's relevant.  No, having both LEDs lit doesn't mean the processor is locked up and heating rapidly.

I see no evidence that this is damaging cameras, do you have something specific?  It's explained that ML can crash and temporarily lock up your cam, and that you should remove the battery if this happens.

Thanks for the new build names_are_hard, and thanks Walter for confirming issue! You carry a lot of legitimacy, and with that, a lot less resistance to your words.

My apologies for using weird terminology (i understand "bricked" is a specific hardware term and it might not apply here). I was corroborating the 2 posts above my post which say the same thing, hoping if enough of us bump thread something maybe happens (even though i dislike saying that because it places expectation on others, but im not good at coding). Also sorry i should have specified i was talking about last iso-build, i said that nowhere and just assumed other will be able to guess, that was not ideal of me.. if i had to rewrite that post there's a lot i could change to make it clearer.



Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on July 23, 2023, 01:35:40 PM
No problem, phrasing was just a bit confusing :)

Did you test my fixed build?  And, why are you using this slightly weird iso-research build anyway?
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Markelian on July 24, 2023, 02:35:21 PM

Quote from: names_are_hard on July 23, 2023, 01:35:40 PM
No problem, phrasing was just a bit confusing :)

Did you test my fixed build?

Yes sir, works a treat.
Quote from: names_are_hard on July 23, 2023, 01:35:40 PMAnd, why are you using this slightly weird iso-research build anyway?
I wanted to play with the ISO registers and see if i can get any more DR out of this camera, and that was the latest build on the site (what other could i have used?) . I also tried looking for some older ones within the thread but all that i found using search seem expired (404)..
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: names_are_hard on July 24, 2023, 04:06:05 PM
Cool, works for two people, better confidence.

Sounds like you're using the build for the right purpose - we get people who end up on some random build due to strange search results or whatever, just wanted to check.  Good luck, and let us know if you find anything interesting!
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Markelian on July 24, 2023, 05:36:53 PM
Ah i see :) Will do, and thank you for the help! 
Title: Re: CMOS/ADTG/Digic register investigation on ISO
Post by: Skinny on July 24, 2023, 09:02:46 PM
Please report here if you find anything interesting :) I love 5D2