Author Topic: CMOS/ADTG/Digic register investigation on ISO  (Read 553574 times)

Greg

  • Contributor
  • Hero Member
  • *****
  • Posts: 589
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #500 on: February 26, 2014, 04:39:46 PM »
My shutter unit says, give me a break.
Another 500 images...  :P

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #501 on: February 26, 2014, 04:40:51 PM »
I forgot to unscrew the lens for the above results.

Patching iso_regs to allow adjustments up to 3148, just by checking that my code changes worked, I can see that SaturateOffset is not reducing the noise.  It fluctuates around the default Canon measured stdev.

The DR increase comes from lowering of the black level (crushing the blacks).

My shutter unit says, give me a break.
Another 500 images...  :P

Heh.  Mine too :)

Code: [Select]
# HG changeset patch
# User Audionut
# Date 1393431415 -7200
# Branch unified
# Node ID 4ab57c1f286e8b7338feb162e9998ba37ede3431
# Parent  1ca7d872bdb2f18dc5813d69db474a7f25b74951
Increase SaturateOffset adjustment range

diff -r 1ca7d872bdb2 -r 4ab57c1f286e modules/iso_regs/iso_regs.c
--- a/modules/iso_regs/iso_regs.c Wed Feb 26 06:49:12 2014 +0200
+++ b/modules/iso_regs/iso_regs.c Wed Feb 26 18:16:55 2014 +0200
@@ -513,7 +513,7 @@
                 .priv = &saturate_offset,
                 .update = saturate_offset_update,
                 .min = 0,
-                .max = 2000,
+                .max = 3148,
                 .unit = UNIT_DEC,
                 .help  = "Alters black level and stretches the range, keeping white fixed.",
                 .help2 = "Decrease to get more highlight details, but watch out RAW zebras.",

Greg

  • Contributor
  • Hero Member
  • *****
  • Posts: 589
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #502 on: February 26, 2014, 05:35:30 PM »
500D ISO 200
ADTG[7] 0x4046 -> 0x4036
c0F0[819c] 0x66f -> 0x40
c0F0[8034] 0x1991 -> 0x1f11




Vertical banding in the shadows (default vs trick). ACR +5EV

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #503 on: February 26, 2014, 05:47:37 PM »
Here's a hint about where each amplifier might be in the processing chain. I've tried to estimate how much noise is at the input of each stage and how much is introduced after it.

Theory:

In complete darkness (zero input signal), let's consider this simplified noise behavior for any amplification stage:
output = K * input_noise + output_noise

Assumming the input and output noises are not correlated and Gaussian, assumming 100% linear behavior, and also assumming we can modify the linear gain K somehow (either by turning the amplifier on/off or by adjusting its gain with an integer parameter), we can find out the two noise components from the following linear system by measuring the output noise and the value of the gains (see this):
K1^2 * var(input_noise) + var(output_noise) = var(output1)
K2^2 * var(input_noise) + var(output_noise) = var(output2)

I will start from ISO 100, consider this setting the reference configuration (gain = 1.0, or 0.0 EV), and will change one amplifier at a time from iso_regs.

Measurements and results:
Code: [Select]
                      measured gain (EV)     measured OB noise (ADU)     input noise (ADU)     output noise (ADU)
                                                                         normalized to ISO 100 noise level       
Canon ISO 100          0.00                  6.54                         NaN                   NaN 
CMOS[0] = 0x773        5.66                 26.26                        0.50                  6.52 
CMOS[0] = 0xDD3        5.73                 28.00                        0.51                  6.52 
CMOS[0] = 0xFF3        6.47                 48.44                        0.54                  6.52 
CMOS[0] = 0xCC3        4.81                 16.93                        0.56                  6.52 
CMOS[0] = 0x553        4.79                 16.76                        0.56                  6.52 
CMOS[0] = 0x443        3.96                 11.15                        0.58                  6.51 
CMOS[0] = 0xEE3        3.52                  9.33                        0.58                  6.51 
CMOS[0] = 0xBB3        3.90                 10.90                        0.59                  6.51 
CMOS[0] = 0xAA3        2.95                  8.15                        0.63                  6.51 
CMOS[0] = 0x333        2.99                  8.33                        0.65                  6.51 
CMOS[0] = 0x663        2.52                  7.65                        0.70                  6.50 
CMOS[0] = 0x993        1.93                  7.04                        0.71                  6.50 
CMOS[0] = 0x883        0.97                  6.65                        0.72                  6.50 
CMOS[0] = 0x223        1.95                  7.14                        0.77                  6.49 
CMOS[0] = 0x113        0.99                  6.70                        0.85                  6.48 
ADTG 0xFE = 3         -0.35                  5.30                        6.18                  2.14 
ADTG 0xFE = 0         -0.55                  4.70                        6.23                  2.00 
ADTG 0xFE = 7          0.31                  8.03                        6.36                  1.53 
ADTG 8/9/A/B 0        -0.36                  5.17                        6.39                  1.39 
ADTG 888x +1EV         1.00                 13.04                        6.51                  0.59 
ADTG 8/9/A/B 30       -0.17                  5.81                        6.55                  0.00 
ADTG 888x -1EV        -1.00                  3.22                        6.57                  0.00 
ADTG 8/9/A/B 90        0.17                  7.41                        6.76                  0.00 

Code: [Select]
    k = 2 .^ gains(i);
    A = [1   1 ;
         k*k 1];
    b = [noise(1); noise(i)] .^ 2;
    noises = sqrt(A\b);
    input_noises(i) = noises(1);
    output_noises(i) = noises(2);

So, it seems clear that ADTG 0xFE is before the other two, but it's not that clear which one from the other two ADTG amps is the first in the image processing chain.

However, ADTG 8/9/A/B is able to pull more highlight detail than ADTG 888x. Since turning 888x all the way down did not bring the other one out of saturation, my conclusion is that ADTG 8/9/A/B is before 888x.

Something like this:

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #504 on: February 26, 2014, 06:06:05 PM »
Applying that theory @ ISO 100

I can drop [0xFE] to 1 before WL drops giving 11.463 EV.

From here, I can drop ADTG [888x] to -0.04 EV before WL drop, giving 11.545 EV
Or, drop ADTG 8/9/A/B to 46 before WL drop, giving 11.514 EV.

In either case, once 1 of the gains has been optimised, the other can not be reduced without dropping WL.

Using ADTG[0xFE] 1, and ADTG [888x] -0.04, applying Digital Gain 540 increases DR to 11.593

edit:  ISO 200 should provide more useful results since here we can be sure of plenty of gain into the ADTG stage.

Here, I can reduce ADTG[0xFE] to 0 before WL drop, giving 11.460 EV

From here, I can drop ADTG [888x] -0.32 EV before WL drop, giving 11.776 EV
Or, drop ADTG 8/9/A/B to 0 before WL drop, giving 11.772 EV.

Again, in either case, I cannot adjust the other without reducing WL.

@ ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 EV.  Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + ADTG 8/9/A/B 0 = 11.641 + ADTG [888x] -0.03 EV = 11.673 EV
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG [888x] -0.34 EV = 11.671 EV.  Cannot reduce ADTG 8/9/A/B without WL drop.
[0xFE] 0 = 11.335 + SaturateOffset 1150 = 11.528 + ADTG 8/9/A/B 29 = 11.662.  Cannot reduce ADTG [888x] without WL drop.


Looks to me, ADTG [888x] is before ADTG 8/9/A/B

A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.34 EV = 11.679 + digital gain 530 = 11.732 EV.


It would be nice to have more adjustment in ADTG[0xFE], since we can reduce this to 0 (at higher ISOs) and still have adjustment in the other gains.

Greg

  • Contributor
  • Hero Member
  • *****
  • Posts: 589
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #505 on: February 26, 2014, 06:15:05 PM »
500D ISO 200

I lengthen exposure of 1/2EV with a trick.

Trick vs default :


naturalsound

  • New to the forum
  • *
  • Posts: 35
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #506 on: February 26, 2014, 06:57:22 PM »
A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.35 EV = 11.679 + digital gain 530 = 11.732 EV.

I have an idea about this. Could you estimate the gain by comparing both whitelevels?

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #507 on: February 26, 2014, 07:13:51 PM »
I have an idea about this. Could you estimate the gain by comparing both whitelevels?

ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  WL 15283, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 520, WL 15490, stdev 4.09
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 525, WL 15619, stdev 4.07
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 530, WL 15748, stdev 4.06
[0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 540, WL 16007, stdev 4.11

naturalsound

  • New to the forum
  • *
  • Posts: 35
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #508 on: February 26, 2014, 07:59:40 PM »
To cut a long story short: This subtle amount of digital gain alters the shape of the noise probability distribution and thus alters the standard deviation.

Long version:
As the data is already quantized for any amount of digital gain there will be gaps (at value v_gap in the histogram (as seen in your shown histograms a few pages earlier) because there is no value v_old to fulfill

Round[gain*v_old]==v_gap

In the following I assume an histogram centered at 2024 at unity digital gain.

Depending on the gap position the retrieved standard deviation may rise. This occurs the number of gaps in the histogram increases. The initial histogram has no gaps (because of unity gain). Increasing the gain increases the number of gaps inside the data. But those gaps lie outside the noise distribution. So the will not affect the computed stdev. Nevertheless the white level rises, so the retrieved dynamic range rises.
For your gain ([0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 540, WL 16007, stdev 4.11) the first gap is produced within the distribution (at initial value of v_old=2023 -> v_gap=2090). Thus the histogram gets wider and the stdev rises. This bigger stdev is now divided by the white level which did only rise one incremental step, resulting in a decreased Dynamic range.

I think this explanation holds true also for initial histograms that already have gaps, because the additional gaps will appear anyway. I will try to add some graphics.


UPDATE: I am sorry, this only affects the standard deviation of the noise probability distribution. Depending on your calculation of the Dynamic range this may not change the dynamic range at all, if the calculation takes into account the median of the noise's probability distribution.
Maybe these finding at least help to figure out the exact amount of digital gain, as the positions of gaps in a full scale histogram are sharp indicators for digital gain. Maybe those "half gaps" seen in some histograms originate from digital smoothing algorithms or correction matrices?

UPDATE2: This is too obvious. I'm sure it is already stated somewhere in those 21 pages  :-[

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #509 on: February 27, 2014, 04:12:06 PM »
Digital gain, from low values to higher values.












When it comes to log2(white/stdev), there is a sweet spot.  This seems to be around value (530), for lower ISOs.

BTW, 14.5 EV with dual ISO and 11 EV @ ISO 1600 is remarkable.  And is better then all other consumer camera afaik.

Heck, ISO 3200 still gives 10.5 EV.  :)

Greg

  • Contributor
  • Hero Member
  • *****
  • Posts: 589
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #510 on: February 27, 2014, 04:55:56 PM »
500D
CMOS[3] 0x810 -> 0xfff

ISO 100 0.015EV
ISO 200 0.062EV

Also cause vertical stripes...  :-\

So 500D can be changed safely only in black / white and saturate offset.
Gain 0.33 EV ISO 100
Other changes cause banding.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #511 on: February 27, 2014, 06:11:00 PM »
ADTG 8/9/A/B defaults to 85 in movie mode.  Reducing it looks to pull back some highlight detail.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #512 on: February 27, 2014, 09:20:27 PM »
Finally something that has effect in movie mode :D

My current settings for photo mode (based on the above theory, which basically says bump the gains with low input noise and lower the ones with high input noise):

- ISO 66 from 100: unchanged (see my previous posts).
- ISO 200 - 12800 in Canon menu: ADTG 0xFE = 3 instead of 4, ADTG 8/9/A/B as close to 0 as possible (keeping the deltas between them), ADTG gains -0.16, -0.18, -0.18, -0.19, -0.25, -0.31 and -0.35 EV. Resulting ISOs: 100, 200, 400, 800, 1530, 3000, 5700 (that is, I can effectively pull down all the important ISOs from the next full-stop ISO).

This time I did not measure the ISOs; I've simply used the theoretical model from iso_regs and rounded the ISOs. Feel free to compile raw_diag and iso_regs if you think you can get better measurements.

Code: [Select]
dr =        [ 10.989 10.928 10.799 10.594 10.153  9.521 8.584 7.778 ] ;     # measured with raw_diag
isos =      [ 100    200    400    800    1600   3200   6400  12800 ] ;     # ISOs from Canon menu

dr_tweak =  [ 11.589 11.782 11.687 11.463 11.028 10.447 9.704 8.930 ] + ... # measured with raw_diag
            [ 0.1    0.1    0.1    0.1    0.1    0      0     0     ] ;     # correction for nonlinearity
iso_tweak = [ 66     100    200    400    800    1530   3000  5700  ] ;     # estimated with iso_regs, adjusted for nonlinearity
                                                                            # and rounded (e.g. 218->203->200)



Note: in the screenshot, the ISO equivalence model assumes linear response; however, the response near highlights is slightly nonlinear. I estimate this nonlinearity helps capturing another 0.1 EV of highlights. Therefore, with these tweaks applied, the equivalent ISO according to DxO definition (clipping point) is nearly 1 stop lower (so the screenshot shows ISO 200 pulled from 400). For the same reason, I've added 0.1 stops to DR measurements where the nonlinear response was noticeable.

This graph does not include the white level optimization for the raw converters or the digital gain correction from the aperture. On 5D3, I estimate this improvement from 0.03 EV (manual lens) to 0.183 EV (f1.4 lens), from the white levels and aperture logs:
Code: [Select]
log2(15282-2048) - log2(15000-2048) => 0.031
log2(569) - log2(512) => 0.152

Of course, each of these gains is tiny, but when all of them add up to almost 1 full stop, the total improvement should be noticeable.

P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.

SpcCb

  • Member
  • ***
  • Posts: 207
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #513 on: February 28, 2014, 01:09:30 AM »
(...)
@SpcCb: can you share some more details about how you did the math?

(I tried to understand the AMaZE algorithm, but only figured out how to call it; if you can shed some light, I'm interested)

I will try to be simple, I mean I hope I will be.. :)

We can refer to what we discussed in the dual_iso thread about SVE images to explain the DR in this case:

We can see how is the gain with dual_iso in a perfect world (calculations done @10E-15 & results round @10E-3);
With a single frame:
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

-> Wooh, on the paper it is monstrous!

However I suppose you prefer a more physical approach to compare with samples, then from DRSVE we get:
DRdi = log2[(( WL - offset ) / (σSmaxx̄ σSmin) ) (smax / smin)]
Note: we should use the same WL and offset in both frames; because of course _for example_ with a different offset it will generate noise (!).

In my case I used a 5D2;
With a single frame, in the best case and with thermal considerations (short exposure time, up to CMOS @40°C):
DRsf = log2(( 15760 - 1024 ) / 6.47 )
DRsf = 11.153 stops

With dual_iso 100/800:
DRdi = log2[(( 15760 - 1024 ) / √{( 8² + 6.47² ) / 2 }) 64 ]
DRdi = 16.984 stops

Then, now we have basic maths to do projection of our SVE, but be careful it does not take consideration of the PSF + MTF, so the wavelength of the source, etc.
To do that we have to introduce light diffraction model in the equation.
I will not demonstrate the maths here, it's a bit complex and I think not relevant to make projections with samples we use: It requires calibrated sources, lens modelisation, etc.
BTW, in my study with a off-continuum mono-spectral source (λ = 0.656μm, source) I recorded an average difference of 3.9% and a sigma difference of 5.0% (both ±10E-1) between single frame image and dual_iso image by using AMaZE both for CR2->DNG & DNG demosaicing.
This mainly because I use 1 of 4px in the Bayer Matrix in this case and there's an intrication between the dual_iso SVE and the Bayer Matrix.
Beside, a similar effect should be observed in regular photography.

About AMaZE, what is very interesting is this algorithm do multiple analyses to proceed to the reconstruction.
There's a (spatial) vector analyse of structures (macro-blocks & special sophisticate diagonal), color/chromic analyse and _in the lasts versions_ a CA analyse, and there's a Nyquist texture (pattern) analyse to make better area interpolations.
IMHO, it is very smart from Emil Martinec _and certainly not foreign to the fact what Emil works in Astrophysics_ because this is well know in our domain what without taking consideration of the light nature it is hard to be close to the reality (even we never match her).

With this algorithm there's a significant SNR improvement face to other reconstruction algorithms; I ever measured +25% on hard cases face to VNG HAD HFFE etc. So compared to the simple mathematical projection where just an average is considered, we should see a gain, even if it's not +2 stops or -50% of sigma.
When we see SNR gain, we see DR gain. So, voilà. :)

Don't hesitate to point me what you don't find clear, I will see how to make it more simple or from a different point of view.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #514 on: February 28, 2014, 07:42:51 AM »
P.S. Be careful when you say you get 10.5 stops of DR at ISO 3200. After tweaking these parameters, the ISO from Canon menu is no longer relevant, but the equivalent ISO estimated with raw_diag should be good. The linear model from iso_regs should be pretty close too.

Indeed.  The graph clearly shows how there is very little DR gain in extreme ISOs.  Simply shifting the ISO point.



Using the settings a1ex listed above, comparing the highlights @ ISO 200.

Images removed.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #515 on: February 28, 2014, 09:52:19 AM »
Indeed.  The graph clearly shows how there is very little DR gain in extreme ISOs.

This probably just proves that Canon did this right by themselves, they already have superior dr vs. Nikon on higher iso ... and every little gain is a blessing in these situations. Plus since the base dr is lower, a small absolute gain here is a larger relative one :->

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #516 on: February 28, 2014, 10:25:25 AM »
Using the settings a1ex listed above, comparing the highlights @ ISO 200.

Can you share the EXIF info? Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #517 on: February 28, 2014, 10:46:33 AM »
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

-> Wooh, on the paper it is monstrous!

I'm not familiar with these notations, can you define them first?

In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.

From here I'm lost, because the checksum doesn't match.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #518 on: February 28, 2014, 12:39:21 PM »
Are you sure you compared 200 pulled from 400 vs Canon ISO 200 at the same shutter speed and aperture?

I was still thinking in terms of relatively small ISO differences.  So Canon ISO 200 vs ML ISO 100, they were at the same exposure though.

In essence, the gain is all in the shadows now?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #519 on: February 28, 2014, 12:51:42 PM »
If you just turn down some amps starting from some base ISO from Canon menu, the gain will be in highlights.

Since the gain is almost 1 stop, you can now bump Canon ISO one stop and move some of this gain to the shadows (you know the story, you always lose 1 EV of highlights, but you gain back a little less than 1 EV in shadows, and this shadow improvement vanishes at high ISOs).

So, now that we can re-create the full-stop ISOs with different settings (e.g. pull them down from the next higher ISO), we can compare them to the unmodified Canon ISOs. There are two things you should look for:

1) clipping point (should be the same, because this is how ISO is defined - the input signal required to saturate the sensor)
2) noise (our modified ISOs should be cleaner: higher DR => higher SNR)

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #520 on: February 28, 2014, 02:21:23 PM »
Can you confirm recommended settings for ISO 100 please?

And is it ok to use the CMOS tweak and WL settings inside mini_iso?  With ADTG gain set to 0.0?

I have a set of data, but this has the above mentioned tweaks, and I need to redo the test with correct ISO 100 settings.

Code: [Select]
Canon shutter file WL    stdev
100 6    6880 13027 194.8
200 13  6896 13074 204.7
400 25  6909 13221 221.0
800 50  6922 13116 238.1
1600 100 6938 13115 271.1
3200 200 6951 13091 326.0
6400 400 6965 13318 433.8

Code: [Select]
ML  shutter file WL    stdev
100 6    6984 14016 180.9
200 13  6995 14106 193.4
400 25  7007 14060 203.5
800 50  7019 13964 223.5
1600 100 7033 13502 247.0
3200 200 7047 13180 203.5
6400 400 7062 12965 383.8

note: that ML ISO is reported as Canon ISO -1 EV.  Also, reported WL has not been black subtracted.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #521 on: February 28, 2014, 02:27:47 PM »
If you use both modules, there might be some conflicts over these settings; so, to be safe, just use iso_regs and a manual lens.

White level from raw_diag should be 15282 in all cases. There might be exceptions at high ISOs because of hot pixels, where WL might be estimated above 15282 (ignore them).

So:
- unmodified ISO 100
- ISO 200 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.16 EV (do the math) and ISO estimated in iso_regs around 109. The difference until 100 should be covered by nonlinearity.

- unmodified ISO 200
- ISO 400 in Canon menu, ADTG 0xFE = 3, ADTG 8/9/A/B (preamp) = as close to 0 as you can (usually 1 or 2), ADTG gain -0.18 EV (do the math) and ISO estimated in iso_regs around 218. The difference until 200 should be covered by nonlinearity.

and so on.

The CMOS and black/white tweaks are separate and do not have a noticeable impact over the measured ISOs. You can analyze these ones separately (to make sure their impact is indeed minor) or combined (to check the overall effect of all these tweaks).

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3565
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #522 on: February 28, 2014, 05:19:11 PM »
Lens unscrewed, last stop before full saturation of the green channel.
WL is the average WL measured in the sample patch of one of the green channels.  stdev is the standard deviation measured from the sample patch, in the green channel.

Code: [Select]
Canon shutter file WL    stdev
100 15  7316 13255 158.2
200 30  7323 13330 170.6
400 60  7330 13513 185.8
800 125 7337 13414 208.9
1600 250 7344 13471 247.3
3200 500 7351 13607 308.5
6400 1000 7358 13963 419.6
1280 2000 7365 14489 543.4*
1280 2500 7366 12010 523.9

* Max value in channel pushed to saturation.

Code: [Select]
ML  shutter file WL    stdev
100 10  7380 14305 158.9*
100 13  7381 12028 144.9
100 15  7382 10071 121.5
200 15  7386 14169 148.7
400 30  7393 14295 157.3
800 60  7400 14208 169.7
1600 125 7407 14176 195.9
3200 250 7414 13702 227.3
6400 500 7421 13627 285.8
1280 1000 7428 13366 370.1

* Last stop before total saturation in channel.
The ISO rating displayed is that in Canon menu, ie: the pulled ISO.  The gains used are as above, and for ISO 100 here.
CMOS tweak was applied.


If I have my maths right, The noise difference between ML ISO 100 and Canon ISO 100 in the highlights.

log2(14169/13255) - log2(158.2/148.7) = 0.0068 EV

And the exposure difference is log2(14169/13255) = 0.096 EV.  Which puts Canon ISO 200 @ ISO 107, subject to any nonlinear behaviour, not captured.

Canon ISO 100 vs Canon pulled ISO 100

log2(15/10) - log2(14305/13255) = 0.4750 EV

Which puts ISO 100 @ ISO 77, again, subject to any nonlinear behaviour, not captured.

Is the noise level difference log2(15/10) - log2(14305/13255) - (158.9/158.2) = 0.529 EV?



Having a quick look at the shadows.
Click thumbnails for full resolution JPGs.  These are around 10mb.

Canon ISO 100


Canon pulled ISO 200


We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.

SpcCb

  • Member
  • ***
  • Posts: 207
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #523 on: February 28, 2014, 05:45:26 PM »
(...)
DRsf = 20log10[(imax / imin)] = 20log10[16384]
DRsf = 84.288 dB

With a SVE image with ISO 100/800 like in dual_iso:
DRSVE = 20log10[(imax / imin) (emax / emin)]
ISO 100/800, so e1 = 64e0
DRSVE = 20log10[16384×64]
DRSVE = 120.412 dB

(...)
I'm not familiar with these notations, can you define them first?

In particular, from "ISO 100/800, so e1 = 64e0", I guess e should be the noise variance, assumming equal DR for 100 and 800, and the ISO 800 image darkened to match the ISO 100 one. However, when you say DRSVE = 20log10[16384×64], you are using this 64 as if it were the standard deviation of the noise, so you've got a theoretical 6-stop improvement for 100/800 (which means the noise stdev for the darkened ISO 800 would be 64x lower than the one for ISO 100). Under ideal conditions (equal DR for 800 and 100), the stdev ratio would be 1/8, not 1/64.

From here I'm lost, because the checksum doesn't match.

OK, let's see this like that:
emax and emin are the maximum and minimum energy scale (not a correct naming, just to get a picture of this) present in the exposure pattern of the SVE.
We took as example to do maths ISO 100/800 in dual_iso, so we get:
ISO 100 - 200 - 400 - 800 range
Corresponding to:
e3 = 4e2 = 16e1 = 64e0
Remember we are in power of 2 world here, to be in accord with the ADU range (16384). Intercourse between ADU and ISO numbers, this is weird. :)
There's no standard deviation here as we are in the 'perfect world' example; it is just to give us the absolute viewing and to know what are the system limits. I always do it to be sure we will not go outside these limits after, when we add 'more physical' parameters, and to give the most 'simple' figure at the beginning (like an introduction).

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12255
  • Emergencies only
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #524 on: February 28, 2014, 06:02:02 PM »
Lens unscrewed, last stop before full saturation of the green channel.

WL: from your description, this is not white level (you have useful data above it).

Gains: you linked the settings before finding the 0xFE register. Once you have changed this one from 4 to 3, the sweet spot for ADTG gains is now the one from here.

Shutter: please use SI units.

Noise: you are probably trying to compute the upper limit of the SNR curve (best-case SNR). Signal is that thing you named WL (assumming you subtracted the black level), noise is stdev. SNR from some more signal levels would be more interesting (the highlights are already clean; most of the noise is in the shadows).

Exposure difference: yes, overall brightness is higher by 0.1 stops. If you measure patches at 1 stop below clipping point, you will not see the nonlinear behavior. If you compare the exact clipping point, you will see it.

You quoted Canon ISO in modified ISOs, which is irrelevant and makes it hard to choose what to compare with what. Moreover, since the equivalent ISOs above 800 are no longer full-stop ISOs, these lines can't be compared directly (you can either plot a graph showing the equivalent ISO vs measured SNR, or you can pull exactly one stop to get ISOs that can be compared directly).

@SpcCb: I'm lost. Where did you get these numbers from? Why ISO 800 has 64x more "energy" than 100? What kind of "energy" is this?

I use the engineering definition of DR: log2(full well capacity / noise RMS), which is log2(white - black) - log2(dark frame stdev), approximated to log2(white - black) - log2(optical black stdev). How does this "energy" fit here?