Dynamic range and Equivalence

[a1ex]

Topic split from CMOS/ADTG/Digic register investigation on ISO.



These days I've fiddled with these noise models quite a bit, and I have some interesting results.

First, I've imported all the data from sensorgen.info and created some plots from there. If you remember, my noise model only uses the FWC and the read noise. Here are some plots (take them with a grain of salt):



(I can pick any combination of cameras from there, so if you are interested in some particular comparisons, just ask).

Typo on sensorgen.info: 6D resolution is not 3840x5760, but 3670x5496. If you know how to contact these guys, please let me know.

Note the dynamic range I've used is not log2(full_well) - log2(dark_noise) - this formula is just an approximation, and a very rough one at high ISO - it's overestimated by as much as 0.5 stops on 60D at ISO 12800, but only by 0.05 stops at ISO 100 (just do the math from my SNR plots below). Instead, I've used the DR as defined in ISO 15739 - quote from http://dougkerr.net/Pumpkin/articles/ISO_Dynamic_range.pdf :

the ratio of the maximum luminance that receives a unique coded representation (the "saturation" luminance) to the lowest luminance for which the signal to noise ratio (SNR) is at least 1.0

So, from the SNR curve, I pick the point where SNR is 0 EV (that means 1.0) and I measure the dynamic range from here until the white level, like this:


Then, I've tried to compare sensorgen data with the measurements obtained with this method. I took the 5D3 and 6D data from Audionut and Levas, and also measured the 60D and the 5D3 myself.

SNR curves for 60D: https://www.dropbox.com/sh/eq2yz563gvx283e/AADD1g8nGxdeetb1FwHmhDqea?dl=0
SNR curves for 5D3: https://www.dropbox.com/sh/jx7byevkt8sl6in/AADy9z2zHZZkbA56N6x1evi3a?dl=0

Comparison (my method vs sensorgen data):


Notice the outlier on sensorgen's data at 60D ISO 3200 (so, take their numbers - and also mine - with a grain of salt).

If you looked closely at the SNR plots, you can see I've tried to compute some confidence intervals. I know almost nothing about this subject (statistics are not exactly my cup of tea), so I simply changed the model parameters (read noise and full well) until the curve fitting error (sum of absolute deviations) increased by 20% (arbitrary threshold that appears to give reasonable results). It's good for identifying bad fits, but it probably has no statistical interpretation (like some probability of the true values being in this interval or whatever). This stuff is way beyond my math skills.

Example of bad fit (the datasheet parameters for this test case are FWC=13500 and noise=13 electrons):


Just for fun, I've also measured the Nokia Lumia 1020 (test data from g3gg0). You can see the higher uncertainity for dynamic range, because the test data did not include really deep shadows. For some reason, the high-ISO result seem bogus, figure out why.

SNR curves for Lumia 1020: https://www.dropbox.com/sh/j90a5no22hucvuo/AACaPOfxoTJh2bsZ5Ynn41Nva?dl=0




So, at this point I think we have a pretty good method for evaluating the improvements of the ISO tweaks (both evaluating the DR improvements, and telling where these improvements are - in highlights or shadows). That means, I will be able to tell the exact performance of the tweaks, without the fear of over-promising

There's still something not considered in my tests - the response curve (I've assumed it's linear).

For the octave scripts used for these plots, just ask.

@Levas: did you keep the CR2 files for the dynamic range tests? I was curious to check the uncertainity levels.

[Audionut]

Have you found a reliable way to detect any artifacts in the highlights?  The last time I looked (sorry can't link to my previous tests, poor internet connection here), it appeared that I had to relax the amount of gain reduction on the 5D3 to ensure very clean highlights.  I haven't had the time to do some decent testing on the 6D.

On this SNR curve:


I don't understand how you measure a DR of 10.52, when the scale of the graph appears to only go to 10.  More interesting, is the measurement data (the blue dots) that rise vertically @ 0EV.  Is this some non-linear data, or a measurement/display error?

[a1ex]

The DR is measured horizontally (red line), not vertically.

The maximum SNR is about 6.5 EV, and the points above that are outliers (the noise there is no longer Gaussian, because it's clipped). I ignore the last half-stop when doing the curve fitting.

The highlight autodetection is still sensitive to artifacts in highlights (so if you lower the gains too much, raw_diag may report a little more DR than you actually get).

[a1ex]

After watching this video, I was curious to see how the graphs would look like if I would use the full-well capacity of the entire sensor instead of ISO on the DR graphs.

With same-size sensors, there should be no difference. The difference appears when comparing sensors of different sizes: at the same ISO, the smaller sensor would get less light (fewer photons). Imagine you put your 50mm lens on 5D3, then on 60D, and take a test picture at ISO 200 1/100 f/4 on both cameras (just an example). You will get 2 identically exposed pictures (of course, with different framing). But on the 60D, some of the light will end up outside the sensor.

You may think: what would happen if you would concentrate the image to fit the smaller sensor? (that is, imagine you use some sort of adapter on the 60D, so you get the same framing as with the 5D3). In this case, both sensors would receive the same amount of light, correct?

So, let's compare how the sensors would behave with such an adapter, in the ideal case (think of a hypothetical Metabones 0.63x for the APS-C Canon, if one would exist). That's what the following graphs will show - how good the sensor technology is, regardless of sensor size.

In the first graph, you can see the 5D3 and the 60D are just as good. That is, if you would use a 80mm/2.8 lens on 5D3 and a 50mm/1.8 on 60D, you would get the same results (same framing, same low-light performance at high ISOs). The 6D is clearly better (about 0.5 stops) and the old 5D shows its age.


Second graph (sensorgen data) shows a few more Canons. The surprise here is the S100 compact - it's sensor is really good.


Note: the X axis in my graphs shows the total number of photons that should fall on the sensor in order to saturate it (assuming perfectly uniform brightness). It's reversed, so the graphs are similar to the previous ones (which have ISO on the X axis). However, I don't know how many photons correspond to what ISO (but it's easy to do the math).

Third graph (also sensorgen data) shows that GH4, Nikon 1 V2 and 6D have similar sensor technology. Sony A7S curve is a little weird - probably outliers in sensorgen data, or some sort of noise reduction, can't tell (if you have one, I'd like to see some test images).


From the isoless sensors, no big surprises: Nikon D5300, Pentax K5 and GH4 are pretty simlar; with a hypothetical Metabones adapter, they might have a small edge over the larger D800/D810. Also, the Olympus E-M1 is worth checking.


Caveat: the Metabones adapter doesn't quite convert your M4/3 camera into full frame. My graphs shows the ideal performance of such hypothetical adapter, if one would exist. So don't rush to sell your 6D for a Oly E-M1 + Metabones for astro

If you have used a Metabones adapter, feel free to confirm or debunk this theory.

[SpcCb]

After watching this video, I was curious to see how the graphs would look like if I would use the full-well capacity of the entire sensor instead of ISO on the DR graphs.

With same-size sensors, there should be no difference. The difference appears when comparing sensors of different sizes: at the same ISO, the smaller sensor would get less light (fewer photons).
(...)

I'm not sure to understand : Are you saying FWC and QE are function of the sensor size? I mean the entire sensor die, because of the area differences between them?

(I watched first minutes of the video and stopped when I saw tones of b.l.ch.t errors)

[a1ex]

I've plotted the full well capacity of the entire sensor (that is, fwc of a pixel multiplied by resolution).

How I've got the QE is explained in my earlier post.

I don't remember any obvious errors in the video, except that the SNR he's talking about is what you would get with an ideal sensor with no read noise (which is a nitpick). But if you have a solid proof that what the guy says in the video is wrong, please show it.

(just be sure to read this and this and this before trying)

[SpcCb]

Ah, the FWC x nbr of pixels. I understand know.

But what the goal? Do you plan to make a bin(full sensor size) image of 1x1px?

[a1ex]

Just to see how the sensor technology compares between models. That is:
- if you would glue 4 GH4 sensors to create a full-frame one, how it would compare to another FF sensors
- if you would use an ideal Metabones adapter, what you would get (compared to FF)
- in other words, is it better to have a large sensor, or a small sensor + an optical adapter?

[Levas]

About dynamic range, DxO has test results of the Canon powershot g7 X.
And guess what, it's linear, doesn't flat out by the lower iso's.

[a1ex]

It's normal. The G7X has a 13.2mm x 8.8mm sensor, so its area is 7.44x smaller than full frame (about 3 stops). This means ISO 100 on the G7X sensor will be equivalent to ISO 800 on full frame. If you start from ISO 800 on 6D, the curve is pretty much linear.

About equivalence:
http://www.dpreview.com/articles/2666934640/what-is-equivalence-and-why-should-i-care
http://www.josephjamesphotography.com/equivalence/
http://www.clarkvision.com/articles/f-ratio_myth/
and, of course, that "controversial" video linked above.

[Levas]

Damn, didn't think about ISO equivalence...
I've seen the "Tony" video before, and still my brain thinks about ISO as a fixed number like Celsius degrees is for temperature.
But ISO is relative. It's a relative number and it differs between sensor size.

Thanks for the interesting articles 

[whumber]

About dynamic range, DxO has test results of the Canon powershot g7 X.
And guess what, it's linear, doesn't flat out by the lower iso's.

It's also using a Sony sensor.

[a1ex]

Some food for thought regarding sensorgen data:
http://www.dpreview.com/forums/thread/3565952
http://www.dpreview.com/forums/thread/3568630

edit: sensorgen applied the fixes discussed above, so my graphs might be a bit outdated

[Audionut]

Can you plot SNR curves from an equal photon count?  Like taking the SNR data from DxO, and photon count matching it. 

With base ISO plots, from say an 6D and an S100, we could see the SNR advantage for the 6D thanks to the higher photon count. 
Combined with an SNR plot from an ISO on the 6D, that is photon count matched to the S100.  This one, rather then defining the DR of equivalent ISO's, would show the SNR curve of the equivalent ISO's.

So we could see the crux point, above which the 6D is preferred (Poisson distribution), and below which the S100 is preferred (read noise), and the SNR curve differences between the equivalent ISO's.

[a1ex]

Will do.

Meanwhile, sensorgen updated their formulas as discussed here: http://www.dpreview.com/forums/thread/3565952
The data I've downloaded last month no longer matches the data from the website, so I'd first double-check my math and update the graphs.

edit: something's fishy with the new sensorgen data: they now say the 6D's QE is lower than 5D3's...

[a1ex]

Can you plot SNR curves from an equal photon count?  Like taking the SNR data from DxO, and photon count matching it. 

Something like this?



(these are Full SNR curves from DxO, normalized: horizontally to match the photon count per sensor area, and vertically to 8 Mpix)

Looks like the only way to get the S100 ahead the 6D is to shoot at night, expose properly on the S100 and choose ISO 100 for the 6D
And even then, the S100 will be a little ahead only in very deep shadows. Overall, the 6D image will be better.

[Audionut]

Thanks.

Looks the S100 is somewhat ISOless, which is to be expected I guess, since it should be tuned for an higher sensitivity to low photon counts..  I'm not sure if matching the photon count per sensor area is showing results that one would expect in real use (or I'm just having problems reading the graph with this normalization).

Equivalent exposures.
This says, if we use equivalent aperture (for the same photon count from the lens, DOF, and framing), then we adjust ISO to generate equivalent exposures.  In this case, we should have the same amount of actual photons hitting the sensor (also same DOF/framing), with the sensor sensitivity adjusted to render the same brightness.  The normalization to 8MP should negate any resolution differences.

I guess I am (was) expecting some cross section area where the SNR curves from both cameras intersect (other then deep shadows).  The DxO data says that when all of the unicorns are crying in harmony under a full moon, the S100 can deliver 37.9 dB (SNR) @ ISO 80.  But where is this in relation to the 6D?  Does the 6D at (say) ISO 1600, render the same brightness as the S100 (@ ISO 100) at equivalent apertures?  Wouldn't using equivalent apertures already take equivalence into equation, without matching the photon count per sensor area?

edit: something's fishy with the new sensorgen data: they now say the 6D's QE is lower than 5D3's...

They now list some results for the 5D3 @ ISO 50, with a slightly higher saturation compared to ISO 100.  In either case, both cameras saturate at the same scene luminance (ISO), yet the 6D records a greater number of photons (Saturation (e-)), but has a lower QE.

I'm fairly sure, that previously, the data from sensorgen closely matched the data from raw_diag.  Now there are some differences between sensorgen and raw_diag on my 6D.  @ ISO 6400, sensorgen lists the read noise as 2.0e, saturation as 1242e and DR as 9.3.  raw_diag is 1.7e, 1244e and 9.06.  The DR numbers don't seem to add up.

I think this highlights that the best way to obtain data, is directly from the source, not some fancy mathematical equation from a subset of data from an 3rd party.

If I adjust CMOS4 from 0x4f0 to 0x8f0, I get this.


I haven't tried playing with digic registers to raise the brightness yet.

[a1ex]

Regarding equivalence:

The S100 has a 5.2-26.0mm f/2.0-5.9 lens. Let's consider just the wide end: 5.2mm f/2.0, and let's choose ISO 100 from Canon menu.

From sensorgen data, the sensor size is 7.5x5.5mm, which gives a crop factor of 4.8. This means our little lens is equivalent to a 25-125mm lens. Note: dpreview says it's equivalent to a 24-120, probably rounded - source.

Let's assume the crop factor is 4.8.

To get a similar picture on a 6D (same framing, same DOF, same clipping point, same noise levels assuming identical sensor technology, and so on), you would need to use 25mm f/9.6 and choose ISO 2300 (roughly). That's what the equivalence theory says.

the S100 can deliver 37.9 dB (SNR) @ ISO 80

This is the peak SNR, at the far right of the curve - it's what you get in nearly-clipped highlights. 37.9 dB is 6.3 EV.

Does the 6D at (say) ISO 1600, render the same brightness as the S100 (@ ISO 100) at equivalent apertures?

The 6D at ISO 1600 would be about half-stop darker.

If we match the photon count on the entire sensor (not per area unit), we can compare the sensor technology. But the result would not be very useful in practice, unless you had some crazy Metabones-like adapter that would undo the crop factor completely (here, 4.8 ), or if you could glue a bunch of small sensors to get a full frame version.



This graph will also show that ISO 100 on S100 will be equivalent to something between ISO 1600 and 3200 on full frame - it's somewhere in the middle (look at the horizontal position).

Why it's not exactly in the middle? Because 6D's ISO 1600/3200 were measured at 1210/2400, while S100's ISO 100 was measured as ISO 85. That's why the red dot is a little closer to the yellow one.

So, indeed, the S100 sensor technology seems a little better than 6D's. Maybe it's just easier to do this with smaller sensors?

I think this highlights that the best way to obtain data, is directly from the source, not some fancy mathematical equation from a subset of data from an 3rd party.

Correct.

[Audionut]

Thanks.  Quite a can of worms. 

If we match the photon count on the entire sensor (not per area unit), we can compare the sensor technology. But the result would not be very useful in practice, unless you had some crazy Metabones-like adapter that would undo the crop factor completely (here, 4.8 ), or if you could glue a bunch of small sensors to get a full frame version.

Why exactly?  I understand that if you screw an FF lens onto a 1.6 crop body, a bunch of photons will land outside of the sensor, since for the same lens mount, there is a smaller chunk of silicon.

But surely, the little 4.8 crop factor body has been designed, so as not to have some significant percentage of captured photons, delivered somewhere other then the sensor?  One would assume that the lens mount has been designed, to help focus all photons from the lens, onto the senor.  If we assume that this is indeed the case, then the above graph is an accurate representation of the sensors, for equivalent apertures.

For me, the above graph is significantly easier to read.  I can see the maximum SNR of the S100, I can assume that it is shot noise limited (since the maximum SNR falls on the 6D curves), I can see directly how the 6D delivers an increased SNR thanks to Poisson distribution.  And indeed we can see, that in low scene luminance circumstances, the S100 delivers an edge.

If I'm understanding this right, the graph with matched counts per unit area, shows us the performance if we were to mount both bodies on say, an telescope.

So, indeed, the S100 sensor technology seems a little better than 6D's. Maybe it's just easier to do this with smaller sensors?

I think the biggest issue, is that we don't know why exactly the S100 seems a little better.  Is it all sensor related, or does the downstream electronics play a role?

ie: The 6D has to be tuned for an photon count of say 100,000 maximum, through 1 minimum. 
The S100 has an sensitivity advantage, since it only needs to be tuned for an photon count of say 6,000 maximum, through 1 minimum.  Four and a bit stops of light that it simply does not care about.

[a1ex]

But surely, the little 4.8 crop factor body has been designed, so as not to have some significant percentage of captured photons, delivered somewhere other then the sensor?

Yes, but for this reason, it has a much smaller lens (which captures much less light).

And indeed we can see, that in low scene luminance circumstances, the S100 delivers an edge.

This would be true only if you could attach a big DSLR lens and concentrate all the light to the tiny sensor. Or, if you could make a 35mm sensor with the same technology (sensor + electronics).

Roger Clark explains it better than me:
http://www.clarkvision.com/articles/f-ratio_myth/