CMOS/ADTG/Digic register investigation on ISO

[dmilligan]

I thought 160 was pulled, not pushed?

[Audionut]

In LV you should be able to auto adjust ADTG for maximum white point, correct?  So in manual mode you can choose your exposure settings, and ML applies ADTG gain as needed. 

An analog auto ISO only limited by the accuracy of the white point detection.

[a1ex]

LV ADTG settings are only relevant for raw video (and I didn't study them yet). In photo mode you get ExpSim, so the only interesting thing here is to match the clipping points for ETTR.

I thought 160 was pulled, not pushed?

Yes, but I was talking about the attempts to hide digital trickery.

[Audionut]

What would a development thread be without shots of boring stuff!

Canon ISO 100 (1,055 KB)


ADTG tweak (853 KB)


ADTG tweak + Dual_ISO 100/1600 (437 KB)

[Marsu42]

What would a development thread be without shots of boring stuff

Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.

[hjfilmspeed]

ML iso is much better!!!! Cant wait! I will always have ML iso on if it makes it to the nightly builds. (i sure it will be a little more complex then turn ml iso on but i hope its that simple) You guys rock!!!

[Steve Kahn]

Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.

I was going to ask the same question.  I have a 5D3 and have never seen that.  Also, it is not there on the dual ISO shot for some reason.

[hjfilmspeed]

Have you ever tried to shoot a light bulb outside at night, without clipping the highlights, then recover the shadows around it? If there are not a lot of details in the shadows you will see the sensor noise (not banding). I can assure you if you shot this exact situation with your 5d3 or 6d, recovering the shadows will give you this result. Its not in the dual iso image because thats what dual iso was designed to do. To give you better highlight and shadow detail.

These ML iso options will be a great advancement in the mysteries surrounding canon iso and cleaner, better iso.

[Audionut]

That's deep into the shadows.



http://graphics.stanford.edu/courses/cs448a-10/sensors-noise-14jan10-opt.pdf  <---  Page 43

http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
http://www.clarkvision.com/articles/evaluation-canon-6d/index.html

[Stedda]

WOW.... that tweak and Dual ISO shot looks great!

[hjfilmspeed]

Especially considering where it came from, that is incredible.

[Marsu42]

If there are not a lot of details in the shadows you will see the sensor noise (not banding).

I'd consider this noise pattern banding, I know that very well from the 60d where it appears very soon. At some point you might see it on any camera, that's why I was asking if this is usual or how much ev the shadows were raised to make it appear... and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.

[Audionut]

and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Dual-ISO looks like it did fine 

[Steve Kahn]

Dual-ISO looks like it did fine 

At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   

[engardeknave]

This is the problem with showing off any sort of impressive capture of a wide DR scene. Nobody really gets how bad it was before recovery. I may have to create a before/after section for my real estate photography.

[Audionut]

At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   

Indeed there is.  I'll post some other comparisons and updated SNR graphs after sleep.

[Marsu42]

Dual-ISO looks like it did fine 

No doubt, it's great and but this is really the type of shot made for it - but once you get details in the shadows and highlights it's a tough decision how much resolution to lose to gain a certain amount of dynamic range.

That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?

[arturochu]

i don't understand 90% of this thread, but seeing so many "DR" talk makes me so excited about the progress of the topic haha.

[1%]

Yea that banding is on almost everything at higher ISOs, 6D, 7D, etc. After ACR or other post processing its mostly gone. Its hardest to remove it from LV dngs.

[a1ex]

Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.

On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.

I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.

[SpcCb]

Yes, but the interest is in the distribution.  Specifically, the effects by digital manipulation.

a1ex called it pattern.  The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain.  If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.

When you get this (5D2 distribution, but I think it's close to the 5D3):


It is because of this (average of 300 frames with winsorized sigma clipping to reject random noises + bin2 + pix value x16):


With a 2D FFT we can better see how is the pattern:

[a1ex]

The banding pattern accounts for only a small part of these histogram distortions.

Proof:

ISO 100 dark frame (5D3), original and corrected


Histograms of horizontal and vertical banding correction (per-line and per-column)


Image histogram before and after correction:


How I've estimated the correction:

Code: [Select]

# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im);  fv = fv - mean(fv);

# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png

# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;

# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png


[SpcCb]

When I do:
* sub *

I get this:


Of course it's not a perfect gaussian yet, but it's close.

[a1ex]

May I see a single dark frame minus the averaged one?

(I know, I could probably take 300 frames myself, get a PixInsight license and try it)

[SpcCb]

Ah, sorry, I don't have it right now. I just have the histograms of those frames (here in my last post).
But I could do it later. I'v planed to publish the same operation on a video RAW frame.


About Pixinsith, mh... You can use Iris too, it's free.