Author Topic: CMOS/ADTG/Digic register investigation on ISO  (Read 801549 times)

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #75 on: January 16, 2014, 02:56:59 PM »
I thought 160 was pulled, not pushed?

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #76 on: January 16, 2014, 03:19:29 PM »
In LV you should be able to auto adjust ADTG for maximum white point, correct?  So in manual mode you can choose your exposure settings, and ML applies ADTG gain as needed. 

An analog auto ISO only limited by the accuracy of the white point detection.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #77 on: January 16, 2014, 03:24:18 PM »
LV ADTG settings are only relevant for raw video (and I didn't study them yet). In photo mode you get ExpSim, so the only interesting thing here is to match the clipping points for ETTR.

I thought 160 was pulled, not pushed?

Yes, but I was talking about the attempts to hide digital trickery.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #78 on: January 16, 2014, 06:37:19 PM »
What would a development thread be without shots of boring stuff!

Canon ISO 100 (1,055 KB)


ADTG tweak (853 KB)


ADTG tweak + Dual_ISO 100/1600 (437 KB)



Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #79 on: January 16, 2014, 06:44:32 PM »
What would a development thread be without shots of boring stuff

Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.

hjfilmspeed

  • Senior
  • ****
  • Posts: 479
  • 5D III and IV
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #80 on: January 16, 2014, 07:05:10 PM »
ML iso is much better!!!! Cant wait! I will always have ML iso on if it makes it to the nightly builds. (i sure it will be a little more complex then turn ml iso on but i hope its that simple) You guys rock!!!

Steve Kahn

  • New to the forum
  • *
  • Posts: 23
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #81 on: January 16, 2014, 08:07:01 PM »
Where does this vertical banding after raising shadows come from - is this standard on 5d3? I've never seen this on the 6d, probably the latter is really an improvement in this category, but of course hard to say w/o taking identical shots.

I was going to ask the same question.  I have a 5D3 and have never seen that.  Also, it is not there on the dual ISO shot for some reason.

hjfilmspeed

  • Senior
  • ****
  • Posts: 479
  • 5D III and IV
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #82 on: January 16, 2014, 08:26:09 PM »
Have you ever tried to shoot a light bulb outside at night, without clipping the highlights, then recover the shadows around it? If there are not a lot of details in the shadows you will see the sensor noise (not banding). I can assure you if you shot this exact situation with your 5d3 or 6d, recovering the shadows will give you this result. Its not in the dual iso image because thats what dual iso was designed to do. To give you better highlight and shadow detail.

These ML iso options will be a great advancement in the mysteries surrounding canon iso and cleaner, better iso.


Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #84 on: January 16, 2014, 08:40:17 PM »
WOW.... that tweak and Dual ISO shot looks great!
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

hjfilmspeed

  • Senior
  • ****
  • Posts: 479
  • 5D III and IV
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #85 on: January 16, 2014, 08:43:16 PM »
Especially considering where it came from, that is incredible.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #86 on: January 16, 2014, 09:52:37 PM »
If there are not a lot of details in the shadows you will see the sensor noise (not banding).

I'd consider this noise pattern banding, I know that very well from the 60d where it appears very soon. At some point you might see it on any camera, that's why I was asking if this is usual or how much ev the shadows were raised to make it appear... and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #87 on: January 16, 2014, 10:49:01 PM »
and the areas were near black, so it's not surprising and a little dr gain from ml is unable to fix such a worst case situation.

Dual-ISO looks like it did fine  :P

Steve Kahn

  • New to the forum
  • *
  • Posts: 23
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #88 on: January 16, 2014, 11:10:27 PM »
Dual-ISO looks like it did fine  :P

At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   

engardeknave

  • Senior
  • ****
  • Posts: 339
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #89 on: January 16, 2014, 11:21:34 PM »
This is the problem with showing off any sort of impressive capture of a wide DR scene. Nobody really gets how bad it was before recovery. I may have to create a before/after section for my real estate photography.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #90 on: January 16, 2014, 11:23:15 PM »
At first glance Dual-ISO looks softer but there is a focus difference between the shots.  This can be seen by examining the "MA" letters near the top left.   


Indeed there is.  I'll post some other comparisons and updated SNR graphs after sleep.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #91 on: January 16, 2014, 11:53:17 PM »
Dual-ISO looks like it did fine  :P

No doubt, it's great and but this is really the type of shot made for it - but once you get details in the shadows and highlights it's a tough decision how much resolution to lose to gain a certain amount of dynamic range.

That's why I'm very excited that now there's every possibility that ML might conjure up 1/2ev of dynamic range out of thin air ... makes you really, really wonder if there's a hidden catch and why Canon doesn't include this, maybe they don't want to confuse users with too much iso wizardry?

arturochu

  • Member
  • ***
  • Posts: 129
  • Filmmaker based in Mexico City
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #92 on: January 17, 2014, 12:49:32 AM »
i don't understand 90% of this thread, but seeing so many "DR" talk makes me so excited about the progress of the topic haha.
Chu

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #93 on: January 17, 2014, 01:14:28 AM »
Yea that banding is on almost everything at higher ISOs, 6D, 7D, etc. After ACR or other post processing its mostly gone. Its hardest to remove it from LV dngs.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #94 on: January 17, 2014, 03:13:51 AM »
Btw & ot: it would be interesting to see if DxO's new PRIME algorithm is able to do better than ACR (or whatever oss) which fail at removing patterns - DxO afaik uses a noise pattern detection to clean up something like this w/o too much plastic look.

On my test shot with Greg's car, it didn't remove the pattern noise and actually interpreted it as detail.

I have found a math model for this pattern noise though, and I hope to have a clean fix for it in the near future.

SpcCb

  • Member
  • ***
  • Posts: 207
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #95 on: January 17, 2014, 03:18:50 AM »
Yes, but the interest is in the distribution.  Specifically, the effects by digital manipulation.

a1ex called it pattern.  The point being, as the distribution does not change with ADTG or SaturateOffset, we can safely assume that the register is making changes in the analog domain.  If a register changes the histogram distribution, we can safely assume it's making changes in the digital domain, in which case, it's probably pointless.

When you get this (5D2 distribution, but I think it's close to the 5D3):


It is because of this (average of 300 frames with winsorized sigma clipping to reject random noises + bin2 + pix value x16):


With a 2D FFT we can better see how is the pattern:

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #96 on: January 17, 2014, 04:41:23 AM »
The banding pattern accounts for only a small part of these histogram distortions.

Proof:

ISO 100 dark frame (5D3), original and corrected


Histograms of horizontal and vertical banding correction (per-line and per-column)


Image histogram before and after correction:


How I've estimated the correction:
Code: [Select]
# estimate horizontal and vertical banding
fh = mean(im'); fh = fh - mean(fh);
fv = mean(im);  fv = fv - mean(fv);

# banding histograms
hist(fh, -3:0.1:3)
print -dpng -r60 hband.png
hist(fv, -3:0.1:3)
print -dpng -r60 vband.png

# compute the flat field
[m,n] = size(im);
ffh = fh(1:m)' * ones(1,n);
ffv = ones(m,1) * fv(1:n);
ff = ffh + ffv;

# image histogram before and after correction
hist(im(:), 2048 + (-50:50))
print -dpng -r60 hist0.png
hist(im(:) - ff(:), 2048 + (-50:50))
print -dpng -r60 hist1.png

SpcCb

  • Member
  • ***
  • Posts: 207
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #97 on: January 17, 2014, 05:32:56 AM »
When I do:
* sub *

I get this:


Of course it's not a perfect gaussian yet, but it's close. :)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #98 on: January 17, 2014, 05:42:53 AM »
May I see a single dark frame minus the averaged one?

(I know, I could probably take 300 frames myself, get a PixInsight license and try it)

SpcCb

  • Member
  • ***
  • Posts: 207
Re: CMOS/ADTG/Digic register investigation on ISO
« Reply #99 on: January 17, 2014, 05:57:00 AM »
Ah, sorry, I don't have it right now. I just have the histograms of those frames (here in my last post).
But I could do it later. I'v planed to publish the same operation on a video RAW frame.


About Pixinsith, mh... You can use Iris too, it's free.