HowTo? Best practice for using Dual-ISO?

[a1ex]

This is how I found the bug. With the original ufraw, you need to change the white level in the DNG to 1-2 stops above what is now in order to get a correct rendering.

Just for fun:

Canon JPEG (extracted from the CR2):


dcraw -H 9:

[Marsu42]

LR 5.3 demo, -3EV, developed by me (side question: why it's everything green at 5000K and zero tint?!)

That's why I was talking about when describing my wb/tint problems with LR, and even when you copy/past a valid wb setting from a non-dual iso shot it's not the same (I want to give you a proper report on this & that's why I didn't report back yet, but everything I tried results in a wb shift in LR). Is it really different with the standalone ACR converter?! They are supposed to use the exact same algorithms, just like ACR in PS.

As for highlight recovery in LR: Yes, it's good, but it's too weak for many dual_iso shots because even recovery -100 has still blown whites but if you add some -ev there still appears data out of nowhere. The only solutions are either to use tone curves or do a "one-shot hdr" assembly in enfuse or similar.

Also note that highlight recovery in LR is very different for crop & ff sensors; on crop it's stronger but tends to introduce real clipping, on ff it's weaker but never loses the natural roll-off into clipped areas.

[Luiz Roberto dos Santos]

a1ex, the RT team develop a algorithm with similar result: http://rawtherapee.com/forum/viewtopic.php?f=7&t=4500

Very good results! 
The actual RT 'color propagation' algorithm is good, but create much chroma artefact and is very [very] slow...

There are denoising algorithms that work well, but they come at the expense of to much fine detail loss for myself.

Any news about RawImageCleaner by Reiner Wittmann (aka kassandro)?
I test this, many times, and the results is incredible... really.
The stratagy of using the RemoveGrainHD on non-interpolate mosaiced data is really genial.
I really want to see that, one day, implemented and working 'smooth'.

[just idea]Maybe this could be implemented in the future at some command line tool for processing .MLV:

MLV2DNG > DNGRecoverEdges (don't make difference?)> Fix possible vertical banding> Fix possible dead pixels> RawImageClearer> CR2HDR> Demosaicing (AMaZE, IGV or AHD)> Compile to some format (like Prores or Cineform 444).

[...]  much fine detail loss [...]

Do you try deconvolution algorithms? The Lucy-Richardson make a good job to 'recover' it.

[l_d_allan]

So, if we ever manage to implement dual shutter

I'm curious what "dual shutter" is.

[l_d_allan]

5d2 and Dec 14th Nightly Build.

ISO 100 seems to be the only usable “base ISO”. With 1600 as the “Recovery” ISO, it shows an estimated 2.8 EV of dynamic range (DR) gained. Super!

But ... if I select ISO 400 as the “base” ISO, and attempt to use 6400 as the “Recovery” ISO, it doesn’t seem to be working as expected. When I “Q” into the details of Dual-ISO, 6400 is shown as “Recovery” but there is only an estimated 1.1 EV gained.

When I get back to the Main “Expo” tab, the info for Dual-ISO shows 400/1600, with estimated 1.1 EV gain. I was expecting to see 400/6400 and closer to 2 to 2.5 EV of DR gained.

Am I doing something wrong? Is this “as expected” and I have a flawed understanding of Dual_ISO?

Are others seeing this behavior?

[Audionut]

Any news about RawImageCleaner by Reiner Wittmann (aka kassandro)?

No idea. 

Removegrain was/is excellent, and was used in many avisynth scripts.  I used it frequently to reduce the bitrate requirements without visual difference.

I don't keep up with denoisers, I really do not like to use them.  Everyone I have seen so far has either blurred detail, and/or produced artifacting.  I make sure to ETTR and crush blacks as needed.

I'm curious what "dual shutter" is.

Double shutter actuation for single photo.  Note that it wouldn't even need the shutter to actuate twice for a large benefit.  Being able to scan the entire frame at different ISOs  before flush would be rather excellent.

[Audionut]

Are others seeing this behavior?

I've had a few drinks, so not keen on being able to explain things in a manner than anyone could understand 
Check this link:  http://www.magiclantern.fm/forum/index.php?topic=7139.msg59971#msg59971

Notice how the DR gain reduces with increasing ISO.

[a1ex]

Max real ISO on 5D2 is 1600.

[l_d_allan]

Thanks!
That clarifies some fuzziness in my head.

BTW: Canon should have you on their payroll, and paying you Big Bucks.

Hmmmm ... wonder if "Auto-ISO" should be smart enough to take this into account?  But that's another OP for another day.

[l_d_allan]

Double (dual) shutter actuation for single photo.  Note that it wouldn't even need the shutter to actuate twice for a large benefit.  Being able to scan the entire frame at different ISOs  before flush would be rather excellent.

Now THAT sounds interesting. Almost beyond "rather excellent".

With the power of the DIGIC-5 and 5+ cpu, I would speculate that the time between read-outs of the sensor values would be in the range of milliseconds ... or possibly even milliseconds?

So ... is this more or less how it would work? ML would scan the sensor at ISO 100, and "tuck the values away somewhere". Then re-scan a millisecond later at ISO 400 and get a second set of values? And then a third at ISO 1600? On newer cameras with "Real" ISO 3200 or ISO 6400, maybe rescan at highest ISO? Or not?

But as far as how much processing this would take, the "pipeline" for prep of JPEG's would seem at least as involved. Or not?

Potentially, there might essentially be No Ghosting.  DR as good as the human eye?

[l_d_allan]

As for highlight recovery in LR: Yes, it's good, but it's too weak for many dual_iso shots because even recovery -100 has still blown whites but if you add some -ev there still appears data out of nowhere. The only solutions are either to use tone curves or do a "one-shot hdr" assembly in enfuse or similar.

As I use Dual-ISO more, I'm encountering what you describe ... need the sliders to go past +/- 100 ... "twist those dials to 11"

In ACR, Auto-Exposure seems to only go to +/- 50, and after 50, it becomes less and less linear.

I used Dual-ISO with my 5d2 and T3i last night at a Christmas concert with Very contrasty interior lighting. HDR wasn't an option (too much movement). Dual-ISO was much more able to handle the contrast compared to non-Dual-ISO.

I'm still unclear on "Best Practice" for such a situation. Sorry for being slow. I started with ISO 100, and did some trial-and-error to get a best guess. I wonder if I should have started with ISO 1600, and done ETTR?

Or start with ISO 100, and do ETT-Left?

Is this a situation where Auto-ETTR and Dual-ISO work well together? I'm aware they can be used together, but am otherwise ignorant about how these ML capabilities combine.

BTW: are you familiar with Datacolor's SpyderCube? Interesting gadget, especially the "black trap" and "chrome ball" for the potential of "by the numbers" dynamic range.

I rigged up a DIY black trap (hole in 55-250 box with old black sock inside it) to go along with my SpyderCube to simulate a spotlight situation. The SpyderCube was in the spotlight, and the DIY black trap was partially in the shade from the tripod leg. 

http://berean.zenfolio.com/misc/hC87E705#hc87e705
 
The ACR eyedropper can actually see the differences:

• DIY black-trap in shade from tripod leg = RGB(0,0,0)
• Unshaded DIY black-trap = RGB(1,1,1)
• SpyderCube black-trap in spotlight = RGB(5,6,5)
• Chrome ball in spotlight = RGB(255,255,255)

 I'm hoping it will help clarify "best practice" between my ears.

 

[Audionut]

So ... is this more or less how it would work?

No idea, I was dreaming   a1ex is using existing Canon functionality for dual-ISO, so unless the option to scan the sensor twice before flushing the data is present in the Canon code, I doubt it would ever be possible.

[Audionut]

I'm still unclear on "Best Practice" for such a situation.

There are 2 exposures.  Base ISO exposure and recovery ISO exposure.  You need to meter for both.

For your highlights, you meter off the lower ISO.  For your shadows, you meter off the higher ISO.  I personally always meter the highlights first.  So ISO 100 and check histogram for over exposure.  Then enable dual-ISO with a recovery ISO as needed.  You can use zebras to check what the recovery ISO is doing.  Where colored zebras appear, the recovery ISO is overexposed and you lose resolution.  Where grey zebras remain, shadows are still in the noise floor.

I'll run some tests myself, but don't expect them until Sat/Sun GMT+10.

Power supply died last night.  Will be a couple of days before I get a replacement.

[a1ex]

... but no doubt this is the way to go, I'm processing all my HDR as EXR recently in apps and then convert to floating point DNG in Lightroom. This is a major quality boost when doing further postprocessing over 16 bit, and the EXR-based floating point hasn't got an as hilarious filesize as floating point TIFF:

Got it working (my first experiment on 32-bit float DNG here). Can you share how you are processing the floating point DNG in Lightroom?

I've opened my DNGs in LR 5.3, boosted exposure to +5, shadows to max, and it seems their internal processing is on 16-bit integer. So right now, the only tool that can process 32-bit DNGs properly seems to be Adobe's dng_validate.exe from the DNG SDK.

[Marsu42]

I've opened my DNGs in LR 5.3, boosted exposure to +5, shadows to max, and it seems their internal processing is on 16-bit integer.

Unfortunately you seem to be correct, your fp raw dng is limted to -5....+5 ev exposure correction, while my non-raw demosaiced hdr dng files are -10....+10. Interesting the Adobe sdk supports it, I guess they didn't add it to ACR/LR yet simply because there aren't any cameras known to them that output that format yet :-(

The only solutions I can think of is to either ask an Adobe dev to add this format (that non-oss for you...) or convert the fp raw dng to non-raw, ACR/LR is able to deal with the latter, though it probably increases the flle size and you lose the ability for lossless wb correction.

[l_d_allan]

Max real ISO on 5D2 is 1600.

Anyone happen to know the max "real" ISO on the 6d?
5d3?
T3i/600d?

[1%]

5DIII - 128k
6D - 6400
others - 3200

gen 1 like 500D/50D/5DII are 1600

[Marsu42]

5DIII - 128k
6D - 6400

Are you sure?! The code in lens.h only makes a difference between digic4 (3200) and digic5 (12800), if you're correct and the 6d is also "only" 6400 this should be changed in the code.

[a1ex]

There seems to be 4 categories of ISOs (according to how they behave internally):

a) the ones that can be configured via CMOS[0] (on 5D2: 100-1600; they may be different in photo vs lv; the dual iso module uses this one as max iso)
b) the ones that use digital gain, and you can undo that gain via ML digital ISO controls (on 5D2: 6400); this is used for tricks like gradual exposure, hdr video...
c) in-between (maybe this is also hardware but it has some secondary amplifier path)
d) max ISO that can be applied via exposure override in LiveView

So it's quite complex, and the numbers from lens.h may not be 100% correct.

For practical purposes, I consider the max useful ISO to be 1600. The higher ones are not any cleaner (maybe up to 0.5 stops), and the price to pay is a large amount of clipped highlights.

[1%]

I don't really like the results I get from isos above A in general, especially when its dark. For both CR2 and raw wouldn't they be equivalent to photoshop?

[a1ex]

Well, if you compare 1600 untouched with 6400 darkened in post by 2 stops (all other variables being equal), you may see that 6400 is a tiny bit cleaner on 5D3 (maybe also on 6D).

But for that tiny bit, you lose 2 stops of highlights.

Anyway, if ETTR chooses ISO 1600 or more, it probably is the least noisy option (of course, assuming you have already maxed out shutter and aperture).

[1%]

ETTR usually makes good choices, I don't really set much anymore. Only have to override or turn it off when I just want the highlights in the dark and can't have a long exposure.

But my 128K iso is not so useable. The 6400 is still in the A range. So 5DIII has 1 "analog" iso above what 6D has.

[dmilligan]

Here's some data:
5D3:

Code: [Select]

              Table 1
-------------------------------------------------
               Apparent  Maximum     Measured
  ISO  Gain   Read Noise  signal    Dynamic range
       e/DN  (electrons) (electrons)   stops
 
   100  5.04     34.9     68900        10.9
   200  2.52     18.3     32400        10.8
   400  1.26      9.8     16200        10.7
   800  0.63      5.6      8100        10.5
  1600  0.315     3.6      4050        10.1
  3200  0.157     2.7      2030         9.6
  6400  0.079     2.5      1000         8.6
 12800  0.039     2.1       500         7.9
 25600  0.0197    2.05      250         6.9

maximum DN: ISO 100 = 15331 (includes offset)
DN offset = 2048

Pixel pitch= 6.25 microns.
22.3 megapixels.
-------------------------------------------------



src: http://clarkvision.com/articles/evaluation-canon-5diii/index.html

5D2:

Code: [Select]

              Table 1
-------------------------------------------------
               Apparent  Maximum     Measured
  ISO  Gain   Read Noise  signal    Dynamic range
       e/DN  (electrons) (electrons)   stops

   50  4.2     24.2       65700       11.41
  100  4.1     23.5       59400       11.30
  200  2.03    11.9       29700       11.20
  400  1.01     6.4       14800       11.18
  800  0.51     3.7        7425       10.97
 1600  0.25     2.5        3710       10.54
 3200  0.127    2.5        1860        9.54
 6400  0.063    2.5         930        8.54
12800  0.032    2.5         460        7.54
25600  0.016                230

 
Pixel pitch: 6.4 microns.
S/N on 18% gray card, ISO 100 = 103.
Sensor Full Well Capacity at lowest ISO: 65,700 electrons.
Sensor dynamic range = 65700/2.5 = 26,280 = 14.7 stops.
ISO at unity gain (scaled to 12 bit) = 1600 (14-bit unity gain = ISO 404).
Low Light sensitivity Factor: 640.
Apparent Image Quality, AIQ = 109
All data derived by R. Clark, December, 2008.



src: http://clarkvision.com/articles/evaluation-canon-5dii/index.html

[l_d_allan]

Well, if you compare 1600 untouched with 6400 darkened in post by 2 stops (all other variables being equal), you may see that 6400 is a tiny bit cleaner on 5D3 (maybe also on 6D).

But for that tiny bit, you lose 2 stops of highlights.

Soooooo ... is it a valid interpretation that on my DSLR's (5d2, T3i/600d, and 6d), it is "best practice" to limit ISO to 1600 max? Even on the 6d? Then lighten in pp? I use ACR and LR.

[a1ex]

That's what I do.