Dual ISO - massive dynamic range improvement (dual_iso.mo)

[Marsu42]

Is there a way to have dual iso enabled in photo mode and not in video mode?

Nope, but you could add a feature request so the idea can be discussed on its own and doesn't get lost along the way. It's easy enough to implement, but would add yamlmo (yet another ml menu option).

[awesnap]

First time poster here.

I would like to start by saying a super huge THANK YOU to the Magic Lantern team, i've been hesitating to make the jump, but a spit of inspiration hit me last night and I installed ML and i'm VERY impressed!!

On the Dual ISO topic, I have a question.
Maybe i'm not doing something right, but....
In the EXPO menu, where the option for Dual ISO lay, if the camera's iso is at (lets say) 400, I want to do 3 stops up (3200) to get a good amount of low light coverage.  I enable this, but when I exit out of that menu, it still reads 400/1600. Then, if I goto the main ML menu, and go all the way to the right, it shows me everything thats modified (which is only DUAL iso) and in this menu it reads DUAL ISO 400/1600 (recovery ISO 3200) under that.

Also, I've noticed that the "Dynamic Range Gained" never goes up from 1.1EV when I change the recovery ISO from 1600 and up.

So I apologize for the noob question, but..
If i'm setting my Dual ISO at 3200 Recovery ISO,
Is it really stuck at 1600 ISO, or is it actually doing 400/3200?

[ayshih]

Is it really stuck at 1600 ISO, or is it actually doing 400/3200?

It's really doing ISO 400/1600.  You haven't specified what camera model you're using, but the available recovery ISOs are restricted depending on the particular model.  I know that my 50D tops out at ISO 1600.  The technical reason for this restriction is that Dual ISO is implemented by changing the CMOS-gain registers, while the ISOs greater than 1600 on the 50D have the same CMOS gain as ISO 1600 (a different amplifier gets you to the higher ISOs).

[Audionut]

That and ISOs above 1600 on most cameras are useless.

[Marsu42]

It's really doing ISO 400/1600.

I admit the necessity for this is beyond me, if I wouldn't be so busy atm I'd add a patch to my local ml that locks dual_iso @iso100 - if I need faster shutter speeds the return of invest diminishes and I simply bite the bullet and shoot in default raw setting w/o the need for cr2hdr postprocessing time and white balance hassle.

[Audionut]

I'm happy with 100/800.  And I suspect with the ADTG gains, using 200 or even 400 ISO as the base won't be so bad, it will just be that pesky photon noise.

To me, postprocessing isn't a hassle, having to redo the shoot or dig detail out of the noise is 

edit:  BTW, CMOS values 663, 773, 883, 993 produce some other ISO values.  I can't recall exactly, but with 1 of them I got ISO 300 and another looked to be over ISO 3200.

edit2:  I would have been using 603, 703, 803, 903 for the recovery ISO.

[Marsu42]

I'm happy with 100/800.  And I suspect with the ADTG gains, using 200 or even 400 ISO as the base won't be so bad, it will just be that pesky photon noise.

With recent cr2hdr versions, I also have good experiences with 100/1600 resulting in 14.5ev dr, that's on 6d...

... at for the effect of mini_iso I still have no clue at all how much this will really boost dual_iso, but I hope sooner or later alex will tell the user crowd :-). In the next days, I'll calibrate the 6d for the latest mini_mo so at least this part is covered on all current camera models.

[a1ex]

edit:  BTW, CMOS values 663, 773, 883, 993 produce some other ISO values.  I can't recall exactly, but with 1 of them I got ISO 300 and another looked to be over ISO 3200.

Yes, would be very nice to map their approximate ISO and dynamic range and seeing if they are worth using. I guess raw spotmeter on a static scene should be enough to estimate the ISO.

...yamlmo

Config presets. They need some major refactoring to remove the restart requirement (basically, attaching the toggle functions to the config entries instead of menu entries), but it's doable (as long as the two presets use exactly the same set of modules loaded). Module reloading on the fly is next to impossible to do without resource leaks IMO.

... at for the effect of mini_iso I still have no clue at all how much this will really boost dual_iso

I have a clue here: http://www.magiclantern.fm/forum/index.php?topic=10111.msg97541#msg97541

[Audionut]

Yes, would be very nice to map their approximate ISO and dynamic range and seeing if they are worth using.

I was getting different results up through 1133, 2233 etc also.  I haven't had a chance to look at the data in a reasonable fashion and mainly just mentioned it so I didn't forget about it.

Here is the output of 1 of them (I think this is 663 (603)).

Code: [Select]

Noise levels    : 5.10 4.29 4.32 5.06 (14-bit)
ISO difference  : 2.51 EV (569)
Black delta     : -3.05
Dynamic range   : 11.32 (+) 11.07 => 13.58 EV (in theory)
ISO overlap     : 5.8 EV (approx)
Noise level     : 3.16 (16-bit), ideally 3.16
Dynamic range   : 13.76 EV (cooked)
I was using ADTG gains and CMOS[4] so you can ignore the ISO value.  But this is a half stop between 2 EV and 3 EV.  The DR numbers were in the middle of the full stop EVs also

My first thought was increasing the ISO overlap.  For me, 100/1600 is generally a little to far apart, 100/800 generally gives me almost the same results (maybe I don't dig into the shadows enough, or I am more inclined to just crush them in post), well we get 0.4 EV back in the midtones with ADTG, so if I can find an ISO difference of 3.5 EV this should give me almost the same DR as 100/1600 with another 0.4 EV (0.8 EV total vs Canon ISO) in the midtones.

Here are the ISO results either side of the above.

Code: [Select]

Noise levels    : 4.77 4.31 4.29 4.76 (14-bit)
ISO difference  : 1.96 EV (388)
Black delta     : -9.79
Dynamic range   : 11.32 (+) 11.17 => 13.13 EV (in theory)
ISO overlap     : 6.4 EV (approx)
Noise level     : 4.69 (16-bit), ideally 4.69
Dynamic range   : 13.19 EV (cooked)

Code: [Select]

Noise levels    : 5.52 4.30 4.32 5.54 (14-bit)
ISO difference  : 2.97 EV (781)
Black delta     : -1.11
Dynamic range   : 11.32 (+) 10.95 => 13.92 EV (in theory)
ISO overlap     : 5.4 EV (approx)
Noise level     : 2.58 (16-bit), ideally 2.58
Dynamic range   : 14.06 EV (cooked)
I haven't had a chance to look at the actual output yet and judge differences.  It obviously needs further investigation, but not sure I will have the time until next week.


edit:  Disregarding dual_iso, I think these other ISOs will be useful in general shooting also.  With real half stop ISOs, there should be no excuse for not getting the exposure nailed in the field 

[Audionut]

With recent cr2hdr versions, I also have good experiences with 100/1600 resulting in 14.5ev dr, that's on 6d...

Yes, but are you just trusting the DR numbers or judging the output?

100/800 vs 100/1600 gives me these typical results.

Code: [Select]

Noise levels    : 5.52 4.30 4.32 5.54 (14-bit)
ISO difference  : 2.97 EV (781)
Black delta     : -1.11
Dynamic range   : 11.32 (+) 10.95 => 13.92 EV (in theory)
ISO overlap     : 5.4 EV (approx)
Noise level     : 2.58 (16-bit), ideally 2.58
Dynamic range   : 14.06 EV (cooked)

Code: [Select]

Noise levels    : 7.57 4.34 4.35 7.54 (14-bit)
ISO difference  : 3.97 EV (1563)
Black delta     : 0.76
Dynamic range   : 11.31 (+) 10.50 => 14.47 EV (in theory)
ISO overlap     : 4.3 EV (approx)
Noise level     : 1.77 (16-bit), ideally 1.77
Dynamic range   : 14.60 EV (cooked)


I apparently gained 0.54 EV in the shadows which I can't see, but I lost 1 EV of midtone detail.
There is a point in the shadows where a pattern noise emerges, and no amount of higher ISO for the recovery seems to fix this.  I assume it's a result of not being able to get any more detail from the base ISO.  ie:  You suffer aliasing in the highlights (only 1 ISO with detail), and this pattern noise is a result of only 1 ISO having good useable detail in the shadows.

[a1ex]

Would be very interesting to plot the SNR with a dual ISO shot. However, you can't use my SNR algorithm for this, because the dual ISO output is basiclly an adaptive blend of the two ISOs. So, the blending will go towards full-res image - both ISOs - in high-detail areas to minimize aliasing and bring back resolution, and will go towards the higher ISO to minimize noise.

SNR in the log is estimated from optical black area (which gets processed in the same way as the entire image). There, the blending factor is almost entirely biased to the higher ISO (you can check by comparing output noise level with the "ideally" value, which is from the higher ISO only).

So, probably the cleanup effect should be obvious in out-of-focus and really dark areas. The estimated DR is likely the best-case improvement (I should probably simulate the worst-case one too).

Quick check with a recent test shot from Marsu.

Dynamic range   : 11.88 (+) 11.32 => 14.36 EV (in theory)
Dynamic range   : 14.60 EV (cooked)  // defaults
Dynamic range   : 12.34 EV (cooked)  // hacked cr2hdr to output fullres only

So... the real improvement was between 0.45 and 2.25?

(estimated improvement from dual ISO menu was probably 2.44 from my offline simulation at 100/800 with DxO data)

Some SNR testing would show exactly what's going on. To get the ground truth image, you can use CeroNoice with a bracketed set of images like the ones from engardeknave. You may even build fake dual ISO DNGs from two bracketed shots, just to study the behavior of the converter (I'll probably try it).

edit: here's a quick and dirty tool for creating fake dual ISOs from two regular images: fake_dual_iso.exe

[Marsu42]

Yes, but are you just trusting the DR numbers or judging the output?

I'm *only* judging the output, and maybe there's something to be gained here by heavier "single-shot-hdr" style postprocessing outside acr and 100% proper auto-ettr exposure, I usually just underexpose manually with dual_iso which is quickest for dynamic scenes.

When I enable dual_iso it's for a specific purpose: to capture at least some details in blown whites, either in a sunlit fur rim or a white spot on a dark-grey horse - and using 100/1600 instead of 100/800 lets me underexpose some more. I never use dual_iso "just" to improve the general shadow noise, for that the postprocessing time, lower detail, dual storage cr2+dng and the current wb/tint problems isn't "worth it" to me personally.

[Marsu42]

fake_dual_iso.exe

Um, zis ist an elf application, Mr. Torvalds, if you name it as .exe you're creating some expectancy under the Windows guys around here :-p

[a1ex]

F5

[Audionut]

When I enable dual_iso it's for a specific purpose: to capture at least some details in blown whites, either in a sunlit fur rim or a white spot on a dark-grey horse - and using 100/1600 instead of 100/800 lets me underexpose some more. I never use dual_iso "just" to improve the general shadow noise,

ISO doesn't fix photon noise.

But yes, dual_iso is excellent for small changes in the EV of the scene.  Expose for the highlights and let dual_iso take care of the shadows.  Lazy mans exposure 

[a1ex]

Now that I have a ground truth image, I ran a quick test on the interpolation methods to see how much resolution you are really losing in the extreme highlights or shadows (worst case):



1) ground truth (used as the high-ISO image)
2) half-resolution image (2 lines sampled, 2 lines discarded and interpolated as nearest neighbour)
3) mean23 interpolation (some sort of bilinear; fast).
4) amaze-edge interpolation (default setting in cr2hdr)

Images 3 and 4 were both interpolated from 2.

Full DNGs:
DUAL7603.CR2 from engardeknave as ground truth
out.dng (fake dual ISO, open with dcraw)
nointerp.dng
mean23.dng
amaze-edge.dng

To get these DNGs, I've modified the cr2hdr source to disable all other processing except interpolation.

To get these crops:

Code: [Select]

ufraw-batch --temperature=4000 --exposure=4.5 DUAL7603.CR2   --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 ground-truth.jpg
ufraw-batch --temperature=4000 --exposure=5.5 amaze-edge.dng --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 amaze-edge.jpg
ufraw-batch --temperature=4000 --exposure=5.5 mean23.dng     --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 mean23.jpg
ufraw-batch --temperature=4000 --exposure=5.5 nointerp.dng   --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 nointerp.jpg

And now, a clear proof that we have reached the 16-bit limit (so adding floating-point DNG output may be a good idea). Here's the output from amaze-edge, first image is the same as before, the others were darkened before saving the DNG by 2, 3 and 4 stops (as it happens with ISO 100/400, 100/800 and 100/1600):



edit: half of the issue was that my modified ufraw sacrifices 2 shadow bits for clip=film, so I've re-created the images with --clip=digital which doesn't have the bug. The issue is still there, but not quite as bad.

Code: [Select]

ufraw-batch --clip=digital --temperature=4000 --exposure=5.5 amaze-edge.dng     --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 amaze-edge-0ev.jpg
ufraw-batch --clip=digital --temperature=4000 --exposure=7.5 amaze-edge-2ev.dng --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 amaze-edge-2ev.jpg
ufraw-batch --clip=digital --temperature=4000 --exposure=8.5 amaze-edge-3ev.dng --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 amaze-edge-3ev.jpg
ufraw-batch --clip=digital --temperature=4000 --exposure=9.5 amaze-edge-4ev.dng --out-type=ppm --output=- | convert - -crop 500x500+1790+1435 amaze-edge-4ev.jpg

Full DNGs:

amaze-edge-2ev.dng
amaze-edge-3ev.dng
amaze-edge-4ev.dng

[a1ex]

Repeated the experiment with DUAL7605.CR2 as overexposed image (considered ground truth for shadows), and the previous one (DUAL7603.CR2) as the underexposed image. This example is a bit closer to reality, no more deep shadows, no more bit problems. I'm showing 100% crops.

Ground truth (overexposed image), and another one underexposed by 4 stops with shutter (so the overlap is similar to 100/1600, but the noise difference is roughly 1 stop larger):


Interpolations:
nearest neighbour (half of the lines from overexposed images), mean23, amaze-edge:

(notice the detail reconstructed by amaze-edge from only half of the lines)

Full-res and half-res blending:


Full-res simply combines both exposures after scaling and offseting them to match. Notice some more detail recovered, at the expense of noise. This is the worst case performance regarding DR.

Half-res blending is a weighted average between the two exposures (after interpolation with amaze-edge), as described in the PDF, figure 6. The noise is much better, at the expense of aliasing and lost detail.

And the final output is basically a weighted average between the last two:

Code: [Select]

ufraw-batch --temperature=4000 --exposure=4.5 DUAL7603.CR2     --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 underexposed.jpg
ufraw-batch --temperature=4000 --exposure=0.5 DUAL7605.CR2     --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 ground-truth.jpg
ufraw-batch --temperature=4000 --exposure=1.5 nointerp.dng     --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 nointerp.jpg
ufraw-batch --temperature=4000 --exposure=1.5 amaze-edge.dng   --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 amaze-edge.jpg
ufraw-batch --temperature=4000 --exposure=1.5 mean23.dng       --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 mean23.jpg
ufraw-batch --temperature=4000 --exposure=5.5 fullres.dng      --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 fullres.jpg
ufraw-batch --temperature=4000 --exposure=5.5 halfres.dng      --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 halfres.jpg
ufraw-batch --temperature=4000 --exposure=5.5 final-output.dng --out-type=ppm --output=- | convert - -crop 900x600+1400+1360 final-output.jpg

Caveats:
- This was a fake dual ISO, done with shutter bracketing, not ISO (if you have a similar bracket with ISO, let me know)
- Still, the results are valid regarding resolution and noise recovery (how close it can get to the overexposed image?)
- I'm only showing the shadow performance (relevant for noise); feel free to experiment with highlights, but I think aliasing it's the same
- For some reason, my DNG files in ufraw are rendered 1 stop darker than CR2's with the exact same content (figure out why; for now I added 1 stop of exposure compensation when developing the DNGs)

My conclusions:
- deep shadows will benefit from more bits or floating point output (from previous post)
- there is room for improvement regarding noise in low-detail areas (compare final output with half-res blending)
- resolution loss is noticeable at 1:1, so it's not a substitute for tripod bracketing (compare final output with ground truth)
- however, resolution loss in worst case should not be obvious in practice (compare amaze-edge with ground truth; note that amaze-edge interpolates from half of the lines). Also compare amaze-edge with final output and notice how resolution improves in midtones.
- when you have to underexpose in order to capture the highlights, go ahead and use it (compare final output with underexposed)

[Audionut]

You've increased photon noise with shutter.  The results would be even better without it.

The shadow areas in Full Res and Weighted Average blends I find extremely distracting.  It's the noise I was describing earlier also.

If you could apply half res blending on flat textures, and apply weighted average blending on the detailed textures, with some edge detection, and weight the blending towards full res on edge detail, that should help to give the best of both worlds?

[a1ex]

Yes, the weighted average is weighted by both exposure and local details (so on the underexposed res charts from Ted it will increase noise but it will avoid aliasing). In midtones I use only full-res, but this balance can be improved.

I would start by tweaking fullres_start, fullres_transition, and playing around with alias map. These parameters are not exposed on the command line (you can turn off alias map, but here it doesn't improve the noise, so the first two parameters are the first suspects).

If you have a set of brackets with ISO, I can use that one instead.

[Marsu42]

- resolution loss is noticeable at 1:1, so it's not a substitute for tripod bracketing (compare final output with ground truth)
- however, resolution loss in worst case should not be obvious in practice (compare amaze-edge with ground truth; note that amaze-edge interpolates from half of the lines). Also compare amaze-edge with final output and notice how resolution improves in midtones.
- when you have to underexpose in order to capture the highlights, go ahead and use it (compare final output with underexposed)

This - good summary and exactly my conclusion after using dual_iso for some time, it's terrific for moving high-dr scenes and nice for added safetly if there's no time for a 100% proper exposure.

From my shots, the resolution loss shows most in semi-blown highlights with fine details, at least my human perception isn't tuned to shadow detail - as long as it's not clipped to black, it's fine.

[a1ex]

it's terrific for moving high-dr scenes

Exactly. I don't like to carry a tripod, and for hand-held shots with sun in front of you it's essential.

I'm looking into increasing the bit depth at postprocessing stage and overcoming the 16-bit limitation, stay tuned.

[Marsu42]

I'm looking into increasing the bit depth at postprocessing stage and overcoming the 16-bit limitation, stay tuned.

I hope it'll work in acr, I remember you having problems with the tags or the format was only supported in the dng sdk?

Anyway, I'm looking forward to another opportunity to re-convert my dual_iso cr2 stock, adding up how many times I did that I contributed significantly to global warming by now :-p

[Erik]

@A1ex - sorry for not responding sooner, and thanks for the help.

@erik: it's indeed a white level problem. I should provide a way to override it in the command line, because right now it's autodetected. It's important to have it correct - or a little underestimated - during the conversion; I've changed it in white_detect_brute_force to 10000 and the output seems fine.

Is there a way to pass a saturation max and white point level argument to cr2hdr?

And is there room to add WP_delta/time variable?

[a1ex]

1) you quoted the answer
2) what is a "WP_delta/time variable" ?

[Audionut]

and for hand-held shots with sun in front of you it's essential.

Anything above 8 EV of dynamic range is essential.  Once you get past 6 EV of DR, you start to gain benefits. 

Yes, the weighted average is weighted by both exposure and local details (so on the underexposed res charts from Ted it will increase noise but it will avoid aliasing). In midtones I use only full-res, but this balance can be improved.

I would start by tweaking fullres_start, fullres_transition, and playing around with alias map. These parameters are not exposed on the command line (you can turn off alias map, but here it doesn't improve the noise, so the first two parameters are the first suspects).

The trick will be applying halfres in the shadows for the least noise, since this will increase the true DR of the scene, not just the numbers, and applying fullres on edge detail to prevent aliasing.  I'll start tweaking and see if I can find something that's any better.