Dual ISO - massive dynamic range improvement (dual_iso.mo)

[a1ex]

Use raw zebras. Solid color appear where both ISOs are clipped, weak zebras appear where only one ISO is clipped (half-res) and no zebras appear where the scene is exposed properly.

In shadows, the issue is not as bad because data from the lower ISO can be still used, even if it's noisy. But in highlights, clipped data is pretty much synonymous to "I have no idea what might be there" (well, a clipped pixel does give a lower bound for guessing its value, but that's pretty much it).

About your DNG:
- I couldn't reproduce issue 1 (check your debayering method, I tried all of the methods from ufraw and rawtherapee, and only eahd seemed to struggle a little)
- issue 3 should be handled by the amaze-edge interpolation, can you upload the interlaced DNG so I could troubleshoot further?

[stevefal]

I used ACR to debayer.

Here's the original: http://popspring.com/drop/M21-1222_00038_interlace.dng

[stevefal]

Use raw zebras. Solid color appear where both ISOs are clipped, weak zebras appear where only one ISO is clipped (half-res) and no zebras appear where the scene is exposed properly.

I assume that by 'solid color' you mean black zebras?

By this I would expect black zebras when clipping at 100/100, since both ISOs must be clipped. But that's not the case, as the initial zebras are green.

In either case, I can't corroborate the statement through experimentation. Here is my repro:

1) Set Dual ISO to 100/100
2) Open iris until first zebras appear
3) Close iris 1/3 stop (zebras disappear)
4) Set Dual ISO to 100/800
5) Expect: zebras to appear, since the ISO 800 lines must now be clipping
6) OBSERVE: no zebras appear.

[a1ex]

Reproduced (it's a bug, probably over-optimization for speed).

Workaround: take a still picture to check the exposure.

[a1ex]

Got some improvement in edge aliasing.

Before/after:


I've also included an anti-posterization trick, which should improve shadow detail a little (no samples yet).

Updated binary: cr2hdr-20bit.exe

Did not run any tests, so watch out for regressions.

edit: I think I've just created a regression, bug #1 in stevefal's sample.
edit2: fixed and reuploaded.

[dubzeebass]

a1ex, can you build one for Mac possibly?

[a1ex]

Nope, but I uploaded the source code too.

[dubzeebass]

Call me a dummy but can you paste the link here? I need a project for today and compiling a program would be it!

[g3gg0]

old algorithm
new algorithm

looks a lot better at the edges.
image is a bit darker, but rarely visible.

good job

btw here is the windows .exe for this build.
use it on your own risk

[IliasG]

Updated binary: cr2hdr-20bit.exe

Did not run any tests, so watch out for regressions.

edit: I think I've just created a regression, bug #1 in stevefal's sample.
edit2: fixed and reuploaded.

These last versions give many totally dark pixels (I mean absolute 0 without subtracting BL) spread over the frame and the OB areas,
- and the third line up from the bottom is corrupted with most pixels again at level 0 and some at 65535 ..

Sample is http://filebin.net/qlh1lwzjwt/DUAL3185.CR2

[stevefal]

edit: I think I've just created a regression, bug #1 in stevefal's sample.
edit2: fixed and reuploaded.

I haven't been able to get cr2hdr-20bit.exe working yet. I use cr2hdr for OSX.

In either case, I want to point out that the aliasing I saw in #1 of my sample became prominent after I lifted the shadows. So I'm not sure how the fix looks under the same circumstances.

[Marsu42]

Bug: I can confirm latest 20bit outputs the last scanline of the dng as trash (I tested both 6d & 60d). The latest version also has some other regressions (i.e. I think it used to work better, but I didn't test all older cr2hdr) with detecting the white point on some tricky shots, I already pm'ed alex about that.

I am cross-linking a sample cr2 archive here, originally for the dual_iso wb thread, but I also made this archive for alex to have some 6d (vs. 60d) samples with a proper color checker. Beware, 600mb archive:

NEW FIXED ARCHIVE (60d shots were taken with mini_iso by mistake): https://bitbucket.org/Marsu42/ml-aiso/downloads/dual_iso_samples_6d_60d_FIXED.zip

[Luiz Roberto dos Santos]

Bug: I can confirm latest 20bit outputs the last scanline of the dng as trash

Same with 600D (g3gg0 compile).

[combat squirrel]

Ok I dont get what im doing wrong, I drag the files over cr2hdr and its not doing anything? confused.

[Audionut]

old algorithm
new algorithm

looks a lot better at the edges.
image is a bit darker, but rarely visible.

Is this from a1ex build above, or your own code?

The reduction in aliasing is excellent.

[g3gg0]

Is this from a1ex build above, or your own code?

The reduction in aliasing is excellent.

this is of course alex' latest change at this time.
(i uploaded a compiled one as his original post hadnt an .exe D/L)

[Luiz Roberto dos Santos]

Ok I dont get what im doing wrong, I drag the files over cr2hdr and its not doing anything? confused.

The g3gg0 compile is command line, no bath (see a1ex compile above it).



@a1ex

Is possible do a software for apply the function "--soft-film=%f" for non-dual_iso photos? I really like it, but sometimes I don't shot dual-iso...
For simulating analog photos, the presets do a curve push the white point to down, and the information is loss, but with soft-film, no.

[a1ex]

Yes, my modified version of ufraw (ufraw-mod) uses the same soft-film curve.

OT1: SET+MainDial exposure compensation from ML uses the same curve.

OT2: I submitted it to ufraw a few months ago, but I guess my samples were not very convincing, or I submitted it in the wrong place, because there was no feedback from them.

[Luiz Roberto dos Santos]

Yeah, I see your mod of UFRAW, really impressive. But, I think putting the UFRaw in the middle of the workflow is a little bit complicated. I mean, obviously for images refined process ("fine art", II dont like this term), this is very useful, but for editing events (aka weddings, birthdays, etc), I think it is inefficient (for speed processes).
I was thinking of a bin batch file, like a cr2hdr, but that only applied their algorithm for soft-film one would export lossless DNG. Would it be possible? I personally consider very useful to use, for example, with the presets VSCO for a wedding or something.

[Audionut]

OT2: I submitted it to ufraw a few months ago, but I guess my samples were not very convincing, or I submitted it in the wrong place, because there was no feedback from them.

Looks like the 2 developers are no longer active.

[a1ex]

for editing events (aka weddings, birthdays, etc), I think it is inefficient (for speed processes).

What could be more efficient than a batch process that you start, go out and have a beer, come back in a few hours and find everything almost done?

www.magiclantern.fm/forum/index.php?topic=7022

(well, still have to automate photo selection, white balance, rotation, cropping, maybe denoising, but at least I get the colors right out of the box for 80% of pics, mixed normal and dual iso files)

(I had an attempt to automate rotation with the built-in electronic level - partial success, the implementation was only good as proof of concept, was imprecise and crashing often)

Lossless DNG: interesting idea (some sort of a raw pre-cooker). This could include many other things like FPN correction, deflicker (via the soft-film curve), highlight recovery, bad pixel fix, raw operations like average/median/max and so on.

BTW, you can hack the cr2hdr source to accept regular cr2's (just trick it into believing it's a dual ISO file, let it accept ISO 100/100, enter some valid line configuration, like 0,1,1,0, and optionally skip all the processing and jump to output).

[a1ex]

A few updates to cr2hdr-20bit.exe

- new exposure matching algorithm (no longer using histogram matching, but medians - should be more robust to noise and outliers)
- new white level heuristic (10th max for low ISO, 100th max for high ISO, minus safety margins - should fix cases where the signal doesn't clip harshly to white)
- fixes green tint (wrong black level) on 650D and maybe others
- fixes shots heavily overexposed at the top edge (6D and maybe others)
- fixes a line of gibberish at the bottom
- a tiny bit less noise (nothing worth showing though)
- from ayshih: fixed EXIF WB info to match Canon tags (note: Canon WB tags may be wrong with dual iso, but at least they will get copied to the DNG)

These last versions give many totally dark pixels (I mean absolute 0 without subtracting BL) spread over the frame and the OB areas

Could not reproduce this one, can you show a crop where I should look?

[dubzeebass]

For any Mac user, if you have Wine installed, move cr2hdr-20bit.exe, dcraw.exe, and exiftool.exe to an empty folder, move all your Dual-ISO CR2s to that folder, open up Terminal and write "wine cr2hdr-20bit.exe *".

It works.

Proof...

[IliasG]

Could not reproduce this one, can you show a crop where I should look?

The crop, used exe , and two dngs one with the guilty cr2hdr-20bit (did not try the last one) and one with the official cr2hdr from the first post (no problem there).
Edit: black pixels exist with the last version also, not sure if they are the same ..

http://filebin.net/3379r86b43

http://filebin.net/3379r86b43/dual3185-patched.dng-crop00-738620-300.2.png
crop is 0,0 - 738,620 .. there are at least four black pixels visible in this crop ..

With the last build ..
- the crappy line at the bottomn disappeared
- (after excluding the 2 top lines) there are 122-125-126-130 black pixels in R-G1-B-G2 same as with the previous 20bit versions

[a1ex]

With b13fcd5 (original 20-bit version), the darkest pixel from (5:620, 1:740) is 7884. With e6bba36 (today's version), it's 8090. Checked with dcraw -4 -E DUAL3185.DNG followed by octave:

Code: [Select]

im = imread('DUAL3185.pgm');
crop = im(5:620, 1:740);
min(crop(:))