Dual ISO - massive dynamic range improvement (dual_iso.mo)

[tron]

@alex: I have used the latest amaze_edge_5 version (downloaded a few minutes ago so it is 100% the latest):

https://www.dropbox.com/s/lqey54l0kfhsmdb/dual650_amaze-edge-5.JPG

[a.d.]

@Ivycreek
I suppose you are still on PowerPC based Mac. Sorry, no support!
I already said in the readme: "Lion and Mountain Lion".

[Ivycreek]

@Ivycreek
I suppose you are still on PowerPC based Mac. Sorry, no support!
I already said in the readme: "Lion and Mountain Lion".

LOL figures, I don't know much about Macs, it's the only place I have ever used one. This just gives me an excuse to work a little bit from home! 

[1%]

I bet all this can be re-compiled for powerpc.

[Danne]

Hi A.D. Did you update the cr2hr? Says there was something going on at bitbucket 8 hours ago but it,s still the 1.5 oct 11th version ?
Thanks
//D

[a.d.]

update:
Oh, I think it's a bug in bitbucket cache server setting. I already deleted the old file. Somehow it still provides the old version instead the new one! Therefore I ned to rename the file.

[Danne]

Million thanks A.D! and Alex of course...

[Marsu42]

From your shot, I've cleaned up some of the dots with this trick, but finding a general solution that doesn't break resolution charts is a bit harder. Also, locating hot pixels where the high ISO is overexposed is hard and I didn't try to solve it yet (a median filter with thresholding should help a bit).

I tried the "confetti" long exposure shot with the new interpolation method, and the dots are definitely better/smaller than before, but still there so I'd still consider this shot "broken" with the latest cr2hdr.

If this is indeed so hard to solve, could you give a guideline what to shots to *avoid* that result in these colorful dots so we can avoid doing dual_iso on these? The sample I provided was a 30sec daylight longtime exposure with a 10x nd filter.

[a1ex]

So far I've only noticed these dots on long exposures. Not sure if ISO 100/400 avoids them, but you can try.

[a1ex]

Another experiment: I've replaced the ISO estimation step with a simple histogram matching.

cr2hdr-histmatch.exe

Advantages:
- much faster (of course, only the ISO estimation step)
- no interpolation needed, so ISO is estimated from real pixels only
- got rid of nonlinear exposure corrections, which were probably just estimation errors from the previous method
- solves the helicopter shot
- a major step towards solving the fire shots from jOt.

There's still room for improvement in very dark shots, where the noise influences the histogram in significant ways. If you can find some theory about histogram matching for noisy images, it will be very helpful.

[a1ex]

And, finally solved the 4 dark shots from jOt!

Now I'm confident the autodetection is really robust and should handle pretty much anything you throw at it. If it doesn't, upload a CR2.

cr2hdr-histmatch2.exe

[toze3]

Hi @a1ex

So many updates, but i have a question, should i use just the last one (now the cr2hdr-histmach2) assuming this one have alll the abilities of the previous versions or all are diferente and for diferent problems ?

Thanks for the excelent work

[a1ex]

Ideally, the latest should result in the best overall output. But since it's only half an hour old, you may want to compare it to previous versions, and if you find any regressions, report them (with a CR2 attached).

[Bassaidai]

Hi Alex,

thanks for the new cr2hdr version.

I seem to have run into a different visual artefact, that consistently occurs no matter edge5 or histmatch2 exe:


(image contrast enhanced for demonstration)

The artefacts occur on a 100/1600 file, a similar 100/800 file (out of a series 100/100 to 100/1600 for shadow/noise comparison) was ok.

Please check your mailbox for the corresponding CR2 file.

[Yaros525]

Hi Alex,

latest cr2hdr-histmatch2.exe gives me interpolation lines in dark areas, while prev cr2hdr-histmatch.exe did not.

I've altered luma curve to push the effect: histmatch | histmatch2

[Marsu42]

Alex, do you think you can make cr2hdr compress the dng like acr does (I don't know the algorithm though)?

I reckon that re-converting the dual_iso cr2 might become a regular habit, and it would be nice to be able to skip also re-compressing the ml dng in Lightroom - the acr version is about half the size of the ml one. It would also be nice for people who don't use acr at all and take the ml dng as it is.

[a1ex]

@Bassaidai: should be OK now, only had to do some minor adjustments.

cr2hdr-histmatch3.exe

@Yaros525: I can't diagnose it from a JPEG. Try with this one, and if you still have trouble, send me the CR2.

@Marsu42: a script that calls cr2hdr and then adobe DNG should do the trick. I'm not aware of any open source software that can do this.

[Bassaidai]

some more tests:

both hismatch2 and edge5:

(nasty overall garbage. 100/400)

Left hismatch2, right edge5:

(ISO-line pattern, greenish color cast. 400/1600)

Both CR2s are provided, see PM.

Cheers
Bass

[Yaros525]

Alex,

cr2hdr-histmatch3.exe is Perfectly O.K. and all-around amazing! )

[engardeknave]

If there's a command line utility to compress dng my script would be well suited to do that after cr2hdr.

[a1ex]

Adobe DNG Converter.exe -dng14 IMG_1234.DNG

[papkee]

a1ex,

I can't seem to get the program to work with my RAW videos. I drag the DNGs over the program, and it briefly flashes open for a second before closing (I hate that command line programs don't stay open to alert you of an error)

I have both dcraw and exiftool in the same folder, but it just doesn't want to work. I extract the DNG's using your raw2dng_cs3x3_ahdlike_noise.exe (because Java can't install on my system) so they should work together, correct?

Thanks for the help.

[engardeknave]

Adobe DNG Converter.exe -dng14 IMG_1234.DNG

Done.

http://www.magiclantern.fm/forum/index.php?topic=8520.0

[a1ex]

I have both dcraw and exiftool in the same folder, but it just doesn't want to work. I extract the DNG's using your raw2dng_cs3x3_ahdlike_noise.exe (because Java can't install on my system) so they should work together, correct?

I didn't try both of them, but I don't think it can work in this way. Try with the plain raw2dng.

[orim]

Hi, A1ex,

I may found a bug:
new cr2hdr-histmatch3.exe is creating DNGs from non DualISO files -
please, see text output in comparison with cr2hdr-amaze-edge5.exe result:

Code Select Expand

C:\dualISO>cr2hdr-amaze-edge5.exe 2013-10-27_3626.CR2
cr2hdr: a post processing tool for Dual ISO images

Last update: af9a60f on 2013-10-24 18:44:01 UTC by alex:
cr2hdr: always use last 8 lines from top optical black bar (should fi...


Input file     : 2013-10-27_3626.CR2
Canon EOS 50D detected
Full size      : 4832 x 3228
Active area    : 4770 x 3177
White level    : 12500
Black borders  : 62 left, 51 top
Black level    : 994
ISO pattern    : BdBd GBRG
Interlacing method not supported
ISO blending didn't work

C:\dualISO>cr2hdr-histmatch3.exe 2013-10-27_3626.CR2
cr2hdr: a post processing tool for Dual ISO images

Last update: 6982824 on 2013-10-26 16:54:22 UTC by alex:
cr2hdr: fine-tuned horizontal stripe correction


Input file     : 2013-10-27_3626.CR2
Canon EOS 50D detected
Full size      : 4832 x 3228
Active area    : 4770 x 3177
White level    : 12500
Black borders  : 62 left, 51 top
Black level    : 994
ISO pattern    : dBBd GBRG
Noise levels   : 5.24 5.24 5.24 5.31 (14-bit)
Doesn't look like interlaced ISO
ISO difference : 0.00 EV (100)
Black delta    : 1.01
Dynamic range  : 11.10 (+) 11.08 => 11.08 EV (in theory)
Interpolation  : amaze-edge-chroma2x2-alias
AMaZE interpolation ...
Amaze took 3.57 s
Edge-directed interpolation...
Semi-overexpo'd: 0.00%
Deep shadows   : 2.83%
Horizontal stripe fix...
Looking for hot/cold pixels...
Cold pixels    : 359
Full-res reconstruction...
ISO overlap    : 8.1 EV (approx)
Half-res blending...
Chroma filtering...
Building alias map...
Filtering alias map...
Smoothing alias map...
Final blending...
Noise level    : 21.28 (16-bit), ideally 18.14
Dynamic range  : 11.08 EV (cooked)
Black adjust   : -7
Output file    : 2013-10-27_3626.DNG
    1 image files updated

It (cr2hdr-histmatch3) however detects "Doesn't look like interlaced ISO",
but dng is created anyway. It happened with a few of my cr2 files from today´s shooting.

orim