Dual ISO - massive dynamic range improvement (dual_iso.mo)

[painya]

I'll try to update it these days. Sorry, I'll be busy with non-ML stuff till the end of the year.

No need to apologize! You are already doing something incredible, and changing the world of photography forever. Thank you.

[Marsu42]

Sorry, I'll be busy with non-ML stuff till the end of the year.

Oh my, then let me take the opportunity to thank you again for all your terrific work, it's incredible how you were available nearly 24/7 during the last time - if your near absence will show one thing it's probably how important you are for ML and how helpful you are for all other devs ... so don't get hit by a bus when crossing the street, will you :-p ?

[Danne]

Thanks Alex

[a1ex]

The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.

[Marsu42]

The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.

Great, and you using ML not only for test doorknob shots means that you have a good incentive to keep improving cr2hdr :-> and maybe even come up with a fix to set auto white balance and eliminate all confetti pixels from longer time exposures :-) ...

... but you found a good time for a vacation, dual_iso is working fine just like it is now, save some usability improvements in the front-end (namely auto-disable dual_iso if ev gain is too low). Btw be careful to use flash+ettl metering with dual_iso, next to needing massive -fec this seems to be somewhat erratic, but I haven't pinpointed the problem enough to submit a detailed report/proposal yet - I'll wait until 2014 then :-)

[Legushka]

a1ex, spasibo!

[Luiz Roberto dos Santos]

Hi Guys!
I'm following the progress here and I think it all very exciting!
I'm producing full_raw dual iso with 600D now [for professional photography], and am finding amazing what gives me in dynamic range. I would just like to ask about the exif, if there is any method that even when unaware of this, because I would like to use dcp VisionLog curves for the grade in Lightroom, however he does not identify.
I'm currently using RawTherapee for the entire workflow, can I apply the curve dcp without exif info's, however, would be of great value can make direct changes in Lightroom to process less time-consuming and less tiring.
I thank you and I can help with respect to the development, testing and ideas, I am available (although I'm not a programmer). Hug!

[tron]

The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.

Have FUN! 

[tron]

Just a quick question please: The cr2_hdr.exe in the first page is Oct 15. There is the denoise edition and the amazing edge edition. Is there a latest version that combines all? (with the mentioned switches).

In

https://bitbucket.org/a_d_/magic-lantern/downloads

the date is Oct 28. But change log in nightly builds suggests ... Nov 4th.

Of course I can wait for periodic updates, just asking for the place to look for the very latest cr2hdr.exe builds...

Thanks

[a1ex]

Here's a version that should be pretty good for everyday use.

cr2hdr-hot.exe

What's new:
- better hot pixel handling (solves the shot from Marsu42 with --really-bad-pix, also fireworks are no longer mistaken for bad pixels)
- fixed a bug that caused massive shadow aliasing in the version with many command-line switches
- minor fixes

What's not:
- I've removed the denoising option for now, because DxO puts it to shame. I'll add it back when I'll get comparable results in terms of recovered detail, without the plastic look.

Have a quick play with it; if there are no major bugs (like stopping after 30 pictures, crashing, or regressions in image quality) I'll update the first post tonight. I've tested it on 700 pics in a single batch with --fast, 100 pics with default settings, and 2 pictures in valgrind, so I believe it should be pretty solid now.

Update: after running my usual tests, I've noticed the new hot pixel algorithm (in aggressive mode) has little or no effect on resolution charts. Another milestone completed!

[daancalo2013]

Alex eres lo maximo, muy bien

I'm experimenting with some noise reduction, with an algorithm along these lines:

http://www.computersdontsee.net/index.php/post/2013/02/09/Introducing-NL-means
http://bengal.missouri.edu/~kes25c/nl2.pdf
http://scholar.harvard.edu/files/stanleychan/files/chan_zickler_lu_2013_0.pdf
http://www.cs.tut.fi/~foi/3D-DFT/ - watch the animation




According to the logs, this squeezes roughly one more stop of dynamic range, and it seems to whiten the noise too.

Download: cr2hdr-denoice.exe

The denoising step is not enabled by default; you need to specify --denoice in the command line.

Now the big question: how does this compare with state-of-art denoising software? (Lightroom, Neat Image, Noise Ninja, whatever you use). I don't have license for any of these, so I don't know if I'm reinventing the wheel or not.

For comparison, I suggest processing the CR2 with default options, then with --denoice, then postprocessing both DNGs in your favorite editor and denoise them so they have similar noise levels, and see which one holds more detail. Your quest is to find out whether my denoising step helps or hurts, and how much.

Original CR2 files for these examples: 7O4C8804.CR2, 6X8A0381.CR2 and IMG_5762.CR2 (credits: AdamTheBoy, daancalo2013 and Danne). To get the DNGs from the top row, process them with default settingss, and to get the DNGs from the bottom row, process them with --denoice.

Please note that each test run will create a different DNG (the denoising algorithm uses monte carlo sampling). It may be interesting to see what happens if you average the output from a few test runs (didn't try). Also it may be worth trying --denoice --cs5x5.

[tron]

@alex: Don't be hard on yourself regarding denoise. You are doing SUPERB WORK!

Plus, I read that DXO works on .CR2 files not DNG. So if that is the case denoise option is helpful ANYWAY!

NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.

OK that is a crazy scenario 

[painya]

NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.

OK that is a crazy scenario 

Why would a .CR2 output be preferable over the .DNG?

[glubber]

What's not:
- I've removed the denoising option for now, because DxO puts it to shame. ...

@alex: Don't be hard on yourself regarding denoise. You are doing SUPERB WORK!

Plus, I read that DXO works on .CR2 files not DNG. So if that is the case denoise option is helpful ANYWAY!
NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.
OK that is a crazy scenario 

Wondering the same... I just downloade a trial version of DxO Optics 9 and it can't read Dual-ISO or RAW DNGs.

Are you doing the denoising on tiff/jpg in DxO?

[a1ex]

I tried on a CR2, but I'm sure you can find workarounds to open a DNG. IIRC, DXO opens the DNGs from cameras that use this format natively, so you probably need to fake the EXIF somehow.

To mangle the data from a CR2, a clumsy hack would be to modify the code from dual ISO preview (since it can burn the preview in the CR2 if you don't call the undo hook). It's nothing more than a proof of concept, not a usable solution. But it proves that it can be done.

[Marsu42]

- I've removed the denoising option for now, because DxO puts it to shame. I'll add it back when I'll get comparable results in terms of recovered detail, without the plastic look.

Indeed, it wasn't even hard before your attempt to get a shot with zero noise but plastic look :-> ... my guess is that the DxO algorithm is rather clever and tuned/tested on a lot of sample shots by a horde of beta testers - to me this seems to be beyond simple denoising and what you can achieve with one "catch all" cr2hdr algorithm... but fee free to prove me wrong :-)

I tried on a CR2, but I'm sure you can find workarounds to open a DNG. IIRC, DXO opens the DNGs from cameras that use this format natively, so you probably need to fake the EXIF somehow.

I'll include you in my will if you manage to make DxO read raw dng or write a dng->cr2 converter (no matter the wb or metadata, just the mosaic pixels) ... I've converted all my cr2 to dng for various reasons and now cannot use the new PRIME denoising on my older shots.

[1%]

Here are the duds from 7D:

http://www.filedropper.com/mg4149
http://www.filedropper.com/ual4148
I used AutoETTR + Dual + auto expo, dunno if turning on auto expo causes this.

Also sometimes the preview is jacked like on 6D:
http://www.filedropper.com/ual3926
I shot single shooting always.


Did you make any more changes to get the FS preview/etc that aren't in that repo?

[tron]

Why would a .CR2 output be preferable over the .DNG?

Because DXO PRIME does not work with .DNG.

So imagine a DUAL ISO converted file to have DXO PRIME noise reduction applied afterwards!

[lureb74]

Hello everybody!
I'm a proud owner of a 5D3 and I use everyday ML with the latest nightly build, with many many satisfaction! Specially the dual iso feature, is AWESOME!!!

I have a question: when the changelog of a nightly says a change in cr2hdr (as the last 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm), where exactly is possible to download it from?

Maybe mine is a stupid question, but I follow this thread and I never find a corresponding link to the updated version (as mentioned in the changelog), and sometimes I try to download the cr2hdr.zip (from http://acoutts.com/a1ex/cr2hdr.zip), but it is the same from october 15...

Thanks guys!
Lorenzo

[tron]

There is a newer download link from alex in:

http://www.magiclantern.fm/forum/index.php?topic=7139.msg86767#msg86767

[tron]

There is also this

https://bitbucket.org/a_d_/magic-lantern/downloads/WIN_cr2hdr_exe.zip

(It contains just the cr2hdr.exe (28/10) so you have to use the other supporting .exes from a previous download like Oct 15)

[lureb74]

There is also this

https://bitbucket.org/a_d_/magic-lantern/downloads/WIN_cr2hdr_exe.zip

(It contains just the cr2hdr.exe (28/10) so you have to use the other supporting .exes from a previous download like Oct 15)

Thank you, but the change 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm is dated 5th of november! I'm just wondering if it means that there is a newer cr2hdr version somewere...

[dmilligan]

Thank you, but the change 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm is dated 5th of november! I'm just wondering if it means that there is a newer cr2hdr version somewere...

That's just a change to the source code, it doesn't necessarily mean a new compiled binary has been published. You can download the source code and compile it yourself if you want, or you can wait until the various folks that publish binaries update them.

[tron]

@alex: your denoising work may prove even more useful than you think even if you reinvent the wheel: Apart from the fact that it removes noise from .DNG files it runs in Windows XP. I tried to setup a DXO Pro 9 demo and I got the message that Windows XP are not supported 

[engardeknave]

Have you tried just copying all the tags and color profile with exiftool?