Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo)  (Read 2157116 times)

painya

  • Senior
  • ****
  • Posts: 321
  • 6d 50d 550d 5dc
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1550 on: November 03, 2013, 07:35:18 AM »
I'll try to update it these days. Sorry, I'll be busy with non-ML stuff till the end of the year.
No need to apologize! You are already doing something incredible, and changing the world of photography forever. Thank you.
Good footage doesn't make a story any better.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1551 on: November 03, 2013, 09:48:32 AM »
Sorry, I'll be busy with non-ML stuff till the end of the year.

Oh my, then let me take the opportunity to thank you again for all your terrific work, it's incredible how you were available nearly 24/7 during the last time - if your near absence will show one thing it's probably how important you are for ML and how helpful you are for all other devs ... so don't get hit by a bus when crossing the street, will you :-p ?

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7549
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1552 on: November 03, 2013, 11:07:11 AM »
Thanks Alex

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1553 on: November 03, 2013, 11:19:30 AM »
The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1554 on: November 03, 2013, 12:19:05 PM »
The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.

Great, and you using ML not only for test doorknob shots means that you have a good incentive to keep improving cr2hdr :-> and maybe even come up with a fix to set auto white balance and eliminate all confetti pixels from longer time exposures :-) ...

... but you found a good time for a vacation, dual_iso is working fine just like it is now, save some usability improvements in the front-end (namely auto-disable dual_iso if ev gain is too low). Btw be careful to use flash+ettl metering with dual_iso, next to needing massive -fec this seems to be somewhat erratic, but I haven't pinpointed the problem enough to submit a detailed report/proposal yet - I'll wait until 2014 then :-)

Legushka

  • New to the forum
  • *
  • Posts: 25
  • Rus
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1555 on: November 03, 2013, 03:50:39 PM »
a1ex, spasibo!
Canon eos 60d, firmware 1.1.1/sigma 10-20 + ML v2.3.RC2 + Nodal Ninja NN4

Luiz Roberto dos Santos

  • Member
  • ***
  • Posts: 162
  • be.net/LuizRoberto vimeo.com/luizroberto
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1556 on: November 04, 2013, 01:51:53 AM »
Hi Guys!
I'm following the progress here and I think it all very exciting!
I'm producing full_raw dual iso with 600D now [for professional photography], and am finding amazing what gives me in dynamic range. I would just like to ask about the exif, if there is any method that even when unaware of this, because I would like to use dcp VisionLog curves for the grade in Lightroom, however he does not identify.
I'm currently using RawTherapee for the entire workflow, can I apply the curve dcp without exif info's, however, would be of great value can make direct changes in Lightroom to process less time-consuming and less tiring.
I thank you and I can help with respect to the development, testing and ideas, I am available (although I'm not a programmer). Hug!

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1557 on: November 04, 2013, 03:00:36 AM »
The good news is that I expect to make heavy use of ML and dual ISO, so I'll try to share some pics.
Have FUN!  :)

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1558 on: November 04, 2013, 03:22:51 AM »
Just a quick question please: The cr2_hdr.exe in the first page is Oct 15. There is the denoise edition and the amazing edge edition. Is there a latest version that combines all? (with the mentioned switches).

In

https://bitbucket.org/a_d_/magic-lantern/downloads

the date is Oct 28. But change log in nightly builds suggests ... Nov 4th.

Of course I can wait for periodic updates, just asking for the place to look for the very latest cr2hdr.exe builds...

Thanks

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1559 on: November 04, 2013, 12:00:55 PM »
Here's a version that should be pretty good for everyday use.

cr2hdr-hot.exe

What's new:
- better hot pixel handling (solves the shot from Marsu42 with --really-bad-pix, also fireworks are no longer mistaken for bad pixels)
- fixed a bug that caused massive shadow aliasing in the version with many command-line switches
- minor fixes

What's not:
- I've removed the denoising option for now, because DxO puts it to shame. I'll add it back when I'll get comparable results in terms of recovered detail, without the plastic look.

Have a quick play with it; if there are no major bugs (like stopping after 30 pictures, crashing, or regressions in image quality) I'll update the first post tonight. I've tested it on 700 pics in a single batch with --fast, 100 pics with default settings, and 2 pictures in valgrind, so I believe it should be pretty solid now.

Update: after running my usual tests, I've noticed the new hot pixel algorithm (in aggressive mode) has little or no effect on resolution charts. Another milestone completed!

daancalo2013

  • New to the forum
  • *
  • Posts: 6
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1560 on: November 04, 2013, 06:08:10 PM »
Alex eres lo maximo, muy bien


I'm experimenting with some noise reduction, with an algorithm along these lines:

http://www.computersdontsee.net/index.php/post/2013/02/09/Introducing-NL-means
http://bengal.missouri.edu/~kes25c/nl2.pdf
http://scholar.harvard.edu/files/stanleychan/files/chan_zickler_lu_2013_0.pdf
http://www.cs.tut.fi/~foi/3D-DFT/ - watch the animation




According to the logs, this squeezes roughly one more stop of dynamic range, and it seems to whiten the noise too.

Download: cr2hdr-denoice.exe

The denoising step is not enabled by default; you need to specify --denoice in the command line.

Now the big question: how does this compare with state-of-art denoising software? (Lightroom, Neat Image, Noise Ninja, whatever you use). I don't have license for any of these, so I don't know if I'm reinventing the wheel or not.

For comparison, I suggest processing the CR2 with default options, then with --denoice, then postprocessing both DNGs in your favorite editor and denoise them so they have similar noise levels, and see which one holds more detail. Your quest is to find out whether my denoising step helps or hurts, and how much.

Original CR2 files for these examples: 7O4C8804.CR2, 6X8A0381.CR2 and IMG_5762.CR2 (credits: AdamTheBoy, daancalo2013 and Danne). To get the DNGs from the top row, process them with default settingss, and to get the DNGs from the bottom row, process them with --denoice.

Please note that each test run will create a different DNG (the denoising algorithm uses monte carlo sampling). It may be interesting to see what happens if you average the output from a few test runs (didn't try). Also it may be worth trying --denoice --cs5x5.

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1561 on: November 05, 2013, 03:56:53 AM »
@alex: Don't be hard on yourself regarding denoise. You are doing SUPERB WORK!

Plus, I read that DXO works on .CR2 files not DNG. So if that is the case denoise option is helpful ANYWAY!

NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.

OK that is a crazy scenario  ;D

painya

  • Senior
  • ****
  • Posts: 321
  • 6d 50d 550d 5dc
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1562 on: November 05, 2013, 04:32:40 AM »

NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.

OK that is a crazy scenario  ;D
Why would a .CR2 output be preferable over the .DNG?
Good footage doesn't make a story any better.

glubber

  • Member
  • ***
  • Posts: 118
  • 550D
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1563 on: November 05, 2013, 09:25:35 AM »
What's not:
- I've removed the denoising option for now, because DxO puts it to shame. ...
@alex: Don't be hard on yourself regarding denoise. You are doing SUPERB WORK!

Plus, I read that DXO works on .CR2 files not DNG. So if that is the case denoise option is helpful ANYWAY!
NOW, imagine if only cr2hdr could produce a .CR2 as output! That could be processed with DXO PRIME.
OK that is a crazy scenario  ;D

Wondering the same... I just downloade a trial version of DxO Optics 9 and it can't read Dual-ISO or RAW DNGs.

Are you doing the denoising on tiff/jpg in DxO?
EOS 550D // Sigma 18-200 // Sigma 18-70 // Canon 10-18 STM

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1564 on: November 05, 2013, 09:31:18 AM »
I tried on a CR2, but I'm sure you can find workarounds to open a DNG. IIRC, DXO opens the DNGs from cameras that use this format natively, so you probably need to fake the EXIF somehow.

To mangle the data from a CR2, a clumsy hack would be to modify the code from dual ISO preview (since it can burn the preview in the CR2 if you don't call the undo hook). It's nothing more than a proof of concept, not a usable solution. But it proves that it can be done.

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1565 on: November 05, 2013, 05:37:35 PM »
- I've removed the denoising option for now, because DxO puts it to shame. I'll add it back when I'll get comparable results in terms of recovered detail, without the plastic look.

Indeed, it wasn't even hard before your attempt to get a shot with zero noise but plastic look :-> ... my guess is that the DxO algorithm is rather clever and tuned/tested on a lot of sample shots by a horde of beta testers - to me this seems to be beyond simple denoising and what you can achieve with one "catch all" cr2hdr algorithm... but fee free to prove me wrong :-)

I tried on a CR2, but I'm sure you can find workarounds to open a DNG. IIRC, DXO opens the DNGs from cameras that use this format natively, so you probably need to fake the EXIF somehow.

I'll include you in my will if you manage to make DxO read raw dng or write a dng->cr2 converter (no matter the wb or metadata, just the mosaic pixels) ... I've converted all my cr2 to dng for various reasons and now cannot use the new PRIME denoising on my older shots.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1566 on: November 05, 2013, 05:51:03 PM »
Here are the duds from 7D:

http://www.filedropper.com/mg4149
http://www.filedropper.com/ual4148
I used AutoETTR + Dual + auto expo, dunno if turning on auto expo causes this.

Also sometimes the preview is jacked like on 6D:
http://www.filedropper.com/ual3926
I shot single shooting always.


Did you make any more changes to get the FS preview/etc that aren't in that repo?

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1567 on: November 06, 2013, 02:08:37 AM »
Why would a .CR2 output be preferable over the .DNG?
Because DXO PRIME does not work with .DNG.

So imagine a DUAL ISO converted file to have DXO PRIME noise reduction applied afterwards!

lureb74

  • New to the forum
  • *
  • Posts: 46
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1568 on: November 08, 2013, 02:31:51 PM »
Hello everybody!
I'm a proud owner of a 5D3 and I use everyday ML with the latest nightly build, with many many satisfaction! Specially the dual iso feature, is AWESOME!!!

I have a question: when the changelog of a nightly says a change in cr2hdr (as the last 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm), where exactly is possible to download it from?

Maybe mine is a stupid question, but I follow this thread and I never find a corresponding link to the updated version (as mentioned in the changelog), and sometimes I try to download the cr2hdr.zip (from http://acoutts.com/a1ex/cr2hdr.zip), but it is the same from october 15...

Thanks guys!
Lorenzo

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1569 on: November 08, 2013, 02:37:47 PM »

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1570 on: November 08, 2013, 02:42:27 PM »
There is also this

https://bitbucket.org/a_d_/magic-lantern/downloads/WIN_cr2hdr_exe.zip

(It contains just the cr2hdr.exe (28/10) so you have to use the other supporting .exes from a previous download like Oct 15)

lureb74

  • New to the forum
  • *
  • Posts: 46
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1571 on: November 08, 2013, 03:32:09 PM »
There is also this

https://bitbucket.org/a_d_/magic-lantern/downloads/WIN_cr2hdr_exe.zip

(It contains just the cr2hdr.exe (28/10) so you have to use the other supporting .exes from a previous download like Oct 15)

Thank you, but the change 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm is dated 5th of november! I'm just wondering if it means that there is a newer cr2hdr version somewere...

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1572 on: November 08, 2013, 04:41:23 PM »
Thank you, but the change 8813: cr2hdr: moved hot pixel correction before interpolation and fine-tuned the algorithm is dated 5th of november! I'm just wondering if it means that there is a newer cr2hdr version somewere...

That's just a change to the source code, it doesn't necessarily mean a new compiled binary has been published. You can download the source code and compile it yourself if you want, or you can wait until the various folks that publish binaries update them.

tron

  • Member
  • ***
  • Posts: 162
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1573 on: November 08, 2013, 07:40:24 PM »
@alex: your denoising work may prove even more useful than you think even if you reinvent the wheel: Apart from the fact that it removes noise from .DNG files it runs in Windows XP. I tried to setup a DXO Pro 9 demo and I got the message that Windows XP are not supported  >:(

engardeknave

  • Senior
  • ****
  • Posts: 339
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #1574 on: November 09, 2013, 10:13:03 AM »
Have you tried just copying all the tags and color profile with exiftool?