Dual ISO - massive dynamic range improvement (dual_iso.mo)

[escho]

I played a bit with dual-iso in video-mode on my 600D. (dual-iso compiled from latest sources of TL2.0 on Linux). Just a little report:

Compilation on Linux:
No problem for an english system. But with my german language-settings, there is this pgm_with-error using cr2hdr. I made a workaround for me for this problem. I asked, whether this could be coded in the way, I told in this thread: http://www.magiclantern.fm/forum/index.php?topic=7693.0 . But i got no answer, so I guess, this way is not the best way and not wanted.

Colors:
Much cleaner colors on raw-vido than on mov-file. And dual-iso let a lot of noise disappear. Great!

Disadvantage:
The hot-pixels get much more visible  using dual_iso.
On the mov-file, canon map most of these pixels out. In the RAW-file, these hotpixels can be surpressed using rawtherapee, other opensource linux-software, I know, is not able to render them out effectively. But using dual-iso, there is a little problem:
In the dng-files from raw2dng (./raw2dng *.RAW ) these pixels cannot be surpressed anymore. and in the output-files of cr2hdr (./cr2hdr *.dng ) I also had no success in deleating the hot pixels.

For me this doesn´t matter, because I only make videos for my astronomical shootings. And after stacking 5000 frames of saturn there is no more noise in the picture and all the hotpixels have gone magically

This was my little report of using dual-iso with my 600D in video-mode

Edgar

[laneo]

Hi there.
I'm trying to get rid of the horizontal lines but i can't.
I'm on a mac. I'm using the last version of raw2dng *0.13).
When i drop the dng files to the last version of cr2hdr it says: name.dng File not supported.
I tried creating the dng files using the last version of RAWMagic (beta7b). CR2HDR doesnt work with those dng files.
Am I missing any steps or it should be working ?

[lourenco]

I was taking photos at the local drag strip today. I accidentally left dual ISO on. Sometimes I am worried about certain custom paint designs not coming out correctly, so I turn it off when needed. I very happy to see this photo turned out great! This is not an extreme dual ISO example. The owner of the car bought multiple prints of this photo.

[engardeknave]

Wouldn't it be easy to multi-thread when batch processing? I made a quick script to do this. It's kind of stupid, though. It makes a new folder for each .cr2, copies the .cr2 and cr2hdr/dcraw/exiftool to that folder and then runs cr2hdr for each one. I do like 10-15 files at once this way on my i5 3570k. I haven't really timed it, but it doesn't take long. There's got to be a less dumb way of doing this though.

It would at least make scripting a lot easier if the temp files were named something different for each file being processed, so you don't have to put everything in separate folders to accomplish the above.

[Shizuka]

pretty much the cheap way to do this is to use a multithread capable dispatcher + modify cr2hdr to use temp files named from the CR2's checksum. it's not really possible to speed up cr2hdr if dcraw is not multithread-capable (amdahl's law etc)

[mediavr]

Dual ISO is really good for shooting interactive 360 panoramas -- where you constantly have to deal with high dynamic range subjects and these look really bad in 360 panoramas if there are large areas with blown out highlights or shadow areas without details. Here is one I shot on the weekend -- 5DMkIII, 100/1600
http://www.mediavr.com/botgardens/botgardens1.htm

To extract all the detail I could from the dngs I made 3 "exposures" jpgs from each dng with ACR -- for shadows, midtones and highlights and then loaded all image variants into the stitching program I use (PTGui). This can use Enfuse, which is an exposure blending plugin
http://wiki.panotools.org/Enfuse
also available apparently as a plugin for Lightroom
http://www.photographers-toolbox.com/products/lrenfuse.php

I think the combination of Enfuse plus extracted exposure variants can be a useful technique for dealing with the vast dynamic range available in Dual ISO dngs

[zuzukasuma]

Wouldn't it be easy to multi-thread when batch processing? I made a quick script to do this. It's kind of stupid, though. It makes a new folder for each .cr2, copies the .cr2 and cr2hdr/dcraw/exiftool to that folder and then runs cr2hdr for each one. I do like 10-15 files at once this way on my i5 3570k. I haven't really timed it, but it doesn't take long. There's got to be a less dumb way of doing this though.

It would at least make scripting a lot easier if the temp files were named something different for each file being processed, so you don't have to put everything in separate folders to accomplish the above.

I'm using multiple virtual machines for this job, now 4 virtual 1 real windows 7 crunching numbers to process photos***.



***for those who still wants multithread processing, you must use industrial grade harddrives or ssds for this job. temp file and final dng file at multithread kills the drive. I'm using intel 520 series, which has best price/performance/security ratio on the market.

[Audionut]

Most of it is memory transfer/cpu processing.  Having 1 drive for each execution will certainly help though

Why are you using virtual machines with all of their overhead?  Make 5 (or whatever) copies of the executable in the base OS.

edit:  Any decent HDD connect via sata should be fine.

[zuzukasuma]

Most of it is memory transfer/cpu processing.  Having 1 drive for each execution will certainly help though

Why are you using virtual machines with all of their overhead?  Make 5 (or whatever) copies of the executable in the base OS.

it doesn't work, I've tried every possible combination. I have a network server has 5 drives but my pc has only 2 gigabit connections, still a bottleneck. I'm considering switching to E3 Xeon machine for better stability and faster processing. also I already have my virtual machines for other purposes now helping out crunching numbers of Dual ISO

*edit: virtual windows 7s have 1gigs ram only, not really overhead for the real system, theoretically I can use 7 virtual machines on 8gigs ram.

[Audionut]

it doesn't work,

Check my edit above.

4 copies of the executables in different folders.  Bunch of dual ISOs in each folder.  Drag and drop just like normal.

[zuzukasuma]

Check my edit above.

4 copies of the executables in different folders.  Bunch of dual ISOs in each folder.  Drag and drop just like normal.

your version supports EOS M's crazy dual iso with multiple sequences?

*edit: found it. testing

[Audionut]

Huh??

100 dual ISO cr2's.
4 folders on the 1 HDD.
In each folder, a copy of each of the executables (cr2hdr, dcraw, exiftool), and 25 of the cr2's
Open 1 folder and drag and drop the cr2's onto cr2hdr.
Open the next folder and do the same
Yada
Yada

Batch processing using poor mans multithreading just like you were doing above.  Difference being, you don't need SSDs, or virtual machine, or xeons, or server racks or any else.
A decent HDD connect via sata and enough cores in the processor are all that's needed.

[zuzukasuma]

Huh??

100 dual ISO cr2's.
4 folders on the 1 HDD.
In each folder, a copy of each of the executables (cr2hdr, dcraw, exiftool), and 25 of the cr2's
Open 1 folder and drag and drop the cr2's onto cr2hdr.
Open the next folder and do the same
Yada
Yada

Batch processing using poor mans multithreading just like you were doing above.  Difference being, you don't need SSDs, or virtual machine, or xeons, or server racks or any else.
A decent HDD connect via sata and enough cores in the processor are all that's needed.

latest version sucked at processing



well you know the saying "the best way is you know"

*also, I've tried it before. unfortunately 5400rpm laptop harddrives doesn't help much with multiple instances.

**I'm switching it because of long-awaited "Intel® TSX-NI" support, today not so important but xeon for a decent price with greater performance than core i7, excellent ram support, dual gigabit ethernet and wait lower power usage? tech miracle.

[Audionut]

*also, I've tried it before. unfortunately 5400rpm laptop harddrives doesn't help much with multiple instances.

I wouldn't all a 5400rpm drive decent   Especially if that's in a laptop.

Let me rephrase.  And decent 7200rpm HDD connect via sata to a desktop should be fine!  That's still well under the recommendation of SSD's, virtual machines and xeons though

latest version sucked at processing

Have you uploaded a CR2 for a1ex to look at? 
And just to clarify, latest version date stamped 3/9/13.

[zuzukasuma]

I wouldn't all a 5400rpm drive decent   Especially if that's in a laptop.

Let me rephrase.  And decent 7200rpm HDD connect via sata to a desktop should be fine!  That's still well under the recommendation of SSD's, virtual machines and xeons though

sorry, I have one laptop and a home server. used 7200rpm seagate 500gb drives for once and now them in enclosures, noisy and overheating drives with only %10 performance difference? no thanks.

edit: this is the exe : https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/cr2hdr.exe

edit 2: found another exe, date 2/9/13 can kick 6 thread same time

[Audionut]

sorry, I have one laptop and a home server. used 7200rpm seagate 500gb drives for once and now them in enclosures, noisy and overheating drives with only %10 performance difference? no thanks.

I agree.  But it's useful to point out that those drives are sufficient for this purpose, for those people without the funds for new toys.

edit: this is the exe : https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/cr2hdr.exe

This is the official one http://acoutts.com/a1ex/cr2hdr.zip
Bug 1% about that version you are using.

[zuzukasuma]

...for those people without the funds for new toys.

currently I'm considering to upgrade my machines before getting new lenses or cameras. because I have t2i and EOS M, if I buy EF-M lens it'll be EOS M only, if I buy EF-S lens it'll be t2i and EOS M only. if/when I upgrade to 5D.III then all other equipment will stay dead for weeks.

edit: found the bug, my DUAL ISO files are older than normal. new version supports dBBd files, some of mine are dBdB.

[a1ex]

dual iso should really be combined with an ettr/ettl option to expose the first image to one side and then automatically figure out the required ev spacing for the 2nd shot, in essence minimizing the required dual_iso ev spacing.

Almost working: https://bitbucket.org/hudson/magic-lantern/commits/eb3cc38d1870

How it works:
- remember that ETTR has SNR limits for midtones and shadows (it will try really hard to meet or exceed that SNR).
- if dual ISO is not enabled, it may have to clip many highlights in order to meet that SNR. If you turn off SNR limits, it will underexpose, but keep all the highlights.
- if dual ISO is enabled, ETTR will adjust it to meet or exceed the SNR you have set, and keep all the highlights (except the specular ones, handled by "Highlight ignore")

Problem: ETTR is iterative, and it relies on the previous image (where it measures the white point, some SNR numbers and stuff like that). SNR seems to change quite a bit with dual ISO (even if it shouldn't); need to investigate why. So, at this point, ETTR with dual ISO may be a little undecided: it may say 100/800 or 100/1600.

How to use it:
- Make sure you have raw zebras and histogram enabled (so you know what's going on, what's clipped, what may be aliased...)
- Enable Auto ETTR (Always ON) and dual ISO.
- Point the camera at your scene and take a few test pics.
- If the exposure doesn't converge, try underexposing a bit. When the image is overexposed, it has to guess by how much (and it doesn't always work). When it's a little underexposed, it can compute exactly what settings are needed. When it's heavily underexposed, noise will affect the metering.

ETTR will try to:
- use a low ISOs and a low EV spacing
- meet or exceed the selected SNR minus 0.5 EV
- do not clip more highlights than you have selected for "Highlight ignore"
- do not exceed max auto ISO from Canon menu

So, if that SNR can be met without clipping highlights, the scene does not require dual ISO. If not... it does.

Enable debug info to show the math behind it.

[zuzukasuma]

Problem: ETTR is iterative, and it relies on the previous image (where it measures the white point, some SNR numbers and stuff like that). SNR seems to change quite a bit with dual ISO (even if it shouldn't); need to investigate why. So, at this point, ETTR with dual ISO may be a little undecided: it may say 100/800 or 100/1600....

do you think its possible to save dual iso RAW + normal JPG at the same time? maybe ETTR can use info from the JPG?

edit: or highlight LOG?

[a1ex]

What?!

[zuzukasuma]

What?!

create and use log of previous shot for ETTR correction.

[a1ex]

create and use log of previous shot for ETTR correction.

How's that different from what I'm doing now?

[zuzukasuma]

How's that different from what I'm doing now?

I'm not sure if we're talking about same stuff. get some rest a1ex

[Shizuka]

I'm using multiple virtual machines for this job, now 4 virtual 1 real windows 7 crunching numbers to process photos***.



***for those who still wants multithread processing, you must use industrial grade harddrives or ssds for this job. temp file and final dng file at multithread kills the drive. I'm using intel 520 series, which has best price/performance/security ratio on the market.

too much work
i wrote custom utilities for this sort of thing before
http://www.mediafire.com/download/5n5pjih98b80909/MT_cr2hdr.zip
note that cr2hdr is hacked to use unique tempfiles based off input file name.

[a1ex]

Can you share the temp file hack?