Dual ISO - massive dynamic range improvement (dual_iso.mo)

Started by a1ex, July 16, 2013, 06:33:50 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

escho

I played a bit with dual-iso in video-mode on my 600D. (dual-iso compiled from latest sources of TL2.0 on Linux). Just a little report:

Compilation on Linux:
No problem for an english system. But with my german language-settings, there is this pgm_with-error using cr2hdr. I made a workaround for me for this problem. I asked, whether this could be coded in the way, I told in this thread: http://www.magiclantern.fm/forum/index.php?topic=7693.0 . But i got no answer, so I guess, this way is not the best way and not wanted.

Colors:
Much cleaner colors on raw-vido than on mov-file. And dual-iso let a lot of noise disappear. Great!

Disadvantage:
The hot-pixels get much more visible  using dual_iso.
On the mov-file, canon map most of these pixels out. In the RAW-file, these hotpixels can be surpressed using rawtherapee, other opensource linux-software, I know, is not able to render them out effectively. But using dual-iso, there is a little problem:
In the dng-files from raw2dng (./raw2dng *.RAW ) these pixels cannot be surpressed anymore. and in the output-files of cr2hdr (./cr2hdr *.dng ) I also had no success in deleating the hot pixels.

For me this doesn´t matter, because I only make videos for my astronomical shootings. And after stacking 5000 frames of saturn there is no more noise in the picture and all the hotpixels have gone magically :)

This was my little report of using dual-iso with my 600D in video-mode

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

laneo

Hi there.
I'm trying to get rid of the horizontal lines but i can't.
I'm on a mac. I'm using the last version of raw2dng *0.13).
When i drop the dng files to the last version of cr2hdr it says: name.dng File not supported.
I tried creating the dng files using the last version of RAWMagic (beta7b). CR2HDR doesnt work with those dng files.
Am I missing any steps or it should be working ?

lourenco

I was taking photos at the local drag strip today. I accidentally left dual ISO on. Sometimes I am worried about certain custom paint designs not coming out correctly, so I turn it off when needed. I very happy to see this photo turned out great! This is not an extreme dual ISO example. The owner of the car bought multiple prints of this photo.

5D Mark III, CF Lexar 1000X 32GB, 24-105 F4L

engardeknave

Wouldn't it be easy to multi-thread when batch processing? I made a quick script to do this. It's kind of stupid, though. It makes a new folder for each .cr2, copies the .cr2 and cr2hdr/dcraw/exiftool to that folder and then runs cr2hdr for each one. I do like 10-15 files at once this way on my i5 3570k. I haven't really timed it, but it doesn't take long. There's got to be a less dumb way of doing this though.

It would at least make scripting a lot easier if the temp files were named something different for each file being processed, so you don't have to put everything in separate folders to accomplish the above.

Shizuka

pretty much the cheap way to do this is to use a multithread capable dispatcher + modify cr2hdr to use temp files named from the CR2's checksum. it's not really possible to speed up cr2hdr if dcraw is not multithread-capable (amdahl's law etc)

mediavr

Dual ISO is really good for shooting interactive 360 panoramas -- where you constantly have to deal with high dynamic range subjects and these look really bad in 360 panoramas if there are large areas with blown out highlights or shadow areas without details. Here is one I shot on the weekend -- 5DMkIII, 100/1600
http://www.mediavr.com/botgardens/botgardens1.htm

To extract all the detail I could from the dngs I made 3 "exposures" jpgs from each dng with ACR -- for shadows, midtones and highlights and then loaded all image variants into the stitching program I use (PTGui). This can use Enfuse, which is an exposure blending plugin
http://wiki.panotools.org/Enfuse
also available apparently as a plugin for Lightroom
http://www.photographers-toolbox.com/products/lrenfuse.php

I think the combination of Enfuse plus extracted exposure variants can be a useful technique for dealing with the vast dynamic range available in Dual ISO dngs


zuzukasuma

Quote from: engardeknave on September 09, 2013, 08:38:51 AM
Wouldn't it be easy to multi-thread when batch processing? I made a quick script to do this. It's kind of stupid, though. It makes a new folder for each .cr2, copies the .cr2 and cr2hdr/dcraw/exiftool to that folder and then runs cr2hdr for each one. I do like 10-15 files at once this way on my i5 3570k. I haven't really timed it, but it doesn't take long. There's got to be a less dumb way of doing this though.

It would at least make scripting a lot easier if the temp files were named something different for each file being processed, so you don't have to put everything in separate folders to accomplish the above.

I'm using multiple virtual machines for this job, now 4 virtual 1 real windows 7 crunching numbers to process photos***.



***for those who still wants multithread processing, you must use industrial grade harddrives or ssds for this job. temp file and final dng file at multithread kills the drive. I'm using intel 520 series, which has best price/performance/security ratio on the market.
in a complicated relationship with eos m.

Audionut

Most of it is memory transfer/cpu processing.  Having 1 drive for each execution will certainly help though :)

Why are you using virtual machines with all of their overhead?  Make 5 (or whatever) copies of the executable in the base OS.

edit:  Any decent HDD connect via sata should be fine.

zuzukasuma

Quote from: Audionut on September 09, 2013, 11:51:30 AM
Most of it is memory transfer/cpu processing.  Having 1 drive for each execution will certainly help though :)

Why are you using virtual machines with all of their overhead?  Make 5 (or whatever) copies of the executable in the base OS.

it doesn't work, I've tried every possible combination. I have a network server has 5 drives but my pc has only 2 gigabit connections, still a bottleneck. I'm considering switching to E3 Xeon machine for better stability and faster processing. also I already have my virtual machines for other purposes now helping out crunching numbers of Dual ISO :)

*edit: virtual windows 7s have 1gigs ram only, not really overhead for the real system, theoretically I can use 7 virtual machines on 8gigs ram.
in a complicated relationship with eos m.

Audionut

Quote from: zuzukasuma on September 09, 2013, 11:54:40 AM
it doesn't work,

Check my edit above.

4 copies of the executables in different folders.  Bunch of dual ISOs in each folder.  Drag and drop just like normal.

zuzukasuma

Quote from: Audionut on September 09, 2013, 12:04:03 PM
Check my edit above.

4 copies of the executables in different folders.  Bunch of dual ISOs in each folder.  Drag and drop just like normal.

your version supports EOS M's crazy dual iso with multiple sequences?

*edit: found it. testing
in a complicated relationship with eos m.

Audionut

Huh??

100 dual ISO cr2's.
4 folders on the 1 HDD.
In each folder, a copy of each of the executables (cr2hdr, dcraw, exiftool), and 25 of the cr2's
Open 1 folder and drag and drop the cr2's onto cr2hdr.
Open the next folder and do the same
Yada
Yada

Batch processing using poor mans multithreading just like you were doing above.  Difference being, you don't need SSDs, or virtual machine, or xeons, or server racks or any else.
A decent HDD connect via sata and enough cores in the processor are all that's needed.

zuzukasuma

Quote from: Audionut on September 09, 2013, 12:12:13 PM
Huh??

100 dual ISO cr2's.
4 folders on the 1 HDD.
In each folder, a copy of each of the executables (cr2hdr, dcraw, exiftool), and 25 of the cr2's
Open 1 folder and drag and drop the cr2's onto cr2hdr.
Open the next folder and do the same
Yada
Yada

Batch processing using poor mans multithreading just like you were doing above.  Difference being, you don't need SSDs, or virtual machine, or xeons, or server racks or any else.
A decent HDD connect via sata and enough cores in the processor are all that's needed.

latest version sucked at processing



well you know the saying "the best way is you know" :)

*also, I've tried it before. unfortunately 5400rpm laptop harddrives doesn't help much with multiple instances.

**I'm switching it because of long-awaited "Intel® TSX-NI" support, today not so important but xeon for a decent price with greater performance than core i7, excellent ram support, dual gigabit ethernet and wait lower power usage? tech miracle.
in a complicated relationship with eos m.

Audionut

Quote from: zuzukasuma on September 09, 2013, 12:16:33 PM
*also, I've tried it before. unfortunately 5400rpm laptop harddrives doesn't help much with multiple instances.

I wouldn't all a 5400rpm drive decent ;)  Especially if that's in a laptop.

Let me rephrase.  And decent 7200rpm HDD connect via sata to a desktop should be fine!  That's still well under the recommendation of SSD's, virtual machines and xeons though :P

Quote from: zuzukasuma on September 09, 2013, 12:16:33 PM
latest version sucked at processing

Have you uploaded a CR2 for a1ex to look at? 
And just to clarify, latest version date stamped 3/9/13.

zuzukasuma

Quote from: Audionut on September 09, 2013, 12:23:46 PM
I wouldn't all a 5400rpm drive decent ;)  Especially if that's in a laptop.

Let me rephrase.  And decent 7200rpm HDD connect via sata to a desktop should be fine!  That's still well under the recommendation of SSD's, virtual machines and xeons though :P

sorry, I have one laptop and a home server. used 7200rpm seagate 500gb drives for once and now them in enclosures, noisy and overheating drives with only %10 performance difference? no thanks.

edit: this is the exe : https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/cr2hdr.exe

edit 2: found another exe, date 2/9/13 can kick 6 thread same time


in a complicated relationship with eos m.

Audionut

Quote from: zuzukasuma on September 09, 2013, 12:26:10 PM
sorry, I have one laptop and a home server. used 7200rpm seagate 500gb drives for once and now them in enclosures, noisy and overheating drives with only %10 performance difference? no thanks.

I agree.  But it's useful to point out that those drives are sufficient for this purpose, for those people without the funds for new toys.

Quote from: zuzukasuma on September 09, 2013, 12:26:10 PM
edit: this is the exe : https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/cr2hdr.exe

This is the official one http://acoutts.com/a1ex/cr2hdr.zip
Bug 1% about that version you are using.

zuzukasuma

Quote from: Audionut on September 09, 2013, 12:29:08 PM
...for those people without the funds for new toys.

currently I'm considering to upgrade my machines before getting new lenses or cameras. because I have t2i and EOS M, if I buy EF-M lens it'll be EOS M only, if I buy EF-S lens it'll be t2i and EOS M only. if/when I upgrade to 5D.III then all other equipment will stay dead for weeks.

edit: found the bug, my DUAL ISO files are older than normal. new version supports dBBd files, some of mine are dBdB.
in a complicated relationship with eos m.

a1ex

Quote from: Marsu42 on August 31, 2013, 05:38:38 PM
dual iso should really be combined with an ettr/ettl option to expose the first image to one side and then automatically figure out the required ev spacing for the 2nd shot, in essence minimizing the required dual_iso ev spacing.

Almost working: https://bitbucket.org/hudson/magic-lantern/commits/eb3cc38d1870

How it works:
- remember that ETTR has SNR limits for midtones and shadows (it will try really hard to meet or exceed that SNR).
- if dual ISO is not enabled, it may have to clip many highlights in order to meet that SNR. If you turn off SNR limits, it will underexpose, but keep all the highlights.
- if dual ISO is enabled, ETTR will adjust it to meet or exceed the SNR you have set, and keep all the highlights (except the specular ones, handled by "Highlight ignore")

Problem: ETTR is iterative, and it relies on the previous image (where it measures the white point, some SNR numbers and stuff like that). SNR seems to change quite a bit with dual ISO (even if it shouldn't); need to investigate why. So, at this point, ETTR with dual ISO may be a little undecided: it may say 100/800 or 100/1600.

How to use it:
- Make sure you have raw zebras and histogram enabled (so you know what's going on, what's clipped, what may be aliased...)
- Enable Auto ETTR (Always ON) and dual ISO.
- Point the camera at your scene and take a few test pics.
- If the exposure doesn't converge, try underexposing a bit. When the image is overexposed, it has to guess by how much (and it doesn't always work). When it's a little underexposed, it can compute exactly what settings are needed. When it's heavily underexposed, noise will affect the metering.

ETTR will try to:
- use a low ISOs and a low EV spacing
- meet or exceed the selected SNR minus 0.5 EV
- do not clip more highlights than you have selected for "Highlight ignore"
- do not exceed max auto ISO from Canon menu

So, if that SNR can be met without clipping highlights, the scene does not require dual ISO. If not... it does.

Enable debug info to show the math behind it.

zuzukasuma

Quote from: a1ex on September 09, 2013, 12:46:25 PM
Problem: ETTR is iterative, and it relies on the previous image (where it measures the white point, some SNR numbers and stuff like that). SNR seems to change quite a bit with dual ISO (even if it shouldn't); need to investigate why. So, at this point, ETTR with dual ISO may be a little undecided: it may say 100/800 or 100/1600....

do you think its possible to save dual iso RAW + normal JPG at the same time? maybe ETTR can use info from the JPG?

edit: or highlight LOG?
in a complicated relationship with eos m.


zuzukasuma

in a complicated relationship with eos m.

a1ex

Quotecreate and use log of previous shot for ETTR correction.

How's that different from what I'm doing now?

zuzukasuma

Quote from: a1ex on September 09, 2013, 01:16:51 PM
How's that different from what I'm doing now?

I'm not sure if we're talking about same stuff. get some rest a1ex
in a complicated relationship with eos m.

Shizuka

Quote from: zuzukasuma on September 09, 2013, 11:26:58 AM
I'm using multiple virtual machines for this job, now 4 virtual 1 real windows 7 crunching numbers to process photos***.



***for those who still wants multithread processing, you must use industrial grade harddrives or ssds for this job. temp file and final dng file at multithread kills the drive. I'm using intel 520 series, which has best price/performance/security ratio on the market.

too much work
i wrote custom utilities for this sort of thing before
http://www.mediafire.com/download/5n5pjih98b80909/MT_cr2hdr.zip
note that cr2hdr is hacked to use unique tempfiles based off input file name.

a1ex