Dual ISO - massive dynamic range improvement (dual_iso.mo)

[nachordez]

Well, the previous version build up without problems in my netbook with Ubuntu, but I didn't see the 20 bit one, I'll look for it. In my desktop with opens use, it builds, but doesn't work, I have to find some time to find why... But meanwhile, with wine in my desktop one is much faster than my poor netbook with native... :-)

[surami]

I tried the "cr2hdr same levels" build, what I found in this post to post process the pieces of a spherical panorama and 2 from the 7 dngs got vertical stripes after the drag and drop (all of the 7 cr2s).

My camera was in M mode, Dual ISO 100/1600, f8, 1/30 and -1 EV, I'm using the nighlty apr. 14.

In Lightroom: shadow +69 and highlight -69. Exported to jpg 100% quality, 300dpi produces this. (deleted file, problem solved see #2031)

Am I doing something wrong or is this a bug?

[a1ex]

Looks like a bug, can you upload a CR2?

[surami]

yepp, it's here. (deleted file, problem solved see #2031)

sorry, the stripe is vertical if we see in landscape view, but my camera was in portrait direction because of the panoramic head.

[a1ex]

Could not reproduce (tried with the 16-bit one and two 20-bit versions). Can you post the conversion log?

[surami]

It happens with that cr2hdr bulid, where the same levels command is embeded.

For anyone interested, here is the build from a1ex above, with the --same-levels command parameter set by default.
https://dl.dropboxusercontent.com/u/34113196/Dual-ISO/forum/cr2hdr-same-levels.zip
This will allow you to simply drag and drop files on the exe, for auto processing with --same-levels.

With the 20-bit version and with the version what is in the main post I can't see the stripes.

How to save the conversion log and how could I compile a 20-bit cr2hdr, where the same levels command is embeded? I'm on windows.

[a1ex]

To save the conversion log, you can use copy/paste. I no longer need it; it's just FYI.

To enable drag and drop with some custom options (in this case, --same-levels), you can create a .bat file (no need to compile a custom binary). I think the syntax is something like this:

Code: [Select]

cr2hdr --same-levels %*

Found the problem:

Code: [Select]

The first good revision is:
changeset:   10751:4c6fa8b9faba
branch:      cr2hdr-20bit
user:        a1ex
date:        Wed Apr 09 16:44:35 2014 +0300
summary:     cr2hdr: fix overflows in RGGB/GBRG/bright/dark detection


[ShaunWoo]

updated the folder with screenshots of the crash, versions of cr2hdr that crash and dont crash, and dcraw
https://www.dropbox.com/sh/d5g3jlugmqqkjdl/8-fIwn3kRg
i tried it on two computers, maybe im using wrong version of dcraw or something? im using the one from the first page of this forum topic
cant think of any other variables, hope you can give some advice as to what im doing wrong

[Walter Schulz]

exiftool located in the directory where cr2hdr and dcraw are?

[ShaunWoo]

i removed it, the first page said it was optional, i tried it with it being in the same folder, same results, but i will try again and include it in the dropbox folder

update: yup, tried it with exiftool in same folder, same results as before with 20bit cr2hdr (crashes), but with  2014.01.23 version, converts correctly, added exit tool to folder: https://www.dropbox.com/sh/d5g3jlugmqqkjdl/8-fIwn3kRg

[Walter Schulz]

Same error message?

[surami]

Thanks for the explanation a1ex.

I have this "cr2hdr-20bit-same-levels.bat" file

Code: [Select]

cr2hdr-20bit --same-levels %*
pause
in the folder, where my cr2s are and I setup the path environmental variable for the cr2hdr folder (cr2hdr-20bit.exe, exiftool.exe and dcraw.exe), so it should run well, but I get this:

Unknown option: --same-levels

update: I think I have to define the input and output somehow and after that run the "--same-levels" command only for the dngs or how?

[a1ex]

Just updated cr2hdr-20bit.exe.

Fixes:

• fixed some valgrind warnings (ShaunWoo, can you try?)
• yet another algorithm for exposure matching, which should solve some corner cases like this

New features:

• --skip-existing: if you pass a large number of files in a single command line, this will skip the input files for which the output file already exists (useful if it crashes in the middle, and you want to resume). There is a small question for you in the commit message.
• --embed-original: this moves the original CR2 into the output DNG, so you'll be able to re-process the DNG with either different options or with another version (of course, newer than this one). Discussed here.
• --wb=graymed: another auto white balance algorithm (median of R-G and B-G). I still like --wb=graymax, so I made this one the default (will battle-test it these days). Discussed here.

@surami: try again with this one.

[ShaunWoo]

oh wow, congrats a1ex, you fixed it, now i will try with the parameter same levels, in a batch convertion of 100 files from 0 upwards, hopefully it wont crash and no flickering, if all good, ill be very happy, well done dude, few notes and suggestions:

may i ask, what was the issue, did the screen shot help figure it out?

few days ago before i knew the cause of flickering i used the script for running multiple instances of cr2hdr, i asume it wont work since the same levels needs to be used within the initial single process, right? if thats the case, how about instead of using "same levels", we assing a specific variable to run with a parameter that sets the exposure, so this way each instance using the script would process the same level, speeding up the processing for video sequences greatly (hopefully its not something i missed in the forum)

the above i believe is necessary for longer sequences with many frames, if i try to click and drag even a 100 files, it will give me an error saying file path too long

also, how about putting a list of versions of cr2hdr in the very first post, i found the actual list of versions, but its the code only not the exe file, and im no where near a coder so i wouldnt know what to do with the code, im sure it would benefit many people

maybe theres a way to put the latest file in the first post? i gotta be honest it took me a few days to figure out the flickering cause, at first i found its ACR, shadow and highlight changes, then cr2hdr, i processed some NON dual iso sequences and there was no flickering what so ever, (even with process 2012 in acr), so i dont think its ACR issue, so yeah, would be useful for many people i think if the latest file was in the first post

big respects for your work and the speed of your fixes, just making suggestions for problems ive encountered as a regular user that im sure many other users would experience (finding the right information. i.e latest cr2hdr file)



NOTE FOR THOSE WHO DO NOT KNOW ABOUT RUNNING PARAMETERS, make a shortcut of the "cr2hdr-20bit.exe" file, right click on shortcut > properties, in the target, copy and paste  --same-levels at the end of the taget, if the target has spaces in the url, put " at the beginning of the url and at the end of it, for example:
"C:\camera raw dual iso converter\cr2hdr-20bit.exe"  --same-levels
putting this info here to save time for people who dont know (like me)

also do you know anyway of evening out the flickering in the dngs after processing them with the old cr2hdr that didnt have the same levels parameters

[a1ex]

Right, using --same-levels with parallel processing is a little harder (because all the files are analyzed at the end, and then it decides a white level that hopefully matches all of them). But it can be done after a regular conversion, like this:

Code: [Select]

exiftool *.DNG "-WhiteLevel<BlackLevel"
exiftool *.DNG "-WhiteLevel+=50000"

You may need to fiddle with the values though; lower the value if you get pink highlights, increase it if you think you can squeeze some more highlight detail.

[ShaunWoo]

Thank you for your responce, just processed 100 files, seems the exposure is the same, but the luma changes, ill post process with after effects ACR now

When you have some free time could you please go over the complete messege i sent you above please, i think i highlight some points many other people would question and come across, especially high frame counts, (i cant process 600 for example using 1 process)

and thank you again for your batch processing advice, but i really dont understand what to do with those commands, any chance you could link me to a discussion, guide, or anything that would point me in the correct direction? do i still use cr2hdr and the script for multiple instances? where would i put in those parameters, in the script?

Hope you understand my ignorance in coding

[surami]

@ShaunWoo: I just thought out the same as you, that there is no need for .bat file, it works with shortcut.

@a1ex: Thanks for the update. The same levels command works with the shortcut trick, but the results for a spherical panorama (situation: daylight, indoor: no electriclights, outdoor: full bright sunshine) isn't enough good. As I see the dngs, they should be analyzed again after the same levels algorithm and there should be a match overall brightness trick after that, which would be based on the same leveled dngs' histograms. This algorithm should find the middle one from the batch processed dngs and there should be an exposure compensation (something like deflicker). Now I make the matching manually and the result is verygood. Tomorrow (today) I have much to do, but as I have time, I will upload the a full set of the sphere pieces and I'll send a pm with a download link for testing.

[Audionut]

we assing a specific variable to run with a parameter that sets the exposure

What should the exposure be?  Are you aware of the problems from setting the white/black levels, to high/low?

maybe theres a way to put the latest file in the first post?

For the 20 bit version?  The 20 bit version is still somewhat experimental. 


Thanks for the shortcut trick.

[ShaunWoo]

isnt there a default exposure? like when you open with ACR its on 0, so i asume theres a default, am i wrong? if that is true, then is there a way to set on default= 0, or something like that, rather than the program auto adjusting it

im having a difficult time because i need to process around 700 frames sequence but i cant since it says the file path is too long, i am able to process 500, but what is the solution for this problem, does anybody know?

[Audionut]

isnt there a default exposure? like when you open with ACR its on 0, so i asume theres a default, am i wrong? if that is true, then is there a way to set on default= 0, or something like that, rather than the program auto adjusting it

You mean midtones at 0 EV?  Because dual_iso files are rendered dark with default settings?

You almost always have to increase exposure in LR, because you are underexposing your midtones with dual_iso.
Lets me try and show an example.  Let's say that this is our scene lighting.

+------+------+

Lets say that the above represents 14 EV of dynamic range.  The middle + represents the midtones, and these midtones are 7 EV below the highlights. Remember, normal ISO is rated for around 11 EV of DR.
So in normal circumstances, you might expose as such.

+------+---+

In other words, you sacrifice 2-3 stops of highlight detail (overexpose them), to expose your midtones correctly.  In raw ADU's, the midtones are around 4 EV from sensor saturation (overexposure). 

So where we use dual_iso to capture these highlights (that would otherwise be overexposed), the midtone point has been shifted 2-3 stops lower (instead of being 4 EV below sensor saturation, it is now 6-7 EV below sensor saturation).  Hence, in PP, you need to increase exposure 2-3 stops to compensate.  This isn't a deficiency with dual_iso, it is an expected outcome.  a1ex's soft-film highlight compression can be used to automatically compress these highlights, so that the midtone point of your exposure (the rendered image), is more closely aligned with the midtone point of a standard Canon image.

Here, you might not need to increase the exposure in PP, because soft-film is applying an exposure correction automatically.

The files are being rendered correctly, because there is significantly more data above the midtone point, then without dual_iso.
You can use --soft-film to compress the highlights.

Code: [Select]

        "Highlight/shadow handling", (struct cmd_option[]) {
            { (int*)&soft_film_ev,    1, "--soft-film=%f",  "bake a soft-film curve to compress highlights and raise shadows by X EV\n"
                                          "                  (if you use this option, you should also specify the white balance)"},
However IIRC, soft-film is nonlinear.  So this would not be useful in a situation where your rendered images need to be exposure matched.

im having a difficult time because i need to process around 700 frames sequence but i cant since it says the file path is too long, i am able to process 500, but what is the solution for this problem, does anybody know?

Right, using --same-levels with parallel processing is a little harder (because all the files are analyzed at the end, and then it decides a white level that hopefully matches all of them). But it can be done after a regular conversion, like this:

Code: [Select]

exiftool *.DNG "-WhiteLevel<BlackLevel"
exiftool *.DNG "-WhiteLevel+=50000"

You may need to fiddle with the values though; lower the value if you get pink highlights, increase it if you think you can squeeze some more highlight detail.

Rather then fiddling with values, I would probably process a bright and dark image with cr2hdr --same-levels, note the black/white levels printed, and use these with exiftool.

If you use the shortcut trick from above, you should just be able to drag and drop all the files onto the shortcut.  With 700 files though, and --same-levels requiring single threaded processing, again, I would probably just process a bright and dark image, and use the printed output with exiftool.  This way, you can process the files with cr2hdr, with a multi-threaded solution (significantly faster), and just use exiftool at the end.

[ShaunWoo]

in acr process 2012, just below white balance, its exposure first, i was refering to that, im afraid im not as well educated as you guys, but as far as im seeing theres exposure, contrast, shadows, highlights, etc, i dont see a midtone option in acr, so yeah, i was refering to absolute exposure, but i guess its different with cr2hdr

what do you mean process a bright and dark image? do you basically mean the dual iso file? note the black/white levels printed, would they be printed as results for each dng processed individually in cr2hdr?

and nope, when i click and drag around 700+ files onto the cr2hdr, or the shortcut for it, it will give me the error, file name too long, im very sure youve come across that error as ive seen people post about it before in this forum, but i havent seen a solution, if you havent i can post a screenshot, so im still limited to around 500 files

oh and i think i understand some of what you said now, you say process a black and white level image, im sure you must mean a dual iso now, once that is process, cr2hdr will print on cmd the black and white levels on screen, right? and use this setting printed with the rest of the sequence using the script that runs multiple instances of cr2hdrm with the code a1ex posted?:
exiftool *.DNG "-WhiteLevel<BlackLevel"
exiftool *.DNG "-WhiteLevel+=50000"
just change the variable with the one printed in cmd?
however once again i dont know what to do with this code, where do i input it in, is it for the cr2hdr file? use it the same way as the --same-levels parameter? the only way i know how to run THAT parameter alone is single line, how would i input two line code in there

also you say use exiftool at the end, do you mean after the cr2hdr process? what is the purpose of this if the file has already been processed and ready for PP

[Audionut]

There are 2 ways to process dual_iso files that need to be exposure matched.


cr2hdr --same-levels

exiftool *.DNG "-WhiteLevel<BlackLevel"
exiftool *.DNG "-WhiteLevel+=50000"

Exiftool.

So, rather then using --same-levels with cr2hdr, you can use exiftool.  And yes, use exiftool after processing with cr2hdr.  And since you are using exiftool, you do not need to use --same-levels with cr2hdr.

in acr process 2012, just below white balance, its exposure first, i was refering to that, im afraid im not as well educated as you guys, but as far as im seeing theres exposure, contrast, shadows, highlights, etc, i dont see a midtone option in acr, so yeah, i was refering to absolute exposure, but i guess its different with cr2hdr

I don't understand what you want to do with this "absolute exposure".

[a1ex]

The same levels command works with the shortcut trick, but the results for a spherical panorama (situation: daylight, indoor: no electriclights, outdoor: full bright sunshine) isn't enough good. As I see the dngs, they should be analyzed again after the same levels algorithm and there should be a match overall brightness trick after that, which would be based on the same leveled dngs' histograms.

The --same-level option is meant for pictures taken with the same exposure settings in the camera (in manual mode). It's not a substitute for deflickering.

Deflickering for panoramas is very hard, because, when deflickering, the algorithm must know which parts of the image are going to overlap. The robust statistics from my deflicker algorithm (median, percentiles) are not robust enough for matching two shots with very different framing.

Also be careful of the flicker introduced by ACR. If in doubt, try to process the images with ufraw (none of its adjustments will introduce flicker). Make sure you are using manual white balance.

when i click and drag around 700+ files onto the cr2hdr, or the shortcut for it, it will give me the error, file name too long

If you reach the command-line length limit, I'm afraid the only options are:
- don't use --same-levels, but run exiftool to equalize the levels after a regular conversion
- try an operating system with a larger limit (mine has around 2 MB, Windows has 8K or 32K)
- bug Microsoft to increase the limit (you paid for it, so they should help you, right?)

If you don't want to fiddle with levels, your best bet is to copy them from an overexposed shot from your sequence. If you have used manual exposure settings in the camera, and you want the exposures matched when rendering, you need to have the WhiteLevel - BlackLevel (their difference) constant for all your files (and that's why my exiftool commands do).

For example, to copy the settings from one image to all the others:

Code: [Select]

# read the levels from a reference image
exiftool IMG_1234.DNG -WhiteLevel -BlackLevel
White Level                     : 47752
Black Level                     : 2032

# now, copy the levels difference (WhiteLevel - BlackLevel) to all other files in the sequence
exiftool *.DNG "-WhiteLevel<BlackLevel"
exiftool *.DNG "-WhiteLevel+=45720"


[a1ex]

Updated cr2hdr-20bit.exe as discussed here (mostly tweaked the --embed-original functionality).

[dubzeebass]

a1ex can you provide a new binary for Mac too?