Dual ISO - massive dynamic range improvement (dual_iso.mo)

Started by a1ex, July 16, 2013, 06:33:50 PM

Previous topic - Next topic

0 Members and 3 Guests are viewing this topic.

zuzukasuma

Quote from: Shizuka on September 09, 2013, 01:51:12 PM
too much work
i wrote custom utilities for this sort of thing before
http://www.mediafire.com/download/5n5pjih98b80909/MT_cr2hdr.zip
note that cr2hdr is hacked to use unique tempfiles based off input file name.

this is what I'm looking for! tell devs to import this to pack! it works like a charm (missing dcraw.exe but no prob). thank you!

edit: worked like a charm :) when you close the example.cmd cr2hdr.exe's running in the background till the all CR2s converted.
in a complicated relationship with eos m.

Shizuka

Quote from: a1ex on September 09, 2013, 02:01:47 PM
Can you share the temp file hack?
it's assembly patched - source would be the bindiff sorry

I would imagine the changes in source are here:
//line 145
        printf("\nInput file     : %s\n", filename);
        char tempname[4];
             tempname[0] = filename[4];
             tempname[1] = filename[5];
             tempname[2] = filename[6];
             tempname[3] = filename[7];
             char tempname_txt[8];
             char tempname_pgm[8];
             snprintf(tempname_txt,8,"%s.txt",tempname);
             snprintf(tempname_pgm,8,"%s.pgm",tempname);

        char dcraw_cmd[1000];
//line 148
        snprintf(dcraw_cmd, sizeof(dcraw_cmd), "dcraw -v -i -t 0 \"%s\" > %s", filename, tempname_txt);
        int exit_code = system(dcraw_cmd);
        CHECK(exit_code == 0, "%s", filename);
       
        unsigned int model = get_model_id(filename);
        exit_code = get_raw_info(model, &raw_info);

        CHECK(exit_code == 0, "RAW INFO INJECTION FAILED");
//line 157
        FILE* t = fopen(tempname_txt, "rb");
        CHECK(t, tempname_txt);

also replace "tmp.pgm" with tempname_pgm (line 184, 188, 189)

finally, replace the unlink() parameters at 283/284


a1ex: a suggestion: turn on sse floating point instructions - it'll give a 20-30% speedup for free on many speed critical routines. -msse -msse2 -mfpmath=sse are the compiler flags for gcc, and will produce executables that run on pentium4/amdK8 or higher
edit: maybe not. I only found two routines

a59192

Currently I'm still experiencing problems on dng converter, set up in the same shot, CR2 conversion to DNG is good and bad, and attach using 5DII CR2 shoot, ask if there is any way to stabilize conversion?

https://mega.co.nz/#!QIVmVCzS!RKZgVGp2-ntkv2OFPqZ3w3ZO2XYcUCnDRkYJmAbhuZQ

Audionut

Quote from: Shizuka on September 09, 2013, 02:14:38 PM
a1ex: a suggestion: turn on sse floating point instructions - it'll give a 20-30% speedup for free on many speed critical routines. -msse -msse2 -mfpmath=sse are the compiler flags for gcc, and will produce executables that run on pentium4/amdK8 or higher
edit: maybe not. I only found two routines

Fprofiling might help also.

a1ex

Experimental cr2hdr:
- converts the CR2 for a59192 (but didn't check if it breaks any others)
- uses pipes instead of temporary files (should work better if you start 5 copies in the same directory)

I didn't try it on Windows, only compiled the exe. On Linux I've tried on 2 pics in parallel (got 57 seconds for both, compared with 45 for a single picture).

Audionut

It's complaining that everything doesn't look like interlaced ISO on my windows box.


Audionut

I:\1>cr2hdr_exp _UAL5976.CR2

Input file     : _UAL5976.CR2
Canon EOS 5D Mark III detected
Full size      : 5920 x 3950
Active area    : 5796 x 3870
Doesn't look like interlaced ISO



a59192

Quote from: a1ex on September 09, 2013, 04:02:17 PM
Experimental cr2hdr:
- converts the CR2 for a59192 (but didn't check if it breaks any others)
- uses pipes instead of temporary files (should work better if you start 5 copies in the same directory)

I didn't try it on Windows, only compiled the exe. On Linux I've tried on 2 pics in parallel (got 57 seconds for both, compared with 45 for a single picture).

Used cr2hdr_exp.exe with the current new construction cr2hdr, can not normally turn out, do not know which part wrong, ask how to use the uses pipes? windows can use it?



a1ex

Can you try again? Got it working under Wine.

I had to set binary mode on the pipe and stop using fseek.

Audionut

Yeah that got it working.  Can you add a command line option for the user to select how many instances to run at any one time.  And then when a bunch of CR2s are dropped on it, it's spawns all those instances?

I tried adding a heap of compiler options (-msse etc etc), and it made no difference to the processing time :(

RenatoPhoto

Camera tested 5D3
I tried the latest commit (da752ca33b18) and the when using the AETTR + Dual ISO I get a problem. 
I am setting the Dual Iso to +3EV
The first picture I take shows as dual ISO possilbly at base iso + 3EV
The next picture the Dual ISO module has the base iso = recovery iso.  So no Dual Iso Picture is take.
Repeatable.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

telecastDaveRI

Quote from: Audionut on September 09, 2013, 07:19:15 PM
Yeah that got it working.  Can you add a command line option for the user to select how many instances to run at any one time.  And then when a bunch of CR2s are dropped on it, it's spawns all those instances?

I tried adding a heap of compiler options (-msse etc etc), and it made no difference to the processing time :(


If I can ask, how did you get it working? Did you just download a1ex's experimental upload (seen above) again? (which I assume has been updated)

Audionut

Quote from: telecastDaveRI on September 10, 2013, 05:43:50 AM
Did you just download a1ex's experimental upload (seen above) again? (which I assume has been updated)

That's correct.

Quote from: RenatoPhoto on September 10, 2013, 04:10:09 AM
Camera tested 5D3
I tried the latest commit (da752ca33b18) and the when using the AETTR + Dual ISO I get a problem. 
I am setting the Dual Iso to +3EV
The first picture I take shows as dual ISO possilbly at base iso + 3EV
The next picture the Dual ISO module has the base iso = recovery iso.  So no Dual Iso Picture is take.
Repeatable.

Are you sure it's not this?

Quote from: a1ex on September 09, 2013, 12:46:25 PM
ETTR will try to:
- use a low ISOs and a low EV spacing
- meet or exceed the selected SNR minus 0.5 EV
- do not clip more highlights than you have selected for "Highlight ignore"
- do not exceed max auto ISO from Canon menu

So, if that SNR can be met without clipping highlights, the scene does not require dual ISO. If not... it does.

----------------

Using autoexpo for the base exposure with AETTR enabled in autosnap with dual ISO.  All images in a test batch had the dual ISO file prefix.  Some images were not dual ISO as not needed.

If I just have dual ISO enabled with alternate frames, the file prefix is working as expected.

RenatoPhoto

When I take a photo with AETTR 02% h.i. and SNR OFF I get
DR = 10.9
Same scene with Dual Iso only with Recovery ISO = +3 EV I get
DR = 13.4

Now I turn on AETTR 0.2% h.i plus and SNR = OFF + Dual ISO with Recovery ISO = +3 EV I get
DR=10.9  and Base ISO = Recovery ISO  DUAL ISO is turned OFF automatically

Now I set DUAL ISO TO 800 ISO plus the AETTR as above and I get
DR=10.9 and Base Iso = Recovery ISO , my ISO 800 has been automatically replaced by base ISO

Conclusion:  Dual ISO + AETTR is not working properly.  If I roll back to Sept 07 build then the results are totally different and DUAL ISO works as expected!
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Audionut

Enable some SNR limits.  This will give the code something to aim for :)

Both functions are now linked together when both are enabled.  With no SNR limits applied, you're telling it, I don't care how noisy the midtones/shadows are.
With SNR limits applied, previously AETTR would blow the highlights to attain the required SNR, now it will enable dual ISO instead to keep the highlights in check while producing the required SNR.

I guess it could be coded so that SNR limits off mean restore previous functionality.  But to be honest, I think that will cause to much confusion for users in the future.

Audionut

Quote from: Audionut on September 10, 2013, 05:57:49 AM
Using autoexpo for the base exposure with AETTR enabled in autosnap with dual ISO.  All images in a test batch had the dual ISO file prefix.  Some images were not dual ISO as not needed.

I tried adding 100ms delay to autosnap but it still wanted to use the same prefix as the previous shot regardless.  And 100ms was becoming noticeable.

a1ex can you send the output of cr2hdr to a logfile?  It would be nice to check the results of processed files without having to manually process each one.

RenatoPhoto

Quote from: Audionut on September 11, 2013, 12:44:16 AM
Enable some SNR limits.  This will give the code something to aim for :)
I want to keep all of the highlights, that is why I turned off SNR.  So my only option to conserve the highlights is Midtones SNR = 1 if I want to use Dual Iso with Recoveru ISO +3EV
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

RenatoPhoto

Ok I tested with SNR and it works with Midtones and Shadows = 1.  So I dont have the option to keep the highlights by doing a standard AETTR at h.i.=0.2% + DUAL ISO.

In my opinion this is not good.  I will be using September 07 build until this gets resolved.
Thanks anyway.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Audionut

AETTR works just like previously.  You decide how many highlights to blow (or not).
Enabling SNR limits will try and keep those highlights, but will blow highlight detail to ensure the SNR limits are met.
Enabling SNR limits and dual ISO will keep those highlights and use dual ISO as necessary to ensure the SNR limits are met without blowing the highlights.

If the SNR limit can be met without the use of dual ISO, dual ISO will not be used.  Highlight detail is controlled by the settings in AETTR just as it's always been.

Audionut

Enable AETTR and set your required highlight settings.  Have dual ISO and SNR limits off.  Observe the results.
Now enable dual ISO and set the SNR limits (use the default settings for a start 6/3), observe the results.
Set the midtone SNR limit to 8EV.  Observe the results.  Highlight detail should not change, dual ISO settings should change (automatically) to boost the midtone luminance.

RenatoPhoto

Quote from: Audionut on September 11, 2013, 04:23:08 AM
Enabling SNR limits will try and keep those highlights, but will blow highlight detail to ensure the SNR limits are met.

Just a simple question.  I want to keep all of the highlight therefore I use AETTR with h.i. = 0

Now I want to boost the shadows and midtones without blowing the highlihgts using DUAL ISO.  What should I set the Dual ISO module to accomplish this?
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Audionut

Nothing.  Dual ISO is controlled automatically.  Just turn it on.

Control the midtones/shadows with the SNR limits.  With dual ISO on, it won't blow your highlight detail like previously, it will adjust dual ISO settings as needed to boost the midtones/shadows.