Dual ISO - massive dynamic range improvement (dual_iso.mo)

[RenatoPhoto]

Reporting a repeatable bug 5D3, compile ae68c60000f2

Tried to use AETTR+DUAL ISO for a night-to-day timelapse.

The module started ok at typical setting ie. 30 seconds at iso 6400.  As light increased the exposure never changed and remained at 30 seconds iso 6400.  I caught it early but if the sun hit my sensor it could have fried it.

I repeated the test under my desk in a very dark place and then pointed to my monitor with lots of light.  The exposure never changed from 30 sec at iso 64000.

Settings used:

Manual lens f/22, iso 6400, 30 second shutter, Canon in M mode, Image review 2 sec

AETTR settings: Trigger mode: Always on, Slowest shutter: 32", Exposure target: -0.5, H.I:02%, Allow clipping:Green channel, Midtone SNR limit: 7EV, Shadow SNR limit:2EV, Link to Canon:OFF, Link to DUAL:ON

DUAL ISO: ON

Intervalometer: Take pic every 1 m, Start after 3s
Post Deflicker: Adobe xmp, 50%, -4 EV

[a1ex]

Good catch.

What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

If you change the exposure target at -1 EV, it will start correcting the overexposure slowly (one stop at a time). It will continue to do that until the white level returns to normal.

At 8 seconds, the white level looks normal (from there it converged from half-screen blown out to perfect exposure in 2 steps). Didn't try in-between shutter speeds.

The issue is quite subtle and I'm not sure how to fix it in a portable way. Maybe with some guessing like in cr2hdr.

[RenatoPhoto]

What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

Maybe this is a good time to add a check based on canon brightness value.

[Audionut]

The new metered areas is handy.

Currently it just continues to draw over itself.  Is there a way to determine if the scene has changed dramatically and clear the overlay for a fresh draw?  I don't know how to do this, so not sure of the best way to stop it continually updating where it doesn't need to.
edit:  Since it seems to just draw everything again (when updating), what about just clearing the old overlay first before drawing again?

What about linking it to zebras?  An option to show either the zebras as they are now, or the metered areas.  Not sure of the cleanest way to do this.

Can it be enabled for QR?

[a1ex]

Heh, I've only tested it in QR.

[Audionut]

I must need sleep then.  It works fine in LV, and works fine on press set, but nothing shows in the quick image review.

[RenatoPhoto]

Good catch.

What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

If you change the exposure target at -1 EV, it will start correcting the overexposure slowly (one stop at a time). It will continue to do that until the white level returns to normal.

At 8 seconds, the white level looks normal (from there it converged from half-screen blown out to perfect exposure in 2 steps). Didn't try in-between shutter speeds.

The issue is quite subtle and I'm not sure how to fix it in a portable way. Maybe with some guessing like in cr2hdr.

Thanks Alex, I tested with Timelapse today and worked properly.  Now I have to figure out how to process some of the dual iso images in Timelapse.

[NateVolk]

Did a shoot last night with the newest build.  Used auto ettr linked to dual iso.  Not sure I'm doing it right, but got good results with the standard settings.  After cr2hdr, the dng files seemed under exposed by about 1.5-2 stops but I was able to raise them without a problem.  Is this the way it is supposed to work, or am I missing something?  Thanks!

https://docs.google.com/file/d/0B_Id0y3JG-KEdU9qZGQ2WkFuQTQ

[Danne]

I think you get the dark exopsure which is the way it was exposed keeping the highlights. As you say, to lift the shadows are not any problem

[RenatoPhoto]

Reporting new test of TIMELAPSE with ETTR and DUAL ISO link:

5D3 with yestedays compile before memory back-end.  be77944f2b0d

I tested again night to day transition using ETTR+DUAL ISO, SNR 6/3.  The previous exposure problems are gone.

The DUAL ISO module did not become necessary until the dynamic range increased when daylight started to increase the brightness of the sky.  All cr2 files where able to read the the xmp in ACR.  All the xmp files had adjustment applied to cr2 and then exported to DNG. 

I was able to process the DUAL ISO (DNG) files with this workflow:

1. Opened all cr2 with ACR and exported as DNG, then processed all DNG with cr2hdr, then dng to tiff with ACR.

And the beginning of DUAL ISO enhancement, there were times when one image would be dual iso and then the next did not, after development and final review the images blended very well.  Except for a couple of times when a small flicker appeared.  By simple analysis I could tell which was dual iso and which was standard.

I did not let the system go into full daylight but I think that it is important that the dual iso system does not go beyond 100/800 iso to maintain good images.  For this reason I think the use of dual iso module should be limited to SNR settings no higher than 6/3.   A setting like 8/4 would raise the dual iso to 100/6400 which would deteriorate the image resolution and colors.

Hopefully will be able to run a new test with extended times for a more complete analysis.

The dickering with slide car files is amazing!

[Canon eos m]

Reporting new test of TIMELAPSE with ETTR and DUAL ISO link:

5D3 with yestedays compile before memory back-end.  be77944f2b0d

Hi RenatoPhoto, where can I get hold of the binary for the be77944f2b0d? Searched, the forums but could not find any refernce to this specific version.

[a1ex]

To limit the max ISO, change it in Canon menu (max auto ISO).

[RenatoPhoto]

Hi RenatoPhoto, where can I get hold of the binary for the be77944f2b0d? Searched, the forums but could not find any reference to this specific version.

You would have to compile it yourself.  I suppose today's build should be as good, I don't know if there are any issues.

[RenatoPhoto]

To limit the max ISO, change it in Canon menu (max auto ISO).

The problem is I need the base ISO of 6400 for the night shots but I dont want DUAL ISO to go very high.

[Canon eos m]

You would have to compile it yourself.  I suppose today's build should be as good, I don't know if there are any issues.

Don't know anything about building. Guess, will wait till I a complied version is available.

[Audionut]

Don't know anything about building. Guess, will wait till I a complied version is available.

http://builds.magiclantern.fm/#/

[Marsu42]

The problem is I need the base ISO of 6400 for the night shots but I dont want DUAL ISO to go very high.

That's why I recently proposed to turn the "dr gained" field into a setting - if you for example set it to at least 1.0 ev, dual_iso would be disabled on high iso settings

[Canon eos m]

http://builds.magiclantern.fm/#/

Thanks.

[RenatoPhoto]

I used ETTR + DUAL ISO to do a timelapse from night to day.

The workflow is:

Open all cr2 images and save them as DNG.  This puts the deflicker adjustment (in xml) in the DNG file for further processing.
Process the files with cr2hdr.  This processed the DUAL-ISO dngs and saves them as a normal DNG.
At this point I can import the DNG into Vegas and make the video.  It is perfect at this point.
Now I want to generate tiffs which can be handled by other video editors, not all can read the dngs.
To do this I open the DNGs in ACR and notice that the deflicker exposure is in the exposure field.  If I export the tiffs with this exposure number then it generates flicker, it appears that this exposure has already changed the image itself so the exposure number is not needed for further processing.  So I select all the dngs in ACR and set the exposure to 0.  Problem fixed, now the tiffs can be imported to video editor and there is no flicker.

Finally I get another problems that I cannot resolve:
The Dual ISO dngs that were converted to tiffs get some really strange effects like if there was a crop!  After analyzing the faulty images in ACR I noticed that the DUAL ISO dngs have a CAMERA PROFILE set to embedded and cannot be changed, while the standard dngs have CAMERA PROFILE = Adobe Standard.  I cannot find anything else that is different from the two files so I wonder if this is causing this strange effect.

Something interesting to note is that the video made with the dngs coming out of cr2hdr is ok.  I assume that Vegas cannot read the much of the embedded information and is just importing the image.  When I import these dngs into ACR to make the tiff is when this problems arises.

[a1ex]

Do they both have the same resolution? If not, try developing the CR2 files with dcraw.

[RenatoPhoto]

Do they both have the same resolution? If not, try developing the CR2 files with dcraw.

They are cropped  The tiff from standard dng is 5796x3870  while the tiff form dual iso dng is 5760x3840. 

[a1ex]

Swap the numbers

[RenatoPhoto]

Swap the numbers

Right: standard dng is 5760x3840 while the tiff form dual iso dng is 5796x3870.

Ok I was able to trim the dual iso tiffs using Photoshop action and batch.
The timelapse is good but I wll try it again with different settings.

[jOt]

I got a problem with conversion from interlaced dual iso dng to normal dng. I shot fire in the dark with fps override to 65 fps. Brighter shots are converted correct, but darker aren't recognized as interlaced. Even in the same take one (brighter) frame is recognized well...

Code: [Select]

Input file     : E:\81_22-0544_00016
5.DNG
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : BddB RGGB
Noise levels   : 8.00 8.00 8.00 8.00 (14-bit)
Estimating ISO difference...
ISO difference : 4.61 EV (2450)
Black delta    : 67.67
Interpolation  : mean23-chroma5x5-alias
Dynamic range  : 10.35 (+) 10.35 => 14.97 EV (in theory)
Matching brightness...
Looking for hot/cold pixels...
Full-res reconstruction...
ISO overlap    : 3.0 EV (approx)
Half-res blending...
Chroma filtering...
Building alias map...
Filtering alias map...
Smoothing alias map...
Noise level    : 8.00 (16-bit), ideally 8.00
Dynamic range  : 12.35 EV (cooked)
Output file    : E:\81_22-0544_00016
5.DNG
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 149) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 151) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 154) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 156) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
    1 image files updated
...and next other not.

Code: [Select]

Input file     : E:\81_22-0544_00016
6.dng
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : BdBd RGGB
Interlacing method not supported
ISO blending didn't work

Input file     : E:\81_22-0544_00016
7.dng
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : dBBd RGGB
Noise levels   : 8.00 8.00 8.00 8.00 (14-bit)
Estimating ISO difference...
Doesn't look like interlaced ISO
ISO blending didn't work


Is there a way to force cr2hdr.exe to convert dngs or other method than cr2hdr.exe?
Please, help. This footage is extremely important to me.

Camera: 5DM3
tried different versions of cr2hdr.exe

[a1ex]

Well, if you really want to remove the error checks, feel free to do so, but you'll get just garbage.

Hint: I can't troubleshoot anything from these logs (other than reaching the above conclusion).