Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo)  (Read 2160477 times)

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1100 on: September 16, 2013, 02:09:39 PM »
Reporting a repeatable bug 5D3, compile ae68c60000f2

Tried to use AETTR+DUAL ISO for a night-to-day timelapse.

The module started ok at typical setting ie. 30 seconds at iso 6400.  As light increased the exposure never changed and remained at 30 seconds iso 6400.  I caught it early but if the sun hit my sensor it could have fried it.

I repeated the test under my desk in a very dark place and then pointed to my monitor with lots of light.  The exposure never changed from 30 sec at iso 64000.

Settings used:

Manual lens f/22, iso 6400, 30 second shutter, Canon in M mode, Image review 2 sec

AETTR settings: Trigger mode: Always on, Slowest shutter: 32", Exposure target: -0.5, H.I:02%, Allow clipping:Green channel, Midtone SNR limit: 7EV, Shadow SNR limit:2EV, Link to Canon:OFF, Link to DUAL:ON

DUAL ISO: ON

Intervalometer: Take pic every 1 m, Start after 3s
Post Deflicker: Adobe xmp, 50%, -4 EV
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1101 on: September 16, 2013, 03:46:14 PM »
Good catch.

What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

If you change the exposure target at -1 EV, it will start correcting the overexposure slowly (one stop at a time). It will continue to do that until the white level returns to normal.

At 8 seconds, the white level looks normal (from there it converged from half-screen blown out to perfect exposure in 2 steps). Didn't try in-between shutter speeds.

The issue is quite subtle and I'm not sure how to fix it in a portable way. Maybe with some guessing like in cr2hdr.

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1102 on: September 16, 2013, 04:00:27 PM »
What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

Maybe this is a good time to add a check based on canon brightness value.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Dual ISO - massive dynamic range improvement
« Reply #1103 on: September 17, 2013, 11:52:05 AM »
The new metered areas is handy.

Currently it just continues to draw over itself.  Is there a way to determine if the scene has changed dramatically and clear the overlay for a fresh draw?  I don't know how to do this, so not sure of the best way to stop it continually updating where it doesn't need to.
edit:  Since it seems to just draw everything again (when updating), what about just clearing the old overlay first before drawing again?

What about linking it to zebras?  An option to show either the zebras as they are now, or the metered areas.  Not sure of the cleanest way to do this.

Can it be enabled for QR?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1104 on: September 17, 2013, 11:58:21 AM »
Heh, I've only tested it in QR.

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Dual ISO - massive dynamic range improvement
« Reply #1105 on: September 17, 2013, 12:23:42 PM »
I must need sleep then.  It works fine in LV, and works fine on press set, but nothing shows in the quick image review.

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1106 on: September 17, 2013, 03:43:42 PM »
Good catch.

What's happening here is that sensor saturation point (at least on 5D3) goes lower than usual, so the zebra indicator and ETTR analysis don't even realize the image is overexposed (they think it's just perfect).

If you change the exposure target at -1 EV, it will start correcting the overexposure slowly (one stop at a time). It will continue to do that until the white level returns to normal.

At 8 seconds, the white level looks normal (from there it converged from half-screen blown out to perfect exposure in 2 steps). Didn't try in-between shutter speeds.

The issue is quite subtle and I'm not sure how to fix it in a portable way. Maybe with some guessing like in cr2hdr.

Thanks Alex, I tested with Timelapse today and worked properly.  Now I have to figure out how to process some of the dual iso images in Timelapse.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

NateVolk

  • New to the forum
  • *
  • Posts: 28
Re: Dual ISO - massive dynamic range improvement
« Reply #1107 on: September 18, 2013, 12:30:40 AM »
Did a shoot last night with the newest build.  Used auto ettr linked to dual iso.  Not sure I'm doing it right, but got good results with the standard settings.  After cr2hdr, the dng files seemed under exposed by about 1.5-2 stops but I was able to raise them without a problem.  Is this the way it is supposed to work, or am I missing something?  Thanks!

https://docs.google.com/file/d/0B_Id0y3JG-KEdU9qZGQ2WkFuQTQ

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7599
Re: Dual ISO - massive dynamic range improvement
« Reply #1108 on: September 18, 2013, 12:50:45 AM »
I think you get the dark exopsure which is the way it was exposed keeping the highlights. As you say, to lift the shadows are not any problem

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1109 on: September 18, 2013, 03:56:38 AM »
Reporting new test of TIMELAPSE with ETTR and DUAL ISO link:

5D3 with yestedays compile before memory back-end.  be77944f2b0d

I tested again night to day transition using ETTR+DUAL ISO, SNR 6/3.  The previous exposure problems are gone.

The DUAL ISO module did not become necessary until the dynamic range increased when daylight started to increase the brightness of the sky.  All cr2 files where able to read the the xmp in ACR.  All the xmp files had adjustment applied to cr2 and then exported to DNG. 

I was able to process the DUAL ISO (DNG) files with this workflow:

1. Opened all cr2 with ACR and exported as DNG, then processed all DNG with cr2hdr, then dng to tiff with ACR.

And the beginning of DUAL ISO enhancement, there were times when one image would be dual iso and then the next did not, after development and final review the images blended very well.  Except for a couple of times when a small flicker appeared.  By simple analysis I could tell which was dual iso and which was standard.

I did not let the system go into full daylight but I think that it is important that the dual iso system does not go beyond 100/800 iso to maintain good images.  For this reason I think the use of dual iso module should be limited to SNR settings no higher than 6/3.   A setting like 8/4 would raise the dual iso to 100/6400 which would deteriorate the image resolution and colors.

Hopefully will be able to run a new test with extended times for a more complete analysis.

The dickering with slide car files is amazing!
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Re: Dual ISO - massive dynamic range improvement
« Reply #1110 on: September 18, 2013, 06:40:52 AM »
Reporting new test of TIMELAPSE with ETTR and DUAL ISO link:

5D3 with yestedays compile before memory back-end.  be77944f2b0d


Hi RenatoPhoto, where can I get hold of the binary for the be77944f2b0d? Searched, the forums but could not find any refernce to this specific version.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1111 on: September 18, 2013, 07:53:14 AM »
To limit the max ISO, change it in Canon menu (max auto ISO).

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1112 on: September 18, 2013, 01:51:17 PM »
Hi RenatoPhoto, where can I get hold of the binary for the be77944f2b0d? Searched, the forums but could not find any reference to this specific version.
You would have to compile it yourself.  I suppose today's build should be as good, I don't know if there are any issues.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1113 on: September 18, 2013, 01:52:57 PM »
To limit the max ISO, change it in Canon menu (max auto ISO).
The problem is I need the base ISO of 6400 for the night shots but I dont want DUAL ISO to go very high.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Re: Dual ISO - massive dynamic range improvement
« Reply #1114 on: September 18, 2013, 05:16:08 PM »
You would have to compile it yourself.  I suppose today's build should be as good, I don't know if there are any issues.

Don't know anything about building. Guess, will wait till I a complied version is available.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3657
  • Blunt and to the point
Re: Dual ISO - massive dynamic range improvement
« Reply #1115 on: September 18, 2013, 05:29:13 PM »
Don't know anything about building. Guess, will wait till I a complied version is available.

http://builds.magiclantern.fm/#/

Marsu42

  • Contributor
  • Hero Member
  • *****
  • Posts: 1557
  • 66d + flashes
Re: Dual ISO - massive dynamic range improvement
« Reply #1116 on: September 18, 2013, 08:40:37 PM »
The problem is I need the base ISO of 6400 for the night shots but I dont want DUAL ISO to go very high.

That's why I recently proposed to turn the "dr gained" field into a setting - if you for example set it to at least 1.0 ev, dual_iso would be disabled on high iso settings

Canon eos m

  • Hero Member
  • *****
  • Posts: 562
  • 5D Mark III, Canon EOS M
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1118 on: September 18, 2013, 10:35:12 PM »
I used ETTR + DUAL ISO to do a timelapse from night to day.

The workflow is:

Open all cr2 images and save them as DNG.  This puts the deflicker adjustment (in xml) in the DNG file for further processing.
Process the files with cr2hdr.  This processed the DUAL-ISO dngs and saves them as a normal DNG.
At this point I can import the DNG into Vegas and make the video.  It is perfect at this point.
Now I want to generate tiffs which can be handled by other video editors, not all can read the dngs.
To do this I open the DNGs in ACR and notice that the deflicker exposure is in the exposure field.  If I export the tiffs with this exposure number then it generates flicker, it appears that this exposure has already changed the image itself so the exposure number is not needed for further processing.  So I select all the dngs in ACR and set the exposure to 0.  Problem fixed, now the tiffs can be imported to video editor and there is no flicker.

Finally I get another problems that I cannot resolve:
The Dual ISO dngs that were converted to tiffs get some really strange effects like if there was a crop!  After analyzing the faulty images in ACR I noticed that the DUAL ISO dngs have a CAMERA PROFILE set to embedded and cannot be changed, while the standard dngs have CAMERA PROFILE = Adobe Standard.  I cannot find anything else that is different from the two files so I wonder if this is causing this strange effect.

Something interesting to note is that the video made with the dngs coming out of cr2hdr is ok.  I assume that Vegas cannot read the much of the embedded information and is just importing the image.  When I import these dngs into ACR to make the tiff is when this problems arises.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1119 on: September 18, 2013, 10:48:36 PM »
Do they both have the same resolution? If not, try developing the CR2 files with dcraw.

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1120 on: September 18, 2013, 10:52:43 PM »
Do they both have the same resolution? If not, try developing the CR2 files with dcraw.

They are cropped  The tiff from standard dng is 5796x3870  while the tiff form dual iso dng is 5760x3840. 
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1121 on: September 18, 2013, 10:57:44 PM »
Swap the numbers ;)

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: Dual ISO - massive dynamic range improvement
« Reply #1122 on: September 19, 2013, 04:12:07 AM »
Swap the numbers ;)
Right: standard dng is 5760x3840 while the tiff form dual iso dng is 5796x3870.

Ok I was able to trim the dual iso tiffs using Photoshop action and batch.
The timelapse is good but I wll try it again with different settings.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

jOt

  • New to the forum
  • *
  • Posts: 12
Re: Dual ISO - massive dynamic range improvement
« Reply #1123 on: September 19, 2013, 09:16:12 PM »
I got a problem with conversion from interlaced dual iso dng to normal dng. I shot fire in the dark with fps override to 65 fps. Brighter shots are converted correct, but darker aren't recognized as interlaced. Even in the same take one (brighter) frame is recognized well...

Code: [Select]
Input file     : E:\81_22-0544_00016
5.DNG
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : BddB RGGB
Noise levels   : 8.00 8.00 8.00 8.00 (14-bit)
Estimating ISO difference...
ISO difference : 4.61 EV (2450)
Black delta    : 67.67
Interpolation  : mean23-chroma5x5-alias
Dynamic range  : 10.35 (+) 10.35 => 14.97 EV (in theory)
Matching brightness...
Looking for hot/cold pixels...
Full-res reconstruction...
ISO overlap    : 3.0 EV (approx)
Half-res blending...
Chroma filtering...
Building alias map...
Filtering alias map...
Smoothing alias map...
Noise level    : 8.00 (16-bit), ideally 8.00
Dynamic range  : 12.35 EV (cooked)
Output file    : E:\81_22-0544_00016
5.DNG
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 149) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 151) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 154) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
Image/ExifTool/MinoltaRaw.pm did not return a true value at (eval 156) line 3.
Can't find table Image::ExifTool::MinoltaRaw::PRD
    1 image files updated

...and next other not.

Code: [Select]
Input file     : E:\81_22-0544_00016
6.dng
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : BdBd RGGB
Interlacing method not supported
ISO blending didn't work

Input file     : E:\81_22-0544_00016
7.dng
Canon EOS 5D Mark III detected
Full size      : 1920 x 508
Active area    : 1920 x 508
White level    : 12500
Black subtract didn't work
ISO pattern    : dBBd RGGB
Noise levels   : 8.00 8.00 8.00 8.00 (14-bit)
Estimating ISO difference...
Doesn't look like interlaced ISO
ISO blending didn't work



Is there a way to force cr2hdr.exe to convert dngs or other method than cr2hdr.exe?
Please, help. This footage is extremely important to me.

Camera: 5DM3
tried different versions of cr2hdr.exe

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Dual ISO - massive dynamic range improvement
« Reply #1124 on: September 19, 2013, 09:23:13 PM »
Well, if you really want to remove the error checks, feel free to do so, but you'll get just garbage.

Hint: I can't troubleshoot anything from these logs (other than reaching the above conclusion).