ETTR + DUAL ISO = 14 Stop Dynamic Range Timelapse - Postprocessing Guide

[RenatoPhoto]

Success!  After a bit of struggling I managed to get a very nice (night to day) timelapse with clean shadows and almost not clipping in the highlights. 

USED 5D3 + ETTR + DUAl ISO

ETTR settings:
Trigger mode. Always on
Slowest shutter: 32"
Highlight ignore: 0.5%
Allow clipping: OFF
Midtone SNR limit: 6EV
Shadow SNR limit: 2EV
Link to Canon Shutter: OFF
Link to Dual Iso: ON
Shadow metered areas: OFF
Show debug info: OFF

Dual ISO (enabled)

Intervalometer:
Take pic every:  1m, Start after: 3 sec, Stop after: Disabled, Manual FocusRamp: OFF

Post Deflicker:
Sidecar file type: Adobe XMP
Deflicker percentile: 50%
Deflicker target level : -4EV


The modules performed very well automatically without any human intervention.  The only thing I noticed is that when Dual Iso kicked in, there were some images that alternated between DUAL ISO 100-200 and no DUAL ISO.  After a while the system stabilized and all of the photos thereafter where DUAL ISO.

POST-PROCESSING with XMP slidecar files (ACR and Photoshop required)
This proved to be the more difficult part since the exposure compensation is not passed through the cr2hdr process.  Maybe this can be updated in a future release making this process a slicker deflicker. 

1. Open all the CR2 files with ACR (xmp files will adjust the exposure) and save them as DNG.
2. Process all of the DNG with cr2hdr
3. Change the extension of all DNG back to CR2.  Make sure the xmp files are put back with the CR2
4. Open the CR2 with ACR (again) and save them as tiff, this will adjust the exposure of the tiff files by the deflicker exposure.   Save then to a separate folder.  Make sure that the exposure compensation is applied in ACR.
5. In Photoshop create a small action to open a file and change the canvas to the correct size of 5760x3840, then save the file.  This is required since the cr2hdr process is increasing the size of the images to 5796x3870. Again this is something that may be able to program in cr2hdr so that the output images do not change in size.
6. In Photoshop open the Batch command (in File, Automate) and select the action you created by selecting the Set and the Action. 
7. Click on Source and select Folder.
8. Click on Choose and select the Folder where you saved the tiff files.
9. Select the Override Action "Open" Commands
10. Select the Override Action  Suppress file Open Options Dialogs.
11. Click on Destination and select Folder
12. Click on Choose and select a folder for the new tiffs
13. Select Override Action "Save As" Command and Click OK.
14. When needed the images are automatically re-sized , you can import these tiffs in your favorite video editor.

Technical Notes;
I also tried sending all the cr2 files directly to the cr2hdr which converted the DUAL ISO images to DNG.  Then rename all the DNG back to CR2 and then along with the xmp open the cr2 to save as tiff.  The problems is that the converted DNGs (converted to CR2) in this manner will never accept the exposure compensation from xmp file.  I tried and tried and could never get them to accept the deflicker exposure.

There are a couple of places (1. and 4. above) where you can adjust shadows, highlights, etc.  I only use tonal curve in ACR to avoid flickering.

[Danne]

Nicely done. Any link to the actual timelapse?

[RenatoPhoto]

Some day when I have bandwidth.

[Danne]

my xmp gets a prefix like this 042A1046.XMP while my cr2 get the regular IMG_1046.CR2? Any ideas? It,s not the biggest problem to rename to get the xmp to work, but still, wonder why?

Could you explain why you set the aettr auto midtone and shadow SNR limit to 6/2ev and not 8/4ev? I assume you tried and found out 6/2 to two be optimal? Curious since I find myself using the 8/4 setting mostly.

Found a way to get the xmp to work with a slightly different method
1 - importing all dual-iso cr2 with xmps to lightroom(Had to change prefix to same as my CR2-files). This gives me the flickerfree timelapsestream with unconverted cr2(still with lines on them)
2 - Convert the CR2 with cr2hdr. Rename the converted files with suffix .dng to .CR2
3 - Overwrite the CR2 originals in the lightroomfolder with the converted files. This will have lightroom "remember" the xmp settings from the files in the very beginning. Should work with acr in photoshop as well. Didn,t try.

Maybe not the most beautiful solution but since I,m on a mac I can,t convert dng:s to dng,s
Thanks for the nice tutorial @renatophoto


Postprocessing
Tried working with the tonal curve tool in lightroom to lift shadows etc to my likings. Also applied contrast, saturation etc and then synchronized all the pictures to apply the changes. See to it you don,t set the exposure slider in the synchronizing process cause it will set the exposure the same for all pictures.

[Audionut]

Could you explain why you set th aettr auto midtone and shadow SNR limit to 6/2ev and not 8/4ev? I assume you tried and found out 6/2 to two be optimal? Curious since I find myself using the 8/4 setting mostly.

You have to weight up the benefits of increased midtone/shadow SNR vs all of the negative effects of dual ISO (reduced resolution in the highlights etc etc).  At least, that's what I do.  RP might have a different reason.

[Danne]

Ok, less aliasing can,t hurt
Updated my xmp "hack workflow" a little.

@Renatophoto
Question. When you get have all the converted dual iso files to tiff without any flicker do you adjust any shadow and highlights and keep the exposure alone to get most out of the files or do you avoid touching any settings in post process?
Only tried a little but my files needs a little postprocessing to get to out the best of the timelapse

Do you experience the same here? "my xmp gets a prefix like this 042A1046.XMP while my cr2 get the regular IMG_1046.CR2? Any ideas? It,s not the biggest problem to rename to get the xmp to work, but still, wonder why?"

[RenatoPhoto]

my xmp gets a prefix like this 042A1046.XMP while my cr2 get the regular IMG_1046.CR2? Any ideas? It,s not the biggest problem to rename to get the xmp to work, but still, wonder why?

xmp uses the canon standard prefix.  For my camera 5D3 it is HP6A, yours must be something different.  It does not pickup your custom preset.
I use www.bulkrenameutility.co.uk for quick changes.

Could you explain why you set the aettr auto midtone and shadow SNR limit to 6/2ev and not 8/4ev? I assume you tried and found out 6/2 to two be optimal? Curious since I find myself using the 8/4 setting mostly.

8/4 raises dual iso to high and the image quality deteriorates beyond my liking.

Found a way to get the xmp to work with a slightly different method
1 - importing all dual-iso cr2 with xmps to lightroom(Had to change prefix to same as my CR2-files). This gives me the flickerfree timelapsestream with unconverted cr2(still with lines on them)
2 - Convert the CR2 with cr2hdr. Rename the converted files with suffix .dng to .CR2
3 - Overwrite the CR2 originals in the lightroomfolder with the converted files. This will have lightroom "remember" the xmp settings from the files in the very beginning. Should work with acr in photoshop as well. Didn,t try.

Interesting that you did not experience the change is image size.  In my case the dng processed by cr2hdr increase the image size (border pixels) to the maximum sensor size i.e.  5796x3870  The problem only becomes apparent when you mix cr2hdr images with images where there was no dual iso process required.

Tried working with the tonal curve tool in lightroom to lift shadows etc to my likings. Also applied contrast, saturation etc and then synchronized all the pictures to apply the changes. See to it you don,t set the exposure slider in the synchronizing process cause it will set the exposure the same for all pictures.

Yes I use tonal curve extensively as much as +/-100% change.  I do not touch the settings in the exposure tab of ACR to prevent flickering.  I should probably mention in the OP

[Danne]

Thanks for answers.
Havn, t mixed dualiso with non dualiso files yet. Now I know what to do when that happens.
//D

[Danne]

Went out for two hours creating a timelapse with aettr and dual iso. Went for the 6/2 midtone, shadow setting which worked very well. Thanks for sharing all info.

[RenatoPhoto]

Finally uploaded my 14 stop dynamic range night-to-day timelapse with ETTR+DUAL ISO on 5D Mark III
SNR 6/2 h.i. 0.5%
Intervalometer at 1 minute

[a1ex]

I've expected a little less overexposure, TBH.

0.5% is a lot bigger than you think; use "show metered areas" to see how big it really is.

[Danne]

@Alex
The settings for my timelapse "Vallbybron" was set to 0,2 hi, and also, by mistake to -2ev, exposure target. What I find difficult is going from light to dark. The xmp post deflicker seem to even out the dark areas overexposing going from daylight to nighttime? Am I missing something?

With post deflicker, as in the finished movie


Without post deflicker, exposure, shadows, whitebalance corrected manually synchronizing all cr2 files

[a1ex]

You need to adjust the exposure in post; obviously, post deflicker only takes care about flicker, it doesn't know about what look you want to get.

I'm working on a deflicker tool that does ramping too.

[Danne]

Ok, ramping then. Thanks again for all the tools you bring to the camera. The best

[RenatoPhoto]

I've expected a little less overexposure, TBH.

0.5% is a lot bigger than you think; use "show metered areas" to see how big it really is.

I thought so, I will have to try a new one. 
Since each timelapse takes hours and early rises with good conditions, it is not easy to do.  Should I go for 0.2% or 0.1%, or 0% h.i.?
Any other suggestions?
Thanks

[a1ex]

On my last experiment (sunrise) I went with 0.1% and target -1. Lowering the target helps avoiding overexposure if the ramping gets faster than ETTR.

Will post the results soon.

[Doyle4]

I'm working on a deflicker tool that does ramping too.

Nice, cant wait too see this.

Separate to your tool you are working on, Iv read people are wanting to gradually wb change, think this will be possible to do with xmp's? kinda how deflicker works but increasing/decreasing wb numbers. If and when this can be done it'll change way people do time lapses again

[a1ex]

If you teach me how to change WB with XMP, I'll try.

Right now, my tool is mostly based on dcraw+ufraw, but the backend is easy to change.

[Doyle4]

Sounds like its going be a nice tool to have, looking forward to it.

haha i wish i could teach you the WB gradual change sorry i was meaning it'll be great when its possible, i can see it been a problem as wb numbers go up/down by 1 and not decimal changes. Iv seen a tutorial on youtube that shows you how to change wb on a jpeg using hue/saturation, tricky and long process but could be a start to building a wb change maybe later in future if someone can figure out the math.

[RenatoPhoto]

If you teach me how to change WB with XMP, I'll try.

Right now, my tool is mostly based on dcraw+ufraw, but the backend is easy to change.

Here is a copy of an xmp file which I was able to modify the wb and have ACR accept.

It is the same one produced by the camera just added the WB stuff and changed the order to match the order in which it is placed in the ACR original xmp file, and added the crs:AlreadyApplied="False"> line in the end.

This I got to work by trial an error, so I cannot be sure if it the correct way of doing it.  The file can also be open with Lightroom 4 and it read the correct wb and exposure adjustments.

Code: [Select]

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Magic Lantern">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
    xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
   photoshop:DateCreated="2050-01-01T00:00:00:00"
   photoshop:EmbeddedXMPDigest=""
   crs:ProcessVersion="6.7"
   crs:WhiteBalance="Custom"
   crs:Temperature="3000"
   crs:Tint="+13"
   crs:Exposure2012="-1.09"
   crs:AlreadyApplied="False">
   <dc:subject>
    <rdf:Bag>
     <rdf:li>ML Post-Deflicker</rdf:li>
    </rdf:Bag>
   </dc:subject>
  </rdf:Description>
 </rdf:RDF>
</x:xmpmeta>

[Doyle4]

Nice!! excellent find RenatoPhoto!

[Welles]

Here is my second attempt of an AETTR + DUAL ISO time-lapse.

I am mostly satisfied, but the end result does not match the quality I want to achieve (yet). I followed the method of the first post of the thread.

Around 10% of the images did not survive the cr2hdr process (latest official zip, PS or LR were unable to reopen the DNGs renamed to CR2 (step 4)). I skipped a few other images, which exposure compensation was not good, I mean a perceived difference of more than 1EV from one image to the next. I go with default settings, except the intervalometer set to 30s, max shutter 10s. I also use a 1s mirror lockup.

I would appreciate if you have any advice about that. I'll try next without dual iso (to see If I still get those glitches in exposure compensation) and then update the Magic Lantern to the latest beta.

[a1ex]

The deflicker method is not really usable with dual iso, mainly because it doesn't know what will happen during postprocessing. Your best bet is deflickering in post, for example with this script: http://www.magiclantern.fm/forum/index.php?topic=8850.0

Can you upload a few troublesome frames? (some that couldn't be processed and some that resulted in large exposure difference)

[Welles]

Thanks! I will have a look at the deflicker script. Do you know how if this script is equivalent to DEFlicker from RE:Vision Effects?

Here is an example of a frame that is processed by cr2hdr (the link below is to the original CR2), but is unreadable afterwards by either PS or LR (following the steps above CR2-->DNG (from ACR)-->cr2hdr-->rename extension back to CR2).

https://filesender.renater.fr/?s=download&token=5ad39abc-7b1d-06df-0646-096dcb9b378c

Here is a frame that is significantly darker than the previous and the next one (included, with the XMP file):

https://filesender.renater.fr/?s=download&token=00ec267e-dfd3-a7b1-d225-763b59600f5f

[a1ex]

I have no idea about RE:Vision Effects, but the script I've linked uses the same idea as ML deflicker module (robust statistics). The difference is where you apply the deflickering: on raw data before dual iso processing, or after.

The first one opens just fine here in ufraw, RawTherapee and darktable. FYI, the latter also includes a deflicker module with the same algorithm (it was ported from ML). Can't help with PS/LR, sorry.

The second image set is pretty much the worst case for the median (which is the default setting for deflicker). If you keep the sky line in the middle of the image, try setting the percentile to something around 75. Otherwise, let the sky cover a larger area of the image (more than half), so the median value (50th percentile) would fall right there. You could use the debug options from ETTR to see exactly where it falls - that part of the image will have the least flicker. However, if the median falls on a sharp transition between light and dark, its value is not exactly a great choice for deflickering (but I expected this to be a corner case, given the usual composition rules).

Side note: I'm thinking to add a deflicker option directly to cr2hdr; will use this set as test data if I'll ever manage to implement it. It's actually an easy coding task, if anyone feels like brushing up his/her C and copy/pasting skills