Flicker Free ETTR Timelapse: - -Beginners Guide & Basic Post Processing --

[RenatoPhoto]

If you have never seen a Timelapse go here: http://timescapes.org/

If you want to learn all the basics and advanced methods to obtain timelapses go here: https://timelapse.org

Make sure you go to the Forum for the latest discussions on this subject. https://timelapse.org/forums/

.. But if you want to learn how to use the latest Flicker Free ETTR Timelapse feature in ML continue reading...



Flicker Free ETTR Timelapse -- Camera Options -- Prerequisites

• Make sure camera is in M mode, and ISO not in Auto.
• Raw based exposure feedback.
• Auto ETTR.
• Intervalometer.

For the actual post deflicking, you may want to use the post deflicker module with instructions below, or this Script for deflickering and ramping ACR (.xmp) settings in Bridge by dmilligan.



ETTR Flicker-Free Timelapse  (ETTR-FFT) -- HOW TO USE IT (step-by-step)

First we set up all of the ETTR features:

1. Go to Canon Menu and set Image Review to some other setting then OFF.  2 seconds is fine.

2. Put  the camera in manual mode M.

3. Press trash button to enter ML menu.

4. Go to the Expo Menu and enable Auto ETTR by pressing set.

5. Press Q if you want to change ETTR settings.  Default settings are good but you may want to adjust Slowest shutter depending on your needs.

6. Press Q and move to Overlay menu.  Make sure Global Draw in On, all modes.  Enable some raw based exposure feedback.

7. Go to the Shoot menu.

8. Scroll to Intervalometer and press Q to change it to wanted settings.

9. Before we enable the intervalometer we want to preset the camera ETTR so press trash button to return to photo mode.

10. Take two or three pictures until the RAW histogram ETTR hint is less than 0.2

11. Erase the pictures you took and press trash button to return to ML.

12. You should be in the Shoot menu, scroll to the intervalometer and Press Set to ON.

13. Press trash button to exit ML menu.  The intervalometer is now functioning and the red led will blink as long as intervalometer is working.  The intervalometer will start depending on the start trigger setting.

NOTE:  If you set your intervalometer at 30 seconds and your slowest shutter speed at 30 sec, you will have problems.  Set the intervalometer at least = Slowest Shutter speed plus the review image setting, in Canon Menu, plus 3 more seconds.  So if the slowest shutter is set to 30 seconds, the intervalometer should be set to at least 40 seconds or more.


If you are using the Post Deflicker module, there are some other options you should set.

1. Scroll to Post Deflicker and press Q.

2. Set Sidecar file type to UFRaw or XMP , and check the other settings i.e 50% and 4 EV.





Watch the camera for a couple of shots to make sure that the images are good and ETTR is settling nice for a long day or night of work. 

Come back from time to time to make sure it is doing a nice job...

wait all day .........

While you are waiting for the timelapse lets get going on setting up the software to process the files.


Post Processing ETTR-FFT Basic Steps


XMP WORKFLOW for PC

1: Make sure your camera time is not ahead of the computer time you will be using.

2: If the name of your CR2 file does not agree with the xmp, change the XMP always.  In this manner the xmp will be modified after the CR2.  If you do it the other way ACR will not read the files.  Always the CR2 Modified Time Stamp mus be before the xml Modified Time Stamp.

3. Once you have the cr2 and the xmp ready for processing you open all of the cr2 files with ACR at once.  If you have too many files for your pc capabilities then do them in groups.

4. In ACR click on Select All. Top left.

5. Make sure that the sliders are not modified in the Basic tab.  Do all the exposure adjustments in the Tone Curve tab.

6. Click on Save Images (Bottom Left) and choose your preferred type.  I use tiff for best results.



Adjusting exposure for XMP with exiftool

With this little tool you can also adjust the exposure of the xmps but you cannot ramp the exposure.

Dowload exiftool here: http://download.cnet.com/ExifTool/3000-2193_4-10791213.html
Place it in the directory where the xmp files are.  (Always make a copy of the original xmp)
Open the DOS promt (Type cmd at the windows comand line)
Navegate to the directory where the xmp files are located
Type: exiftool -Exposure2012+=x *.xmp
Where x is the ammount of adjustment, could be +ve or -ve.
The exiftool will report back how many files where adjusted.
If you do not put the + sign after the 2012 exfitool will just set the x exposure to all the files.  With the + sign it will add that exposure to the xmp.



TROUBLE SHOOTING XMP WORKFLOW

ETTR is a module to expose your images to the right so as to maximize the signal-to-noise ratio, meaning have lower noise in your shadows and probably better color.  ETTR has nothing to do with deflicker, it is just exposing each image to the best of the cameras abilities to try to capture all the details possible.  This module has become more complicated and powerful and now you have better control of what is that you are trying to capture best.  For more information go here: http://www.magiclantern.fm/forum/index.php?topic=5693.0

If the change in exposure from one shot to the next is extreme, then ETTR will be going all over the place trying to adjust to each exposure.  For this reason it could seem that exposure ramp via AV or Auto ISO is smoother.  I have experienced this kind of variation when the interval time  in the intervalometer is very long, ie 2 minutes.  I never use ETTR with and interval time longer then 1 minute.  I had a situation with passing clouds where the exposure was varying quickly and ETTR produced very bad results so a shorter interval time is always better.  In some cases it may be better to use other exposure methods.

Also do not set the intervalometer to take pictures every 2 secods because there will not be enough time for ETTR and Postdeflicker calculations.  Minimum intervalomenter setting should be above 10 seconds.

ETTR is not an instant perfect photo mode, what I mean is that it can take more than two shots to get the software to converge into an ideal exposure to the right.  So if the exposure changes from one shot to the next then ETTR is in a hunting mode, AND in this case do not use ETTR.

Post Deflicker is another beast all by itself.  I dont know the math behind this but it must assume some kind of average exposure ramp and it adjusts each shot via xmp file so as to bring the exposure of that shot very close to the average exposure ramp.  I assume that this algorithm can also get somewhat screwed if there are a lot of exposure changes from one shot to the next.

Assuming that all is working properly, when you import a cr2 file into ACR or lightroom, it has an exposure adjustment of zero, but when the software reads the xmp file, then the exposure will be adjusted by some amount sufficient to bring the exposure to some predefined standard by the Post Deflicker algorithm.  For example when I open the files in ACR without the xmp files and I walk through the images, it is clear that the exposure is changing from one image to the next.  If I open the same files accompanied by the xmp file, then the exposure change from one image to the other is minimal, and I can see how the exposure slider changes from one image to the next.



XMP WORKFLOW to ADD an Exposure OFFSET for MAC

Doyle; many thanks, your tutorial worked a treat.

Haven't quite worked 'how' it has worked, but I seem to have ended up with some additional XMP files with the suffix 'original' at the end of them. Perhaps Exiftool is being friendly and making backups of the originals? 

To clarify (if anyone else is having the same issues) this was my process;

1 - Download and install exiftool FOR MAC (There is a specific Mac OSX package)
2 - Prepare/move CR2 files and .xmp files from Magic Lantern in one folder
3 - Open Terminal
3 - Enter the command as Doyle explained above;

exiftool -Exposure2012+=+*.** /drag folder here containing cr2's and .xmp's

Where the *.** is enter the amount you want to set
eg 0.05, next to the number you have chosen
Too the left is +
+=over expose
-= under exposure

4 - Allow process to run (New/backup XMP files were created for me. I waited until the list had finished)
5 - Select all CR2 files, open in Adobe Camera Raw
6 - FILES SHOULD OPEN, DE-FLICKERED
7 - Select all, make White Balance auto. (Unsure about next bit?...)
Make any edits should you want them? Note, that if you change exposure now, you'll undo the de-flickering!
(Perhaps edits should be done first, then you start the de-flickering command afterwards? Needs testing!)
8 - Close Adobe Camera Raw
9 - In After Affects, import the first CR2 image, select 'Camera Raw Sequence.' Create timelpase!

Many thanks Doyle! Maybe this Mac step by step should be added to that top post/tutorial thread?

More on this subject on the original post here: http://www.magiclantern.fm/forum/index.php?topic=6789.msg78647#msg78647

Also if you need to rename multiple files in Mac:  http://namechanger.en.softonic.com/mac

Exiftool plugin for lightroom. (Thanks Dane)  http://www.robcole.com/Rob/ProductsAndServices/xEmPLrPlugin/



UFRAW WORKFLOW


1. Download UFRAW basic version.  You can find it here in many flavors:
http://ufraw.sourceforge.net/

I used the windows for dummies version from here:
http://sourceforge.net/projects/ufraw/files/ufraw/ufraw-0.19.2/ufraw-0.19.2-2-setup.exe/download

2. Install it all default settings.

3. Download this little .bat file I created to run it in batch mode here:
https://docs.google.com/file/d/0B9z8Y0rg-pu8bE9XaUdQcnJPQzA/edit?usp=sharing

or.. make your own bat file with the following commands:

cd C:\Program Files (x86)\ufraw\bin
ufraw-batch I:\ufrawconvert\*.ufraw --out-type=jpeg

Note that you must have the correct directory for   \ufraw\bin

Also note that you must have another directory selected for the place where you put the timelapse files, in the above example it is I:\ufrawconvert\

3a. If you downloaded the bat file you will have to edit.  Note that if you placed that bat file in the C: drive you will have to start the notepad as an administrator; this also depends on what type of operating system you have.  so before you move it to the C dir, just edit with the necessary changes. 

3b. Now save it and copy it to the \ufraw\bin directory.  Again you may need administrator permits to do this.

3c. Now create a short-cut (select the file, right mouse, Shortcut) to this file and copy it to your desktop.

If you double click on this link the ufraw batch system will kick in and start processing the files.  But before you do this lets go to step 4.

4. Wait for the timelapse to finish, and then we are going to copy the files to the I:\ufrawconvert\ directory, or to whatever directory you set in the bat file.

MAKE SURE YOU COPY THE .CR2 and the .UFR files.  There should be a corresponding .UFR  file for the .CR2  If the names don't match then we will have to rename them.  More on that later.  For now just copy them.

5. If the names are not identical then download Bulk Rename Utility here:
www.bulkrenameutility.co.uk  (Also in Mac:  http://namechanger.en.softonic.com/mac)

6. Open Bulk Rename Utility and navigate to where your files are.   There are a lot of options on this little thing but we will only use a couple.  Our goal will be to make the .CR2 files names match the UFR files and then to rename the .UFR files to .UFRAW.

Let's assume that the CR2 file says IMG_7064.CR2 and that you want to change it to HP6A7064.CR2

Click on the CR2 file, now look at the third selection column (below the files) on the top row and it should say Remove.  On the little box that say first you can enter 4 to remove the first 4 letters from the file.  Now the CR2 file should  read 7064.CR2.  Now look to next selection column where it says ADD and in prefix put HP6A.  Now it should read HP6A7064.CR2  If you have it right then select all of the CR2 files and press Rename on the bottom right corner of the application.

But note that the numbers must be corresponding so make sure you do not change any numbers.

Now select one of the . UFR files and reset the boxes that you previously modified.  Find the selection column number 11, this is Extensions.  Click on the drop-down box and select Fixed, type the UFRAW extension.  Notice how the HP6A7064.UFR is now renamed to HP6A7064.UFRAW  If all looks good then select all of the UFR files and  press Rename on the bottom right corner of the application.

Done!

7.  Go back to your desktop and double click on the shrortcut-link you created for the batch file.

If all goes well you will see UFRAW going to work... for a long time

8.   Watch the program details because it may not send the files to the destination folder.  The first time I ran, it worked flawlessly but the second time it sent the files to a \ufraw\bin directory but I could not find them.  In my case it says it is going to the C:\Program Files (x86)\ufraw\bin directory but I found them in the C:\Users\RE\AppData\Local\VirtualStore\Program Files (x86)\UFRaw\bin   

To simplify the whole process I went to this directory, went up one level, and made a short-cut which I copied back to my Desktop.  So now I have a shortcut for the batch file and a shortcut for the directory where it saves.

After all of the files are processed get them out of there to another DIR and import them to your video software.



Todo:  Add some silent pic specific stuff.

[platu]

At the moment, when using the silent pictures option and post deflicker, sidecar files are not created.  Is it possible to create sidecar files for the silent pictures DNGs?  Or are there technical differences from CR2s that would make this a more difficult prospect?

[RenatoPhoto]

Support to XMP slidecar files as follows:

The XMP file need to be converted to UTF-8 format.

1. Download this little converter: http://www.softpedia.com/get/Science-CAD/UTF-8-converter.shtml

2. Process all of you xmp files to change their format

3. Open the images in ACR and the exposure will be automatically adjusted based on the information of the XMP file


I also tested with Lightroom 4.2, ACR 7.1, and ACR 6.2 and all work!!!

Note: make sure you .XMP files have the same name as the .CR2 files.  Look in the guide above for Bulk Rename Utility.

[minimimi]

@RenatoPhoto

Are you using Adobe on Windows??

It's just my guess, Current code  generated  a "\n" return code.
But Windows using "\r\n". 
Ofcource Windows understanding both "\n" and "\r\n" but it's depend on software programmer(depend on which Windows API use).

So UTF-8 converter is automatically convert retrun code, then we can use xmp , I guess.
If you have both(befor/converted) zip and share it. I can confirm it

[1%]

ETTR will work on 600D, just needs to stop checking for "raw" quality. Its ok in plain old modes where you don't set sraw.

XMP functionality I haven't tested.

[minimimi]

Alex,
The first 1byte is some special char. But it's really strange. I'm checked codes, But i cant find bug.
- FILE* f is not move when we do my_printf
-xmp_template has no special char.



@RenatoPhto
Got it.
Notepad delete first special char.
UTF-8 converte is successfully convert 1byte strange char to UTF-8 , so we can read it.

My guess is fault. All files are using "\n".

[RenatoPhoto]

Yea, that is the problem.  Text wise there are no differences between the files that work and the files that dont work so it is that special character that we cannot write via ML that prevents the xmp to be read directly by ACR.

So up to now we still have to use the utf-8-converter.exe (8K) to do the dirty work.

On the other thread there is the code for a LRTimelapse XMP.  Now it will be a good thing to figure out what is important for the latest ACR profile to work and to put that code into the XMP.

[a1ex]

It's not the special character either, because ML can write it and did at some point. It's the timestamps. See here for my current hypothesis: http://www.magiclantern.fm/forum/index.php?topic=5724.msg46537#msg46537

[RenatoPhoto]

OK, I though the previous commit tested that idea, so I already gave up!  As i said before, if I just open and save the files in notepad they work.

[minimimi]

Ah! You already delete it Alex.


Anyway, I think ASCII and UTF-8 is compatible when we use only ASCII chars.
I will try to find convert issue on my own equipment.

Modified
You already solved this , I can read current XMP to ACR directly.

[Makaveli6103]

I tried to convert the ufraw files to tiff using back processing but it still converted them to jpeg. I just changed jpeg to tiff in the command. Did anyone try this and get it to work?

[Marsu42]

The first 1byte is some special char.

Afaik this is a BOM, there actually are two ways to write a UTF-8 file: https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark ... for example Notepad++ supports both encodings so you can see the difference: http://notepad-plus-plus.org/

[glubber]

I made my first timlapse with ETTR this week. I'm very pleased with the results, the transitions are smooth and without flicker.
Sadly the final timelapse was not really exposed ETTR, but i guess that was more a problem of where to target the spotmeter. 

I was using the Ufraw-sidecar, which left me with some pink highlights.


IMG_2918-UFR

A research via google returned like its a known bug for some Canon DSLRs.
As i'm using Windows (or no ability to compile myself) and i'm not familliar with Ufraw i cant' give a workaround at the moment.
I will look further into this.
I just wan't to give a hint to other users it's no fault of ML or the user themselves.

https://bugs.launchpad.net/ubuntu/+source/ufraw/+bug/780754
http://sourceforge.net/p/ufraw/discussion/434060/thread/b43582e3/

What amazed me most with the Timelapse is despite of stopping the invalvometer midway
(coz the pictures were getting to dark) the ML-ETTR catched up almost flawless thanks to right ETTR-processing!

Well Done Devs!

[a1ex]

Did you use ISO 160 multiples? At these ISOs, the white level is lower than what's declared in dcraw source code, and I'm not sure ufraw takes care of this.

Can you upload a CR2 file?

[glubber]

I shot at ISO 100.
I justopened the original CR2 with Rawtherapy4.0 (i think uses also dcraw), and when i enable "highlight reconstructions" the pinks are gone.

https://www.dropbox.com/s/3oppxb9j1lh8jhc/IMG_2918.CR2

[alexunderboots]

please share a link to autoETTR module

[glubber]

please share a link to autoETTR module

It's in the nightly builds already.

[screamer]

hi all,
i've already posted somewhere else, because i don't know where is the right place.
anyway, i'm on a 6d with the latest build. i'm trying to figure out how to use auto ettr module with intervalometer and post deflickering, using xmp files and lightroom.
I've already readed all around about that, and i have the xmp working in lightroom, using bluk renamer for the date issue.
But the problem is that i have alway bad values for deflickering.
i will try to explain with realworld examples:

Deflicker target level: -1ev
values from +8 to +9 (instead of what i will expect, from -1 to +1 or something similar. the scene is correctly exposed, and the autoettr seem to work fine)

Deflicker target level: -8ev
the most usable (but not at all) with values from 0 to 5 (or something like that).
now the point is that acr and lightroom can handle only 4 ev stops, and all the xmp is written with 3.99.

am i doing something wrong?

[noix222]

thanks...

[swinxx]

is this possible with the ml firmware?
if yes, perhaps someone can explain how it can be done. thx

http://www.youtube.com/watch?NR=1&feature=endscreen&v=PO514P2u8NI

[Danne]

This is quite a remarkable feature, creating flickerfree pics with the ettr function. Even works with silent mode on. Xmp works fine i lightroom. A question. I,m getting out to do i sunset timelapse and I,m not sure whether to keep the settings in manual mode or Av mode? Won,t the xmp level upp the darks in ettr mode both in av and in manual mode?

[a1ex]

You can expose with any method you want (even by turning the aperture ring manually); post deflicker doesn't care about that.

I recommend to expose with ETTR, as it will try to keep the highlights and minimize the noise.

[Danne]

Thanks, gonna try it out

[DJHaze596]

I cannot get the Batch to work at all.  I've done everything it said.  I just don't get it lol.  Here's what my batch says. and yes i have UFRaw installed.

cd C:\Program Files (x86)\UFRaw\bin
ufraw-batch C:\Users\Johnnie\Desktop\Timelapse\ufrawconvert*.ufraw --out-type=jpeg

[glubber]

I cannot get the Batch to work at all.  I've done everything it said.  I just don't get it lol.  Here's what my batch says. and yes i have UFRaw installed.

cd C:\Program Files (x86)\UFRaw\bin
ufraw-batch C:\Users\Johnnie\Desktop\Timelapse\ufrawconvert*.ufraw --out-type=jpeg

You are missing the "\" before "*.ufraw"
ufraw-batch C:\Users\Johnnie\Desktop\Timelapse\ufrawconvert\*.ufraw --out-type=jpeg