Script for deflickering and ramping ACR (.xmp) settings in Bridge

[Joachim Buambeki]

Looks like you selected the undo data file to be deflickered (which is not an image).

Of course, I just pressed Cmd+A to select all files...
Could you modify the script to only process image files and ignore everything else (like undoData, subfolders, etc.)?

I tried it again and now it gets interesting:
In Bridge you can see the adjustments made by the script in the thumbnails and when you open the files in ACR but After Effects ignores the adjustments - not all of them its really weird, the adjustments seem to be taken into account for about half of the sequence but then it stops. Do you have any explanation for that behaviour?
I then rendered the images from the ACR dialogue in Bridge and put the image sequence through compressor to see if the changes are applied - they are (though there still is some shimmering - not sure if I would call it flicker).


I can send you the DNGs for examination via Dropbox if you need them, they are only 200 MB.

JB

[dmilligan]

In Bridge you can see the adjustments made by the script in the thumbnails and when you open the files in ACR but After Effects ignores the adjustments - not all of them its really weird, the adjustments seem to be taken into account for about half of the sequence but then it stops. Do you have any explanation for that behaviour?

Had you already loaded and viewed part of the sequence in AE? If so, it's because you are seeing AE's preview cache (notice the green areas above the timeline). When you make changes inside AE, it knowns to clear the preview cache. But you made changes outside of AE that it wasn't aware of. Simply clear the preview cache.

[Joachim Buambeki]

Had you already loaded and viewed part of the sequence in AE? If so, it's because you are seeing AE's preview cache (notice the green areas above the timeline). When you make changes inside AE, it knowns to clear the preview cache. But you made changes outside of AE that it wasn't aware of. Simply clear the preview cache.

I suspected something like that and made sure this isn't what is causing it. I quit AE and imported the sequence again (actually did that multiple times) but the sequence was still screwed up.

[dmilligan]

quitting AE does not clear the preview cache!

[Joachim Buambeki]

I am not sure if this is the RAM cache, but the problem is still there after cleaning the "Media & Disk Cache" in preferences. If there would be still something in the disk cache it should be visible by beeing blue lines instead of green, right? I don't see those blue lines after importing the sequence again in a new project.

[dmilligan]

then I have no idea

[Joachim Buambeki]

I also get this error after the first pass now:



After pressing OK it stays stuck with purging the cache. (I have plenty of free HDD space if that is a possible cause).

I am also trying to break the script in other ways:
When picking three keyframes (start+middle+end) and choose ramp multiple it only ramps the exposure and clarity but not the temperature that was also changed in the middle keyframe. I tried that a few times and made sure everything is checked in the "Ramp Multiple" dialogue.


When you want to ramp and deflicker you have to ramp mutliple first and then deflicker, right? What about combining that into one process?


EDIT - More testing: 3 keyframes, start underexposed, middle overexposed and high colour temp, end underexposed and same colour temp as start.
When applying the script to CR2 files instead of DNGs the ramping is working as expected when inported into AE (except colour temperature like mentioned above). I assume AE needs the sidecar files for some reason, otherwise it applies the settings of the first frame to the entire sequence. I can't verify that because I don't know how to create .xmp files when processing DNG files.


In case you missed that in the last posts: CC2014, OSX

[getho]

Sorry for being dumb, if you had values of 50, 50, 50 ,50  would that be a rectangle that starts at 50% of the width of the image, and is 50% of the width of the image?  (in which case what does the 200 "analysis size" value above it refer to?

Another dumb question (I will get the hang of this)

Percentiles: are we talking histogram? in which case how does the amount relate to the preview? And how do you analyse your shot to know where these values should be?

[milank]

When applying the script to CR2 files instead of DNGs the ramping is working as expected when inported into AE (except colour temperature like mentioned above). I assume AE needs the sidecar files for some reason, otherwise it applies the settings of the first frame to the entire sequence. I can't verify that because I don't know how to create .xmp files when processing DNG files.

Yes, this is well known issue. I am used to create xmp files using LightRoom rc_xEmP plugin. which works fine, and AE accepts them.
However, the workflow gets somehow complex and I am still looking for easier one.
Either having XMP files directly generated by dmilligan script or to figure out how to avoid ACR processing in AE.
Premiere would be better choice, but it does not seem to understand DNG produced by ML silent module.
DNG->TIFF conversion would be a choice, too.

[dmilligan]

Ramping temperature isn't working because you need to set it to "Custom" for all photos (this doesn't happen automatically, yeah I know: "Can you make this automatic?")

I've never seen the purgeFolderCache error and I have no idea what it could be.

For DNGs Adobe stores the metadata in the DNG itself so there will be no sidecars. If it's not working here's what I would do: select all the DNGs in the ACR dialog. Make some type of change and synchronize it to all of them. This will sort of "initialize" the XMP metadata within the DNG, otherwise the script may not be able to write the metadata correctly (yeah I know: "Can you make this automatic?" Well I'm already calling the function to init the metadata in the xmp API, and it basically seems to not always work. I have no idea why and there's nothing I can do about it, but at least this workaround seems to work)


The preview should show you the analysis area and it will highlight the percentile you selected.

Analysis size refers to how big the image is scaled down to before computing the histogram. The smaller you make it the faster the script but the histogram will be less accurate. (the reason for this setting is that the rescale operation is native and very fast, however computing the histogram is done in JavaScript which is very very slow, so rescaling the image down makes the histogram calculation much faster)

Yes percentiles refers to the histogram, it shouldn't make much difference, the default should be fine.

[ansius]

I recently encountered a glitch with the script behavior, it either did not adjust or deflickering was poor. It took me a while to run down the bug, but basically in adobe bridge you have to set "Options for thumbnail quality and preview generation" to "always high quality".



Any other setting gave me either unpredictable results (like constant +1 stop in exposure for embedded thumbnails if I had shot -1EV) or left a lot of flickering. I have not done more research why, but just for someone who encountered the same problem. (Bridge CS6 5.0.2.4 + ACR 8.4).

As for ramping color temperature - that explains why the setting did not work until I did set in ACR to a certain value for all pictures. maybe this should be an option in script, set it to custom if not set?

[Joachim Buambeki]

Yes, this is well known issue. I am used to create xmp files using LightRoom rc_xEmP plugin. which works fine, and AE accepts them.
...
Either having XMP files directly generated by dmilligan script or to figure out how to avoid ACR processing in AE.
...

Would it be possible to create those sidecar files with the script instead of another Plugin for another software?

Ramping temperature isn't working because you need to set it to "Custom" for all photos (this doesn't happen automatically, yeah I know: "Can you make this automatic?")

For DNGs Adobe stores the metadata in the DNG itself so there will be no sidecars. If it's not working here's what I would do: select all the DNGs in the ACR dialog. Make some type of change and synchronize it to all of them. This will sort of "initialize" the XMP metadata within the DNG, otherwise the script may not be able to write the metadata correctly (yeah I know: "Can you make this automatic?" Well I'm already calling the function to init the metadata in the xmp API, and it basically seems to not always work. I have no idea why and there's nothing I can do about it, but at least this workaround seems to work)

As for ramping color temperature - that explains why the setting did not work until I did set in ACR to a certain value for all pictures. maybe this should be an option in script, set it to custom if not set?

Commercial software seems to have found a way to correctly initialise metadata, so there might be a way.

I've never seen the purgeFolderCache error and I have no idea what it could be.

 

Analysis size refers to how big the image is scaled down to before computing the histogram. The smaller you make it the faster the script but the histogram will be less accurate. (... rescaling the image down makes the histogram calculation much faster)

Did you try cascading the resolution when multiple passes are going to be used? Make the the first pass faster by running it with a lower resolution and the latter passes with higher resolution for histogramm calculation. Maybe this saves some time in the end (and increases precision in the last pass if run with an even higher resolution)?
Also: Is it possible to let the run as many passes as needed instead of giving out the warning that more passes are needed?

I recently encountered a glitch with the script behavior, it either did not adjust or deflickering was poor. It took me a while to run down the bug, but basically in adobe bridge you have to set "Options for thumbnail quality and preview generation" to "always high quality".

IMAGE

Any other setting gave me either unpredictable results (like constant +1 stop in exposure for embedded thumbnails if I had shot -1EV) or left a lot of flickering. I have not done more research why, but just for someone who encountered the same problem. (Bridge CS6 5.0.2.4 + ACR 8.4).

Maybe it would be good to put things like that and the DNG and colour temp issue on the main page of the script under known bugs and limitations.

Also, why don't you explain the settings and options on the site? The documentation is bit lacking at the moment IMO.

Cheers
JB

[milank]

Would it be possible to create those sidecar files with the script instead of another Plugin for another software?

Exiftool is able to do that.

[Joachim Buambeki]

Exiftool is able to do that.

Now that you say it, I remember that commercial software also relies on Exiftool. Would it be possible to call an external app like Exiftool from within the script?
Or maybe there is some documentation that says what Exiftools does and then this can be recreated in the script?

[dmilligan]

There's no need to use exiftool. The script is already using Adobe's built in XMP library.

[Joachim Buambeki]

There's no need to use exiftool. The script is already using Adobe's built in XMP library.

If Exiftool works reliably and Adobe's API doesn't I would consider using EXIFtool. Not saying that EXIFtool does but I have never managed to get this script working reliably over a course of more than a year on totally different systems.

[PaulJBis]

Hi, Joachim.

First, let me say thank for those two links. I am not sure how exactly the info in the second link can be translated to the recent version of ACR. The problem is that when the sliders are at zero the image is far away from beeing linear. The problem was discussed by Christian Bloch here aswell. I assume the settings he recommends approximate a linear gamma pretty well tough. Writing this post I would think that one should be able to find out pretty close what settings create a linear gamma on PV2012 with Photoshop and the colour picker when comparing the same image processed with PV2010.

Sorry for not replying sooner; I've been out of the loop lately, and yesterday I spent half an hour fighting with the forum software trying to quote automatically your message  . Anyway, you are right: the problem is that the "linear" curve in PV2012 isn't really linear, due to the magic that Adobe added to extract more detail from the highlights (I found a thread in Adobe's forums discussing this months ago, but can't find it right now).

I don't know about trying to reverse-enginner whatever Adobe has done in PV2012, but it seems easier to me to just keep using PV2010 for the shots we take specifically for timelapses, which is why I added that option to dmilligan's script.

[PaulJBis]

As for the "undo" feature I added:

I noticed that after I installed the latest version of the script, but I am not sure how to use that really. Maybe Paul can chime in to explain it to me?

Basically, before you perform any operation using the script (deflicker, ramp values...), it will automatically save the current state of all the XMP attributes in all your selected pictures into a "set", which is represented by the menu entry in the Undo dropdown. When you later select an option in that dropdown, the script takes all those XMP attributes and loads them again into your pictures, erasing whatever modification you just made (that is, "undoing" it). Basically, it's like the "Backup XMP sidecars" feature, except you don't have to manually admin several sets of XMP files.

There can also be several options in the dropdown, representing several undo levels. Basically, I wrote this to behave pretty much like the undo feature in After Effects, Photoshop, etc. (Of course, there can be bugs  ).

[Audionut]

Much better results with this small timelapse, although I didn't get the chance to continue till dark since it started raining.

Does it render dark, or is it only my browser?  The source file:  https://dl.dropboxusercontent.com/u/34113196/test.mkv

[NickZee]

Hello Guys.

I have a question.

After I deflickered my sequence, I decided that I needed to lower the exposure at a few places.  I added 1 star to 2 frames (around 1/3rd and around 1/2 way) and set the exposure(lowered).  I also set (lowered) the exposure to the first and last in the sequence.   

I ramped the exposure, (not ramp multiple) and checked Additive.  My results are not what they should be.  The exposure is being added to making the exposure brighter.

In this screen shot.  I ran ramp about 6 times, with different scenarios.  Like, adding 1 star to 1st and last frame, using PV2010 and changing exposure to the four frames.  Each time, the exposure was increased.  The difference between frame 1 and frame 2 wasn't that extreme on the first run.

https://www.dropbox.com/s/ywxoxnh1xjch0i9/ramp.JPG?dl=0


Can ramp exposure additive be used with 1 star keyframes?

Thank you!

[dmilligan]

the "additive" exposure ramping does not work with keyframes, it simply adds the amount you put in the box to all the frames you have selected

[NickZee]

Thank you for clarifying @dmilligan

[itsskin]

I'm using the script on different mac now and got the same problem with process being stuck at some point of time. Last time you fixed it with delay for file render, so this is not the case now I think. Tried to find //$.writeln in new versions to give some logs, but seems you got rid of this feature How do I provide more info?
Thank you!

[dmilligan]

It's in a different branch in the repo "debug". You can just download that version and use it unmodified

[itsskin]

ok! I also wonder if it is possible to code the following behaviour:
1. do 1 pass of deflicker
2. wait for bridge to finish rendering new previews
3. start 2nd pass
4. go to step 2

Or I guess adding dialog with "delay between iterations" will also work awesome

I have a feeling its going to work faster and more stable
thx!

EDIT:
Got debug version, replaced old script. Running Bridge CC 8.1 on mac 10.9. Nothing is displayed while script is running. Just usual progress bar.