Offline/Online Workflow for DNG in OS X

[Oedipax]

I'm trying to figure out a good workflow where I can edit proxies of my 5Dmk2 raw footage and then relink back to the original DNGs for grading in AE at the end. I figured I'd make this thread to write about what's working right now and hopefully people can chime in with ways on improving.

First off, the only way this works is with proper file management and good renaming. I've been renaming all my shots so that the folder looks like (for example):

/5d2raw-2013-05-27-shot001/
/5d2raw-2013-05-27-shot002/
/5d2raw-2013-05-27-shot003/

Within this, the files would be numbered like:

/5d2raw-2013-05-27-shot001-image00001.dng
/5d2raw-2013-05-27-shot001-image00002.dng
/5d2raw-2013-05-27-shot001-image00003.dng

And so on. No more of this /M0000001/000001.dng stuff. Right now I'm just doing it with Name Mangler, which automatically derives the folder name from its creation date, and then derives the image file names from the parent folder. It still requires me manually dragging and dropping each folder and clicking rename, because if I do a whole shoot (say 10 clips) as a batch, the image numbers increase even when a new shot starts. It would still work that way (the first file for shot002 might be 00185.dng) but it seems kinda funky to me.

Okay, everything's been renamed and organized. Now it's time to convert to something I can actually edit in realtime. Right now I'm opening up an entire folder of DNG's in Photoshop CS6, finding my hero shot, making a quick adjustment for exposure/highlights/shadows/etc, then syncing that adjustment to all the other DNGs in the sequence and saving that out as a series of jpg files with the exact same name, just a different extension.

It's a pretty slow process to grind through many thousands of DNGs to get them into JPG form, so I hope that's something that can be accomplished faster. Even if the output wasn't top notch quality (which these are right now), that'd be okay. It's just a proxy.

Okay, onto Premiere CS6. I import my JPG image sequence; seems you have to do this one at a time, and check that it is in fact an image sequence - any better way to do this? Once that's done, drop it on a timeline and start editing. Plays back at full quality without a hitch. Edit away.

When done, I'm exporting an XML file of my Premiere sequence and then using Pro Import AE in After Effects CS6 to bring it in. That gets my sequence in with all the cuts intact.

Now the cool part - if I then reconnect the image sequence in the AE bin to my DNG files, I've now got an edited sequence of raw DNG files. I can then grade away at these, including re-opening Camera RAW and making all the adjustments I want.

When it's time to render out, I can do it from the highest quality source possible (the original DNG) to whatever format I want.

I've seen Windows workflows for rendering out proxy files in formats like mp4, but for me it makes a lot more sense to stay with image sequences for the relinking ability at the end.

What do you guys think? Obviously the holy grail here would be if we could get these files natively into Resolve - then it's basically the R3D/BMCC workflow... import the raw files, do a one light, export proxies, edit, conform and color at the end with a lot less steps!

[squig]

Resolve won't be the holy grail unless someone writes some code that allows you to mess with the raw files like you can with individual frames in ACR.

I quickly tried a Premiere workflow today. ACR/Photoshop save DNG as DNG then follow GingerHDR workflow http://19lights.com/wp/tutorials/raw-and-cinemadng-wrapper-workflow/

[Oedipax]

Resolve won't be the holy grail unless someone writes some code that allows you to mess with the raw files like you can with individual frames in ACR.

What do you mean - just that you prefer ACR's interface? Or it actually does stuff Resolve can't?

I thought ACR was looking less desirable on account of the highlight/shadow stuff fluctuating during shots...

[squig]

Yeah ACR is practically useless with the exposure shifts. I'm a Resolve noob but looking at it there's limited raw data manipulation, red raw appears to have the most options, cinemaDNG has practically none, am I missing something here?

Raw video's been around for a while, where are the raw post tools? Call me stupid but I expected you'd be able to manipulate raw video like you could raw stills albeit a lot slower.

[Oedipax]

Yeah, agreed that Resolve's RAW implementation needs to come a lot further. My dream app would basically just be Lightroom wrapped around RAW image sequences. And maybe add in a tracker for secondaries. I hate how many different windows and panels I have to toggle through to work in Resolve. Part of why people use a control surface, I guess. As it is right now, I can get my shots looking good in Lightroom/ACR quickly where I continue to struggle to get what I'm after in Resolve. But then you get to the export stage and it's damn near realtime!

Anyway, remembered one other trick to getting image sequences into Premiere faster (i.e. without individually needing to bring them in, and interpret them to 23.976 since they default to 29.97). You do it in After Effects (you can use a script called Immigration to batch import image sequences) then literally just copy them from your AE bin and paste that into Premiere.

[swinxx]

i admit that resolve is really different than ae or premiere, but when you work with it for some hours, watch some tutorials, and read in the manual, this is the program to go!!

check this when you are interested in RAW handling in Resolve

http://www.premiumbeat.com/blog/blackmagic-cinema-dng-raw-files/

[squig]

As it is right now, I can get my shots looking good in Lightroom/ACR quickly where I continue to struggle to get what I'm after in Resolve.

Ok so I'm not stupid (arguably) just naive. The power of raw video isn't being utilised, I don't get it. The market's wide open for someone to capitalise on it, but I hope to see an open source option first.

[Oedipax]

@swinxx, thanks - I'll check it out. The weird thing is I've used Resolve a decent amount, and I do like it - I graded the last couple music videos I shot using it, but that was transcoded GH2 prores stuff. It responded much more the way I'd expect it to.

Something about RAW in Resolve is a whole different story for me. I have the hardest time getting an image I'm happy with. People say "That's because RAW is a completely different animal!" and it's true, but how do I square that with the fact I've been shooting RAW stills for years and never had a problem? Argh.

[marten]

Yeah ACR is practically useless with the exposure shifts. I'm a Resolve noob but looking at it there's limited raw data manipulation, red raw appears to have the most options, cinemaDNG has practically none, am I missing something here?

Raw video's been around for a while, where are the raw post tools? Call me stupid but I expected you'd be able to manipulate raw video like you could raw stills albeit a lot slower.

Have you tried Lightroom 3. I'm using it and I haven't got any flicker/exposure shift yet. I haven't done more than 50 clips though.

[Oedipax]

Interesting - Lightroom 3 in particular rather than 4 or 5 beta? Is that because the old image processing algorithm behaves differently than the new one? Certainly worth a shot.

[squig]

Have you tried Lightroom 3. I'm using it and I haven't got any flicker/exposure shift yet. I haven't done more than 50 clips though.

I don't have Lightroom but it uses ACR just like AE and Photoshop so I expect the same result. Can you do a test on a shot where the light changes, like a person walking in front of the camera with the sky in the background (just panning the camera in my living room with daylight coming through the window triggers it). Then in Lightroom turn the highlight slider down to -100 and render out.

[squig]

i admit that resolve is really different than ae or premiere, but when you work with it for some hours, watch some tutorials, and read in the manual, this is the program to go!!

check this when you are interested in RAW handling in Resolve

http://www.premiumbeat.com/blog/blackmagic-cinema-dng-raw-files/

I watched that tutorial and as I discovered there's no raw highlight and shadow control in Resolve which makes me really wonder what all the hype over the BMCC is about. From my experience shooting raw stills the real power of raw is the highlight and shadow recovery.

[Oedipax]

Is anyone here good at writing shell scripts?

I've been experimenting with a script that takes a folder of DNG files and converts them to JPG. It uses the white balance of the DNG file and doesn't do any other adjustments (focus here is on speed). It's running dcraw to generate a standard output and then piping that to cjpeg from the old Netbpm toolkit.

Right now using this method, I'm processing about 111 DNG files in a minute, versus the ~70 I was getting in Photoshop/ACR.

But the best part is, I can run multiple instances of the script (it's really a matter of how many processor cores you have), so on my 8-core MacPro I stopped at 6 to leave a little overhead for other stuff.

The rates stayed more or less consistent with that many processes running, I think the lowest dip I saw was 105 images/minute, so obviously this is way better than just running it one at a time. That's a rate of ~630 frames/minute, or about 9x what I was getting doing the same thing in ACR. If I raised that to 7 or 8 instances it could probably hit 50% of realtime. Granted these are just proxies, but that sounds good to me!

Now the big question - can someone with script experience tell me if this is easily implementable:

1) Launching multiple instances within the script, either on the same folder/shot, or multiple folders/shots

2) Running the script recursively, so that I can give it a base folder and have it process everything (including additional folders) within that

I'm not a programmer, so I'm mostly fumbling around at this. The script is extremely basic right now:

Code: [Select]

#!/bin/bash

mkdir ./output/

for i in *.dng; do dcraw -c -w ${i} | cjpeg > ./output/${i}.jpg

done
Right now that makes an /output/ folder with the JPG files, although they have the extension .dng.jpg which I've been batch renaming in a separate step. I'm sure that can be scripted too.

Anyone want to help out?

[notdabod]

Is anyone here good at writing shell scripts?

I've been experimenting with a script that takes a folder of DNG files and converts them to JPG. It uses the white balance of the DNG file and doesn't do any other adjustments (focus here is on speed). It's running dcraw to generate a standard output and then piping that to cjpeg from the old Netbpm toolkit.

Right now using this method, I'm processing about 111 DNG files in a minute, versus the ~70 I was getting in Photoshop/ACR.

But the best part is, I can run multiple instances of the script (it's really a matter of how many processor cores you have), so on my 8-core MacPro I stopped at 6 to leave a little overhead for other stuff.

The rates stayed more or less consistent with that many processes running, I think the lowest dip I saw was 105 images/minute, so obviously this is way better than just running it one at a time. That's a rate of ~630 frames/minute, or about 9x what I was getting doing the same thing in ACR. If I raised that to 7 or 8 instances it could probably hit 50% of realtime. Granted these are just proxies, but that sounds good to me!

Now the big question - can someone with script experience tell me if this is easily implementable:

1) Launching multiple instances within the script, either on the same folder/shot, or multiple folders/shots

2) Running the script recursively, so that I can give it a base folder and have it process everything (including additional folders) within that

I'm not a programmer, so I'm mostly fumbling around at this. The script is extremely basic right now:

Code: [Select]

#!/bin/bash

mkdir ./output/

for i in *.dng; do dcraw -c -w ${i} | cjpeg > ./output/${i}.jpg

done
Right now that makes an /output/ folder with the JPG files, although they have the extension .dng.jpg which I've been batch renaming in a separate step. I'm sure that can be scripted too.

Anyone want to help out?

Wow nice work!

can dcraw export to exr files instead of dng?

I read something somewhere that you could.

Thanks

[CaptainHook]

I watched that tutorial and as I discovered there's no raw highlight and shadow control in Resolve which makes me really wonder what all the hype over the BMCC is about. From my experience shooting raw stills the real power of raw is the highlight and shadow recovery.

In Resolve you do it all yourself using keys (chroma/luma/saturation) etc. You can do exactly the same stuff (like highlight/shadow recovery if you want), just with a LOT more control. The log controls are actually great for highlight/shadow recovery and i do that all the time. The downside is there's no easy "slider" for a lot of stuff so you have to put it all together yourself for the most part, but the upside is grade management is amazing and it's also easy to store stuff like that as power grades and just call it up in any shot in any project as you need. And now that BMD own Resolve each new release is taking major leaps towards being "user friendly" in the way typical prosumers would expect an app to behave, rather than those coming from 3D/Compositing and other node based apps/workflows (mostly in linux or ported from it) where resolve is actually SIMPLE compared those apps.

It's really different to using normal still RAW apps (Capture One is my personal choice for colour etc), but it's AMAZING for grading motion footage once you learn it and find out how to get the most from it. Don't get me wrong, it could definitely learn from the RAW still apps too, and BMD are working at getting it there.

[Oedipax]

Wow nice work!

can dcraw export to exr files instead of dng?

I read something somewhere that you could.

Thanks

I don't see any options directly in dcraw (might be there, but I don't see the option just looking at the list of parameters), but if you have some other command line tool to encode to EXR then you could pipe the standard output from dcraw into that, just like I'm doing with cjpeg.

Also, an update on multiple instances - it looks like GNU Parallel is a good way to do this. But I'm having difficulty getting the syntax right for the command line. My hangup right now seems to be I'm calling Parallel for the first step (i.e. dcraw to stdout) but then when I pipe that into cjpeg it doesn't understand the input it's being given and can't retain the filenames from the first part (it just keeps overwriting the same filename which is supposed to be a Parallel variable, {}).

If anyone wants to chime in, that would be great. Otherwise I don't know if I'll have much time to tinker with this until next weekend.