Batching rushes: temp files and storage constraints

[saf34]

Quick question for those shooting *lots* of footage:

What is the space-efficient way to quickly render out low-resolution rushes for the director?

For example, yesterday I shot 5TB of MLV (14-bit lossless).  If I mlv-dump this lot to dng (my usual workflow to Cinelog) it's around 10TB. 

So my questions:
  - If I mount everything using MLVFS to render out low res, does this need 10TB of temp files?
  - If I import the whole lot to MLVApp, does this require 10TB of temp files?

Or how do others do this for long shoots?

Best,
Stewart

[Kharak]

I am a little confused with how you end up with twice the data as Lossless MLV ? Or do you mean, all data combined, like MLV + DNG?

If so, why not convert to Lossless DNG and do a 2nd lossless compression with SlimRaw to take off another ~12%. (Perhaps more). SlimRaw does a better compression than MLV_Dump.

SlimRaw can also do Lossy compression from 3:1 to 7:1 ratio and if for only rushes, set Resolution to Half for another 50% save. But you will have to keep on to your Source MLV for a Full Quality Conversion again in the future. This is if you choose to Override the DNG's with SlimRaw's Compression.

If you separate your Batch processing in to 200-400 GB chunks (or what have you) in separate folders, which I assume you already are, you can Compress as you go.

You can also Compress the folder you are converting to as it goes (a la Watch Folder), but SlimRaw is way faster than MLV_Dump, so make sure it does not overtake MLV_dump, because you will end up with parts Compressed of the folder and parts not and it can be a pain looking for where SlimRaw stopped and will require recompressing parts of the footage, especially if you did really long takes at a time. But you will see the Checksum file in the DNG folders where it has compressed the footage, so look for that.

So to be clear, SlimRaw has no Official Watch Folder Support, you can compress from the folder you are converting to, but if SlimRaw reaches the last DNG available in that folder before MLV_dump is done converting, SlimRaw will stop as it thinks it finished. So take care of that, therefore better to just do Chunks at a time in separate folders.

SlimRaw Generates Checksums and can run a Verification pass aswell.

You can even do Dual output for backup.

I probably sound like a SlimRaw commercial, but I am not not, I just really like that software.

But let me know what your process is, I am not sure what you mean about Cinelog-c when talking about DNG. Are you running an AE Workflow?

[saf34]

Thanks for answering - much appreciated.  My workflow is a quick setup from a few years back when I bought Cinelog-C:

1) Copy MLVs to folder on internal fast SSD (D:\MLV\*.mlv)
2) Run a batch file that runs mlv_dump on each MLV file in that folder and deletes the MLVs in turn.  So now I have D:\DNG\[filename]\*.dng
3) Import each DNG folder into After Effects using my Cinelog profile, and render out Prores 4444 masters D:\cinelog\*.mov

Originally this was all uncompressed 14-bit MLVs so this was my own way to 'compress' my masters.  Recently I've moved to lossless MLVs (why not) and because my MLVs are now 14-bit lossless and my DNGs are uncompressed, the intermediate DNG folder is twice as big as my MLV folder.  The final Cinelog Prores files are a little smaller than the original MLVs, but mainly they're just a convenient habit for grading.

Then I always just did a render of rushes from my final log masters.  But now - with such big sessions and multicam taking ages to transcode - I'd like to er... rush ... them through first *ahem*.

I'm just setting up MLVFS for the first time, but haven't read how much storage is used when mounting a folder of MLVs.  I.e. is the entire folder in temp files as DNGs or perhaps individual DNGs are pulled one by one from the MLV when required?  If the latter, this is much much better than my hotch-potch method.  Plus is seems it gives you a timestamp on the DNGs and Wavs, which is useful.

MLVApp is seductive too, but takes a long time to load in large amounts of footage.  Waiting for a frozen screen to unfreeze is stressful! 

Anyway, that's the context.  I'm a bit late to the workflow party, but ideally I'd manage to get a bunch of log masters, a bunch of Rec709 H264 rushes, and have them all with a timestamp rather than have to handbuild my timelines.

I'm veering towards MLVFS right now, but am still getting set up.  So I'm definitely interested to hear other possibilities from more experienced MLers!

[Kharak]

I recommend Danne´s MLV_Dump Batch converter.  https://www.magiclantern.fm/forum/index.php?topic=10526.125

You can convert the Lossless MLV's to Lossless DNG, keeping the original size of the Lossless MLV's (NO data lost) and further losslessly compress it with Slimraw, like explained above.

I would be careful with running a Batch tool that deletes the Original MLV's as you go.

And secondly, I can not recommend  Resolve enough, also for further Batch processing. It is literally 100 times faster than AE. If I render DNG to Cineform without any effects, it runs at 100-150 FPS or faster, the bottleneck being the write speed of the drives. So for transcoding it is the obvious solution.

Or don't transcode and work straight on the DNG's.

[Walter Schulz]

Reminds me of a feature request years ago: https://www.magiclantern.fm/forum/index.php?topic=18999
Would come handy to test memory/processor requirements and limits.

[saf34]

Thanks Kharak, sounds like it's about time I get a new workflow down.  Thanks for the tips, which I'll start playing around with.  Coming from a photoshop background, Cinelog in ACR was perfect at the time, but has perhaps made me a little lazy re Resolve etc!