[WINDOWS] MLV to DNG Batch Converter v1.9

Started by tonybeccar, January 27, 2014, 05:08:15 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

SteveScout

Tony, you said above that you´re using the Canopus Codec instead of DNxHd in After Effects. Question - can you export in 16bit in it? Because by standard you cannot export more than 8bit ("millions of colors") into that codec, the wrapper (QT or AVI) does not matter. There was an article once online where they described what text setting for AE´s file formats to change to allow "16bit / trillions of colors" exporting, but I never found out how.

So in case you´re "just" exporting millions of colors, be aware that you´re not getting the full information (no matter if you´re using a log curve or not) from the DNG file as it converts down to 8bit.

cheers,
Steffen

tonybeccar

Quote from: SteveScout on February 05, 2014, 11:33:52 PM
Tony, you said above that you´re using the Canopus Codec instead of DNxHd in After Effects. Question - can you export in 16bit in it? Because by standard you cannot export more than 8bit ("millions of colors") into that codec, the wrapper (QT or AVI) does not matter. There was an article once online where they described what text setting for AE´s file formats to change to allow "16bit / trillions of colors" exporting, but I never found out how.

So in case you´re "just" exporting millions of colors, be aware that you´re not getting the full information (no matter if you´re using a log curve or not) from the DNG file as it converts down to 8bit.

cheers,
Steffen

Hiya! Grass valley has 2 codecs, Canopus HQ or Canopus HQX, the first one is 8bit and the other one is 10bit, I always use the second one. And if you use AVI instead of MOV, you have Canopus Lossless, but the sizes are HUGE. Always use the HQX with AVI form.

tonybeccar

Quote from: Frank7D on February 05, 2014, 11:18:00 PM
tony, I did a little reading on dcraw. I don't know if you've seen this:
http://www.cybercom.net/~dcoffin/dcraw/dcraw.1.html

Based on the above, my suggestions to maybe speed up proxy creation process:
~ Make 8 bit tiffs instead of 16 (drop the -6)?
~ Eliminate metadata in tiffs (drop the -T)?

Mmmmmh.. I don't know.. I dont want to drop the quality until I pass down ffmpeg.. since it's acting as a debayer shouldnt we use the full bit depth? Well really I am not sure.. and the -T is for creating tiffs instead of ppm.. don't know about the metadata... anyway gonna research on Ufraw, to see if its better and faster... when is ffmpeg gonna support DNGs?!?!?

tonybeccar

Quote from: jayzed on February 05, 2014, 05:57:13 PM
This is probably not going to be possible, given that the codec needs to be purchased but as you're asking for AVI codecs - Cineform. I love the active metadata feature - Cineform RAW would be heaven.

Rawanizer works with Cineform but I've not been able to get the Cineform transcode working on my machine (I'm long overdue a rebuild) and, at last check, Rawanizer didn't handle MLV or audio.

Saying that, I've not used CF as much since the birth of RAW as I've been using ACR for much the same purpose - an active basic grade. However, I only work on short sequences so I'm fine loading RAW into After Effects and either rendering from there or transcoding to Cineform for use elsewhere.

Anyway, please don't consider this a moan or a begging letter. I really love this tool and have moved over completely to MLV (with audio) since starting to use it.

Downloading version 1.6 now.

I once used cineform.. and didnt like it one bit.. i dont know if I made some sort of mistake, but it downgraded the quality by a LOT... so never used it again.. also it messed up my system's codecs.. :S. But it's not hard at all to implement.. could give it a try even.. we'll see.. there are other priorities on the program right now, such as individual progress...

tonybeccar

Quote from: Frank7D on February 05, 2014, 05:06:22 PM
Yes, the audio is pretty much in sync, but as I said if it's off by 1 frame it would be hard for me to tell.

The proxy is definitely off by one frame (visually) from the dng sequence however, because of the extra frame at the beginning, which makes editing with the proxies problematic if you want frame-accurate edits.

Ok great, will do some tests with speech myself also.. still a bit iffy on thw workflows myself.. th TIFF thing takes too long for making a proxy! :S

Frank7D

Quote from: tonybeccar on February 05, 2014, 11:54:52 PM
Mmmmmh.. I don't know.. I dont want to drop the quality until I pass down ffmpeg.. since it's acting as a debayer shouldnt we use the full bit depth? Well really I am not sure.. and the -T is for creating tiffs instead of ppm.. don't know about the metadata... anyway gonna research on Ufraw, to see if its better and faster... when is ffmpeg gonna support DNGs?!?!?

You're right, I didn't notice that -T indicates tiff output!

As far as bit depth, we're talking about proxies here, which won't be used for final output, so do we really need 16 bits? Don't know whether using 8 would speed the tiff creation or not though.

arrinkiiii


If you use a "proxy" of prores 444, is very good for final result/output.

Frank7D

Quote from: arrinkiiii on February 06, 2014, 12:13:49 AM
If you use a "proxy" of prores 444, is very good for final result/output.

That's true, but I'm planning to use the dngs for final render. I will use proxies just for editing (and after all, that's what the word "proxy" implies).

I guess you're right though; some people will want high quality "proxies" to use for final render, so I guess the intermediate tiffs need to be high quality.

Frank7D

I did some more testing and found:

1. My proxies are adding duplicate frames but also are dropping frames and are ending up shorter than the dng sequence.

2. The audio in the proxy mov files is out of sync. The audio is the correct length but the video part is too short. The difference is made up by black frames at the end.

3. The dng sequence and the wav file are exactly in sync.

4. The shortening of the video in the proxy mov files is just the same as if they were assumed to be 25 fps (in actuality my videos were shot at 23.976 fps).

Reading here:

http://ffmpeg.org/ffmpeg.html#Stream-specifiers-1

I see some information about frame rates which may be helpful.

Edit: Here is something I found somewhere else which might be applicable (a solution to someone else's problem):

"An image sequence is assumed to be 25 frames per second.  Your command line requests 23.976 frames per second output.  So FFmpeg drops frames from your
video to achieve the requested frame rate.

I think you need to put the "-r 23.976" in front of the "-i".  Options that describe the input file must be given before the "-i" on the command line. "

Edit 2: Also, here are the command lines that Rawanizer uses to make proxies:

-threads 8 -r {InputVideoFps} -i "{InputVideoName}" -i "{InputAudioName}" -strict experimental -vcodec prores -pix_fmt yuv444p10 -s 1920x1080 {Parameters} "{OutputVideoName}.mov"

CoresNZ

Nice job on the converter!

It's really nice to be able to spit .dngs and .wav files out through a nice GUI like this.
And the real time preview window for playback of .mlv files is great!

The only problems I'm having with it after using it for a couple of days are the output naming options. They are a bit limited and force me to end up with files that have way more information than I want to look at. If you could add check boxes to turn on and off all the prefix and numbering tags individually then people could customize the output however they desired.

I personally prefer to have a nice short - camera number_shot number_frame number

Currently the best I can get is this "MLV_A001_001_C00001_f000000.dng"

What I'd like is something more like this "A_001_0000.dng"

The only other thing is the progress bar, Which seem to only update at the end of converting each .mlv file? and calculates the percentage based on the number of clips done rather than the number of total frames?

I only tend to dump .dngs rather than create proxys so a percentage of total frames dumped would be more useful to see for an honest progress bar.

Oh one other thing, is it possible to get the audio to play in the preview player? doesn't seem to work for me at the moment.

Look forward to the future releases, Keep up the great work mate!

tonybeccar

Quote from: Frank7D on February 06, 2014, 05:57:10 AM
I did some more testing and found:

1. My proxies are adding duplicate frames but also are dropping frames and are ending up shorter than the dng sequence.

2. The audio in the proxy mov files is out of sync. The audio is the correct length but the video part is too short. The difference is made up by black frames at the end.

3. The dng sequence and the wav file are exactly in sync.

4. The shortening of the video in the proxy mov files is just the same as if they were assumed to be 25 fps (in actuality my videos were shot at 23.976 fps).

Reading here:

http://ffmpeg.org/ffmpeg.html#Stream-specifiers-1

I see some information about frame rates which may be helpful.

Edit: Here is something I found somewhere else which might be applicable (a solution to someone else's problem):

"An image sequence is assumed to be 25 frames per second.  Your command line requests 23.976 frames per second output.  So FFmpeg drops frames from your
video to achieve the requested frame rate.

I think you need to put the "-r 23.976" in front of the "-i".  Options that describe the input file must be given before the "-i" on the command line. "

Edit 2: Also, here are the command lines that Rawanizer uses to make proxies:

-threads 8 -r {InputVideoFps} -i "{InputVideoName}" -i "{InputAudioName}" -strict experimental -vcodec prores -pix_fmt yuv444p10 -s 1920x1080 {Parameters} "{OutputVideoName}.mov"

Hey Frank.. I'm sorry I did not do a lot of testing for the proxies, I just assumed if you did a simple command to the dngs it will be ok. Just so you know.. of course it detects your framerate, i have the option -r "videoFPS", it is not at the front like rawanizer.. would that change anything???? Does your proxy say 25fps? But anyway.. there must be other parameters to look at.. and the rawanizer line was for .RAW files, no audio involved here.. and so new troubles arise! Will look into it.. Thanks!

tonybeccar

Quote from: CoresNZ on February 06, 2014, 11:06:13 AM
Nice job on the converter!

It's really nice to be able to spit .dngs and .wav files out through a nice GUI like this.
And the real time preview window for playback of .mlv files is great!

The only problems I'm having with it after using it for a couple of days are the output naming options. They are a bit limited and force me to end up with files that have way more information than I want to look at. If you could add check boxes to turn on and off all the prefix and numbering tags individually then people could customize the output however they desired.

I personally prefer to have a nice short - camera number_shot number_frame number

Currently the best I can get is this "MLV_A001_001_C00001_f000000.dng"

What I'd like is something more like this "A_001_0000.dng"

The only other thing is the progress bar, Which seem to only update at the end of converting each .mlv file? and calculates the percentage based on the number of clips done rather than the number of total frames?

I only tend to dump .dngs rather than create proxys so a percentage of total frames dumped would be more useful to see for an honest progress bar.

Oh one other thing, is it possible to get the audio to play in the preview player? doesn't seem to work for me at the moment.

Look forward to the future releases, Keep up the great work mate!

Hey man! About the naming.. I'm trying to force the program to have a standard naming.. which I think is very good.. it takes the model naming of r3d files and blackmagic.. it shows you vital information, and it is minimalistic! On the other hand, I dont suggest you to rename your original MLV files.. :S, in fact you are adding a sequential number, and that's what the clip does.. "C"(clip)00001. Just so you know the naming:

MLV_A001_241238_C00001 = (magiclanternvideo)_(cameraid)_(card1)_(originalfilename)_(uniqueclipnumber)

I guess obviously I can make an option to turn it off.. for sure i'll never use it.. but I suggest you keep the originalfilenames.. you can end up with duplicates or later having trouble tracking sth down!

Oh and regarding progress bar, believe me it's on my top priorities! But it is hard to implement.. making progress though.. will be soon!

Cheers!

Frank7D

Quote from: tonybeccar on February 06, 2014, 03:17:14 PM
Hey Frank.. I'm sorry I did not do a lot of testing for the proxies, I just assumed if you did a simple command to the dngs it will be ok. Just so you know.. of course it detects your framerate, i have the option -r "videoFPS", it is not at the front like rawanizer.. would that change anything???? Does your proxy say 25fps? But anyway.. there must be other parameters to look at.. and the rawanizer line was for .RAW files, no audio involved here.. and so new troubles arise! Will look into it.. Thanks!

Hi Tony, according to the one solution I found (quoted in my last post), you need to put the input frame rate at the front. Otherwise, ffmpeg assumes a 25fps input rate (because it's an image sequence) and drops (or adds) frames to match the 23.976fps (or whatever you request) output rate. Not sure if you need to put the output rate also or not (I'm guessing not).

SteveScout

Hi, Tony!

I know about the two codecs .. but do ANY of those two codecs show the possibility in AE to show "trillions of colors" for you? Because standard is "millions of colors" - and this is only 8bit. Even if you have the 10bit version of that codec, as long as "millions of colors" is the only possible selection, it will store 8bit content in this 10bit codec which reduces it to 8but. DNxHD can be switched to 16bit through its quicktime wrapper.

tonybeccar

Quote from: Frank7D on February 06, 2014, 04:19:36 PM
Hi Tony, according to the one solution I found (quoted in my last post), you need to put the input frame rate at the front. Otherwise, ffmpeg assumes a 25fps input rate (because it's an image sequence) and drops (or adds) frames to match the 23.976fps (or whatever you request) output rate. Not sure if you need to put the output rate also or not (I'm guessing not).

Alright then, will try! Thanks man! ;)

CITY-U1001

50D | EFS 18-55 | last build crop_rec-3744x1080_24fps_50D-eXperimental.4.57pm.2020May06.50D109.zip

joaomoutinho

Hello guys,

Just installed the latest TragicLantern and I was messing around with the MLV.  I made 3 clips, using my 50D, its +/- 10gb each. 

Got to the MLV CONVERTER 1.6 and it took about 2hours to process the data. Is this normal to take this much time? Can I do anything to accelerate this process? Ah, and another issue, it didnt made the PROXYs.

Anyone experienced this issues? Can I do something to make this work better?

Thank you guys

Arthur Rambo

Hi,
Just batch converted 75 Go of mlv files in around 3 hours on a HP pavilion laptop.
It works like a charm !
Sam

SteveScout

Quote from: joaomoutinho on February 06, 2014, 10:31:04 PM
Anyone experienced this issues? Can I do something to make this work better?

Hi, Joaomoutinho,

the app´s speed is just fine. There is no real magic done to it, there is no real processor power etc. (debayering happens at a later stage), the only action which is done is "extracting" the DNG files out of the MLV (or the raw on other apps) file. No conversion or anything.  So the only thing that really counts is - drive speed!

If you had the MLVs on an internal RAID or SSD drive and would write to that drive as well - take a look. It´s fast as lightning. But normally you can´t store so much stuff internally, so it all comes from the CF cards itself or external harddrives. If you read the MLVs from an attached USB 3.0 drive (or E-SATA/Firewire800) and DON`T store the DNGs on the same drive it should be really fast. If the stuff´s read via USB 2.0 - then yes .. it takes a lot of time.

Steffen

tonybeccar

Quote from: joaomoutinho on February 06, 2014, 10:31:04 PM
Ah, and another issue, it didnt made the PROXYs.

Are you SURE? Checked the folder you selected? Has at least one setting? Can you tell us a bit more? :S OS version? It should work well really...

Canon eos m

Hi Tony, Thank you again for the wonderful software. I too have speed issues, since it takes more than 30 minutes to process a 300 - 500 frame mlv video.

Questions:

1. Any advice on the speed up - I use Windows 8, NIVDIA graphics TI660i, 32GB RAM, internal and additional internal SSD and 7200RPM external HDD's
2. The proxies are too dark after extracting - I recall you explaining why. Is there anything that can be done is post. I have tried but trying to recover the blacks / shadows doesn't appear to work best. Could there be something wrong with my workflow - Proxies (or DNG extractions), move to AE CC and color correct (neat video plugin if the video is grainy), move to PPRO CC and edit footage, move to AE CC to render. 
3. Lastly, is it possible to somehow mention the choice of prores (proxy to 4444) used in the file name or metadata encoding. I try to keep track but miss out sometimes.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

gary2013

Quote from: tonybeccar on February 07, 2014, 01:00:05 AM
Are you SURE? Checked the folder you selected? Has at least one setting? Can you tell us a bit more? :S OS version? It should work well really...
I am also not getting any proxies in the folder I made and specified. I have tried diff types, 444, HQ...  I use Windows 7 with all the updates and latest drivers.

tonybeccar

V 1.7 !!!

- INDIVIDUAL PROGRESS!!!!!!!!!!! damn it was hard... all except the ffmpeg part of the proxies.. still have to figure that one :/
- FASTER thumbnails!
- Fixed info panel, and status moved to status bar
- Temporary fix to the "No image" "bug". It can happen only if you scrub reeeeally fast.
- Put the "-r FRAMERATE" on the beginning now... check if it drops frames now...

https://www.dropbox.com/s/se3a5ipf6d337rh/MLV%20Converter%201.7.zip

Will continue on the next few days...

tonybeccar

Quote from: gary2013 on February 07, 2014, 02:22:56 AM
I am also not getting any proxies in the folder I made and specified. I have tried diff types, 444, HQ...  I use Windows 7 with all the updates and latest drivers.

My god that sucks... I wonder whats going on with ffmpeg.. did you move any files inside the program folder???? There's no apparent reason why it shouldnt be working :S, it just skips the process?? No error? No nothing?

tonybeccar

Quote from: Canon eos m on February 07, 2014, 02:00:12 AM
Hi Tony, Thank you again for the wonderful software. I too have speed issues, since it takes more than 30 minutes to process a 300 - 500 frame mlv video.

Questions:

1. Any advice on the speed up - I use Windows 8, NIVDIA graphics TI660i, 32GB RAM, internal and additional internal SSD and 7200RPM external HDD's
2. The proxies are too dark after extracting - I recall you explaining why. Is there anything that can be done is post. I have tried but trying to recover the blacks / shadows doesn't appear to work best. Could there be something wrong with my workflow - Proxies (or DNG extractions), move to AE CC and color correct (neat video plugin if the video is grainy), move to PPRO CC and edit footage, move to AE CC to render. 
3. Lastly, is it possible to somehow mention the choice of prores (proxy to 4444) used in the file name or metadata encoding. I try to keep track but miss out sometimes.

Congrats on the PC, wishing to have something similar to that hehe... mm, i dont know really, gonna investigate that later.. regarding the extraction of DNG files i guess you should speek to g33g0 to speed the program up.. or make it use all cores/threads and all the power of the cpu.. it only uses 30% on mine...

And proxies.. I am gonna investigate another debayer.. or different settings on this one.. it is dark now cause with the other settings it flickered... gonna try and apply some gamma curve or sth.. or use ufraw...