Feature Request: Batch Convert MLV files to DNG

Started by djfremen, December 26, 2013, 01:08:32 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

djfremen

I've been happily using MLV DUMP on mavericks to create image sequences for AE Import. Works great but not having the ability to batch process 50+ files is a real drag.

Wish I knew C#, I'd work something out. Anyone have a status update on when can expect this massive time saving feature?

Thanks Devs!

Danne

Updated batchscript command for mac-users(30dec 2013) Thanks to dubzeebass, g3gg0, daisermac, jerrykil

Ok.
I modified a command script from a member here,  jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.

Instruction would be:

1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command0.2 press enter
6. run the recursive.command by doubleclicking

You can doubleclick the recursive.command0.2 script a couple of times to have them work in parallell.

commandscript https://drive.google.com/file/d/0B4tCJMlOYfirWDd2RldZVXdjck0/edit?usp=sharing

Thanks
/D

*updated recursive.command 29 dec (g3gg0)
*updated recursive.command0.2 30 dec (daisermac moving spanned files to folder)

djfremen

MAGICAL!!! Totally works!!!

Few things I did different were just to hit Shift+Command+G - then go to "/usr/bin/" and drag in mlv_dump.

Everything else the same.

Danne

You can doubleclick the recursive.command script a couple of times to have them work in parallell.

dubzeebass

Glad to hear you have had success! The script was a joint effort from g3gg0, myself, and Danne. Collaboration rules!

Danne

Don, t forget jerrykil whose hdr commandscript I tweaked :). Other than that I say you dubzee and g3ggo are the brains on this one though it feels great to have contributed somehow :)


djfremen

One thing that would be beneficial is including the clip name in the DNG file name: "clipname-frame#.DNG"

This is done manually when you use terminal but not when you execute the script. Currently, it creates a folder but makes searching or indexing multiple DNG files impossible because of duplicate file names.

Also spanned files are not exported correctly. The ".00" files are skipped and not moved into the destination folder.

Thanks again for this über helpful script!

g3gg0

workingDir=`dirname "$0"`
cd "${workingDir}"

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./$FILE ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump -x $FILE
    /usr/bin/mlv_dump --dng -o ${BASE}_frame_ $FILE
    /usr/bin/mlv_dump -m -o meta.mlv $FILE
    /usr/bin/mlv_dump -v meta.mlv >$FILE.txt
    rm ./meta.mlv
    rm ./meta.mlv.wav
    cd ..
done
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

update, added xref creation due to wave misordering
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


dubzeebass

Quote from: g3gg0 on December 28, 2013, 10:59:45 AM
update, added xref creation due to wave misordering

Once the IDX file is created, it won't dump the MLV.  It will dump if I remove that IDX creation line.

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M28-1310.MLV'
   - Convert to DNG frames
   - Output into 'M28-1310_frame_'
File M28-1310.MLV opened
File M28-1310.IDX opened (XREF)
XREF table contains 419 entries
Processing...
0, 341516896
[E] Missing file header
Processed 0 video frames
Done

Danne

I get the same as dubzeebass on the script-update

g3gg0

did you download the latest mlv_dump from the mlv_rec thread?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

Thanks g3gg0! Your script worked with the new mlv_dump :)

Francis Frenkel

Hello,

Is MLV with sound working for 5D Mark2 ?

If yes, where could I download the module containing the mlv_Dump.exe / mlv_rec.mo  and mlv_snd.mo for 5D mark2 ?

Have a nice sunday !

Francis

Francis Frenkel
www.ffrenkel.com

g3gg0

Quote from: Francis Frenkel on December 29, 2013, 10:38:03 AM
Is MLV with sound working for 5D Mark2 ?

If yes, where could I download the module containing the mlv_Dump.exe / mlv_rec.mo  and mlv_snd.mo for 5D mark2 ?

Have a nice sunday !

try and report ;)
mlv_rec is there and the 5D2 binary you have to build yourself or use an old one.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Francis Frenkel

I have in the 24th October for 5D mark2 version and the the MLV_Rec.mo file is in it ! Ok
It works find when I shoot with mlv_rec !

But in the modules file, there is not mlv.snd.mo !

>>> Then when I use mlv_dump the result is a RAW file without sound (I mean without .wave file !)...

My question is : where could I find the module mlv_snd.mo for 5D mark2.
Is it working for 5D Mark2 ?

I got one mlv_snd.mo for 5D mark3 but is it the same ? Could I copy it in the Module folder of my 5D mark2 ?
Francis Frenkel
www.ffrenkel.com

djfremen

Clip named fixed in DNG title. Thanks!

Spanned files still aren't moved into their respective folders (*.00) and don't work.

djfremen

Here's the correct response in terminal:


Here's the scripts:


Danne

Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

dubzeebass

Quote from: Danne on December 29, 2013, 08:59:58 PM
Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

Thanks dude, works great !  I wonder if we should make our own thread...

Danne


g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


semenyuk.yura

Quote from: Danne on December 29, 2013, 08:59:58 PM
Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

Hi,

after step 2 terminal says:"Permission denied"



Any ideas what i'm doing wrong?

g3gg0

Only enter the chmod command. Nothing in front of that.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

semenyuk.yura

Quote from: g3gg0 on January 08, 2014, 01:06:47 PM
Only enter the chmod command. Nothing in front of that.

As you've understood i'm not good at programming,

so i tried 2 variants of how i understand your advice, but none of them ended successfully.



I'm running osx 10.9.1, if that matters.

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

semenyuk.yura


shinokubo

Hello friends,

is there a way for me to set a specific destination folder, by altering or adding a line to the script?

By now I am copying .mlv files from the CF Card to a certain folder and then run the script in there. As the folder is on an external hdd, the time for the copying process and mlv_dump to process the files could be greatly reduced by having different input and output drives.

Any help is deeply appreciated, the same way all your work already is.

Thank you very much in advance,

Ben

djfremen


Quote from: shinokubo on January 13, 2014, 09:06:24 PM
mlv_dump to process the files could be greatly reduced by having different input and output drives.

Explain how this would be faster please. I understand the concept of reading from one drive and writing to another but I don't think it would be a huge difference. I just processed 500 GB of MLV and it took under 2 hours on my i7 4770.

Danne

You can also double click the recursive script several times and have lke four processes working simultaneously.
Would fasten it even further.

shinokubo

I am using a macbook pro with an external USB 3.0 drive that is buspowered. It is a 5400 rpm drive that levels out at around 50Mb/s maximum read+write at the same time. It can write at around 95Mb/s though. As the input and output is on the same drive, I am stuck with that speed.

I would not have to copy the files at ~95Mb/s from CF -> HDD and then convert them at 50Mb/s, but could place mlv_dump and the script on the cf card, read from there and have the output written at (I'm guessing) around 90-95MB/s on the HDD

The limiting factor is drive speed, not the cpu so far.

It should take roughly 1/3 of the time if I am not mistaken here.

cheers, Ben

edit: typo and clarification

arturochu

Chu

alexpolsky

Hello!I do not dump mlv file in raw or in dng with mlv_dump & MlRawViewer (OSX)
In MlRawViewer playing stoping on 46 frames and app hangs(Other video good play and converting on dng.
In mlv_dump this log
Mac-Pro-MacMan:Movie macman$ ./mlv_dump -f30 M07-2323.mlv out.raw

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-2323.mlv'
   - Verify file structure
File M07-2323.mlv opened
File M07-2323.m00 opened
File M07-2323.m01 not existing.
File M07-2323.IDX opened (XREF)
XREF table contains 323 entries
Processing...
Reached end of all files after 323 blocks
Processed 299 video frames
Done
I don't have M07-2323.m01 on SD or CF card.

Is it possible dump to from a particular frame. Because if you miss the 46 frame video on playing in MlRawViewer.
Sorry my english. I am from Ukraine.  Please help!
Modify message

ShootMeAlready

mlv_dump command line examples for Windows??? Cant find any???

I am trying to process a 10bit raw video to dng

Here is what I enter (with mlv_dump in directory)
G:\DCIM\100CANON>mlv_dump -o test02 -b 10 --dng m20-1339.mlv


PROBLEM: The output states its defaulting to 14bpp ??? not 10??? when I try with default it still reports 14 bpp
The picture is too small as file properties reports 128x94 96dpi, this is wrong it should be 1824x776

It seems the command-line is wrong ???

Blow is Windows 10 output:

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'm20-1339.mlv'
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Output into 'test02'
File m20-1339.mlv opened
File m20-1339.m00 not existing.
File m20-1339.IDX opened (XREF)
XREF table contains 328 entries
Processing...
Cold pixels : 0
Reached end of all files after 328 blocks
Processed 304 video frames
Done

T3i+ML & 70D.112+ML, Tokina 11-16 2.8, Sigma 18-35 1.8, 50-150 II 2.8, 50 1.4, Canon 28 1.8, 35 2, 85 1.8 "Shoot Wide and Prosper"