Magic Lantern Forum

Using Magic Lantern => Raw Video => Raw Video Postprocessing => Topic started by: djfremen on December 26, 2013, 01:08:32 AM

Title: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on December 26, 2013, 01:08:32 AM
I've been happily using MLV DUMP on mavericks to create image sequences for AE Import. Works great but not having the ability to batch process 50+ files is a real drag.

Wish I knew C#, I'd work something out. Anyone have a status update on when can expect this massive time saving feature?

Thanks Devs!
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 26, 2013, 01:28:20 AM
Updated batchscript command for mac-users(30dec 2013) Thanks to dubzeebass, g3gg0, daisermac, jerrykil

Ok.
I modified a command script from a member here,  jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.

Instruction would be:

1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command0.2 press enter
6. run the recursive.command by doubleclicking

You can doubleclick the recursive.command0.2 script a couple of times to have them work in parallell.

commandscript https://drive.google.com/file/d/0B4tCJMlOYfirWDd2RldZVXdjck0/edit?usp=sharing

Thanks
/D

*updated recursive.command 29 dec (g3gg0)
*updated recursive.command0.2 30 dec (daisermac moving spanned files to folder)
Title: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on December 26, 2013, 06:03:12 AM
MAGICAL!!! Totally works!!!

Few things I did different were just to hit Shift+Command+G - then go to "/usr/bin/" and drag in mlv_dump.

Everything else the same.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 26, 2013, 11:56:18 AM
You can doubleclick the recursive.command script a couple of times to have them work in parallell.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 26, 2013, 04:17:09 PM
Glad to hear you have had success! The script was a joint effort from g3gg0, myself, and Danne. Collaboration rules!
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 26, 2013, 05:41:45 PM
Don, t forget jerrykil whose hdr commandscript I tweaked :). Other than that I say you dubzee and g3ggo are the brains on this one though it feels great to have contributed somehow :)
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 26, 2013, 11:28:30 PM
Community ^_^
Title: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on December 27, 2013, 03:23:24 AM
One thing that would be beneficial is including the clip name in the DNG file name: "clipname-frame#.DNG"

This is done manually when you use terminal but not when you execute the script. Currently, it creates a folder but makes searching or indexing multiple DNG files impossible because of duplicate file names.

Also spanned files are not exported correctly. The ".00" files are skipped and not moved into the destination folder.

Thanks again for this über helpful script!
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on December 28, 2013, 12:42:02 AM
workingDir=`dirname "$0"`
cd "${workingDir}"

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./$FILE ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump -x $FILE
    /usr/bin/mlv_dump --dng -o ${BASE}_frame_ $FILE
    /usr/bin/mlv_dump -m -o meta.mlv $FILE
    /usr/bin/mlv_dump -v meta.mlv >$FILE.txt
    rm ./meta.mlv
    rm ./meta.mlv.wav
    cd ..
done
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on December 28, 2013, 10:59:45 AM
update, added xref creation due to wave misordering
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 28, 2013, 02:36:44 PM
Nice one thx!
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 28, 2013, 08:11:38 PM
Quote from: g3gg0 on December 28, 2013, 10:59:45 AM
update, added xref creation due to wave misordering

Once the IDX file is created, it won't dump the MLV.  It will dump if I remove that IDX creation line.

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M28-1310.MLV'
   - Convert to DNG frames
   - Output into 'M28-1310_frame_'
File M28-1310.MLV opened
File M28-1310.IDX opened (XREF)
XREF table contains 419 entries
Processing...
0, 341516896
[E] Missing file header
Processed 0 video frames
Done
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 28, 2013, 11:31:51 PM
I get the same as dubzeebass on the script-update
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on December 29, 2013, 02:11:47 AM
did you download the latest mlv_dump from the mlv_rec thread?
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 29, 2013, 02:26:32 AM
Thanks g3gg0! Your script worked with the new mlv_dump :)
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Francis Frenkel on December 29, 2013, 10:38:03 AM
Hello,

Is MLV with sound working for 5D Mark2 ?

If yes, where could I download the module containing the mlv_Dump.exe / mlv_rec.mo  and mlv_snd.mo for 5D mark2 ?

Have a nice sunday !

Francis

Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on December 29, 2013, 11:01:14 AM
Quote from: Francis Frenkel on December 29, 2013, 10:38:03 AM
Is MLV with sound working for 5D Mark2 ?

If yes, where could I download the module containing the mlv_Dump.exe / mlv_rec.mo  and mlv_snd.mo for 5D mark2 ?

Have a nice sunday !

try and report ;)
mlv_rec is there (http://www.magiclantern.fm/forum/index.php?topic=7122.0) and the 5D2 binary you have to build yourself or use an old one.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Francis Frenkel on December 29, 2013, 11:20:52 AM
I have in the 24th October for 5D mark2 version and the the MLV_Rec.mo file is in it ! Ok
It works find when I shoot with mlv_rec !

But in the modules file, there is not mlv.snd.mo !

>>> Then when I use mlv_dump the result is a RAW file without sound (I mean without .wave file !)...

My question is : where could I find the module mlv_snd.mo for 5D mark2.
Is it working for 5D Mark2 ?

I got one mlv_snd.mo for 5D mark3 but is it the same ? Could I copy it in the Module folder of my 5D mark2 ?
Title: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on December 29, 2013, 12:35:37 PM
Clip named fixed in DNG title. Thanks!

Spanned files still aren't moved into their respective folders (*.00) and don't work.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on December 29, 2013, 12:55:08 PM
Here's the correct response in terminal: (http://img.tapatalk.com/d/13/12/29/avusarej.jpg)


Here's the scripts: (http://img.tapatalk.com/d/13/12/29/9u7u2u6e.jpg)
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 29, 2013, 08:55:26 PM
Quote from: Danne on December 29, 2013, 02:26:32 AM
Thanks g3gg0! Your script worked with the new mlv_dump :)

+1
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 29, 2013, 08:59:58 PM
Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 29, 2013, 09:50:11 PM
Quote from: Danne on December 29, 2013, 08:59:58 PM
Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

Thanks dude, works great !  I wonder if we should make our own thread...
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on December 29, 2013, 10:01:40 PM
Probably :)
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on December 29, 2013, 10:03:38 PM
yep thats highly welcome.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: dubzeebass on December 29, 2013, 10:45:22 PM
Will do later
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: semenyuk.yura on January 08, 2014, 02:38:14 AM
Quote from: Danne on December 29, 2013, 08:59:58 PM
Dubzeebas. Tried simplyfying your script to following. Think It,s ok? It,s pretty much doubleclicking the commandscript til it works :)

Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

Hi,

after step 2 terminal says:"Permission denied"

(http://fs54.www.ex.ua/get/679999899378/89594011/Screen%20Shot%202014-01-08%20at%203.01.31%20AM.png)

Any ideas what i'm doing wrong?
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on January 08, 2014, 01:06:47 PM
Only enter the chmod command. Nothing in front of that.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: semenyuk.yura on January 08, 2014, 03:06:05 PM
Quote from: g3gg0 on January 08, 2014, 01:06:47 PM
Only enter the chmod command. Nothing in front of that.

As you've understood i'm not good at programming,

so i tried 2 variants of how i understand your advice, but none of them ended successfully.

(http://fs149.www.ex.ua/get/340328571301/89657526/Screen%20Shot%202014-01-08%20at%203.44.30%20PM.jpg)

I'm running osx 10.9.1, if that matters.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: g3gg0 on January 08, 2014, 03:17:18 PM
chmod +rwx /path/to/your/recursive.command0.3
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: semenyuk.yura on January 08, 2014, 04:30:20 PM
Hooray! Thanks a lot!
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: shinokubo on January 13, 2014, 09:06:24 PM
Hello friends,

is there a way for me to set a specific destination folder, by altering or adding a line to the script?

By now I am copying .mlv files from the CF Card to a certain folder and then run the script in there. As the folder is on an external hdd, the time for the copying process and mlv_dump to process the files could be greatly reduced by having different input and output drives.

Any help is deeply appreciated, the same way all your work already is.

Thank you very much in advance,

Ben
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: djfremen on January 14, 2014, 04:54:25 AM

Quote from: shinokubo on January 13, 2014, 09:06:24 PM
mlv_dump to process the files could be greatly reduced by having different input and output drives.

Explain how this would be faster please. I understand the concept of reading from one drive and writing to another but I don't think it would be a huge difference. I just processed 500 GB of MLV and it took under 2 hours on my i7 4770.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: Danne on January 14, 2014, 07:20:36 AM
You can also double click the recursive script several times and have lke four processes working simultaneously.
Would fasten it even further.
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: shinokubo on January 14, 2014, 02:26:02 PM
I am using a macbook pro with an external USB 3.0 drive that is buspowered. It is a 5400 rpm drive that levels out at around 50Mb/s maximum read+write at the same time. It can write at around 95Mb/s though. As the input and output is on the same drive, I am stuck with that speed.

I would not have to copy the files at ~95Mb/s from CF -> HDD and then convert them at 50Mb/s, but could place mlv_dump and the script on the cf card, read from there and have the output written at (I'm guessing) around 90-95MB/s on the HDD

The limiting factor is drive speed, not the cpu so far.

It should take roughly 1/3 of the time if I am not mistaken here.

cheers, Ben

edit: typo and clarification
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: arturochu on January 16, 2014, 07:35:03 AM
windows? any?
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: alexpolsky on March 14, 2014, 07:06:15 PM
Hello!I do not dump mlv file in raw or in dng with mlv_dump & MlRawViewer (OSX)
In MlRawViewer playing stoping on 46 frames and app hangs(Other video good play and converting on dng.
In mlv_dump this log
Mac-Pro-MacMan:Movie macman$ ./mlv_dump -f30 M07-2323.mlv out.raw

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-2323.mlv'
   - Verify file structure
File M07-2323.mlv opened
File M07-2323.m00 opened
File M07-2323.m01 not existing.
File M07-2323.IDX opened (XREF)
XREF table contains 323 entries
Processing...
Reached end of all files after 323 blocks
Processed 299 video frames
Done
I don't have M07-2323.m01 on SD or CF card.

Is it possible dump to from a particular frame. Because if you miss the 46 frame video on playing in MlRawViewer.
Sorry my english. I am from Ukraine.  Please help!
Modify message
Title: Re: Feature Request: Batch Convert MLV files to DNG
Post by: ShootMeAlready on January 21, 2017, 08:08:12 AM
mlv_dump command line examples for Windows??? Cant find any???

I am trying to process a 10bit raw video to dng

Here is what I enter (with mlv_dump in directory)
G:\DCIM\100CANON>mlv_dump -o test02 -b 10 --dng m20-1339.mlv


PROBLEM: The output states its defaulting to 14bpp ??? not 10??? when I try with default it still reports 14 bpp
The picture is too small as file properties reports 128x94 96dpi, this is wrong it should be 1824x776

It seems the command-line is wrong ???

Blow is Windows 10 output:

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'm20-1339.mlv'
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Output into 'test02'
File m20-1339.mlv opened
File m20-1339.m00 not existing.
File m20-1339.IDX opened (XREF)
XREF table contains 328 entries
Processing...
Cold pixels : 0
Reached end of all files after 328 blocks
Processed 304 video frames
Done