Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r) (Read 500880 times)

Danne · « **Reply #800 on:** October 15, 2016, 09:23:48 PM »

Pressing Q? Are you starting from an empty folder? What are your trying to do?
You could try running MLP via looks_creator.command and check if you get anything of interest in there.
*solved. Deafeye updated former question.

togg · « **Reply #801 on:** October 30, 2016, 05:57:27 PM »

After the whole installing process (by the way do I have to keep the MLP folder? can I move it on the applications folder?) I get this error:

- Impossibile aprire il documento “MLP.workflow” perché è danneggiato o incompleto.

That means that MLP.workflow is damaged or incomplet. I'm on a mbpr 15" mid 2012, 10.11.6

I'm trying to installt to have another option for conversion. By the way can this automatically detect and correct hot pixels in dng export?

Danne · « **Reply #802 on:** October 30, 2016, 07:24:35 PM »

Hi. thanks for notifying. Seems I uploaded an incomplete version to dropbox. Should be fine now.
Regarding hot pixels they an only be manually mapped out by specifying the coordinates of the pixel in this file. It,s called dead_pixel_list.txt
and you put it inside A_lut_hold folder before starting MLP. Note that it will only apply to the ProRes file upon export and not to the actual dng file.

togg · « **Reply #803 on:** October 30, 2016, 07:47:23 PM »

Thanks for the reply! I'm trying again.

edit: tried, no luck. Still same error. And maybe now I've screwed something up because after another try with the installer I can't even see the menu option anymore. Maybe I should uninstall everything first? How is it done?

I'm mainly installing this because I've just noticed some ugly vertical stripes in some highlight that I need to push on some footage. Also in a few shadows but less disturbing. Does MLP correct it on dng export? I've tried both MLRW and RAWMagic, they have some stripe correction option but it doesn't work. I was also wondering if the exported dng can be compressed, that'w a terrific feature in MLRW that saves ton of space.

Danne · « **Reply #804 on:** October 30, 2016, 08:18:58 PM »

Hi togg. Sent you a pm.
About compressing that can be done with MLP if you have slimraw installed to your system. It,s optional of course. It activates compression when MLV_RAW_dng_repack_or_compress_DNG.txt is included in A_lut_hold folder.

zachnfine · « **Reply #805 on:** November 29, 2016, 10:08:16 PM »

Apologies if this is covered somewhere in the previous 32 pages of this thread (I looked a bit but haven't read every post), but I've just installed MLP and tested it by using it on a folder containing a single MLV file. I didn't change any of the 3 txt files that appeared in the A_lut_hold folder. I noticed the following, when I compared the results to the output of MLVFS and to a simple invocation of mlv_dump:

• The generated prores4444 quicktime file seemed to be constructed of all the frames running in some random, jumbled, unwatchable order. I've got 8 cores, maybe they ran in parallel and concatenated frames to the file out of sequence? I'm just puzzled by this result.

• When I open the generated dng sequence in Davinci Resolve, its embedded white balance settings are very different than the white balance settings of a dng sequence created from the same MLV file simply using "./mlv_dump.osx --dng" from the command line --and those have slightly different white balance settings than the dng files generated from the same MLV using MLVFS. Any idea why all 3 of these methods would embed different white balance values in the resulting dng sequence? For what it's worth, in Davinci Resolve's camera raw settings for each clip I've got all 3 set to "Decode Using: Clip" "White Balance: As Shot" "Color Space: Blackmagic Design" "Gamma: Blackmagic Design Film" with "Highlight Recovery" checked. Here's an image of what I'm seeing for frame 1 of each sequence:

https://dl.dropboxusercontent.com/s/3yagz5n2lur1ptg/2016-11-29%20at%2012.57%20PM.jpg

Apart from that white balance metadata, the content of the DNG frames appears to be the same - if I change the color settings of the MLP to match those of the MLVFS they look identical. I'm just curious why these 3 methods are all coming up with different metadata for white balance and why MLP's are so different from the other two.

Thanks for any help or information.

Danne · « **Reply #806 on:** November 29, 2016, 11:01:53 PM »

Hi.
Not sure what the reason could be here. If you can send me a dng over I could check your file for white balance. ProRes output should be in order though. Randomness indicates something is not right maybe with ffmpeg. Not really sure.
By the way. If you could check with this program cr2hdr.app and see if results are better? It,s stand alone and it exports to prores the same way as in MLP only the bash code in here is built around menus to make it all easier to work with.
cr2hdr.app
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822

zachnfine · « **Reply #807 on:** November 29, 2016, 11:27:56 PM »

Thanks for the reply. I'll check with cr2hdr.

After a bit more googling, I came across this page: http://www.cinelogdcp.com/news-updates/2015/1/27/important-information-for-magic-lantern-users
It looks as though the white balance metadata requires the use of a separate matrix for each camera. They claim MLVFS is doing it right and provide a link to the commit that includes the matrices --maybe mlv_dump doesn't use them?

Danne · « **Reply #808 on:** November 29, 2016, 11:33:54 PM »

MLP and cr2hdr.app uses the color matrices needed. You also get the correct unique camera model tags instead of sometimes localized camera model names.
Most code comes from dmilligan ml_dng branch which is then partly modified by bouncyball.
About your white balance issue. You probably filmed with your cam set to AWB. If so then MLP calculates auto white balance for you since that mode is the only one that can,t be found in mlv metadata.
What you see in mlvfs is the underlying kelvin temp which your cam is set to.
Regular mlv_dump exports hard coded wb values around 5500k.

zachnfine · « **Reply #809 on:** November 30, 2016, 01:54:03 AM »

That makes sense.

I ran cr2hdr - it generated a Prores Proxy and Prores 4444 quicktime with the same random-seeming frame order, just like MLP.

On a different note -- is there a way to get cr2hdr to keep its terminal window around so I could watch the output of the commands and thus get visual feedback during processing (might also help for diagnosing problems like the one I'm experiencing)?

Danne · « **Reply #810 on:** November 30, 2016, 06:29:17 AM »

Here is a version of cr2hdr.app which will print text files next to your dng files. They will be called out_01, out_02 etc depending on which process will run. They will show what ffmpeg is doing.
https://drive.google.com/file/d/0B4tCJMlOYfirUmRfX2EwbFZQLTQ/view?usp=sharing

If you want to use MLP that might be even better. You can run MLP through the looks_creator.command script which is inside the folder with the rest of your files. Double click that command file and then choose (r) run MLP(MLV/RAW) and terminal output will be shown while processing.

Could you upload a file that doesn,t work here? What camera are you using? Are you shooting normal footage 14-bit or 10/12-bit?

zachnfine · « **Reply #811 on:** November 30, 2016, 09:18:04 PM »

I'm running "Magic Lantern Nightly.2016Sep10.5D3113" firmware on a Canon 5D mk III, capturing to the MLV 2.0 format with audio at 1080p 23.976. I think the resulting files are 14-bit raw.

I ran cr2hdr on several additional MLV files, and in each case the resulting quicktime file featured random frame order. On the plus side, the dng frames are in sequential order.

I'm in the process of running the version of cr2hdr.app that will print text files next to the dng. I don't see any text files yet and several dng sequences have appeared. I'll leave it running a while.

I'll see if I can find a smallish MLV file that exhibits the problem to post to dropbox.

zachnfine · « **Reply #812 on:** December 01, 2016, 04:53:16 AM »

Here's a folder on dropbox containing a small-ish MLV file, the quicktime file that cr2hdr made from it, and the output text file:

https://www.dropbox.com/sh/76steg8004016hf/AACiMj5cdU_6YcWxtlwj37Uma?dl=0

Danne · « **Reply #813 on:** December 01, 2016, 05:59:13 AM »

Just tried your file and it works fine over here. Could you zip and send me your folder with dng files of that same mlv file over? Maybe there is something in those files that is not working idk.
What OS are you on? What specs on your computer?
Did you try working from the looks_creator.command script in MLP? I couldn,t find anything in your ffmpeg output textfile unfortunately.

zachnfine · « **Reply #814 on:** December 01, 2016, 05:24:58 PM »

I've added the folder containing the dng sequence to the dropbox folder. It should show up in there in a couple of minutes.

I'm running OS X 10.11.6.

The MLV file and the directory to which dng and quicktime output are directed are both stored on a thunderbolt RAID that's a ZFS filesystem. I suppose that's maybe a little bit of weirdness I've injected into the process.

What's the command that sends the dng files through to ffmpeg? Maybe I can just have it echo the filenames in order rather than pipe it into a process, and see if it's getting them jumbled and track that down.

Danne · « **Reply #815 on:** December 01, 2016, 06:33:01 PM »

Your files works fine over here. You can try some other ffmpeg binary to see if that helps.
http://www.osxexperts.net/ffmpeg/ffmpegexperts.html

All binaries are located inside cr2hdr.app/Contents/
You might wanna try another version of dcraw and exiftool as well. Hard to tell in this case what,s going on. Try on another filesystem if you think that,s the case.

or...

If you are working with cr2hdr.app you can change code inside the source folder. When you,re done you double click Build_dmg_package.command to build a new dmg file. You have to change code in four scripts since they work in parallell. You can start by changing stuff in the first script and run tests on one dng sequence.

!A faster way is to work from the command scripts inside the cr2hdr.app itself. The same file are inside cr2hdr.app/Contents only they,re suffixed as command files!

Anyway. Back to source folder...

FFmpeg_produce_01.txt
FFmpeg_produce_02.txt
FFmpeg_produce_03.txt
FFmpeg_produce_04.txt

I,d suggest scroll down to the bottom of FFmpeg_produce_01.txt file and maybe try do something with the find "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw....

The find command at the bottom affects the ProRes4444 output activated from the ProRes4444 to mov menu.

Code: [Select]

#check if ProRes4444 settings file contains information 
    if ! [ x"$(cat /tmp/FFmpeg_settings | grep -v 'HL\|AWB\|dcrawA')" = x ]
    then
    mkdir -p "$out""$out2"$(date +%F)_ProRes4444
#export ProRes4444
    find "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M $h2 $o $S -c -6 -W -q 3 $gam $wb $pix | ffmpeg $wav -f image2pipe -vcodec ppm -r "$fps" -i pipe:0 $sd -vcodec prores_ks -pix_fmt yuv444p10 -n -r "$fps" $cin$cin_01$cin_02$cin_03$cin_01b$cin_02b$cin_03b$scale$denoise$sharpen "$out""$out2"$(date +%F)_ProRes4444/"$name".mov
    fi

zachnfine · « **Reply #816 on:** December 01, 2016, 08:50:32 PM »

I'll have a look when I get a chance -- am about to have a few days away from the computer. My bet is that for whatever reason the find command is coming back with frames out of sequence for zfs. Maybe I need to change it to something like:

find "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | sort | xargs -0 dcraw ...

or maybe 'sort' wouldn't produce the right order but adding the '-s' flag to 'find' would do the right thing.

Danne · « **Reply #817 on:** December 01, 2016, 09:00:56 PM »

If that,s the case some sort command should probably do. Last time I checked wildcard function should sort anyway. Hopefully you,ll narrow it down.

zachnfine · « **Reply #818 on:** December 06, 2016, 12:51:58 AM »

I ran a test on the same MLV file on an HFS+ filesystem and the resulting quicktime was all in order. So I was pretty sure the results I was getting must've been due to the 'find' command somehow returning a list of dng files in jumbled order when run on a directory of dng files that are stored in a ZFS volume.

So I opened all the FFmpeg_produce files and added " -s " to every invocation of "find". I ran it on the MLV stored on HFS+ and there was no adverse effect. Then I ran it on the MLV stored on ZFS and the result was a quicktime file with frames in the proper order. So the find invocations looked like:
find -s "$out""$out3""$name""$sl". -maxdepth 1 -iname '*.dng' -print0 | xargs -0 dcraw +M $h2 $o $S -c -6 -W -q 3 $gam $wb $pix | ffmpeg $wav -f image2pipe -vcodec ppm -r "$fps" -i pipe:0 $sd -vcodec prores_ks -pix_fmt yuv444p10 -n -r "$fps" $cin$cin_01$cin_02$cin_03$cin_01b$cin_02b$cin_03b$scale$denoise$sharpen "$out""$out2"$(date +%F)_ProRes4444/"$name".mov 2> out_01

Just that little "-s" flag did the trick. So that seems to be a solution.

Pretty weird little quirk of find and openzfs. Interesting.
https://openzfsonosx.org/

Danne · « **Reply #819 on:** December 06, 2016, 01:16:12 AM »

Nice find. And fix.

kamranjon · « **Reply #820 on:** December 12, 2016, 05:11:26 PM »

The user guide link is broken. Does anyone have an alternative?

Danne · « **Reply #821 on:** December 12, 2016, 05:59:07 PM »

It,s up again. I made a slight change this afternoon and forgot to link. Enjoy. Link in first post.
Also feel free to check this workflow. All menu based.
cr2hdr.app
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822

hughred22 · « **Reply #822 on:** January 12, 2017, 03:02:03 AM »

Hi sorry noob here.

I want to figure out a workflow to convert my 48fps 1920x508 to 48fps 1920x818 with MLP. Since I will love to avoid AE in all cost. But if this can not be done, I will love to know how should I approach this. So after I ran MLP with a folder of all the dng files, do I important them into After Effects to stretch them to 1920x818? And from After Effects, how can I easily get a ProRES4444 C-log mov file so I can edit inside Adobe Premiere? I use Magic Bullet Colorista for color grading so if it is c-log, I know what to do next. But do I output from After Effects to ProRes4444 after stretching? Or can I stretch the footage and save it back as DNG sequence for Resolve to color grad to create offline edit in Premiere? Sorry I am so confuse the Mark III RAW workflow here and I am noob and trying to figure it out. Lots of guide out there is out of date... Thank you so much in advanced!

hughred22 · « **Reply #823 on:** January 12, 2017, 03:12:51 AM »

Also, I don't see the wav file inside the PNG folder. I thought MLV file should have Audio file embeded in it, right?

Walter Schulz · « **Reply #824 on:** January 12, 2017, 08:49:10 AM »

Only if audio was enabled, though.
You had MLV_SND.mo loaded and audio recording enabled?

News:

Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r) (Read 500880 times)