MLP - Mac OSX batch processing workflow (former cr2hdr-r)

[Danne]

Important
MLP project is abandoned and replaced with:
Switch for macOS Sierra.
Please use that instead to keep yourself updated with the latest.






MLP NO LONGER MAINTAINED! Use Switch above instead.

MLP – Magic Lantern Processor
download_source
user_guide

Forum discussion regarding MLP starts here
MLP_discuss

Word of advice. All processes should be run separately. Keep different  files and processes separated if not stated otherwise. It,s all up to you to test and validate this workflow. I leave no guarantees. It should be treated as beta.

What,s MLP about?
MLP works with the following files:
-   RAW/MLV to dng conversion. Files workable in DaVinci Resolve(embeds audio) and Premiere Pro
-   RAW/MLV and folders of dng files to ProRes4444 files
-   Dual iso CR2 still images are converted to workable DNG files with correct white balance applied
-   HDR processing of both MLV/RAW and H264 mov files
-   HDR processing of CR2 still image files
-   Dual iso dng sequences are processed automatically coming from RAW/MLV files
-   RAW/MLV/dng/DNG/mov previewer
-   Multithreaded

Description
By right clicking a folder, containing any of the above mentioned files, and when selecting MLP from the contextual services menu to specified folder two additional folders will be created prior to running any further processing:
-   A_dan_log_3d_luts
-   A_lut_hold
By adding these folders you are able to add or change different settings before continuing processing such as white balance, brightness, gamma, add different luts. You are also able to change settings for HDR processing.
Adding setting files is easy. Simply pick something from the "A_dan_log_3d_luts" and put it in the "A_lut_hold". Open up the txt file and change setting and save. Then start processing you files. The tricky part is to find the right combinations for how you want your files to be processed.
Whatever setting you create will be viewable as preview tif files if right clicking a RAW,MLV,dng,DNG or mov file directly. For more in depth information the user_guide above is highly recommended.



What you need
You will need some additional files to have MLP fully functional, FFmpeg among them. If you follow the HOWTO.txt in the download package you should be good to go. For doing changes in dng raw files dcraw(Dave Coffin) and exiftool(Phil Harvey) and exiv2(Andreas Huggel) are used. Mlv_dump(g3gg0, A1ex) raw2dng(A1ex) and cr2hdr(A1ex) for  developing dng files. For creating prores dng files are piped to FFmpeg(Fabrice Bellard) via dcraw. (FFmpeg not included in the download). For additional functionality see Howto.txt in the download folder.

For creating 3D luts I used Ben Turley´s cool tool online LUTCalc here https://cameramanben.github.io/LUTCalc/

Thanks to dfort for the tip which lead to being able to have all processes run from the same path. Also for testing and working out ways with the dead pixel fix function in dcraw and detecting a dual iso seg fault which now A1ex has fixed. For feedback and bugtests as always thank you deafeyejedi and dfort.

MLP is an open source project(GPL).
Main reason for this workflow is to optimize processes and get around issues arising in cr2hdr-r when upgrading to OS X El Capitan.
The workflow itself is build around bash scripts run through an automator service. You install it by double clicking the MLP workflow.
The best way to learn MLP workflow is to work with the user_guide.
All MLP sources are in the download. You can easily add changes in the scripts yourself. Add changes in the source_code_MLP folder then double click the A_move_content_to_workflow.command before installing MLP. More about this in the "Howto" file.
MLP still uses the core philosophy from cr2hdr-r but processes are optimized and gives a lot more possibilities to change settings manually. I rewrote the way HDR files are being treated and added some white balance templates. Processes are still multithreaded.


Many ideas and solutions are coming from people here at the forum, stackoverflow and other forums. I tweaked and optimized almost all scripts to fit the way I want it to work. MLP is still a work in progress and everything is open for changes.

Thanks
A1ex, g3gg0, Dave Coffin, Phil Harvey, Fabrice Bellard, Bouncyball, dmilligan, cpc, chmee, Scrax, Jerrykil, Dubzebass, Kichetof, Baldavenger, Andy600, Baldand, dfort, deafeyejedi and others I might forgot to mention here.

GPL (magic lantern)
http://www.magiclantern.fm/forum/index.php?topic=13335.msg128757#msg128757


















cr2hdr-r automator app workflow NO LONGER MAINTAINED! Use MLP instead. For further questions regarding cr2hdr-r please pm me.


- Transcodes MLV/RAW to folders. Dual iso movie files also supported. Dng files will work both in Davinci resolve and Premiere pro CC(thanks cpc for repackto16bit binary)
- ProRes444 creation
- HDR movie (thanks to Jerrykil script)
- CR2 dual iso to DNG conversion
- 3d lut support
- Multithreading
- speed_preview (adding speed and ability to preview one frame)

thread starts around here (lut_lab version)
http://www.magiclantern.fm/forum/index.php?topic=13512.msg144696#msg144696

Always run the app from the parent folder!


Usage
- cr2hdr-r creates two additional folders (HDR_RAW_MLV and PRORES) inside the parent folder on first run, then the app quits.
- Place files in parent folder or in the created subfolders or in all three of them.
- Run the app again from the parent folder and it will convert first from parent folder then search the created subfolders for files.
All RAW/MLV originals will be placed in ORIGINAL folders after conversion

About created folders

Parent folder
- Transcoding MLV/RAW files. Dual iso is supported. dng files will work both in Davinci resolve and Premiere pro CC (Thanks @cpc for 16bit repacking binary).
- Dual iso CR2 files. If adobe dng converter is installed it will reduce the size of the resulting DNG lossless.

PRORES folder (ProRes444 creation)
- Creation of ProRes files.
- White balance control when using silent pic DNG or adding a CR2 with desired WB.
https://www.youtube.com/watch?v=S5SAnakpum0&list=UUomeOeghS6wanMOCQ8BtH_A
- add 3d luts in the mix(up to 20 luts can be used)

- FFmpeg binary needed for ProRes encoding. You find it here http://ffmpegmac.net/
The binary must have the name ffmpeg and have to be placed in /usr/bin/ folder on your computer

More speed and previews
http://www.magiclantern.fm/forum/index.php?topic=13512.msg147712#msg147712
http://www.magiclantern.fm/forum/index.php?topic=13512.msg148522#msg148522
http://www.magiclantern.fm/forum/index.php?topic=13512.msg149104#msg149104

- cr2hdr_r_lut_lab version will install a "speed_preview" workflow. Simply right click the parent folder and run "speed_preview" from services and it will add an extra process to PRORES and HDR_RAW_MLV_MOV folder. If you run the speed_preview by right clicking a mlv/raw/mov/MOV file you get one preview frame from the chosen file. Add CR2 for white balance and add a 3d lut for preset looks.
https://www.youtube.com/watch?v=2ZkXeISWC0c&feature=youtu.be


3D lut support
- Added lut support. add lut inside the PRORES folder or in subfolders together with the RAW/MLV file
- Hardcoded flat looking 3d lut when transcoding dual iso ProRes files. Thanks Andy600.

Up to 20 extra 3d luts can be added. When running the app the first time it creates some additional folder. Inside the PRORES folder some luts and a text description will be provided.

White balance handling
http://www.magiclantern.fm/forum/index.php?topic=13512.msg144750#msg144750
http://magiclantern.fm/forum/index.php?topic=13512.msg148069#msg148069
http://www.magiclantern.fm/forum/index.php?topic=13512.msg148231#msg148231
http://www.magiclantern.fm/forum/index.php?topic=13512.msg149797#msg149797

Preferred workflow in PRORES folder is to add one or more CR2 files( for white balance) and/or up to twenty 3d luts to you files.

Brightness, white balance, denoiser
Change above settings by adding commands to a generated png preview.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg149461#msg149461
works with speed_previews
http://www.magiclantern.fm/forum/index.php?topic=13512.msg149476#msg149476


HDR_RAW_MLV_MOV folder  Updated 2015-09-16! here
http://www.magiclantern.fm/forum/index.php?topic=13512.msg154260#msg154260

Place HDR MLV/RAW footage(48, 50, or 60 fps) in folder and it will transcode to a .mov file.
Place a .MOV HDR file and it transcodes to a merged .mov file
More info here
http://magiclantern.fm/forum/index.php?topic=13512.msg147144#msg147144

FFmpeg binary needed for HDR encoding. You find it here http://ffmpegmac.net/
The binary must be named ffmpeg and have to be placed in /usr/bin/ folder on your computer

All processes are multithreaded!

Thanks Andy600 for provided dual iso luts and N/A for updated dcraw compiles, cpc for DNGRepackto16,
Deafeyejedi and others for tests and feedback.



What is it
Cr2hdr-r relies on script made in bash calling for different binaries through an automator app. Most of the scripts have been gathered and tweaked from users here at the ML forum. The app is also provided as an automator workflow. The processing is done the same for both alternatives.
Main interest for starting this was for fast dual iso conversion(multithreaded) and to apply same levels(prevent flicker) using exiftool, to dual iso movie files but as it emerged it is now a sort of an "all in one" processing solution handling above files.

Usage
Collect all the files you want to convert into a chosen folder. You can add multiple MLV, RAW, CR2, dng, MOV files. They will be converted one after the other. While processing an ORIGINALS folder will be created and original files will be moved to this folder after after processing is done.


*Good to know
Regarding format RAW. If filmed with spanned files such as R.00, R.01 etc the app will automatically cat these files and after the cat process start converting and then erase the spanned files replacing it with the concenated file. I personally I don,t see any reason to keep the spanned files after conversion since you get a new big file instead.


Why use it
I consider it a laid back all in one processing solution taking care of  many of the magic lantern processing needs.
Fast conversion of dual iso dng movie files and dual iso CR2 files (multithreaded).
Exiftool adds back metadata to processed MLV and RAW(fps) files. Also applies the same white level to dual iso movie sequences to avoid flicker.
Audio is shortened(sox) and wav metadata added(bwfmetaedit) to fit with autolinking in DaVinci Resolve.
HDR .MOV files which is converted multithreaded(thanks to Jerrykil).


Additional
automator http://www.macosxautomation.com/automator/
dual iso movie thread http://www.magiclantern.fm/forum/index.php?topic=12796.msg123437#msg123437

Recommended(mac)
MLVFS - dmilligan
http://www.magiclantern.fm/forum/index.php?topic=13152.0

MlRawViewer – baldand
http://www.magiclantern.fm/forum/index.php?topic=9560.0

Lightroom plugin cr2hdr - Kitchehof
http://www.magiclantern.fm/forum/index.php?topic=11056.0

Thanks
This is a great community. No scripting or any creation around this workflow would be possible without the community, and a special thanks to these guys.
Great thanks to A1ex, g3gg0, (cr2hdr20bit, raw2dng, mlv_dump.)
Andy600(dual iso log_lut), @cpc (DNGrepackto16), Dmilligan, Baldand, Kitchehof, Chmee, 
Scrax, Jerrykil, Dubzebass and others I forget to mention.
Tester(s) for feedback.

And of course the other authors of the included binaries.

I think of this as an open, sharing community. I learn and pick up stuff along the way.
This project is me learning scripting and creatng a personlized workflow.
I have no commercial interests. If any questions please feel free to pm me.

GPL (magic lantern)
http://www.magiclantern.fm/forum/index.php?topic=13335.msg128757#msg128757

Source

[kichetof]

Hi Danne,

Cool stuff !!

I've made some changes to your command Feel free to test and implement if you like it !

Code Select Expand

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

echo -n "Enter path to DNG Dual ISO files:"
read path

find $path -name '*.dng' | xargs ./cr2hdr20bit --same-levels

Why did you move cr2hdr ? It works in the folder. So, no need to use sudo, happy user
If you want to move bin, don't forget to try if file exists:

Code Select Expand

if [ -e ./cr2hdr20bit ]
then
#mv ...
fi

[Danne]

Thanks Kitchehof! Will check this out. I,m learning so all input is valuable

[Danne]

So i keep experimenting to be able to use my external hard drive which keeps giving me random errors. Will run a few folders without the same level command and run a script after like this. It seems to be working.

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

find . -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel<BlackLevel" -overwrite_original ; find . -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel+=50000" -overwrite_original 

[dmilligan]

You could also make these into automator workflows (like mlvfs). There's a shell script action, and you can put the cr2hdr binary inside the workflow "package" (which is really just a directory => right click and show package contents), and just reference it from there (workflows are always "installed" to ~/Library/Services/). Open the mlvfs workflow in automator for reference. If you have any more questions on how it works, feel free to ask.

[Danne]

Been lurking around on getting the hang of automator for just that reason. Gonna do some more trying out before I get some questions out there. Good to be in this skilled coding community. Feels real nice.

[DavidSh]

Nice tool, Thanks Danne.

[Danne]

Found a way to multithread the conversion by adding this line this after xargs -P 4 -n 1
The number 4 can be altered but don,t go too high I say. Your computer will run hot. --same-levels didn,t work with this but I could queue the exiftool commands for white and blacklevel so whitelevel is set on 50 000 at the moment. Change the number if you like.

For cr2hdr-r MLVFS.command it would look like this (copy paste):

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

sudo mv ./cr2hdr20bit /usr/bin/cr2hdr20bit
sudo chmod +x /usr/bin/cr2hdr20bit

find /path/to/parent/directory -name '*.dng' | xargs -P 4 -n 1 /usr/bin/cr2hdr20bit ;
find /path/to/parent/directory -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel<BlackLevel" -overwrite_original ;
find /path/to/parent/directory -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel+=50000" -overwrite_original

For the cr2hdr-r.command like this:

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

sudo mv ./cr2hdr20bit /usr/bin/cr2hdr20bit
sudo chmod +x /usr/bin/cr2hdr20bit

find . -name '*.dng' | xargs -P 4 -n 1 /usr/bin/cr2hdr20bit ;
find . -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel<BlackLevel" -overwrite_original ;
find . -name '*.DNG' | xargs /usr/bin/exiftool "-WhiteLevel+=50000" -overwrite_original 

Note. Once the cr2hdr20bit is placed you can erase the two sudo lines in the script.
sudo mv ./cr2hdr20bit /usr/bin/cr2hdr20bit
sudo chmod +x /usr/bin/cr2hdr20bit 

[Danne]

An automator app is now working, at least on my macbook pro 15 inch, mavericks. It will run 4 dual iso dng or CR2 files at the time so it is pretty fast working. I followed some steps from dmilligan (Thanks for pointing me in a direction) using automator and added a y/n question when moving binaries to /usr/bin/ with the script (Thanks Kitchehof).
The app should work straight as is if you have cr2hdr20bit installed in /usr/bin/ folder together with exiftool and dcraw in the same location. If not you could try run the install.command provided with necessary binaries.

Howto
Double click the cr2hdr-r.app and point it to a specific folder, a parent folder or a mlvfs mount. That,s it. Should start conversion when folder is selected. It will convert both dual iso dng and dual iso CR2 files.

@dmilligan. I used your LICENSE text (GNU) from your mlvfs app. Not for myself but for the binaries and respecting the developers whose binaries are used. Is this the right way? Also put in the GPL statement text from magic lantern forum in another document. Please get back to me if anythng seems out of the order here.

Automator cr2hdr-r.app download link (also on the first post)
https://drive.google.com/folderview?id=0B4tCJMlOYfirckJfdkRWU09icWc&usp=sharing

Thank you all and please try it out and report back. And as always. Use it at your own risk of course.

[DeafEyeJedi]

@Danne -- Thanks again for your PM regarding this new tool. After watching your tutorial and carefully read your instruction on how to install the command tools and to edit them in text editor as well...

This is what I'm seeing...



I read it somewhere whereas you mentioned that I need to make sure both draw & exiftool must be in the same location?

My question would be should I locate the files and move them within or should I just redownload them and place them into the same location as the new folder that was created for the DUAL ISO files to be exported into?

Thanks again for creating this great tool. Such a time saver (when I get it right! =))

[Danne]

Hi Deafeyejedi!
Try this link and follow the instructions in the package. Please report back if you get it to work.

Automator cr2hdr-r.app download link
https://drive.google.com/folderview?id=0B4tCJMlOYfirckJfdkRWU09icWc&usp=sharing

One thing with dcraw I think you need to have your computer langauge set to english for it to work.

[DeafEyeJedi]

@Danne -- Thanks again for your second PM... after reinstalling the exiftool, dcraw from within the folder that I downloaded per your suggestion.

Now I can see that it actually converts four of CR2/DNG's seamlessly... Macbook Pro 2006 running Mavricks has not been feeling HOT as of yet. LOL



But I'll be online waiting for the converts to be done and then will try to export it as a quicktime ONCE I color grade them in ACR just to see how it turns out, especially with the same levels being applied from the automator (sweet stuff @Danne!!!)

Unless I'm told otherwise... Thanks again mate!

Be in touch...

[Danne]

Nice to see it in action ! Hope it turns out good.

Don,t forget to let the automator work til the end so all level commands are applied.

[DeafEyeJedi]

I hope so too and I just clicked on one of the DNG's and open it in ACR just to take a peek and its quality... I must admit that it is coming out much cleaner than it does directly from MLVFS alone (not sure why) but now my only question is...

In ACR -- is it a good idea to NOT implement with the highlights/shadows or does that not matter because the same levels has already been applied from the cr2hdr app itself?



I actually like doing your trick with adjusting the TONE CURVES within the Darks & Shadows.

P.S. If I still see some hot/pink pixels (EOS-M) do I just use MlRawViewer AFTERWARDS to export as DNG's with the Hot Pixel Remover enabled or is that not necessary? (Because in the newer version of MLVFS which has a built in OS to show the options to choose from such as HPR) 

I'm still waiting for the the converts to be done within the Cr2hdr app.

To be continue...

[Danne]

Nice.
Post production is a whole different story. Only reason I fiddle with acr occasionaly is to get that last bit of highlight out of the file. Only use the curve tools and not contrast, higlight slider or shadow and such. Exposure, white balance and exposure sliders seems to work. I tend to use mlrawviewer which is also giving me nice pro res files and the workflow is so fast.
Can,t help you at all with dot pixel removal unfortunately.
Feel free to discuss post production in the dual iso movie thread 2,5k.

Same levels has nothing to do with flicker caused from acr. Same levels are needed since cr2hdr treats black and whitelevel individually  and not linear. THis is optimal when taking stills but not for movie sequenses obviously .
Keep it up!

[dmilligan]

I took the liberty to modify your workflow so that it shows up as a "service" and you don't need to "install" cr2hdr, dcraw, exiftool first. I put them inside the workflow "package" contents, and reference them from there, this is possible because when you "install" a workflow service it always goes to the same directory: ~/Library/Services . So installation should simply require double clicking the cr2hdr.workflow and clicking "install" (just like mlvfs).

https://bitbucket.org/dmilligan/mlvfs/downloads/cr2hdr.workflow.zip

[Danne]

Thanks! I started doing a workflow service and it worked but couldn,t understand the reference thing from mlvfs.
Is this command line all it takes to reference to the three binaries?

cd ~/Library/Services/cr2hdr.workflow/Contents/

Ok, think  I understand now. Everything is referenced from the ~/Library/Services and the workflow package in there.

[DeafEyeJedi]

@Danne -- I'm currently uploading a few clips onto Vimeo after using the new workflow you proposed from earlier. Thanks & Be back to you shortly!

@dmilligan -- I went ahead and downloaded the modified workflow that you created w Danne's and it is now enlisted under my 'Service' package. Thanks for that, However from what I am seeing... it seems that I am unable to locate the new files after the process is completed? (It seems to be too quick almost as if it didn't work properly) Thanks again for your work though! (I'm more than happy to help improve your workflow as well)




Because earlier today with the cr2hdr-r app that Danne suggested worked flawlessly as far as I'm concerned.

should I stick with Danne's original way or try with the new workflow?

Unless I am missing something?

[DeafEyeJedi]

Left everything alone in ACR except for the tone curve regarding darks/shadows via your 'trick'... came out not too bad, eh? (although I think I do see some strange whitebalance changes in the middle & end of the clip -- not sure where did that come from) Just a step closer to perfection...
Thanks @Danne!

Shot MLV DUAL ISO w EOS-M > converted with MLVFS > cr2hdr-20bit (Danne's workflow w cr2hdr-r) > Export as 16-bit Tiff's from ACR > Image Sequence with Quicktime7 export as ProRes 422 HQ.

P.S. personally I would rather use MlRawViewer than QT7 so I can just apply some of my favorite LUT's... not sure why it just won't run on my macbook anymore after I tried uninstalling to downgrade back to older version but then I decided to erase that and go back to the latest version 1.3.2 and still no luck so far. Sorry @Baldand for my own stupidity! I'll retry to download from your bitbucket site once again. Be in touch!

[Danne]

Nice to see highlights retained without flicker. The cat is green in the end though .

[DeafEyeJedi]

Here's Take 2 of the latest saga with the DUAL-ISO MLV > MLVFS > cr2hdr-R (20-bit) > MlRawViewer (1.3.2) export as ProRes 4444 w LUT's applied. (I tried to export as DNG's but then it crashes every time -- let me know if you would like for me to send you a crash report regarding this situation @Baldand)

Thanks again to @Danne, @dmilligan, @Baldand & @a1ex for their correspondents & dedication!

[Danne]

I,m trying to get my head around extracting dng files with the raw2dng binary. I want the script to move the files to created folders and then make the command extract the dng files within the folders. Tried with xarg and a few other solutions but files either end up outside the folder or in my home directory and sometimes in the right folder? Maybe someone skilled could explain wht I,m missing here? Code below comes from g3gg0, dubzeebass and internet

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

for FILE in `ls -A1 *.RAW *.raw 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./"$BASE".RAW ./$BASE
    done
   

find . -name '*.RAW' -exec raw2dng  {} \;

[Danne]

Found the answer to get my RAW starting conversion in multiple directories. Will try to incorporate this to the app.

find . -name '*.RAW' -execdir raw2dng  {} \;

[DeafEyeJedi]

looking forward to it @Danne! 

[Danne]

Updated my first post with a new "app version". It will now recursively work on RAW, MLV, CR2, dng files.

Simply place all your files in a folder and run the app in the folder. The app starts creating folders and then converts the dual iso files and applies same level within the folder(This goes for RAW, MLV). At the end it puts the originals to an ORIGINALS folder. It will run 4 dng files or CR2 files at the time and your computer will work multithreaded.
It is all BETA and I consider myself noob in this territory. USE and TEST WISELY, backup with copies before testing and let me know what you think.

cr2hdr-r 0.1 "app"
https://drive.google.com/folderview?id=0B4tCJMlOYfirckJfdkRWU09icWc&usp=sharing