MLP - Mac OSX batch processing workflow (former cr2hdr-r)

[Danne]

Sorry, I can,t really follow your steps. You say

I've copied all my RAW files to my LACIE disk, but then i right click i don't get the Service tab. When i right click on a random folder on my desktop, i do however get the Services tab. Why is that?

WHat is it that you are right clicking here? Are you on another harddrive?

but then i right click i don't get the Service tab

Then you say in the next message

This is from when i do it from Finder.

Are we on the lacie harddrive or is this another issue?

Just trying to follow your steps. This is what works for me.

1 - Right click any folder containing RAW/MLW/CR2/MOV/mov files and the services "MLP" option appears.
2 - Right click straight on files like MOV, dng, DNG, RAW, MLV and services "MLP" is showing.

Is there any other way to do this?
/D

[MortenSJ]

First, thanks for helping. I really appreciate it!

Sorry if i'm not thorough enough.

I copied all my RAW files to a folder on my LACIE disk called RAW TEST SHOOT. I right click on the folder, and i don't see any Services tab. Not even if i right click on some of the RAW files. The only time i get the Services tab is when I'm on my Desktop and right click a random folder.

[Danne]

Ok, Could you try this.
Press the buttons cmd g
When opens up go here
~/Library/Services/MLP.workflow/Contents/



Take out some items from the contents folder and then put them back again. Try A_dan_log_3d_luts and HDR_align and put them back again. Sometimes automator has to be "woken up". Now if that doesn.t work you could try renaming your folders to something more casual like My_files  or something.
Do you have the latest download from the first post by the way?

[Danne]

Here,s  a little workflow of how to use HDR "on the fly" function with canon native H.264 MOV files.
By adding following files to the A_lut_hold folder all functions needed will be available.
HDR_tblend_average.txt
HDR_tblend_average_half_FPS.txt
FFplay_Histogram.txt

First file is filmed in 60 fps, the rest is filmed in 50 fps.

[MortenSJ]

Thanks for your help Danne

I upgraded to El Capitan with a clean install and now it seems to work. I'm a little curious though. I started conforming about 64gb of footage and it takes forever.

I've conformed 9 RAW files each about 1 GB. It took about 32 min in total. I used to use RAWMagic to conform my files, and i just did a quick test and i was able to conform a single file of 1GB in less than 30 seconds. What's the difference between MLP and RAWMagic except for all the extra settings?

Maybe it's my workflow. This is what i do. I right click the folder containing my RAW files. I get the two folders A_lut_hold and A_dan_log_3d_luts. I delete the prores text file cause i don't need them. I then run looks_creator.command and hit "r" to conform the files.

Best, Morten

[Danne]

HI, nice to hear that it works. Are yu using the MLV_RAW_dng_repack_16bit.txt file? If you don,t work in premiere it shouldn,t be needed and will save some time. Exiftool is the other process that will prolong processing since it is inserting missing metadata.

[MortenSJ]

No, i'm only using the standard files.

MLV_RAW_dng_dcraw_any_command.txt
MLV_RAW_dng_gamma_2.4_0.txt

That's it.

[aschille84]

Hi,
and thanks for the great app!
Cant seem to get all functions working, lately the HDR conversion. Followed the procedure in the video, but i can only preview, not write a new file. Tried re-installing the app, maybe I should re-install the binaries also?

The app codes for a while and then exits and put the .mov file i the A_HDR_folder. Any suggestions?

[Danne]

What HDR workflow are you using? Seems FFmpeg needs a -strict -2 in the syntax in some circumstances. I have a new version of MLP coming up later tonight.

Could you do following.

1 - upload a sample file
2 - check if all binaries are present. Should say so if you used the script.
2 - what files are you using in the A_lut_hold folder?

[aschille84]

Thanks, Im doing some testing at the moment..
I used the tblend_average.txt and the tblend_average_half_FPS.txt og histogram. I installed all the binaries when first installing MLP, couple of weeks ago.

Here is the file I tried with: https://www.dropbox.com/s/lpahqby6yq9unfo/MVI_6693.MOV?dl=0

[Danne]

I, ll check later. Should work. Meanwhile, download the latest version and run the binary script prior to installing. It will ask if you want to reinstall the old binaries. Not sure if it will grab dcraw as well. It should be updated to the version in latest download.

[Danne]

New version up. Reworked main script a lot due to some changes before. It should also not mix folders named to files name with the actual files now. A lot of different processes to keep track of here.

@aschille84
Just checked your file and it rendered perfectly over here. A lot of aliasing on the 6D, you might wanna turn the sharpening down and also use the disable_sharpening.txt file. I put these in the A_lut_hold folder.
FFplay_Histogram.txt
HDR_MOV_disable_sharpening.txt
HDR_tblend_average_half_FPS.txt
HDR_tblend_average.txt

Here,s you file rendered.
https://sprend.com/download.htm?C=c7b69f70dd4b4192a5df474cc188fa9e

[aschille84]

Thanks @Danne!
Great tip with the sharpening, aliasing is massive In 720pon the 6d. Also got MLP working now, re-installed binaries. Not sure I had Mlrawviewer In there before.

[Piotr.23]

Thanks @Danne!

It works for me; I was able to process the HDR MOV!
However, some points to note:

The
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
command needs the Xcode.app to be pre-installed (and this is > 4 Gb download)

The
brew install ffmpeg --with-fdk-aac --with-ffplay
command installs new version of ffmpeg in /usr/local/Cellar/ffmpeg/2.8.5/bin folder (and adds links to /usr/local/bin/ folder)

So, in order to use the MLP, one needs to remove the old ffmpeg (I renamed it by:
1. cd /Users/piotr/Library/Services/MLP.workflow/Contents/
2. mv ffmpeg ffmpeg.old
)

And, to copy the new ffmpeg installed by brew into MLP folder:
cp -p /usr/local/bin/ffmpeg /Users/piotr/Library/Services/MLP.workflow/Contents/

Then, MLP was able to create the HDR version of the MOV file

One thing I cannot fix is the looks_creator.command
Every time I use it, the OS X says it was downloaded by Chrome and is dangerous... Have to go to the security settings and allow to use it. How to disable this warning?

Thanks again! | P

[Danne]

The
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
command needs the Xcode.app to be pre-installed (and this is > 4 Gb download)

4gb, have to find another way to make FFplay work then. There is actually mpv player https://mpv.io/installation/ , a fork of mplayer which also stream FFmpeg content which works too but without sdl. Lacks the wave monitor it seems unfortunately.

So, in order to use the MLP, one needs to remove the old ffmpeg (I renamed it by:
1. cd /Users/piotr/Library/Services/MLP.workflow/Contents/
2. mv ffmpeg ffmpeg.old

I downloaded a recent version from the link in the HOWTO file which works. Older FFmpeg will probably not work. Are you using the latest MLP version?

One thing I cannot fix is the looks_creator.command
Every time I use it, the OS X says it was downloaded by Chrome and is dangerous... Have to go to the security settings and allow to use it. How to disable this warning?

I believe there,s a setting for this in system preferences.

Thanks for feedback.

[Piotr.23]

Are you using the latest MLP version?

Hmmm... The versions are not numbered or dated, are they? The way I update is by reading this thread. If new binaries are needed, I install them (for instance, the ffmpeg via brew). I then download and unzip the MLP.ZIP file and click MLP.workflow. The system says there is an old one, and shall it be replaced. I say 'yes'.

Is there another way to upgrade the MLP? | P

[aschille84]

@piotr.23 I dont think Xcode is 4gb if installed thru terminal and the homebrew-process. I did this yesterday and it only took like 5-10 min to download. There is a hint when running the ruby command.

[Danne]

@piotr.23 I dont think Xcode is 4gb if installed thru terminal and the homebrew-process. I did this yesterday and it only took like 5-10 min to download. There is a hint when running the ruby command.

I had the same experience, thanks for sharing.


@Piotr.23
I replaced dcraw due to been having a version that needed libraries before. Therefore I would recommend to download the latest version and dcraw will already be included. However, since FFmpeg are updating and changing commandline outputs I had to change some of the commands in MLP. This will be problematic when using the script which asks to redownload the former versions installed. So suggestion. Either redownload all binaries and run the script again or replace FFmpeg after you have run the script and then install. Quite the hassle but that,s what we get in this open source community

[Piotr.23]

Guys, I'm not making things up: using the terminal, the command
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
on my El Capitan 10.11.2 stopped and asked for the Xcode.app to be installed via App Store. And I have no problem with this! just saying it took a day via wifi (which is fine)

Re: dcraw: the version that is in my /Users/piotr/Library/Services/MLP.workflow/Contents/ folder is v9.26.
So it seems to be the same as in MLP.ZIP/source_binaries/dcraw_GPLv2+.txt

MLP is pretty... experimental and needs some knowledge in shell use.
I'm just sharing my observations, perhaps some of ML users may find them useful | P

[Danne]

No one is making things up. Simply sharing different experiences.
Well my advice is this. Change to a fresh FFmpeg if anyone been using MLP for a while and wants to download the latest MLP version. First time users can simply follow instructions in howto.
Regarding xcode hassle. This will only be needed if you want FFlay as a previewer.
Anybody can feel free to participate in building a static version of FFplay. I havn, t figured out how yet.

*Updated MLP

If you have an older version of MLP already installed and needs to replace ffmpeg.

1 - Download the latest version of MLP and download ffmpeg from the download link in the HOWTO file and move it to the Place_binaries_here folder.

2 - Run the move_binaries.command and and follow the instructions.(Press yes to recopy and press yes again to overwrite ffmpeg)

If you,re a new user simply follow the HOWTO and disregard what,s written above.

[Danne]

Uploaded a new version. Contains a link to an alternative media player which doesn,t have to install external libraries. It,s called mpv and instructions are in HOWTO file. You can still run FFplay by installing it on the system if you rather have that(will be chosen before mpv if both players are installed) Some additional info on page 23 in the user_guide.

*updated the move_binaries.command now so it should be easier to follow especially if you want to put in something new in the PLace_binaries_here folder continuing with an older build.

[Danne]

Uploaded a MLP version with the following changes.

- folder names for MLV files will contain the date from the actual record date instead of the date you actually transcodes to dng files.
 
- inserted dfort,s fine script which fixes the focus pixel issues for cameras like eos M, 650D, 100D, 700D.

The script handles these problems automatically by calculating the map coordinates on the fly from a collection of fpm files. Still a work in progress. More info in this thread. http://www.magiclantern.fm/forum/index.php?topic=16054.msg162313#msg162313
Note that this pixel maps works when exporting to Prores and not to the actual dng files. Check out MLVFS for dng handling.

Would be really nice if people with the affected cams could test and report the picture quality and if it works as it should. Thanks dfort and others involved in fixing these long lasting issues.

[DeafEyeJedi]

Uploaded a MLP version with the following changes.

- folder names for MLV files will contain the date from the actual record date instead of the date you actually transcodes to dng files.
 
- inserted dfort,s fine script which fixes the focus pixel issues for cameras like eos M, 650D, 100D, 700D.

The script handles these problems automatically by calculating the map coordinates on the fly from a collection of fpm files. Still a work in progress.

Would be really nice if people with the affected cams could test and report the picture quality and if it works as it should. Thanks dfort and others involved in fixing these long lasting issues.

Whew ... once again it has been too long since I last hung out in this fabulous cave of yours, Danne, and for that I apologize. 

Last night I ran a few old files from SL1 just to see how the Focus Pixel Fix works on the fly. I am pleased to say that it works well for Non-Dual-ISO footage but then when I tried running the same quick test with Dual-ISO files which works OK but not as much as it would with the None-Dual-ISO files (I remember @dfort mentioned that this was a still work in progress) however I admit that all of is so fly!

Please see below of my two quick tests comparisons (even tho I am still trying to manage to get the FFplay to show up) and play the footage in previews as oppose to the Tiff's (must be an user error on my part even tho I followed the instructions on the looks_creator.command) but maybe I need to install a missing specific binary that I'll need in order to make this one feature work other than that I AM IN LOVE with the new rendering which spits out the Original Dates from day of shoot instead of when we rendered them. This is HUGE and definitely makes our post lives easier by lightyears! Thanks for implementing @dfort's brilliant concepts into your amazing app to keep things as simple as possible. Hence @dmilligan's infamous words of courtesy "KISS" which applies to everything else in life, right?

Also Special THANKS to @dfort for your drudgery work on the FPF -- it was indeed such an uphill battle so far!

More to come...

[dfort]

Eliminating the focus pixels on dual iso is still a bit of a challenge. Combining some chroma smoothing with the focus pixel fixer helps.

I like how Danne incorporated focus pixel fixing so it is invisible to the user.

Yes, it is a work in progress, like all things in Magic Lantern.

[Danne]

Thanks for testing.

even tho I am still trying to manage to get the FFplay to show up

To use FFplay/mpv media previewer you simply right click a folder containing dng files.

If you want the 5 fps media previewer you right click a MLV or RAW file. To activate this function add the MLV_RAW_player.txt to the A_lut_hold folder. This works outside the looks_creator.command.

Also updated MLP with the latest pixel map files from dfort
http://www.magiclantern.fm/forum/index.php?topic=16054.msg162567;topicseen#msg162567