Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r)  (Read 490034 times)

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #600 on: January 17, 2016, 11:35:12 PM »
Sorry, I can,t really follow your steps. You say
Quote
I've copied all my RAW files to my LACIE disk, but then i right click i don't get the Service tab. When i right click on a random folder on my desktop, i do however get the Services tab. Why is that?

WHat is it that you are right clicking here? Are you on another harddrive?
Quote
but then i right click i don't get the Service tab


Then you say in the next message
Quote
This is from when i do it from Finder.
Are we on the lacie harddrive or is this another issue?

Just trying to follow your steps. This is what works for me.

1 - Right click any folder containing RAW/MLW/CR2/MOV/mov files and the services "MLP" option appears.
2 - Right click straight on files like MOV, dng, DNG, RAW, MLV and services "MLP" is showing.

Is there any other way to do this?
/D

MortenSJ

  • New to the forum
  • *
  • Posts: 32
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #601 on: January 18, 2016, 12:30:36 AM »
First, thanks for helping. I really appreciate it!

Sorry if i'm not thorough enough.

I copied all my RAW files to a folder on my LACIE disk called RAW TEST SHOOT. I right click on the folder, and i don't see any Services tab. Not even if i right click on some of the RAW files. The only time i get the Services tab is when I'm on my Desktop and right click a random folder.


Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #602 on: January 18, 2016, 12:54:52 AM »
Ok, Could you try this.
Press the buttons cmd g
When opens up go here
~/Library/Services/MLP.workflow/Contents/



Take out some items from the contents folder and then put them back again. Try A_dan_log_3d_luts and HDR_align and put them back again. Sometimes automator has to be "woken up". Now if that doesn.t work you could try renaming your folders to something more casual like My_files  or something.
Do you have the latest download from the first post by the way?

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #603 on: January 18, 2016, 07:53:49 PM »
Here,s  a little workflow of how to use HDR "on the fly" function with canon native H.264 MOV files.
By adding following files to the A_lut_hold folder all functions needed will be available.
HDR_tblend_average.txt
HDR_tblend_average_half_FPS.txt
FFplay_Histogram.txt


First file is filmed in 60 fps, the rest is filmed in 50 fps.


MortenSJ

  • New to the forum
  • *
  • Posts: 32
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #604 on: January 19, 2016, 12:28:22 AM »
Thanks for your help Danne

I upgraded to El Capitan with a clean install and now it seems to work. I'm a little curious though. I started conforming about 64gb of footage and it takes forever.

I've conformed 9 RAW files each about 1 GB. It took about 32 min in total. I used to use RAWMagic to conform my files, and i just did a quick test and i was able to conform a single file of 1GB in less than 30 seconds. What's the difference between MLP and RAWMagic except for all the extra settings?

Maybe it's my workflow. This is what i do. I right click the folder containing my RAW files. I get the two folders A_lut_hold and A_dan_log_3d_luts. I delete the prores text file cause i don't need them. I then run looks_creator.command and hit "r" to conform the files.

Best, Morten

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #605 on: January 19, 2016, 12:55:09 AM »
HI, nice to hear that it works. Are yu using the MLV_RAW_dng_repack_16bit.txt file? If you don,t work in premiere it shouldn,t be needed and will save some time. Exiftool is the other process that will prolong processing since it is inserting missing metadata.

MortenSJ

  • New to the forum
  • *
  • Posts: 32
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #606 on: January 19, 2016, 01:12:16 AM »
No, i'm only using the standard files.

MLV_RAW_dng_dcraw_any_command.txt
MLV_RAW_dng_gamma_2.4_0.txt

That's it.

aschille84

  • Freshman
  • **
  • Posts: 72
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #607 on: January 19, 2016, 08:16:40 PM »
Hi,
and thanks for the great app!
Cant seem to get all functions working, lately the HDR conversion. Followed the procedure in the video, but i can only preview, not write a new file. Tried re-installing the app, maybe I should re-install the binaries also?

The app codes for a while and then exits and put the .mov file i the A_HDR_folder. Any suggestions?

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #608 on: January 19, 2016, 08:23:10 PM »
What HDR workflow are you using? Seems FFmpeg needs a -strict -2 in the syntax in some circumstances. I have a new version of MLP coming up later tonight.

Could you do following.

1 - upload a sample file
2 - check if all binaries are present. Should say so if you used the script.
2 - what files are you using in the A_lut_hold folder?

aschille84

  • Freshman
  • **
  • Posts: 72
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #609 on: January 19, 2016, 08:29:01 PM »
Thanks, Im doing some testing at the moment..
I used the tblend_average.txt and the tblend_average_half_FPS.txt og histogram. I installed all the binaries when first installing MLP, couple of weeks ago.

Here is the file I tried with: https://www.dropbox.com/s/lpahqby6yq9unfo/MVI_6693.MOV?dl=0


Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #610 on: January 19, 2016, 08:40:00 PM »
I, ll check later. Should work. Meanwhile, download the latest version and run the binary script prior to installing. It will ask if you want to reinstall the old binaries. Not sure if it will grab dcraw as well. It should be updated to the version in latest download.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #611 on: January 19, 2016, 09:53:36 PM »
New version up. Reworked main script a lot due to some changes before. It should also not mix folders named to files name with the actual files now. A lot of different processes to keep track of here.

@aschille84
Just checked your file and it rendered perfectly over here. A lot of aliasing on the 6D, you might wanna turn the sharpening down and also use the disable_sharpening.txt file. I put these in the A_lut_hold folder.
FFplay_Histogram.txt
HDR_MOV_disable_sharpening.txt
HDR_tblend_average_half_FPS.txt
HDR_tblend_average.txt


Here,s you file rendered.
https://sprend.com/download.htm?C=c7b69f70dd4b4192a5df474cc188fa9e


aschille84

  • Freshman
  • **
  • Posts: 72
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #612 on: January 20, 2016, 01:28:26 PM »
Thanks @Danne!
Great tip with the sharpening, aliasing is massive In 720pon the 6d. Also got MLP working now, re-installed binaries. Not sure I had Mlrawviewer In there before.

Piotr.23

  • New to the forum
  • *
  • Posts: 10
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #613 on: January 20, 2016, 06:38:26 PM »
Thanks @Danne!

It works for me; I was able to process the HDR MOV!
However, some points to note:

The
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
command needs the Xcode.app to be pre-installed (and this is > 4 Gb download)

The
brew install ffmpeg --with-fdk-aac --with-ffplay
command installs new version of ffmpeg in /usr/local/Cellar/ffmpeg/2.8.5/bin folder (and adds links to /usr/local/bin/ folder)

So, in order to use the MLP, one needs to remove the old ffmpeg (I renamed it by:
1. cd /Users/piotr/Library/Services/MLP.workflow/Contents/
2. mv ffmpeg ffmpeg.old
)

And, to copy the new ffmpeg installed by brew into MLP folder:
cp -p /usr/local/bin/ffmpeg /Users/piotr/Library/Services/MLP.workflow/Contents/

Then, MLP was able to create the HDR version of the MOV file

One thing I cannot fix is the looks_creator.command
Every time I use it, the OS X says it was downloaded by Chrome and is dangerous... Have to go to the security settings and allow to use it. How to disable this warning?

Thanks again! | P

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #614 on: January 20, 2016, 06:58:17 PM »
Quote
The
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
command needs the Xcode.app to be pre-installed (and this is > 4 Gb download)

4gb, have to find another way to make FFplay work then. There is actually mpv player https://mpv.io/installation/ , a fork of mplayer which also stream FFmpeg content which works too but without sdl. Lacks the wave monitor it seems unfortunately.

Quote
So, in order to use the MLP, one needs to remove the old ffmpeg (I renamed it by:
1. cd /Users/piotr/Library/Services/MLP.workflow/Contents/
2. mv ffmpeg ffmpeg.old

I downloaded a recent version from the link in the HOWTO file which works. Older FFmpeg will probably not work. Are you using the latest MLP version?


Quote
One thing I cannot fix is the looks_creator.command
Every time I use it, the OS X says it was downloaded by Chrome and is dangerous... Have to go to the security settings and allow to use it. How to disable this warning?

I believe there,s a setting for this in system preferences.

Thanks for feedback.


Piotr.23

  • New to the forum
  • *
  • Posts: 10
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #615 on: January 20, 2016, 07:13:17 PM »
Are you using the latest MLP version?

Hmmm... The versions are not numbered or dated, are they? The way I update is by reading this thread. If new binaries are needed, I install them (for instance, the ffmpeg via brew). I then download and unzip the MLP.ZIP file and click MLP.workflow. The system says there is an old one, and shall it be replaced. I say 'yes'.

Is there another way to upgrade the MLP? | P

aschille84

  • Freshman
  • **
  • Posts: 72
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #616 on: January 20, 2016, 07:23:48 PM »
@piotr.23 I dont think Xcode is 4gb if installed thru terminal and the homebrew-process. I did this yesterday and it only took like 5-10 min to download. There is a hint when running the ruby command.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #617 on: January 20, 2016, 07:41:26 PM »
Quote
@piotr.23 I dont think Xcode is 4gb if installed thru terminal and the homebrew-process. I did this yesterday and it only took like 5-10 min to download. There is a hint when running the ruby command.
I had the same experience, thanks for sharing.


@Piotr.23
I replaced dcraw due to been having a version that needed libraries before. Therefore I would recommend to download the latest version and dcraw will already be included. However, since FFmpeg are updating and changing commandline outputs I had to change some of the commands in MLP. This will be problematic when using the script which asks to redownload the former versions installed. So suggestion. Either redownload all binaries and run the script again or replace FFmpeg after you have run the script and then install. Quite the hassle but that,s what we get in this open source community :)

Piotr.23

  • New to the forum
  • *
  • Posts: 10
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #618 on: January 20, 2016, 08:02:23 PM »
Guys, I'm not making things up: using the terminal, the command
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
on my El Capitan 10.11.2 stopped and asked for the Xcode.app to be installed via App Store. And I have no problem with this! just saying it took a day via wifi (which is fine)

Re: dcraw: the version that is in my /Users/piotr/Library/Services/MLP.workflow/Contents/ folder is v9.26.
So it seems to be the same as in MLP.ZIP/source_binaries/dcraw_GPLv2+.txt

MLP is pretty... experimental and needs some knowledge in shell use.
I'm just sharing my observations, perhaps some of ML users may find them useful | P

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #619 on: January 20, 2016, 08:49:23 PM »
No one is making things up. Simply sharing different experiences.
Well my advice is this. Change to a fresh FFmpeg if anyone been using MLP for a while and wants to download the latest MLP version. First time users can simply follow instructions in howto.
Regarding xcode hassle. This will only be needed if you want FFlay as a previewer.
Anybody can feel free to participate in building a static version of FFplay. I havn, t figured out how yet.

*Updated MLP

If you have an older version of MLP already installed and needs to replace ffmpeg.

1 - Download the latest version of MLP and download ffmpeg from the download link in the HOWTO file and move it to the Place_binaries_here folder.

2 - Run the move_binaries.command and and follow the instructions.(Press yes to recopy and press yes again to overwrite ffmpeg)

If you,re a new user simply follow the HOWTO and disregard what,s written above.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #620 on: January 21, 2016, 01:55:28 PM »
Uploaded a new version. Contains a link to an alternative media player which doesn,t have to install external libraries. It,s called mpv and instructions are in HOWTO file. You can still run FFplay by installing it on the system if you rather have that(will be chosen before mpv if both players are installed) Some additional info on page 23 in the user_guide.

*updated the move_binaries.command now so it should be easier to follow especially if you want to put in something new in the PLace_binaries_here folder continuing with an older build.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #621 on: February 15, 2016, 06:55:24 PM »
Uploaded a MLP version with the following changes.

- folder names for MLV files will contain the date from the actual record date instead of the date you actually transcodes to dng files.
 
- inserted dfort,s fine script which fixes the focus pixel issues for cameras like eos M, 650D, 100D, 700D.

The script handles these problems automatically by calculating the map coordinates on the fly from a collection of fpm files. Still a work in progress. More info in this thread. http://www.magiclantern.fm/forum/index.php?topic=16054.msg162313#msg162313
Note that this pixel maps works when exporting to Prores and not to the actual dng files. Check out MLVFS for dng handling.

Would be really nice if people with the affected cams could test and report the picture quality and if it works as it should. Thanks dfort and others involved in fixing these long lasting issues.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3413
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #622 on: February 19, 2016, 01:07:29 AM »
Uploaded a MLP version with the following changes.

- folder names for MLV files will contain the date from the actual record date instead of the date you actually transcodes to dng files.
 
- inserted dfort,s fine script which fixes the focus pixel issues for cameras like eos M, 650D, 100D, 700D.

The script handles these problems automatically by calculating the map coordinates on the fly from a collection of fpm files. Still a work in progress.

Would be really nice if people with the affected cams could test and report the picture quality and if it works as it should. Thanks dfort and others involved in fixing these long lasting issues.

Whew ... once again it has been too long since I last hung out in this fabulous cave of yours, Danne, and for that I apologize.  :-\

Last night I ran a few old files from SL1 just to see how the Focus Pixel Fix works on the fly. I am pleased to say that it works well for Non-Dual-ISO footage but then when I tried running the same quick test with Dual-ISO files which works OK but not as much as it would with the None-Dual-ISO files (I remember @dfort mentioned that this was a still work in progress) however I admit that all of is so fly!

Please see below of my two quick tests comparisons (even tho I am still trying to manage to get the FFplay to show up) and play the footage in previews as oppose to the Tiff's (must be an user error on my part even tho I followed the instructions on the looks_creator.command) but maybe I need to install a missing specific binary that I'll need in order to make this one feature work other than that I AM IN LOVE with the new rendering which spits out the Original Dates from day of shoot instead of when we rendered them. This is HUGE and definitely makes our post lives easier by lightyears! Thanks for implementing @dfort's brilliant concepts into your amazing app to keep things as simple as possible. Hence @dmilligan's infamous words of courtesy "KISS" which applies to everything else in life, right?

Also Special THANKS to @dfort for your drudgery work on the FPF -- it was indeed such an uphill battle so far!



More to come... :)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

  • Guest
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #623 on: February 19, 2016, 03:14:09 AM »
Eliminating the focus pixels on dual iso is still a bit of a challenge. Combining some chroma smoothing with the focus pixel fixer helps.

I like how Danne incorporated focus pixel fixing so it is invisible to the user.

Yes, it is a work in progress, like all things in Magic Lantern.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7701
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #624 on: February 19, 2016, 08:53:38 AM »
Thanks for testing.

Quote
even tho I am still trying to manage to get the FFplay to show up

To use FFplay/mpv media previewer you simply right click a folder containing dng files.

If you want the 5 fps media previewer you right click a MLV or RAW file. To activate this function add the MLV_RAW_player.txt to the A_lut_hold folder. This works outside the looks_creator.command.

Also updated MLP with the latest pixel map files from dfort
http://www.magiclantern.fm/forum/index.php?topic=16054.msg162567;topicseen#msg162567