MLP - Mac OSX batch processing workflow (former cr2hdr-r)

Started by Danne, October 05, 2014, 04:09:34 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Danne

folder_ProRes444_lut Download in first post

There is some more "howto" info in the download.

Some different updates amongst them being able to include a single dng file in lut_hold folder for linear white balance. Mlvfs is delivering correct wb when shooting in "k", manual wb mode, but sometimes you might want the same wb for all clips if camera was set differently for some reason. Recommended to simply drag a dng(or CR2)  file of choice through mlvfs straight to lut_hold folder. Wb multiplier settings will apply to all ProRes files.

01 - Install the the folder_ProRes444 service.


02 - click done.


03 - Select one folder in the mlvfs partition to start the transcoding(see pic below).


04 - PRORES folder and two more folders inside PRORES folder will be created next to your mlvfs partition for a 3d lut workflow. After the folders are created the workflow will quit. This will give you time to select 3d luts which you put in the lut_hold folder.


05 - When luts are ready start the workflow a second time. More info inside dan_log_3d_luts folder


06 - lut_hold folder now supports linear white balance when including a single dng or CR2 file. Wb multipliers from the selected dng will be used to all mov files.


07 - Create a HDR folder in PRORES folder for HDR transcoding


08 - fill the batch_folder with different luts.


09 - Work_with_previews_also_in_HDR



                                                            tutorial

dfort

Hi Danne -- excellent work and exciting to see active development in action.

I took a look at cr2hdr-r a few months ago but at the time I was only using dual_iso for still images and I was also learning how to use Lightroom so I decided to go with kichetof's Lightroom Dual ISO Plugin. Now I'm looking into shooting a project in raw video so I'm trying to figure out a workflow to deal with the MLV files and whoa! I'm confused by the sheer number of options available. Of course everyone seems to have their opinion on which is the "ideal" workflow.

Revisiting your project I'm also finding it somewhat confusing because it is not only for dual_iso but it also encompasses raw video, export to ProRes, LUTs, etc. I'm sure this all makes perfect sense to you and other users who have been following your project but I've already run into issues trying to install the command line tools your project depends on. For example, in your source code you have:

/usr/bin/enfuse

Mac users should be aware of the restrictions that the latest OS, "El Capitan" is putting on where files can be installed and /usr/bin is now off limits. There are ways to work around this but maybe we should ask ourselves if defeating these new Apple policies is something that we really want to do in order to get things working as they used to. The flip side of that argument is that Apple is doing things wrong because OS-X (a.k.a. Darwin) is a Unix type of system and "The UNIX Philosophy" is to work with a bunch of small programs installed on the system each doing a specialized task, the output of one piped into another and rely heavily on shell scripts to glue them all together. So, why not have it both ways? It seems that it would make everything much simpler if all of the dependencies are bundled inside of the app so the user doesn't have to install half a dozen apps in various directories.

I heard of issues regarding "El Capitan" even before updating my system. A cr2hdr Lightroom Plugin user was on a location job and the system update broke the plugin. It seems that not only was /usr/bin off limits but the plugin couldn't use anything that was installed in /usr/local/bin either. I was asking for a while why was it that the dependencies for the plugin were included in the package but the Mac couldn't utilize them. By the way, Windows did use them so the plugin had to have different installation instruction for Windows and Mac. I started playing around with the code (Lightroom plugins are actually written in lua) and with the help of dmilligan and johnrellis at the Adobe Lightoom SDK forum the plugin now works the same on Windows and Mac without the user having to install any dependencies--well, except for maybe the Adobe DNG Converter for compressing DNG's.

The trick (by dmilligan) we ended up using in the plugin looks like this:

if MAC_ENV then
    command = 'export PATH="'..LrPathUtils.child(_PLUGIN.path, 'bin"')..':${PATH} && '..command
end
result = LrTasks.execute(command)


However I was also experimenting with a shell script (suggested by johnrellis) that would probably work for this project.

PATH=`dirname -- "$0"`:$PATH
cr2hdr $@


That just might be a fancy way of adding the current directory (represented by a single dot) to the path but it should persist through the rest of the terminal session. On the Lightroom SDK the path setting doesn't persist and needs to be used every time a call is made to cr2hdr. I would try this out on your project but since you don't have your source code on a distributed version control server (like bitbucket.org which is used on most Magic Lantern projects), it makes it rather difficult to submit suggestions (a.k.a. pull requests) to your project.

FYI to the readers of this topic, Danne and I have been sending PM's back and forth and helping each other out but we believe that this conversation might be best done on the forum so anyone can chime in. After all, this is an open source project and user input is not only welcome but encouraged.

Danne

Thanks dfort. I,ll check into this asap. Meanwhile all scripts are provided in the download. Really cool if this would work. all binaries are being called upon from usr/bin. Not sure how I would implement the solution here.

dfort

I made a demonstration repository for you:

https://bitbucket.org/daniel_fort/cr2hdr-r

Note there is an issue, check it out. There is also a pull request showing a suggestion on how to set the path. In addition there is a download area with your distribution files in it.

Of course this isn't really going to work yet, just a demonstration. I haven't been able to install your project on my "El Capitan" equipped Mac so I can't verify that my pull request even works. I'm also not sure how you have things setup. Is it an Xcode project? I've never worked with one of those.

It takes a while to set up the repository but it is rather easy with the SourceTree app which made to work with bitbucket.org. In addition, dmilligan has posted Mercurial tips and he has helped me out on several occasions--I see he's helped you out too.

The great part about this is that other developers, and some users, can easily submit their suggestions yet you still maintain full control of your project.

dfort

Quote from: Danne on October 09, 2015, 12:15:41 AM
Thanks dfort. I,ll check into this asap. Meanwhile all scripts are provided in the download. Really cool if this would work. all binaries are being called upon from usr/bin. Not sure how I would implement the solution here.
Your scripts are there and the app is there but how did you build your app? If you setup an Xcode project you will discover that it supports Git or Subversion repositories but the Magic Lantern repositories are Mercurial. There is an Xcode plugin available and it is hosted on bitbucket:

https://bitbucket.org/creaceed/mercurial-xcode-plugin

Getting back to the "El Capitan" /usr/bin issues, here's what happens when trying to install cr2hdr-r.

You've got a script that installs the binaries in "cr2hdr-r_11.7_align/Install these first/install_uninstall.command" and the binaries to install are located in "cr2hdr-r_11.7_align/Install these first/Binaries_cr2hdr-r" so let's see what happens on OS-X 10.11, I'll use dcraw as an example:

First of all check to see if dcraw is installed on this system.
Rosie-Forts-Computer:Install these first rosiefort$ which dcraw
Rosie-Forts-Computer:Install these first rosiefort$ ls /usr/bin/dcraw
ls: /usr/bin/dcraw: No such file or directory


The install script is meant to be run by double clicking on it but since it is from an "unidentified developer" you need to control click -> Open the first time you run it, here's the terminal output:

/Users/rosiefort/Desktop/cr2hdr-r/cr2hdr-r_11.7_align/Install\ these\ first/install_uninstall.command ; exit;
Rosie-Forts-Computer:~ rosiefort$ /Users/rosiefort/Desktop/cr2hdr-r/cr2hdr-r_11.7_align/Install\ these\ first/install_uninstall.command ; exit;
Password:
ffmpeg binary is missing. Download and place in /usr/bin/ folder for Prores and HDR transcoding.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]


So it looks like except for the optional ffmpeg that isn't in your Binaries_cr2hdr-r everything got installed but on "El Capitan" nothing got installed, not in /usr/bin or anywhere else in the system path. Running install_uninstall.command again results in:

Rosie-Forts-Computer:~ rosiefort$ /Users/rosiefort/Desktop/cr2hdr-r/cr2hdr-r_11.7_align/Install\ these\ first/install_uninstall.command ; exit;
cr2hdr-r_speed_preview.workflow
do you wish to uninstall Y/N?ylogout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]


So it seems like it uninstalled but there was nothing to uninstall.

It would probably work following wadehome's instructions:

  • Restart the computer, while booting hold down Command-R to boot into recovery mode.
  • Once booted, navigate to the "Utilities > Terminal" in the top menu bar.
  • Enter csrutil disable in the terminal window and hit the return key.
  • Restart the machine and System Integrity Protection will now be disabled.
  • Run install from cr2hdr package to install the files.
  • (optional) To reenable System Integrity Protection follow the steps above, except use the csrutil enable command for step 3.
However, wouldn't it be better if the app would work without having to install dependencies or work around the new security policies? Placing the dependencies inside of your app, like was done with the Lightroom plugin, and running them from there would certainly simplify the installation. Users could still install custom versions of the binaries if they desire by simply control clicking on the app's icon and Show Package Contents then replacing the binaries.

Danne

It, s shell scripts running through automator. I have no idea how to work with xcode. It, s all bash scripts.
I have no time right now but install dependencies will be checked if I could do something about.
Wadehome solution works but I could also direct all binaries to usr/local/bin before a more stable solution is made.
Thanks for taking the time dfort.

dfort

Got it--I never worked with automator. Checked out the documentation and it looks like an interesting app. Reminds me of HyperCard which I never learned how to use either. Wow, I just dated myself.

Completely understand about not having the time to do this. I retired recently and can finally put my time into the projects that interest me the most. However I'm getting called back to work to fill in for an editor that landed in the hospital. Didn't know operating an Autodesk Flame was that dangerous  :o

Anyway, I took the first (and maybe final) step--changed everything in your ".workflow" files and source code from /usr/bin to /usr/local/bin. The binaries now install in /usr/local/bin and everything seems to be working fine on "El Capitan". One side benefit is that you don't need administrator privileges so I also took out your sudo commands and now the user doesn't need to enter a password to install-uninstall.

Here you go: https://bitbucket.org/daniel_fort/cr2hdr-r/downloads/cr2hdr-r_11.7_align_OS_10.11_test.zip

Danne

Hi dfort. I actually have a version with the local/bin folder. Wanted to make sure it was working in el capitan before putting it up.
Something has doubled the size of the "install these first" folder of yours from around 105 to 239mb? I also noticed that when I don,t enter the full /usr/local/bin path to both dcraw and exiftool they won,t work in tha workflow as it would from usr/bin without full paths. They are found in terminal alright but not in cr2hdr-r. usr/local/bin/exiftool/dcraw will work. This goes for OSX maverick anyways.
THanks for pushing this forward.

dfort

I'm still trying to figure out how to use this so if something doesn't work I wouldn't even know about it. Now that you mention it I do remember seeing some calls that didn't have the full path to the binary but if I type "which exiftool dcraw" it is in the system path. You're probably right though, we had similar issues with the Lightroom plugin.

dfort

By the way, the difference in file size was my mistake. I've got a bunch of extra stuff in your folder_ProRes444_lut.workflow package - no idea how that happened.


Danne

I put up a cr2hdr-r version with usr/local/bin paths. Should be ok. Test it if you like.

DeafEyeJedi

Excellent progress gents and now I'm itching closer towards to updating to El Captain but I refuse to do so on my brand new MBP -- I don't mind biting the bullet with my old MBP just in case.

Looking forward to testing this out and report back soon!

Thanks!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Oh come on Sean -- get on the bleeding edge!

Seriously, like it or not at some point you are going to have to update or fade into obsolescence.

So back to the topic, this is so much more than batch converting dual_iso files. You can also batch convert MLV files to ProRes in 10-bit log color space? That is huge.

DeafEyeJedi

Actually you're right, Daniel! [emoji28]

Yes, sir! Actually this app is one of the greatest stand alone tool that does all the dirty work for you. I seriously cannot wait to meet Danne one of those days and get hammered in Germany!

Try shooting Dual-ISO MLV or RAW and honestly cr2hdr-r is all you need.

Simply start it and sit back for a dip of Tea of have it run overnight for you with peace of mind!

Let's do lunch next week and we'll talk more over this!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Quote from: DeafEyeJedi on October 10, 2015, 04:49:12 AM
I seriously cannot wait to meet Danne one of those days and get hammered in Germany!

I thought Danne was in Sweden. Whatever, count me in too.

Quote from: DeafEyeJedi on October 10, 2015, 04:49:12 AM
Simply start it and sit back for a dip of Tea of have it run overnight for you with peace of mind!

So is the tea so you'll get up several times at night to check on the progress?

Quote from: DeafEyeJedi on October 10, 2015, 04:49:12 AM
Let's do lunch next week and we'll talk more over this!

Next week is good for me. I could use some Magic Lantern tutoring.

DeafEyeJedi

Shoot another user error on my part... Sweden it is and for the most part once I'm done with Tea I would probably just swing by and check on it before hitting the sack for the evening!

Other than that it is pretty solid and reliable for the most part!

Trust me I can't imagine life without cr2hdr-r especially when it comes to Dual-ISO, HANDS DOWN!

Don't forget that it also deals with not only Dual-ISO but HDR (both H264 & RAW) files as well as spitting out ProRes4444 in either with a log or LUT applied (even for HDR & Dual-ISO files too) right off the bat hence the title of this particular thread.

After all there's No ACR nor After Effects. No DR12. No Joke!

Basically a poor man's best batch conversion tool especially when it comes to me being lazy at times.

Nice, eh?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

Getting hammered in Germany. Too funny  :P.
You, re both welcome to Sweden :).
If you, re going prores try out the previewer and the batch_folder function.you quickly spit out batches of looks. I developed this even further in folder_prores444_lut and included some more nice 3d luts.
Might do a video tutorial soon.

dfort

I got a chance to try out the installation on a virgin Mac running El Capitan and thought I'd pass along some notes.

First of all it won't launch the "install-uninstall.command" script because:

Quote"install_uninstall.command" can't be opened because it is from an unidentified developer.

Your security preferences allow installation of only apps from the Mac App Store and identified developers.

Google Chrome downloaded this file today at 9:34 AM from docs.google.com.

The workaround for this is to control-click on "install_uninstall.command" and choose Open from the pop up menu. That tip should be in "install notes.rtf" unless Danne becomes an "identified developer" or puts this on the Mac App Store.



Here's the terminal session from that script:

Last login: Fri Oct  9 13:28:28 on ttys000
Rosies-MacBook-Air:~ Fort$ /Users/Fort/Downloads/cr2hdr-r_11.7_local_align/Install\ these\ first/install_uninstall.command ; exit;
Password:
mkdir: /usr/local/bin: Operation not permitted
ffmpeg binary is missing. Download and place in /usr/local/bin/ folder for Prores and HDR transcoding.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
Deleting expired sessions...none found.

[Process completed]


Checking out the posts about this issue it seems that if the system had a /usr/local directory before the El Capitan update there's no problem making modifications to it but a brand new Mac like the one I'm testing out doesn't have a /usr/local directory even though it is in the default path:

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Apparently there is no way around this. You have to disable System Integrity Protection (SIP) just to create /usr/local. Once that is done you can (and probably should) turn it back on. If you have a /usr/local you shouldn't need administrator privileges "sudo" to make changes.

[UPDATE: I just went into recovery mode, turned off SIP, rebooted, created a /usr/local directory (needed root privileges to do this) then reenabled SIP. Now I can't make any modifications in /usr or /usr/local. It looks like this is going to be a bit more complicated.]

Danne

mkdir: /usr/local/bin: Operation not permitted
That is one crappy situation.
Thanks for sharing dfort.

I guess it,s possible to direct the whole thing to ~/Library/Services/ or something similar?

dfort

That is certainly a possibility or you can use Library/Application\ Support or anywhere that you have write access. Then you will either have to set the path or call the binaries using the full path name. My suggestions to put the binaries inside of your cr2hdr-r package was before I understood how it runs--not that I really know now.

The best solution that I have found is to turn off SIP then run this command:

sudo mkdir /usr/local && sudo chflags norestricted /usr/local && sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local

Then immediately go back and reenable SIP. This will allow full access to /usr/local. Now for the fun part, you can easily delete /usr/local but you can't recreate it without doing the csrutil disable/enable song and dance.

[EDIT: I should give credit to that command that sets the permissions for /usr/local. I found it here: http://digitizor.com/2015/10/01/fix-homebrew-permissions-osx-el-capitan/ ]

dfort

Just found out why cr2hdr-r breaks after the "El Capitan" update. The update moves files located in /usr/bin and other locations into a quarantine folder located in /Library/SystemMigration/History/Migration-(UUID)/QuarantineRoot/


Danne

Interesting.
I should have made it into a service workflow. Too late now. When I tried it in the beginning I had the problems with cr2hdr calling exiftool from system rather than from where I point it too. Interesting you found a solution in the lightroom  app.
I just made a tutorial about how to use the stand alone folder_ProRes444 workflow. This stores all that,s needed inside the workflow so should work straight away in el capitan.
http://www.youtube.com/watch?v=nllfXNd-U58

dfort

Danne--I was going through the steps and believe that your folder_ProRes444_lut.workflow instructions are showing folder_ProRes444.workflow screenshots.

Still figuring this stuff out but hey, it works great on El Capitan and I was able to make a good looking ProRes Log file without having to install any dependencies.

Danne

Thanks for checking. Good that it works on El Capitan. I plan on installing the new OS on a laptop today.
About the screenshots I just named them incorrectly. They are about the lut version. See if I, ll get it fixed soon :).