MLP - Mac OSX batch processing workflow (former cr2hdr-r)

[DeafEyeJedi]

Thanks Danne for the update. Definitely useful features for CR2 Dual-ISO. Will test them and report back.

Meanwhile, I feel that it is also important to note how useful (extremely useful) it is to use DarkFraming Average Process within MLP and yet it's so easy!

Here are some samples for those who are still curious or unsure whether to take a quick dive into the Dark Side even tho I am a Rebel.

5D3 @ ISO 3200 (pushed exposure to +5.00 in ACR for this test purpose):

Pre-DF Avg


Post-DF Avg

7D @ ISO 6400 (pushed exposure to +3.00 in ACR):

Pre-DF Avg


Post-DF Avg


Still unsure whether or not this helped for the 7D? Check again while zoomed in @ 200% in ACR :

Pre-DF Avg


Post-DF Avg


Another good one from 7D @ ISO 3200:

Pre-DF Avg


Post-DF Avg

[Danne]

New version

a1ex added some really nice progress around crop_rec.mo module and I will try to follow the progress in MLP.
Thread here
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171148#msg171148

I added the changes recently added from here and slightly modified some default switches in this version of mlv_dump.
Code in sources_binaries upon download. The modifications are work in progress.
Original commits
https://bitbucket.org/hudson/magic-lantern/branch/bad_pixels

So if you download the crop_rec.mo and put in the ML folder(onto your memory card) according to instructions you can develop the MLV files the usual way in MLP.
What you have to do to get rid of the colorsplotches is to add this file in the A_lut_hold folder
MLV_mlv_dump_settings.txt

Inside the MLV_mlv_dump_settings.txt file specify this at the top row to disable vertical stripes cotrection and add fix cold pixels.
--no-stripes --fixcp

Post your findings in the crop_rec.mo thread. Hopefully this will help get this module a little closer to the nightlies

[calebdescognets]

I'd really love to give MLP a chance considering the good things I'm reading about it but when I click the source download link on pg. 1 it says the file was not found.  Where can I download MLP?  Other methods I've tried have given me unusable ghosting and it seems like it's minimized with this process.

 I have a shot looking out a train window and panning to someone sitting that was shot in mlv with dual iso and I'm afraid no matter what method I try I'll get too much ghosting because of the fast motion in the shot (shot at 24 fps).  Anyone have suggestions as to which way to go about processing?

Cheers!

[Danne]

Dropbox had a problem creating links before. Should work now. Try downloading again.
Regarding you ghosting problem. Dual iso isn,t giving ghosting but HDR will. Have you mixed up your workflows?

[calebdescognets]

Dropbox had a problem creating links before. Should work now. Try downloading again.
Regarding you ghosting problem. Dual iso isn,t giving ghosting but HDR will. Have you mixed up your workflows?

Yes, I did end up confusing the two.  As far as this shot goes I think I'm shit out of luck with the HDR video ghosting since I shot it in 24fps.  If I shot HDR mlv video in 48fps and then used the HDR_tblend_average_half_FPS.txt would it give me no ghosting (max record time with those settings must be next to nothing)?  What's the advantage of using HDR video over dual iso?  Seams like they do the same thing but HDR video is more for landscape/slo-moving shots because of ghosting issues? 

I just did a little dual iso test in my home for practice purposes.  I used cinestyle, at 24fps, iso 100/1600 w/ -0.3 ev, then loaded the dual iso gamma 3.6, dang straight to prioress, and dcraw in MLP and ran.  My pores clips look great but are strangely playing back at double fps? not sure whats going on.  I shot this handheld and sure, it's the morning and I haven't had my whiskey yet, but the shake is ridiculous.  Very difficult for a first time user to dive in head first with this sort of DIY program but I'm slowly realizing how much of a benefit MLP is.   Any help would be much appreciated!

[Danne]

That looks like an earthquake  . Never seen it before .
Welcome aboard the HDR train. Quite some workflows here to learn. Firstly. Dual iso and HDR don,t work together. Are you working HDR? MLV/RAW or H.264? 48 fps or more? 24 or 25 fps? If you give me this information I can give you some handy advice.
By the way. Really liked your Wicked game video.

[DeafEyeJedi]

Even if it wasn't intended for but that footage looked real good if it were an actual movie about earthquakes. So bad ass!

Give us the detailed info and we'll be glad to get this train moving smoothly for you @calebdescognets!

[calebdescognets]

That looks like an earthquake  . Never seen it before .
Welcome aboard the HDR train. Quite some workflows here to learn. Firstly. Dual iso and HDR don,t work together. Are you working HDR? MLV/RAW or H.264? 48 fps or more? 24 or 25 fps? If you give me this information I can give you some handy advice.
By the way. Really liked your Wicked game video.

Even if it wasn't intended for but that footage looked real good if it were an actual movie about earthquakes. So bad ass!

Give us the detailed info and we'll be glad to get this train moving smoothly for you @calebdescognets!

haha you two bring a smile to my face.  Thanks Danne! Currently working on another one for the same artist and you can expect to see it here when I'm done 

I laid out what I did in my first post.. " I used cinestyle, at 24fps mlv raw video, duel iso mod at iso 100/1600 w/ -0.3 ev, shot these two clips, popped cf card in reader, loaded the dual iso gamma 3.6, dang straight to prioress, and dcraw in MLP and ran."  Did I need to add any other commands to the A_Lut_hold folder for duel iso conversion? The user guide seemed to think I only needed the 3 I stated above and MLP would automatically recognize if the .mlv clips were duel iso or not and adjust accordingly

[Danne]

If you shoot dual iso gamma 3.6 can be a good idea. If HDR not so good.

If you shoot HDR 48fps than to put this in the A_lut hold can be a good idea. Don,t set camera to dual iso while shooting HDR.
Try this in A_lut_hold folder
HDR_tblend_average_half_FPS.txt
HDR_tblend_average.txt
You can throw out the HDR_tblend_average_half_FPS.txt is you want to keep 48fps instead of 24fps.

An alternative if you downloaded hugin alignment binary you can align footage. THis will take a really long time but then use these files in A_lut_hold. Footage will end up in 24fps.
HDR_enfuse_settings.txt
HDR_enfuse_straight_from_dng_settings.txt

If you shoot 5d mark III I can really recommend 50 or 60fps h.264. Use the same files as in first suggestion.
HDR_tblend_average_half_FPS.txt
HDR_tblend_average.txt


There,s a lot of stuff going on in this sentence.

I used cinestyle, at 24fps mlv raw video, duel iso mod at iso 100/1600

[calebdescognets]

If you shoot HDR 48fps than to put this in the A_lut hold can be a good idea. Don,t set camera to dual iso while shooting HDR.
Try this in A_lut_hold folder
HDR_tblend_average_half_FPS.txt
HDR_tblend_average.txt
You can throw out the HDR_tblend_average_half_FPS.txt is you want to keep 48fps instead of 24fps.

wouldn't keeping footage at the original framerate (shooting at 48 and finishing in 48 or shooting in 24 and ending in 24) only introduce ghosting with HDR video? Isn't that the appeal of dual iso so that you don't have to double your frame rate (shoot 48fps) if you want to end in 24fps?

An alternative if you downloaded hugin alignment binary you can align footage. THis will take a really long time but then use these files in A_lut_hold. Footage will end up in 24fps.
HDR_enfuse_settings.txt
HDR_enfuse_straight_from_dng_settings.txt

What do you mean by alignment?  Does that eliminate ghosting?  You can tell that Im really just after shooting higher dynamic range video without ghosting.  That's why I shot the dual iso test video above.  I asked before but What's the advantage of using HDR video over dual iso?  Is it really just ghosting with HDR video vs. moire and aliasing with dual iso?  Pick your poison kind of thing?  My apologies if this was already made clear

 Still any ideas as to why I might be getting this weird earthquake effect on my PRORES clips after MLP processes the .mlv clips?

[Danne]

Sorry, thought you were still mixing HDR with dual iso. Realize you are shooting dual iso now. Hm, not sure why you,re footage comes out like that. Was all files successfully processed? Check your sequence and see so cr2hdr didn,t skip any dng;s.
If the file isn,t to big you could maybe upload it for testing?
By the way. Cinestyle or any picture style won,t bite on raw mlv files. Pic styles works only with H.264 material.

[calebdescognets]

Sorry, thought you were still mixing HDR with dual iso. Realize you are shooting dual iso now. Hm, not sure why you,re footage comes out like that. Was all files successfully processed? Check your sequence and see so cr2hdr didn,t skip any dng;s.
If the file isn,t to big you could maybe upload it for testing?
By the way. Cinestyle or any picture style won,t bite on raw mlv files. Pic styles works only with H.264 material.

Realized I had it on cinestyle for no reason right after I uploaded the footage haha 
 No DNGs were skipped in the sequence and when I scroll through the frames one by one there isn't any of this earthquake effect we're getting in the final prores clips so it's gotta be when it's converted from the DNG sequence to prores.

Here is a dropbox link to the original .mlv files: https://www.dropbox.com/sh/5ws86clk1tmkgga/AABJ4tshiYuDezoRgpHfbbZQa?dl=0

[Danne]

Hi @calebdescognets.
I just ran one of your files and the exported fine without any of the earthquake effect. Did you reproduce the issue? Did you interrupt processing in any way, like running MLP while it was already running for instance?

These were my files in A_lut_hold folder
MLV_RAW_dng_dcraw_any_command.txt
MLV_RAW_dng_dual_iso_gamma_3.6_0.txt
MLV_RAW_dng_straight_to_prores444.txt

[calebdescognets]

Hi @calebdescognets.
I just ran one of your files and the exported fine without any of the earthquake effect. Did you reproduce the issue? Did you interrupt processing in any way, like running MLP while it was already running for instance?

These were my files in A_lut_hold folder
MLV_RAW_dng_dcraw_any_command.txt
MLV_RAW_dng_dual_iso_gamma_3.6_0.txt
MLV_RAW_dng_straight_to_prores444.txt

I just re-processed the files and I'm getting the same shake in the final proves clips.  Maybe it's my version of MLP?  I'm unsure what may be causing this anomaly 

I could always just import the processed DNG sequence in AE and export as cineform/prores there but it'd be nice not to have to switch back and forth

[Danne]

Let,s try to get to the bottom of this. Try these suggestions.

If you have your folder with dng files or MLV file in your folder do following.

1 - double click looks_creator.command located in your folder. From the menu select (r) and enter. Terminal will tell you what is happening when processing. Copy paste this content into a txt file and upload the file and sent it to me. You can even reprocess starting with your MLV file if you want.

2 - Send me over your first three DNG files from your sequence.

3 - If I can,t find what,s wrong you can zip your version of MLP and upload it for me to check. Write in terminal
open ~/Library/Services/ hit enter. You,ll find MLP in there, zip and send.

[calebdescognets]

Let,s try to get to the bottom of this. Try these suggestions.

If you have your folder with dng files or MLV file in your folder do following.

1 - double click looks_creator.command located in your folder. From the menu select (r) and enter. Terminal will tell you what is happening when processing. Copy paste this content into a txt file and upload the file and sent it to me. You can even reprocess starting with your MLV file if you want.

2 - Send me over your first three DNG files from your sequence.

3 - If I can,t find what,s wrong you can zip your version of MLP and upload it for me to check. Write in terminal
open ~/Library/Services/ hit enter. You,ll find MLP in there, zip and send.

Awesome, thanks Danne.  Click the dropbox link below.  I included the first 3 DNGs from both sequences, the original mlv files, and the txt doc

https://www.dropbox.com/sh/5ws86clk1tmkgga/AABJ4tshiYuDezoRgpHfbbZQa?dl=0

[yunkai]

Hi guys ! Magic Lantern is really awesome , but honestly I cant get the MAC cr2hdr or any other APPS to transfer DUAL ISO into DNG in China . Because the dropbox is blocked here .......... REALLY BAD THING . So if there anyone who can see this , could plz send the app to me by email ? THANK YOU VERY MUCH !!!   Here it is 1014300852@qq.com

[Kharak]

Could someone here with a Mac upload a sequence of Cinema/DNG's with Darkframe Averaging. Mininum 5 Sec.

Preferably with high ISO and lots of noise or if with low ISO, then  a dark scene where I can push the Exposure to get noise. Also low darks, ETTR.

I would like to do some NR tests on the footage.

Post link to download here or PM me.

[DeafEyeJedi]

@Kharak -- check your PM. Let me know if you need anything else.

[Danne]

Thanks to @calebdescognets for lending me a dng frame from his video here.
http://www.magiclantern.fm/forum/index.php?topic=17927.msg172410#msg172410

Here is a little something useful when working with dng sequences in acr. One issue when opening up dng files is the lack of retrieving highlight information. Dng files are simply clipped in display referred working space as a starting point. Main reason I work with raw files is to get into scene referred working space having all dynamic range represented unclipped as a starting point. Ok, so working with still images using the highlight slider is possible to use for retrieving highlight information back but with movie dng sequences this will give you flicker being non linear treating every image indivdually.
Solution is to create a linear dcp profile in adobe dng profile editor. Free for download here.
https://www.adobe.com/support/downloads/product.jsp?product=195&platform=Macintosh

Once your in the editor it is very easy to create this profile. Open up a dng file in the editor and select linear under tone curve. Then export this linear profile as a dcp and you will be able to select the dcp from within acr when importing your dng files. It,s under "camera profiles"
Here is a linear dcp profile for 5D mark III
https://drive.google.com/file/d/0B4tCJMlOYfirQ21wVk9SYTg3Z3c/view?usp=sharing

Linear setting


Exporting the dcp profile


Examples with and without the profile

Without the dcp profile


With the dcp profile


Example from MLP using dcraw exporting linear and –H 2 –b 2 then adding linear_to_cineon
and then adding a Cineon_to_rec709.cube. Dcraw is doing fine job here.



What is also very cool in acr is that you can retrieve even more highlight information
by changing parametric settings in tone curve.


Tone curve

[DeafEyeJedi]

I am digging this yet another new hidden trick big time, @Danne and gonna take a dip with this one!

Thanks for sharing and will report my results when I can. If I remember correctly that MLP does by default go with -H 2 -b 2 if we were to add the linear_to_cineon/cineon_to_rec709 combo within Dcraw?

[Danne]

Well I think you can do like this. Erase any gamma files and simply add -H 2 -b 2 into the dcraw_any_command.txt file.

[Danne]

Updated to Mac OS Sierra and everything seems to work as before when it comes to MLP. Of course there is still the gatekeeper problem when double clicking a novel command file so that part still have to be handled(ctrl + right click or make a gatekeeper exception in privacy/security settings). I tried getting around this by creating an installer through an apple script put into an automator app but to no avail. However I decided to keep the installer and replaced the original installer script. The HOWTO tell you what to do.
I managed to create a dmg package of the bundle as well but since most stuff isn,t bundled in MLP the benefit of creating the dmg is none. Creating a writable dmg isn,t working good either since it will acting as a big balloon file into which one can fill it up with stuff until the limit size says stop. I used dmilligans very nice script for this coming from here(below link). Just change size and content (source) and you can create your own little dmg package .
https://bitbucket.org/dmilligan/mlvfs/src/9f8191808407bb49112b9ab14c27053ae5022749/build_installer.sh?at=master&fileviewer=file-view-default

[bobbyshots2]

Hey Danne,
Ive been away for a while just checking ad seeing this last post about acr... Looks awesome great work.
Im going to do some testing with those settings in dcraw.
If you need any help wth testing let me know.

[DeafEyeJedi]

Updated to Mac OS Sierra and everything seems to work as before when it comes to MLP.

Got the gatekeeper issue fixed (Thanks @reddeercity & @dfort) and just downloaded MLP from OP with the new installer (really nice!) and now when I tried to run MLP after selecting 'q' to spit the usual Folders for me to work with but it doesn't do anything. Spinning wheel comes on for a second then goes out. Tried reinstalling and restarted my Mac to no avail. I had just used this last night (before I decided to give the download link a try this monring) so I guess I shouldn't have done that? 

My wrong @Danne -- ran into another hiccup with OS X 10.12 Sierra when connecting NTFS drivers (external drive was from client who wanted me to batch process H.264 files to ProResHQ) so now it all seems normal after I learned that I had to reinstall NTFS for Mac OS X since I guess it wasn't corresponding and sorry for yet another false alarm!