Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r)  (Read 460616 times)

ciriuss925

  • New to the forum
  • *
  • Posts: 36
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #675 on: March 20, 2016, 08:05:43 PM »
Hey Danne thank you :)

I decide to give MLP a try - am pretty 'skittish' as it seems to be more of a video processor than a stills photo :D

So I have downloaded the MLP zip and uncompressed it - I am looking at the instructions in the Howto file and it says to download quite a few files.  I googled a few - sox, ffmpeg - and found that these are for audio/videos.   If I just need to do stills photos, do I need to download all of them or just a select few? 

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #676 on: March 20, 2016, 08:24:40 PM »
For still CR2 dual iso files you don,t need to download anything additional. Check page 2 and page 4 in the user_guide after you installed MLP. Good luck :)

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #677 on: March 21, 2016, 07:58:02 PM »
New MLP in first post.
Been spending some time trying to learn basics on timecode. Been disussing with dfort checking and verifying information sharing valuable knowledge on different angles on using timecode in real life plus I,ve been reading different posts on the topic. It,s just a little number in the dng file but for some people probably very useful. Anyway. I had some necessary help from Phil Harvey, creator of exiftool who created a config script which made it work in MLP. Big thanks for this help.
http://u88.n24.queensu.ca/exiftool/forum/index.php/topic,7117.msg35940.html#msg35940

No dropframe inclusion.

This is a start.
For now it will count frames like this
24, 23.976, 23.98, 48 fps              00-23
25, 50 fps                                    00-24
30, 29.97, 60                               00-29

                                               

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #678 on: March 23, 2016, 04:08:59 AM »
@Oswald -- curious to see your file samples from your 100D as I also own a SL1 myself and works well with MLP as far as I'm concerned.

@Danne -- Excellent progress on the TimeCode feature (Thanks @Dfort for the suggestion) and I am actually pleased by @Phil Harvey's work with exiftool.

Will definitely download this and test it out. Thanks, D!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

bobbyshots2

  • New to the forum
  • *
  • Posts: 12
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #679 on: March 24, 2016, 04:44:58 AM »
Hi,
Im new to mlp and i cant get ta pro res mov from dual iso dngs.
I keep getting a zero bytes mov.
I reinstalled MLP twice.
My non-dual iso files are processed fine.

BTW, Im using a MAC running yosemite
Camera a 5dIII
Im also running MLVFS

Any help in the direction would be great.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #680 on: March 24, 2016, 11:08:58 PM »
Darkframe processing through mlv_dump

New version of MLP in the first post.

Always been curious about how to work with the average function in mlv_dump but never got around to do it until now. Been having some conversations with Baldavenger about how this works and I finally think I got the hang of it. Take a look here on Baldavengers excellent post on the topic.
http://www.magiclantern.fm/forum/index.php?topic=15801.msg164620#msg164620

This is a dark frame. You,ll need one. Quote from link above.
Baldavenger
Quote
So for the example above, put the lens cap on the camera (with the correct settings) and record a few seconds. If you were to view the frames of the MLV you would see almost black frames (except for a magenta looking noise). A Dark Frame is a single frame that is to be subtracted from every frame of the destined (actual footage) MLV, so an average frame is made from the few seconds that were shot with the lens cap on.

Big gratitude toward g3gg0 for fixing a bug in the mac version of mlv_dump and for implementing the average darkframe function in mlv_dump.

I put in quite a few things to be used around darkframes. Some ideas are related to Baldavengers thoughts on a workflow, (storing sets of darkframes), other ideas are naturally following the MLP philosophy.
I will try to present my ideas through a few scenarios through pictures and short youtube tutorials. Don,t hesitate to ask or point out things or ideas.

Scenario 1
Full automation
1 – Right click a folder and select MLP to build the folder structure
2 – Place your darkframe footage inside A_lut_hold folder and your footage next to the A_lut_hold folder
3 – Right click your folder once again to start processing

During the process MLP will create the averaged files in the A_lut_hold folder and they will be renamed with camera model, resolution, iso and fps. The original darkframes will be placed in a folder named darkframe_orig inside the A_lut_hold folder. Next MLP will start processing each file in your folder and if it matches the info from the darkframe it will apply the averaged mlv file from the A_lut_hold folder to your movie footage. If no match it will process the mlv files as it normally would.

Movie tutorial





Scenario 2
Creating darkframe mlv files
1 – Right click on a folder containing your filmed darkframe footage
2 – Place your darkframe footage in the A_lut_hold folder
3 – Run MLP on the folder and your averaged darkframes will be created and renamed

Movie tutorial





Scenario 3
Storing darkframe footage to 05_my_darkframes folder
1 – Once your darkframe mlv files are done you can move them to a storage folder called 05_my_darkframes folder. You can do this by manually move them or simply right click the A_lut_hold folder
2 – If your 05_my_darkframes folder contains darkframe files you can store them to your system by right clicking selecting MLP on the folder. The files will then be stored to ~/Library/Services/looks/05_my_darkframes folder
3 – By selecting the MLV_RAW_my_darkframes.txt file and putting it in the A_lut_hold folder you will activate darkframe processing of mlv footage only now it will match the footage with the files in the 05_my_darkframes folder instead of from the A_lut_hold folder
4 – If you want to erase the darkframes stored in your system you can do this by right clicking MLP on the A_dan_log_3d_luts folder.
Check the movie

Movie tutorials





Scenario 4
Adding and renaming darkframe mlv files to the 05_my_darkframes folder
1 – In order for MLP to accept the files in the workflow the darkframes must be prefixed with avg_. If you choose to add random darkframe mlv files to the 05_my_darkframes folder they will be renamed according to metadata info following camera model, resolution, iso and fps
2 – Add your files to the 05_my_darkframes folder and rename the content by right clicking MLP on the 05_my_darkframes folder

Movie tutorial



DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #681 on: March 25, 2016, 04:36:29 AM »
Off I go to join into the Darkside ... Man I'm not even sure if I should do 'Scenario #2' first to create my average dark frame files from MLV (shoot some with caps on) prior to do 'Scenario #1' or vice versa?

I just downloaded the latest MLP and about to shoot some quick test files in ISO 1600, ISO 3200 and ISO 6400 ... Thanks D and may the force be with you!

Once again SPECIAL THANKS to @baldavenger for the implementation of your work as well as to @g3gg0 for the mlv_dump update!

*edit*

For the DarkFrames MLV's that I shot (ISO 1600, 3200, 6400) for each ... do I rename them into a proper name without ruining the MLV file itself, correct? (Just did and all seems fine and normal)

*edit #2*

Looking good so far...



However, I just noticed that Crop-Mode MLV's won't respond to DarkFrame's rendering ... is this to be expected? My wrong I forgot to shoot a dark frame (w caps on) for crop-mode. D'oh!

I also plan on doing another test in increments of 200, 400, 800, 2000, etc just to see the differences between analog and digital ISO when using this feature. Should be interesting.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #682 on: March 25, 2016, 09:43:44 PM »
Here's a quick glimpse of what I was able to accomplish from last night's late-night out testing.

Just for this particular test comparisons it was shot on 5D3 in 1/48th 24p ISO 6400 w 50mm @ 1.2.

•Here's the screenflow of how I organized the the Files into their respective locations prior to render/export within MLP re: Scenario #1.


•Here are the comparisons in AE (Cinelog-C vs Cinelog-C REC709 FM Wide DR v2).


•Screenshots Comparisons (cropped @ 200% in AE).

Pre_DF


Post_DF


•Here are the GIF's for DF comparisons.

@ 200% (cropped):


@ 100%:


Here's the DF overall video comparisons.


Now the question is ... is it worth all the extra effort to reduce the noise w DF processing?

I very much think so even tho it may not be noticeable but I can definitely tell.  8)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #683 on: March 26, 2016, 02:54:55 AM »
Hell yeah! Any noise reduction that doesn't degrade the image is worth it  8)

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #684 on: March 26, 2016, 08:12:05 AM »
Quote
Hell yeah! Any noise reduction that doesn't degrade the image is worth it

I tend to agree. It,s still the same dng but hot pixels, strange colors and banding will dissapear. Maybe the effect is bigger with frsp and shooting nighttime timelapses on higher isos. I welcome any deeper insights on the the darkframe theme and more examples too.

Those gif don,t really show what,s happening. The gif itself seems full of artifacts. Better if you show cropped examples of before, after @deafeyejedi

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #685 on: March 26, 2016, 10:43:04 AM »
Those gif don,t really show what,s happening. The gif itself seems full of artifacts. Better if you show cropped examples of before, after @deafeyejedi

Looks like you were right ... I took them down and replaced them with somewhat better examples (non-Cinelog-C, etc and they are cropped per your suggestion) and let me know if there's anything else that I can do to help.

More to come this weekend!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Oswald

  • Freshman
  • **
  • Posts: 67
  • 7D & Eos M & 100D.100B
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #686 on: March 27, 2016, 09:16:52 PM »
https://onedrive.live.com/redir?resid=CF04D6F1ABB25DA8!239&authkey=!AOmutAiKS-Ov5ug&ithint=file%2cRAW

I got it working, Instal-lion did not succeed well on my hackintosh. I don't know why, but I reinstalled MLP and then it works without any problem.  :) But here is link.
7D, EOS-M & 100D.100b ¶  Sigma 18-35mm, Canon 50mm F1.8, 22 STM, 8-48mm f1.0, 18-55 EF-M STM

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #687 on: April 01, 2016, 01:12:18 AM »
7D Crop-Mode won't respond to DF Avg Process re: MLP atm.

Here are the example Files for whoever wants to try them.

https://mega.nz/#F!6wN1jATb!QWq_l7waSfGuox0jMfELGw
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #688 on: April 01, 2016, 01:28:32 AM »
It says empty folder?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3411
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #689 on: April 01, 2016, 02:13:21 AM »
Had to reboot MBP. Froze up during upload. It's completed and online atm.

Keep in mind other crop-mode's from 7D worked fine during DF Avg. process except for this particular file for some reason.

Which was why I thought it was worth sharing. Thanks, D!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Macke1

  • New to the forum
  • *
  • Posts: 47
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #690 on: April 02, 2016, 12:30:10 PM »
I am a novice to use the MLP. Is there an opportunity to preview different settings? it says about tiff? As I do now, I get to drive the entire process to see what the different 3D LUTs provide image. hope you understand the question

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #691 on: April 02, 2016, 01:59:10 PM »
What happens when you right click MLP on the actual MLV or RAW file?
see page 13 in the user_guide, first post

Macke1

  • New to the forum
  • *
  • Posts: 47

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #693 on: April 02, 2016, 03:16:19 PM »
Did you run the move_binaries.command before installing the latest version? Otherwise you could be missing all your files needed. in this case ffmpeg. If so you can redownload what,s missing following the HOWTO file.

Macke1

  • New to the forum
  • *
  • Posts: 47
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #694 on: April 02, 2016, 03:52:34 PM »
ok, that was the problem. Thanks!!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #695 on: April 22, 2016, 09:14:28 AM »
Updated first post with latest MLP

- For some time there has been support for exporting to openexr and dpx files if imagemagick is installed. More info on this on p27 in the user_guide

- Also included in this version is a new repack txt file called MLV_RAW_dng_repack_or_compress_DNG.txt Which mainly will repack dng files from 14-16bit files but, there will be an alternative here. I,ve been having some conversations with cpc, author of slimraw about adding some commandline options to his program. Now there is so as a non dependent alternative MLP will call for the -premiere option if the program is installed in the application folder. Dng files will be turned to lossless cdng files after they have been extracted to folders. This is something I personally find useful but of course it,s optional for the user.

Macke1

  • New to the forum
  • *
  • Posts: 47
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #696 on: April 23, 2016, 08:22:39 PM »
I was testing to shot some raw mlv today, and the result was gr8, @dfort gr8 work no dots and MLP worked very well!
check it out!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #697 on: April 23, 2016, 08:32:57 PM »
Looks great @Macke1. Could you share settings in cam and what luts you used? Did you film any dual iso or regular MLV? What cam?

Macke1

  • New to the forum
  • *
  • Posts: 47
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #698 on: April 24, 2016, 11:01:30 AM »
I used canon 700d, regular raw movie MLV. Settings in cam - 1280 720 and allow frame skipping.
I used MLV_RAW_dng_straight_to_prores444 - MLV_RAW_dng_gamma_2.4_0 - MLV_RAW_dng_dcraw_any_command - brightness_+2 - 01_HDR_MOV_sRGB_01.cube.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7366
Re: MLP - Mac OSX batch processing workflow (former cr2hdr-r)
« Reply #699 on: May 08, 2016, 06:56:13 PM »
New version out in first post.

A simplistic menu will appear when including legacy RAW files in your parent folder. Choose either to create MLV files from legacy RAW files or to simply process RAW to dng the regular way. It will batch process all RAW files in the directory. Nothing will be deleted or moved if converting to MLV. After conversion you can manually delete or choose to keep the RAW files.
Converting RAW to MLV is all possible thanks to bouncyball. Big thanks for this effort. Check his thread.
Also note that processing RAW to dng files now works without the need of concatenating spanned files. Bouncyball fixed this as well.
http://www.magiclantern.fm/forum/index.php?topic=17185.0

Menu selector



The script part looks like this
Code: [Select]
#courtesy bouncyball at magiclantern.fm
if grep 'RAW' /tmp/magic_l_prores/list_RAW
then
ls *.RAW > /tmp/magic_l_prores/list_RAW_00
echo > /tmp/magic_l_prores/RAW_demolish
open ~/Library/Services/MLP.workflow/Contents/bmcc_ixml_luts/MOV_set_output.command
while ls /tmp/magic_l_prores/RAW_demolish
do sleep 1
done
if ! ls /tmp/RAW
then
exit 0
fi
if grep '01_legacy_RAW_to_MLV' /tmp/RAW
then
while grep 'RAW' /tmp/magic_l_prores/list_RAW_00
do
~/Library/Services/MLP.workflow/Contents/raw2dng "$(cat /tmp/magic_l_prores/list_RAW_00 | awk 'FNR == 1 {print; }')" --mlv $(ls A_lut_hold/*.MLV | head -1)
echo "$(tail -n +2 /tmp/magic_l_prores/list_RAW_00)" > /tmp/magic_l_prores/list_RAW_00
done
exit 0
fi
fi