MLP - Mac OSX batch processing workflow (former cr2hdr-r)

Started by Danne, October 05, 2014, 04:09:34 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dfort

Mind if I post a feature request?

Rescale image.

Except for maybe the 5D3, most cameras can't shoot 1920x1080 raw video. It is best to do the resizing before creating the ProRes movie file. Then of course there are several scaling algorithms to choose from. It looks like ffmpeg is fully capable of resizing and it also has several scaler options.

Your source code hinted at this possibility:
#FFmpeg used to transcode to ProRes444 with or without audio. FFmpeg will also scale certain kind of stretched footage to get a normal looking file.
#commands/effects used
#-vf scale=iw:ih*1.66

but how to implement rescaling options without overly complicating it? How about the same way you're handling dcraw settings by putting the command line options in text files that you can drop into the lut_hold folder. There are a lot of possible combinations of height x width options when recording MLV. Some need to be stretched or squeezed to create a normal looking image and many won't fit neatly into a 16x9 frame. Add in the rescaling algorithm choices and there's a mind boggling amount of options. Of course you can't include all the options but maybe some of the more popular choices? I'd really like to be able to take MLV files shot at 1280x720 and batch convert them to 1920x1080 Log-C ProRes 444.

As long as we're on the subject of rescaling, dare I mention 4K? Sure, you can't get more resolution when you upscale a single image but what using multiple images? I played around with image stacking with video footage and though I haven't figured it all out yet there may be some way to create a script that will make this practical. One idea is vary the the depth of the image stack depending on how many images can be stacked before it is no longer possible to align the images.

Danne

Been thinking about scaling options. Placing them as droplets in lut_hold folder is possible and probably the best idea. Is 1920x1080 the only wish or could there be more scaling options?
By the way, I skipped all stretching x1.66 in folder_prores444_lut since dcraw reads the default scale tag implemented by dmilligan in mlvfs. Really cool.
About the 4k things I have no idea how to do that right now. What software is used right now? Are you working through command line?

dfort

The image stacking experiments I did with Photoshop. No big deal doing it for still photographs but video would require setting up a rather elaborate action or writing a Photoshop scipt. Your method of using freely available command line tools like dcraw, enfuse, ffmpeg, etc. along with some shell scripts to automate the process is also a possibility.

For regular scaling I would think that getting the image to fit in a 1920x1080 frame would be the best place to start because it is a well established standard. Sure, some users like working with ratios other than 16x9 but that could be done with custom settings instead of trying to come up with all the possible rescale options.

Is is very cool that MLVFS takes care of any necessary stretching so you can just concentrate on rescaling to fit the frame. Maybe just crop to fit or letterbox to fit?

Danne

I think something along the way like -vf scale=1920:-1 Should keep a constant 1920 and the height will follow accordingly and leave the footage unstretched.
It would also be possible to change the aspect ratio straight in the text document which would leave it up to user to write whatever aspect ratio wanted.

Scaling techniques here
https://trac.ffmpeg.org/wiki/Scaling%20(resizing)%20with%20ffmpeg

dfort

How do you set the scaler flags?

Looks like there's a lot there to experiment with to try and find the best quality rescale.

sws_flags
Set the scaler flags. This is also used to set the scaling algorithm. Only a single algorithm should be selected.

It accepts the following values:

'fast_bilinear'
Select fast bilinear scaling algorithm.

'bilinear'
Select bilinear scaling algorithm.

'bicubic'
Select bicubic scaling algorithm.

'experimental'
Select experimental scaling algorithm.

'neighbor'
Select nearest neighbor rescaling algorithm.

'area'
Select averaging area rescaling algorithm.

'bicublin'
Select bicubic scaling algorithm for the luma component, bilinear for chroma components.

'gauss'
Select Gaussian rescaling algorithm.

'sinc'
Select sinc rescaling algorithm.

'lanczos'
Select Lanczos rescaling algorithm.

'spline'
Select natural bicubic spline rescaling algorithm.

'print_info'
Enable printing/debug logging.

'accurate_rnd'
Enable accurate rounding.

'full_chroma_int'
Enable full chroma interpolation.

'full_chroma_inp'
Select full chroma input.

'bitexact'
Enable bitexact output.

Danne

Good question. I uploade a version which simply scales without any scaler flags. Not sure what the default scaler is set to.
You can write straight to the text file to change settings.
folder_Prores444_lut, top of first page.

Danne

Updated the download again with more info on how to choose between scaler options. Double click scale.txt to read more. This document is also the one you put in the lut_hold folder.

In short. This is base nside the document scale=1920:-1, If you want to change scaler flag it should look like this (example) scale=1920:-1:flags=bicubic,
Simply change the first in the document and put it in lut_hold folder. Somebody mentioned "spline" as a good scaler, scale=1920:-1:flags=spline,

dfort

Well that was quick. Only thing is, I can't find the scale.txt in the latest download.

[EDIT: Tried it again and now it is there. Maybe I didn't blow off the old PRORES directory? The scale.txt script is very well commented. Thanks!]

Danne

Thanks for the suggestion. Good to have it. Once the dan_log folder or PRORES folder is erased the latest one will be copied to the prores folder with the scale.txt file.
I also must rename or exclude the folder_ProRes444_lut from cr2hdr-r since it, s not updated like the stand alone version.

dfort

Only thing is, I can't seem to get it to rescale. Tried it with a lut and tried scale.txt by itself but the size of the ProRes file stays the same as shot. Am I missing something?

Danne

Strange. Worked here when I tried yesterday. The scale.txt is in the lut_hold folder?

dfort

It was me.

I've too many services and ran folder_ProRes444 instead of folder_ProRes444_lut the second time. Maybe I should remove folder_ProRes? So is this going to replace cr2hdr-r?

DeafEyeJedi

Gonna give this a try once my MBP is finished updating to El Captain...

Stand by!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

You can skp the folder_ProRes444 and keep the one with the lut at the end. It can,t replace cr2hdr-r since this is only working on folders with dng files. However I had plans on including folder_ProRes444_lut to cr2hdr-r as well but I think I,ll skip that part. I,m about to put up a new cr2hdr-r with some fixes.

Danne

I updated first post with version cr2hdr-r 12.0_align

- Included install procedure when using El Capitan.

Will have to do for now since I couldn,t get the migration to usr/local/align to work properly. Procedures are all provided in the install folder. Maybe if I get the the time I give the migration thingy another go with dfort :).

- Dual iso CR2 will now obtain the correct white balance values.

This is working through getting the multipliers from dcraw and then recalculated to fit AsShotNeutral values.
Thanks Chmee, dmilligan, A1ex and others discussing and solving problems related to white balance.
More info here
http://www.magiclantern.fm/forum/index.php?topic=16024.msg155670;topicseen#msg155670

DeafEyeJedi

Thanks, D!

Running El Captain and your download link for cr2hdr-r 12.0 from OP seems broken?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne


rullebullefrulle

@Danne

If I've got "creeping" in the image what am I doing wrong? Don't know what it's called but it's as if there are loads of pixels in the image that are moving / changing exposure fast. 

Firmware: 1.2.3
Canon 5D Mark iii

Modules in ML:
Raw Video (mlv) 1920x1080
Dual iso 100 / 800

Video settings in camera:
Manual mode 
Shutter 1/50
(AWB or set WB gives the same result)
FPS: 23.976

Workflow:
Record -> transfer through kingston 3.0 usb card reader -> OSX.
Edit with your program cr2hdr-r (put MLVs in folder and run the program, yields folders with dngs and a wav)
DNGs are first created, and then re-created (for the dual iso conversion I assume).

When I edit these files in Davinci Resolve 11 the creeping is present. If I record under same circumstances but with standard RAW (not dual iso) then there is no creeping in the image.

What gives?

Danne

Hmm, could you upload dng example? Tried recording with global draw off?

Danne

Thanks dfort for /usr/bin/csrutil status. Updated install script in cr2hdr-r to check if SIP is enabled. Also added some previewing changes in folder_ProRes444_lut and some fixes related to batch_lut folder.

Danne

@rullebullefrulle.
I checked your files you sent and I say that the creeping is normal when shooting dual iso in non crop mode. To get the best result in dual iso it,s recommended to shoot in 3x crop mode. Examples here
http://magiclantern.fm/forum/index.php?topic=12796.msg125268#msg125268

rullebullefrulle

@danne

thanks for pointing me in the right direction. I found this video to be quite informative for those stumbling on the thread and don't understand what 3x crop means:

http://www.magiclantern.fm/forum/index.php?topic=6039.0

Would you all be interested in a sample video if I would shoot?

ciriuss925

Hello forum,

Yesterday I took a few DUAL-ISO shots and was trying to run cr2hdr-r to convert the dualiso files to DNG, but nothing happened.  its been months since I tried to convert Dual ISO raw images so I decided to go with the latest cr2hdr-r version (cr2hdr_12_3_align) and downloaded it.

Once downloaded, I first ran the install_uninstall.command  (I assume - and hope - that this command would remove the contents installed by the old cr2hdr-r versions?) - as expected it prompted for my password and I entered it, and the command ended with [Process completed].

Then i created a folder called "ML NOV 15" - and ran the cr2hdr-r.app - it proceeded to create those folders (HDR, PRORES, ORIGINAL) etc under the new folder.   I copied the DUAL ISO raw images into the  "ML NOV 2015" folder.     Then ran cr2hdr-r again.   It only moves  the CR2 files into the ORIGINAL folder - I didn't see DNGs created.

I also saw your DUALISO_TO_DNG app (which seems to be an easier app to use) - followed the steps from youtube https://www.youtube.com/watch?v=Q48HhBDH4Z4&feature=youtu.be ...   i created a new folder "DUAL ISO CR2" - right click on services and selected "dualiso_to_DNG".  It created   a "commands_list.txt" file and a new "ORIGINAL" folder, copied the CR2 files to the "ORIGINAL" folder and... there is no DNG file in sight :(    i tried to reboot my machine and re-do the steps - still no dice :(

Did i miss anything?  Your inputs are appreciated.

Danne


ciriuss925

Hi Danne, I am still on Yosemite.

i am wondering that maybe because there's "way to many versions" of cr2hdr-r that i downloaded and installed over past few months may have messed something up?

Oh btw I forgot to mention that when I ran the cr2hdr-r and pointed it to the folder with my CR2 (there are only two CR2 files) - it just moved one CR2 , and i had to run cr2hdr-r one more time for it to move the 2nd CR2 to the ORIGINAL folder.