Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r) (Read 500542 times)

Danne · « **Reply #700 on:** May 10, 2016, 06:54:10 PM »

Been busy scripting a CR2 still image HDR workflow. Since magic lantern has a really good hdr bracket function and MLP already works with enfuse I might as well put in something that works with that. Put in the CR2_HDR_enfuse.txt file in A_lut_hold to activate the menu below. Parent folder must contain CR2 files. Some more info in p4 in the user_guide. Processing is multithreaded.

Danne · « **Reply #701 on:** May 23, 2016, 10:37:05 AM »

New version in first post

This one goes out to bouncyball. The guy that reworked raw2dng so that we can convert RAW to MLV. On a sidenote he also solved the spanned file problem in raw2dng.
http://www.magiclantern.fm/forum/index.php?topic=17185.0
After some discussion I pointed out a few things I always wanted to fix in MLP regarding mlv_dump and to make this happen there is no better branch than the work from dmilligan here https://bitbucket.org/dmilligan/magic-lantern/branch/ml_dng
There is actually quite a few things that dmilligan shares that I personally thinks deserves a lot more attention.

Ok, so what happened is that bouncyball reworked a lot of things in the ml_dng branch that was either missing or was somehow problematic and put the code up here.
https://bitbucket.org/bouncyball/ml_dng-branch/src/cca020d2d94ec0bb1bd082f6189227ec36a1c229/?at=ml_dng

Following is now implemented and tested working in mlv_dump coming from bouncyball ml_dng branch/fork?
-   White balance handling(same as mlvfs – ufraw matrix math) All modes working except auto white balance in camera)
-   Default scale tag(automated)
-   Timecodes(as fixed in mlvfs. Chmee code originally?)
-   Removed forward matrices(my wish, not cdng standard)
-   Added focal resolution tags

Maybe I missed something here. I fill up with information as developing evolves.
The reworked mlv_dump together with a speedup in dng developing(converting 4 chunks in parallell, works since you can select frame range in mlv_dump, thanks g3gg0 – Thanks so-rose for pointing out.) will make developing dng files a lot faster. No need depending on exiftool and exiv2(actually very fast). All metadata handling is baked from within the mlv_dump binary.

Big thanks bouncyball listening, continuing developing, refining dmilligan´s stellar work in ml_dng branch. As always g3gg0, A1ex for mlv_dump, Chmee for a lot of early work around matrices, timecodes etc, and of course dmilligan for groundbreaking work in mlv_dump. Please tell if I missed thanking somebody here.

DeafEyeJedi · « **Reply #702 on:** May 23, 2016, 06:32:55 PM »

Daaaaaaammmmmnn @Danne's at it again. This time it's no turbo nor a supercharger. It's a freakin' 74-Z Military Speeder. So quick 'n dangerous like laser speeds! Sings beautifully on the Mac's while running MLP as usual. It's just even more pure Magic!

Funny how when one experiment actually solves one another and remarkably solves a lot of things overall together. That's the beauty of collaborations from certain people on this Forum. I am not even part of that crew.

This is beyond legacy stuff and much credit goes to you for your never ending work on this beautiful script in MLP which seems to get the most Updates (frequently) out of most if not all apps out there. For good reasons. He obviously cares for those that uses his fine work along with other amazing apps/converters as well out there on this forum. Thanks again Danne for yet another Christmas gift for all of us.

Remember none of this would have been possible without the amazing discoveries that @bouncyball recently made for us and has contributed so much in so little time. Keep on rolling, Buddy!

Special Thanks also goes out to @dmilligan, @g3gg0, @a1ex, @dfort, @Chmee, @so-rose & those that have contributed to keep this community well alive and always reviving.

Off I go into the speed comparisons experiment between the older and newer versions of MLP and will report the results as soon as I can. Fasten seat belts and best be sure to hold on to these speedy 74-Z's!

Danne · « **Reply #703 on:** June 04, 2016, 10:41:13 AM »

Some findings about the format EXR started out as a side topic so I grabbed the information and put it up here instead.

Seems photoshop is pushing exr files to gamma 2.2 when it should be linear 1.0. DaVinci Resolve is respecting linear output for exr format.
A "fix" is to set midpoint to 0.45 to get back to linear in PS.

http://forums.cgarchitect.com/80010-saving-exr-gamma-settings.html

Quote

If you were comping your EXR's you'd want them to remain linear, and as such they shall appear too dark. But you want to do the maths on them like this. To see what you are getting as a result, you would put a gamma of 2.2 on top of the comp. In PS I would do this with a levels adjustment layer at the top of the stack. The mid point is set to zero by default, but you can type in 2.2 (or 0.45 if required) to adjust the gamma output.

If you're not concerned with linear compositing, it's still useful to know.

Closer

Photoshop

DaVinci Resolve

Some more stuff in AE. By selecting Preserve RGB(interpret footage) the output seems linear.

Changing color settings in AE as well

Pretty much matches DaVinci Resolve output
After effects

Davinci Resolve

Set to camera RGB profile seems even closer

In photoshop you can set RGB to convert to actual rgb and it will show the correct linear gamma. But we,re suddenly in RGB and not sRGB color space right now, hmm

Answer from Andy600

Quote

@Danne - Instead of setting gamma to .45 try the inverse of the sRGB transfer function (as a lut). I'll bet it matches precisely.

My answer

Quote

Not sure how to create inverse luts but I added the sRGB to linear lut from VFX IO(Davinci Resolve) lut to the clip in AE and I assume it,s the same thing? Looks right to me

Andy600 · « **Reply #704 on:** June 04, 2016, 10:54:56 AM »

@Danne - The Resolve 'sRGB to Linear' lut is fine for this but I personally wouldn't set the workspace to Camera RGB ICC because it's non-standard. Try it with the sRGB or Rec709 ICC profile and a grayscale chart image if possible. You can't know for sure what is happening without some kind of reference.

Danne · « **Reply #705 on:** June 04, 2016, 11:03:43 AM »

I restarted AE so I believe it choose the sRGB colorspace(EXR colorspace) as default. Here it is again with color settings actively set to sRGB.
I checked how to get into linear gamma in photoshop too and they have some "actual RGB" setting which overrides the sRGB setting in the exr file but that seems plain wrong to do? Applying the sRGB to linear lut for gamma applied to the EXR sRGB colorspaced image must be the better approach. What do you say?

AE color settings set to sRGB

Andy600 · « **Reply #706 on:** June 04, 2016, 01:49:45 PM »

An EXR workflow should use a linear float workspace so AE's workspaces ICC should be linear too. Looking at your AE settings you need to click 'Linearize working space' before doing anything else or you can end up with double gamma issues which may be why the image looks like it does i.e. it looks almost log to me - there may be an additional gamma compound issue to deal with.

Assuming the image was produced in MLP, have you tried creating an EXR in Resolve using the original raw i.e. without MLP? How different is it to MLP?

Danne · « **Reply #707 on:** June 04, 2016, 03:14:19 PM »

Exr from DaVinci Resolve produces the same results when previewed on mac or imported into PS or AE(overbrights) Will upload comparison files later.
Double linear gamma happens in DaVinci when selecting -colorspace RGB in MLP(imagemagick). Or would this actually be the linear file we,re looking for?

Andy600 · « **Reply #708 on:** June 04, 2016, 03:53:41 PM »

I would trust Resolve's interpretation above ImageMagik because, as you say, it produces the same results in AE and PS but also in Nuke and Baselight. If AE color management is set up correctly it will produce EXR's the same as Resolve. You should aim at getting ImageMagik to match Resolve EXR but that may involve tweaking the source code - I don't know ImageMagik well enough and have only used it a couple of times for calibration/testing.

Danne · « **Reply #709 on:** June 04, 2016, 04:07:55 PM »

Sorry if I was unclear. Both Imagemagick and Resolve output delivers overbright gamma in preview and photoshop.
The really hard part here is to understand where in the pipe things goes wrong. I don,t even know what is supposed to be the correct starting point here

Please have alook at these two examples from Davinci resolve and image_magick.
DR(half float)
https://drive.google.com/file/d/0B4tCJMlOYfirbkxXLWpCdGZLaVk/view?usp=sharing

IM
https://drive.google.com/file/d/0B4tCJMlOYfirVkl3MlZpVW5RSUU/view?usp=sharing

DeafEyeJedi · « **Reply #710 on:** June 13, 2016, 08:40:33 PM »

Sorry @Danne to be a party pooper but I tried to download latest MLP from OP and seems the link is dead?

On both Chrome and Safari I've been getting this...

Anxious to try out the latest FFplay menu features ... Thanks, D!

Danne · « **Reply #711 on:** June 13, 2016, 08:46:29 PM »

Seems to work over here?

DeafEyeJedi · « **Reply #712 on:** June 13, 2016, 08:52:35 PM »

Actually just tried it on my iPhone's safari which does show the latest MLP (updated 19 hours ago) so I'll restart my MBP and try again. Sorry for the false alarm, D.

Danne · « **Reply #713 on:** June 14, 2016, 10:17:06 PM »

New upload in first post.

There is still things to develop around HDR
http://www.magiclantern.fm/forum/index.php?topic=17334.msg168376#msg168376

By duplicating an image sequence one can obtain the original framerate instead of having HDR halfen fps due to fusing images. Instead of going A+B, A+B, A+B you can do consecutive blending by fusing A+B, B+A, A+B and so on.

To activate correct framerate in HDR processing add HDR_enfuse_correct_FPS.txt into the A_lut_hold folder. This works automated both with dng and H.264 movie processing.
Now this workflow is already working with ffmpeg tblend average filter however with enfuse you also have the possibilty to work with aligning via Hugin.

Also added in MLP is a small menu helper when right clicking folders with dng files which activates FFplay previewer.

DeafEyeJedi · « **Reply #714 on:** June 16, 2016, 12:34:52 AM »

Thanks for the update, D!

Here are my findings when using H264 HDR footage...

Is this normal for the a_lut_hold to be a File rather than a Folder? Because when I go into the dcraw_FFmpeg_settings Folder to pick out a few HDR features which then won't go into the a_lut_hold because it's not a Folder? Should I delete that 'file' and create a new Folder with the same name?

Danne · « **Reply #715 on:** June 16, 2016, 06:24:24 AM »

You should select 'q' exit and quit from the menu if you are working from the same folder as your files.
Recommend you to test some short 30 fps since the quality is better and aligning takes a while.
I have a new version coming up soon which fixes an interpolation issue creating flickery parts from enfuse.

Danne · « **Reply #716 on:** June 16, 2016, 07:31:27 AM »

New version uploaded which fixed align/enfuse issue by auto crop with hugin.
When running MLV/RAW files include following files in A_lut_hold folder
example:
HDR_enfuse_correct_FPS.txt
HDR_enfuse_straight_from_dng_settings.txt
HDR_MOV_disable_sharpening.txt

When running H.264 mov material include:
example:
HDR_enfuse_correct_FPS.txt
HDR_enfuse_settings.txt
HDR_MOV_disable_sharpening.txt

Try 24,25 or 30fps to see how it works. If you want fast enfusing add HDR_disable_aligning.txt only the ghosting will be severe. Remove the HDR_MOV_disable_sharpening.txt
if you want a slight sharpening added to your files.

Danne · « **Reply #717 on:** June 20, 2016, 12:56:49 PM »

Here,s a little something filmed in HDR iso 100-1600(H.264 MOV), 24 fps keeping the original framerate by copying and enfusing double amount of frames.
Following settings in A_lut_hold folder.
HDR_enfuse_correct_FPS.txt
HDR_enfuse_settings.txt
HDR_MOV_disable_sharpening.txt

aschille84 · « **Reply #718 on:** June 20, 2016, 01:58:10 PM »

Wow, this looks suprisingly good! None of the ghosting and jittering from my own tests a little while ago. Good job

Danne · « **Reply #719 on:** June 20, 2016, 02:24:54 PM »

Thanks. Did you try it out yourself with aligning function enabled? Could you share some results @aschille84?

aschille84 · « **Reply #720 on:** June 20, 2016, 02:53:59 PM »

I should definitely try it. Ill see what I am able to come up with in the coming weeks!

aschille84 · « **Reply #721 on:** June 21, 2016, 10:29:45 PM »

Ok, so I tried the enfuse-workflow with the newest version. And it works really well. On my 6D I would prefer working with HDR h264 over RAW any day. The only drawback is the rendering time, on my 2012 macbook pro, rendering 30sek HDR took like 25min.

here is an example:

https://www.dropbox.com/s/pbcpwh9woc0hdfj/HDR_MVI_8093.mov?dl=0

I was thinking if its possible to have some kind of proxy workflow.
Is the tblend average option faster?

Danne · « **Reply #722 on:** June 21, 2016, 10:38:46 PM »

I,ll check soon. Oh yes, the tblend filter is in realtime. Caveat you don,t have the aligning function so 50 or 60fps is recommended. I consider the tblend filter workflow the most useful actually.

Danne · « **Reply #723 on:** June 21, 2016, 10:57:30 PM »

Just checked you vid. ANd also the one you posted before here http://www.magiclantern.fm/forum/index.php?topic=17444.msg168711;topicseen#msg168711. Nice work.
Do try the tblend filter. I think it could be useful

Danne · « **Reply #724 on:** June 26, 2016, 12:42:10 PM »

Updated MLP so it works with the latest FFmpeg. Planning on maybe implementing the new atadenoiser filter which seems good.
Had to change some commands used in ffmpeg in older versions so when updating see to it that the latest ffmpeg is present in the Place_binaries_here folder before migrating binaries from the older version.
Latest FFmpeg can be found here
http://www.osxexperts.net/ffmpeg302osx108.dmg

Clarification
Old user(already MLP installed)
1 - Place the updated FFMpeg in Place_binaries_here folder(download package)
2 - Double click the move_binaries.command and follow instructions(say yes to both questions)

New user
1 - Follow the HOWTO.txt in the download package

News:

Author Topic: MLP - Mac OSX batch processing workflow (former cr2hdr-r) (Read 500542 times)