OpenFX MLV file Reader for Natron Compositor

[cedricp]

I just released an OpenFX plugin to read MLV files with Natron compositor (https://natrongithub.github.io/). I spent some days to make it fast (multiprocessing, optimizations, etc.), currently, I can playback the timeline at about 20 FPS (HD resolution).
There are some options for raw processing (like in MLVApp) and some options for output colorspaces. I used the AMPAS/rawtoaces method to convert RGB data to a "good" ACES IDT.
You can test it by downloading a zipped pre release here : https://github.com/cedricp/openfx-CMS/tags
Just copy the content of the ZIP in the Plugin/OFX folder of Natron application and voila.
you can also install focus pixel maps, just create a new directory in the Content plugin folder copy them inthere.
The plugin also include some nodes to bake 3D LUTs, I'll maybe explain how it works later.

Hope you'll like it 

[Walter Schulz]

Thanks for your work and contribution!

Q: Dual-ISO support?

[cedricp]

Yes, on the way !

[Walter Schulz]

Nice!

May I ask you to change subject line?
Title is quite generic and that's not user friendly (IMO). Title should make transparent that you created an import plug-in for MLV footage for OpenFX.

I already moved your post to "Raw Video Postprocessing".

[cedricp]

Done

[Walter Schulz]

Thank you!

I think you are aware that Dual-ISO processing is hampered by design. In its current implementation it works with a single thread and this is not only a shame but slows down processing big time!

If (if!) someone finds a way to make it multithreaded and it - cream on top - is portable ... big win!

This is much to ask for and I don't: If you go with "plain" Dual-ISO processing it would be great anyway! ;-)

[cedricp]

Just finished dual ISO support, it works in multithreaded environement, but several frames in parallel, no MT support for 1 frame (too complicated for me  ), this is a good improvement, though !

EDIT : I can get 4-6 FPS with a EOSM 4.2K footage with dual ISO enabled. (Natron supports disk caching, that can help a lot)

[iaburn]

Wooow!!!!!! Awesome work!   
Double thank you, first for the plugin and second for introducing this software, it's the first time I hear about it :O

It took me a while just to get the MLV on the screen and export the video, but now I start to love it!
The focus pixels fix works (very important), and it seems to have a lot of options.
I tried to play a 12 seconds (300 frames) 2160x1620 clip with focus pixels fix, and the first time playing it was jumping, but after it's cached, it plays easy at real time speed.

On a Ryzen 5 3600 + Nvidia RTX 3070 + 16GB DDR4 3000Mhz, the export to mp4 h264 high quality at original resolution took 77 seconds on MLVApp, with only rec709 profile applied and focus pixels fix.
Exporting from Natron with your widget with the same configuration + highlight processing took just 40 seconds, so significantly faster. (Export format and resolution seems to not have any effect, so it's limited by the MLV decoding speed)

I need to play more with this software, but it looks very promising, thank you

[iaburn]

I don't have the tools to compile right now, but I will test the Dual ISO on your next release

[cedricp]

Thank you for your feedback, dual ISO build is on the release GitHub page, have a nice weekend !

[iaburn]

Thank you for your feedback, dual ISO build is on the release GitHub page, have a nice weekend !

Thanks, just gave it a try with a Dual ISO 2520x1054 clip, 142 frames and got mixed results, but potentially very good 

Exporting the 142 frames with Dual ISO corrected, linear debayer and rec709 profile, took only about 12 seconds, WOW! That's when the clip is not cached, if it's cached, it's almost instant :O

Now the funny thing: on MLVApp, activating the "Chroma Smoothing" has basically no cost, but activating the Chroma Smoothing on the plugin to 5x5 to get rid of the color artifacts on Dual ISO, takes the exporting time from 12 seconds to around 210 seconds. That's exponentially slower, there must be something wrong there 

For reference, in MLVApp the export with same Dual ISO settings takes around 97 seconds (much slower), but enabling or disabling Chroma Smoothing has no impact.

Something else that I've noticed is that a simple "grade" node has a significant impact on performance, between 10 and 20%, which I would expect to have almost none. Not sure if this is related to the plugin or not, though...

As I said, it seems to have a lot of potential

[cedricp]

That should be a bug, indeed, if I recall correctly, chroma smoothing should be bypassed with dual iso enabled.
I will check that later.

[cedricp]

Oh, and most nodes on natron are CPU computed. I generally create a 3d LUT of my grade to speed up the export process. Look at disk cache node in natron, it helps a lot when correctly used.

[cedricp]

That should be a bug, indeed, if I recall correctly, chroma smoothing should be bypassed with dual iso enabled.
I will check that later.

Yes I checked that and chroma smoothing is disabled (not in preview mode) when dual ISO is on (this is normal, we don't want to filter an image with different exposures on every lines)

BTW, I've nearly finished a new version with GPU accelerated demosaic program borrowed from darktable), it's a bit faster.
That was a great experience, though ! I had to do all the work libRaw does... camera matrix conversions, work with OpenCL, etc.

[cedricp]

Binaries for testing : https://github.com/cedricp/openfx-CMS/releases/tag/OpenCL-preview

[Danne]

Yes I checked that and chroma smoothing is disabled (not in preview mode) when dual ISO is on (this is normal, we don't want to filter an image with different exposures on every lines)

BTW, I've nearly finished a new version with GPU accelerated demosaic program borrowed from darktable), it's a bit faster.
That was a great experience, though ! I had to do all the work libRaw does... camera matrix conversions, work with OpenCL, etc.

Interesting. Will test on occasion. Thanks for contribution.

[iaburn]

BTW, I've nearly finished a new version with GPU accelerated demosaic program borrowed from darktable), it's a bit faster.

It's A LOT faster! On a RTX 3070 it plays realtime and the green line indicating cache progress is advancing even faster, really cool.
I guess you know already, but when activating OpenCL the image is much darker and highlight processing is disabled.

A bigger issue is that with the new version, as soon as I add a ColorLookup or a Grade node to the tree, the exported video from the Write node is black, sometimes with a few correct frames only.
It's very fast, but it's not really working, I guess that's why it's fast xD
I've tried several combinations (with and without OpenCL, different debayer, etc.) it just works when I have no other nodes between the MLVReader and the Write nodes.
Here is my example with a 2 seconds MLV, if you want to try and see if you have the same problem:
https://mega.nz/file/wtkXkBCZ#P77T999siW7eGsK3vbsxXQe_aDB7ITv8ojGf0ZAtIvQ

[cedricp]

I had to swicth from RGB to RGBA (for performance issue) and indeed Natron is weird when alpha channel is there. I generally uncheck "premult" in nodes to prevent this to happen, I'll try to make it better next time

Edit : you can also insert a "shuffle" node after the read node to suppress the Alpha component.
I'll try to add a highlight reconstruction filter later...

[cedricp]

A bigger issue is that with the new version, as soon as I add a ColorLookup or a Grade node to the tree, the exported video from the Write node is black, sometimes with a few correct frames only.

Found why ! The image has some values > 1 and the colorlookup node doesn't like it at all ! Try to stay in 0-1 range and it fast again !!!

[cedricp]

Found why the render is slow with a color grade node ! The image has some values > 1 and the colorlookup node doesn't like it at all ! Try to stay in 0-1 range and it fast again !!

[iaburn]

Found why the render is slow with a color grade node ! The image has some values > 1 and the colorlookup node doesn't like it at all ! Try to stay in 0-1 range and it fast again !!

Good catch! This software is definitely not noob friendly... you need to know it from inside out 

[names_are_hard]

This inspired me to have a bit of a look around, and I found this fairly nice looking repo: <https://github.com/cheind/pytorch-debayer>

MIT licensed, choice of different debayer algs, pytorch based so it runs on GPU or CPU.  QT has a C++ torch binding so porting it to mlvapp might not be too hard.