Topics

[Make a feature request as an issue on GitHub]

Hello!

As I was developing color management for convmlv, I ran into a lack a lack of free/OS tools to handle and apply LUTs (especially 1D LUTs), Color Matrices (for gamut transforms), and just simple gamma functions, to/from production image formats (especially 16-bit EXR files). I will be updating convmlv to use this soon.

Essentially, this solves that problem as a Python 3.5+ library for practical color management ("apply this LUT to this image and save it!"). Take a look at the feature list!

A big thanks to @Danne and @bouncyball for testing & feedback !

Download

Download the zip from GitHub: https://github.com/so-rose/openlut

Features

Code Select Expand


* Functional/OOP design scales from simple one-liners to complex systems.
* Load & save images in countless formats, including EXR and DPX, at up to 16 bits per pixel. The guts (numpy array) are easily accessible.
* Create 1D LUTs easily, from arbitrary contrast functions, arbitrary mappings, and more.
* Load and save 1D LUTs in the .cube format.
* Create, compose, invert, combine, and apply color matrices easily for gamut transformations.
* Apply any Transform, including LUTs, Matrices, and perfect Gamma functions, to any loaded image with ease (the .apply() method).
* Batteries included - comes with an image viewer, functions for sRGB, sLog2, Reinhard, Rec709, and more, as well as matrices for XYZ, sRGB, and Adobe RGB.
* Fast - transforms themselves are done in parallel C++ bound to Python.


More to come! Make a feature request as an issue on GitHub.

Info

See the GitHub page for the latest info on dependencies and usage. I'd rather just maintain that than also maintain this post .

I suggest looking at the test code at the bottom of the GitHub page.

If you need 3D LUT support today, take a look at pylut. It should fit your needs!

Plans

• 3D LUT support. A hard thing, but it would be a good thing!
• CLI Script. For those who don't know Python!

See Issues in GitHub for a more comprehensive list.

Feedback
I'd appreciate any feedback and testing (especially bugs)! I'm very happy to try feature requests too (within time & reason, naturally).

[Huge thanks]

Hello,

This is, simply enough, an offline image processing tool converting ML formats (RAW, MLV, DNG sequences) to many usable formats (EXR, DPX, and Prores). It runs on Linux and Mac, and focuses on features and output quality (More in the Features section).

It's essentially a huge swath of glue between a bunch of great tools.

I use it myself - these two short films were developed, graded, and edited in a single weekend, made possible due to convmlv (v1.7)!

http://youtu.be/yi-G7sXHB1M
http://youtu.be/yi-G7sXHB1M

Huge thanks to @Danne and @bouncyball for figuring out the color management. Thanks to @dfort for his badpixels script and to @dmilligan for MLVFS! Another huge thanks to the whole Magic Lantern community for the tools without which nothing here would be possible

Download/Install

Getting convmlv on your machine goes like this:

• Download the release tarball for your platform and extract it.
• Optional: Download the .sig file and verify the tarball integrity (so you know that nobody has modified the software from me to you)
• Install dependencies - the bundled docs.pdf.

Latest Release: https://github.com/so-rose/convmlv/releases/tag/v2.0.3 <-- Download here!
Repository: https://git.sofusrose.com/so-rose/convmlv <-- The cutting edge!
GitHub Mirror: https://github.com/so-rose/convmlv <-- A simple mirror to GitHub.

Dependencies on other ML tools are bundled in the release. Refer to the help page (under MANPAGE, or run './convmlv.sh -h') for links to these.

Make sure all binaries, and the script, have execution permissions (run 'chmod +x file'), otherwise convmlv will fail! It will tell you what it's missing & where to get it if you try!

General Info

Documentation: You can find an up to date tutorial/documentation PDF I made under 'docs/docs.pdf' in the repository. It's also bundled in the release.

Here's the link to it: https://git.sofusrose.com/so-rose/convmlv/raw/master/docs/docs.pdf?inline=false


v2.0.3: Some more bug fixes, based on a bug report.
*Fixed bug with split MLV files by piping the FRAMES output through "paste -s -d+ - | bc"
*Fixed the color-ext LUTs, which were unfixed due to laziness.
*Fixed mlv2badpixels.sh when mlv_dump wasn't on the PATH. Requred modding the script; a patch is in the main repo, and is distributed alongside binaries.
*Added documentation that symbolic linking support is required.

Full changelog can be found in the repository, under CHANGELOG.


Bad Pixels File Example: You can find a sample .badpixels file in the repository (the Download link), which you can adapt to remove bad pixels (which all cameras have) from your footage.

Darkframe Information: Read more about darkframe subtraction here: http://magiclantern.fm/forum/index.php?topic=15801.msg164620#msg164620

Config File Examples: You can find example config files in the download repository, under configs/*.conf.

All command line options must go before the list of MLV/RAW files or DNG folders.

Features
convmlv is essentially a big piece of interface glue between many great image processing tools, whose features it in many cases inherits directly!

Code Select Expand


-Easy usage w/good defaults - specify -m, follow with a list of .MLV files to develop. (Type in terminal: convmlv -m *.MLV)
-Create ready to edit image sequences/video files in all kinds of losslessly compressed formats.
-Offline image quality with good, highly multithreaded performance.
-Develop a specific frame range easily. MLRawViewer is a great companion to find desired frame ranges.
-Complete control over the RAW process itself: Highlight reconstruction, demosaicing, color space, chroma smoothing, white balance, etc. .
-Color managed, with a variety of color space options. The philosophy is one of no quality loss from sensor --> output.
-Several noise reduction techniques, from wavelet to high quality temporal-spatial denoising, to astro derived darkframe subtraction, to experimental FFMPEG modules.
-Easy HDR (Dual ISO) processing, with the -u option.
-Easy bad pixel removal. The -b option (courtesy of @dfort) removes pink dots, and can be combined with our own .badpixels file mapping out the dead pixels on your camera.
-Since the output can be very heavy to edit with, it's simple to create edit-friendly color managed JPG/MP4 proxies alongside.
-Several FFMPEG filters (multiple 3D LUTs, temporal denoising, hqdn3d, removegrain, unsharp, and deshake currently - request more; they're very easy to implement).
-Reads Camera WB, but can also use a homegrown AWB algorithm or no WB at all.
-Extracts MLV sound to to a WAV, and metadata into a settings.txt file.
-Portable Bash script - Fully compatible with Linux and Mac (Windows is untested)
-Production-ready config file format lets you specify any option on a global or per-MLV basis in a single config, saving enormous amounts of time on a deadline.


Documentation/How To

Tutorial: You can find an up to date tutorial/documentation PDF under 'docs/docs.pdf' in the repository. It's also bundled in the release.

Here's a link to it: https://git.sofusrose.com/so-rose/convmlv/raw/master/docs/docs.pdf?inline=false

Help Page: The primary, most esoteric (but most updated) documentation is the help page. You can access it in docs/MANPAGE in the repo, partially in the code block below, or by typing 'convmlv -h' once it's installed.

Human Help: I tried to cover everything with the above, but if you're still having trouble I'm happy to help. Just respond to the thread or send me a PM somewhere.


Workflow

A typical task is to convert an MLV file to a high quality MOV file for editing. Doing so with convmlv is quite simple:

Code Select Expand

convmlv -m cool.MLV

Simple as that! My personal preference is to edit using losslessly compressed EXR sequences, like so:

Code Select Expand

convmlv -i cool.MLV

Of course, you have a very wide range of features available to aid you in this process. Here are a couple

Code Select Expand

convmlv -i -m -p 3 -C config.conf -s 25% -b -k -d 3 -g 3 -G 2 -o ./output test.MLV test2.MLV cool.RAW

I'll break this specific command apart:

• -i: We will output a full-quality EXR sequence.
• -m: We will also output a high-quality Prores 444 file.
• -p 3: This proxy mode will generate JPG sequence and MP4 (H.264) proxies.
• -C: We'll use a local config file, which lets us automatically specify options without having to type them out each time in the command.
• -s 25%: Our proxies will be 25% the size of the original video.
• -b: We'll remove any egregious focus pixel issues.
• -k: We will also output undeveloped DNG files.
• -d 3: Our demosaic mode (a big part of why we shoot RAW) will be the high quality, but slower, AHD mode.
• -g 0: My output file will have Standard Gamma (close to 2.2; gamut dependent. In this case, a Rec709 Gamma).
• -G 2: My output file will have a Rec.709 Gamut.
  
• -o ./output: We'll place an additional folder per converted shot/file inside of the folder ./output.
• The Rest: These are all the shots/files (.MLVs, .RAWs, or a folder containing .DNGs) to develop. This works with bash wildcards; for example, to convert all MLV files in a directory, simply use *.MLV.

You can of course go crazy. This is a valid command:

Code Select Expand

convmlv -i -t 2 -p 2 -s 75% -r s-e -d 3 -f -H 2 -c 2 -n 50 -g 3 -w 0 -S 9000 --white-speed 5 -a ../../7D_badpixels.txt --threads 36 --uncompress test.MLV

Avoiding commands like this is the point of Config Files. Configs can specify options globally, but can even do different options depending on the file name!


Bugs
Known Bugs: See Issues in the Repository.

If possible, please report bugs as Issues in the main Repository (GitLab), instead of GitHub. I don't mind if you do it in GitHub; just, the other way makes things easier for me.

Future Ideas
Please see (or make) an Issue in the Repository!

Feedback
I'd appreciate any feedback (especially bugs)!