Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Topics - revast

#1
Raw Video Postprocessing / ML RAW-tools for Linux
November 15, 2013, 03:05:58 PM
ML RAW-tools & more how to deal with .raw/mlv & cDNG files in Linux

News
Updated on 05-2017




MLFVS (MLV files can now be simply mounted thanks to MLFVS. 64bit binary; thread; homepage; It's fast and efficient, real-time playback possible with fast storage. I made a Python-based MLVFS-AppIndicator, which allows to control MLVFS from systray)

Installation & Usage

QuoteA MLV "converter" that provides a virtual file system of converted data 'on the fly' using Filesystem in Userspace (FUSE). It allows you to "mount" an MLV file(s) which shows up as a directory of converted CDNGs. The data for these CDNGs is provided 'on the fly' by MLFVS as the data is being requested by whatever raw editor or post processing software you are using. It's fast and efficient, and allows for real-time playback.

QuoteAdvantages
- You don't have to convert first, you more or less "instantly" have "converted" DNGs
- You don't have to choose between keeping the original MLVs or converted DNGs or doubling disk usage to keep both.
- There's no need for a GUI, there already is one provided by the OS (i.e. the OS's file browser).
- Possible to vary the quality, pre-processing, bit depth, etc, of the DNGs on the fly, via webinterface.





convmlv (convmlv + required tools; thread; homepage, written in Bash. A featurefull converter from ML formats (RAW, MLV, DNG sequences) to many formats (EXR, DPX, 10bit Prores). You have to install libc6:i386 to get its dependencies run on 64bit systems.)

QuoteFeatures
-Bash script, based on mlv_dump, raw2dng, mlv2badpixels.sh
-Uses mlv/raw files as arguments
-Create image sequences/movies in EXR, DPX, Prores
-Develop specific frame range easily, making MLRawViewer an ideal companion
-Complete control over RAW development: Highlight reconstruction, demosaicing, color space, chroma smoothing, white balance...
-100% colormanaged, output gamma/gamut (including Cineon, S-Log3, DCI-P3, Rec709...), no quality loss from sensor --> output
-Create/use darkframe subtraction, temporal denoising, wavelet denoising for noise reduction
-Multithreaded
-HDR Dual ISO processing
-Easy bad pixel removal
-Autom. applies lossless compression
-Option to create jpg/mp4 proxies
-Apply ffmpeg-supported filters
-Optional Auto White Balance
-Sound is available as a .wav file
-Metadata dumped from MLV or DNG files to settings.txt or terminal
-Config file interface lets you automate all, saving time

documentation: convmlv -h, pdf

Usage (excerpt)
    -i: output full-quality EXR sequence
    -m: output high-quality Prores 4444 file
    -p 3: generate proxies both JPG seq, H.264 video.
    -C: local config file
    -s %: denotes proxy scale, in %
    -b: seamlessly removes focus pixels, mlv2badpixels.sh.
    -k: keep DNGs
    -d 3: hq AHD demosaicing
    -g 0: Standard Gamma
    -G 2: Rec.709 Gamut
    -o ./output: One folder per converted file, each containing all materials
   
All options must go before the MLV/RAW files.



MLVApp (MLVApp+static FFMpeg, 64bit; thread; homepage, written in qt/C++. A RAW player/converter for MLV files with ProRes export.)

QuoteFeatures
-Free and Open Source!
-Fast (The main goal), Fast playback as well.
-Develop your raw video files like in Adobe Camera Raw (Exposure, Temperature, Tint, Saturation, Dark Strength, Dark Range, Light Strength, Light range, Lighten, Highlight reconstruction)
-Gamma curves: Tonemapping, Alexa Log-C, Cineon Log, Sony S-Log3
-Correct ML Raw files issues (Focus dots, Bad Pixel, Chroma Smooth, Pattern noise, Vertical Stripes, Deflicker target)
-Histogram/WaveformMonitor/Zebras
-Support for Lossless compressed MLV
-Support for 10/12 bit
-Batch export
-ProRes (and other formats) export (Prores 4444, 422 Proxy, 422 LT, 422 HQ, 422, uncompressed AVI so far)

Linux compile howto

ubuntu 16.04 with kubuntu packports ppa activated, so qt 5.6.1:

sudo apt-get install libpng16-dev qtmultimedia5-dev qt5-qmake qtbase5-dev libz-dev

I had to change line 22 of MainWindow.cpp to: 

#include <libpng16/png.h>

and line 55 of MLVApp.pro to:   

linux-g++*: LIBS += -L/usr/local/lib/ -lz -lpng16

then the ususal:
qt5-qmake MLVApp.pro
make


ffmpeg binary must be in the same directory as MLVApp for working export.



[UNMAINTAINED]MlRawViewer ( 64bitBinary2016-08-15; thread; homepage) .mlv/raw Player (no 10/12 bit .mlv support); converter; GUI & command-line; written in Python. Needs python-opengl python-numpy python-tk python-pyaudio libglfw 3.1+ python-scandir; While unmaintaind, works, particularly for previewing/playing footage, also has powerful LUT support. Prores export 10bit only (ffmpeg). Great for quickly going through a card full of MLVs as you can preview and export only what you need very easily. mostly keyboard controlled

Quotesupports RAW MLV in both single and multi-file versions, sound playback from MLV_SND audio, external WAV, playback of directories containing sequences of "CinemaDNG" .DNG files; BSD-style license.

Features
    RAW, MLV and CinemaDNG playback in real-time (if you have fast file read and modest GPU), single and multi-file
    Progress bar and timestamp overlay. Click to jump in file
    Pause playback: SPACE key or play/pause icon.
    Toggle fullscreen and windowed mode: TAB key or fullscreen icon
    Jump to next/previous RAW or MLV files in same directory as current file using O/P keys.
    Drag a RAW/MLV/DNG file onto the window to view that file. Drag a WAV file to play with the current video.
    Step forward and backward through frames by one second (LEFT/RIGHT cursor keys) or one frame (,/. keys) while playing or paused
    Brightness control while playing: UP/DOWN cursor keys. Or click/drag in brightness box.
    Fast GPU Bilinear demosaicing for playback
    High-quality (but slow) CPU demosaicing (AMaZE) when paused or enabled (Q key)
    Export current marked range with audio: E key or red button.
    Choose export folder: W key.
    Choose MOV (ProRes) or DNG sequence as export type using D key.
    White Balance adjustments. Keys: 1=WhiteFluoro,2=Daylight,3=Cloudy,0=None, 4/7=Red down/up, 5/8=Green down/up, 6/9=Blue down/up. Or click/drag in Red/Blue box.
    sRGB colour output
    sRGB or Rec.709 gamma, global (HDR) tone mapping, Linear mapping, Log mapping toggle "T" key or mapping icon
    Drop-frame+audio and non-drop-frame modes. Toggle "F" key
    Slide audio sync -0.5,-0.05,+0.05,+0.5 seconds with V/B/N/M keys.
    Mark IN/OUT, Jump to PREV/NEXT mark with J/K and H/L keys.
    Loop or one play mode: I key or loop icon.
    Exit viewer: ESC key

1D and 3D LUTs can be used.

Keyboard shortcuts


Compilation

Quote
Usage:
             ./mlrawviewer.py  [<path_to_ML_RAW_or_MLV_base_file>] [<wav_file_to_play_along_with_video>] [<output_file_name_for_encoding>]  (if no argument given, a file-open dialog pops up)
             


raw2cdng (download latest on website) ML .raw to cinemaDNG ConverterGUI; (10bit LOG, 12bit & 16bit), batch convert, 15% white level adjust; dotnet40 based, runs via Wine; .mlv support, supports audio from mlv

Installation

drag&drop works flawlessly, converting too.

now GPL open source, native MLV and RAW Support, AudioSupport, multithreaded, code at https://bitbucket.org/chmee/raw2cdng





MLVProducer (download, choose amd or intel optimizedthread; Converter; GUI (Visual Basic); runs via Wine. currently crashes at export, but export is fine)

Installation

QuoteFeatures:
-open Source, Visual Basic
-Correction, Curves, Filters
-Encode to  h264, ProRes, DNxHD, FFVH, JPEG, PNG, TIFF, CinemaDNG, DNG
-HDR Processing
-Dual-ISO(in progress)
-Focus Dots suppression
-Vertical Stripes suppression
-Horizontal noise reducing
-Apply 3DLut: .cube, .3dl
-Color Spaces: Linear, sRGB, AdobeRGB, ClogC, CanonLogC, AlexaLogC, REDLog, BMDFilm, Cineon, REC709, SLog, SLog2, SLog3, SLogF35, Panasonic V-Log,...
-Multithreading
-Demosaic with different algorithms.
-Group\Global editing
-Cache frames almost like AE. 1.5GB Limit.
-Process MLV and RAW Files
-.M00,M01,Mxx.. Files
-.R00,R01,Rxx.. Files
-Process Huge files.
-Process some broken files.
-Video Import
-Audio Support
-16bit precessing
-measuring tools: Histogram, Waveform, RGBParade, RGB Balance


raw2dng (raw2dng2016-08-14; thread) Converter; reference implementation, commandline; 32bit; compiling for 64bit does NOT work, so you have to install libc6:i386 to get it run on 64bit systems; NO MLV SUPPORT

Quoteusage:

./raw2dng file.raw [prefix]

=> will create prefix000000.dng, prefix0000001.dng and so on.




mlv_dump (mlv_dump.zip; thread) Converter; reference implementation, commandline; 32bit; libc6:i386 on 64bit. Unzip, rename mlv_dump.linux > mlv_dump, make it executeable (chmod +x ./mlv_dump)


Quote
mlv_dump
- used for debugging and converting .mlv files
- can dump .mlv to legacy .raw files  + .wav files
- can dump .mlv to CinemaDNG  + .wav
- can compress and decompress frames using LZMA
- convert bit depth (any depth in range from 1 to 16 bits)
Quotecompression:
you can get a data reduction of ~60% with 12 bit files.
down-converting to 8 bits gives you about 90% data reduction.
this feature is for archiving your footage.
converting back to e.g. legacy raw doesn't need any parameters - it will decompress and convert transparently without any additional parameter.





[UNMAINTAINED]mlv2dng (mlv2dng2014-04-30; thread) ML .raw to DNG converter; commandline; 64bit; maybe still useful. NO MLV SUPPORT




[UNMAINTAINED]MLVViewSharp (MLVViewSharp.zip; thread) ML .mlv Player; GUI written in c#, runs with mono-runtime, libmono-corlib2.0-cil libmono-winforms2.0-cil installed

Quotea _very_ simple and hackish MLV viewer to check your footage
will read uncompressed MLV files and display the frames with a few frames per second.





[UNMAINTAINED]Magic Lantern RAW (MagicLanternRAW-latest.zip; thread) Converter GUI; written in QT; 32bit, so needs libc6:i386 and 32bit qt libs to run on 64bit systems;  NO MLV SUPPORT





cr2hdr / Dual ISO(cr2hdr2014-04-29; thread). commandline tool; 32bit; libc6:i386 @ 64bit systems.

Quote
Increases dynamic range by sampling the sensor at two different ISOs, switching ISO for every other line pair. This trick cleans up shadow noise, resulting in a dynamic range improvement of around 3 stops, at the cost of reduced vertical resolution, aliasing and moire.

Works for both raw photos (CR2) and raw videos (DNG sequences). You need to postprocess these files with a tool called cr2hdr.

After postprocessing, you will get a DNG that looks like a dark ISO 100 shot, but you can bring the exposure back up and be delighted by how little noise is present in those recovered shadows.

Compilation

Quote
Usage:

LC_ALL=C cr2hdr *.CR2





Linux Applications known to work on/with dng, cinemaDNG, respectively .raw/.mlv files support:

Digikam raw photo developer; open source; (ML .raw support via their kipi-plugins; tutorial)https://extragear.kde.org/apps/kipi/


Darktable raw photo developer; open source; (ML .raw import should already work ; also powerful commandline interface for processing frames; thread ; proof of concept tutorial of working with raw image sequences in Darktable )


Rawtherapee, Lightzone, Photivio, Rawstudio are open source raw (photo) developers which are also dealing with .dng, with Rawtherapee having the by far most feature-rich demosaicing possibilities and settings.


Natron opensource compositor modeled after Nuke; node-based; .dng support via openimageIO , exr, tiff dpx etc. supported, ofx-host;


Digital Fusion compositor; freeware(slightly less features)/commercial; now owned by BlackMagicDesign, native Linux version, .dng support; ofx-host;


The Foundry Nuke/NukeX compositor; crossplatform; node-based; free for noncommercial use Edition (watermarked, special file format, features disabled); from $4213; You can use J_Ops , tutorial which includes J_rawReader and allows direct reading of .dng (as well as other raw formats) into nuke. Widely used industry standard compositor, if big Shops do not use inhouse tools, they use this. ofx-host;


Mistika grading/compositing, editing/finishing; commercial; only available as a turnkey system, bundled with hardware $100000+; originally Runs on Suse/Novell Enterprise Linux; ofx-host;


MambaFX grading/compositing; commercial, crossplatform; cinemaDNG only; FREE FOR 1080p! MambaFX is derived from Mistika, without timeline and editing capabilities, so a single Shot based compositor, with most of the powerful compositing/grading features of the full Mistika, for ~ $300!!, with cheap and expensive plugin options ( codecs, 4k, DCP, 3D-rig=$9000) available; recent Nvidia card & proprietary drivers needed.


Ifxsoftware Piranha grading/compositing/editing; crossplatform; commercial, but free for 720p; $1000; EDL; supports cinemaDNG via gpu based plugin (FAST), or dng via DCRAW plugin (slow) screenshot


Davinci Resolve for Linux grading/compositing, editing/finishing; freeware(slightly less features)/commercial; $20000+;commercial version only available bundled with hardware console;  FCPX xml, EDL; Supports CinemaDNG and DNG; ofx-host;


Autodesk Flame Premium editing/conforming/finishing. compositing/grading; $10000+; originally runs on RedHat / CentOS
overview Training edition available. 16 GB ram & recent Nvidia (also geforce 8xx+ works now) card & proprietary drivers needed., Supports DNG, DPX, OpenEXR


Apple Shake compositing; commercial, abandonware; was $20,000 before EOL in 2006; supports DNGs through this openSource dcraw plugin: smshplugins; Very capable  RGBa-only node-based piece of Software from Apple, which is still used in some VFX shops; training material still available; ofx-host;


Speedgrade NX grading/compositing; commercial, abandonware; was $19,999; Abandonware on Linux (as now owned by Adobe, they dropped Linux Version);  cinemaDNG only;




The apertusProject is also dealing with DNG for their axiom, they also want to build an openSource DNGraw processing tool, Open Cine. This software is in the development. You can help out!



common errors:
- "command not found": you are on a 64bit system, but no 32bit C library installed:  install libc6:i386;
- NO you do NOT need wine, except for MLVProducer, raw2cdng