MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[dmilligan]

MLVFS - Magic Lantern Video File System

What is it?
An MLV "converter" that provides a virtual file system of converted data 'on the fly' using Filesystem in Userspace (FUSE). It allows you to "mount" an MLV file(s) which shows up as a directory of converted CDNGs. The data for these CDNGs is provided 'on the fly' by MLFVS as the data is being requested by whatever raw editor or post processing software you are using.

It's fast and efficient, and I've managed to get real-time playback in Premiere/SpeedGrade (on my 2012 MacBook Air!).

The code is written in C and works on Linux, Mac (via the OSX port of FUSE http://osxfuse.github.io/ ), and Windows (via the dokany project: https://github.com/dokan-dev/dokany thanks to g3gg0)


Advantages
- You don't have to convert first, you more or less "instantly" have "converted" DNGs
- You don't have to choose between keeping the original MLVs or converted DNGs or doubling disk usage to keep both.
- There's no need for a GUI, there already is one provided by the OS (i.e. the OS's file browser).
- Possible to vary the quality, pre-processing, bit depth, etc, of the DNGs on the fly


Updates
2014-09-02 Working Proof of Concept => There's much more work to be done (see below), but there is a working proof of concept, and a Mac build is available.
2014-09-08 Audio is working!
2014-09-09 Mac installer/service
2014-09-19 White balance mostly working
2014-09-20 AE working! and filesystem is now read/write (files are stored in .MLD directory in the real filesystem)
2014-09-26 Vertical banding correction
2014-09-27 Chroma smoothing
2014-10-08 Bad pixel correction
2014-10-09 Embedded mongoose webserver for configuring options at runtime
2014-10-13 Full quality dual ISO working (cr2hdr-20bit)
2014-10-30 Auto-linked audio in DaVinci Resolve
2015-02-05 Compressed file support
2015-02-17 Animated GIF Previews
2016-02-05 Fix for Focus Pixels
2016-02-07 Full Windows Support thanks to g3gg0 (via dokany)
2017-05-05 Lossless JPEG Support thanks to bouncyball

Source Code
https://bitbucket.org/dmilligan/mlvfs


Download
Mac
Install OSXFUSE
Download MLVFS.dmg
Double click the "MLVFS.workflow" file and you should see the following prompt:



(Click Install)

Now, you can right click a folder (with MLV files in it) and go to "Services" > "MLVFS"



You will be prompted to select a mount directory (it must be empty)
To stop mlvfs, simply click the eject button that shows up next to the mount point

If you get a security error, you may need to change your security settings to allow unsigned/downloaded programs to run

Linux
Install FUSE via the normal way for your distribution. It's probably called something like libfuse-dev

Code: [Select]

apt-get install libfuse-dev
Download the code with git and compile using make

Windows
Install Dokany
Download MLVFS_x86

Usage
Start it via the command line like this:

Code: [Select]

cd <mlvfs_exe_dir>
mlvfs.exe <mount point> --mlv_dir=<directory with MLV files>


Web based GUI



Known issues and things not yet implemented
- Audio
- Raw image pre-processing
    - hot/cold pixel correction
    - vertical banding correction
    - chroma-smoothing
- Support for compressed MLV files
- I'm sure there's more stuff...


Contributors
ayshih
dmilligan
g3gg0 (Windows port)

Credits
ML developers (esp. g3gg0 and a1ex for designing the MLV format and providing reference converters)
Developers of open source MLV=>CDNG converters: chmee and baldand => your work and source code was a great help in understanding the CDNG format and requirements


Some helpful tutorials from @reddeercity and @DeafEyeJedi

Original Post

So, what would be the 'best' or 'easiest' possible workflow for MLV? Native support for MLV format in whatever raw video processing tool you use (via open source code!), obviously.

There are ways to do this, many programs provide an API for third party plugins to implement something like this, but any implementation would be a lot of work and limited to working with one single app (as APIs are going to vary). Then I had the thought, what if we went down to a lower level of abstraction that all apps will have in common: the file system itself. So I started looking around the internet for an easy way to create virtual file systems, and found this awesome API that allows you to do just this in a user land program: FUSE

So the basic idea is that you can "mount" an MLV file(s). The converter program provides info and converted data to the OS "on demand" for this virtual file system. This is sort of like how you can mount a disk image or traverse into zip files in windows without actually extracting it.

Here's how it works: you just start the program, point it to some 'live' directory (where you can drop MLV files) and give it a mount point. The mount point directory then shows up as having folders for each MLV file containing CDNGs for each frame.

So I have started working on this and I have some sort of VERY preliminary a proof of concept working. It is by no means usable yet, there are all kinds issues and things not implemented yet, but I can at least setup the file system and provide some semblance of the data that's going to be in the DNG (not readable yet, b/c I haven't created the DNG header with all the correct tags and such, and there are probably issues with my bit unpacking code) so that I can test speed and such. Basic DNG conversion is working and it seems to be pretty fast, so far it's not really noticeable that you're using a virtual file system.

If you're a coder and would like to help, that'd be fantastic, esp if you have a good knowledge of the DNG spec (I don't and I'm learning all of this as I go).

[Audionut]

Looking forward to see where this ends up.  Nice work dmilligan.

Something useful for windows here?  http://sourceforge.net/p/fuse/wiki/OperatingSystems/#windows

[ayshih]

It sounds like a promising approach!

I'm interested in helping, but it'd be nice if there was a way for it to work on Windows.  Unfortunately, it looks like the only actively developed implementation may be DokanX (https://github.com/BenjaminKim/dokanx/), and I don't think it uses the FUSE API.  (See the list of relevant projects on this Wikipedia page.)  But, I do have an OS X machine and Linux VMs, so it's not an non-starter.

I've only started reading up on this stuff, but I'll point out that OSXFUSE says that it provides an API that is a superset of the FUSE API, so you may want to make sure you're using just the FUSE API to ensure Linux portability.  Of course, at such an early stage, that's a very low-priority concern.

[dmilligan]

Great! I'm pretty sure I've only used the basic FUSE API (I've only included fuse.h, which I assume is the basic API), and I only use basic C libs and general posix stuff (no apple specific libs), so it should be totally portable to linux. Most of the FUSE related stuff is pretty well implemented (we could do more advanced things later on with it), the real areas that needs help is:

Bit unpacking - I didn't want to use the bit fields struct stuff like ML and mlv_dump use, since this is non-standard compiler stuff (and not supported by Apple's compiler), so I wrote a more generic one, but IDK if it's really correct or efficient

DNG generation - I didn't want to use chdk-dng.c b/c that code is an absolute mess, also I need to write specific sections of the file to buffers on demand, not the entire file at once (that's the way the FUSE api works, a specific section of the file is requested, and you write that section of the file into a buffer). Also, it's not thread safe (nor is the vast majority of stuff from mlv_dump), which this needs to be since the FUSE API is multithreaded and requests for data can be concurrent (multiple programs accessing different or even the same file at the same time, etc.).

there are also areas of mlv parsing that will need to be improved: there's no support for chunks yet, it should generate and use index files to make finding the frame blocks faster, no audio support, etc.

If you have any questions about the code I've written so far, I'll be happy answer them.

[chmee]

@dmilligan. good idea.

[surami]

@dmilligan: awsome thought.

I know it's too early to write about it, but I think it would be also nice to add and option to apply DNG Camera Profiles (.dcp files) too. I mean the index file should contain the information, that during the virtual file system upbuilding which .dcp file should be applied to the DNGs. There is a utility, the dcpTool which could be a good starting point if a xml file is needed for this.

[dmilligan]

okay, proof of concept is basically working, big thanks to ayshih who has done a lot of work (and fixed my bugs!)

ACR and Resolve can both correctly read converted DNGs. Resolve playback is slow, but it's slow on my system anyway, I think it's the GPU. Hopefully I can get Premiere/SpeedGrade working, I think it's just a matter of getting all the right DNG tags, any thoughts @chmee?

Code: [Select]

david$ ./dng_validate -v /Users/david/mlvfs/M23-1157.MLV/00000000.DNG
Validating "/Users/david/mlvfs/M23-1157.MLV/00000000.DNG"...

Uses little-endian byte order
Magic number = 42

IFD 0: Offset = 8, Entries = 33

NewSubFileType: Main Image
ImageWidth: 1728
ImageLength: 972
BitsPerSample: 16
Compression: Uncompressed
PhotometricInterpretation: CFA
FillOrder: 1
Make: "Canon"
Model: "Canon EOS 60D"
StripOffsets: Offset = 65536
Orientation: 1 - 0th row is top, 0th column is left
SamplesPerPixel: 1
RowsPerStrip: 972
StripByteCounts: Count = 16896
PlanarConfiguration: 1
Software: "MLVFS"
*** Warning: IFD 0 DateTime has unexpected count (1) ***
*DateTime: ""
CFARepeatPatternDim: Rows = 2, Cols = 2
CFAPattern:
    Red      Green
    Green    Blue
ExifIFD: 410
DNGVersion: 1.4.0.0
UniqueCameraModel: "Canon EOS 60D"
LinearizationTable: 16383 entries
    Table [0] = 0
    Table [1] = 1
    Table [2] = 2
    Table [3] = 3
    Table [4] = 4
    Table [5] = 5
    Table [6] = 6
    Table [7] = 7
    Table [8] = 8
    Table [9] = 9
    Table [10] = 10
    Table [11] = 11
    Table [12] = 12
    Table [13] = 13
    Table [14] = 14
    Table [15] = 15
    Table [16] = 16
    Table [17] = 17
    Table [18] = 18
    Table [19] = 19
    Table [20] = 20
    Table [21] = 21
    Table [22] = 22
    Table [23] = 23
    Table [24] = 24
    Table [25] = 25
    Table [26] = 26
    Table [27] = 27
    Table [28] = 28
    Table [29] = 29
    Table [30] = 30
    Table [31] = 31
    Table [32] = 32
    Table [33] = 33
    Table [34] = 34
    Table [35] = 35
    Table [36] = 36
    Table [37] = 37
    Table [38] = 38
    Table [39] = 39
    Table [40] = 40
    Table [41] = 41
    Table [42] = 42
    Table [43] = 43
    Table [44] = 44
    Table [45] = 45
    Table [46] = 46
    Table [47] = 47
    Table [48] = 48
    Table [49] = 49
    Table [50] = 50
    Table [51] = 51
    Table [52] = 52
    Table [53] = 53
    Table [54] = 54
    Table [55] = 55
    Table [56] = 56
    Table [57] = 57
    Table [58] = 58
    Table [59] = 59
    Table [60] = 60
    Table [61] = 61
    Table [62] = 62
    Table [63] = 63
    Table [64] = 64
    Table [65] = 65
    Table [66] = 66
    Table [67] = 67
    Table [68] = 68
    Table [69] = 69
    Table [70] = 70
    Table [71] = 71
    Table [72] = 72
    Table [73] = 73
    Table [74] = 74
    Table [75] = 75
    Table [76] = 76
    Table [77] = 77
    Table [78] = 78
    Table [79] = 79
    Table [80] = 80
    Table [81] = 81
    Table [82] = 82
    Table [83] = 83
    Table [84] = 84
    Table [85] = 85
    Table [86] = 86
    Table [87] = 87
    Table [88] = 88
    Table [89] = 89
    Table [90] = 90
    Table [91] = 91
    Table [92] = 92
    Table [93] = 93
    Table [94] = 94
    Table [95] = 95
    Table [96] = 96
    Table [97] = 97
    Table [98] = 98
    Table [99] = 99
    ... 16283 more entries
BlackLevel: 2047.00
WhiteLevel: 15000
DefaultCropOrigin: H = 0.00 V = 0.00
DefaultCropSize: H = 1728.00 V = 972.00
ColorMatrix1:
      0.6719  -0.0994  -0.0925
     -0.4408   1.2426   0.2211
     -0.0887   0.2129   0.6051
AsShotNeutral: 0.4736 1.0000 0.6240
BaselineExposure: +0.00
ActiveArea: T = 0 L = 0 B = 972 R = 1728
*Tag51044: SRational = 24000/1001
BaselineExposureOffset: +0.00
NextIFD = 0

Exif IFD: Offset = 410, Entries = 5

ExposureTime: 1/30.0 sec
FNumber: f/1.80
ISOSpeedRatings: 400
SensitivityType: ISO Speed
ExifVersion: 2.30
NextIFD = 0

*** Warning: Too little padding on left edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on top edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on right edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on bottom edge of CFA image (possible interpolation artifacts) ***
Raw image read time: 0.008 sec
Linearization time: 0.005 sec
Interpolate time: 0.041 sec
Validation complete


[g3gg0]

just... wow
amazing!

[dmilligan]

Premiere/SpeedGrade are working and playback is realtime!!!!!

[Danne]

 
holy crap!

[LRF]

Guys, you never cease to amaze me 

If people with such skills and attitude were working in one company... Damn, you would turn the market inside-out within six months!

[dmilligan]

Okay, now that it is basically working I have posted a preliminary binary for people to try. You'll need to be somewhat familiar the command line to get it started, but eventually I plan on adding some kind of installer or setup utility. See the OP for details. Also please keep in mind that there are all sorts of things not implemented yet (like audio, white balance, processing options such as banding correction and chroma smoothing, etc.).

[reddeercity]

This look very good ! would this work with other Software like FCPX or is this just for Adobe Workflow ?
Never the less I will give it a try .
Thanks   

[Frank7D]

But not for Windows (yet), right?
Edit: Never mind, I read the OP.

[tin2tin]

Pismo is crossplatform (mac,linux, win): http://www.pismotechnic.com/
Can it be used as an alternative to fuse?

(It was used for a virtual filesystem for Avisynth: http://www.turtlewar.org/avfs/)

[chmee]

@dmilligan
the original bmcc-linearization-table is inside my cdng's. only copy/paste the tag and the values. or pipe the dng_validate -v output into a file and parse them into a binary. (or look into my sources. There's the dng-header named dngtemplate20.dng.

[ayshih]

Pismo is crossplatform (mac,linux, win): http://www.pismotechnic.com/
Can it be used as an alternative to fuse?

(It was used for a virtual filesystem for Avisynth: http://www.turtlewar.org/avfs/)

Interesting!  It looks Pismo's file mounting may actually be only for Windows, but it's not a problem to have platform-specific code.  I'll definitely look into this.

Edit: Okay, there is cross-platform support for Pismo, but it may still be preferable to stick with fuse for OS X and Linux.

Alternatively, it hasn't been explicitly stated, but the MLVFS proof of concept works on Linux as well.  You'll have to grab the MLVFS code and compile your own executable, but you probably already have fuse installed.  I imagine one could configure a Linux VM in such a way as to achieve Windows support.

[baldand]

I just tested mlvfs on Linux (Ubuntu 14.04). Nice work! :-)

I tried to playback the DNGs with MlRawViewer. There were a couple of small problems.

I had to mount using full path names, not with ~ in the mlv_argument. It didn't give an error when mounting, but the mount dir could not be opened.

The BaselineExposure tag gave a rational value of 0/0 which caused a divide by zero for me. Easy for me to workaround with a test though.

Another problem was that because the dir is read only, I can't write my per-clip config files into the DNG directory, so it's not possible to store any parameters or LUT settings for use when exporting. Export to MOV works, but with limited functionality. Well, not much to be done about that one.

I also noticed that CPU usage was quite high in the mlvfs process - something like 70% of one i7 core. Playing back through mlvfs was a bit heavier overall than direct playback, but it was still real time.

[Danne]

This is simply amazing! Just created a new preference folder in davinci resolve 10 and the mlv files plays in realtime. No conversion. Dng:s magically appears in the osxfuse mount . Dmilligan breakthrough. All creds to participant coders and so on.

Couldn,t get sequence to work with After effects cs 6 nor to play in mlrawviewer, tried version 1.2.3, 1.1.7 and 1.1.6. (noticed Baldands reply above now regarding full pathnames)
Smooth as butter in davinci resolve. I,m on a macbook pro 15 inch 2.6 ghz, OSX mavericks.

Tried to convert some dual iso movie files but it wouldn,t sing. Tried Alex lates cr2hdr20bit binary and in lightroom. Got the following:

Output file     : /Users/Daniel/Desktop/mlv/M05-1941.MLV/00000005.DNG (already exists, overwriting)
Bus error: 10

Only info and of low priority of course.

[dmilligan]

Thanks for the feedback!

I'll fix the BaselineExposure tag (interestingly, dng_validate didn't complain about that).

I'm not sure what the deal is with ~ is, but I have noticed that too. It seems that the stdio.h functions don't accept it.

About it being readonly: I can fix that (and I plan to), but it's going to be some work. Basically I can do a loopback to the original directory and store the actual data there (FUSE supports both read and write), but it means implementing all the FUSE API's write callbacks (since it's read only right now I only need to implement the read functions) and doing some type of path wrangling to manage where the files are put and how they map between the two file systems. It's certainly possible, but a lot of work.

The readonly thing seems to also be the reason AE isn't working, it tries to write it's own XMP settings, which it can't b/c the filesystem is readonly.

CPU usage is probably from the 14 =>16 bit conversion. There may be some possibility for performance improvement there (IDK how efficient my bit unpacking code is or could be, but it works), but if it's already realtime, it doesn't seem particularly important

[Danne]

ah, ok read only. Fun is i can open the dng:s and store changes made in acr and photoshop.
Very promising this indeed.

[Canon eos m]

Will be watching this thread but will stay away from trying this awesome sw just yet since the discussion is too technical at the moment. Hope you folks can release the installer soon.

[Danne]

Cool! When I right click a dual iso movie file I can give the file read and right priviligies and the files can convert fine with cr2hdr20 bit and also open in after effects.

I have to drag the folders to desktop to be able to change permissions.


Apply permissions to all items in a folder or a disk

    Select a folder or a disk, then choose File > Get Info.
    Click the lock icon to unlock it, then enter an administrator’s name and password.
    Choose "Apply to enclosed items" from the Action pop-up menu (looks like a gear).

[surami]

I will experiment with the Linux (Ubuntu 14.04.1) in a Virtual Box on Windows 7 in the evening. It should work in theory, I will report back:
1. Run the Linux in Virtual Box on Windows 7.
2. Run the MLVFS in the virtualized Linux system.
3. Open the DNG sequence in Windows 7.

Update: I can't get it working, the latest stable (4.3.15) Virtual Box drops error for me, but I'm on that to solve somehow.
Update2: I could get it working with the new stable 4.3.16 VB. It's a very long journey to do, but it works. As I have time I write a step by step "how-to".

[ayshih]

I had to mount using full path names, not with ~ in the mlv_argument. It didn't give an error when mounting, but the mount dir could not be opened.

Paths now have typical shell expansion (e.g., ~), and there is error handling for bad paths.  I'll let dmilligan update the binaries for those who can't compile.