MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[Danne]

Created a short video tutorial on how to mount the fuse app according to the terminal commands provided in the first post from David Milligan.
Got some questions about this in my inbox.
This workflow will likely change soon due to the early development stages. But for now...


*Update Dmilligan created a automated gui through services in the first post. Terminal workflow on mac not needed anymore.

[nigel]

Oh wow. Is it time to get a Mac? Hahah

[ayshih]

Cool! When I right click a dual iso movie file I can give the file read and right priviligies and the files can convert fine with cr2hdr20 bit and also open in after effects.

It's easy enough to understand why After Effects complains; the virtual filesystem is configured to not have write permission, and you work around that.  What I don't understand is your statement about being able to convert with cr2hdr.  The filesystem doesn't have any actual implementation of the write call.  So, what exactly happens when you run cr2hdr?  Where do the converted DNGs go?  I don't have a dual-ISO video to try this out myself.

[Danne]

- Drag the dual iso dng folders to my desktop, then give the folder read, write permissions
- Convert the files using cr2hdr20bit as usual. Original dng files are being overwritten to the converted one.

Question. How do you explain acr being able to store write changes directly to dng files in the original fuse folder but not in after effects which needs permissions? Is acr overriding the write, read permission structure?

Thanks
/D

[ayshih]

- Drag the dual iso dng folders to my desktop, then give the folder read, write permissions
- Convert the files using cr2hdr20bit as usual. Original dng files are being overwritten to the converted one.

Ah, okay.  To be clear, when you drag the folders to your desktop, you are converting the virtual DNGs to actual DNGs.

Question. How do you explain acr being able to store write changes directly to dng files in the original fuse folder but not in after effects which needs permissions? Is acr overriding the write, read permission structure?

When you use ACR on files in a read-only directory (e.g., on a CD), it can't create the sidecar XMP files (which is also its fallback for read-only DNG files), so the changes are instead stored in a central Camera Raw database.  That's how ACR can work with the (current) read-only nature of the MLVFS filesystem.

I don't know how files are distinguished in the Camera Raw database, so it's possible you may lose your ACR modifications if you mounted the MLV file in a different location.  Certainly, the ACR modifications would not persist if you mounted the MLV file on a different computer.

[dmilligan]

Theoretically, dual-ISO support would be built into this tool (that would be the best case scenario). However, I don't see this as a viable option. I don't think there's really any way to make dual-ISO conversion fast enough to be useful with this on-the-fly way of doing things (unless somebody wants to implement dual-ISO conversion in hardware on an FPGA or something ), and make it fit in with the way the FUSE API works (processing small chunks of a frame on demand, rather than the whole frame at once).

I think for dual-ISO you're just going to need to go a more standard processing route, permanently converting the DNGs and then permanently converting those with cr2hdr (you can use this tool for permanent conversion, by copying the DNGs out of the virtual filesystem, but then there's no real advantage to using this tool over any of the other ones).

It would be possible for this tool to generate "low-res proxies" of dual-ISO clips for editing or previewing (such as by just skipping the bright rows in the output) though.

[Danne]

Ah, okay.  To be clear, when you drag the folders to your desktop, you are converting the virtual DNGs to actual DNGs.

Yes, that must be the case. But I am simply copying them from the mount.

When you use ACR on files in a read-only directory (e.g., on a CD), it can't create the sidecar XMP files (which is also its fallback for read-only DNG files), so the changes are instead stored in a central Camera Raw database.  That's how ACR can work with the (current) read-only nature of the MLVFS filesystem.

Thanks for claryfiyng.

I think for dual-ISO you're just going to need to go a more standard processing route, permanently converting the DNGs and then permanently converting those with cr2hdr (you can use this tool for permanent conversion, by copying the DNGs out of the virtual filesystem, but then there's no real advantage to using this tool over any of the other ones).

It is till faster to copy the dng files from the fuse mount than exporting from a converter right?
So, there won,t be a possibility to overwrite the mounted dng:s since they,re only virtual ones from the orginal mlv files?

[dmilligan]

It is till faster to copy the dng files from the fuse mount than exporting from a converter right?

Theoretically, no. When you copy from the FUSE mount, you essentially turn this program into a traditional converter, so it should take about the same amount of time as a traditional converter, unless my conversion implementation is more efficient. Which might actually be the case right now, since I don't do any pre-processing on the files like banding correction or chroma smoothing, and also MLFVS supports multithreading via the FUSE API, so you might be getting more CPU usage. However, I imagine speed is mostly I/O limited, not CPU limited, and if this is the case, multithreading would have no effect. Feel free to do a speed comparison, though, I'd be interested to know (hint: use time to time a cp of the virtual DNGs to the real filesystem).

So, there won,t be a possibility to overwrite the mounted dng:s since they,re only virtual ones from the orginal mlv files?

No, that's not really possible, though I guess theoretically you could implement something where the CDNGs are writeable and the written data goes back into the MLV file VIDF blocks, modifying the original MLV file. That sounds VERY hard to implement.

[surami]

I could install Ubuntu (14.04.1) inside the VirtualBox (test build 4.3.15) but the MLVFS isn't working. I think it's because I'm trying inside a virtual machine, so there is too much virtualisation "layer" for the system. Now I will try the Cygwin (never used it), maybe it will work.

Update: I could get it working with the new stable 4.3.16 VB. It's a very long journey to do, but it works. As I have time I write a step by step "how-to".

[ayshih]

MLVFS works fine in a virtual machine, but just within that virtual machine's local filesystem.  The way that VirtualBox and Parallels (at least) provide support for shared folders, they are local folders in the host machine, and thus MLVFS can't create its virtual filesystem there.

Instead, you'd probably have to use more traditional methods of sharing the MLVFS mount point: share the folder over a network as if they were two separate computers (e.g., using SMB).

[surami]

@ayshih: Thanks for your explanation, so I should use Samba server. I will try, but I never did this yet, so it will be a long journey for me. Cygwin is not for this job.

[Danne]

So, compared the mlv_dump v 1.0 which ran vertical stripes correction to the speed of the mlvfs conversion. Used the "hints" (time, cp) commands from links provided from dmilligan (thanks).

The file is 9.52gb big.

The mlvfs conversion

real   0m35.285s
user   0m0.043s
sys   0m12.761s

The mlv_dump conversion

real   2m4.544s
user   1m36.526s
sys   0m10.662s

I also did some batch conversion around 38gb comparing mlrawviewer and mlvfs and the mlvfs was about 30 seconds faster around 5min30sec while mlrawviewer finished around 6min. Not whole lot of difference there.

[dmilligan]

cool, thanks for the tests...


Audio is working!!!!

[senzazn12]

Praying this gets ported to Windows somehow. 

[ayshih]

Instead, you'd probably have to use more traditional methods of sharing the MLVFS mount point: share the folder over a network as if they were two separate computers (e.g., using SMB).

Oof, it's not for the faint of heart, but I've figured out how to get this approach working for Windows users.  You have to jump through a couple of extra hoops, and I'm sure performance will suffer because of the layers.

In the Linux VM:

• Make sure that the Linux VM is networked with the Windows host
• Configure Samba to share the local folder that will be used as the MLVFS mount point
• Configure FUSE to allow other users (namely smbd) to use your mount points by editing "/etc/fuse.conf" (requires sudo) and uncommenting the line "user_allow_other" (and also make sure that fuse.conf is world-readable)
• Run the MLVFS mount command with the additional command-line option "-o allow_other", e.g.:

Code: [Select]

mlvfs <mount point> --mlv_dir=<directory with MLV files> -o allow_other

In the Windows host, connect to the shared folder, e.g.,

Code: [Select]

\\<Linux hostname>\<mount point share>

Praying this gets ported to Windows somehow. 

I'm still looking into using Pismo File Mount to provide native support for Windows, because the above is painful.

[g3gg0]

oh i just remind of my good old symbian days.
back then i made a webdav server in C#.

using that tool we could get a cheap and easy-to-use MLVFS pendant on windows.
let me see....

[dmilligan]

updated the OP with the Windows instructions

[g3gg0]

short update:

i patched my mini WebDAVServer to virtualize MLV files just like MLVFS does.

on windows you just have to mount the file system using
> net use x: \\localhost@8080
where 8080 is the port you specified in configuration dialog.
then drive X: shows you the content of the shared directory/card and allows you to go into MLV files.
you can even mount it via network. please dont use auth string, as windows confuses it with NTLM auth crap.

it only supports
 - MLV->JPG
 - MLV->WAV
 - experimental saving files in MLV folder (creates a <mlv-file>_store directory)
 - no DNG support

i will create a separate post for that and ask people to help.
creating sane DNGs is not really fun and takes its time to test and tweak.

@chmee:
interested in adding your C# DNG code to this tool?
will post it on bitbucket.org

[dmilligan]

There's now a simple installer and service for Mac, see the OP (and let me know that it works)

[jose_ugs]

wow! mounting MLVs! keep it up, and hopefully you can figure out a Win7 solution

[Danne]

Works perfect! With audio aswell. How did you do that. Amazing work. Very convenient to have integration with services.
Folder names can,t contain spaces on mount point folder. One word only.
cool stuff.

[menoc]

I installed it and works pretty well on my mac pro. Only thing is, for some reason Premiere  Pro CC 2014 won't let me import the dngs as an image sequence - the check box is greyed out. Anyone knows how to fix it?

[chmee]

Menoc. You dont need to. It will be recognised AS sequence.

[dmilligan]

Folder names can,t contain spaces on mount point folder. One word only.

should be fixed now

I installed it and works pretty well on my mac pro. Only thing is, for some reason Premiere  Pro CC 2014 won't let me import the dngs as an image sequence - the check box is greyed out. Anyone knows how to fix it?

Just select the first DNG and click import, the whole thing will be imported (don't worry about the check box). Or you can just use the "Media Browser", the DNGs should show up as a single clip and you can drag/drop it into your timeline.

[Danne]

Folder names working.