MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[Ethan]

Has this program been compiled into a single downloadable file that I can download and run like any other program?

I have no knowledge of coding or running commands and I am completely lost on how to use this program. I even followed the video tutorial but no luck.

Thanks,
-Ethan

[andrew_dotdot]

@Ethan...

Once you have it installed (on Mac anyway), the idea is that you right click the folder where you have your MLV files and set that up as a mounted "drive". You go down the right-click menu to where it says Services, then select MLVFS. It's not an "application" situation, but rather more transparent -- it just starts to show you a new drive in the Finder window's list.

The thing that I always found tricky is how the next dialog that comes up (where you select the empty directory that will act as the "drive") can be hidden under other Finder windows. But MLVFS doesn't work unless you make that selection. So once you've started using MLVFS, you know you MUST do this; but the first time, it can be tricky-tricky.

[andrew_dotdot]

If I have Xcode installed, how do I then use it to monitor what MLVFS is doing, as was suggested earlier in the thread? To recap, MLVFS works as expected except that it is stoooopid slow.

Do I need to open MLVFS in Xcode as a project somehow? I was thinking that it might come up in the "Debug...Attach to process...", but there, it doesn't seem to be referring to processes running at large, but rather to processes inside the project that's loaded into Xcode.

I found the MLVFS.workfow, which just refers to "./mlvfs", but I'm not sure where that actually refers to. I guess it's in an environment variable somewhere, but that's at the outer reaches of my knowledge.

I'm not sure what to point Xcode at, and how.

Thx! 

:A)>

[dmilligan]

You do have to download the MLVFS source code and then open the xcode project in xcode.

Code: [Select]

git clone https://bitbucket.org/dmilligan/mlvfs.git

or https://bitbucket.org/dmilligan/mlvfs/get/9f8191808407.zip

Then open the mlvfs.xcodeproj in xcode.
Go to Product > Scheme > Edit Scheme to modify the command line parameters for debugging to suit your environment (mainly you just need to change the first argument to some empty mount directory, and the "--mlv_dir=..." argument to somewhere with MLV files). Now you can just hit "Build and Run" button at the top left of the main window. Once it's running you can view some basic information about CPU, memory, etc. or on the CPU info screen and click "Profile in Instruments" for complete and detailed information. Once the "instruments" is recording, do some stuff with MLVFS (play something back in Resolve, etc.). Save the trace and send it to me.

[kaco]

Hi,

I have a problem with windows port.
When called on a lot of files (1000+), after calling localhost it will process/index only approx first 500. first 345. The rest will start indexing after I open the mounted folder, which crashes the Resolve while waiting for the folder to list.

I think this must be some kind of hardcoded file limit? Is there a chance to higher this number? I get hunders of files from every shooting day.

[kaco]

The file limit displayed (which means also indexed) by localhost is 345.

In the source code I have found that you have disabled indexing after files are mounted. What was the reason for this? For me, it would be much better to mount, wait until everything is indexed and then start working with it.

Now it crashes even Windows Explorer, if I try to open big not indexed files.

[daniel.t]

Hello, guys. I'm just starting my MLVFS experience but I met a huge performance drop compared to normally converted dng files.

I'm using MacBock Pro 2016 and it gives me realtime playback with dng files. But when i try MLVFS playback fps drops down to 2-3, sometimes it just "freeze". Also it takes few minute just to import files into media page. MLV files located on external HDD (exfat file system) don't know is that matters. Have someone same issues?

Just tested files from the internal SSD, smooth as butter. But normal dng files plays from external HDD pretty easily in realtime. Tried to choose directory on SSD and on external HDD, nothing helps.

[dmilligan]

It's probably the fact that it's formatted exFAT. exFAT support in macOS is not very good, and I've seen the issues I think you are probably experiencing (I have SD cards formatted as exFAT). Unfortunately, I couldn't find a viable solution. IIRC, fclose() takes a really long time and eats CPU for seemingly no reason (there's nothing intensive or complicated fclose should be doing since the files were not written to, only read from). I could make it just not close the files, but then you eventually run into the max number of allowed open file handles.

You can try using a different format on the HDD and see if that fixes your issue.

[dmilligan]

In the source code I have found that you have disabled indexing after files are mounted. What was the reason for this?

I have no clue what you are talking about. Could you reference what source file and line number you are referring to?

[kaco]

I have no clue what you are talking about. Could you reference what source file and line number you are referring to?

Comment for this file: https://bitbucket.org/dmilligan/mlvfs/src/9f8191808407bb49112b9ab14c27053ae5022749/mlvfs/index.h?at=master&fileviewer=file-view-default
Windows: IDX files no longer created upon mounting

The problems with 345 files limit is not on macOS, I am working under WIN10 64bit, disk formatted in NTFS.

[dmilligan]

That is a very old and irrelevant comment that only has to do with the old pismo version which doesn't exist anymore.

Are you saying that the webgui will only show you the first 345 files if you have more than that in a single folder? Have you tried a different set of files? Is the limit still 345? Could you be more precise about what exactly is happening and maybe include screen shots? Have you looked at the console output? Have you tried separating the files into sub folders so there's not so many in a single folder?

[kaco]

That is a very old and irrelevant comment that only has to do with the old pismo version which doesn't exist anymore.

Are you saying that the webgui will only show you the first 345 files if you have more than that in a single folder? Have you tried a different set of files? Is the limit still 345? Could you be more precise about what exactly is happening and maybe include screen shots? Have you looked at the console output? Have you tried separating the files into sub folders so there's not so many in a single folder?

Windows 10, 64bit, RAID drive, exfat, folder with more then 345 files (1000+):
1. mount the folder with mlvfs, console shows nothing
(if I will open mounted drive in Resolve or under Windows Explorer, it will hang and then crash, because files are not indexed yet. MLVFS will however start to index those being opened by Resolve/Explorer)
2. when called locahost:8000, it will start to index. However, it will index only first 345, it will also print only the first 345 on the localhost page. Console displays just indexing info, nothing unsual. All files are valid. Same behaviour for different folders.
All files are in the root folder, no subfolders.

So for now, I have to run mlvfs, call localhost, let index first 345, move indexed files to another folder and then refresh localhost to let another 345 files to index.

Let me know, what could I do to help. Record youtube video?

[dmilligan]

There is a maximum size for the HTML generated by the MLVFS webgui. You can make it bigger if you like.  More work, but also possible would be to dynamically allocate that buffer, eliminating the limit.

If you don't put so many files in a single folder you shouldn't have this issue, and you also shouldn't have to force MLVFS to generate index files using the webgui like you are doing. When you browse to a folder in the virtual file system, all the MLV files in the corresponding real directory have to be indexed before that operation can complete. Did windows explorer actually crash or was it just taking a long time and you gave up and killed it? It may take quite a while to index 1000+ MLV files.

[kaco]

There is a maximum size for the HTML generated by the MLVFS webgui. You can make it bigger if you like.  More work, but also possible would be to dynamically allocate that buffer, eliminating the limit.

If you don't put so many files in a single folder you shouldn't have this issue, and you also shouldn't have to force MLVFS to generate index files using the webgui like you are doing. When you browse to a folder in the virtual file system, all the MLV files in the corresponding real directory have to be indexed before that operation can complete. Did windows explorer actually crash or was it just taking a long time and you gave up and killed it? It may take quite a while to index 1000+ MLV files.

With Resolve, if the file is big (sometimes single file can be 100Gb+), it takes longer then Resolve can wait and it imports not all of the cdng. With no warning. So that is why I generate index files via webgui.

Also with Windows Explorer, if I copy not indexed files from mounted drive (sometimes I need to copy cdng for postproduction studio), for some very big files it will copy empty folder (probably the windows explorer copy function has some timeout?).

Would you be so kind and help me where to change the size of html?

[kaco]

Or would it be to complicated to change the mlvfs so that it will automatically start indexing all of the files after mounting?

[dmilligan]

in webgui.c change the line:

Code: [Select]

#define HTML_SIZE 65535

Automatically starting to index files when mounting is probably a bad idea. It would significantly hurt performance while indexing is happening, requiring one to basically wait for MLVFS to search and index the entire directory structure before being able playback any files. If anything I would like to make indexing even more "lazy loaded". That might actually fix the issue you are seeing, and if you take my suggestion to use sub folders, I think you will also see improvement.

If you really want to, you should be able to use mlv_dump -x to force generate index files.

[Teamsleepkid]

I've got Focus Pixels with the new eos m 203 experimental build. And 2x2 produces large blue red and white artifacts. I'm using 10 bit and 12 bit. Not sure if it's the build or if it's mlvfs. It used to work in 10 bit and 12 bit. Been hearing a lot of talk about people using mldump. Does mlvfs use this? Definitely not complaining so please don't take it that way:)

[dmilligan]

MLVFS does not use mlv_dump. Theoretically, chroma smoothing shouldn't be necessary, MLVFS uses some bad pixel maps to remove the focus pixels. If you think there's an issue, posting some examples would be a good idea.

[a1ex]

Most likely this issue.

[dfort]

@Teamsleepkid -- If you are shooting with the crop_rec module it has a different focus pixel pattern than the regular mv720 mode so you need a special focus pixel map file. I created an experimental map file that covers both the mv_720 and crop_rec modes for the EOSM. You need to install this inside the MLVFS application. It replaces the 80000331_1808x727.fpm file.

EOSM_crop_rec_mv720_combo_map_file

[Teamsleepkid]

sweet. how do i do it? how to locate 80000331_1808x727.fpm file? looked but its not like a normal program.

[dfort]

If you are on a Mac, go to where MLVFS.workflow is installed. It should be in your home Library in the "Services" directory. Control click -> Show Package Contents -> Contents and you should see all the .fpm files in there. If you are on Windows, I'm not sure but it should be similar.

[Teamsleepkid]

cool thank you

[futografia]

Does anybody is running this on sierra?

It doesn't show at right-click/services for me.

[dmilligan]

Works fine here. You're right clicking on a folder not an MLV file right?