MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

Started by dmilligan, August 31, 2014, 02:01:24 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

haeki

Quote from: haeki on February 10, 2018, 10:53:01 AM
Hi, sadly I can't bring MLVFS back to life. I had it working back in 2017, but now it's not working anymore. Updated both, fuse and mlvfs but the dialog to choose the mountingfolder doesn't pop up and the gear keeps on turning staying at 0%. Any suggestions?

I'm on a mac 10.12.6



Kind regards

It's all good, the dialogbox was hidden somewhere far away. :-)

Markus

Trying to get mlvfz working for compressed mlvs. I have normal mlvfs working on this computer but when issuing mount command with mlvfz instead of mlvfs.exe nothing happens. Is their some step of the install process I'm missing/ have forgotten?

Mlvfz works fine on my other computer...

Win10

Any help is appreciated.

bouncyball

Here is your older post and my answer. It seams you have to reinstall win.

And please stop calling it mlvfZ! It was just typo made by me ;). It is plain 32bit mlvfs with compressed MLV support.


clanlee

MLVFS Installation on Windows 10 x64

Install latest version of Dokany:
https://github.com/dokan-dev/dokany/releases

Reboot if necessary.

MLVFS_x64
Download http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip
Extract MLVFS_x64.zip

Delete the following files from the extracted folder:
dokan1.dll
dokanfuse1.dll

Using MLVFS:
Note the path of the extracted MLVFS_x64.exe
Open Command Prompt

Run program from within directory into the executable:
The web gui interface only works if you call the executable from the same directory
cd mlvfs_x64
mlvfs_x64.exe Z:\ --mlv-dir=C:\Videos\Files

Run from any location by typing full path to executable:
If you run this from a remote directory, the web gui will not work
C:\mlvfs_x64\mlvfs_x64.exe Z:\ --mlv-dir=C:\Videos\Files

If your folder name contains spaces, use quotation marks for path:
"C:\Extracted files\mlvfs_x64\mlvfs_x64.exe" Z:\ --mlv-dir="C:\Videos from Camera\Files"

Or:
C:\mlvfs_x64\mlvfs_x64.exe Z:\ --mlv-dir="C:\Videos from Camera\Files"

Or:
"C:\Extracted files\mlvfs_x64\mlvfs_x64.exe" Z:\ --mlv-dir=C:\Videos\Files

Notes:
The Z:\ option is setting a mount point.  You'll have to use an unused drive letter in your computer.  When you open Windows Explorer, you'll see a list of all the drives in your computer.  Use a drive letter that is not used.


Quote from: g3gg0 on February 26, 2016, 08:54:54 PM
@Canon eos m:

win10:
please uninstall dokan from your control panel
install dokan v1.0.0-rc1
unpack mlvfs_x64 into your folder f:\mlvfs_x64\
open a commandline

f:
cd \mlvfs_x64
mlvfs_x64.exe Z:\ --mlv-dir=F:\


osx:
did you do a chmod +x FILENAME ?

Markus

Thanks clanlee for detailed instructions and links for MLVFS win 10 x64 Install!
Got it running now. I had to go in and manually delete all old dokan entrys in regedit to be able to install new dokan. Even though I had uninstalled previous versions there were so much crap left there so the new dokan thought old versions were still installed and refused to install at first.

Mounting some really large files now, several around 200GB. It seems to take quite awile for dokan/mlvfs to load them. Is this normal? Previous versions i think it was pretty much intant even if the files were very large?

Also, does this version support mlvs with lossless compression? All the files I'm testing to mount now are 12bit lossless.

Switched back to old version with support för lossless. Files mounted instantaneously and it still worked with new version of Dokan.

clanlee

I've run test with 15-45 seconds of RAW video clips.  With those small clips, MLVFS and Dokan (1.1.0.2) does not seem to have delay.

Which version are you running that allows large files to mount immediately?

Part of the reason that I posted instructions was to help write updated information.  This is the first one that I was successfully able to figure out.

Markus

I said too much to fast, takes some time to load them when the files are large. When I tried to mount 2TB+ files at the same time Mlvfs got stuck on loading forever. I'm going to try to add one big file at a time to the mount directory to see if It has a easier time indexing them one by one.

I'm using a x86 version of mlvfs I got from bouncyball that has support for compressed mlvs. Latest Dokan version.

Markus

Just wanted to share my latest experience dealing with large amounts of data on Windows 10 using bouncyballs x86 compile of MLVFS with support for lossless.

We filmed a live show with a runtime of around 2h with 5 cameras. We filmed the show twice the same day and collected over 3tb of 12bit raw lossless data.
You can film around 93min with sound on one 256GB CF card and it turns out that one Original LP-E6 has just enough power to power the camera for that amount of time.

Had no big problems filming but the postprocessing was extremely slow. The idx-file generation on on these big and long file must have taken 24h or more. I just left my office computer on and checked in on it remotely from time to time.
After all idx files had been generated I started converting all mounted dngs through slimraw to disk. Strange thing here was that mlvfs stopped serving files after converting around 170K frames. So remotely I had to remount and restart Slimraw at 170K intervals to get through the 1.5 million frames (thank god for remote desktop :-)). After that everything went smooth exporting DNXHD files from resolve.

Is the slow experience with MLVFS  consistent with using it on Mac or Linux?

/end rant

Danne

Wow, that's a lot of dng files :).
It sounds about right that indexing would take this long. I'm even surprised you got all working in the end.
I would also suggest exporting to dng files directly through mlv_dump using the -c option which also compresses the dng lossless. Nothing against slimraw(really cool and fast) but you wouldn't need to go through mlvfs.
I wonder how batch_mlv will work on a humongous mlv file.

clanlee

I figured out the problem with the web gui not working.  I looked thru a bunch of post and saw this one:

Quote from: bouncyball on April 27, 2017, 10:00:10 AM
Don't know about resolve issue but for correctly display settings web page you absolutly have to change dir (cd) to MLVFS folder when running it from the command line. Otherwise mlvfs can not find needed files to display page corectly. Also it can not find focus pixel maps.

I'll leave the post of just in case someone else needs the information.  You must change directory to the mlvfs.exe or mlvfs_x64.exe directory to get the web gui working.


---Original post starts here---

I realized that my setup works great and I can access the dng files via windows explorer but the web interface isn't working. 

Using:
C:\mlvfs_x64\mlvfs_x64.exe Z:\ --mlv-dir=C:\Videos\Files


I wasn't getting any error messages or able to stop the drive.  Instead, I added -f and I was able to get error messages and by pressing CTRL + C, it would umount the drive:
C:\mlvfs_x64\mlvfs_x64.exe -f Z:\ --mlv-dir=C:\Videos\Files


I'm getting the following message when attempting to access the web gui (localhost:8000):

webgui: scanning C:\Videos\Files/...
load_resource: fopen error
webgui: scanning C:\Videos\Files/favicon.ico...
load_resource: fopen error


Any thoughts as to why web gui stopped working?


Markus

Quote from: Danne on April 13, 2018, 04:38:15 PM
Wow, that's a lot of dng files :).
It sounds about right that indexing would take this long. I'm even surprised you got all working in the end.
I would also suggest exporting to dng files directly through mlv_dump using the -c option which also compresses the dng lossless. Nothing against slimraw(really cool and fast) but you wouldn't need to go through mlvfs.
I wonder how batch_mlv will work on a humongous mlv file.

After checking all the files from the live event where we filmed ~20h 12bit lossless raw there is one file that's only outputting 50K frames when It should have 80K+ in it. I know this because the filesize and number of frames that I get out of It does not add up. I can also see the missing gap in the timeline I made In premiere with all cameras synced up from the event.
I guess It's a metadata corruption where the file thinks it only have 50K frames in it even though there is much more data there. Does anyone have any experience with this kind of problem? Is it possible to get the missing files out of the mlv or is this a lost cause?

Got the same amount of frames both from MLVFS and mlv dump/mlv batch, both missing 30K frames...

D_Odell

Hi,

So I tried this evening with a new project, haven't filmed for a while. Now on a new computer, slower, but newer. Running latest MLVFS and Davinci 14.2. When I start Davinci MLVFS stops working, all files goes offline meanwhile Davincis running. When I reload MLVFS it works, then power Davinci and it goes offline.. Anyone experienced this? I can't seem to get around it? Do I need to use another tool than MLVFS?

MacOS Sierra 10.12.6
MLVs is compressed and normal MLVs.

Thanks a bunch!
David
5D3 [size=6pt](OLPF removed)[/size] :: 1.1.3 :: Canon FD L Serie

Markus

@D_Odell
I always convert mounted dngs with slimraw and put them on disk. Working directly with mounted dngs isn't stable with MLVFS on Windows either. It was pretty stable and faster with pismo but no support except standard 14bit raw there and mlvfs has all metadata handling working correctly.

D_Odell

Quote from: Markus on April 23, 2018, 01:59:44 PM
@D_Odell
I always convert mounted dngs with slimraw and put them on disk. Working directly with mounted dngs isn't stable with MLVFS on Windows either. It was pretty stable and faster with pismo but no support except standard 14bit raw there and mlvfs has all metadata handling working correctly.
Thanks for replying. I see the problem being much less hassle with converted DNGs of course, but since I really love MLVFS and virtual DNGs its a blast for storage and low budgets. So the solution to get it fixed again, it wasn't MLVFS but Davinci 14... Going back to 12.55 it works like a dream again.

Weird..

Best,
David
5D3 [size=6pt](OLPF removed)[/size] :: 1.1.3 :: Canon FD L Serie

ArcziPL

Quote from: clanlee on April 04, 2018, 11:36:34 PM
MLVFS_x64
Download http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip
Extract MLVFS_x64.zip

Hi, is there a newer windows build available, supporting MLV with lossless JPEG compression? Tried already several builds, out of which this one seems to be the newest one, but only uncompressed MLV are opening properly.


EDIT:
huh, here it is...

https://www.magiclantern.fm/forum/index.php?topic=19429.0

https://bitbucket.org/bouncyball/mlv-tools/downloads/

Thank you, bouncyball!


All in all it's a fantastic tool! But all the docus are totally outdated...
M50.110 [main cam] | G7X III [pocket cam] | 70D.112 [gathers dust] | M.202 [gathers dust] | waiting for M5II

papasapien

hey can someone help me?my mlvfs isnt working. i do the correct command lines and it just takes me back to the root folder like nothing ever happened please help. it worked a couple months ago but now im just coming back to it and it doesnt work.

sys73r

hey guys, I have been using for a long time the stable build recording 1080p with sound for long time and post processing with the latest MLVFS without any issues.

Today I'm testing the last build crop_rec4k.2018Mar10 with mlv_lite and mlv_snd to try  to record 3.5K with sound, It records fine but when I post process the image I see no sound at all. So think it's because the version of MLVFS I have is no longer able to grab the data from the latest files.

is there any new version or patch I can install for OSX to keep using MLVFS?



bouncyball

Here

However if you use as you said latest MLVFS this build (crop_rec4k.2018Mar10) does something different.

freely

Hello.

It seems trivial but I found something a little cumbersome for me. If you select Naming Scheme as DaVinci Resolve in MLVFS, "MLV" is inserted in the middle of the file name.





It is not included in other conversion programs (MLV App, Switch).



Here is my mlv file.
https://drive.google.com/open?id=1GqMSKjZ-EqFx3KaJG6XzJ2zBs6AMFYS-

I used https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS.dmg

I'm not sure, but I think this is a bug in MLVFS, what's your opinion? If this is a bug, I hope it is fixed.

Thank you.

Danne

It´s not a bug. Bouncyball once helped me find how to exclude it. Somewhere along these lines. FIddle with it and recompile MLVFS and it should work:

Something like following. Can´t remember exactly:
In main.c changed to this to erase the MLV part from the name to match Resolve naming scheme with other cams for instance

sprintf(*mlv_basename, "%s_1_%d-%02d-%02d_%04d_C%04d", start, 1900 + frame_headers.rtci_hdr.tm_year, frame_headers.rtci_hdr.tm_mon + 1, frame_headers.rtci_hdr.tm_mday, 1, 0);

....

static int get_mlv_name_from_basename(const char *path, char ** mlv_name)
{
    if(mlvfs.name_scheme == 1)
    {
        struct slre_cap caps[2];
        memset(caps, 0, sizeof(caps));
        if(slre_match("(.+)_1_\\d+-\\d+-\\d+_\\d+_[C|c]\\d+", path, (int)strlen(path), caps, 1, 0) >= 0)
        {
            char * result = get_capture(caps[0]);
            *mlv_name = concat_string(result, ".MLV");
            free(result);
            return 1;
        }
        else
        {
            return 0;

freely

Thank you.
Thanks to the guide, I was able to get what I wanted. Thank you.

In addition, I would like to tell you what I have learned and what I have thought about in this process.
I did a search on Google with the DaVinci Resolve Naming Scheme. So I found something like the following on the following page.
https://forum.blackmagicdesign.com/viewtopic.php?f=2&t=32836



You can see that this is named CAMERA ID / Date / Time / Clip #. After knowing this, there is one question to be asked. For example, it has the name M13-1450_1_2018-06-13_0001_C0000. Why do MLVFS, MLV App, and Switch all display time information in clip name as 0001? Personally, it is more intuitive to display the time information than the meaningless information of 0001, so I think that there will be some profit because I can know the shooting time.

I tried to modify some of the code as a test and changed the name like M13-1450_1_2018-06-13_1450_C0000.

sprintf(*mlv_basename, "%s_1_%d-%02d-%02d_%02d%02d_C%04d", start, 1900 + frame_headers.rtci_hdr.tm_year, frame_headers.rtci_hdr.tm_mon + 1, frame_headers.rtci_hdr.tm_mday, frame_headers.rtci_hdr.tm_hour,  frame_headers.rtci_hdr.tm_min, 0);





I think the naming scheme is more useful for displaying time information. Then I thought that "_1" after the camera "M13-1450" had no meaning, so I thought it seemed unnecessary.

I tried it and found that it was possible.



For your reference, let me tell you why I want to remove "MLV" from the clip name. My system will drop frames when playing clips loaded with MLVFS. So I have to convert it to dng for smooth editing. When I am done editing, the dngs are deleted, and sometimes I need to make some changes later. At this time, it is too time-consuming and energy-intensive to convert all back to dng, so I want to load it into MLVFS instead. At this time, if the clip names are the same, I can simply continue the work by specifying a new folder.

Thank you for your cooperation.




Danne

Or maybe it could simply be:
M13-1450_2018-06-13_C0000_000000

papasapien

do i need windows 10 professional Intel virtualization technology to run mlfvs on windows?

briwil

Quote from: clanlee on April 04, 2018, 11:36:34 PM
MLVFS Installation on Windows 10 x64

Install latest version of Dokany:
https://github.com/dokan-dev/dokany/releases

Reboot if necessary.

MLVFS_x64
Download http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip
Extract MLVFS_x64.zip

Delete the following files from the extracted folder:
dokan1.dll
dokanfuse1.dll

Using MLVFS:
Note the path of the extracted MLVFS_x64.exe
Open Command Prompt

Run program from within directory into the executable:
The web gui interface only works if you call the executable from the same directory
cd mlvfs_x64
mlvfs_x64.exe Z:\ --mlv-dir=C:\Videos\Files

Run from any location by typing full path to executable:
If you run this from a remote directory, the web gui will not work
C:\mlvfs_x64\mlvfs_x64.exe Z:\ --mlv-dir=C:\Videos\Files

If your folder name contains spaces, use quotation marks for path:
"C:\Extracted files\mlvfs_x64\mlvfs_x64.exe" Z:\ --mlv-dir="C:\Videos from Camera\Files"

Or:
C:\mlvfs_x64\mlvfs_x64.exe Z:\ --mlv-dir="C:\Videos from Camera\Files"

Or:
"C:\Extracted files\mlvfs_x64\mlvfs_x64.exe" Z:\ --mlv-dir=C:\Videos\Files

Notes:
The Z:\ option is setting a mount point.  You'll have to use an unused drive letter in your computer.  When you open Windows Explorer, you'll see a list of all the drives in your computer.  Use a drive letter that is not used.

I've done all the steps here, and everything seems to work, except all my DNG's end up looking like pink noise, even if I open in Photoshop. What am I missing here?
On Windows 10, btw.