MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[dmilligan]

new builds uploaded
* fixed the webgui issue on windows (windows file paths contain "\" and json doesn't like that)
* streamlined buffer management (and eliminated 'prefetch' option)
* lots of stability improvements from g3gg0


still todo: embed git version info on the Windows version like the Mac/Linux version

[mothaibaphoto]

Last mlvfs.exe with Resolve:

Code: [Select]

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>cd /d D:\_soft\4\MLVFS\MLVFS_x86

D:\_soft\4\MLVFS\MLVFS_x86>mlvfs.exe X: --mlv-dir=Z:\out
mlvfs_wrap_read(1426): caught exception 0xC0000005

D:\_soft\4\MLVFS\MLVFS_x86>
Resolve can be downloaded here
Start with any new project, Media Storage - try to navigate to any MLVFS folder - immediate crash of mlvfs.exe

[bouncyball]

Hi Again,

Problem 1
E:\MLVFS>mlvfs E:\DNG --mlv-dir=E:\MLV
mlvfs_wrap_read(1440): caught exception 0xC0000005
mlvfs_wrap_read(1440): caught exception 0xC0000005

no difference if you mount it to drive letter or empty dir.
exceptions mostly caught when accessing WAV file. Wave shows correct size but I can not copy it. However I could copy it with the previous mlvfs version.

Problem 2
It seams that MLVs bigger than 2048mb (also MLVs + M00...) can not be mounted at all?! they are not shown on dir list. Checked all available versions of mlvfs.

OS: Windows 10 Pro x64, Dokan 0.8 (checked Dokan 1.0-beta and it shows the same behavior)

@dmilligan: Webui works. BTW it lists every MLV even the >2gb ones which are not mounted (seen) in file explorer.

regards
bb

[g3gg0]

please redownload mlvfs.exe and try.
it will still crash but show an address that might help to find the exact crash position.

meanwhile i am downloading resolve.
guess it's some off-by-one-error or such with the wave decoder that causes the troubles.

[mothaibaphoto]

No output to CMD window, problem details from popup:

Code: [Select]

Problem signature:
  Problem Event Name: APPCRASH
  Application Name: mlvfs.exe
  Application Version: 0.0.0.0
  Application Timestamp: 56b89cfa
  Fault Module Name: StackHash_8e87
  Fault Module Version: 6.3.9600.17415
  Fault Module Timestamp: 54504b0d
  Exception Code: c0000374
  Exception Offset: PCH_80_FROM_ntdll+0x0003CEEC
  OS Version: 6.3.9600.2.0.0.256.4
  Locale ID: 1049
  Additional Information 1: 8e87
  Additional Information 2: 8e87aeb06e784aea107beb1d064656d5
  Additional Information 3: dc1c
  Additional Information 4: dc1c75fae54762c904db111a92126c62


[bouncyball]

Just accessing dir or accesing wav:
E:\MLVFS>mlvfs_debug E:\DNG --mlv-dir=E:\MLV
mlvfs_wrap_read(1440): caught exception 0xC0000005 at address 0x739AE21C

Sometimes (very rearly) DokanMounter service stops and needs to be started manually.
E:\MLVFS>mlvfs_debug E:\DNG --mlv-dir=E:\MLV
Unmount

!!! Checked on 3 PCs MLVs >2GB are not mountig at all and can not be seen in file explorer (win10prox64, unfortunatelly have no win7 or 8.1 to test it on) !!!

@mothaibaphoto: What ver of win do you use and do you have the problem with MLVs bigger than 2GB

bb

[mothaibaphoto]

@mothaibaphoto: What ver of win do you use and do you have the problem with MLVs bigger than 2GB

1. win 8.1x64
2. Yes.

Code: [Select]

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>dir X:\
 Volume in drive X is DOKAN

 Directory of X:\

05.02.2016  16:18    <DIR>          M05-1617.MLV
05.02.2016  16:19    <DIR>          M05-1619.MLV
05.02.2016  16:20    <DIR>          M05-1620.MLV
05.02.2016  16:21    <DIR>          M05-1621.MLV
05.02.2016  16:29    <DIR>          M05-1629.MLV
05.02.2016  16:30    <DIR>          M05-1630.MLV
05.02.2016  16:31    <DIR>          M05-1631.MLV
05.02.2016  16:36    <DIR>          M05-1636.MLV
               0 File(s)              0 bytes
               8 Dir(s)               0 bytes free

C:\Users\VOLD>dir Z:\out
 Volume in drive Z is EVO850
 Volume Serial Number is F873-BE21

 Directory of Z:\out

08.02.2016  00:30    <DIR>          .
08.02.2016  00:30    <DIR>          ..
08.02.2016  00:30            11 836 M05-1617.IDX
05.02.2016  16:18       396 013 620 M05-1617.M00
05.02.2016  16:18     1 714 848 352 M05-1617.MLV
07.02.2016  20:54            31 216 M05-1618.IDX
05.02.2016  16:19     1 137 176 628 M05-1618.M00
05.02.2016  16:19     4 599 571 040 M05-1618.MLV
07.02.2016  17:55             1 684 M05-1619.IDX
05.02.2016  16:19        61 763 636 M05-1619.M00
05.02.2016  16:19       207 090 272 M05-1619.MLV
07.02.2016  17:55            12 088 M05-1620.IDX
05.02.2016  16:20       421 445 684 M05-1620.M00
05.02.2016  16:20     1 729 380 960 M05-1620.MLV
07.02.2016  20:54            14 248 M05-1621.IDX
05.02.2016  16:21       486 842 420 M05-1621.M00
05.02.2016  16:21     2 070 897 248 M05-1621.MLV
07.02.2016  20:54            31 384 M05-1622.IDX
05.02.2016  16:22     1 104 478 260 M05-1622.M00
05.02.2016  16:22     4 588 671 584 M05-1622.MLV
07.02.2016  20:54            23 056 M05-1623.IDX
05.02.2016  16:24       802 926 644 M05-1623.M00
05.02.2016  16:24     3 436 962 400 M05-1623.MLV
07.02.2016  20:54            60 280 M05-1624.IDX
05.02.2016  16:27     2 132 660 276 M05-1624.M00
05.02.2016  16:27     8 875 790 944 M05-1624.MLV
07.02.2016  20:54             3 568 M05-1629.IDX
05.02.2016  16:29       153 260 084 M05-1629.M00
05.02.2016  16:29       672 367 200 M05-1629.MLV
07.02.2016  20:54             2 464 M05-1630.IDX
05.02.2016  16:30        98 877 492 M05-1630.M00
05.02.2016  16:30       449 892 960 M05-1630.MLV
07.02.2016  20:54             2 944 M05-1631.IDX
05.02.2016  16:31       168 091 700 M05-1631.M00
05.02.2016  16:31       726 749 792 M05-1631.MLV
07.02.2016  20:54             2 740 M05-1636.IDX
05.02.2016  16:36        79 102 004 M05-1636.M00
05.02.2016  16:36       543 826 528 M05-1636.MLV
07.02.2016  20:54            15 832 M05-1639.IDX
05.02.2016  16:40       821 092 404 M05-1639.M00
05.02.2016  16:40     2 395 668 480 M05-1639.MLV
              39 File(s) 39 875 661 952 bytes
               2 Dir(s)  24 641 163 264 bytes free

C:\Users\VOLD>


[g3gg0]

WAV: nailed it down and fixing it.

[g3gg0]

please redownload mlvfs.exe and try.

[g3gg0]

4GiB files also fixed: please redownload mlvfs.exe and try.

[mothaibaphoto]

Good news:
Resolve imports and plays great, with sound.
Not so good:
Big files still ignored. The directory list I posted recently looks the same.
Several consecutive refreshing of webGUI with changing parameters causes smiley BSOD. Reproduced several times - comp. restart - webGUI - attempt to save anything - BSOD.

[bouncyball]

@g3gg0: Great! Wav plays nicely . However, as mothaibaphoto mentioned above MLVs >2GB ignored again. Can you compile the test binary against the Dokan version 1.0-beta1? The new version gets rid of a DokanMounter service and adds windows mount manager support.

Several consecutive refreshing of webGUI with changing parameters causes smiley BSOD. Reproduced several times - comp. restart - webGUI - attempt to save anything - BSOD.

Think I nailed exact reason of it. Switching "Naming Scheme" from "Deafult" to "DaVinci Resolve" and after that accessing folders, every time, ends up with exception. Sometimes restarting DokanMounter service helps and you can remount the folder, but sometimes OS freezes and only hard reset is a solution.
Other parameters in webgui are working just fine and only "Fix Pattern Noise" slows the process down a little bit. I think It's a normal behavior.

bb

[g3gg0]

@bouncyball:
tried with a forged 19GiB file and that works well.
will have to do some tests what may cause that.

will try 1.0

[bouncyball]

tried with a forged 19GiB file and that works well.

hmm... will try to record some huge mlvs on exfat. From ~50 test files of mine worked none which is bigger than 2048mb. They just do not show up in mounted folder.

[bouncyball]

@dmilligan:
What does exacty switching "Naming Scheme" from "Default" to "DaVinci Resolve"? Changes some cdng metadata in favor of resolve? Or something else?

[dmilligan]

It changes how the DNGs that show up in the mount folder are named. If you use a particular naming convention that Resolve likes (one that mimics the naming convention used by BM cameras), then the parent folders of cDNGs show up as video clips in media browser.

It makes sense that it would be the source of issues on Windows since file naming rules are different (case insensitive, different path separator, etc.), and it's adding a layer of complexity on top of the path translations mlvfs is doing (FUSE sends mlvfs a file path, mlvfs has to figure out what data to send back).

forged 19GiB file

Heh, this made me think of using FUSE to generate fake, huge MLV files for testing. It could just take an existing small MLV file and basically just repeat the frame chunks over and over. Then we can have huge testing MLV files that don't waste a lot of hard drive space.

[DeafEyeJedi]

Not trying to be the party pooper in here for Windows users ... I'm very excited for all of you and it's been a long wait!

However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work and how can I help troubleshoot this or at least get a log for you?

[dfort]

...If you use a particular naming convention that Resolve likes (one that mimics the naming convention used by BM cameras)...

Speaking of DaVinci Resolve Naming Scheme and Black Magic camera file names--

I shot this test today and the filename has a date for last month. The date is correct in the browser window.

More on DaVinci Resolve Naming Scheme--I found this information about how the file name is formatted:

File Naming Convention
The Blackmagic Cinema Camera uses the following file naming convention when recording video.
[Camera ID]_[Reel Number]_[yyyy-mm--dd]_[hhmm]_C[Clip number].mov
This is an example of how this would appear on the touch screen of the camera:
BMC01_1_2012-08-08_1631_C0002.mov QuickTime Movie Filename
BMC01_1_2012-08-08_1631_C0002.mov Camera ID
BMC01_1_2012-08-08_1631_C0002.mov Reel Number
BMC01_1_2012-08-08_1631_C0002.mov Date (2012 Aug 08)
BMC01_1_2012-08-08_1631_C0002.mov Time (16:31pm - 24hrs)
BMC01_1_2012-08-08_1631_C0002.mov Clip Number
For CinemaDNG files, the folder of the image sequence will also be named the same way.

If you check discussions from people that are using these files in post you'll find a lot of dissatisfaction. Besides the long filename the information that editors need to see, the clip number, is at the end of the name and depending on the editing system it might be cut off.

Still, the date should be correct and it would be an improvement to add the time.

If we take an MLV file as it comes off the camera:
M02-0828.MLV

The Black Magic naming would probably go something like this:
M_02_2012-08-08_1631_C0828

I'm not suggesting such a radical change because it isn't a very practical name for most editing systems and the "M" is way too generic for a camera name but at least the fields in the name have some meaning instead of being padded out with zeros. Just something to think about.

Maybe something like this?
M02-0828_02_2012-08-08_1631_C0828

[Danne]

How is folder/audio embed working with the dng files in the case of shortening the folder name but not the dng files? The longer extension of zeroes are there to give each frame a sequential number.

[dmilligan]

I shot this test today and the filename has a date for last month. The date is correct in the browser window.

Ah yes. You found a bug The tm.month field is zero based (0-11), so we need to add 1 before we print it out. This is already corrected in some places, but you found a place it was wrong.

However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work

I think I know what's wrong: forgot to include some resource files in the Mac build. Will fix.

[g3gg0]

@4Gib:

for me it works :-S
no clue what the problem really is.

[dmilligan]

However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work and how can I help troubleshoot this or at least get a log for you?

I think I know what's wrong: forgot to include some resource files in the Mac build. Will fix.

Actually, everything seems fine. Web GUI works just fine here.

[g3gg0]

@dmilligan:
get_mlv_filename needs to be reworked.
currently it's not possible to create files in directories within a MLV file.
and we have to use dokan v1 since file creation doesnt work with 0.8.

[dmilligan]

and we have to use dokan v1 since file creation doesnt work with 0.8.

v1 doesn't seem to work on my win7 VM, fuse_main() just returns 1 immediately

get_mlv_filename needs to be reworked.

I think all the path handling should be much better now

[g3gg0]

oh thats odd. works fine on my win 10.

wow you did a lot of improvements. nice.

the problem with virtual folder is still persistent.
for a requested file:

/path/to/some/file.mlv/with/some/subfolder/leads/to.txt

we have to iterate over every element and check if it is a (existing!) MLV.

/path
/path/to
/path/to/some
/path/to/some/file.mlv

and cut the remaining part
/with/some/subfolder/leads/to.txt

and recombine it again with the MLD
/path/to/some/file.MLD/with/some/subfolder/leads/to.txt