MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[bouncyball]

@dmilligan, g3gg0
You really did some magic to the source code

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb

[g3gg0]

oh i am glad to read this.
then we are on a good path

[mothaibaphoto]

@dmilligan, g3gg0
You really did some magic to the source code

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb

Redownloaded the exe after reading this and everything the same (:

[bouncyball]

Redownloaded the exe after reading this and everything the same (:

Have you downloaded this file?
https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip

[bouncyball]

Did some test copy of 4,918,485,826 bytes (1168 files) from one SSD to second SSD

result (average of 3 runs):
Dokan 1.0 - 58mb/sec, mlvfs.exe consumed ~13% of CPU (Intel Xeon E3 @ 3.4Ghz)
Dokan 0.8 - 52mb/sec, mlvfs.exe consumed ~12% of CPU

A little bit subjective and depends on hardware but v1.0 has some performance boost (maybe new fs driver optimizations?)

bb

[mothaibaphoto]

No, This one, as posted by g3gg0
I tried this one too: it asks for vcruntime140d.dll, I find it and put to program dir, than ucrtbased.dll, than...
it just show the message "The application was umable to start correctly (0x000007b)." Click OK to close.

[bouncyball]

yes it's linked against debug version of dlls.

get 2 files below and drop to the saem dir with mlvfs.exe:
http://www.dll-files.com/dllindex/dll-files.shtml?vcruntime140d
http://www.dll-files.com/dllindex/dll-files.shtml?ucrtbased

oh forgot to mention get 32bit versions

[mothaibaphoto]

I find these files myself on that site, but it anyway don't run with message from my previous post.

[bouncyball]

If you got 32bit versions and still have an error message then something messed up in your system with ms c++ redistributables.

[mothaibaphoto]

Ups, it seems I mixed versions, now it works and I see all the files !!!
Great thanks !!!
Edit:
I think it's meaningless to measure performance of the debug version: it definitely less. With that version Resolve plays with 24 FPS(real time) with this - 14.

[bouncyball]

Good point. We'll see... happy bright future is almost over here! Not only for MAC users

Just kidding, have 2 old hackintoshes myself. But love doing/testing video/audio things under win or linux.

[g3gg0]

125MiB/s for 19GiB with plain old HDD and my x64 build for dokan v1.0-rc

[mothaibaphoto]

Wow...
And where did you get that dokan v1.0-rc?
here:
https://github.com/dokan-dev/dokany/releases
only 1.0.0-BETA1

[g3gg0]

yeah sorry, its the beta1, you are right.
thats the correct one.

here some more optimized builds
choose the one that works for your CPU.

AVX2 is the best of course, but not all CPUs support it.

[mothaibaphoto]

Congratulations, you did a great job,
Great thanks to everyone involved!!!
Now it would be perfect if some more people try this asap to localize possible issues.
I tried for now only 64 build without avx(as i have old bloomfield) -
Resolve plays 30 fps FHD project in full screen, cpu load 90%(but 75% by Resolve itself), video - 40%,
I can't estimate disk load as Task manager shows 0%(probably due to indirect access).
Edit:
And regarding that Naming Scheme - i can't reproduce anymore with 12.3 version of Resolve that "show but not play" behavior. Maybe they relaxed naming requirements.

[g3gg0]

@bouncyball:
can you do a another comparison of the MLVFS speed with the optimized builds?
as you seem to have rea-world footage and a high end machine i think thats a good basis for comparisons

i have those variants:

x86 vs. x64  [CPU architecture]
nopar vs. par [multicore parallelization]
noSSE, SSE, SSE2 on x86 [vectoring engine]
noAVX, AVX, AVX2 on x64 [vectoring engine]

i am the most curious about the nopar/par performance boost (if any).
so if you want to do a minimum setup, it would be the best you choose x64/avx2 and compare nopar vs. par.
if the tool crashes, your CPU probably doesnt support avx2, try avx then.

[Ottoga]

@g3gg0

I have ben watching this thread with interest for some time now and decided to try mlvfs.

My Gear:
---------------------------------------
CPU:            Intel i5-3317U quad core
Ram:           6GB
Graphics:     Intel HD 4000
OS:             w10 Pro 64bit updated with the most recent patches.

I did have Pismo file mount for Windows installed but uninstalled it to avoid any potential conflicts with DOKAN.

The initial installation of DOKAN (v1) displayed the following message;

     http://1drv.ms/1WfsIs3
    (note the screen shot was from v0.800 but apart from the version number in the message, the message was the same).

I clicked "YES" then proceeded to download and install MS Visual C++ - Redistributable (x64). It installed without error.

DOKAN v1 appears to have installed correctly as verified via the cmd line screenshot below.

     http://1drv.ms/1KGNvnA
     Note: version was checked before and after manually starting the Driver and network service. Given the absence of error messages I
              assume that this is working correctly.

When I try to launch MLVFS.exe I initially had errors re: vcruntime140.dll and msvcp140.dll not found yet both of these files were in the C:\Windows\System32 directory. Based on some google research I moved copies of these files into the C:\Windows\SysWOW64 directory.

Subsequent to this the "dll not Found" error messages no longer appears but the following one does.
     http://1drv.ms/1KGO0xG

I acknowledge the error and the application fails to launch. Note that I have tried to launch it at the cmd line level with the same results.

Any thoughts, ideas? Is there an architecture dependency (e.g.: graphics card) that I'm missing?

Edit add;  If I run dokanctl.exe from the C:\Program Files\Dokan\DokanLibrary directory it appears to work correctly. However,
                 If I run it from the C:\Program Files (x86)\Dokan\DokanLibrary directory, it fails with the same error as mlvfs.exe.


Might be a clue to what is going on.


Cheers... Otto

[mothaibaphoto]

@Ottoga:
1. Docan install always ask to install VC Redist, whether it installed, or not. No problem to reinstall it.
2. The message you got after manually adding dll files in respond to  "dll not Found" meaning you mix versions, ie, run 32 bit exe with 64 bit dll or vice versa.
To fix everything try:
1. Discard all the dll files you copied manually.
2. Install/reinstall Docan & VC redist. No need to run anything manually.
3. Unpack Optimized builds
to directory where you previosly unpacked mlvfs.zip.
4. As you on 64 win, run 64-bit exe, mlvfs_x64_rc.exe or even mlvfs_x64_rc_avx2.exe as your processor seems support that.
Make text file with 2 next rows and save it with .bat extension:
cd /d path_to_your_mlvfs_x64_rc.exe
mlvfs_x64_rc X: --mlv-dir=path_to_your_MLV_dir
where X - the Letter for disk not yet present in your computer.
Then just run that bat file and configure MLVFS with your web browser pointing it to http://localhost:8000/.
For example, my working bat file:
cd /d D:\PortableSoft\video\bin\mlvfs_optimized
mlvfs_x64_rc X: --mlv-dir=Z:\out

[g3gg0]

@Ottoga:
as mothaibaphoto already wrote - MSVC runtime is a bit bitchy. never just download the DLLs.
always search for "vc_redist" packages when you miss the DLLs.
the optimized builds (which are still experimental of course) do not require the debug versions of the DLLs.

@dmilligan:
can you upload the "release" build of the package?

[squig]

I'm having some issues with the new Mac build. MLV previews aren't visible in the Resolve media window. Folders are visible where there used to be previews. When I go into the folders most of them look empty. I can't load any MLV files into Resolve at all. 

Also when I try to refresh the browser I get a "can't connect to the server" error message.

[squig]

I'm running Yosemite 10.10.3, Davinci Resolve 12.0, Safari 8.06, and Fuse 2.8.2

I went back to the Feb 2015 build: all good. One thing I noticed: my MLV folder takes a few seconds to load (spinning wheel of death). There's no spinning wheel of death showing up when I try to load the MLV folder in Resolve with the new build.

[Licaon_Kter]

VC Redist issues, maybe fix it with the pack: http://forums.guru3d.com/showthread.php?t=393405
(right click run as Admin)

[bouncyball]

@g3gg0

Machine: cpu Xeon E3-1245 @ 3.4ghz, mem 16gb, 2x 120 gig kingston ssds <- they are no highend ones but not so bad anyway
Folder with MLVs reside on one ssd, mounted to drive Z: -> copied to second ssd.
Copied 19,852,533,609 bytes, 4717 files, 5 folders

Without runing mlvfs:
Result of sequential copy of a large file from 1st to second gives ~220mb/sec.
Result of sequential copy of a dir with a real DNGs of the same size gives ~175mb/sec.

results are below (orange line - mlvfs cpu usage):
























regards
bb

[bouncyball]

@g3gg0

I think every binary performance is about the same. Par versions are just CPU hogs (would be nice to test them in random or parralel DNG file access mode). Anyway it is nice to have native 64bit binary and avoiding WOW64
Forgot to mention, test was performed with mlvfs default parameters.

bb

[g3gg0]

@bouncyball:
wow, nice. thanks a lot!