Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter  (Read 939168 times)

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #900 on: February 10, 2016, 10:11:20 AM »
@dmilligan, g3gg0
You really did some magic to the source code :)

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #901 on: February 10, 2016, 10:12:48 AM »
oh i am glad to read this.
then we are on a good path ;)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #902 on: February 10, 2016, 02:06:28 PM »
@dmilligan, g3gg0
You really did some magic to the source code :)

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb
Redownloaded the exe after reading this and everything the same (:

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #903 on: February 10, 2016, 04:38:56 PM »
Redownloaded the exe after reading this and everything the same (:

Have you downloaded this file?
https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #904 on: February 10, 2016, 05:06:40 PM »
Did some test copy of 4,918,485,826 bytes (1168 files) from one SSD to second SSD

result (average of 3 runs):
Dokan 1.0 - 58mb/sec, mlvfs.exe consumed ~13% of CPU (Intel Xeon E3 @ 3.4Ghz)
Dokan 0.8 - 52mb/sec, mlvfs.exe consumed ~12% of CPU

A little bit subjective and depends on hardware but v1.0 has some performance boost (maybe new fs driver optimizations?)

bb

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #905 on: February 10, 2016, 05:10:11 PM »
No, This one, as posted by g3gg0
I tried this one too: it asks for vcruntime140d.dll, I find it and put to program dir, than ucrtbased.dll, than...
it just show the message "The application was umable to start correctly (0x000007b)." Click OK to close.
:(

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #906 on: February 10, 2016, 05:16:36 PM »
yes it's linked against debug version of dlls.

get 2 files below and drop to the saem dir with mlvfs.exe:
http://www.dll-files.com/dllindex/dll-files.shtml?vcruntime140d
http://www.dll-files.com/dllindex/dll-files.shtml?ucrtbased

oh forgot to mention get 32bit versions

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #907 on: February 10, 2016, 05:19:13 PM »
I find these files myself on that site, but it anyway don't run with message from my previous post.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #908 on: February 10, 2016, 05:22:28 PM »
If you got 32bit versions and still have an error message then something messed up in your system with ms c++ redistributables.

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #909 on: February 10, 2016, 05:29:54 PM »
Ups, it seems I mixed versions, now it works and I see all the files !!!
Great thanks !!!
Edit:
I think it's meaningless to measure performance of the debug version: it definitely less. With that version Resolve plays with 24 FPS(real time) with this - 14.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #910 on: February 10, 2016, 06:03:56 PM »
Good point. We'll see... happy bright future is almost over here! Not only for MAC users ;)

Just kidding, have 2 old hackintoshes myself. But love doing/testing video/audio things under win or linux.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #911 on: February 10, 2016, 06:33:15 PM »
125MiB/s for 19GiB with plain old HDD and my x64 build for dokan v1.0-rc ;)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #912 on: February 10, 2016, 06:47:16 PM »
Wow...
And where did you get that dokan v1.0-rc?
here:
https://github.com/dokan-dev/dokany/releases
only 1.0.0-BETA1

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #913 on: February 10, 2016, 06:49:27 PM »
yeah sorry, its the beta1, you are right.
thats the correct one.

here some more optimized builds
choose the one that works for your CPU.

AVX2 is the best of course, but not all CPUs support it.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #914 on: February 10, 2016, 07:44:06 PM »
Congratulations, you did a great job,
Great thanks to everyone involved!!!
Now it would be perfect if some more people try this asap to localize possible issues.
I tried for now only 64 build without avx(as i have old bloomfield) -
Resolve plays 30 fps FHD project in full screen, cpu load 90%(but 75% by Resolve itself), video - 40%,
I can't estimate disk load as Task manager shows 0%(probably due to indirect access).
Edit:
And regarding that Naming Scheme - i can't reproduce anymore with 12.3 version of Resolve that "show but not play" behavior. Maybe they relaxed naming requirements.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #915 on: February 10, 2016, 09:19:06 PM »
@bouncyball:
can you do a another comparison of the MLVFS speed with the optimized builds?
as you seem to have rea-world footage and a high end machine i think thats a good basis for comparisons

i have those variants:

x86 vs. x64  [CPU architecture]
nopar vs. par [multicore parallelization]
noSSE, SSE, SSE2 on x86 [vectoring engine]
noAVX, AVX, AVX2 on x64 [vectoring engine]

i am the most curious about the nopar/par performance boost (if any).
so if you want to do a minimum setup, it would be the best you choose x64/avx2 and compare nopar vs. par.
if the tool crashes, your CPU probably doesnt support avx2, try avx then.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Ottoga

  • Member
  • ***
  • Posts: 203
  • The Ox is Slow but the Earth is Patient
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #916 on: February 11, 2016, 06:05:14 AM »
@g3gg0

I have ben watching this thread with interest for some time now and decided to try mlvfs.

My Gear:
---------------------------------------
CPU:            Intel i5-3317U quad core
Ram:           6GB
Graphics:     Intel HD 4000
OS:             w10 Pro 64bit updated with the most recent patches.

I did have Pismo file mount for Windows installed but uninstalled it to avoid any potential conflicts with DOKAN.

The initial installation of DOKAN (v1) displayed the following message;

     http://1drv.ms/1WfsIs3
    (note the screen shot was from v0.800 but apart from the version number in the message, the message was the same).

I clicked "YES" then proceeded to download and install MS Visual C++ - Redistributable (x64). It installed without error.

DOKAN v1 appears to have installed correctly as verified via the cmd line screenshot below.

     http://1drv.ms/1KGNvnA
     Note: version was checked before and after manually starting the Driver and network service. Given the absence of error messages I
              assume that this is working correctly.

When I try to launch MLVFS.exe I initially had errors re: vcruntime140.dll and msvcp140.dll not found yet both of these files were in the C:\Windows\System32 directory. Based on some google research I moved copies of these files into the C:\Windows\SysWOW64 directory.

Subsequent to this the "dll not Found" error messages no longer appears but the following one does.
     http://1drv.ms/1KGO0xG

I acknowledge the error and the application fails to launch. Note that I have tried to launch it at the cmd line level with the same results.

Any thoughts, ideas? Is there an architecture dependency (e.g.: graphics card) that I'm missing?

Edit add;  If I run dokanctl.exe from the C:\Program Files\Dokan\DokanLibrary directory it appears to work correctly. However,
                 If I run it from the C:\Program Files (x86)\Dokan\DokanLibrary directory, it fails with the same error as mlvfs.exe.


Might be a clue to what is going on.


Cheers... Otto
EOS 7D.203, EFS 55-250mm, EF 75-300 III, Tamron 16-300 DiII VC PZD Macro, SpeedLite 580EX II.

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #917 on: February 11, 2016, 07:55:09 AM »
@Ottoga:
1. Docan install always ask to install VC Redist, whether it installed, or not. No problem to reinstall it.
2. The message you got after manually adding dll files in respond to  "dll not Found" meaning you mix versions, ie, run 32 bit exe with 64 bit dll or vice versa.
To fix everything try:
1. Discard all the dll files you copied manually.
2. Install/reinstall Docan & VC redist. No need to run anything manually.
3. Unpack Optimized builds
to directory where you previosly unpacked mlvfs.zip.
4. As you on 64 win, run 64-bit exe, mlvfs_x64_rc.exe or even mlvfs_x64_rc_avx2.exe as your processor seems support that.
Make text file with 2 next rows and save it with .bat extension:
cd /d path_to_your_mlvfs_x64_rc.exe
mlvfs_x64_rc X: --mlv-dir=path_to_your_MLV_dir
where X - the Letter for disk not yet present in your computer.
Then just run that bat file and configure MLVFS with your web browser pointing it to http://localhost:8000/.
For example, my working bat file:
cd /d D:\PortableSoft\video\bin\mlvfs_optimized
mlvfs_x64_rc X: --mlv-dir=Z:\out

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #918 on: February 11, 2016, 08:31:17 AM »
@Ottoga:
as mothaibaphoto already wrote - MSVC runtime is a bit bitchy. never just download the DLLs.
always search for "vc_redist" packages when you miss the DLLs.
the optimized builds (which are still experimental of course) do not require the debug versions of the DLLs.

@dmilligan:
can you upload the "release" build of the package?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #919 on: February 11, 2016, 08:38:55 AM »
I'm having some issues with the new Mac build. MLV previews aren't visible in the Resolve media window. Folders are visible where there used to be previews. When I go into the folders most of them look empty. I can't load any MLV files into Resolve at all.  :'(

 Also when I try to refresh the browser I get a "can't connect to the server" error message.

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #920 on: February 11, 2016, 08:40:42 AM »
I'm running Yosemite 10.10.3, Davinci Resolve 12.0, Safari 8.06, and Fuse 2.8.2

I went back to the Feb 2015 build: all good. One thing I noticed: my MLV folder takes a few seconds to load (spinning wheel of death). There's no spinning wheel of death showing up when I try to load the MLV folder in Resolve with the new build.

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 519
  • M
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #921 on: February 11, 2016, 11:32:08 AM »
VC Redist issues, maybe fix it with the pack: http://forums.guru3d.com/showthread.php?t=393405
(right click run as Admin)

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #922 on: February 11, 2016, 12:33:21 PM »
@g3gg0

Machine: cpu Xeon E3-1245 @ 3.4ghz, mem 16gb, 2x 120 gig kingston ssds <- they are no highend ones but not so bad anyway :)
Folder with MLVs reside on one ssd, mounted to drive Z: -> copied to second ssd.
Copied 19,852,533,609 bytes, 4717 files, 5 folders

Without runing mlvfs:
Result of sequential copy of a large file from 1st to second gives ~220mb/sec.
Result of sequential copy of a dir with a real DNGs of the same size gives ~175mb/sec.

results are below (orange line - mlvfs cpu usage):
























regards
bb

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 850
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #923 on: February 11, 2016, 12:50:22 PM »
@g3gg0

I think every binary performance is about the same. Par versions are just CPU hogs (would be nice to test them in random or parralel DNG file access mode). Anyway it is nice to have native 64bit binary and avoiding WOW64 :)
Forgot to mention, test was performed with mlvfs default parameters.

bb

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3184
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #924 on: February 11, 2016, 03:48:25 PM »
@bouncyball:
wow, nice. thanks a lot!
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!