Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 942193 times)

g3gg0 · « **Reply #925 on:** February 11, 2016, 03:48:25 PM »

@bouncyball:
wow, nice. thanks a lot!

Ottoga · « **Reply #926 on:** February 11, 2016, 03:56:48 PM »

@mothaibaphoto @g3gg0

Thanks for your replies. I've got it working albeit I'm still driving it manually via the cmd line. I'll set-up the batch file and figure out how to point the application to my web browser after I've had some zzz's

One thought/suggestion: I've had to manually run dokanctl to unmount the mapped drive (i.e.: X) which in turn automatically closes the MLVFS application. However, if I exit the MLVFS application via say ctl/c then I'm unable to unmount the mapped drive letter via Dokanctl.

Does MLVFS monitor for a hidden/undocumented "quit" command that would send the unmount command to DOKAN and then gracefully exit MLVFS? or is that part of the web GUI that I've yet to figure out how to set up?

Other than that, fantastic job porting this to windows to all involved. I've successfully tested it with normal and FRSP mlvs. I use Cyberlink photo and video editing software and both happily imported the dngs from the mounted mlv's. one less step in the workflow. Who could complain about that.

Will play some more with the optimized builds tomorrow. mlvfs_x64_rc_avx2.exe crashes but mlvfs_x64_rc_avx.exe seems to work.

Tomorrows another day. Thanks again gentlemen.

g3gg0 · « **Reply #927 on:** February 11, 2016, 04:00:46 PM »

ok my conclusion:
parallelization for the 12bit->16bit conversion is pointless due to probably thread start overhead.
too much CPU usage and no visible performance boost.

the vectorization. hmm also not convincing.
does anyone have a "clearly visible" performance boost with some version?
guess only x64 gives a proper speedup.

g3gg0 · « **Reply #928 on:** February 11, 2016, 04:03:07 PM »

@Ottoga:
you should be able to just run mlvfs and it will automatically mount it at the drive letter (or folder) you specified.
to unmount you simply close/ctrl-c mlvfs.exe and thats it.
30 seconds later the drive letter should disappear.

Ottoga · « **Reply #929 on:** February 11, 2016, 04:23:38 PM »

@g3gg0

Thanks. Absolutely correct, I just didn't wait long enough to observe this.

bouncyball · « **Reply #930 on:** February 11, 2016, 04:36:57 PM »

@g3gg0: u r welcome

Just did copy same files with microsoft tool "robocopy /MT:8" 8 threads turned on. mlvfs_x64_avx2_nopar vs mlvfs_x64_avx2_par - 2:17min vs 2:14min accordingly.
Result is 3 milisecond difference

bb

g3gg0 · « **Reply #931 on:** February 11, 2016, 05:38:43 PM »

i guess seconds?

nah thats not worth it. hoped there is a lightweight multithreading implementation involved, but seems not.

can you do another try with mlvfs_x64_avx2_par.exe ?
i modified it to use OpenMP. perhaps ist a bit better.
just redownload, the file date is 11.02.2016 17:37

bouncyball · « **Reply #932 on:** February 11, 2016, 07:23:58 PM »

Yup, my bad 3 seconds

Windows 10 standard copy:

robocopy time was 2:18. In both cases max rate was at 158-162mb/sec and min rate 75-85. Rate rise at the beginning ~30sec I guess is a cache effect then it normalizes with some spikes. There are 5 mlvs (under 1gb, under 2gb, under 3gb, under 4gb and more than 4gb). Disk usage in resource monitor shows that 3 or 4 MLV files are open simultaneously for reading by mlvfs process.

bb

g3gg0 · « **Reply #933 on:** February 11, 2016, 08:05:37 PM »

okay thanks again.
so no parallelization method helps speeding up RAW->DNG pixel operations.

g3gg0 · « **Reply #934 on:** February 12, 2016, 09:00:49 AM »

btw. with dokan v0.8 and win10:

if you copy DNGs from the MLVFS and cancel copying (no matter when) does it crash?

or is this only a dokoan v1.0 issue?

bouncyball · « **Reply #935 on:** February 12, 2016, 11:54:50 AM »

u r welcome again

Returned to Dokan v0.8, did tests with x64_avx2_par and x64_avx2_nopar versions ubraptly terminating copy process with ESC. Par version crashed more often.
Once printed "mlvfs_wrap_read(1497): caught exception 0xC0000005 at address 0x7F64C85B" but still running!
Once OS rebooted?!

Installed Dokan 1.0 again, did same tests (also copying to the not cleaned dest dir) par version crashed several times.
Once printed "smlvfs_wrap_read(1497): caught exception 0xC0000005 at address 0x2627C85B" but still running.
Nopar never crashed.

Switched to x64_noavx_nopar version. Several times ran 3 copy processes simultaneously from the same source to the same destination (file overwrite on) and terminated with ESC or mouse (X) click. No crashes.

bb

g3gg0 · « **Reply #936 on:** February 12, 2016, 05:51:33 PM »

ok another speedup.
reduced the instruction count for RAW->DNG pixel conversion from 18-20 to 14 instructions on x64.
x86 should also benefit, its now 15 instructions.

here the optimized builds

bouncyball · « **Reply #937 on:** February 12, 2016, 08:39:15 PM »

robocopy no threads

new builds:
2:35 mlvfs_x64.exe
2:36 mlvfs_x64_avx.exe
2:36 mlvfs_x64_avx2.exe

old builds:
2:37 mlvfs_x64_noavx_nopar.exe
2:38 mlvfs_x64_avx_nopar.exe
2:37 mlvfs_x64_avx2_nopar.exe

Don't even know what to say

I guess 4717 files are not sufficient to measure the good difference and I'm too lazy to test it with 15K

In real usage prog is quite snappy and rearly crashes. Used with resolve and it's fine.

Weekend is coming!

Will test on my haswell 4790k 4.5ghz in premiere and after effects at home tomorrow.

Thanx g3gg0
bb

Licaon_Kter · « **Reply #938 on:** February 12, 2016, 10:53:08 PM »

Those pesky kids and their Windows builds...

Linux now fails

Somewhat expected: https://bitbucket.org/dmilligan/mlvfs/commits/097ffc1508d3daf7b827d4bfb1b39247debefc9b

Code: [Select]


gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 -DVERSION="\"584941c\"" -DBUILD_DATE="\"2016-02-12 21:51:31\"" -DFUSE_USE_VERSION=26 main.c dng.o index.o wav.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose/mongoose.o webgui.o resource_manager.o lj92.o gif.o patternnoise.o LZMA/7zAlloc.o LZMA/7zBuf.o LZMA/7zBuf2.o LZMA/7zCrc.o LZMA/7zCrcOpt.o LZMA/7zDec.o LZMA/7zFile.o LZMA/7zIn.o LZMA/7zStream.o LZMA/Alloc.o LZMA/Bcj2.o LZMA/Bra.o LZMA/Bra86.o LZMA/BraIA64.o LZMA/CpuArch.o LZMA/Delta.o LZMA/LzFind.o LZMA/Lzma2Dec.o LZMA/Lzma2Enc.o LZMA/Lzma86Dec.o LZMA/Lzma86Enc.o LZMA/LzmaDec.o LZMA/LzmaEnc.o LZMA/LzmaLib.o LZMA/Ppmd7.o LZMA/Ppmd7Dec.o LZMA/Ppmd7Enc.o LZMA/Sha256.o LZMA/Xz.o LZMA/XzCrc64.o -pthread -lfuse -lm -o mlvfs
main.c: In function ‘mlvfs_getattr’:
main.c:1017:23: error: storage size of ‘mlv_stat’ isn’t known
         struct stat64 mlv_stat;
                       ^
main.c:1019:26: warning: implicit declaration of function ‘stat64’ [-Wimplicit-function-declaration]
         int mlv_status = stat64(mlv_filename, &mlv_stat);
                          ^
main.c:1017:23: warning: unused variable ‘mlv_stat’ [-Wunused-variable]
         struct stat64 mlv_stat;
                       ^
Makefile:35: recipe for target 'mlvfs' failed

g3gg0 · « **Reply #939 on:** February 12, 2016, 11:01:56 PM »

well, things develop step by step.

add a
#define stat64 stat
and try again

Licaon_Kter · « **Reply #940 on:** February 12, 2016, 11:09:15 PM »

Looks like that was that... 10q

The mlvfs file is now ~40kb smaller ( 466kb vs 504kb ), is that expected?

Should I bother with a PR or ?

Ottoga · « **Reply #941 on:** February 13, 2016, 01:03:34 AM »

A possible bug and some questions please.

Today I have launched MLVFS_x64_avx to a local directory containing a single MLV file mapped as drive X: All is well as confirmed via the localhost:8000 view in my browser.

http://1drv.ms/1o7Mtqk

While I was randomly viewing individual frames (by double clicking on them) I noticed in the MLVFS cmd screen a number of loadchunk error messages. These related to mlv files tested yesterday that reside in a different (external HD) location. This suggests that MLVFS is storing previously analysed MLV file names/details somewhere. In itself, these error messages are not an issue however from a performance perspective MLVFS probably shouldn't be trying to open MLV files other than those in the mapped directory.

http://1drv.ms/1o7OHpy

Edit:
The generated errors messages may have been my fault. I had originally launched:
mlvfs_x64_avx2.exe X: --mlv-dir=H:\Video\20150825_Banff_Surrouds\MLVs (where the 4 MLVs related to the error messages reside - it crashed), I then launched

mlvfs_x64_avx.exe X: --mlv-dir=D:\MLVs (where the single MLV resided)

As there was no appreciable delay between the crashing of the mlvfs_x64_avx2 instance vs the launching of the mlvfs_x64_avx instance it suggests that maybe I should
have waited a bit longer to allow the first instance to clean up after itself.

Questions

Where does MLVFS store details of historical MLVs processed? (Maybe irrelevant).
Configuration options set in the localhost view don't appear to be persistent. Is this the expected behaviour?
Having set the configuration options in localhost at what point will they apply e.g: on the fly upon opening, viewing or extracting the frames?
MLVS caters for a number of parameters:
- what are the defaults settings?
- what the purpose of --port and is its sub-parameter a directory name or port number (or both)?

Possibly the above questions could be answered by adding a little more commentary to the options list provided when running MLVFS without parameters.

http://1drv.ms/1TgUrdK

Thanks in advance.... Otto

dmilligan · « **Reply #942 on:** February 13, 2016, 04:50:41 AM »

--port is the port number for the webserver, default is 8000, it's so that you can have multiple instances running at the same time

default values are the ones you see in the webgui initially

Ottoga · « **Reply #943 on:** February 14, 2016, 02:06:09 AM »

@dimilligan.... Thanks

jogodedentro · « **Reply #944 on:** February 14, 2016, 03:55:37 AM »

just reached this forum after having no joy converting mlv files for use with davinci resolve, there seems to be a solution in these threads, but my understanding of coding and command line use isn't great, but please can someone send me a command line sequence that has worked for them in a similar set-up to me, I have windows 10 64bit, I have already downloaded Dokany and mlvfs_86 and the optimized mlvfs zip, I also have oracle virtual box. How do I get from having all this to successfully converting mlv files for use with davinci resolve....thanks to all for working on this.

Ottoga · « **Reply #945 on:** February 14, 2016, 04:30:36 AM »

@jogodedentro

I had problems with a flaky Visual studio Redistribution 2015 runtime library install that was causing the application to fail.

The problem was fixed by re-installing all of the MS redistributable runtimes via the package advised to me by Licaon_Kter in the following post:
http://www.magiclantern.fm/forum/index.php?topic=13152.msg162133#msg162133

The following post contains guidance by Mothaibaphoto on how to run it.
http://www.magiclantern.fm/forum/index.php?topic=13152.msg162127#msg162127

There is a wealth of information within this topic that will help with your understanding. My suggestion, if you haven't already done so is, read the 1st couple of posts and then as the Windows developments are more recent go to the last post and work your way backwards.

A small tip, If you have already installed dokany and MLVFS open a cmd line window. Then:

for Dokany - cd /d C:\Program Files\Dokan\DokanLibrary
dokanctl.exe [enter]
and

for MLVFS - CD to wherever you have unpacked MLVFS to
mlvfs.exe [enter]

Both applications will present you with a list of applicable parameters.

Hope this helps.

jogodedentro · « **Reply #946 on:** February 14, 2016, 05:52:38 AM »

thanks for your reply, yes I can see the options for DokanyLibrary using command prompt, so I assume I need to create a mount point but I can't tell how you do that from the options listed, what option in the DokanyLibrary is used to create a mount point?

Ottoga · « **Reply #947 on:** February 14, 2016, 06:50:42 AM »

@jogodedentro

2nd link in my original post to you.

bouncyball · « **Reply #948 on:** February 14, 2016, 12:29:11 PM »

SemiOffTopic info

Bad news for windows users using SlySoft's great image mounter "Virtual CloneDrive". Dokan 1.0-BETA driver conflicts and renders useless that mounter. after dokanctl /r d (removing driver) and rebooting things are sorting out. No problem with Dokan 0.8 though (0.8 version has it's own issues, and with latest version of mlvfs I'd recommend 1.0)

bb

Walter Schulz · « **Reply #949 on:** February 14, 2016, 12:34:19 PM »

That's a pity for Windows 7 users! Are other image mounters affected as well?

News:

Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 942193 times)