MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[Danne]

Try this:

Code: [Select]

mlv_dump -d -o OUTPUT.MLV M01-2316.MLV

[squig]

Try this:

Code: [Select]

mlv_dump -d -o OUTPUT.MLV M01-2316.MLV

Same stuttering playback in Resolve.

[bouncyball]

@Danne: Thanks for correcting me.

@squig:

Same stuttering playback in Resolve.

Hmm... that means decompression is not a bottleneck. At least on your system. When I have time will check this under win.
Thanks for the answer and sorry about forgetting "-d" .

[squig]

Thanks for the answer and sorry about forgetting "-d" .

No problem. It seems more like a disk access issue, but my SSD and raid have 400MB/s+ read speeds. The 3K file is only 185MB/s, and that's not an issue playing DNGs in Resolve at that data-rate.

[Danne]

Squig. Could you post a short sample file? I´d like to test it over here on my computer and check how slow/fast it is.

[squig]

Squig. Could you post a short sample file? I´d like to test it over here on my computer and check how slow/fast it is.

https://drive.google.com/open?id=0BzJ3L6nv6Fn0RG9tUzBmY3phb1k

[bouncyball]

@squig:

Tested your file at work:
CPU Xeon E3-1245 3.4Ghz
GPU (very slow) Nvidia GTX 650
SSD
~18fps (both mlvfs/no mlvfs)  CPU load ~64%
Looks like DR GPU bottleneck here.

Edit:
Tested at home:
CPU - 4790K 4.2Ghz
GPU - Nvidia GTX 1080
SSD
23.976 fps (realtime both mlvfs/no mlvfs) CPU Load ~43%

bb

[squig]

Tested at home:
CPU - 4790K 4.2Ghz
GPU - Nvidia GTX 1080
SSD
23.976 fps (realtime both mlvfs/no mlvfs) CPU Load ~43%

bb

Hmm. Mac OS or Windows?

[bouncyball]

Hmm. Mac OS or Windows?

Win10 x64, DR 14b1

[erikbaldwinson]

I'm having the same problem stuttering problem with 12bitlossless mlvfs in both Windows 7 64 and linux centos7 in Resolve 14 beta2. In windows its awful, in linux it usually sorts itself out in a couple seconds but it makes editing tiring.

I've been running trial and error with nvidia gpus.

GTX 970 = bad
GTX 1060 = not good
GTX 1070 = not bad still not good

I've overclocked my 3770k to 4.4Ghz, turned off intel speedstep and other power saving stuff in bios and it's working at around 40%-60% capacity in Linux.

The stuttering seems like a hard drive issue but I have a samsung m2 drive in the pci port that does well on the crystal mark test.

The GTX 970 was not doing any work at all according to the gpu monitor at around 12-20% and 1GB of memory max used. This is basically the same case with the 1060 and 1070 yet playback performance is quite obviously a noticeable improvement from the 970 as the stuttering is less.

Why are the GPU monitors stating only 20% load and why is Resolve stating 23.98fps playback when it is clearly stuttering and playing back maybe 12fps?

I also turned off v-sync.

I have a feeling there is something else going on and upgrading the gpu is just easing off the tension indirectly helping the problem.

Any ideas? Trying to finish a project.

[voblaunsane]

Hi all,

It's been a while since I used mlvfs and now that I have fresh windows and reinstalled everything, took me a while to get this beast running..

Anyway, I am experiencing a massive stutter and general performance issue with 23.9 footage to the point where it is hardly possible to edit. I reported this long ago and someone found a timecode problem. I was under impression this was fixed and I think I was using it with no problems at some point..err..or maybe I'm dreaming? hahaha..

Or maybe using a wrong build or something..here how I got it working:

1. Installed Dokan 1.0.3.1000 (x64) from https://github.com/dokan-dev/dokany/releases/tag/v1.0.3

2. Unzipped MLVFS_x64.zip from http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/

3. Unzipped optimized.zip to MLVFS_x64 folder and renamed files adding "_opt" from http://www.magiclantern.fm/modules/modules/mlvfs_optimized.zip/

4. Renamed dokanfuse1.dll (in MLVFS_x64 folder) to dokanfuse.dll and dokan1.dll to dokan.dll

I tried all (mlvfs_x64.exe, mlvfs_x64_avx.exe, mlvfs_x64_opt.exe and mlvfs_x64_avx_opt.exe) and finaly capitalised on using mlvfs_x64_avx_opt.exe which seemed to be the fastest (super subjective/based on feeling)


all the settings on localhost panel are set to off, disabled, default or none (default values)

Same issues occur with footage copied from mlvfs drive onto HDD;
Yet unpacked using raw2cdng 1.7.5 works really well, not using mlvfs that is;

Also 25fps with mlvfs works like a charm too..


Thanks a ton! Good night..

[erikbaldwinson]

Hi all,

It's been a while since I used mlvfs and now that I have fresh windows and reinstalled everything, took me a while to get this beast running..

Anyway, I am experiencing a massive stutter and general performance issue with 23.9 footage to the point where it is hardly possible to edit. I reported this long ago and someone found a timecode problem. I was under impression this was fixed and I think I was using it with no problems at some point..err..or maybe I'm dreaming? hahaha..

Or maybe using a wrong build or something..here how I got it working:

1. Installed Dokan 1.0.3.1000 (x64) from https://github.com/dokan-dev/dokany/releases/tag/v1.0.3

2. Unzipped MLVFS_x64.zip from http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/

3. Unzipped optimized.zip to MLVFS_x64 folder and renamed files adding "_opt" from http://www.magiclantern.fm/modules/modules/mlvfs_optimized.zip/

4. Renamed dokanfuse1.dll (in MLVFS_x64 folder) to dokanfuse.dll and dokan1.dll to dokan.dll

I tried all (mlvfs_x64.exe, mlvfs_x64_avx.exe, mlvfs_x64_opt.exe and mlvfs_x64_avx_opt.exe) and finaly capitalised on using mlvfs_x64_avx_opt.exe which seemed to be the fastest (super subjective/based on feeling)


all the settings on localhost panel are set to off, disabled, default or none (default values)

Same issues occur with footage copied from mlvfs drive onto HDD;
Yet unpacked using raw2cdng 1.7.5 works really well, not using mlvfs that is;

Also 25fps with mlvfs works like a charm too..


Thanks a ton! Good night..

Hey man, thank you for sharing your installation workflow in great detail but unfortunately the optimized versions are outdated and do not support the Canon HW compressed MLVs.

The problem I am experiencing is playback with the 12bit lossless files. Some files playback perfectly and some files, for whatever reason, do not. Large files (ex.40gb+) both lossless and raw do not playback well - maybe your installation suggestions will solve that, I'll check next project.
Also, perhaps some of these optimized mlvfs.exe files could include the lossless playback support to resolve this playback problem.

I think having a latest greatest CPU would just strengthen through this lossless playback problem possibly. Although that being said in Linux my 3770K was not working hard at all (40-60%) and the playback was still stuttering as if the drive was failing or the CPU was maxed. Then suddenly playback was perfect with all files all sizes.. until I hit stop and then play.. and stuttering ensues..

 Sometimes with big files on Windows the CPU is maxed.

[loknar]

Hi,
i tried to use mlvfs on 10-bit mlv dual iso video from EOS M and i thought this is working combination; however, when i try to use dng file from inside mlvfs it opens interlaced and in console is:

Code: [Select]

ISO pattern     : BBdd RGGB
White levels    : 10000 5000
Noise levels    : 8.00 8.00 8.00 8.00 (14-bit)
Doesn't look like interlaced ISOAm I doing something wrong?

cr2hdr does the same:

Code: [Select]

Input file      : /data/mlvfs/M30-1955.MLV/M30-1955_000100.dng
Camera          : Canon EOS M
Full size       : 2512 x 1068
Active area     : 2512 x 1068
Doesn't look like interlaced ISO
EDIT: It does, however, works in preview mode 

[reddeercity]

There seems to be reports that some people are having playback problems with MLVFS (PC) and Blackmagic Resolve
on large files . I just ran a few tests to fine any bottle necks or issues in general , here are the details :
Test Machine :
CPU : AMD 8 core 8350 (Overcocked to 4.8 GHz on water cooler)
MB  : Asus Sabertooth 990FX (ver.1.0)
Ram: DDR3 2x4GB kit(8GB total) Crucial Ballistix 1600 MHz Dual Channel
GPU: (2)x NVidia GTX 580 1.5GB Vram ea. (resolve only used 1 GPU for cuda processing  as per the free licence  and 2nd for GUI)
SSD: OS Windows 7 Pro
Media File Storage : On board 8TB Raid0 (Striped)(4x2GB 7200 rpm Seagate Barracuda SATA 6 Gb/s NCQ 64MB Cache drives R/W 500+MB/s @ 50% free)
Test File: Green Screen Talking Head 10bit Full MLV 2144x1076 23.976fps +audio 12,000 frames = about 7:30 (from 5d2 so 8 spanning files = 30GB)
Blackmagic Resolve ver. 12.5.2 (free ver.)
Resolve setting:
2144x1076 23.976 project
32bit floating point
DaVinci YRGB Color science
Cache frames in Uncompressed 10bit
Enable background caching after 5 seconds
------------------------------------------------
Optimized Media:
Resolution (choose automatically)
format-Umcompressed 10-bit
---------------------------------------
Working Folders:
Cache files location (cache clips on D drive 8tb raid0
------------------------------------------------------------
Basic raw adjustment for log output

MLVFS with Dokan + Web GUI 10bit input file  >16bit output Cdng's+Audio (no options enabled) (location on 8TB raid0)
Play file in full res @ 23.976 with 3 frame drops at 2:00 , 5:00 & 7:00+ 1-2 frames audio out of sync. (but a quick pause & start sync is restored)
All 8 CPU working @ about 60% with 6.3GB being used with resolve .

MLVFS with Pismo (File Mount Audit Package build 171) (right click quick mount) No web GUI or options for processing just extracts 16bit Cdng+Audio to virtual folder in the same folder
no CMD prompt to deal with , mlv's can be mounted anywhere on the pc system . Mounted the 30GB file on 8TB raid0.
Only 5 CPU's working @ 30% with the same 6.3GB of ram being used with resolve running.
Play file on timeline @ 23.976 and no frame drops of sync loss .

So with dokan there twice the load with 8 core working @60%
and with Pismo only 5 core working @ 30% load .

So if performance is the issue where you need real-time playback (for editing) and not just transcoding
then used Pismo (look at the bottom of page) I use both systems on my PC.

[goose]

Hi everyone, i followed the installation instructions for MLVFS and it appears as a service to run when i right click the folder. Once i select it though nothing all happens.

Any ideas?

Thanks in advance.
Modify message

[dmilligan]

There's probably a file selection dialog that came up behind other windows (hide other open windows and you should see it). It's really annoying that Automator does that, but not anything I can do about it.

[Danne]

There´s actually an easy way to make the dialog box open as the front window. Add an applescript and print tell me to activate before the upcoming "Ask for finders items" and the promt window will always be placed on top of everything else.

[dmilligan]

Cool!

[goose]

that was the answer thank you, cant believe that was the answer, feel so thick lol

[voblaunsane]

so I've tested bouncyball's version of mlvfs (with dokan 1.0.3) as it seems to be the latest mlvfs compile out there and still have 23.9 fps stuttering issue..

to demonstrate I would like to share Performance Monitor graphs produced while scrubbing ~3 minute long footate in Premiere

25 fps graph looks nice: constant load on hdd, processor cruncing numbers, works lika a charm and the whole take can be previewed in a blink


while 23.9 fps is a real mess: hdd constantly choking and processor is having a nap, visual performance - stuck on one frame mostly...



anyone else experienceing this?.. I find this is the case with all versions of mlvfs I found out there...

[reddeercity]

so I've tested bouncyball's version of mlvfs

What is this ? Can someone explain bouncyball's  MLVFS !
That's dmilligan's project from my understanding on PC & MAC.
I can't even find his source code , why would bouncyball be modifying mlvfs as it work flawlessly as is .
Can't even find any threads about this to follow .

[DeafEyeJedi]

Can't even find any threads about this to follow .

mlvfs on steroids

[reddeercity]

Thanks @ DeafEyeJedi , OK it's just mlv_dump not mlvfs , I thought I missed something .
really nothing new I can see specially for any digic4 cams looks like digic5 & up thou.

[bouncyball]

@reddeercity

I can't even find his source code , why would bouncyball be modifying mlvfs as it work flawlessly as is .
Can't even find any threads about this to follow .

The correct thread link is here

Anyway, after some testing I did pull request and David merged it to main MLVFS repo. It's all there now

regards,
bb

[voblaunsane]

What is this ? Can someone explain bouncyball's  MLVFS !

sorry for not being clear enough.. mlvfs.exe (2017-05-06) https://bitbucket.org/bouncyball/mlv-tools/downloads/

<...> as it work flawlessly as is .

do you not experience problems with 23.9fps on PC?



It appears I might be but a singleton...