Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 959300 times)

dmilligan · « **Reply #500 on:** March 01, 2015, 08:39:52 PM »

Delete your IDX files and let MLVFS re-create them and it works just fine (same with mlv_dump).

My guess what happened is that you tried to playback in camera or you tried to process the files without them named correctly first, which triggered the IDX file to be created, but this IDX files does not contain info about the missing frames in the spanned file. Then when you copied the spanned file and gave it the correct name, the index did not get rebuilt with the frames from the spanned file (because the IDX was already created).

Also, for the situation where you had a whole bunch of m0X files because recording stopped, you can just delete all of those extra files (as long as the file is not larger than a couple MB, there's nothing it). Those are just created by mlv_rec when recording starts and the unused ones are normally cleaned up, but they don't get cleaned up if recording stops prematurely.

swinxx · « **Reply #501 on:** March 01, 2015, 08:44:18 PM »

oh. thank you a lot.
i will try that and will get back as soon as possilbe.

best wishes. swinxx

swinxx · « **Reply #502 on:** March 02, 2015, 10:34:23 PM »

@dmilligan:

can you please give me an example of an mlv_dump compression terminal command which is working with mlvfs, cause i tried to compress a mlv but with no success.. (the file got smaller, nearly 45% but mlvfs can not convert it anymore.)
my command was:

mlv_dump -o Compressed.mlv -v -e -c -b 16 -l 9 -x <filenamexyz.mlv>

thank you very much.

dmilligan · « **Reply #503 on:** March 03, 2015, 12:20:18 AM »

delta encode mode (-e switch) is not supported by MLVFS, also mlv_dump doesn't handle white and black level properly when changing bit depth (IMO), so if you use -b 16, the output black (and white) level from MLVFS will be wrong. mlv_dump will actually output the correct black level if you change bit depth, but that is because it assumes the black level is 14 bit (IOW, it assumes the original data is 14 bit). I don't like to make that assumption.

Here's a fix: https://bitbucket.org/dmilligan/magic-lantern/commits/a22302855d05c6cb587336943c8b5fb623a96bb4

The other thing is that LZMA is quite slow, and you'll never get realtime playback with it. That's why I also implemented the new compression scheme from here: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92

swinxx · « **Reply #504 on:** March 03, 2015, 10:00:34 AM »

hello dmilligan

thank you for the explanation.
so how can i compress them with the new option?
is there a simple batch possibility to compress - lets say a complete directory..

thank you very much.
sw

dmilligan · « **Reply #505 on:** March 03, 2015, 11:14:14 PM »

Quote from: swinxx on March 03, 2015, 10:00:34 AM

so how can i compress them with the new option?

use this mlv_dump
like this

Code: [Select]

./mlv_dump -o <output.mlv> -c <input.mlv>

a1ex · « **Reply #506 on:** March 04, 2015, 10:02:34 AM »

Finally managed to record a timelapse with fullres silent pics (nothing spectacular, just something to try if it works), and I installed MLVFS to postprocess it. Wanted to try it for a long time, and this was the perfect occasion (since I don't really record raw videos anyway).

Already did some minor tweaks, to address the small installation difficulties I had:
https://bitbucket.org/dmilligan/mlvfs/pull-request/5/minor-tweaks-for-command-line-interface/diff

Problems found:
- ~~an invalid MLV crashes MLVFS right away~~ SOLVED
- ~~the web interface doesn't work, browser gets stuck at "Request sent"~~ SOLVED
- running cr2hdr in the virtual FS crashes after processing a few files (will try to fix, but I think I need more time)
- copying DNGs from the virtual FS to the real FS (same HDD drive, not SSD) is about twice as slow as a straight file copy (~20MB/s vs ~40MB/s, will look into it)
- exiftool can't change black level of the DNGs created by MLVFS (minor, probably some bad header)
- with spanned files, there is a black frame for each chunk (!), most likely the last frame, need to double-check (mlv_dump handles these frames properly)

Pyriphlegethon · « **Reply #507 on:** March 06, 2015, 12:01:02 AM »

dmilligan · « **Reply #508 on:** March 08, 2015, 08:59:34 PM »

Quote from: a1ex on March 04, 2015, 10:02:34 AM

- the web interface doesn't work, browser gets stuck at "Request sent", not sure why (tried to change port to 80000, no luck)

I think this might have to do with background mode and the threading model. Try running with -f. (I always use -f so I can debug and watch console output, so I've never noticed this issue). When I tried without -f on OSX, the webgui didn't work.

a1ex · « **Reply #509 on:** March 08, 2015, 11:10:59 PM »

Correct. When I run it with -f, I get "Segmentation fault" right when I connect to a web browser on 127.0.0.1:8000.

That's the only message I get printed on the console.

Valgrind log:

Code: [Select]

==26190== Memcheck, a memory error detector
==26190== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==26190== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==26190== Command: ./mlvfs MLVFS -f --mlv-dir=/path/to/mlvs
==26190== 
webgui start
webgui run
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
==26190== Thread 2:
==26190== Invalid read of size 8
==26190==    at 0x409059: xref_sort (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4098A7: make_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409B0C: build_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E23: force_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E79: get_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409F15: mlv_get_frame_count (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4375D5: webgui_generate_mlv_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x437F1E: webgui_generate_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4385A0: webgui_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42AE5A: call_user (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42FD78: call_request_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4309DF: call_request_handler_if_data_is_buffered (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==26190== 
==26190== 
==26190== Process terminating with default action of signal 11 (SIGSEGV)
==26190==  Access not within mapped region at address 0x0
==26190==    at 0x409059: xref_sort (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4098A7: make_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409B0C: build_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E23: force_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E79: get_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409F15: mlv_get_frame_count (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4375D5: webgui_generate_mlv_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x437F1E: webgui_generate_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4385A0: webgui_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42AE5A: call_user (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42FD78: call_request_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4309DF: call_request_handler_if_data_is_buffered (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==  If you believe this happened as a result of a stack
==26190==  overflow in your program's main thread (unlikely but
==26190==  possible), you can try to increase the size of the
==26190==  main thread stack using the --main-stacksize= flag.
==26190==  The main thread stack size used in this run was 8388608.
==26190== 
==26190== HEAP SUMMARY:
==26190==     in use at exit: 868,724 bytes in 105 blocks
==26190==   total heap usage: 1,096 allocs, 991 frees, 1,604,352 bytes allocated
==26190== 
==26190== LEAK SUMMARY:
==26190==    definitely lost: 917 bytes in 34 blocks
==26190==    indirectly lost: 0 bytes in 0 blocks
==26190==      possibly lost: 1,088 bytes in 4 blocks
==26190==    still reachable: 866,719 bytes in 67 blocks
==26190==         suppressed: 0 bytes in 0 blocks
==26190== Rerun with --leak-check=full to see details of leaked memory
==26190== 
==26190== For counts of detected and suppressed errors, rerun with: -v
==26190== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

edit: it was caused by an invalid MLV (created by latest fullres silent code). Deleting that MLV solved the crash.

edit2: fix commited, thanks dmilligan for write access.

swinxx · « **Reply #510 on:** March 09, 2015, 08:46:06 PM »

@dmilligan

would it be possible to compile a new mac version with the latest code changes at the bitbucket download section

thank you very much swinxx

dmilligan · « **Reply #511 on:** March 10, 2015, 01:42:33 AM »

DeafEyeJedi · « **Reply #512 on:** March 10, 2015, 08:56:21 AM »

Thanks for the update regarding the latest cr2hdr_20-bit from @a1ex into your app @dmilligan!

dmilligan · « **Reply #513 on:** March 10, 2015, 11:54:09 AM »

huh? I have not updated the cr2hdr code. It is based on, but slightly different from the main cr2hdr code. The main differences are:
it doesn't use dcraw, the data comes internally straight from the MLV file
it doesn't use exiftool, dng header data is written by mlvfs itself
it's thread safe
some gcc specific language features were removed so it would compile with llvm

DeafEyeJedi · « **Reply #514 on:** March 10, 2015, 06:33:16 PM »

Okay gotcha -- my fault for assuming that you got the updated cr2hdr compiled for Mac by N/A that he released just few few days ago...

Will give it another test run w MLVFS with a quick Dual-ISO MLV footage (hopefully it won't come out pinkish again) although it still comes out in different kind of pink cast for non Dual-ISO MLV's as well.

I also updated the Adobe DNG to the latest firmware in case you were wondering.

Thanks again!

N/A · « **Reply #515 on:** March 10, 2015, 08:56:41 PM »

I think dmilligan knows how to compile cr2hdr as well

DeafEyeJedi · « **Reply #516 on:** March 10, 2015, 10:34:06 PM »

Ofc, I would assume so.

Doesn't hurt to ask.

Thx.

N/A · « **Reply #517 on:** March 13, 2015, 04:37:30 AM »

Getting crazy vertical stripes and green tint occasionally with the latest mac mlvfs. Doesn't happen all the time and not every mlv either, seems to happen when I first log on and mount my folder. Ejecting the drive, closing my browser and re-mounting it seems to fix it. Toggling vertical stripe correction and bad pixel fix doesn't fix it. Going to try to replicate the issue and see if its random or if certain mlv's and settings cause it.

vertigopix · « **Reply #518 on:** March 13, 2015, 02:21:10 PM »

Just want to share my "MLVFS server based converter"

Server is clearos-community-6.6-x86_64
Installed FUSE (from clearos-core repo)
Compiled MLVFS from BitBucket
Created 2 shares (Flexshare) mlv-in and mlv-out
Created the file /etc/fuse.conf with user_allow_other argument
Finely lauched mlvfs -f /var/flexshare/shares/magiclantern/mlv-out --mlv_dir=/var/flexshare/shares/magiclantern/mlv-in -o allow_other &

So now, from my W7 PC i can put any MLV file in the mlv-in share and get the DNGs in the mlv-out folder share

Perfect !

mario1000 · « **Reply #519 on:** March 13, 2015, 04:34:43 PM »

Quote from: vertigopix on March 13, 2015, 02:21:10 PM

Just want to share my "MLVFS server based converter"

Server is clearos-community-6.6-x86_64
Installed FUSE (from clearos-core repo)
Compiled MLVFS from BitBucket
Created 2 shares (Flexshare) mlv-in and mlv-out
Created the file /etc/fuse.conf with user_allow_other argument
Finely lauched mlvfs -f /var/flexshare/shares/magiclantern/mlv-out --mlv_dir=/var/flexshare/shares/magiclantern/mlv-in -o allow_other &

So now, from my W7 PC i can put any MLV file in the mlv-in share and get the DNGs in the mlv-out folder share

Perfect !

Do I understand it right that this is a solution for windows users to use the latest update of dmilligan? If this is the case could you please provide a more detailed step by step guidance for users. What I mean is: does your solution needs linux or just windows... How do I setup Flexshare?

Sorry if this question seems to be a little stupid but I am a user and not a very skilled programmer.

Thanks in advance

dmilligan · « **Reply #520 on:** March 13, 2015, 05:02:58 PM »

You need a Linux box to do this (or a VM running Linux).

arrinkiiii · « **Reply #521 on:** March 13, 2015, 09:26:04 PM »

....almost !!! grrrrrrrr ...for windows users!!!

Walter Schulz · « **Reply #522 on:** March 13, 2015, 09:35:50 PM »

Arrinkiii: Running a VM is not that complicated. Try VMware Player.

DeafEyeJedi · « **Reply #523 on:** March 13, 2015, 09:46:17 PM »

I've ran VMware on PC @ work since they don't have Mac's and MLVFS works quite well as per @dmilligan's suggestion.

However, I prefer to do it solely Mac's system alone.

*feedbacks*:

@dmilligan -- Based on my recent quick tests from earlier it seems I'm still getting strange pink frames from dual-ISO MLV files. Looks like it doesn't complete process during render before exporting from PP) Am I to export from the mount itself (with dual ISO and cs 2x2 enabled ON) or do I need to drag files onto system before I export it from PP?

Thanks!

dmilligan · « **Reply #524 on:** March 14, 2015, 02:22:35 PM »

I'm not completely sure what you are asking. When I have time, I'll look into the dual ISO issue and see if I can duplicate it (I don't use dual ISO very much)

News:

Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 959300 times)