MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[spnsir]

So far so good, working here! Thank you for the quick fix David!

[swinxx]

YES!! Great! It WORKS!!! BBIIGG TTHHHUUMMBBSS UP!! This is really great and a big step forward.

btw. i have found out that chroma smoothing (especially 2x2) causes those jumping hot? pixels in the dark areas..
can there anything be improved? perhaps from a1ex code?
thx swinxx

[Danne]

Nice work

[chmee]

could someone show me a screenshot with the file and path? am i right, the filenaming-convention for

path is [name]_1_yyyy-mm-dd_0050_C0000
dng is [name]_1_yyyy-mm-dd_0050_C0000_[sequencenumber].dng
wav is [name]_1_yyyy-mm-dd_0050_C0000.wav

@swinxx
i assume chroma smoothing and banding are time consuming, so it wont be realtime? prove me wrong. if it is rt (and the dng's look right in premiere as well) its time to stop developing my tool. because the filesystem-idea is much better than the converting-thing.

regards chmee

[dmilligan]

Actually the filename convention seems to have nothing to do with the audio showing up correctly, even if I don't use the filename convention at all, the audio and dng sequence still show up as one media file. The only thing the filename convention really does is make the folder itself show up as a media file. I think it has to do with what the extra metadata in the wav file is. See my code (wav.c) for what I'm doing. I actually got most of that information from you (the bext and iXML stuff), so I'm surprised to hear it doesn't work for you.

chroma smoothing is quite slow, but I think banding is realtime or almost realtime if your computer is fast enough (it only calculates the coefficients once for one frame and then reuses them for the rest of the frames, it's the calculating that's slow, applying the correction is quite fast, all we have to do is subtract-multiply-add integers).

One thing you can do though, is to simply not use any of slow options until final render. So you can edit, CC, etc. until you are completely finished, then turn on the options, start the render and let it chug away (just make sure to clear any render caches when doing this).

[swinxx]

@chmee
I have a fast computer;) so it runs nice also with chroma smoothing options on, the only thing is that thise pixels are spinning around, so it is not possible to really use it. I wonder that no one else had seen them before..?

@dmilligan:
So if i have a directory, with mixed mlv files (normal and dual iso ones) how would i mount them? All together or dual isos seperately?  Would it be a smart addition to add the possibility for making a file dependent dual iso on off option in the webserver??
Thx

[dmilligan]

It should be able to detect pretty quickly that it's not dual ISO. It may hurt performance slightly to have dual ISO turned on for non-dual ISO files, but I think it will probably still be realtime. Just try it and let us know.

I don't really understand how the chroma smooth algorithm actually works because I've never dug into the code. I just implemented the code straight from cr2hdr/mlv_dump. So I have no idea what could be causing the artifacts you're seeing or how to fix it.

[chmee]

@dmilligan
thanks. by now the main difference - it seems - is, you are starting on 00:00:00 with timecode, will do some tests.

@swinnx
[fast pc] thats good for you, but it limits the usability for the slower pc owners. indeed, it should be an option, enabling cs and vb.

[swinxx]

@chmee.
yes fast read speeds (i have a raid system) are very useful for working with raw video. the write speed doesn´t matter, cause when delivering a video or playback a sequence, the limiting factors are gpu and cpu. and i have trained myself to cut before denoising and cc.

what i do not understand is, why do so less people download that tool... i think the problem is that they simply are afraid of a complicated workflow. all i can say to all the magic lantern users out there.. just try that program, the workflow is sooo simple and soo fast! the best program out there..

what i do not understand when working with resolve is that how do you handle the mounting points? do you apply the service at the main video folder where all the different directories with the mlv video files are placed into or do you apply more than one mlvfs service (one for every project, or one for every scene?) i think when this mystey is solved mlvfs will probably my solution with no proir conversion. so far i always copy the files out of the mounting point to a dedicated directory. - i like to be on the safe side..

@dmilligan
why are there 2 different naming schemes? isn´t it possible to work with resolve naming scheme in ae or premiere?
I have a scene of the moon (will update this post with the link to dngs as soon as possilbe).. there are 3 white pixels which are bad or hot or whatever pixels.. with option 1 (Bad Pixel Handling) they simply do not go away.. but it is a black scene and the pixel is white..
when i use agressive Bad pixel handling the pixels go away but the moon surface gets really strange.. the algo destroys the moon surface.. so is it possible to adjust the threshold a bit? perhaps with my delivere dng..?

All in All Thank You for you HARD WORK!!
I LOVE THIS TOOL SO MUCH! Everything is working so fine now (except ChromaSmoothing and Bad Pixel handling)

Now MLV is a breeze to work with.

@TO ALL PEOPLE -> https://bitbucket.org/dmilligan/mlvfs/downloads AND TRY THAT TOOL.. ITS AWESOME!

[vertigopix]

Can somebody help me ?

I can't compile MLVFS under Mint 17 x64...
When i try, i get a lot of errors.

Code: [Select]

gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 stripes.c -o stripes.o
In file included from mlvfs.h:25:0,
                 from stripes.c:26:
raw.h:167:5: error: unknown type name ‘uint32_t’
     uint32_t api_version;           // increase this when changing the structure
     ^
In file included from mlvfs.h:26:0,
                 from stripes.c:26:
mlv.h:42:5: error: unknown type name ‘uint8_t’
     uint8_t     blockType[4];
     ^
mlv.h:43:5: error: unknown type name ‘uint32_t’
     uint32_t    blockSize;
     ^
mlv.h:44:5: error: unknown type name ‘uint64_t’
     uint64_t    timestamp;
     ^
mlv.h:48:5: error: unknown type name ‘uint8_t’
     uint8_t     fileMagic[4];    /* Magic Lantern Video file header */
     ^


[ayshih]

Oops.  For Linux, in "mlvfs.h", you need to add an include prior to the include for "raw.h":

Code: [Select]

#include <stdint.h>

[vertigopix]

Better ! 
But...   

Code: [Select]

gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 dng.c -o dng.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 index.c -o index.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 wav.c -o wav.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 kelvin.c -o kelvin.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 stripes.c -o stripes.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 cs.c -o cs.o
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64   -c -o amaze_demosaic_RT.o amaze_demosaic_RT.c
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 hdr.c -o hdr.o
hdr.c: In function ‘identify_bright_and_dark_fields’:
hdr.c:553:13: warning: variable ‘changed’ set but not used [-Wunused-but-set-variable]
         int changed = 0;
             ^
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 histogram.c -o histogram.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 mongoose.c -o mongoose.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 webgui.c -o webgui.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 resource_manager.c -o resource_manager.o
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 main.c dng.o index.o wav.o kelvin.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose.o webgui.o resource_manager.o -pthread -lfuse -lm -o mlvfs
main.c:33:18: fatal error: fuse.h: No such file or directory
 #include <fuse.h>
                  ^
compilation terminated.
make: *** [mlvfs] Error 1


[vertigopix]

Found !
Need to add

Code: [Select]

libfuse-dev
Now it compile.

[vertigopix]

But webserver doesn't work under Linux.  (http://localhost:8000)
nmap report the port open

Code: [Select]

8000/tcp open  http-altbut nothing...

[dmilligan]

@swnixx,

There are two different naming schemes in case anyone was using the old naming scheme and had imported stuff into whatever projects => with a different naming scheme none of their paths would be correct anymore and they would have to re-link everything. It's also much more compact, those huge long filenames are kind of annoying if you don't need them.

what i do not understand when working with resolve is that how do you handle the mounting points?

Handle it however you like, you can run multiple MLVFS instances if you wish. If you like a project to be contained entirely within a directory you could do something like this:

~/projects/project1/                                         <-root dir of project
~/projects/project1/project1.prprj                     <-project file for whatever NLE
~/projects/project1/other_media/
~/projects/project1/mlv/                                  <-original mlvs here
~/projects/project1/mlv/scene1/M00-0001.MLV
~/projects/project1/mlv/scene1/M00-0002.MLV
~/projects/project1/mlv/scene2/M00-0003.MLV
~/projects/project1/mlv/scene3/M00-0004.MLV
~/projects/project1/mlvfs/                               <- mount point

Whenever you're working on this project, mount "~/projects/project1/mlv/" to "~/projects/project1/mlvfs/" and import from "~/projects/project1/mlvfs/..." directly inside your NLE.

[swinxx]

@dmilligan
thank you for your detailed explanation. that makes sense.
will try it.
greets. swinxx

[ayshih]

But webserver doesn't work under Linux.  (http://localhost:8000)

Sorry, I'm still out of commission with respect to working on ML or MLVFS for at least another week, so I haven't been able to maintain the Linux version.  I encourage you (or others) to try to figure out why it's not working!

[dmilligan]

updates: hopefully improved hot pixel detection, try it out and let me know, also if you have some MLV samples with hot pixels that would be helpful (post them here: http://www.magiclantern.fm/forum/index.php?topic=11899.0)

[DeafEyeJedi]

Thanks @dmilligan for the update -- will definitely try them out and let you know!

[swinxx]

@dmilligan:
I had a problem where only the bad pix slgo worked, so i will try it out again.
Thank you!! That is awesome.

swinxx

[swinxx]

hello dmilligan!

i have found a bunch of old raw files and wanted to throw them into a mlvfs service.. but they are not displayed. now i realized that the "mlvfs" only supports mlv files..
will your service ever support raw files (the old v1 mlv) or does anyone know a possibility to convert some raw files to mlvs.

thx in advance, swinxx

[swinxx]

hello dmilligan again.

i would have another question regarding your newest update with the new bad pixel adaptation:
when i deselect the bad pixel option in the webserver, will the service try to remove some pixels anyway?
i ask because my video seems clean now, without anything selected :=)

thank you very much

[dmilligan]

will the service try to remove some pixels anyway?

No

will your service ever support raw files (the old v1 mlv)

IDK. Probably not. I want to encourage people to use MLV and not RAW. It would also be a lot of work. There's lots of important metadata missing too.

[bennyray1]

Not sure exactly what happened but only up until this latest update MLVS is not showing up in my services after right clicking on my .MLV files...is anyone else experiencing this? @dmilligan is there away I can start over with a fresh install, instead of replacing MLVS? Running on OS X 10.9.3,I even updated OSX Fuse.

[dmilligan]

In the Finder menu click on Go > Go to Folder... and type ~/Library/Services/
You should see the MLVFS.workflow there. You can delete it and try installing it again. I sort of doubt it will help.

after right clicking on my .MLV files...

Just to be sure, you are right-clicking a folder right? and not the actual files? You mount a folder containing files, not the files themselves.

(I'm not seeing any issues with the automator service on my end)