MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[Licaon_Kter]

And you did restart MLVFS right?

[Simonwb]

It's a hidden file in your home folder.

Open the terminal and type

Code: [Select]

cat ~/.mlvfs.log

Try a different folder name. The name "dng" could potentially confuse MLFVS (b/c DNG files end in "dng".)

Hi @dmilligan Thanks for the tip on where to find the log.  Here's the contents:

Last login: Fri Apr 29 21:01:04 on console
Simons-MBP-2:~ simonwilkinson-blake$ cat ~/.mlvfs.log
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
nohup: ./mlvfs: No such file or directory
Simons-MBP-2:~ simonwilkinson-blake$

I did try a different folder name, still the same error.

Thanks for your help with this!

Simon

[dmilligan]

download and install again, it looks like the actual mlvfs executable is missing (it should be inside the automator workflow package)

[Simonwb]

download and install again, it looks like the actual mlvfs executable is missing (it should be inside the automator workflow package)

@dmilligan thanks.  Reinstalled OSXFUSE and MLVFS - same error as before, unfortunately

However, looking in my Library>Automator folder, I can't see anything that looks obviously MLVFS.  Doing a search on my Mac for 'MLVFS' doesn't bring up anything that looks like an executable file.  But then, neither can I find a MLVFS file on my other Mac, on which MLVFS works fine

[dmilligan]

the executable should be located at: ~/Library/Services/MLVFS.workflow/Contents/mlvfs

in the terminal type

Code: [Select]

ls ~/Library/Services/MLVFS.workflow/Contents/

You can also run mlvfs from the terminal, if you do without arguments you should see the command line help:

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs


[Simonwb]

the executable should be located at: ~/Library/Services/MLVFS.workflow/Contents/mlvfs

in the terminal type

Code: [Select]

ls ~/Library/Services/MLVFS.workflow/Contents/

You can also run mlvfs from the terminal, if you do without arguments you should see the command line help:

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs


@dmilligan thanks

On your first code, I get:

Last login: Tue May  3 15:34:24 on ttys000
Simons-MBP-2:~ simonwilkinson-blake$ /Library/Services/MLVFS.workflow/Contents/
-bash: /Library/Services/MLVFS.workflow/Contents/: is a directory
Simons-MBP-2:~ simonwilkinson-blake$

On your second, I get

Last login: Tue May  3 13:27:23 on ttys000
Simons-MBP-2:~ simonwilkinson-blake$ ~/Library/Services/mlvfs
-bash: /Users/simonwilkinson-blake/Library/Services/mlvfs: No such file or directory
Simons-MBP-2:~ simonwilkinson-blake$ ~/Library/Services/MLVFS.workflow/contents/mlvfs
-bash: /Users/simonwilkinson-blake/Library/Services/MLVFS.workflow/contents/mlvfs: No such file or directory
Simons-MBP-2:~ simonwilkinson-blake$ mlvfs
-bash: mlvfs: command not found
Simons-MBP-2:~ simonwilkinson-blake$ /library/services/mlvfs
-bash: /library/services/mlvfs: No such file or directory
Simons-MBP-2:~ simonwilkinson-blake$ /library/services/MLVFS.workflow/Contents/mlvfs
main.c:1995:main(): MLVFS: no mlv path specified

usage: mlvfs mountpoint [options]

general options:
    -o opt,[opt...]        mount options
    -h   --help            print help
    -V   --version         print version

FUSE options:
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation

no mount point

MLVFS options:

- File/folder options:
    --mlv-dir=%s           Directory containing MLV files
    --resolve-naming       DNG file names compatible with DaVinci Resolve

- Processing options:
    --cs2x2                2x2 chroma smoothing
    --cs3x3                3x3 chroma smoothing
    --cs5x5                5x5 chroma smoothing
    --bad-pix              Fix bad pixels (autodetected)
    --really-bad-pix       Aggressive bad pixel fix
    --fix-pattern-noise    Fix row/column noise in shadows (slow)
    --stripes              Vertical stripe correction in highlights (nonuniform column gains)
    --deflicker=%d         Per-frame exposure compensation for flicker-free video
                           (your raw processor must interpret the BaselineExposure DNG tag)

- Dual ISO options:
    --dual-iso-preview     Preview Dual ISO files (fast)
    --dual-iso             Render Dual ISO files (high quality)
    --amaze-edge           Dual ISO: interpolation method (high quality)
    --mean23               Dual ISO: interpolation method (fast)
    --no-alias-map         Dual ISO: disable alias map
    --alias-map            Dual ISO: enable alias map

- Web GUI options:
    --port=%s              Port used for web GUI (default: 8000)
    --fps=%f               FPS used for playback in web GUI

- Diagnostic options:
    --version              Display MLVFS version

Simons-MBP-2:~ simonwilkinson-blake$

[Simonwb]

in the terminal type

Code: [Select]

ls ~/Library/Services/MLVFS.workflow/Contents/

@dmilligan sorry, missed out the 'ls' in the code.  When I include it, I get:

Last login: Tue May  3 15:42:25 on ttys000
Simons-MBP-2:~ simonwilkinson-blake$ ls /Library/Services/MLVFS.workflow/Contents/
80000301_1808x1190.fpm   80000326_2592x1108.fpm   80000346_1872x1060.fpm
80000301_1808x727.fpm   80000331_1808x1190.fpm   80000346_2592x1108.fpm
80000301_1872x1060.fpm   80000331_1808x727.fpm   Info.plist
80000301_2592x1108.fpm   80000331_1872x1060.fpm   document.wflow
80000326_1808x1190.fpm   80000331_2592x1108.fpm   html_template.html
80000326_1808x727.fpm   80000346_1808x1190.fpm   jquery-1.12.0.min.js
80000326_1872x1060.fpm   80000346_1808x727.fpm   mlvfs
Simons-MBP-2:~ simonwilkinson-blake$

[dmilligan]

FYI, file paths are case sensitive.

There doesn't appear to be anything wrong. The error message: "nohup: ./mlvfs: No such file or directory" is a mystery. You can try actually launching MLVFS from the command line (like you did when you saw the help), but actually supply the required parameters:

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs <mount point> --mlv_dir=<directory with MLV files>

where <mount point> and <directory with MLV files> are actual file paths (hint: drag and drop folders onto the terminal window will insert the file path for that folder at the cursor position)

[Simonwb]

FYI, file paths are case sensitive.

There doesn't appear to be anything wrong. The error message: "nohup: ./mlvfs: No such file or directory" is a mystery. You can try actually launching MLVFS from the command line (like you did when you saw the help), but actually supply the required parameters:

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs <mount point> --mlv_dir=<directory with MLV files>

where <mount point> and <directory with MLV files> are actual file paths (hint: drag and drop folders onto the terminal window will insert the file path for that folder at the cursor position)

@dmilligan I tried and got this:

Last login: Tue May  3 19:29:11 on ttys000
Simons-MBP-2:~ simonwilkinson-blake$ /Library/Services/MLVFS.workflow/Contents/mlvfs </Users/simonwilkinson-blake/Desktop/test> --mlv_dir=</Users/simonwilkinson-blake/Desktop/mlv>
-bash: syntax error near unexpected token `newline'
Simons-MBP-2:~ simonwilkinson-blake$

[dmilligan]

the tilde ("~") at the beginning is very important, and you don't actually include the "<" and ">"

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs /Users/simonwilkinson-blake/Desktop/test --mlv_dir=/Users/simonwilkinson-blake/Desktop/mlv


[Simonwb]

the tilde ("~") at the beginning is very important, and you don't actually include the "<" and ">"

Code: [Select]

~/Library/Services/MLVFS.workflow/Contents/mlvfs /Users/simonwilkinson-blake/Desktop/test --mlv_dir=/Users/simonwilkinson-blake/Desktop/mlv


@dmilligan thanks for the corrections.

When I add that command as you've written it into Terminal, I get back:

Last login: Tue May  3 19:35:37 on ttys000
Simons-MBP-2:~ simonwilkinson-blake$ ~/Library/Services/MLVFS.workflow/Contents/mlvfs /Users/simonwilkinson-blake/Desktop/test --mlv_dir=/Users/simonwilkinson-blake/Desktop/mlv
-bash: /Users/simonwilkinson-blake/Library/Services/MLVFS.workflow/Contents/mlvfs: No such file or directory
Simons-MBP-2:~ simonwilkinson-blake$

[dmilligan]

this is getting weird, try:

Code: [Select]

otool -L ~/Library/Services/MLVFS.workflow/Contents/mlvfs


[Simonwb]

Code: [Select]

otool -L ~/Library/Services/MLVFS.workflow/Contents/mlvfs
[/quote]

@dmilligan I got:

Simons-MBP-2:~ simonwilkinson-blake$ otool -L ~/Library/Services/MLVFS.workflow/Contents/mlvfs
xcode-select: note: no developer tools were found at '/Applications/Xcode.app', requesting install. Choose an option in the dialog to download the command line developer tools.
Simons-MBP-2:~ simonwilkinson-blake$

I didn't install any tools. If you want me to, pls let me know which, I wouldn't know where to start!

[dmilligan]

Meh, IDK what is wrong, this is very strange. Maybe something wrong with OSXFuse installation. It's really strange that MLVFS will display the command line help, but you get the no such file error when you actually try to use it.

[Simonwb]

Meh, IDK what is wrong, this is very strange. Maybe something wrong with OSXFuse installation. It's really strange that MLVFS will display the command line help, but you get the no such file error when you actually try to use it.

Many thanks for taking a look, I really appreciate what you and the other developers here have done with ML.

[canneloni]

Isn't there anyone with an idea for what might be the problem in my case? I would really appreciate an answer.

Thank you very much for your work on Magic Lantern and for the tools to make the most out of it!

[Ottoga]

@cannelloni

Re: the m22-1744 in your --mlv-dir path. Is this the actual mlv file or the directory where that mlv resides. The path should be to the directory where the mlv files reside.

Also, have you actually checked that dokan is loaded? See below for how to check (in a cmd window):

C:\Program Files\Dokan\Dokan Library-1.0.0>cd /d C:\Program Files\Dokan\Dokan Library-1.0.0

C:\Program Files\Dokan\Dokan Library-1.0.0>dokanctl
Driver path: 'C:\WINDOWS\system32\drivers\dokan1.sys'
dokanctl /u MountPoint (/f)
dokanctl /m
dokanctl /i [d||n]
dokanctl /r [d||n]
dokanctl /v

Example:
  /u M                : Unmount M: drive
  /u C:\mount\dokan : Unmount mount point C:\mount\dokan
  /u 1                : Unmount mount point 1
  /u M /f             : Force unmount M: drive
  /i d                : Install driver
  /i n                : Install network provider
  /r d                : Remove driver
  /r n                : Remove network provider
  /d [0-9]            : Enable Kernel Debug output
  /v                  : Print Dokan version

C:\Program Files\Dokan\Dokan Library-1.0.0>dokanctl /v
Driver path: 'C:\WINDOWS\system32\drivers\dokan1.sys'
dokanctl : Mar  8 2016 13:07:15
Dokan version : 100
Dokan driver version : 0x190

C:\Program Files\Dokan\Dokan Library-1.0.0>


The above is for the 64bit version. If you don't get any or, the correct version displayed then you probably need to install the dokan driver and network provider.

Do this via the following commands:
dokanctl /i d
dokanctl /i n

You only need to do this once, it will automatically load at boot time from then on.

You might also like to create a batch file to launch MLVFS. below is an example of mine.

cd /d C:\OtherPrograms\MagicLantern\MLVFS_x64
mlvfs_x64.exe X: --mlv-dir=D:\MLVs


You won't see the drive mounted in the command window, check under "This PC" in file manager to see the dokan virtual drive.  Also, you need to keep the command window active the entire time that you are using MLVFS. Just minimise it. If you close the command window, the dokan virtual drive will be unmounted.

Assuming that you can now see your virtual dokan drive. Fire up you browser and enter http://localhost:8000/ You should now see all of the MLVs listed.

A little tip:  Choose your MLVFS configuration options in the browser before accessing the mlv file in the dokan virtual drive. It saves you having to force or wait for a refresh cycle in file manger.

Hope this helps....

[marekk]

Hi dmilligan,
Thanks for your hard work with mlvfs.

I'm currently converting mlv footage from my holidays trip to dng sequences and I've found that vertical stripes fix doesn't work. I've tested it with mlv_dump and it works fine. Is it possible for you to implement full vertical stripes fix from original raw2dng code ? I've found your fix code is different than raw2dng..

Thanks in advance
marekk

[DeafEyeJedi]

@marekk -- Did you forget to refresh the webpage (local host) after changing your options choices?

[marekk]

@marekk -- Did you forget to refresh the webpage (local host) after changing your options choices?

I've tried it but it doesn't work correctly. I've found that part of the mlvfs code responsible for vertical stripes fixing is the same but procedure to fix is much shorter.

Code: [Select]

void stripes_apply_correction(struct frame_headers * frame_headers, struct stripes_correction * correction, uint16_t * image_data, off_t offset, size_t size)
{
    if(correction == NULL || !correction->correction_needed) return;
    if(frame_headers->rawi_hdr.xRes % 8 != 0) return;
   
    uint16_t black = frame_headers->rawi_hdr.raw_info.black_level;
    uint16_t white = frame_headers->rawi_hdr.raw_info.white_level;
    size_t start = offset % 8;
    for(size_t i = 0; i < size; i++)
    {
        double correction_coeffficient = correction->coeffficients[(i + start) % 8];
        if(correction_coeffficient && image_data[i] > black + 64)
        {
            image_data[i] = (uint16_t)MIN(white, (image_data[i] - black) * correction_coeffficient / FIXP_ONE + black);
        }
    }
}



It removes vertical stripes only from some parts of the dng file. Full code from raw2dng removes all vertical stripes. I shot 256GB of mlv footage (6-10 sec), so I've verified it in a lot of files.
Mlv_dump output is nice but in Davinci Resolve 12.5 is presented as a folder with video and audio file. Correct naming scheme doesn't help. I think that Davinci REsolve is looking for specific tags in DNG file metadata, maybe "Reel Name"..

[dmilligan]

MLVFS caches the correction coefficients so it runs faster. Everything else is the same.

[marekk]

MLVFS caches the correction coefficients so it runs faster. Everything else is the same.

I think that correction coefficients is done in the same way, but "white" is calculated in different way..

[dmilligan]

White is only used to constrain the output values (to make sure the correction calculation doesn't produce values greater than white), therefore the only place you will see any effect is in areas that have been clipped.
Here's the equivalent code for the correction calculation in the code you posted:

Code: [Select]

    image_data[i] = (image_data[i] - black) * correction_coeffficient / FIXP_ONE + black;
    if(image_data[i] > white) image_data[i] = white;

Caching the white value as computed by the original raw2dng is a bad idea for our purposes, b/c this white level can actually be quite wrong if applied to a different frame (raw2dng does not compute the true white level, it computes a heuristic that is good "*for this particular purpose*"). If the analyzed frame is quite dark, the white level computed by this method will be much lower than the true white level; if you then take that value and apply it to a brighter frame, you will end up clipping a whole bunch of values that aren't actually clipped. So instead, we rely on the one from metadata because computing the true white level for every frame would severely impact performance.

Code: [Select]

        /** 
         * inexact white level will result in banding in highlights, especially if some channels are clipped 
         * 
         * so... we'll try to use a better estimation of white level *for this particular purpose* 
         * start with a gross under-estimation, then consider white = max(all pixels) 
         * just in case the exif one is way off 
         * reason: 
         *   - if there are no pixels above the true white level, it shouldn't hurt; 
         *     worst case, the brightest pixel(s) will be underexposed by 0.1 EV or so 
         *   - if there are, we will choose the true white level 
         */ 
        int white = raw_info.white_level * 2 / 3; 

        struct raw_pixblock * row; 
         
        for (size_t i = 0; i < size; i++) 
        { 
            white = MAX(white, image_data[i]); 
        } 


For these reasons, I really don't think it has anything to do with white level. My guess as to why it might not be working in some cases is actually the caching. Caching the coefficients makes two assumptions: the defect is constant over time (for at least for the duration of a clip), and all frames will show the defect (so we can analyze any frame and use it's coefficients to correct other frames). I think the first assumption is pretty safe, but the second one might not be if there's a dramatic change in the scene/lighting/exposure/etc.

You have a couple of options then:
- analyze every frame (slow -> you might loose real-time playback, IDK)
- analyze a random sample of frames, choose the 'best' coefficients (this has the potential of actually working better than the original algorithm, since you might be able to correct a frame that doesn't show the defect very strongly, based on other frames that do)
- analyze every N frames (maybe N is user settable)

[marekk]

Thanks for explanation.

How can I force it to analyze every frame ? I don't need real-time playback.

[dmilligan]

comment out line 983 in main.c

Code: [Select]

               //if(correction == NULL)