Magic Lantern Forum

Using Magic Lantern => Raw Video => Raw Video Postprocessing => Topic started by: dmilligan on August 31, 2014, 02:01:24 AM

Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 31, 2014, 02:01:24 AM
MLVFS - Magic Lantern Video File System


What is it?
An MLV "converter" that provides a virtual file system of converted data 'on the fly' using Filesystem in Userspace (FUSE). It allows you to "mount" an MLV file(s) which shows up as a directory of converted CDNGs. The data for these CDNGs is provided 'on the fly' by MLFVS as the data is being requested by whatever raw editor or post processing software you are using.

It's fast and efficient, and I've managed to get real-time playback in Premiere/SpeedGrade (on my 2012 MacBook Air!).

The code is written in C and works on Linux, Mac (via the OSX port of FUSE http://osxfuse.github.io/ ), and Windows (via the dokany project: https://github.com/dokan-dev/dokany thanks to g3gg0)



Advantages
- You don't have to convert first, you more or less "instantly" have "converted" DNGs
- You don't have to choose between keeping the original MLVs or converted DNGs or doubling disk usage to keep both.
- There's no need for a GUI, there already is one provided by the OS (i.e. the OS's file browser).
- Possible to vary the quality, pre-processing, bit depth, etc, of the DNGs on the fly



Updates
2014-09-02 Working Proof of Concept => There's much more work to be done (see below), but there is a working proof of concept, and a Mac build is available.
2014-09-08 Audio is working!
2014-09-09 Mac installer/service
2014-09-19 White balance mostly working
2014-09-20 AE working! and filesystem is now read/write (files are stored in .MLD directory in the real filesystem)
2014-09-26 Vertical banding correction
2014-09-27 Chroma smoothing
2014-10-08 Bad pixel correction
2014-10-09 Embedded mongoose webserver for configuring options at runtime
2014-10-13 Full quality dual ISO working (cr2hdr-20bit)
2014-10-30 Auto-linked audio in DaVinci Resolve
2015-02-05 Compressed file support
2015-02-17 Animated GIF Previews
2016-02-05 Fix for Focus Pixels
2016-02-07 Full Windows Support thanks to g3gg0 (via dokany)
2017-05-05 Lossless JPEG Support thanks to bouncyball


Source Code
https://bitbucket.org/dmilligan/mlvfs



Download
Mac
Install OSXFUSE (http://osxfuse.github.io/)
Download MLVFS.dmg (https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS.dmg)
Double click the "MLVFS.workflow" file and you should see the following prompt:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fbitbucket.org%2Fdmilligan%2Fmlvfs%2Fdownloads%2Fmlvfs_service_install.png&hash=9e5fa1b0e94861651fa7541aee7ca017)

(Click Install)

Now, you can right click a folder (with MLV files in it) and go to "Services" > "MLVFS"

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fbitbucket.org%2Fdmilligan%2Fmlvfs%2Fdownloads%2Fmlvfs_service.png&hash=ff9b670bd25ab340cf96443a07b20583)

You will be prompted to select a mount directory (it must be empty)
To stop mlvfs, simply click the eject button that shows up next to the mount point

If you get a security error, you may need to change your security settings to allow unsigned/downloaded programs to run (http://stackoverflow.com/questions/19551298/app-cant-be-opened-because-it-is-from-an-unidentified-developer)

Linux
Install FUSE via the normal way for your distribution. It's probably called something like libfuse-dev
Code: [Select]
apt-get install libfuse-dev
Download the code with git and compile using make

Windows
Install Dokany (https://github.com/dokan-dev/dokany/releases/download/v0.8.0/DokanInstall_0.8.0.exe)
Download MLVFS_x86 (https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip)

Usage
Start it via the command line like this:
Code: [Select]
cd <mlvfs_exe_dir>
mlvfs.exe <mount point> --mlv_dir=<directory with MLV files>



Web based GUI
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fbitbucket.org%2Fdmilligan%2Fmlvfs%2Fdownloads%2Fmlvfs_gui.png&hash=01f87f1a6a336e28b746fb07ff50547e)



Known issues and things not yet implemented
- Audio
- Raw image pre-processing
    - hot/cold pixel correction
    - vertical banding correction
    - chroma-smoothing
- Support for compressed MLV files
- I'm sure there's more stuff...



Contributors
ayshih
dmilligan
g3gg0 (Windows port)

Credits
ML developers (esp. g3gg0 and a1ex for designing the MLV format and providing reference converters)
Developers of open source MLV=>CDNG converters: chmee and baldand => your work and source code was a great help in understanding the CDNG format and requirements



Some helpful tutorials from @reddeercity and @DeafEyeJedi





Original Post
Quote
So, what would be the 'best' or 'easiest' possible workflow for MLV? Native support for MLV format in whatever raw video processing tool you use (via open source code!), obviously.

There are ways to do this, many programs provide an API for third party plugins to implement something like this, but any implementation would be a lot of work and limited to working with one single app (as APIs are going to vary). Then I had the thought, what if we went down to a lower level of abstraction that all apps will have in common: the file system itself. So I started looking around the internet for an easy way to create virtual file systems, and found this awesome API that allows you to do just this in a user land program: FUSE (http://fuse.sourceforge.net/)

So the basic idea is that you can "mount" an MLV file(s). The converter program provides info and converted data to the OS "on demand" for this virtual file system. This is sort of like how you can mount a disk image or traverse into zip files in windows without actually extracting it.

Here's how it works: you just start the program, point it to some 'live' directory (where you can drop MLV files) and give it a mount point. The mount point directory then shows up as having folders for each MLV file containing CDNGs for each frame.

So I have started working on this and I have some sort of VERY preliminary a proof of concept working. It is by no means usable yet, there are all kinds issues and things not implemented yet, but I can at least setup the file system and provide some semblance of the data that's going to be in the DNG (not readable yet, b/c I haven't created the DNG header with all the correct tags and such, and there are probably issues with my bit unpacking code) so that I can test speed and such. Basic DNG conversion is working and it seems to be pretty fast, so far it's not really noticeable that you're using a virtual file system.

If you're a coder and would like to help, that'd be fantastic, esp if you have a good knowledge of the DNG spec (I don't and I'm learning all of this as I go).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: Audionut on August 31, 2014, 02:16:57 AM
Looking forward to see where this ends up.  Nice work dmilligan.

Something useful for windows here?  http://sourceforge.net/p/fuse/wiki/OperatingSystems/#windows
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: ayshih on August 31, 2014, 05:04:54 AM
It sounds like a promising approach!

I'm interested in helping, but it'd be nice if there was a way for it to work on Windows.  Unfortunately, it looks like the only actively developed implementation may be DokanX (https://github.com/BenjaminKim/dokanx/), and I don't think it uses the FUSE API.  (See the list of relevant projects on this Wikipedia page (http://en.wikipedia.org/wiki/Filesystem_in_Userspace).)  But, I do have an OS X machine and Linux VMs, so it's not an non-starter.

I've only started reading up on this stuff, but I'll point out that OSXFUSE says that it provides an API that is a superset of the FUSE API, so you may want to make sure you're using just the FUSE API to ensure Linux portability.  Of course, at such an early stage, that's a very low-priority concern.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: dmilligan on August 31, 2014, 05:44:27 AM
Great! I'm pretty sure I've only used the basic FUSE API (I've only included fuse.h, which I assume is the basic API), and I only use basic C libs and general posix stuff (no apple specific libs), so it should be totally portable to linux. Most of the FUSE related stuff is pretty well implemented (we could do more advanced things later on with it), the real areas that needs help is:

Bit unpacking - I didn't want to use the bit fields struct stuff like ML and mlv_dump use, since this is non-standard compiler stuff (and not supported by Apple's compiler), so I wrote a more generic one, but IDK if it's really correct or efficient

DNG generation - I didn't want to use chdk-dng.c b/c that code is an absolute mess, also I need to write specific sections of the file to buffers on demand, not the entire file at once (that's the way the FUSE api works, a specific section of the file is requested, and you write that section of the file into a buffer). Also, it's not thread safe (nor is the vast majority of stuff from mlv_dump), which this needs to be since the FUSE API is multithreaded and requests for data can be concurrent (multiple programs accessing different or even the same file at the same time, etc.).

there are also areas of mlv parsing that will need to be improved: there's no support for chunks yet, it should generate and use index files to make finding the frame blocks faster, no audio support, etc.

If you have any questions about the code I've written so far, I'll be happy answer them.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: chmee on August 31, 2014, 07:38:38 AM
@dmilligan. good idea.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: surami on August 31, 2014, 08:04:48 PM
@dmilligan: awsome thought.

I know it's too early to write about it, but I think it would be also nice to add and option to apply DNG Camera Profiles (.dcp files) too. I mean the index file should contain the information, that during the virtual file system upbuilding which .dcp file should be applied to the DNGs. There is a utility, the dcpTool (http://dcptool.sourceforge.net/Introduction.html) which could be a good starting point if a xml file is needed for this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: dmilligan on September 02, 2014, 11:08:46 PM
okay, proof of concept is basically working, big thanks to ayshih who has done a lot of work (and fixed my bugs!)

ACR and Resolve can both correctly read converted DNGs. Resolve playback is slow, but it's slow on my system anyway, I think it's the GPU. Hopefully I can get Premiere/SpeedGrade working, I think it's just a matter of getting all the right DNG tags, any thoughts @chmee?

Code: [Select]
david$ ./dng_validate -v /Users/david/mlvfs/M23-1157.MLV/00000000.DNG
Validating "/Users/david/mlvfs/M23-1157.MLV/00000000.DNG"...

Uses little-endian byte order
Magic number = 42

IFD 0: Offset = 8, Entries = 33

NewSubFileType: Main Image
ImageWidth: 1728
ImageLength: 972
BitsPerSample: 16
Compression: Uncompressed
PhotometricInterpretation: CFA
FillOrder: 1
Make: "Canon"
Model: "Canon EOS 60D"
StripOffsets: Offset = 65536
Orientation: 1 - 0th row is top, 0th column is left
SamplesPerPixel: 1
RowsPerStrip: 972
StripByteCounts: Count = 16896
PlanarConfiguration: 1
Software: "MLVFS"
*** Warning: IFD 0 DateTime has unexpected count (1) ***
*DateTime: ""
CFARepeatPatternDim: Rows = 2, Cols = 2
CFAPattern:
    Red      Green
    Green    Blue
ExifIFD: 410
DNGVersion: 1.4.0.0
UniqueCameraModel: "Canon EOS 60D"
LinearizationTable: 16383 entries
    Table [0] = 0
    Table [1] = 1
    Table [2] = 2
    Table [3] = 3
    Table [4] = 4
    Table [5] = 5
    Table [6] = 6
    Table [7] = 7
    Table [8] = 8
    Table [9] = 9
    Table [10] = 10
    Table [11] = 11
    Table [12] = 12
    Table [13] = 13
    Table [14] = 14
    Table [15] = 15
    Table [16] = 16
    Table [17] = 17
    Table [18] = 18
    Table [19] = 19
    Table [20] = 20
    Table [21] = 21
    Table [22] = 22
    Table [23] = 23
    Table [24] = 24
    Table [25] = 25
    Table [26] = 26
    Table [27] = 27
    Table [28] = 28
    Table [29] = 29
    Table [30] = 30
    Table [31] = 31
    Table [32] = 32
    Table [33] = 33
    Table [34] = 34
    Table [35] = 35
    Table [36] = 36
    Table [37] = 37
    Table [38] = 38
    Table [39] = 39
    Table [40] = 40
    Table [41] = 41
    Table [42] = 42
    Table [43] = 43
    Table [44] = 44
    Table [45] = 45
    Table [46] = 46
    Table [47] = 47
    Table [48] = 48
    Table [49] = 49
    Table [50] = 50
    Table [51] = 51
    Table [52] = 52
    Table [53] = 53
    Table [54] = 54
    Table [55] = 55
    Table [56] = 56
    Table [57] = 57
    Table [58] = 58
    Table [59] = 59
    Table [60] = 60
    Table [61] = 61
    Table [62] = 62
    Table [63] = 63
    Table [64] = 64
    Table [65] = 65
    Table [66] = 66
    Table [67] = 67
    Table [68] = 68
    Table [69] = 69
    Table [70] = 70
    Table [71] = 71
    Table [72] = 72
    Table [73] = 73
    Table [74] = 74
    Table [75] = 75
    Table [76] = 76
    Table [77] = 77
    Table [78] = 78
    Table [79] = 79
    Table [80] = 80
    Table [81] = 81
    Table [82] = 82
    Table [83] = 83
    Table [84] = 84
    Table [85] = 85
    Table [86] = 86
    Table [87] = 87
    Table [88] = 88
    Table [89] = 89
    Table [90] = 90
    Table [91] = 91
    Table [92] = 92
    Table [93] = 93
    Table [94] = 94
    Table [95] = 95
    Table [96] = 96
    Table [97] = 97
    Table [98] = 98
    Table [99] = 99
    ... 16283 more entries
BlackLevel: 2047.00
WhiteLevel: 15000
DefaultCropOrigin: H = 0.00 V = 0.00
DefaultCropSize: H = 1728.00 V = 972.00
ColorMatrix1:
      0.6719  -0.0994  -0.0925
     -0.4408   1.2426   0.2211
     -0.0887   0.2129   0.6051
AsShotNeutral: 0.4736 1.0000 0.6240
BaselineExposure: +0.00
ActiveArea: T = 0 L = 0 B = 972 R = 1728
*Tag51044: SRational = 24000/1001
BaselineExposureOffset: +0.00
NextIFD = 0

Exif IFD: Offset = 410, Entries = 5

ExposureTime: 1/30.0 sec
FNumber: f/1.80
ISOSpeedRatings: 400
SensitivityType: ISO Speed
ExifVersion: 2.30
NextIFD = 0

*** Warning: Too little padding on left edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on top edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on right edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on bottom edge of CFA image (possible interpolation artifacts) ***
Raw image read time: 0.008 sec
Linearization time: 0.005 sec
Interpolate time: 0.041 sec
Validation complete
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: g3gg0 on September 03, 2014, 10:58:08 AM
just... wow :)
amazing!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: dmilligan on September 03, 2014, 09:50:29 PM
Premiere/SpeedGrade are working and playback is realtime!!!!!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: Danne on September 03, 2014, 09:53:23 PM
 :o
holy crap!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: LRF on September 03, 2014, 10:48:54 PM
Guys, you never cease to amaze me  :D

If people with such skills and attitude were working in one company... Damn, you would turn the market inside-out within six months!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: dmilligan on September 03, 2014, 11:17:28 PM
Okay, now that it is basically working I have posted a preliminary binary for people to try. You'll need to be somewhat familiar the command line to get it started, but eventually I plan on adding some kind of installer or setup utility. See the OP for details. Also please keep in mind that there are all sorts of things not implemented yet (like audio, white balance, processing options such as banding correction and chroma smoothing, etc.).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: reddeercity on September 03, 2014, 11:36:40 PM
This look very good ! would this work with other Software like FCPX or is this just for Adobe Workflow ?
Never the less I will give it a try .
Thanks   8)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: Frank7D on September 03, 2014, 11:58:06 PM
But not for Windows (yet), right?
Edit: Never mind, I read the OP.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: tin2tin on September 03, 2014, 11:58:35 PM
Pismo is crossplatform (mac,linux, win): http://www.pismotechnic.com/
Can it be used as an alternative to fuse?

(It was used for a virtual filesystem for Avisynth: http://www.turtlewar.org/avfs/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: chmee on September 04, 2014, 01:02:21 AM
@dmilligan
the original bmcc-linearization-table is inside my cdng's. only copy/paste the tag and the values. or pipe the dng_validate -v output into a file and parse them into a binary. (or look into my sources (https://bitbucket.org/chmee/raw2cdng/src/3b267af91e6caaf81bbe47131773a85e4500d09d/csharp2010_code/raw2cdng_v2/?at=master). There's the dng-header named dngtemplate20.dng.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to DNG converter (WIP)
Post by: ayshih on September 04, 2014, 04:08:43 AM
Pismo is crossplatform (mac,linux, win): http://www.pismotechnic.com/
Can it be used as an alternative to fuse?

(It was used for a virtual filesystem for Avisynth: http://www.turtlewar.org/avfs/)
Interesting!  It looks Pismo's file mounting may actually be only for Windows, but it's not a problem to have platform-specific code.  I'll definitely look into this.

Edit: Okay, there is cross-platform support for Pismo, but it may still be preferable to stick with fuse for OS X and Linux.

Alternatively, it hasn't been explicitly stated, but the MLVFS proof of concept works on Linux as well.  You'll have to grab the MLVFS code and compile your own executable, but you probably already have fuse installed.  I imagine one could configure a Linux VM in such a way as to achieve Windows support.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: baldand on September 05, 2014, 06:42:17 PM
I just tested mlvfs on Linux (Ubuntu 14.04). Nice work! :-)

I tried to playback the DNGs with MlRawViewer. There were a couple of small problems.

I had to mount using full path names, not with ~ in the mlv_argument. It didn't give an error when mounting, but the mount dir could not be opened.

The BaselineExposure tag gave a rational value of 0/0 which caused a divide by zero for me. Easy for me to workaround with a test though.

Another problem was that because the dir is read only, I can't write my per-clip config files into the DNG directory, so it's not possible to store any parameters or LUT settings for use when exporting. Export to MOV works, but with limited functionality. Well, not much to be done about that one.

I also noticed that CPU usage was quite high in the mlvfs process - something like 70% of one i7 core. Playing back through mlvfs was a bit heavier overall than direct playback, but it was still real time.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 05, 2014, 07:39:46 PM
This is simply amazing! Just created a new preference folder in davinci resolve 10 and the mlv files plays in realtime. No conversion. Dng:s magically appears in the osxfuse mount :o :o. Dmilligan breakthrough. All creds to participant coders and so on.

Couldn,t get sequence to work with After effects cs 6 nor to play in mlrawviewer, tried version 1.2.3, 1.1.7 and 1.1.6. (noticed Baldands reply above now regarding full pathnames)
Smooth as butter in davinci resolve. I,m on a macbook pro 15 inch 2.6 ghz, OSX mavericks.

Tried to convert some dual iso movie files but it wouldn,t sing. Tried Alex lates cr2hdr20bit binary and in lightroom. Got the following:

Output file     : /Users/Daniel/Desktop/mlv/M05-1941.MLV/00000005.DNG (already exists, overwriting)
Bus error: 10

Only info and of low priority of course.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 05, 2014, 08:15:59 PM
Thanks for the feedback!

I'll fix the BaselineExposure tag (interestingly, dng_validate didn't complain about that).

I'm not sure what the deal is with ~ is, but I have noticed that too. It seems that the stdio.h functions don't accept it.

About it being readonly: I can fix that (and I plan to), but it's going to be some work. Basically I can do a loopback to the original directory and store the actual data there (FUSE supports both read and write), but it means implementing all the FUSE API's write callbacks (since it's read only right now I only need to implement the read functions) and doing some type of path wrangling to manage where the files are put and how they map between the two file systems. It's certainly possible, but a lot of work.

The readonly thing seems to also be the reason AE isn't working, it tries to write it's own XMP settings, which it can't b/c the filesystem is readonly.

CPU usage is probably from the 14 =>16 bit conversion. There may be some possibility for performance improvement there (IDK how efficient my bit unpacking code is or could be, but it works), but if it's already realtime, it doesn't seem particularly important :P



Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 05, 2014, 08:29:21 PM
ah, ok read only. Fun is i can open the dng:s and store changes made in acr and photoshop.
Very promising this indeed.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on September 06, 2014, 05:39:40 AM
Will be watching this thread but will stay away from trying this awesome sw just yet since the discussion is too technical at the moment. Hope you folks can release the installer soon.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 06, 2014, 10:49:20 AM
Cool! When I right click a dual iso movie file I can give the file read and right priviligies and the files can convert fine with cr2hdr20 bit and also open in after effects.

I have to drag the folders to desktop to be able to change permissions.


Apply permissions to all items in a folder or a disk

    Select a folder or a disk, then choose File > Get Info.
    Click the lock icon to unlock it, then enter an administrator’s name and password.
    Choose "Apply to enclosed items" from the Action pop-up menu (looks like a gear).

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on September 06, 2014, 04:54:15 PM
I will experiment with the Linux (Ubuntu 14.04.1) in a Virtual Box (https://www.virtualbox.org/) on Windows 7 in the evening. It should work in theory, I will report back:
1. Run the Linux in Virtual Box on Windows 7.
2. Run the MLVFS in the virtualized Linux system.
3. Open the DNG sequence in Windows 7.

Update: I can't get it working, the latest stable (4.3.15) Virtual Box drops error for me, but I'm on that to solve somehow.
Update2: I could get it working with the new stable 4.3.16 VB. It's a very long journey to do, but it works. :) As I have time I write a step by step "how-to". ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 06, 2014, 11:25:48 PM
I had to mount using full path names, not with ~ in the mlv_argument. It didn't give an error when mounting, but the mount dir could not be opened.
Paths now have typical shell expansion (e.g., ~), and there is error handling for bad paths.  I'll let dmilligan update the binaries for those who can't compile.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 07, 2014, 02:54:33 PM
Created a short video tutorial on how to mount the fuse app according to the terminal commands provided in the first post from David Milligan.
Got some questions about this in my inbox.
This workflow will likely change soon due to the early development stages. But for now...


*Update Dmilligan created a automated gui through services in the first post. Terminal workflow on mac not needed anymore.

Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: nigel on September 07, 2014, 05:45:28 PM
Oh wow. Is it time to get a Mac? Hahah
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 08, 2014, 07:21:16 AM
Cool! When I right click a dual iso movie file I can give the file read and right priviligies and the files can convert fine with cr2hdr20 bit and also open in after effects.
It's easy enough to understand why After Effects complains; the virtual filesystem is configured to not have write permission, and you work around that.  What I don't understand is your statement about being able to convert with cr2hdr.  The filesystem doesn't have any actual implementation of the write call.  So, what exactly happens when you run cr2hdr?  Where do the converted DNGs go?  I don't have a dual-ISO video to try this out myself.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 08, 2014, 10:42:36 AM
- Drag the dual iso dng folders to my desktop, then give the folder read, write permissions
- Convert the files using cr2hdr20bit as usual. Original dng files are being overwritten to the converted one.

Question. How do you explain acr being able to store write changes directly to dng files in the original fuse folder but not in after effects which needs permissions? Is acr overriding the write, read permission structure?

Thanks
/D
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 08, 2014, 02:18:59 PM
- Drag the dual iso dng folders to my desktop, then give the folder read, write permissions
- Convert the files using cr2hdr20bit as usual. Original dng files are being overwritten to the converted one.
Ah, okay.  To be clear, when you drag the folders to your desktop, you are converting the virtual DNGs to actual DNGs.

Question. How do you explain acr being able to store write changes directly to dng files in the original fuse folder but not in after effects which needs permissions? Is acr overriding the write, read permission structure?
When you use ACR on files in a read-only directory (e.g., on a CD), it can't create the sidecar XMP files (which is also its fallback for read-only DNG files), so the changes are instead stored in a central Camera Raw database.  That's how ACR can work with the (current) read-only nature of the MLVFS filesystem.

I don't know how files are distinguished in the Camera Raw database, so it's possible you may lose your ACR modifications if you mounted the MLV file in a different location.  Certainly, the ACR modifications would not persist if you mounted the MLV file on a different computer.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 08, 2014, 02:58:38 PM
Theoretically, dual-ISO support would be built into this tool (that would be the best case scenario). However, I don't see this as a viable option. I don't think there's really any way to make dual-ISO conversion fast enough to be useful with this on-the-fly way of doing things (unless somebody wants to implement dual-ISO conversion in hardware on an FPGA or something :P), and make it fit in with the way the FUSE API works (processing small chunks of a frame on demand, rather than the whole frame at once).

I think for dual-ISO you're just going to need to go a more standard processing route, permanently converting the DNGs and then permanently converting those with cr2hdr (you can use this tool for permanent conversion, by copying the DNGs out of the virtual filesystem, but then there's no real advantage to using this tool over any of the other ones).

It would be possible for this tool to generate "low-res proxies" of dual-ISO clips for editing or previewing (such as by just skipping the bright rows in the output) though.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 08, 2014, 03:28:25 PM
Ah, okay.  To be clear, when you drag the folders to your desktop, you are converting the virtual DNGs to actual DNGs.
Yes, that must be the case. But I am simply copying them from the mount.

When you use ACR on files in a read-only directory (e.g., on a CD), it can't create the sidecar XMP files (which is also its fallback for read-only DNG files), so the changes are instead stored in a central Camera Raw database.  That's how ACR can work with the (current) read-only nature of the MLVFS filesystem.
Thanks for claryfiyng.

I think for dual-ISO you're just going to need to go a more standard processing route, permanently converting the DNGs and then permanently converting those with cr2hdr (you can use this tool for permanent conversion, by copying the DNGs out of the virtual filesystem, but then there's no real advantage to using this tool over any of the other ones).
It is till faster to copy the dng files from the fuse mount than exporting from a converter right?
So, there won,t be a possibility to overwrite the mounted dng:s since they,re only virtual ones from the orginal mlv files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 08, 2014, 05:29:30 PM
It is till faster to copy the dng files from the fuse mount than exporting from a converter right?
Theoretically, no. When you copy from the FUSE mount, you essentially turn this program into a traditional converter, so it should take about the same amount of time as a traditional converter, unless my conversion implementation is more efficient. Which might actually be the case right now, since I don't do any pre-processing on the files like banding correction or chroma smoothing, and also MLFVS supports multithreading via the FUSE API, so you might be getting more CPU usage. However, I imagine speed is mostly I/O limited, not CPU limited, and if this is the case, multithreading would have no effect. Feel free to do a speed comparison, though, I'd be interested to know (hint: use time (http://en.wikipedia.org/wiki/Time_(Unix)) to time a cp (http://en.wikipedia.org/wiki/Cp_(Unix)) of the virtual DNGs to the real filesystem).

So, there won,t be a possibility to overwrite the mounted dng:s since they,re only virtual ones from the orginal mlv files?
No, that's not really possible, though I guess theoretically you could implement something where the CDNGs are writeable and the written data goes back into the MLV file VIDF blocks, modifying the original MLV file. That sounds VERY hard to implement.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on September 08, 2014, 05:33:28 PM
I could install Ubuntu (14.04.1) inside the VirtualBox (test build 4.3.15) (https://www.virtualbox.org/ticket/13187#comment:57) but the MLVFS isn't working. I think it's because I'm trying inside a virtual machine, so there is too much virtualisation "layer" for the system. Now I will try the Cygwin (http://cygwin.com/) (never used it), maybe it will work.

Update: I could get it working with the new stable 4.3.16 VB. It's a very long journey to do, but it works. :) As I have time I write a step by step "how-to". ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 08, 2014, 06:33:35 PM
MLVFS works fine in a virtual machine, but just within that virtual machine's local filesystem.  The way that VirtualBox and Parallels (at least) provide support for shared folders, they are local folders in the host machine, and thus MLVFS can't create its virtual filesystem there.

Instead, you'd probably have to use more traditional methods of sharing the MLVFS mount point: share the folder over a network as if they were two separate computers (e.g., using SMB).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on September 08, 2014, 07:25:32 PM
@ayshih: Thanks for your explanation, so I should use Samba server. I will try, but I never did this yet, so it will be a long journey for me. Cygwin is not for this job.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 08, 2014, 08:59:37 PM
So, compared the mlv_dump v 1.0 which ran vertical stripes correction to the speed of the mlvfs conversion. Used the "hints" (time, cp) commands from links provided from dmilligan (thanks).

The file is 9.52gb big.

The mlvfs conversion

real   0m35.285s
user   0m0.043s
sys   0m12.761s

The mlv_dump conversion

real   2m4.544s
user   1m36.526s
sys   0m10.662s

I also did some batch conversion around 38gb comparing mlrawviewer and mlvfs and the mlvfs was about 30 seconds faster around 5min30sec while mlrawviewer finished around 6min. Not whole lot of difference there.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 08, 2014, 10:58:03 PM
cool, thanks for the tests...


Audio is working!!!!

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: senzazn12 on September 09, 2014, 03:52:44 AM
Praying this gets ported to Windows somehow.  ::)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 09, 2014, 04:51:57 AM
Instead, you'd probably have to use more traditional methods of sharing the MLVFS mount point: share the folder over a network as if they were two separate computers (e.g., using SMB).
Oof, it's not for the faint of heart, but I've figured out how to get this approach working for Windows users.  You have to jump through a couple of extra hoops, and I'm sure performance will suffer because of the layers.

In the Linux VM:
Code: [Select]
mlvfs <mount point> --mlv_dir=<directory with MLV files> -o allow_other
In the Windows host, connect to the shared folder, e.g.,
Code: [Select]
\\<Linux hostname>\<mount point share>
Praying this gets ported to Windows somehow.  ::)
I'm still looking into using Pismo File Mount to provide native support for Windows, because the above is painful.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 09, 2014, 11:15:43 AM
oh i just remind of my good old symbian days.
back then i made a webdav server in C#.

using that tool we could get a cheap and easy-to-use MLVFS pendant on windows.
let me see....
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 09, 2014, 12:50:10 PM
updated the OP with the Windows instructions
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 09, 2014, 05:46:51 PM
short update:

i patched my mini WebDAVServer (http://upload.g3gg0.de/pub_files/d224e3b20042ab2ecce9a0912a347cf2/WebDAVServer.zip) to virtualize MLV files just like MLVFS does.

on windows you just have to mount the file system using
> net use x: \\localhost@8080
where 8080 is the port you specified in configuration dialog.
then drive X: shows you the content of the shared directory/card and allows you to go into MLV files.
you can even mount it via network. please dont use auth string, as windows confuses it with NTLM auth crap.

it only supports
 - MLV->JPG
 - MLV->WAV
 - experimental saving files in MLV folder (creates a <mlv-file>_store directory)
 - no DNG support

i will create a separate post for that and ask people to help.
creating sane DNGs is not really fun and takes its time to test and tweak.

@chmee:
interested in adding your C# DNG code to this tool?
will post it on bitbucket.org
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 09, 2014, 11:51:56 PM
There's now a simple installer and service for Mac, see the OP (and let me know that it works)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jose_ugs on September 10, 2014, 12:55:40 AM
wow! mounting MLVs! keep it up, and hopefully you can figure out a Win7 solution
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 10, 2014, 01:07:14 AM
Works perfect! With audio aswell. How did you do that. Amazing work. Very convenient to have integration with services.
Folder names can,t contain spaces on mount point folder. One word only.
cool stuff.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: menoc on September 10, 2014, 01:34:55 AM
I installed it and works pretty well on my mac pro. Only thing is, for some reason Premiere  Pro CC 2014 won't let me import the dngs as an image sequence - the check box is greyed out. Anyone knows how to fix it?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: chmee on September 10, 2014, 02:24:46 AM
Menoc. You dont need to. It will be recognised AS sequence.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 10, 2014, 02:34:27 AM
Folder names can,t contain spaces on mount point folder. One word only.
should be fixed now

I installed it and works pretty well on my mac pro. Only thing is, for some reason Premiere  Pro CC 2014 won't let me import the dngs as an image sequence - the check box is greyed out. Anyone knows how to fix it?
Just select the first DNG and click import, the whole thing will be imported (don't worry about the check box). Or you can just use the "Media Browser", the DNGs should show up as a single clip and you can drag/drop it into your timeline.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 10, 2014, 07:23:18 AM
Folder names working.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: KurtAugust on September 10, 2014, 08:55:35 AM
Seeing this evolve so fast is like watching the Big Bang from your couch with a lovely cup of coffee. Very clever thinking!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: timbytheriver on September 10, 2014, 10:37:19 AM
Thank-you @dmilligan et al for this breakthrough work.

After following your new .dmg based instructions I'm getting an error running on Mac OSX 10.7.5

Operation fails after after selecting a mount folder:

=====

! The action "Run Shell Script" encountered an error.

I click > Show Workflow

The workflow appears > I click it. The workflow dialogue boxes appear, containing:

~/Library/Services/MLVFS.workflow/Contents/mlvfs "$2" --mlv_dir="$1"

Could you help please?

Cheers

tim
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 10, 2014, 12:44:08 PM
Mac OSX 10.7.5
I have not tested it on anything but Mavericks (10.9). You may need to compile it yourself if you want it to work on older versions.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: timbytheriver on September 10, 2014, 01:20:37 PM
Thank-you! I'd better learn to compile then… ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: menoc on September 11, 2014, 01:50:09 AM
Quote
Just select the first DNG and click import, the whole thing will be imported (don't worry about the check box). Or you can just use the "Media Browser", the DNGs should show up as a single clip and you can drag/drop it into your timeline.

Great thanks guys! That works in Premiere and Media Encoder. It does not work in After Effects CC 2014 though- I get an error
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fwww.dropbox.com%2Fs%2F67k7r7mjhfxgn8x%2FScreen%2520Shot%25202014-09-10%2520at%25207.28.46%2520PM.png%3Fdl%3D0&hash=41fd321d0e272c10976fcbad495565b2)
https://www.dropbox.com/s/67k7r7mjhfxgn8x/Screen%20Shot%202014-09-10%20at%207.28.46%20PM.png?dl=0
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 11, 2014, 02:16:59 AM
read this thread starting at reply #18
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mg_dev on September 11, 2014, 05:38:44 PM
Hi guys. I started tracing this topic some time ago, and I decided to share my conclusion :)

First, I'm interesting this topic from developer point of view rather than end user. I tried to convert MLV movie to set od DNGs using modified (fork) raw2cdng. I've prapared NVidia CUDA depth converters (in dynamic linked library). Sadly, summary processing time were not reduce too much because of hard drive bottleneck.. Creating 100 or 1000s  1-2Mb DNGs give huge overhead for disk-writing efficiency. 
 
Second, We all know - the idea of mounting MLV movies as a transparent 'drive' or folder is really good. But my approach is different a little bit - I will try to store DNGs in .ISO or .UDF (universal disk format). Output can be mounted on every platform. Writing huge data to one file is often more efficient than little data to  many files..
Or maybe someone tested this solution?..

I know that "ISO" access to DNGs won't be easier than mounting MLV in MLVFS, but it is kind of solution:)

I'll let You know, if I get better processing results :). And of course, I'll share my project if You want.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 13, 2014, 04:19:06 AM
I'm happy to announce that I've gotten an analogous implementation of MLVFS running on Windows using Pismo File Mount Audit Package (http://www.pismotechnic.com/pfm/ap/).  After installing PFMAP, download the MLV formatter DLL from Bitbucket (https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll) and then register it with PFMAP by entering on a command prompt:
Code: [Select]
pfm register mlvfs.dll
You can then mount any individual MLV video by right-clicking on it and selecting "Quick Mount", which will turn that video into a virtual folder containing the virtual DNGs.  The virtual folder also has support for creating and modifying additional files (e.g., XMP sidecar files), but the files are not preserved when the video is unmounted.

It's far better than the workaround with the Linux VM, but I don't know if it's better than g3gg0's WebDAV approach because I haven't tried it. 
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 13, 2014, 05:30:55 AM
Hi ayshih,
I tried your pismo method with no success.
I have been using pismo to mount avisynth files for years now, so I already had PFMAP (I believe).
I downloaded the file (but it was just "mlvfs", with no ".dll" on the end).
I changed the named to add ".dll" on the end.
I opened a command prompt, went to the directory with the file, and registered the file per your instructions.
I tried to quick mount an mlv file, but got a message saying it was an unsupported file format.

Edit:  Never mind, the problem was all me; I was using an old build of PFMAP. Once I installed the latest one I was fine.

Thanks for this great option! Can't wait to play with it!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 13, 2014, 06:54:42 AM
Using pismo, I am getting no wav files in the virtual folders.

Also, the folders are read-only by default, so I have to right-click them and make them writable in order to import the virtual DNGs into After Effects.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 13, 2014, 07:16:59 AM
Using pismo, I am getting no wav files in the virtual folders.
Yup, I haven't implemented audio yet, but it shouldn't be too difficult since it's up and running on the FUSE side.

Also, the folders are read-only by default, so I have to right-click them and make them writable in order to import the virtual DNGs into After Effects.
Hmm, the folder itself should already be writable, but I guess After Effects really wants the DNGs to be writable.  Be careful when making the DNGs writable; things could break in weird and fantastic ways because I didn't protect all of the possible calls (on the presumption that the DNGs were read-only).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 13, 2014, 04:03:15 PM
One other thing (using pismo): When I open the virtual DNGs in ACR or After Effects with white balance set to "As Shot", the "Temperature" is incorrect. For example, my camera (7D) was set to 5100 and the DNG opened with 2600.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 13, 2014, 04:22:14 PM
The DNG writing code for white balance needs work, because it doesn't have model-specific or preset-specific white-balance multipliers.  Perhaps you should create an issue on Bitbucket!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 13, 2014, 05:21:40 PM
Someone explain to me the math behind how to get from temperature to RGB multipliers, and I'll fix it :)

Quote
AsShotNeutral specifies the selected white balance at time of capture, encoded as the coordinates of a perfectly neutral color in linear reference space values.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 13, 2014, 07:08:58 PM
Someone explain to me the math behind how to get from temperature to RGB multipliers, and I'll fix it :)
https://bitbucket.org/hudson/magic-lantern/src/e12ad3d84e7bc5b61e27c9e9e5a7c0423b39598c/contrib/g3gg0-tools/MLVViewSharp/DebayerBase.cs?at=unified

basically this:

            /* get the XYZ --> cone reference whites for requested temperatures */
            Matrix dst = coneDomain * KelvinToXYZ(ColorTemperature);
            /* verify: do all our matrices that came from dcraw really convert raw to D65? */
            Matrix src = coneDomain * KelvinToXYZ(6500);

            /* scale coordinates in cone color space */
            Matrix xyzScale = new Matrix(3, 3);
            xyzScale[0, 0] = dst[0] / src[0];
            xyzScale[1, 1] = dst[1] / src[1];
            xyzScale[2, 2] = dst[2] / src[2];

            /* finally scale colors */
            Matrix xyzKelvinWb = coneDomain.Inverse() * xyzScale * coneDomain;

            /* now combine the whole thing to get RAW --> RAW-WB --> XYZ --> Kelvin-WB --> XYZ --> (s)RGB --> RGB-WB */
            CamToRgbMatrix = WhiteBalanceMatrix * RGBToXYZMatrix.Inverse() * xyzKelvinWb * XYZToCamMatrix.Inverse() * WhiteBalanceMatrixRaw;
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 13, 2014, 11:05:25 PM
EDIT: okay, I think I've basically got it

What about G/M shift?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 14, 2014, 02:19:02 AM
never tried.

would set up a function that gives you the RGB values for the G/M shift, which is quite simple.  ([R, G, B] = [128-x, 128+x, 128-x])
then transform it to XYZ and straight forward to bradford cone domain.
then just do the same scaling as with kelvin.

basically you could do this with any white point in any color space.

so:
Code: [Select]
            Matrix gmShiftRef = new Matrix(3, 1);
            gmShiftRef [0] = 0.5;
            gmShiftRef [1] = 0.5;
            gmShiftRef [2] = 0.5;

            Matrix gmShiftRgb = new Matrix(3, 1);
            gmShiftRgb[0] = 0.5-gmShift;
            gmShiftRgb[1] = 0.5+gmShift;
            gmShiftRgb[2] = 0.5-gmShift;

            Matrix dst = RGBToXYZMatrix* gmShiftRgb;
            Matrix src = RGBToXYZMatrix* gmShiftRef;

            Matrix coneScale = new Matrix(3, 3);
            coneScale [0, 0] = dst[0] / src[0];
            coneScale [1, 1] = dst[1] / src[1];
            coneScale [2, 2] = dst[2] / src[2];

            Matrix xyzGMShift = XYZBradford.Inverse() * coneScale * XYZBradford;

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 14, 2014, 06:19:10 AM
MLVFS for Windows using PFM now extracts audio as well (DLL download (https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll)).  Let me know if you encounter any bugs!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mk11174 on September 14, 2014, 06:27:25 AM
This is pretty cool, can the RAW files somehow be added to work with this. Or is it only possible with MLV?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mario1000 on September 14, 2014, 07:23:14 AM
MLVFS for Windows using PFM now extracts audio as well (DLL download (https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll)).  Let me know if you encounter any bugs!

Hi ayshih
thanks a lot for your work but the download link does not work (it says that it found a dead link). Could you please fix this.
Thanks in advance

Edit: download link works!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 14, 2014, 07:41:47 AM
This is pretty cool, can the RAW files somehow be added to work with this. Or is it only possible with MLV?
I don't think there's any limitation precluding implementing RAW support, although the lack of metadata is a little annoying.  However, I myself am more interested in making sure we fully support MLV files rather than worrying about the "older" format.  I encourage any interested developers to contribute to the code, of course!

thanks a lot for your work but the download link does not work (it says that it found a dead link). Could you please fix this.
Thanks for letting me know!  Looks like Bitbucket got a little confused, so I've re-uploaded it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 14, 2014, 06:41:48 PM
(using pismo)
Now I have audio (.wav) files also; thanks!
One thing I notice: when I import the (virtual) sequence into After Effects, an .xmp (sidecar) file appears in the virtual folder. I believe these files normally contain the ACR settings used to import the sequence.
However, when I unmount the mlv and then mount it again, the xmp is gone. Maybe this is a good thing, since it implies AFX is not altering the mlv file.
If I want to retain the AFX/ACR import settings more permanently, I can save an xmp manually in the ACR settings folder (or maybe just drag the xmp from the virtual folder into a real folder).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 14, 2014, 07:03:37 PM
(using pismo)
What controls the naming of the (virtual) DNGs and WAVs?
All my DNG names are of the form ffffffff.DNG
All the WAVs are "_AUDIO.WAV"
This can be confusing when dealing with multiple MLV files.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 14, 2014, 07:40:00 PM
However, when I unmount the mlv and then mount it again, the xmp is gone. Maybe this is a good thing, since it implies AFX is not altering the mlv file.
If I want to retain the AFX/ACR import settings more permanently, I can save an xmp manually in the ACR settings folder (or maybe just drag the xmp from the virtual folder into a real folder).
Yup, while the virtual filesystem allows the creation of real files (e.g., XMP sidecars), I haven't yet implemented any preservation of those files after unmounting.  Dragging/copying the XMP files out of the virtual filesystem should work fine.

What controls the naming of the (virtual) DNGs and WAVs?
The naming scheme, which is the same as on the FUSE versions, can certainly be changed.  I presume you want the filename of the MLV video prepended to the individual DNG files and WAV file?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 14, 2014, 07:45:19 PM
Is this xmp creation present on mac as well working with ae or is it a windows thing?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 14, 2014, 07:45:57 PM
"I presume you want the filename of the MLV video prepended to the individual DNG files and WAV file?"

That would be perfect (for me at least).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 14, 2014, 07:56:10 PM
Is this xmp creation present on mac as well working with ae or is it a windows thing?
For now, just on the Windows side.  Because of the differences between FUSE and PFM and the rapid nature of MLVFS development, I imagine this won't be the last time you'll see features show up on one version before they show up on the other versions.

"I presume you want the filename of the MLV video prepended to the individual DNG files and WAV file?"

That would be perfect (for me at least).
Does anyone else have any thoughts?  I don't want to make this change if there turns out to be a compelling reason to keep it as it currently is (or to change it to some third option).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 14, 2014, 08:06:17 PM
Cool, would it be able to store a new folder as well containing copies of dng;s like storing an xmp? I,m thinking Kitchehof lightroom app workflow in first hand. This would entirely skip the first converting step when working dual iso movie files.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on September 14, 2014, 08:55:35 PM
Quote from: ayshih
Does anyone else have any thoughts?  I don't want to make this change if there turns out to be a compelling reason to keep it as it currently is (or to change it to some third option).
I prefer this kind of file naming, source MLV: M18-1424.MLV, DNG: M18-1424-00000.DNG ... M18-1424-00250.DNG
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 14, 2014, 09:07:45 PM
99,999 frames at 23.976 fps would give about 69.5 minutes of video, which should be more than enough for one take, so I agree with surami's suggestion.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 15, 2014, 01:01:14 AM
I chose "_AUDIO.WAV" so that the audio would appear first. I'm fine with adding the MLV filename as a prefix, but we probably need to decide something and then not change it anymore, as it would mess up any projects already created that point to specific filenames. The other option is making it configurable with various options from MLV metadata (such as cam name/serial, text/take tags, date/time, etc). Some kind of string format could be a command line parameter. Ex: "$CAMERA_$TAKE_$TIME_" => 7D_003_07-30_00000000.DNG. The advantage to making it customizable is that we can avoid breaking old projects that depend on specific paths. Of course the disadvantage is this is significantly more complex.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 15, 2014, 01:17:50 AM
ive chosen M26-1335_000000.dng in my windows MLV client.
thought thats a good choice. frame count should be enough and it contains the MLV name as prefix.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 15, 2014, 07:05:13 AM
I've changed the naming scheme in the FUSE and Windows/PFM versions to match g3gg0's scheme.  (dmilligan: please update the OS X binaries)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 15, 2014, 07:40:20 AM
So right now, the audio file is named like "M13-1837_audio.wav"
Could it just be "M13-1837.wav"?
It would work better with the After Effects script I am using ("Smart Import 2.jsx").
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 15, 2014, 08:10:33 AM
scripts can be simply changed ;)
main question is, if it makes sense to remove _audio or to keep it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 15, 2014, 08:25:48 AM
I agree. Is there a reason to have the "_audio" in the name?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 15, 2014, 01:32:06 PM
I don't really see a reason to have "_audio" in the name anymore using the prefixes. Before, when not using prefixes, it had to have some kind of name so I just named it what it was, "audio", and the "_" makes it show up first. Now, I see no reason not to just use "M12-3456.WAV"
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 15, 2014, 03:47:17 PM
Yeah, there really doesn't appear to be any benefit for having "_audio" in the filename when it already has the extension ".wav", so the cleaner approach would be to remove the "_audio".
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 15, 2014, 03:52:08 PM
will do it the same as MLVFS does. so you both decide.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 15, 2014, 05:39:40 PM
With no objections to removing "_audio", I have made that change.  New Windows/PFM DLL uploaded.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 15, 2014, 10:40:49 PM
mac binary updated
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 16, 2014, 06:04:36 AM
Quick note/tip for pismo method:
If having difficulty importing the (virtual) DNG sequences into your favorite software, try:
1. Making each virtual MLV folder writable.
2. Deleting the IDX files that get created alongside the folders when you mount the MLV.

I had to do both of the above when using the smart import scripts in After Effects.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 18, 2014, 12:02:58 AM
@g3gg0,

Well, I'm sort of stuck, this math seems to be over my head. I just sort of blindly implemented your C# code, but mathematically, I have no idea what's going on or what's supposed to be happening. I'm pretty sure I've computed xyzKelvinWb matrix correctly, but I don't really know where to go from there to get AsShotNeutral. If you don't mind, take a look: https://bitbucket.org/dmilligan/mlvfs/branch/white_balance
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on September 18, 2014, 12:14:12 AM
If it helps, I've extracted the ufraw kelvin routines in a separate C file:
https://bitbucket.org/hudson/magic-lantern/src/cr2hdr-20bit/modules/dual_iso/kelvin.c
https://bitbucket.org/hudson/magic-lantern/src/cr2hdr-20bit/modules/dual_iso/kelvin.h

Usage examples in cr2hdr.c (look for AsShotNeutral and kelvin): https://bitbucket.org/hudson/magic-lantern/src/cr2hdr-20bit/modules/dual_iso/cr2hdr.c
For initialization, call adobe_coeff("Canon", "EOS ...") - I'm doing this in dcraw-bridge.c.

Also, a good reference is this: http://users.soe.ucsc.edu/~rcsumner/rawguide/RAWguide.pdf

If you'll still be stuck, I can sit down and figure it out.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 18, 2014, 01:52:45 AM
a very informative site is here: http://www.brucelindbloom.com/index.html?Eqn_XYZ_to_T.html
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 18, 2014, 09:19:10 AM
a good XYZ/xyY/wavelength/kelvin converter is there: https://www.sylvania.com/en-us/tools-and-resources/Pages/led-color-calculator.aspx
it's from osram and meant for LED gamut calculation
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 18, 2014, 10:08:19 AM
if you want to debug your code, here the results of my code:

Code: [Select]
            /* get the XYZ --> cone reference whites for requested temperatures */
            Matrix temp = KelvinToXYZ(ColorTemperature);
            /* verify: do all our matrices that came from dcraw really convert raw to D65? */
            Matrix tempD65 = KelvinToXYZ(6500);

            Matrix dst = coneDomain * temp;
            Matrix src = coneDomain * tempD65;

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs1.postimg.org%2F3y18b7b0f%2Fmatrix.png&hash=ca7dc2f74f4b33350515511b82f55db5)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 18, 2014, 06:25:44 PM
Thanks for the link a1ex, that's the most helpful thing I've read. I think I've started to make some progress in my understanding. Please correct me where I'm wrong:

Compute XYZKelvinWB matrix using the "chromatic adaptation" techniques in the Bradford cone domain (I'm pretty sure now, that I've got this part right). Then use the XYZKelvinWB matrix to transform "white" (1,1,1) to some different color temperature (I assume this is what the XYZKelvinWB matrix does, but I'm not totally sure). Then convert this new "white" back to the camera raw color space using the inverse of the "ColorMatrix1" (which if I understand correctly this matrix is a Cam->XYZ). Now we have a neutral color in camera raw coordinates right? So then the multipliers (aka AsShotNeutral) are simply {R/G, 1, B/G}. Mathematically this would be:

CamNeutral = [ColorMatrix1]-1 * ([XYZKelvinWB] * [1,1,1])
AsShotNeutral = [ CamNeutral R / CamNeutral G, 1, CamNeutral B / CamNeutral G ]

Is this correct, or am I totally off?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 19, 2014, 10:44:35 PM
well none of that worked, but I just refactored kelvin.c from c2hdr20bit to work with mlvfs (had to get rid of some global variables) and its working, yay!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 20, 2014, 09:18:09 AM
Nice work.
Hi David. Love this workflow. Any thoughts on including raw format and maybe also some "write" possibilities(direct use in after effects)? Noticed some of this going on in the windows workflow but I have no idea how complicated this will be for mac mlvfs.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 20, 2014, 05:00:05 PM
For Windows users who can't directly edit the cdngs, I posted a workflow for making proxies or intermediates with After Effects using some batch files to simplify things:
http://www.magiclantern.fm/forum/index.php?topic=13354.0 (http://www.magiclantern.fm/forum/index.php?topic=13354.0)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: senzazn12 on September 20, 2014, 06:07:37 PM
When I try to register the MLVFS.DLL in command prompt I get an error saying this. Anybody know what I'm doing wrong I'm doing wrong?

It says ERROR: Unable to register formater "mlvfs.dll"
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 20, 2014, 06:13:30 PM
Are you sure you are typing the command in the correct directory (i.e., where you downloaded mlvfs.dll to)?  You can type "dir mlvfs.dll" to make sure that the DLL is present in current directory.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: senzazn12 on September 20, 2014, 06:21:45 PM
It is in my downloads folder.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 20, 2014, 06:28:37 PM
What I did was:
1. Renamed the file to add the ".dll" on the end.
2. Copied the file to my Windows folder.
3. Left-clicked somewhere in the Windows folder.
4. Held down the shift key while right-clicking, and chose "Open Command Window Here."
5. Typed the command per ayshih.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: senzazn12 on September 20, 2014, 06:32:39 PM
Thanks ayshih and Frank7D

That worked perfect for me! I'm not too literate with CMD so thanks for helping me out. I'll be trying your workflow next. :D
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 20, 2014, 11:28:59 PM
Any thoughts on including raw format and maybe also some "write" possibilities(direct use in after effects)? Noticed some of this going on in the windows workflow but I have no idea how complicated this will be for mac mlvfs.
Well, I've mostly got a read/write filesystem working via a loopback to the real filesystem (files to show up in the xxx.MLV directory are stored in a xxx.MLD directory in the real filesystem). However, After Effects still wouldn't open the DNG sequence because it wanted to write to the DNGs themselves. I was hoping that AE would fall back to using an XMP sidecar if the DNGs were readonly, but this didn't happen and it appears there's no way force it to (unless you can find one). There's certainly no way to make the DNGs themselves writeable, so it looks like an AE workflow is not going to be possible. IMO this is very poor design on Adobe's part. Maybe Adobe will bring native CDNG support (like in Premiere) to AE soon.

It might still be possible to use Lr or Photoshop.

Though, personally, I much prefer workflows that don't involve ACR. There seem to be a lot of folks that swear by ACR, but I don't buy it, it's just not good for video. There are much better video specific tools out there.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 20, 2014, 11:52:25 PM
I was hoping that AE would fall back to using an XMP sidecar if the DNGs were readonly, but this didn't happen and it appears there's no way force it to (unless you can find one). There's certainly no way to make the DNGs themselves writeable, so it looks like an AE workflow is not going to be possible.
I don't have or use AE, but Frank7D's workflow (http://www.magiclantern.fm/forum/index.php?topic=13354.0) uses the Windows/PFM version of MLVFS, and those virtual DNGs aren't writable either.  He turns off the read-only attribute on the virtual DNGs, and then AE can open the sequence.  If AE actually tries to write to the virtual DNGs, that implementation will return a type of error that is actually soft-ish (i.e., AE might not actually realize that the writes failed), which may also be helping.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 20, 2014, 11:55:17 PM
interesting, though it sounds kind of hackish to make the DNGs writable, I would prefer that the DNGs attributes remain readonly, if they are in fact going to be readonly, but I'll try it
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 21, 2014, 12:13:43 AM
okay well, that worked, so it's now working with AE, binaries updated
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on September 21, 2014, 12:39:38 AM
okay, will update the code to also use .MLD directories.
just to make our tools behave the same ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 21, 2014, 08:59:31 AM
Hi!
Tried the latest update and the workflow crashes. I all seems going good at first sight but when entering the mlv folders and trying opening the dng:s either makes the folders dissapear or the files won,t open in both photoshop acr and AE acr. This is how it looks. I,m on macbook pro os X mavericks adobe cs 6.

[img=http://s22.postimg.org/7u7tqsxf1/Screen_Shot_2014_09_21_at_08_51_11.png] (http://postimg.org/image/7u7tqsxf1/)

[img=http://s22.postimg.org/52ok06ywd/Screen_Shot_2014_09_21_at_08_54_10.png] (http://postimg.org/image/52ok06ywd/)

[img=http://s22.postimg.org/6gg6vhy5p/Screen_Shot_2014_09_21_at_08_54_53.png] (http://postimg.org/image/6gg6vhy5p/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on September 21, 2014, 11:06:38 AM
I am getting the same thing. The last version was working.

The latest version appears to mount the files and as soon as you go to browse - the mounted folder disappears? The folders seem to have disappeared into the ether? However the folders are visible from the Terminal only (they are not visible in Finder).

What's happening?

I'm running Mavericks 10.9.5 with latest OSXFUSE (v2.71)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 21, 2014, 03:29:56 PM
Something's wrong with the latest OS X MLVFS, now that there is a write implementation, and I haven't figured it out yet.  It may have to do with the "._*" files that OS X wants to create for all of the files.  Sorry guys: we'll get this fixed as soon as possible.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 21, 2014, 07:04:53 PM
I can't figure out what's going on either. It was working fine, I even got it to work in AE, but now nothing works, so I've reverted the dmg to the previous working version until we figure it out.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 21, 2014, 09:21:03 PM
I believe I've fixed the problem.  dmilligan, please confirm and update.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 22, 2014, 12:49:49 AM
No, it's still really screwy, I get file not found errors. I think there are more FUSE API functions that need to be implemented for it to work (like statfs and getattr). The strange thing is I can't figure out how I ever got it to work in the first place.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 22, 2014, 12:59:46 AM
Hmm, that's odd.  I'm no longer getting errors on my OS X side, so I don't know if I can help you track down the problems.  If you haven't already, I recommend nuking everything (object files, .MLD directories, any files starting with a period) to make sure everything builds and runs as cleanly as possible.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 22, 2014, 03:44:37 AM
yeah I already tried that, also rebooted and installed the latest xcode updates, everything was the same. I have sort of narrowed it down though, it seems to be an issue with flush, when I get rid of it everything works. If you don't think there's anything wrong with getting rid of flush I'll push the commit.

I have updated the mac binary so please try it everyone.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 22, 2014, 07:12:41 AM
Holy mother of all crap. It is working! What is also working is direct conversion of dual iso moviefiles  :D :D :D. The mld directorys takes the files perfectly. I ran the cr2hdr binary. What a morning man. Awesomeness.
Biggest thanks David and Ayshih.

I you drag and drop several folders at once in after effects, see to it it doesn,t contain any wav file cause it won,t open then. Open the folders one by one works as usual. This is not related to the MLVFS app.
[img=http://s7.postimg.org/d6sbcun9j/Screen_Shot_2014_09_22_at_06_47_49.png] (http://postimg.org/image/d6sbcun9j/)

Dual iso creation!
[img=http://s7.postimg.org/8m64xx3k7/Screen_Shot_2014_09_22_at_07_00_42.png] (http://postimg.org/image/8m64xx3k7/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on September 22, 2014, 01:11:54 PM
A quick question: I have the MLVFS version downloaded from 10 days ago. Do I have to uninstall this from the Mac and reinstall the newer version on my laptop. Any safeguards or precautions to follow while do so?

Are the steps the same as they were before when Danne / Dimiligan guided me through the process.

I appears that the 10 day old version is working fine on AE 2014 for me. However, cannot say the same for the g3gg0's version on my Windows 8 machine. This maybe be because I am doing something horribly wrong.

Will try this files on Photoshop, Premier Pro and Speedgrade to ensure all is working fine and as intended.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 22, 2014, 01:20:03 PM
I,m on mac and I simply install the new version and services asks if I want to replace the old one and then it is good to go. Cannot help on windows.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on September 22, 2014, 01:35:41 PM
thanks but i am getting the message that mlvfs cannot open since it is from an unidentified source. sorry i am new to the osx environment .. would have never asked such a stupid question if i was no the windows environment  :(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 22, 2014, 01:41:41 PM
So if you,re on a mac and you get the unidentified developer notification you have to change settings in your,re OS. Try this or google for more info on the subject. Or are you on windows?
http://support.apple.com/kb/PH14369

http://stackoverflow.com/questions/19551298/app-cant-be-opened-because-it-is-from-an-unidentified-developer

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 22, 2014, 01:42:38 PM
I appears that the 10 day old version is working fine on AE 2014 for me. However, cannot say the same for the g3gg0's version on my Windows 8 machine. This maybe be because I am doing something horribly wrong.
For Windows, you can try the MLVFS version that uses Pismo File Mount Audit Package (see Frank7D's post on his AE workflow (http://www.magiclantern.fm/forum/index.php?topic=13354.0).  The MLVFS formatter can be downloaded here: https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll

@dmilligan: can you add a download link for the DLL to the first post?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 22, 2014, 02:01:34 PM
done, also added the stackoverflow link from Danne about mac security settings
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DavidSh on September 22, 2014, 04:20:03 PM
beautiful.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 22, 2014, 08:01:26 PM
Dual iso movie conversion the fast way. Thanks for all efforts. This is massive.

*Issues with dng exiftool samelevel command in lightroom. Still gets converted though but dng tags doesn,t get written. See if Kitchehof finds some useful info on this.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 26, 2014, 12:52:33 AM
New version of the Windows/PFM MLVFS DLL (https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll): like the other implementations, XMP sidecar files are now preserved in an associated .MLD directory.  Other types of files can be created, but they will be discarded upon unmount.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on September 26, 2014, 02:37:38 AM
Tried activating the Pismo, etc. but gave up.
Find the g3gg0 solution better for now at least. There is so much happening on this forum and so fast difficult to keep up  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 27, 2014, 12:52:55 AM
ayshih, the "smart import 2" script didn't seem to have a problem with the latest mlvfs.dll, but I ran into another issue:
1. Mounted the MLVs
2. Created an AE project
3. Imported the sequence(s)
4. Closed and saved the project
5. Unmounted and then mounted the MLVs again
6. Opened the AE project (so far, so good)
7. Did a Ctrl+H on a sequence (replaces footage; this is a useful way to adjust the import settings after the fact)
8. ACR opened again; when I said "OK" to open in AE, I got this message:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi62.tinypic.com%2F1dypgo.jpg&hash=de82c03efa6a6a633af80ccdb14e9922)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 27, 2014, 04:29:21 AM
I've implemented vertical banding correction code from raw2dng. Please test if you have a 5D3 (I don't).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 27, 2014, 02:55:52 PM
Hi David!
I,m yet to see vertical banding on my 5d mark 3. I think I encountered one occasion with banding that was fixed with mlrawviewer.
Tried to reproduce a mlv file with banding cause the one I got is a raw file.
Just for you to clarify. Do you intend to include raw 1.0 to this workflow or is the philosophy here to stick strictly to mlv conversion? The reason I use raw is purely 3x mode and 2240x1260 24fps hacked preview which gives my several minutes of filming. Mlv is not really getting that speed.

Ok, so my test couldn,t be done cause of errors. It seems the files are getting different information in them with the latest binary. SOme shows with thumbnails, some don,t. I managed to get one dng out on my desktop, only to open up a completely black picture in acr. The mount randomly unmounts, very unstable.
OS X Mavericks macbook pro 15 inch

Here,s the dng sample
https://drive.google.com/file/d/0B4tCJMlOYfirU254U0JxR0l2aU0/edit?usp=sharing

A few error codes when I tried either open a file or drag/drop onto desktop.

I upload some pictures here:
[img=http://s28.postimg.org/l8whed015/Screen_Shot_2014_09_27_at_14_42_23.png] (http://postimg.org/image/l8whed015/)

translation: Photoshop could not follow you request...
[img=http://s28.postimg.org/4b2269g15/Screen_Shot_2014_09_27_at_14_42_55.png] (http://postimg.org/image/4b2269g15/)

[img=http://s28.postimg.org/jii1qm7vt/Screen_Shot_2014_09_27_at_14_43_53.png] (http://postimg.org/image/jii1qm7vt/)

[img=http://s28.postimg.org/p9884cfvt/Screen_Shot_2014_09_27_at_14_44_06.png] (http://postimg.org/image/p9884cfvt/)

[img=http://s28.postimg.org/534q5gk89/Screen_Shot_2014_09_27_at_14_44_15.png] (http://postimg.org/image/534q5gk89/) 

On a sidenote. Could view mlvfiles fine in mlrawviewer with the former mlvfs build but exports to mov don, t work. Is this expected behavior?

Thanks for good work!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on September 27, 2014, 04:06:56 PM
hello,
i have tried the latest mlvfs with my mac 10.9.5, i copied the mlv dir on the desktop also the empty dir. then i mounted the dir with the mlv file, it mounted the files to the empty one, then after 10 seconds it suddenly disappears,..
tried different mlv files, its the same with all other files.
tested the same procedure on an external drive, also not working.

thx. sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 27, 2014, 05:19:56 PM
I think this is a multithreading issue. You can use the -s option to force single threaded mode. I've updated a new dmg that uses the -s option, so try that, there may still be other stability issues.

We're going to need to do some thread synchronization for this to work properly in multithreaded mode, not sure of a good multi-platform, generic way to do this since semaphores and locks are usually OS specific, you got any suggestions @ayshih?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 27, 2014, 05:36:18 PM
@dmilligan: I've only encountered a few problems with cross-platform support for POSIX threads, and even those problems were localized to thread attributes and handling (joining, canceling, etc.).  As long as we're not talking about instantiating threads ourselves, I don't believe we'd encounter any weirdness at all with just mutexes.

@Frank7D: I think I may have tracked down the problem with the Window/PFM version.  Stay tuned.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on September 27, 2014, 05:50:44 PM
And on my hackintosh ... Mavericks 10.9.5 ... I am not experiencing any problems since the 22/09/2014 update of MLVFS.

(@ Danne) It's interesting you are getting those errors? ::) That is what I was getting pre 22/09/2014 (the unmounting and all error messages if you tried to import to After Effects.)

I must say now I am very happy with it as part of the workflow. I can mount, then open, adjust and export with relative ease and stability.

Oh. One thing I haven't really had a chance to try out yet is how it works with Davinci Resolve. I guess that will be fine though?

Thank you so much for all who have made this possible. It's genius! :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 27, 2014, 06:08:25 PM
New test.

Didn,t notice a different in banding with former versions. Probably because my pics don,t have banding?
With the latest mlvfs I think the vertical banding part is causing some irregular behaviour. It goes like this.
Open a dng within the mount folder and a thin pixelborder appears on top. When closing down and opening again, the border issue is gone.
When dragging a dng to desktop a big pixelated border is present on top. See included pics. These are only a sample from the latest mlvfs. These are crops and overexposed.

Straight from mlvfs mount
[img=http://s2.postimg.org/vg8m71bfp/Screen_Shot_2014_09_27_at_18_00_36.png] (http://postimg.org/image/vg8m71bfp/)
after closing down and opening again
[img=http://s2.postimg.org/7qj6icd2d/Screen_Shot_2014_09_27_at_18_00_52.png] (http://postimg.org/image/7qj6icd2d/)
dragged to desktop
[img=http://s2.postimg.org/vishtvf39/Screen_Shot_2014_09_27_at_18_03_24.png] (http://postimg.org/image/vishtvf39/)

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 27, 2014, 06:43:16 PM
@Frank7D: I've uploaded a new version that fixed any crashing on my side, but I'm not using AE.  Can you try it with your workflow?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on September 27, 2014, 07:05:19 PM
It works! Another step eliminated; thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 27, 2014, 07:20:10 PM
@Danne, I think I fixed your issues with the bad data at the top of the file, try the latest.

Didn,t notice a different in banding with former versions. Probably because my pics don,t have banding?
Probably. The correction should be identical to the correction provided by raw2dng/mlv_dump, and the algorithm does not do any correction unless the coefficients are above a specific threshold.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 27, 2014, 08:19:15 PM
Confirmed working! Fast progress. Amazing.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 27, 2014, 10:35:29 PM
okay, I used pthreads_mutex, seems to work pretty well, so the stripe correction now supports multithreading mode. I made a LOCK() and UNLOCK() macro, so just redefine these if you need to use something else on a different platform.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 29, 2014, 12:05:16 AM
Some big improvements! Chroma smoothing is now implemented (I'm not getting real-time playback with it, but it is reasonably fast ~10 fps, perhaps could be realtime on higher end systems), and this has also paved the way for easily implementing dual ISO processing, by implementing a way to process an entire frame all at once. And this has provided a fix for the issue with exFAT on OS X (b/c we avoid constantly opening and closing the MLV file, which was slow on exFAT for some unknown reason).

I haven't posted a binary yet, b/c I'm not quite sure how to provide a way to easily enable/disable the chroma smoothing options. The possible options are to provide two different automator workflows (one with chroma smooth and one without), or to prompt the user to select options (which is some extra mouse clicks). Any preference, or other ideas?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on September 29, 2014, 12:09:44 AM
Whow Great!
What about a choosing option when mounting??
Greets sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 29, 2014, 12:18:48 AM
Did I hear dual iso processing  :o : :D :D

Could the setting be stored and than changed back again as desired than I would go for the click option but otherwise different  automator workflows is cool with me.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on September 29, 2014, 01:17:13 AM
The possible options are to provide two different automator workflows (one with chroma smooth and one without), or to prompt the user to select options (which is some extra mouse clicks). Any preference, or other ideas?
I'm limited on the Windows/PFM side because I can't actually supply any parameters for the mounting.  My likely solution is to have a configuration text file (e.g., "config.ini") in the MLV's directory that can be parsed for such parameters on the fly.

Note that with the "config.ini" saved in the associated .MLD directory, this approach would make possible per-MLV settings (e.g., one file needs more chroma smoothing than others) that are persistent.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 29, 2014, 04:15:28 AM
Mac binaries updated, I just made multiple workflows for now, install the main one and any of the cs ones you want to use. If there are more options in the future, we may have to come up with a better way to select them. (some kind of embedded webserver might be cool)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 29, 2014, 07:37:34 AM
Cool!
Tested on my macbook pro 2.6 ghz 16gb ram 15 inch. The chroma smoothing is giving a not a super smooth playback(tested in mlrawviewer)but I,m not complaining :).
Just to make sure. There is still the exiftool temp issue as before converting dual iso files? Converted as before, still the poop temp file :)
Thanks for nice work David
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 02, 2014, 03:15:12 AM
I've implemented a very experimental dual ISO conversion (I really prefer to think of it more as a preview). It's nowhere near the quality of cr2hdr and probably not all that useable for real work yet, but it's realtime :)

It will try to convert dual ISO if the path name contains "DUAL", so put your dual ISO mlv files in a folder called DUAL or rename them with the word DUAL in them, to try it out.

There are all sorts of issues (like strange color artifacts, banding, bad aliasing, etc.) b/c of all the short cuts I take compared to cr2hdr (and b/c I'm no a1ex when it comes to this computational image processing stuff, I need to read the dual ISO pdf about 10 more times), but I feel like it might at least be somewhat useful for previewing and doing quick edits.

Anyway, I'm putting it out there b/c I don't really have much good test footage, so please do some tests and post some comparisons, and let me know what you think.

Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: senzazn12 on October 02, 2014, 05:25:20 AM

I've implemented a very experimental dual ISO conversion (I really prefer to think of it more as a preview). It's nowhere near the quality of cr2hdr and probably not all that useable for real work yet, but it's realtime...

Where can we try this out? Thanks for providing this to us.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 02, 2014, 07:39:57 AM
Hi David. A realtime viewer on dual iso moviefiles. Not bad at all! Mlvfs should be in camera as well ;).
I did a fast comparison between cr2hdr and mlvfs. Don,t know if it,s relevant. Just two original dng;s and then applied the same settings lifting shadows and such.
I provide the dng;s in two links. Also linking to the original mlv file (about 350mb) if anyone wants to play around.
Off to work then.

screencaptures:
cr2hdr
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2Fdcwg4gc3x%2F0_1_cr2hdr.png&hash=c9b2227cbcaf694a59674789563e1a98) (http://postimg.org/image/dcwg4gc3x/)
mlvfs
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2F7tyi0z4gd%2F0_1_mlvfs.png&hash=6eb1edbf6e54c1ea7994f3a144868cd8) (http://postimg.org/image/7tyi0z4gd/)
cr2hdr
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2F60qzywtvx%2F0_2_cr2hdr.png&hash=883b03aed74207599c754e75f5345461) (http://postimg.org/image/60qzywtvx/)
mlvfs
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2Flowsjfyvh%2F0_2_mlvfs.png&hash=9415b964aa76ab3b466a9fe1a67c048c) (http://postimg.org/image/lowsjfyvh/)

dng files:
https://drive.google.com/file/d/0B4tCJMlOYfirUnBtOHNDdGllTjg/edit?usp=sharing
https://drive.google.com/file/d/0B4tCJMlOYfirWFJaenVhTG50eWc/edit?usp=sharing

Mlv original dual iso file:
https://drive.google.com/file/d/0B4tCJMlOYfirMDg0VV84VlplV28/edit?usp=sharing
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 02, 2014, 12:59:22 PM
Where can we try this out?
Download links are in the first post. Mac version has been updated, ayshih will need to update the Windows version.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: escho on October 02, 2014, 08:46:17 PM
I tried to compile, but I run into an undefinied reference:

Code: [Select]
edgar@linux-indh:~/mlvfs/mlvfs> make
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 dng.c -o dng.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 index.c -o index.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 wav.c -o wav.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 kelvin.c -o kelvin.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 stripes.c -o stripes.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 cs.c -o cs.o
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 main.c dng.o index.o wav.o kelvin.o stripes.o cs.o -pthread -lfuse -lm -o mlvfs
/tmp/ccayy0ec.o: In function `mlvfs_read':
main.c:(.text+0x1b8b): undefined reference to `hdr_convert_data'
collect2: error: ld returned 1 exit status
make: *** [mlvfs] Fehler 1

What did I wrong?

OpenSuse13.1 64bit
Newest sources

Edgar
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on October 02, 2014, 09:11:00 PM
@escho: The Makefile had not been updated for compilation on Linux.  Try again now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: escho on October 02, 2014, 09:14:25 PM
OK, compiles now.
ThankYou

Edgar
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on October 03, 2014, 03:30:10 AM
I've implemented a very experimental dual ISO conversion (I really prefer to think of it more as a preview).
ayshih will need to update the Windows version.
The Windows/PFM version of MLVFS now has the same (experimental) dual ISO support.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 04, 2014, 04:10:45 AM
some stability improvements and fixed the exiftool issue with cr2hdr --same-levels, Mac binary updated
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 04, 2014, 07:14:05 AM
Quote
some stability improvements and fixed the exiftool issue with cr2hdr --same-levels, Mac binary updated

Wow. Just like that :). Confirmed working. Tested on mac binary cr2hdr20bit. This would be a killer now with kitchehof lightroom plugin.
Can I ask you of a terminal command. Lets say I want to convert a bunch of dual iso folder and use some drag and drop command to terminal instead of selecting all files within folders and dragging those to terminal window. I tried a little with recursive command but I really don,t get it.

For now I 1) drag cr2hdr20bit to terminal 2) enter the folder, selects all pictures, drag those to terminal 3) enter.
What I like is able to select all folders and drag those to terminal.
Thanks David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 04, 2014, 02:48:45 PM
You can use 'find' and 'xargs' to recurse into subdirectories:

find /path/to/parent/directory -name '*.dng' | xargs /path/to/cr2hdr --any-cr2hdr-options-here

This will convert all dngs anywhere inside /path/to/parent/directory. Is that what you're looking for?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 04, 2014, 03:15:51 PM
Thanks a lot, gonna check into this later tonight.
Dual iso conversion is a breeze now since you fixed exiftool. Great work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 05, 2014, 01:13:12 AM
Awesome!! With your script command I,m only one click away now. Created a bashscript to a folder I will use on my desktop. Thank you a lot. Most grateful.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 05, 2014, 04:13:00 PM
A little monkey wrenching on a late sunday afternoon :)
Scriptbased batchconversion of dual iso dng files.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 07, 2014, 01:05:26 AM
hello,
i really like the idea and workflow of this tool, but have just realized, that the free version of davinci resolve 11 has no hot cold pixel correction for raw files (as far as i know). so i wonder if  - hot/cold pixel correction will be possilbe in the near future?
i ask because i had a video shoot the other day, where i recognized a bad pixel in the middle of the screen and it is really annoying.

thank you for your time and engagenment. greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 07, 2014, 02:57:31 PM
should be doable, it's on my todo list
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 07, 2014, 06:08:21 PM
@dmilligan sounds great!!

thank you swinxx

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 08, 2014, 11:19:20 PM
bad pixel correction is implemented, please test

it's kinda slow (~2fps on my machine), IDK if there's much way around that, there's a couple more possible optimizations, but I doubt it will ever be realtime
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 08, 2014, 11:52:47 PM
thank you dmilligan, i will test it at the weekend, you are great!
that should help a lot!
so i see a seperate bad pix script, is there chroma smoothing and/or dual iso implemented too?

btw.. i was download number 1 :))

greets. sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 09, 2014, 02:29:36 AM
Yes, chroma smoothing is implemented. There is also a dual ISO "preview", quality is not very good. It will run if you have "DUAL" in the path name.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 10, 2014, 03:11:32 AM
Well, there were getting to be too many options, and I wanted there to be a way to turn them off and on without restarting mlvfs. So I have embedded a little webserver (mongoose (https://github.com/cesanta/mongoose)) in mlvfs for configuring options at runtime (this also eliminates the need for all those different automator workflows). The new mac workflow will automatically popup the configuration page in the web browser when you start it. You can always get to it at http://localhost:8000 when mlvfs is running. This will be a great help in the future as options start to proliferate, it also may be useful for displaying status/error information.

@ayshih, mongoose should work on windows, but this may not be as useful since you have to multiple instances (and the webservers would conflict), so you may want to simply disable this with macros
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 10, 2014, 09:26:44 AM
Very clever. Working nicely storing settings.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 10, 2014, 11:37:35 AM
Sounds promising. Thank you for your hard work. Im looking forward to try the new version at the weekend;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 10, 2014, 10:00:09 PM
Hi dmilligan!

i have tried the new badpixel option, but here in my specific shot my bad pixel is not gone.
so i see no difference in on and off position.
 
please take a look here:
https://copy.com/LjWpLtXeqIrO0AtK

should i contact a1ex, or is my problem not a bad pixel?

thank you very much..

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 10, 2014, 11:00:11 PM
That pixel doesn't appear to be bad enough to trigger correction based on the current algorithm, because I've assumed a value for dark noise rather than calculating it (for speed), and this pixel was just barely below that threshold (so the algorithm basically thinks it's dark noise, not a hot pixel). The threshold was conservatively high, so I'll lower it a little. Thanks for the feedback.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 10, 2014, 11:54:51 PM
thanks dmilligan!

i´m looking forward for the update.

greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 11, 2014, 11:39:40 PM
I added the "aggressive" hot pixel correction, and lowered the dark noise estimate a little, hopefully this will pick up your bad pixels.

Also cleaned up the web GUI a little and added a metadata display table for MLV files.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 12, 2014, 08:22:36 AM
Great. Could there be a downside when using agressive hox pixel correction with other material?
I will try today. Thx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 13, 2014, 07:11:29 PM
@swinxx,
IDK

@all,
Full dual-ISO conversion is now working!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on October 13, 2014, 07:18:25 PM
Wow, cannot wait to test it  :D  Is the first page download link up to date (for windows)?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 13, 2014, 07:20:25 PM
Is the first page download link up to date (for windows)?
No, Windows version has not yet been updated, only the Mac version
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 13, 2014, 08:00:48 PM
@dmilligan

first, let me say thank you very much for your time and knowhow to provide a brilliant solution for converting the mlv files. this is a really great way to speed up the working process.

but for the first time some questions came up after installing it.. perhaps you could explain in a few words.

.) when i select the file and mount the drive (with an empty folder which i have created) can i swap and change options on the fly..? i mean now i see a new web interface (which looks really nice btw :))

so the configuration options are applied at the same time when selecting them? or when i remount them? when i have a dual iso file here, i can not see any difference between "OFF", "PREVIEW" and "FULL"?? i mean, i see a distorted image, please have a look at my posted link. thx

UPDATE: Ok, it confused me cause i tried with dual iso first and there are some issues for me? perhaps because of the resolution aspect ratio?

.) I have tried it with a mlv dual iso file, but had problems (image looks strange)
here is the posted image after conversion:
https://copy.com/WHHtwGsDkm84v8rl

.) is it possible to add a "APPLY" button, that we can be sure the settings are applied?

UPDATE: Ok, after more testing i see, they are applied on the fly :=)

.) althought the web interface shows shutter time 19ms i am quite sure that i shot with 50ms..?
could this be a tag issue? or my fault?

.) in the link from above, there is a directory with 2 tiff images, where i sadly can see, that the agressive hot/bad pixel algo is doing some weird things.. perhaps a1ex can take a quick look at it.


all in all, great tool!! dmilligan you are a goal getter :)
thank you,
greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on October 13, 2014, 09:28:41 PM
Wow, cannot wait to test it  :D  Is the first page download link up to date (for windows)?
I haven't been able to work on the Windows/PFM version lately, so it's fallen a bit behind.  Unfortunately, that's likely to continue to be the case for the next couple of weeks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 13, 2014, 09:43:06 PM
.) althought the web interface shows shutter time 19ms i am quite sure that i shot with 50ms..?
could this be a tag issue? or my fault?
Are you sure you didn't mean 1/50s?
1/50s = 20ms
(it's 19 probably because of either rounding error, or 1/50 only being the approximate shutter speed)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 13, 2014, 09:50:49 PM
ah yes of course..  :D
1/50 for 24fps.. nearly 180 degree :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 13, 2014, 10:04:36 PM
.) I have tried it with a mlv dual iso file, but had problems (image looks strange)
It looks more like what I would expect the preview to look like. Perhaps you had the preview option enabled and loaded the image and it got "cached", so when you switched to "full" it just retrieved the cached preview. Just unmount and try again, select the "full" option before doing anything else.

If that's not it, could you upload the first little bit of the original mlv file?
hint: you can use mlv_dump to trim an mlv file to just a few frames:
Code: [Select]
mlv_dump -o output.mlv -f 5 input.mlv
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 13, 2014, 10:21:40 PM
thank you,
i will have a quick look now, and get back as soon in some minutes..

greets.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 13, 2014, 10:33:05 PM
Yeah man! This is nice. It is a cache thing. Got the dual iso "view" files converted looking fishy but after remounting I could start conversion with dual iso 20bit converter.
Are files hardcoded with 60 000 whitelevel for same level correction? How is that done? Where can I view the code? Curious noob I am.
Compared converting with my cr2hdr-r script but altough my computer was sweating like a pig your conversion was 20 seconds faster with the same laptop and it didn,t make a sound. Go figure  :P.
Wishing for raw support purely for dual iso purposes and until then monkeywrenching i still a good thing ;).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 13, 2014, 10:35:41 PM
hello dmilligan,
 
so i have tried it, and yes my fault, i had to reboot the computer, now it is working, although, when you take a quick look (link posted above) i have some strange artefacts in the converted file (in the window) green and pinkish..
the same file converted with the lr plugin has no artefacts.
thx. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 13, 2014, 11:08:57 PM
Are files hardcoded with 60 000 whitelevel for same level correction? How is that done?
white level is just the one from metadata * 4

Where can I view the code? Curious noob I am.
https://bitbucket.org/dmilligan/mlvfs/src/
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 13, 2014, 11:14:45 PM
Thank you. Heavy reading. A lot of stuff. Thanks for all your work on this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 14, 2014, 01:47:14 PM
Compared converting with my cr2hdr-r script but altough my computer was sweating like a pig your conversion was 20 seconds faster with the same laptop and it didn,t make a sound. Go figure  :P.
It's probably the fact that I'm not using the AMaZE based interpolation. I just wanted to get cr2hdr working internally as simple and stripped down as possible, then add stuff back in.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 14, 2014, 02:18:18 PM
Could be the case. It looked good though but have to test some more clips. Wasn, t amaze "fixing" jagged lines a little better?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 15, 2014, 11:12:08 PM
update: some performance improvements

(mostly if you have multiple cores and are using the processing heavy options, also requires the software accessing the files to try to load multiple files at once, in SpeedGrade there is an option you can set for this, the default is 2 frames at once, a Finder copy only does one file at a time, so to see improvement do multiple copies at the same time)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 16, 2014, 04:22:57 PM
whow. great thank you . will try tonight.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 16, 2014, 09:21:23 PM
Just tested out the latest update of MLVFS -- it is in fact quite impressive especially with the new GUI layout as well as the DUAL-ISO 20-bit option...

I tried to leave everything alone as much as possible beside 'Bad Pixel Fix' and 'Dual-ISO 20-bit' and the DNG's are coming out quite weird...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm4.staticflickr.com%2F3932%2F14930543134_47896f3511_b.jpg&hash=c9dabaf3bfe4d1bd10146a0913da5932) (https://flic.kr/p/oKmWiu)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm6.staticflickr.com%2F5603%2F15364673089_dd9602bc90_b.jpg&hash=e935bcb304f6912fdf4151da608ff317) (https://flic.kr/p/ppHY1H)

Definitely better than the previous version but it's not quite the same quality as if you were just to export as unconverted DNG's and then use a stand alone app (cr2hdr-r) through with Automator OR the 20-bit plug-in for LR5... either one of these gets the job well.

**ALSO when trying export files directly from the MLVFS mount folder -- it freezes up the Photoshop app and have to force to close BUT it seems to work just fine with LR5... just have to wait for it to finish load which takes forever! is there a reason for this?

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm6.staticflickr.com%2F5601%2F14930627194_9c50f343c1_b.jpg&hash=8a64e521281d761efbde62c463c92618) (https://flic.kr/p/oKnnhN)

The only question I have regarding this is whether or not should I apply the same levels through command because I'm not sure if MLVFS does this? (this is why I love the new cr2hdr-r workflow the @Danne created as well @dmilligan's merged into this 'Service' similar to MLVFS.

Unless I am missing something regarding the newly updated MLVFS...

THANKS ALL!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 16, 2014, 10:02:33 PM
@dmilligan

hi,
when i mount an folder it works but safari gives me this error:
https://www.dropbox.com/s/qiylsho8d5gor2u/Screenshot%202014-10-16%2021.59.12.png?dl=0

greets.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 16, 2014, 10:27:01 PM
@DeafEyeJedi, those look like focus pixels, you need to turn on chroma smoothing to get rid of them. Right now you might get some weird artifacts with chroma smoothing, I need to incorporate it into the correct sequence within cr2hdr stuff. There are also several cr2hdr options that I have not implemented yet. These options, when implemented, should improve image quality greatly, and match the result of the standard cr2hdr.

It's slow because Lr is trying to load the DNG, which means MLVFS has to convert it first. Conversion can take several seconds per frame. If you're going to use LR, I would recommend copying the DNGs out of the MLVFS mount into the real filesystem (permanently converting them).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on October 16, 2014, 10:32:43 PM
Note that focus pixels with dual ISO are still unsolved, at least in my implementation (didn't try MLVFS yet, but will definitely do it). So try not using both at the same time - I know you can, from your signature :P
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 17, 2014, 12:08:03 AM
Okay I implemented the chroma smoothing the same way it's in cr2hdr (chroma smooth fullres and halfres before final blending). It helps with the focus pixels, but doesn't remove them completely. If you do the chroma smoothing after cr2hdr is completely finished, the focus pixels are completely gone, but there are strange green blob artifacts.

I haven't updated the binaries yet (I'll wait till some more updates)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 17, 2014, 08:49:45 AM
copy that @dmilligan & @a1ex -- I'm off to bed and will get back on the boat w you guys over the weekend!

Thanks guys!

SJ
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PowerMac84 on October 18, 2014, 09:38:43 AM
Quite often OS X 10.10 during Convert with that AE script the temporary mount point of MLVFS disappears. I usually name a folder like "MLVFS temp folder". This looks like a volume which acts like a folder on another volume. But after some time it just disappears and also the Webserver is not more there. I then recreated it and let AE reconnect the files. After a while it happened again and the AE batch export gave me errors and exported error clips. Any ideas?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PowerMac84 on October 18, 2014, 09:43:33 AM
Well the mount point of that virtual MLVFS volume also appears under the devices of my computer. Which volume should be used for working with the CDNG files? That mount point under devices or the concrete folder (virtual volume) on a other volume?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 18, 2014, 03:19:46 PM
I've only just started testing on 10.10. If the mount point disappears then that means the program crashed.

I've never seen the FUSE mount show up under "devices", but I suppose it doesn't make any difference (it's probably still actually pointing to the same place, the folder where you mounted it to).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 18, 2014, 04:40:12 PM
@swinxx,
I think I fixed the issue with the "%@" error popup (this was an issue with automator in osx 10.10)

@PowerMac84,
I have also some stability updates, please try the latest version and let me know (I also recommend updating FUSE to the latest version, 2.7.1, if you're not already on the latest version). If it's still crashing, let me know all the settings you were using.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 19, 2014, 06:23:19 AM
Just downloaded the latest update few mins ago.. Thanks @dmilligan! Will let you know how it goes..

Question -- is it always reccommended to take the files out from the mount folder and into the file system (desktop, external drives, etc) as oppose to try running apps within the mount folder itself?

Perhaps it would run faster running off the system rather than or does that not matter?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on October 19, 2014, 06:26:22 AM
Copying the files out and not using the mount itself kind of defeats the whole purpose of this program.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 19, 2014, 09:28:30 AM
Hmm. @dmilligan:
I dont think so, how can i work with davinci resolve instead??
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 19, 2014, 09:42:39 AM
You can work with resolve straight from the mount. Worked last time I tried.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 19, 2014, 08:05:27 PM
yes, but when i start with a project, i would like to have all my working files collected in a special folder...
greets.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 19, 2014, 08:14:28 PM
Collect all mlv files in folders on your harddrive and go from there?
Greets
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: delrosestephen on October 20, 2014, 07:39:25 AM
I just wanted to say thank you for this. I recently built a Hackintosh and MLVFS saves me so much time with post. Davinci Resolve works beautifully with this :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 20, 2014, 08:51:55 PM
hello dmilligan!

i updated to yosemite yesterday, and now i can not mount any mlv files anymore..?
have tried it from my desktop, from another hard drive, no chance..
when i try to start the service, my safari web browser pops up but with no web server, there is the message that it is not able to load the page..?

update:
ok, i had a problem with some writing privileges.. ? don´t know why but before the update everything was fine. now i have corrected those things and my issues are gladly gone..

thx.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: yanone on October 21, 2014, 09:11:01 AM
Hi,

I just downloaded and tried MLVFS for the first time. The initial post claims that audio is working, but not for me.

An audio file appears in the file system, but is reported at 0 Bytes size and therefore doesn't play back.
Also I have just tried the MLV module for the first time, coming from RAW, because I need the sound, and have no experience with MLV yet. But I turned the sound module on, and hey, something is there, yet empty.

Any ideas? Thank you!

UPDATE:
Please forgive my post. I have just found that there is another setting in the sound menu that will activate the sound recording. I thought loading the module would already achieve this. Cheers...

Tiny work flow enhancement:
Would be fantastic if MLVFS could remember the mount path, or maybe its parent directory.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 21, 2014, 10:57:32 AM
@dmilligan

do you think that it would be possible to link audio and video - especially in resolve, cause i have here the folders only and must click into all the folders, where else other programs like rawmagic (i know that there is a license problem) links the files that when i import it into resolve audio and video files are linked and i dont see them seperatly :)

thank you.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 21, 2014, 01:12:35 PM
From what I've been able to gather, this requires that the file and folder names conform to some special naming scheme and also the audio file has some special metadata in it. The metadata in the wav file shouldn't be too hard, but the naming scheme presents somewhat of a challenge because of the way mlvfs and the fuse api works. Naming the DNGs something different isn't a problem at all, it's the parent folder that is a problem because mlvfs uses the parent folder's name to find the original mlv in the real filesystem. Instead of a simple translation of the path name, we'll have to have some kind of complicated lookup table.

It's certainly possible, and something I'd like to get working, but it will be a lot of work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on October 21, 2014, 01:55:55 PM
Guys, i am little confused, could you please tell me where is the most recent version for windows - is it the one mlvfs.dll in the first page? I know that windows version is behind the mac but i am going to test it a little bit and would like to work on the latest version available. Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 21, 2014, 02:01:53 PM
is it the one mlvfs.dll in the first page?
yes
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on October 21, 2014, 02:41:11 PM
Ok, great, thanks dmilligan  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 23, 2014, 11:57:34 PM
Update: automatic audio linking in Resolve should now work (you need to select the "DaVinci Resolve" naming option). Even if you don't have audio, if you use the resolve option then your dng folders themselves should show up as "movies" or whatever in the Resolve file browser instead of showing up as a folder containing a dng sequence, which is handy. There is one drawback: you can't use the MLD folders or write anything into the mount (like .xmp files) so AE won't work in this mode (yet).
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 24, 2014, 07:33:19 AM
Dmilligan, you arr the man!! Thank you 1000x
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 24, 2014, 11:04:55 AM
That is cool.
Have a strange thing going on. When I select da vinci resolve in the interface the files vanishes from the folder. However. Tried running the mlv file with audio straight in da vinci resolve without selecting for it in the mlvfs interface and it worked anyway straight out of the box? Audio and picture. Maybe it is set to da vinci as default?
I used Da vinci relove lite 10, OSX maverick mac book pro 15 inch.
NIce work on this. Gotta start using da vinci resolve more often though.

Tried also installing the next latest update and audio wasn,t merged as with latest to rule out audio working before
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 24, 2014, 02:15:03 PM
Quote
There is one drawback: you can't use the MLD folders or write anything into the mount (like .xmp files) so AE won't work in this mode (yet).

Actually, I can use both after effects(storing xmp to MLD folder) and Da Vinci Resolve with audio without selecting da vinci resolve mode in mlvfs. SWEET!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on October 24, 2014, 04:51:14 PM
The audio seems to show up either way, but if you use the naming scheme, the directories themselves just show up as clips (so you don't have to navigate down into them)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 24, 2014, 05:52:41 PM
Ok. Well, then the DaVinci Resolve selection isn,t working on my computer. The mount empties when selected unfortunately.

With davinci option checked
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2Fjmgtyxr6z%2FScreen_Shot_2014_10_24_at_17_46_46.png&hash=6de4cba0540d31306031d24af481e881) (http://postimg.org/image/jmgtyxr6z/)

Entering folder with davinci resolve checked(mount also empty)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2F5hb0x4i5n%2FScreen_Shot_2014_10_24_at_17_46_53.png&hash=8b03e6a2573f7ff7b2516e9d291877df) (http://postimg.org/image/5hb0x4i5n/)


(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2F5wmaq542z%2FScreen_Shot_2014_10_24_at_17_47_12.png&hash=0270bd664b41b659baf73b0ab75980c9) (http://postimg.org/image/5wmaq542z/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 25, 2014, 10:22:50 AM
hello,
i do not understand.. so what´s the problem with mlvfs and davinci now? should i use the option "resolve" in the webserver? or not?
is it also working without it? sorry for asking but i can not test it at the moment.
thx sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 25, 2014, 01:50:11 PM
With resolve option selected folder content is suppose to show the moviefile without entering any folder.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 25, 2014, 02:57:48 PM
Okay, there seems to be some sort of strange issue with Resolve. I noticed you're in the thumbnail view. When I try thumbnail view, Resolve actually completely crashes (it's not MLVFS crashing, in fact MLVFS is still running fine afterwards). In the list view everything is fine (which is what I was using when testing), and the folders show up as clips. Could you verify that in thumbnail view after it disappears for you that MLVFS is still properly running?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 25, 2014, 03:47:56 PM
Hi David. Still no change. MLVFS mount comes up empty upon selecting da vinci resolve mode. Davinci app no change when in list view.

some pics

Selected
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Fbyxxcemon%2FScreen_Shot_2014_10_25_at_15_43_45.png&hash=baa7bafd9288bf3c19b25eabf5dc678c) (http://postimg.org/image/byxxcemon/)

Empty due to davinci selected
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Fyc5nz7nmf%2FScreen_Shot_2014_10_25_at_15_42_33.png&hash=5e1490b9ab0039fe13465f8dfdc47eb6) (http://postimg.org/image/yc5nz7nmf/)

List view emptiness
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Fizlx56ogn%2FScreen_Shot_2014_10_25_at_15_43_12.png&hash=693db1885597891d8b9ed6e0bc21916f) (http://postimg.org/image/izlx56ogn/)

List view emptiness 2
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Feezqq94rb%2FScreen_Shot_2014_10_25_at_15_43_33.png&hash=aca537d922c2b2417d7b4701d282a7b6) (http://postimg.org/image/eezqq94rb/)


Folder reappears when davinci resolve option is not selected
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2F7sd2x2n2v%2FScreen_Shot_2014_10_25_at_15_44_11.png&hash=df42dd915a21d557c3573ca42982c7d9) (http://postimg.org/image/7sd2x2n2v/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 25, 2014, 06:39:31 PM
found and fixed the problem, try again
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 25, 2014, 06:43:42 PM
Will try out tonight. Thx so much for all your hard work
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 25, 2014, 06:56:20 PM
You most certainly did :) That is so great!

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs30.postimg.org%2F4n4dovuhp%2FScreen_Shot_2014_10_25_at_18_53_15.jpg&hash=81df5f3b36d86da602ec48da503594e8) (http://postimg.org/image/4n4dovuhp/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 26, 2014, 07:58:35 AM
Great news regarding Davinci Resolve. I ran out today to shoot a recreational soccer game with 5D3 and shot it all in RAW (few Dual-ISO's) just for the hell of it and it came out quite well, I believe.

I am currently using MLVFS to use  the DNG's in PP_CC(2014) which can read within from the mount itself (which is nice) and can actually playback smoothly in the sequence layout.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm4.staticflickr.com%2F3954%2F15629684635_449ba4ab81_c.jpg&hash=d8bd05944643f49d988bb15f87d62bbc) (https://flic.kr/p/pP9dHi)

My only question now... how come audio isn't being linked together? Do i have to drag the audio folder into it or something?

Thanks again, David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: spnsir on October 26, 2014, 06:47:51 PM
Hey guys,

Resolve audio linked is great but is not working for me. All of the files are loading with a linked audio file, but most of the audio is blank, and only some of my longer clips have the audio start towards the end of the file. I don't know if it makes a difference, but all the clips are shot at 50p. When I play each clip in the source window in Resolve, the audio is linked and works perfectly, but once I load it into the project there's no more audio, or there is but only in a few clips and it starts towards the end of the clip//isn't in sync.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 26, 2014, 07:15:06 PM
Must be a timecode issue. I never actually went so far as adding the clips to projects when I was testing this. I assumed that if they were correct in the browser they'd show up correct once you actually add them. I'll look into it, though, resolve is super slow on my system so it's hard to even tell if the audio is correct because playback is so slow, so testing is a pita.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 26, 2014, 07:42:11 PM
hello,

@dmilligan: thank you so much for the resolve update!!

however, i converted the same sequence 2 times,
first time, audio was right :)
second time, audio was blank in resolve (as described) but i dont know why, cause it was the same file. perhaps different conversion options

if i find out why it was different, i will post my findings..

UPDATE:
in resolve -> Media Tab -> click on the video file -> Play it back -> Sound is here perfectly.
Import into the Media Pool -> Double Click -> Preview with the play button -> Sound has gone..

thats really strange, so when i preview the video file before the import, i can hear the sound, when i import it and play it back from the media directory -> the file is silent.


greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 27, 2014, 10:13:54 PM
@dmilligan

i also found out that in resolve after importing the footage into the media pool, the audio is no longer linked.
when i put the video file into a timeline, a wrong audio file (which is empty) is also placed within the video file onto the timeline, but it is not linked to the video file (it should)
normally i can not take the audio file and drag it to another position - cause it is locked to the video footage.
so to sum it up, at the moment audio files are not imported correctly into resolve, and they are also not linked.
the second interesting obervation is here:
https://www.dropbox.com/s/jnqym9j6wawkl3p/Screenshot%202014-10-27%2021.26.57.png?dl=0
https://www.dropbox.com/s/2ix9qfusvvqmrzb/Screenshot%202014-10-27%2021.26.47.png?dl=0

there are some dots jumping around in playback, they are strange and only in the black areas, but not in all clips.
my options within mlvfs were: resolve naming/Vertical Stripes Correction On/ Chroma Smoothing 2x2

thank you so much, greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 27, 2014, 10:30:04 PM
i can confirm that with rawmagic (sorry but great converter) audio works in resolve as expected.
rawmagic is a little bit brighter, and it has no jumping dots,
mlvfs is darker (dont know which one is right here), and has jumping dots in some darker areas of some (not all) shots

here is the link with 5 dngs from rawmagic (vertical stripe correction on) and from mlvfs (resolve on, vertical stripes correction on, 2x2 on)
perhaps you can find the missing information there..

https://copy.com/YOHU3WPaXbE7YT3q

greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 27, 2014, 11:12:10 PM
The audio thing does appear to be a timecode issue. The DNGs from RAWMagic has the timecode tag. This must need to match the timecode in the wav file. I have no idea how to do this. The CDNG specification merely references "SMPTE 331M-2004", which is not freely available, so I do not know how this field is supposed to work. Maybe @chmee can shed some light?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on October 28, 2014, 01:04:31 AM
Definitely makes sense with the timecode tag...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Audionut on October 28, 2014, 02:04:52 AM
Page 10 of this (http://wwwimages.adobe.com/content/dam/Adobe/en/devnet/cinemadng/pdfs/CinemaDNG_Format_Specification_v1_1.pdf) gives an example.  :-\

Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on October 28, 2014, 02:53:32 AM
Yeah I know, that's the CDNG spec I'm talking about, no real helpful information just a reference to the smpte spec and some meaningless table. None of the things in that table are explained, it is assumed you know what they mean from the smpte spec, which I don't have and isn't freely available. If you can explain it to me feel free...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Audionut on October 28, 2014, 05:16:21 AM
Figured as much, but that table looked so pretty, I figured someone smarter then myself might be able to decode it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on October 28, 2014, 08:23:35 AM
as described in this thread (http://www.magiclantern.fm/forum/index.php?topic=11520.0) (and this (http://www.magiclantern.fm/forum/index.php?topic=10861.0)), there are some more metadata in the wav-file to give it a timecode. cdng has its timecode, yes, and this is quite easy to transform (look on main code raw2cdng, line 1088 (https://bitbucket.org/chmee/raw2cdng/src/772e76f0a55f18a0c83fb0ac5c45d88b2324ea98/csharp2010_code/raw2cdng_v2/MainWindow.xaml.cs?at=master) and calc.cs), but the wavfile needs a timecode in units of samplerate*frames(daytime)
Code: [Select]
long timeRef = (long)(mData.data.audioData.audioSamplingRate * time2frame);the last function in my code io.cs (https://bitbucket.org/chmee/raw2cdng/src/772e76f0a55f18a0c83fb0ac5c45d88b2324ea98/csharp2010_code/raw2cdng_v2/io.cs?at=master).. my converted files does NOT behave like blackmagic-files, which a path looks like a mediafile - our cdng-folders look like folders with audio and video inside - but they react according on auto-sync.

regards chmee
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 29, 2014, 01:19:20 PM
thanks @chmee, I think I figured out the DNG timecode, Resolve is still not working though, perhaps I'm just not setting them to the correct thing. The DNG timecode doesn't need the date part, correct (last 4 bytes)? And it should be set to the time of day? or is it just relative to the start of the sequence?

perhaps you have an example DNG header and wave file header that work correctly (just a screenshot of the hex is fine)?

my converted files does NOT behave like blackmagic-files, which a path looks like a mediafile - our cdng-folders look like folders with audio and video inside - but they react according on auto-sync.
does this mean that auto-sync doesn't work if I try to make them show up like blackmagic files? or are you just saying that you don't do it like that?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on October 29, 2014, 01:50:16 PM
(will search for examples..) So, yes, i take the created/modified-time of file (this is the datem/time of recording) as startpoint, the timecode for first picture.
Quote
..doesn't need the date part..
i didnt used it. maybe its important for datechange (midnight)..

[Resolve][MediaExplorer] cant tell you. my knowledge so far is simple. if i use according naming-convention [name]_C001_0000_[framenumber] and path [name]_C001_0000, it will show up as a mediafile instead of a folder BUT only if theres no audiofile inside. just wanted to say, my convertions are auto-syncable, but they dont show up as mediafile. something i couldnt figure out.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 29, 2014, 03:06:23 PM
it will show up as a mediafile instead of a folder BUT only if theres no audiofile inside. just wanted to say, my convertions are auto-syncable, but they dont show up as mediafile. something i couldnt figure out.
well, I guess I figured it out, though I'm not really sure what exactly makes it work, but I use the naming convention and the audio files and DNG sequence show up as a single mediafile and play correctly in the media browser, however, when you actually import them to the timeline, the audio doesn't work.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: chmee on October 29, 2014, 03:37:27 PM
Enlighten us :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: spnsir on October 29, 2014, 03:55:53 PM
from my experience, it seemed like the audio is properly linked in the timeline, just the timing is not right. it is starting up late, towards the end of the clip. but not sure why the audio timing is correct in the source window.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: spnsir on October 29, 2014, 05:43:46 PM
In Resolve, if you right click and select "Add into Media Pool With Offset" and then enter in a value around 3500, the audio should be close to sync. I'm not sure if 3500 will work for you guys but it worked with multiple clips for me.

Note: it doesn't put the audio perfectly in sync so this isn't a usable workaround unless you want to fiddle with finding the exact offset for each clip, but I think this may help dmilligan get closer to figuring out what's causing this.

Screenshot:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fwww.dropbox.com%2Fs%2Focrwztlbvcais28%2FScreenshot%25202014-10-29%252012.39.39.png%3Fdl%3D1&hash=f272d371f9856bb65744c4c8e0e5d466)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on October 29, 2014, 11:06:29 PM
Okay well it seems to work if I just use 0 for the start time code in the DNG and WAV. Mac download updated so try it and let me know if it works.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: spnsir on October 30, 2014, 05:50:15 AM
So far so good, working here! Thank you for the quick fix David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 30, 2014, 07:06:54 AM
YES!! Great! It WORKS!!! BBIIGG TTHHHUUMMBBSS UP!! This is really great and a big step forward.

btw. i have found out that chroma smoothing (especially 2x2) causes those jumping hot? pixels in the dark areas..
can there anything be improved? perhaps from a1ex code?
thx swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on October 30, 2014, 08:10:26 AM
Nice work :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on October 30, 2014, 11:06:40 AM
could someone show me a screenshot with the file and path? am i right, the filenaming-convention for

path is [name]_1_yyyy-mm-dd_0050_C0000
dng is [name]_1_yyyy-mm-dd_0050_C0000_[sequencenumber].dng
wav is [name]_1_yyyy-mm-dd_0050_C0000.wav

@swinxx
i assume chroma smoothing and banding are time consuming, so it wont be realtime? prove me wrong. if it is rt (and the dng's look right in premiere as well) its time to stop developing my tool. because the filesystem-idea is much better than the converting-thing.

regards chmee
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 30, 2014, 01:16:42 PM
Actually the filename convention seems to have nothing to do with the audio showing up correctly, even if I don't use the filename convention at all, the audio and dng sequence still show up as one media file. The only thing the filename convention really does is make the folder itself show up as a media file. I think it has to do with what the extra metadata in the wav file is. See my code (wav.c) for what I'm doing. I actually got most of that information from you (the bext and iXML stuff), so I'm surprised to hear it doesn't work for you.

chroma smoothing is quite slow, but I think banding is realtime or almost realtime if your computer is fast enough (it only calculates the coefficients once for one frame and then reuses them for the rest of the frames, it's the calculating that's slow, applying the correction is quite fast, all we have to do is subtract-multiply-add integers).

One thing you can do though, is to simply not use any of slow options until final render. So you can edit, CC, etc. until you are completely finished, then turn on the options, start the render and let it chug away (just make sure to clear any render caches when doing this).
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on October 30, 2014, 01:27:38 PM
@chmee
I have a fast computer;) so it runs nice also with chroma smoothing options on, the only thing is that thise pixels are spinning around, so it is not possible to really use it. I wonder that no one else had seen them before..?

@dmilligan:
So if i have a directory, with mixed mlv files (normal and dual iso ones) how would i mount them? All together or dual isos seperately?  Would it be a smart addition to add the possibility for making a file dependent dual iso on off option in the webserver??
Thx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 30, 2014, 02:00:05 PM
It should be able to detect pretty quickly that it's not dual ISO. It may hurt performance slightly to have dual ISO turned on for non-dual ISO files, but I think it will probably still be realtime. Just try it and let us know.

I don't really understand how the chroma smooth algorithm actually works because I've never dug into the code. I just implemented the code straight from cr2hdr/mlv_dump. So I have no idea what could be causing the artifacts you're seeing or how to fix it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on October 30, 2014, 02:53:40 PM
@dmilligan
thanks. by now the main difference - it seems - is, you are starting on 00:00:00 with timecode, will do some tests.

@swinnx
[fast pc] thats good for you, but it limits the usability for the slower pc owners. indeed, it should be an option, enabling cs and vb.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 31, 2014, 08:57:54 AM
@chmee.
yes fast read speeds (i have a raid system) are very useful for working with raw video. the write speed doesn´t matter, cause when delivering a video or playback a sequence, the limiting factors are gpu and cpu. and i have trained myself to cut before denoising and cc.

what i do not understand is, why do so less people download that tool... i think the problem is that they simply are afraid of a complicated workflow. all i can say to all the magic lantern users out there.. just try that program, the workflow is sooo simple and soo fast! the best program out there..

what i do not understand when working with resolve is that how do you handle the mounting points? do you apply the service at the main video folder where all the different directories with the mlv video files are placed into or do you apply more than one mlvfs service (one for every project, or one for every scene?) i think when this mystey is solved mlvfs will probably my solution with no proir conversion. so far i always copy the files out of the mounting point to a dedicated directory. - i like to be on the safe side..

@dmilligan
why are there 2 different naming schemes? isn´t it possible to work with resolve naming scheme in ae or premiere?
I have a scene of the moon (will update this post with the link to dngs as soon as possilbe).. there are 3 white pixels which are bad or hot or whatever pixels.. with option 1 (Bad Pixel Handling) they simply do not go away.. but it is a black scene and the pixel is white..
when i use agressive Bad pixel handling the pixels go away but the moon surface gets really strange.. the algo destroys the moon surface.. so is it possible to adjust the threshold a bit? perhaps with my delivere dng..?

All in All Thank You for you HARD WORK!!
I LOVE THIS TOOL SO MUCH! Everything is working so fine now (except ChromaSmoothing and Bad Pixel handling)

Now MLV is a breeze to work with.

@TO ALL PEOPLE -> https://bitbucket.org/dmilligan/mlvfs/downloads (https://bitbucket.org/dmilligan/mlvfs/downloads) AND TRY THAT TOOL.. ITS AWESOME!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on October 31, 2014, 08:58:28 AM
Can somebody help me ?

I can't compile MLVFS under Mint 17 x64...
When i try, i get a lot of errors.

Code: [Select]
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 stripes.c -o stripes.o
In file included from mlvfs.h:25:0,
                 from stripes.c:26:
raw.h:167:5: error: unknown type name ‘uint32_t’
     uint32_t api_version;           // increase this when changing the structure
     ^
In file included from mlvfs.h:26:0,
                 from stripes.c:26:
mlv.h:42:5: error: unknown type name ‘uint8_t’
     uint8_t     blockType[4];
     ^
mlv.h:43:5: error: unknown type name ‘uint32_t’
     uint32_t    blockSize;
     ^
mlv.h:44:5: error: unknown type name ‘uint64_t’
     uint64_t    timestamp;
     ^
mlv.h:48:5: error: unknown type name ‘uint8_t’
     uint8_t     fileMagic[4];    /* Magic Lantern Video file header */
     ^
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on October 31, 2014, 09:26:07 AM
Oops.  For Linux, in "mlvfs.h", you need to add an include prior to the include for "raw.h":
Code: [Select]
#include <stdint.h>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on October 31, 2014, 10:08:39 AM
Better !  :)
But...   :(

Code: [Select]
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 dng.c -o dng.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 index.c -o index.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 wav.c -o wav.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 kelvin.c -o kelvin.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 stripes.c -o stripes.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 cs.c -o cs.o
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64   -c -o amaze_demosaic_RT.o amaze_demosaic_RT.c
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 hdr.c -o hdr.o
hdr.c: In function ‘identify_bright_and_dark_fields’:
hdr.c:553:13: warning: variable ‘changed’ set but not used [-Wunused-but-set-variable]
         int changed = 0;
             ^
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 histogram.c -o histogram.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 mongoose.c -o mongoose.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 webgui.c -o webgui.o
gcc -c -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 resource_manager.c -o resource_manager.o
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 main.c dng.o index.o wav.o kelvin.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose.o webgui.o resource_manager.o -pthread -lfuse -lm -o mlvfs
main.c:33:18: fatal error: fuse.h: No such file or directory
 #include <fuse.h>
                  ^
compilation terminated.
make: *** [mlvfs] Error 1
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on October 31, 2014, 10:58:30 AM
Found !
Need to add
Code: [Select]
libfuse-dev
Now it compile.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on October 31, 2014, 12:23:06 PM
But webserver doesn't work under Linux.  (http://localhost:8000)
nmap report the port open
Code: [Select]
8000/tcp open  http-altbut nothing...

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 31, 2014, 08:34:43 PM
@swnixx,

There are two different naming schemes in case anyone was using the old naming scheme and had imported stuff into whatever projects => with a different naming scheme none of their paths would be correct anymore and they would have to re-link everything. It's also much more compact, those huge long filenames are kind of annoying if you don't need them.

Quote
what i do not understand when working with resolve is that how do you handle the mounting points?
Handle it however you like, you can run multiple MLVFS instances if you wish. If you like a project to be contained entirely within a directory you could do something like this:

~/projects/project1/                                         <-root dir of project
~/projects/project1/project1.prprj                     <-project file for whatever NLE
~/projects/project1/other_media/
~/projects/project1/mlv/                                  <-original mlvs here
~/projects/project1/mlv/scene1/M00-0001.MLV
~/projects/project1/mlv/scene1/M00-0002.MLV
~/projects/project1/mlv/scene2/M00-0003.MLV
~/projects/project1/mlv/scene3/M00-0004.MLV
~/projects/project1/mlvfs/                               <- mount point

Whenever you're working on this project, mount "~/projects/project1/mlv/" to "~/projects/project1/mlvfs/" and import from "~/projects/project1/mlvfs/..." directly inside your NLE.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on October 31, 2014, 10:36:45 PM
@dmilligan
thank you for your detailed explanation. that makes sense.
will try it.
greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on November 02, 2014, 08:25:42 PM
But webserver doesn't work under Linux.  (http://localhost:8000)
Sorry, I'm still out of commission with respect to working on ML or MLVFS for at least another week, so I haven't been able to maintain the Linux version.  I encourage you (or others) to try to figure out why it's not working!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 05, 2014, 11:23:55 PM
updates: hopefully improved hot pixel detection, try it out and let me know, also if you have some MLV samples with hot pixels that would be helpful (post them here: http://www.magiclantern.fm/forum/index.php?topic=11899.0)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on November 06, 2014, 03:47:14 AM
Thanks @dmilligan for the update -- will definitely try them out and let you know!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 06, 2014, 09:06:40 AM
@dmilligan:
I had a problem where only the bad pix slgo worked, so i will try it out again.
Thank you!! That is awesome.

swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 06, 2014, 08:57:53 PM
hello dmilligan!

i have found a bunch of old raw files and wanted to throw them into a mlvfs service.. but they are not displayed. now i realized that the "mlvfs" only supports mlv files..
will your service ever support raw files (the old v1 mlv) or does anyone know a possibility to convert some raw files to mlvs.

thx in advance, swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 06, 2014, 10:47:08 PM
hello dmilligan again.

i would have another question regarding your newest update with the new bad pixel adaptation:
when i deselect the bad pixel option in the webserver, will the service try to remove some pixels anyway?
i ask because my video seems clean now, without anything selected :=)

thank you very much
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 06, 2014, 11:23:22 PM
will the service try to remove some pixels anyway?
No

will your service ever support raw files (the old v1 mlv)
IDK. Probably not. I want to encourage people to use MLV and not RAW. It would also be a lot of work. There's lots of important metadata missing too.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bennyray1 on November 07, 2014, 01:05:25 AM
Not sure exactly what happened but only up until this latest update MLVS is not showing up in my services after right clicking on my .MLV files...is anyone else experiencing this? @dmilligan is there away I can start over with a fresh install, instead of replacing MLVS? Running on OS X 10.9.3,I even updated OSX Fuse.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 07, 2014, 02:15:06 AM
In the Finder menu click on Go > Go to Folder... and type ~/Library/Services/
You should see the MLVFS.workflow there. You can delete it and try installing it again. I sort of doubt it will help.

after right clicking on my .MLV files...
Just to be sure, you are right-clicking a folder right? and not the actual files? You mount a folder containing files, not the files themselves.

(I'm not seeing any issues with the automator service on my end)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bennyray1 on November 07, 2014, 04:18:21 AM
Thank you sir, I was simply trying it on the file its self, it had been a while since I used it and forgot to click on the folder and not the file...thank you very much, working like a charm sir..thank you for the update, everything is working perfect sir.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silvertonesx24 on November 08, 2014, 08:32:30 PM
Would it be possible to integrate this workflow with split files- like M00, M01, M02, etc?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 08, 2014, 10:04:11 PM
Should already be working with split files automatically no problem (and has been since very early on)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 08, 2014, 10:58:44 PM
hello dmilligan:

i just wanna report that your tool rocks. everything works as expected and better :)
no problems so far, the only thing which would make it better would be a possibility to save my preferences..

e.g.: i would like to always have davinci and vertical stripes correction turned on :)
is that possible?

thx. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 08, 2014, 11:15:34 PM
You can modify the automator service and add command line options. Open the MLVFS.workflow with Automator and add options in the "Run Shell Script". So you would change the default:
Code: [Select]
nohup ~/Library/Services/MLVFS.workflow/Contents/mlvfs "$2" --mlv_dir="$1" -f > /dev/null 2>&1 &

to add whatever options you want after the -f:
Code: [Select]
nohup ~/Library/Services/MLVFS.workflow/Contents/mlvfs "$2" --mlv_dir="$1" -f --resolve-naming --stripes > /dev/null 2>&1 &

here's the complete list of options (in main.c (https://bitbucket.org/dmilligan/mlvfs/src/e30abe780435edaee54c94a762bbc4173f88e303/mlvfs/main.c?at=master)):
Code: [Select]
static const struct fuse_opt mlvfs_opts[] =
{
    { "--mlv_dir=%s", offsetof(struct mlvfs, mlv_path), 0 },
    { "--port=%s", offsetof(struct mlvfs, port), 0 },
    { "--resolve-naming", offsetof(struct mlvfs, name_scheme), 1 },
    { "--cs2x2", offsetof(struct mlvfs, chroma_smooth), 2 },
    { "--cs3x3", offsetof(struct mlvfs, chroma_smooth), 3 },
    { "--cs5x5", offsetof(struct mlvfs, chroma_smooth), 5 },
    { "--bad-pix", offsetof(struct mlvfs, fix_bad_pixels), 1 },
    { "--really-bad-pix", offsetof(struct mlvfs, fix_bad_pixels), 2 },
    { "--stripes", offsetof(struct mlvfs, fix_stripes), 1 },
    { "--dual-iso-preview", offsetof(struct mlvfs, dual_iso), 1 },
    { "--dual-iso", offsetof(struct mlvfs, dual_iso), 2 },
    { "--amaze-edge", offsetof(struct mlvfs, hdr_interpolation_method), 0 },
    { "--mean32", offsetof(struct mlvfs, hdr_interpolation_method), 1 },
    { "--no-alias-map", offsetof(struct mlvfs, hdr_no_alias_map), 1 },
    { "--alias-map", offsetof(struct mlvfs, hdr_no_alias_map), 0 },
    FUSE_OPT_END
};

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 08, 2014, 11:31:50 PM
thx. that worked flawlessly :=)

greets. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on November 09, 2014, 12:07:31 PM
I feel pretty dumb :( ::). I'm getting the same problem as I got some time ago (http://www.magiclantern.fm/forum/index.php?topic=13152.msg129070#msg129070 (http://www.magiclantern.fm/forum/index.php?topic=13152.msg129070#msg129070)) where you go through the normal procedure of mounting the the MLV folder and assigning mount point.

Then it appears to have mounted the folder and shows the server on local host etc. But as soon as I click on the folder it it disappears into the ether? Also the mount point folder disappears?

This is with the latest MLVFS from the first page here. Running on Mavericks 10.9.5 with FUSE for OSX 2.7.2

Could you be so kind as to tell me where I am going wrong? It's probably something I am doing. Also is it likely that Paragon NTFS for Mac could, in any way, interfere  with this?

I apologise for my stupidity! :D
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 09, 2014, 02:10:30 PM
If the mount disappears, that means MLVFS crashed. Perhaps it's a particular MLV file that it's having trouble with. Can you try and narrow it down some? If you can't, I can work on a version that logs some diagnostic info to a file so we can figure out why it's crashing.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on November 09, 2014, 02:50:43 PM
Thanks for the reply. :)

Well first I thought that I should get rid of the *.idx and *.MLD files from the folder so as to allow these to be rebuilt if necessary. So I tried with a folder with lots of *.MLV files and when I mounted it rebuilt the *.idx files.

After that I went to browse and I get the same problem :(.

So I decided to try it with another folder with just a few *.MLV files contained. Unfortunately the same again. :( ???

Yes I think perhaps a way of logging this would be useful to see what's going on here?

EDIT: I found some crash logs for MLVFS in folder /Users/[USERNAME]/Library/Logs/DiagnosticReports/

Do these help?

https://mega.co.nz/#!MoMxGBab!4-4a_ffgtY1T5I9uW25X3spt-TF8C2K5ZxXv52vbqa0
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 09, 2014, 08:10:52 PM
Thanks, the crash log is very helpful. The issue appears to be related to some file path. What is the exact path to the directory you are trying to mount (is the full path really long)? Also, could you please type the following in the terminal:
Code: [Select]
ls -la /path/to/your/mlv/folder/
and send me the output[/code]
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on November 09, 2014, 09:36:08 PM
OK I don't have a particularly long and convoluted path it's: /Volumes/Storage2.5TB/Media/MLV/ (in fact quite logical I think!)

Anyway here's the output from the terminal:

https://mega.co.nz/#!8wFhlRpJ!Ogjh0zzojGIspGFOBjcjc_dDFQGSx-bksllsRgew_jc (https://mega.co.nz/#!8wFhlRpJ!Ogjh0zzojGIspGFOBjcjc_dDFQGSx-bksllsRgew_jc)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 09, 2014, 10:22:37 PM
Might have something to do with the "." in your file path. Try a source path that doesn't have a dot in it. Doesn't appear to be anything wrong with the files themselves, but there are a whole lot of them. If it is an issue with a particular file there's no way to know which one. Try with just one file in a clean directory with no dots in any of the source paths. I think I've found and fixed the issue but I just want to make sure it is in fact the dot that's causing the problems.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on November 09, 2014, 10:39:11 PM
Well aint that interesting!

I just dropped the dot from the drive name and now it's staying mounted and appears to be working. Thank you for your time on this! :D

Anyway why I think it's interesting is that my drive has been called that for some time now and I haven't had that problem before? (apart from the one time I mentioned ::)).

Also, yes there are lots of files in there, but that's not been a problem before either? I did try earlier with just a few of those copied to a folder and I got the same thing, so it wasn't to do with the amount or crappy files.

But what is good is that it works finally! ;D :D

So thanks again. I owe you a beer ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 10, 2014, 01:17:11 AM
Okay I uploaded a new version, it should handle dots in the file path properly and also very long filenames (even though that doesn't seem to be the issue here). Please let me know if it works.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: epyonxero on November 11, 2014, 12:15:09 AM
I just went to give this app a try on my Mac running Yosemite. Installed FUSE and created a new, empty folder on my desktop to be the container. When I select the folder containing MLV files the folder becomes a FUSE drive and my browser tries to open the LocalHost but the browser window fails and the FUSE drive is empty.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 11, 2014, 12:22:53 AM
Sounds like maybe you did it backwards. Right click on the folder with MLV files in it and run the service, when the dialog pops up to select a folder, pick an empty one to be the mount point.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 11, 2014, 12:38:05 AM
Hello dmilligan!
Everything works so great with your provided service. I wonder if there are any other features planned for future updates?
Greets swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mageye on November 11, 2014, 12:46:04 AM
@dmilligan

Yes thanks again :). It's all good now. There were a few other programs which referenced the Volume name with the dots in, so I'm glad you made that possible!  ;D
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 13, 2014, 09:07:15 PM
hello dmilligan.

omg. suddenly my mlvfs mounting procedure is not working anymore? the browser opens and says can not connect to site bla bla bla.
i have made a complete transfer with all my media data to a new raid system, but now i do not get it to work anymore. the system itself has not changed.
any idea?
how can i completely remove and install it again?

Update:
works on the desktop..
but does not work on my new raid drive.. name is in capitals.. nothing else changed.
the old working path was /Volumes/Daten/\[FILM\ SCHNITT\]/01_CODED_2014-06-15_MONOBIT/2_MEDIA/Raw/Kurhaus\ MLV
the new one is /Volumes/THUNDER/\[FILM\ SCHNITT\]/01_CODED_2014-06-15_MONOBIT/2_MEDIA/Raw/Kurhaus\ MLV

as you can see the only change was from drivename "Daten" to drive name "THUNDER"

UPDATE 2: puh, now it works, had to make it read and writeable for everyone..


thx. sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 13, 2014, 10:58:44 PM
Hi!
Did a quick check and gotta ask this. So, if using dual iso I can actually preview the files in real time and cut, edit etc and before export I check the full cr2hdr20bit and start my exports? Like proxy editing? I mean, I thought I first had to convert my dual iso dng files to regular dng? This is amazing really if it works straight on the timeline. Can David or Swinxx confirm. Or I try more tomorrow. THis sure would be good news.

*So it seems christmas is early this year :).
Quick question. Is it not possible to export to prores through Mlrawviewer in conjunction with mlvfs? I hit the E button but nothing. What are other experiencing?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 13, 2014, 11:22:55 PM
preview the files in real time and cut, edit etc and before export I check the full cr2hdr20bit and start my exports? Like proxy editing?
That's the idea. Assuming that the dual ISO preview is fast enough, and hopefully the preview is close enough to the full dual ISO output that it's useful (albeit at a much lower quality and having issues/artifacts). Let me know if it works for you like this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 13, 2014, 11:29:15 PM
Did a quick test in davinci and the preview worked really nice. Only had a short clip so gonna collect some more files to check on this more thoroughly. Crazy good stuff. Opens up some real oppurtunitys with dual iso workflow.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 14, 2014, 03:46:46 PM
@dmilligan:
what i have realized while comparing some dual iso shots with danne´s tool cr2hdr-r is, that converted dualiso dngs do not get displayed properly with the finder preview on mac osx 10.10 ( don´t know if it works on 10.9)
you can take a look at the dual iso dngs after mlvfs conversion here:
https://copy.com/wWXYBHbJwMHyi4Z2

there is a pinkish cast in the frame of the window. although davinci does a great job, it is not able to cancel that out completely. all in all it was much more present in the past conversion attemps with other tools like a1ex command line tool for dual iso conversion.

in the following link i made a short clip, which represents the major problem with my dual iso shot, which was used in a problematic situation (i know)...
https://copy.com/0nLwfWmR7Ebxy9FW
dmilligan, perhaps you or a1ex could tweak the conversion code cause that cast is flickering.. perhaps it can be cancelled out completely. that would be a major step forward, especially for dualiso movie mode.

greets.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on November 14, 2014, 05:00:31 PM
This is super cool =). I've been trying it out on windows with pismo file mount program. Is it possible to control hotpixel fix and stripe removal in windows?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 14, 2014, 06:57:14 PM
Hi David!
I played a little with the dual iso functionality and as I am a little lazy learning davinci and still wait to get my license for latest adobe premiere I have no deep analysis here.
So. I might be the only one creaming out prores files through AE and so I tried this today using mlvfs. It opens up fine but what I noticed it was lagging my computer considerably. I then started a short export and it was very slow. Also tried drag folders to desktop and it took a long while for export. Is it the cdng process taking som extra time or something else? Amaze imlementation? Maybe my computer? I,m on a macbook pro retina 15,5, 16gb ram ssd harddrive.
I havn,t done a proper comparison yet with a1ex original binary.
Thanks for all your help around here.
/D
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 14, 2014, 07:28:09 PM
For full dual ISO processing it usually takes about 1.5 seconds per frame on my computer. If you turn AMaZE off, it's usually about half that or faster. CDNG doesn't take any extra time. I also haven't done a speed comparison with cr2hdr. The dual ISO preview should be realtime or close to realtime.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 14, 2014, 07:30:43 PM
Ok, thanks. It seems about right. I,ll try to do some comparison. The issue would be the lagging effect which I,m gonna try to narrow down.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 15, 2014, 04:19:27 PM
The reason it's "lagging" is because the Finder tries to load the file to show you a preview, which causes MLVFS to do a full render. This takes several seconds if you have dual ISO "Full" on. I would recommend using a Finder view that doesn't show previews, or avoid browsing around in the Finder once you turn the dual ISO on "Full".

I did some investigating on the speed issues as well. On a test image MLVFS is taking about 6.5 seconds to render a single frame, while cr2hdr20bit only takes about 3. Of the 6.5 seconds, 5.2 is spent doing the AMaZE interpolation. If you use mean32 instead of AMaZE, it only takes 1.5 seconds and the mean32 interpolation part only takes 0.2s. I don't really know why MLVFS's implementation is so much slower, could be a compiler optimization thing, who knows.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 15, 2014, 04:32:40 PM
Thanks for sharing. Maybe implement an cr2hdr20bit export option with possibility to multithread? Might be messy and not in philosophy of the mlvfs workflow but it might speed up things a bit.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 15, 2014, 10:11:11 PM
@dmilligan

would it be possible to implement an "EJECT" button in the webserver?
that would be handy, cause in the finder that eject is very laggy and non responsive.

Update:
Sorry Mate, i have not understood the dual iso thing completely.. so my question:
would it be a good idea, to let mlvfs search for a folder called "dualiso" and let only those mlvs be processed like dual iso mlvs? i ask this because dual iso conversion is a slow process and when those shots are placed in a special folder within the mounted folder, mlvfs could speed up conversion process...? no?

thank you.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Audionut on November 15, 2014, 11:19:23 PM
Are the SSE2 additions to AMaZE working as expected?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 15, 2014, 11:59:22 PM
While AMaZE itself is somewhat slower, the vast majority of the slowdown appears to be coming from cr2hdr code that does stuff with the AMaZE output, not AMaZE itself. IDK if that stuff is being vectorized or can be. Perhaps I should try compiling MLVFS with GCC and see if it makes any difference.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andrzej on November 17, 2014, 06:05:02 AM
Hi, i am geting: "safari can't connect to server"
sys wer 10.7.5 security settings changed.
safari 6.1.6
any suggestions?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 17, 2014, 08:02:46 AM
Perhaps a problem with write permissions of the drive with the mounting point.. I had that problem after updating mac osx to yosemite, just change it that everyone can read and write. Greets
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on November 17, 2014, 01:43:03 PM
This is super cool =). I've been trying it out on windows with pismo file mount program. Is it possible to control hotpixel fix and stripe removal in windows?
Not yet, no, but the other demands on my attention have largely subsided, so I should be able to spend time on ML/MLVFS again.  Stay tuned!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 17, 2014, 01:44:34 PM
Hi, i am geting: "safari can't connect to server"
Is it just the webserver that doesn't work? (does the mount still show up?) Or does it not work at all?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andrzej on November 17, 2014, 03:31:25 PM
Thanks for trying to help :)

Perhaps a problem with write permissions of the drive with the mounting point.. I had that problem after updating mac osx to yosemite, just change it that everyone can read and write. Greets
Changed that, still the same.

Is it just the webserver that doesn't work? (does the mount still show up?) Or does it not work at all?
not work at all. MLVFS service is visible under "services"

Maybe i do something wrong with the installation proces? Is this method correct?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silvertonesx24 on November 17, 2014, 04:26:08 PM
I also get the cannot "connect to server error as well", quite often. The problem seems to be when the operation chokes on certain MLV files. There is no indication of why it chokes on these files, and the only way I can tell is by adding groups of MLV files to a folder to find the one that eventually crashes FUSE.

A crash log is below, let me know if you need any more
http://1drv.ms/1wO7Ny6
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on November 17, 2014, 07:50:21 PM
ayshih:
Really looking forward to the possibility to get the webserver options controlable on Windows
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 17, 2014, 10:40:02 PM
@silvertonesx24
divide by zero error on this line:
Code: [Select]
result = sizeof(struct wav_header) + (uint64_t)mlv_wavi_hdr.bytesPerSecond * (uint64_t)mlv_file_hdr.sourceFpsDenom * (uint64_t)mlv_get_frame_count(path) / (uint64_t)mlv_file_hdr.sourceFpsNom;

Which means that the FPS metadata in the file is either wrong, or being read incorrectly. I can prevent the error, but your FPS is still going to be wrong. It's worth investigating *why* the data is bad rather than just suppressing the error (could be an ML issue). Is there anyway you can send a file that has this issue? or perhaps at least the output of mlv_dump -v
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 18, 2014, 11:18:27 PM
updated with some bug fixes (hopefully)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 18, 2014, 11:51:11 PM
thank you.
will test tomorrow.

greets.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on November 19, 2014, 02:47:22 AM
Will test it out tonight as soon as I get home from work. Thanks again @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 19, 2014, 09:28:10 PM
@dmilligan:
i have just converted some DI Shots with MLVFS, and wanted to inform you that in mac osx 10.10 the finder does not show the image correctly. here is a link to one mlvfs converted di image.
thank you very much, great app.
https://copy.com/aCMp94TfSCCF8jK2

sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on November 19, 2014, 10:20:57 PM
finder does not show the image correctly
Can you elaborate?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 19, 2014, 10:26:48 PM
of course.
when hitting the space bar, finder shows a preview.
this previewed dng is pink/white. there is a screenshot in the upper link.
thx.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 19, 2014, 10:45:37 PM
Not much to do about that. It's a known bug with the Mac Finder, it uses a hard coded black level of 2048 and ignores the black level in the file (which is different because of dual ISO processing that actually utilizes 16 bit). Tell Apple to fix their software.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 20, 2014, 02:32:06 PM
ah. ok.
i just wanted to report, cause all other previewed files work as expected

thx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 23, 2014, 09:15:09 PM
Hi David! I have a question.
I somehow managed to create a davinci resolve naming scheme to cr2hdr-r following your example and naming scheme in MLVFS. Without audio the files appears as it should with no need to enter the folders but when including audio of course you have to enter the folder and audio and movie files are separated.
using exiftool I can put in the TimeCodes: 00:00:00.00 in my DNG:s but that alone don,t seem to do the trick. When I view the RIFF info in your wav file I see TimeReference: 0 amongst other info that I don,t find in my wav file. Trying to changing/adding metadata in RIFF section in my wav file seems pretty hard. Don,t really get exiftool to apply here. Would you know how maybe?
Would be really nice to hear what you have to say about this metadata issue and maybe point me in some direction how to solve this?
Using your audio file coming from MLVFS works together with my converted DNG files so changing metadata in wav somehow should do the trick. But how :)?
Update* Used a tool called BWF Metaedit which lets me change metadata but couldn,t really copy the order of things. Still something missing but can,t find out what, hmm
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 24, 2014, 12:13:30 AM
IDK if you'll be able to do it any way besides actually writing code to modify the WAV file itself directly. For reference on what all you need in the WAV file and how to do it see wav.c in the MLVFS source.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 24, 2014, 06:55:47 AM
Thanks, will check on this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 24, 2014, 05:47:10 PM
Thanks David. Got audio and 16-bit dual iso DNG working by shortening audio with sox and adding the ixml info from your wav.c using bwfmetaedit. Also added TimeCode 00:00:00.00 with exiftool. Will wrench it together and put it up in cr2hdr-r in a while. Thanks again.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on November 25, 2014, 08:19:37 AM
Great news, Danne!

Dual ISO MLV would be sweet!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 28, 2014, 03:16:25 AM
Some updates:

I added a "prefetch" option. This will cause MLVFS to preemptively load a certain number of frames to allow utilization of multiple CPU cores even if the program accessing MLVFS isn't loading multiple files at once (this should give a fair amount of speed improvement if you're using processing heavy options with apps that don't already preload a certain number of frames, like a Finder copy). MLVFS was already "multithreaded" but this was only taken advantage of if the accessing program was actually trying to load multiple files at the same time. Now MLVFS will load future frames in expectation that they will be requested at some point soon. Let me know how well this works for you and if you find any bugs/crashing. Probably a good idea not to put a number in there higher than the number of CPU cores you have, or you will just incur extra threading overhead. Some programs like SpeedGrade already do this themselves (and you can set how many frames in the settings), if that's the case, I would recommend using that instead of MLVFS's prefetch.

Added the Adobe calibration matrices (thanks Andy! (http://www.magiclantern.fm/forum/index.php?topic=13948.0)) for all ML cameras to the DNG metadata. This should help improve color "correctness" in non-Adobe programs. Please test (see the linked thread for more info).
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 28, 2014, 06:50:08 AM
@dmilligan:
Thank you very much! Should the new update speed up the appereance of the webgui when mounting a big set of mlv files? Nice.
And should it also improve dualiso conversion?
Would it be possible to add an eject button in the webgui? Had some problems with unmounting it in the viewer..


Big thumbs up and a big Thank You for your improvements, swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on November 28, 2014, 08:04:33 AM
Dual ISO MLV would be sweet!

Oh yes, that would be great
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on November 28, 2014, 09:37:18 AM
i have now tried out the new mlvfs color matrix.

in the following example i seperated the white of the shirt and set it to the center of the vectorscope in both shots

https://copy.com/LvhkenDVEDPDkyj2

to my suprise the blue color changed dramatically.
left side is the old one, right side the new one.

i would prefere the left side, so perhaps dmilligan: would it be possilbe to put the new matix to the webgui as an optional setting?

thx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 28, 2014, 02:16:27 PM
would it be possilbe to put the new matix to the webgui as an optional setting?
It's actually not possible to make this an option, the size of the CDNG IFD is hardcoded. I wouldn't want to make something like this an option anyway, there's probably something else going on that needs to be fixed, and that's the real problem that should be addressed. It might be that the AsShotNeutral is getting computed wrong because it's using routines from dcraw that use the old dcraw matrices. I'll update them and see if it makes any difference.

Should the new update speed up the appereance of the webgui when mounting a big set of mlv files?
Probably not.
And should it also improve dualiso conversion?
Yes, especially since you're using the Finder, which only tries to read the files one by one.
Would it be possible to add an eject button in the webgui?
Probably. You can always type "umount /path/to/mlvfs/mount" in the terminal. (Note there is no "n" in "umount")


Dual ISO MLV would be sweet!
Oh yes, that would be great
Dual ISO conversion has been implemented in MLVFS for some time now.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on November 28, 2014, 04:51:09 PM
Thanks @dmilligan and that quote was actually for @Danne -- my fault on the confusion and again thanks for your kind update!

Enjoy your Holidays!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 28, 2014, 05:22:27 PM
Cool stuff David. Will test speed when time is on my hands.
@deafeyejedi. Cr2hdr-r has been able to convert mlv dualiso files from the very beginning. Check the first post in the cr2hdr-r thread.
 
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on November 28, 2014, 07:30:11 PM
I knew that I guess what I meant was regarding audio directly.

Happy Thanksgiving to you all and everyone else for never stopping the magic!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: baldavenger on November 28, 2014, 11:49:03 PM
The new matrix has seemingly added a red/magenta bias to my images now.  I'm using a 5D Mark III, bought in March this year.  Would it be possible to download an older build of MLVFS so I can make comparisons between before and after?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on November 29, 2014, 02:33:41 AM
Sorry guys,

After looking at the previous matrices I think I might have got the matrix order wrong. I'm gonna check a few shots and if it's what I think I will fork the repo, change the code and submit a pull req.


Edit: Yes, it's my fault  :-[ There is an extra digit in ColorMatrix2 that shouldn't be there (green-green) and the forward matrices are incorrect. I will recheck everything.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on November 29, 2014, 03:11:32 AM
@dmilligan - submitted a pull req that should fix 5D mark III footage. I'll submit a separate pull req for the other cams later. Sorry for the hassle  :-[


Edit: pull req submitted for all cams. See note on Bitbucket.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 29, 2014, 07:17:43 AM
Thx andy, i'll give it a try when it will be updated ;)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: swinxx on November 29, 2014, 08:09:34 AM
And thx dmilligan for the fast update!! You are great!!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 29, 2014, 04:19:42 PM
Updated the binaries with Andy's changes. Let us know if it's better now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on November 29, 2014, 04:46:41 PM
Hi David!
Dragged a folder with dual iso files to desktop but can,t view the files in acr.
Here is one of the converted dng:s
 https://sprend.com/download.htm?C=4085435b91c746f7aebb0ad4821d3baa

Message from AE
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs18.postimg.org%2Fojl658cbp%2FScreen_Shot_2014_11_29_at_16_48_01.jpg&hash=53feecb05eb17865cc581df55d23c6c3) (http://postimg.org/image/ojl658cbp/)

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 29, 2014, 05:16:23 PM
reproduced..., this is what the Adobe DNG SDK has to say about it:
Code: [Select]
*** Error: ForwardMatrix1 does not map equal camera values to XYZ D50 ***

here are the relevant DNG tags:
Code: [Select]
Model: "Canon EOS 60D"
ColorMatrix1:
      0.7428  -0.1897  -0.0491
     -0.3505   1.0963   0.2929
     -0.0337   0.1242   0.6413
ColorMatrix2:
      0.6719  -0.0994  -0.0925
     -0.4408   1.2426   0.2211
     -0.0887   0.2129   0.6051
AsShotNeutral: 0.6215 1.0000 0.4568
CalibrationIlluminant1: Standard light A
CalibrationIlluminant2: D65
ForwardMatrix1:
      0.7550   0.0645   0.1448
      0.2138   0.8936   0.1075
     -0.0005  -0.4306   1.2562
ForwardMatrix2:
      0.7286   0.1385   0.0972
      0.2600   0.9468  -0.2068
     -0.0093  -0.2268   1.0426

any thoughts @Andy600?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 29, 2014, 06:25:05 PM
found the problem, some wrong negative signs, I'll check them for all cameras and then update it
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 29, 2014, 06:42:38 PM
Fixed problems in the 5D3, 5D2, and 60D forward matrices (and verified 550D is correct). Please test (especially for other cameras, which I haven't checked).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on November 29, 2014, 11:09:36 PM
I'll check the others. Looks like I made some typos on Bitbucket  ::)

edit: checked and double checked all cams - submitted pull req
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on November 30, 2014, 10:15:35 AM
I would like to make sure - i am on windows 7 - does updating mlvfs requires to replace mlvfs.dll only or do i need to make something more (like registering in cmd every time for example)? Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on December 06, 2014, 09:17:50 AM
*feedbacks*

I just ran the latest MLFVS w a Dual-ISO MLV file and seems to look better with the thumbnails looking cleaner than ever! Great job on that @dmilligan -- however I ran into this odd pop up window after trying to export through with smartscript2 in AE...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm9.staticflickr.com%2F8582%2F15956889782_b1c74d6971_n.jpg&hash=f1a3c7b5b870f4dcc19aa8769a6d958c) (https://flic.kr/p/qj4epm)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7470%2F15931765046_19d5795859_n.jpg&hash=690640762bf1c62659958127b220e7f6) (https://flic.kr/p/qgQsGf)

same AE error-code as @danne posted from earlier BUT this MLV file was produced directly from an EOS-M body...

thoughts?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 06, 2014, 03:49:37 PM
That error means the color matrices in the DNG metadata are wrong/invalid. I'll double check them for EOSM (make absolutely sure you have the latest version).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 06, 2014, 05:09:19 PM
They appear to be correct. Try updating again.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on December 06, 2014, 08:54:49 PM
Thanks @dmilligan for the quick follow up on the update -- will check as soon as I get home from work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on December 06, 2014, 10:01:42 PM
Thanks dmilligan for fine work. I havn,t really tested dual iso lately but will in time.
Curious about cdng implementation. Answer if you like. Curious how and where to look for recoding cr2hdr20bit to create cdng files. How is it done?
Shortcut question. Would it be possible to get some kind of mac binary of the cr2dr20bit from you with your code implemented or is this fuse workflow a totally different beast? Like to use it in cr2hdr-r workflow if possible.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on December 06, 2014, 11:02:30 PM
This is purely a genius type of an idea...

This definitely would be a nice addition as well...

 :D
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 07, 2014, 12:08:58 AM
Best way would be to rewrite chdk_dng.c (which is used by ML itself, cr2hdr, raw2dng, and mlv_dump) using code from MLVFS's dng.c
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on December 07, 2014, 07:28:59 AM
What a challenge. Thanks for pointers. I, ll see where it takes me.Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on December 12, 2014, 05:36:17 AM
Final got around to Trying "MLVFS"  It seem to work very well so far, maybe its because I have a PC Video card in my Mac but I have pink cast image
in finder, but Looks OK in After Effect with ACR  ::)
I'm on maverick 10.9.3 with a GTX760 video card ,I will post a image tomorrow. 
I can see how this will greatly speed up A.E. workflow.
Thanks for the great work on "MLVFS" :D
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Audionut on December 12, 2014, 09:37:51 AM
Known issue.  The finder uses an fixed black level.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on December 18, 2014, 04:14:36 AM
Hi,
the web interface at http://localhost:8000 is not working for me. I've tried restarting after installing but it is hung up on "waiting for localhost" indefinitely in firefox, chrome and safari. The FUSE disk mounts, but there are no files inside.

Edit: just realized it's still busy unpacking the mlvs and creating idx files. I will wait until it is finished and see if the webpage is accessible then.

edit 2--just finished, looks like it's all working fine.

Is there by chance the possibility of implementing a function that will fix the aspect ratio for slo-mo clips?

Also, is there a way to make these volumes searchable by Spotlight? Right now Spotlight can't see them, I think it's a permissions issue.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on December 18, 2014, 02:36:10 PM
Not sure if this has been mentioned but the latest version of PFMAP-171-win works fine in W8.1 (64bit) for .mlv files.  It doesn't/can't? successfully mount any .M00, M01 etc files.

All the files that I've tested with didn't have any embedded sound so I don't know if it works with an MLV with Audio file. Something to try another day and report the outcome.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on December 18, 2014, 04:32:35 PM


Will ayshih implement the other stuff on the windows version, like strips correction?  Im having problems with g3gg0 windows version... empty folders =(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 18, 2014, 04:48:11 PM
Is there by chance the possibility of implementing a function that will fix the aspect ratio for slo-mo clips?
If only the DNG spec supported a "pixel aspect ratio". It doesn't, so there's basically no way to do it. Re-scaling must be done after demosaicing, so for now you'll have to do the stretch manually in your NLE.

Side Note: It's very strange to me that Adobe would not have support for this in the DNG spec. It would also be very useful for anamorphic lenses.

Also, is there a way to make these volumes searchable by Spotlight? Right now Spotlight can't see them, I think it's a permissions issue.
Probably not, if it doesn't work like it is now. Certain folders pretty much have to remain read only.
Using ls (http://en.wikipedia.org/wiki/Ls) and grep (http://en.wikipedia.org/wiki/Grep) works pretty well though (and regular expressions (http://www.regular-expressions.info/) are way better for searching if you know how to use them, also if you don't like Apple to know about everything you search for (http://arstechnica.com/security/2014/10/mac-os-x-yosemite-reportedly-leaks-location-search-data/)):
ls -R | grep "search expression"

Will ayshih implement the other stuff on the windows version, like strips correction?  Im having problems with g3gg0 windows version... empty folders =(
http://www.magiclantern.fm/forum/index.php?topic=13152.msg133827#msg133827
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on December 18, 2014, 06:08:24 PM
Quote from: dmilligan
Quote from: bwinter88 on Today at 04:14:36 AM
Quote from: bwinter88
Also, is there a way to make these volumes searchable by Spotlight? Right now Spotlight can't see them, I think it's a permissions issue.
Probably not, if it doesn't work like it is now. Certain folders pretty much have to remain read only.
Using ls and grep works pretty well though (and regular expressions are way better for searching if you know how to use them, also if you don't like Apple to know about everything you search for):
ls -R | grep "search expression"

Thanks for the reply. I've read that to enable Spotlight on FUSE volumes you can use something called the "allow_other" option at mount. I won't pretend to understand FUSE or coding at all but what do you think? would it be possible to add this without affecting anything?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on December 18, 2014, 06:17:45 PM
Thanks dmilligan =)    Today i have read the whole post and see that but since it's from 14 of November... Hope ayshih can continuous this app for windows. 
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 18, 2014, 09:34:27 PM
Tried this on my MacBook a couple months ago and couldn't get it working at all. I installed it again last night on my iMac and everything worked fine, right from the start. Going to try it again on the MacBook tonight or tomorrow and see if it was a technical issue, or just a user malfunction  ;)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on December 19, 2014, 02:25:05 AM
Will try this converter again on a new Mac mini that is supposed to arrive tomorrow -- should be quicker than my current 2006 MBP.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 20, 2014, 07:24:11 PM
Thanks for the reply. I've read that to enable Spotlight on FUSE volumes you can use something called the "allow_other" option at mount. I won't pretend to understand FUSE or coding at all but what do you think? would it be possible to add this without affecting anything?
Simply pass the FUSE options you want when you start MLVFS, it will recognize them.

I have also updated the automator workflow to use the necessary options (-o allow_other -o local). So MLVFS will now show up as a 'local' drive (hopefully this won't cause any issues, there's a warning against doing this on the OSXFUSE FAQ, so let me know), and spotlight search should work (I found you actually need both of these for spotlight to work). I also figured out how to make the name show up different in the Finder. So it's no longer "OSXFUSE ....", but it will simply keep the name of the mount directory and have "(MLVFS Mount)" at the end.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 20, 2014, 07:51:15 PM
So far, so good. The local drive is a nice feature, running fine now on my MacBook too. One thing though, did I miss the detail or does this not support .raw, only .mlv?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 20, 2014, 08:39:14 PM
MLV only. I want to discourage people from using RAW format. There's very vital metadata missing amongst other things.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 20, 2014, 11:46:11 PM
Ahh, I see. Had some .raw's from right after raw video was first developed to see how much nicer the converters are now. And then I spent 5 hours looking at different ways to edit and save exif/xmp information so that the .dng's would be universally compatible .   ::)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on December 21, 2014, 08:19:45 AM
@Danne -- just got my mac mini installed and upgraded with two 8gb of RAM (16gb total) along with a 360gb SSD. Looking forward to your cdng implementation test in the near future I hope.

Can't wait to test out MLVFS on this tiny beast...

Hopefully it won't get too HOT!

 ::)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on December 21, 2014, 05:42:20 PM
@dmilligan:
hello, so 2 days ago you commited a silent service update :)
nice. thank you.

great.
greets sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 22, 2014, 03:52:39 AM
Expanding on my previous post about exif metadata, most, if not all, video/image editing programs claim to be able to interpret exif/xmp data but it seems like there are so many exif tags for most of the current digital cinema cameras that our poor little ML Raw files aren't supported very well.
ACR does a great job of adding and interpreting exif, so my idea was to open the mounted .dng's in Bridge ACR, make my adjustments, save the exif/xmp to the images and then import into Resolve or FCPX to skip the transcoding and get the best Raw processing possible. Bridge wouldn't write new exif/xmp to the images or the camera raw database for some reason (parsing error), so is there another way to go about this? I looked into .MXF files but haven't tried them yet.
On another note, Black Magic is developing a CDNG wrapper so that quicktime plays the files natively, hopefully that's something we can have implemented as well.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 22, 2014, 01:03:09 PM
then import into Resolve or FCPX to skip the transcoding and get the best Raw processing possible.
What kind of stuff are you trying to add, that makes Resolve or FCPX work better? The CDNGs from MLVFS should already have all the necessary metadata to open correctly. Thanks to Andy600, all the color calibration matrices from Adobe are also now included. AFAIK, the vast majority of XMP data is ACR specific. XMP is really just an XML schema that Adobe uses to store non-destructive raw adjustments (i.e. the values of ACR sliders). You can also put EXIF tags/info and various other stuff in the XMP format, but there's no reason to, because you can put EXIF tags as IFD entries (TIFF headers) directly into the DNG.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 22, 2014, 02:02:01 PM
Color management. Compatibility. A thorough record of information for archiving, transcoding and sharing footage across programs, platforms and OS's. I've always transcoded to Prores before editing but since MLVFS makes it so much quicker to access the dng's, a raw workflow makes more sense now.

I'm not saying that the files lack anything, just offering a different perspective. I have a decade of audio experience but I've only been shooting and editing a few years. I want my mix and master files to sound clean, full and powerful on any speaker or program its played back with, and I've found that video files can be wildly unpredictable. Basically, a universal wrapper or data file.

Just a "bigger picture" point of view, really.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 22, 2014, 04:28:24 PM
Color management. Compatibility. A thorough record of information for archiving, transcoding and sharing footage across programs, platforms and OS's.
Well, that's sort of the point of the DNG specification, and all that stuff is already there => all the information needed to properly and consistently interpret the DNG is already provided. If certain programs do not abide the DNG spec and ignore particular metadata (e.g. the Apple Finder ignores the blacklevel metadata), which is quite rampant (even for Adobe who created the DNG spec), then there's nothing I can really do about that.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on December 24, 2014, 06:38:26 PM
Yeah I definitely understand, hopefully we'll see a lot more support for CDng in the future.

Interestingly enough, I found a Quicktime and quicklooks component for playing back Dng's, but unfortunately its been discontinued.

http://hamburgpromedia.com/products/mxf4mac/codecs/dng-qt.php (http://hamburgpromedia.com/products/mxf4mac/codecs/dng-qt.php)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Piotruchna on December 28, 2014, 02:59:53 PM
Hello,
I am not sure if anybody had this problem but Ill just type it out:

If you use latin extended characters (for me polish) in names of the folders you will not be able to mount .mlv files. Error window will come out. Just changing name from "kręcić" to "krecic" will make everything work.

Also thanks!
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 28, 2014, 04:08:41 PM
Makes sense, the standard C libraries do not have multi-byte character support (UTF-8 ), only ASCII. This may or may not be a limitation that can be overcome depending on the FUSE library, I'm not sure.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwv656 on January 03, 2015, 07:31:31 PM
I just want to say Thank You to all of you made this possible.
I couldn't believe my eyes watching non-converted MLV playing real-time in resolve.

I haven't tried it on PC side yet but the OSXFUSE+MLVFS works incredibly well.
Thank you so much once again!!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: limey on January 04, 2015, 05:20:46 AM
Help me understand where I'm messing this up.

I'm using Windows.

I've installed PFMAP.
I've registered the DLL.
In a windows explorer window I find a MLV file and use Quick Mount.

I then goto Premiere and import the first image of the image sequence on that quick mounted MLV folder that is created.

When it gets imported the file is black and only the audio works.

**initially was mounting MLV files on an external drive, but wanted to rule out that it wasn't a loading issue and copied the files to the desktop in its own folder (MLV Test). Didn't fix it.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 04, 2015, 09:44:41 AM
Can you post a DNG?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: limey on January 04, 2015, 04:21:18 PM
The M21-0244 DNG is the one that comes out black.

I tried another MLV after quick mounting it and importing to Premiere and it has a Pink overlay. This is (M10-0244)

Both files are totally fine in MLVRawViewer and I've used them both in movies doing different methods before, ie export to DNG, or straight export to .mov file.

https://www.dropbox.com/sh/6sih3x1alfcme9o/AACQj3yySqm72gQ7Cc-z2Nn6a?dl=0

Thanks for your help dmilligan.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 04, 2015, 05:03:48 PM
There's nothing wrong with either DNG you posted. They both appear fine in Premiere and all the metadata looks correct. It may be some kind of issue with how Premiere is reading the data from MLVFS, but I can't really be much help with the Windows port as I don't own a Windows PC.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: limey on January 04, 2015, 05:42:40 PM
I appreciate the time you took to look into it dmilligan.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: limey on January 04, 2015, 07:08:02 PM
As a further sidenote.... When I export from MLVRawViewer to DNG, those will not import to premiere as a sequence, but MLV mounting does (despite the pink or black overlay). Not sure if that helps diagnose anything.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 04, 2015, 07:37:59 PM
You are running the latest Premiere CC? Have you tried SpeedGrade or Resolve or some other program?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on January 05, 2015, 04:33:45 PM
Resolve works for me on pc. Still on premiere cs6 though so I can't try to import them there.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: andy kh on January 05, 2015, 05:34:59 PM
unable to register mlvfs.dll. getin this eror msg "the module mlvfs.dll was loaded but the entry-point DllRegisterServer was not found Make sure that mlvfs.dll is a valid DLL" can someone here help me
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on January 05, 2015, 06:05:19 PM
These are the steps I followed that worked:
1. Downloaded the file and renamed it to add the ".dll" on the end.
2. Copied the file to my Windows folder.
3. Left-clicked somewhere in the Windows folder.
4. Held down the shift key while right-clicking, and chose "Open Command Window Here."
5. Typed the command per ayshih.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: timbytheriver on January 14, 2015, 10:05:12 AM
Hello

I've installed binary on Mac OSX 10.8.5 as per instructions.

I'm not getting the services MLVFS option showing on my right-click menu > services > MLVFS

Have rebooted, repaired prefs. Can anyone help me with a solution please?

Thanks

Tim

PS I have checked in the main Preferences cp and it lists MLVFS as a ticked option for the menu. What gives? :S
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 14, 2015, 01:47:00 PM
Are you right clicking on a folder?

I've installed binary on Mac OSX 10.8.5 as per instructions.
I've never tested MLVFS on 10.8 so it could be this, nor do I really have the capability (or desire) to support old OS versions. I would recommend updating to the latest (10.10)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on January 14, 2015, 01:53:46 PM
Guys, is vertical banding correction works on windows version? Seems not but maybe i missed something.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: timbytheriver on January 14, 2015, 02:57:20 PM
Are you right clicking on a folder?
I've never tested MLVFS on 10.8 so it could be this, nor do I really have the capability (or desire) to support old OS versions. I would recommend updating to the latest (10.10)

@dmilligan I was clicking on the mlv itself. Doh!  :o  It's working fine. My bad, sorry.

PS And thank-you for an inspirational approach in this app.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on January 16, 2015, 01:02:58 PM
I just started to play again with the MLVFS on my freshly reinstalled system.
- OS: Windows 7 Ultimate 64bit
- Pismo File Mount version: 2014.7.11
- mlvfs.dll version: latest from bitbucket (2014-10-03)
- AE CC 2014 version: 13.2.0.49 (2014.2 Release)
- ML: Nightly.2015.Jan15.550D109, mercurial changeset 4fff9f6cb33f (unified) tip

I can't import the DNG sequence into AE. After I adjust the settings in ACR (version 8.7.1) during the import process and hit OK, it gives me an error, saying:
"After Effects error: The file format module could not parse the file 45::35"

This happens only with my new MLV file (https://www.dropbox.com/s/pac7egqarj34kko/M16-0963.MLV?dl=0) (recorded today). If I import an older MLV file (https://www.dropbox.com/s/k1ut4s3oc4da2ny/M18-1424.MLV?dl=0), which was recorded with a nightly build from previous year (I don't know the date), then everything works fine.

Somebody could help me out with checking the MLV files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 16, 2015, 10:12:56 PM
They both work fine on the current Mac version, and there doesn't appear to be much difference between the two. I guess you'll have to wait until somebody with Windows and some programming skills updates the Windows version.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on January 16, 2015, 10:50:40 PM
Is it possible to speed up dual iso conversion (on mac) ?? Thx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 17, 2015, 03:56:20 AM
Yes. Use mean32 interpolation instead of AMaZE, or figure out how to optimize this code from cr2hdr, 75% of the time is spent here (I don't even understand how it works/what its doing, much less being able to optimize it, so good luck):
Code: [Select]
        for (int y = 5; y < h-5; y ++)
        {
            int s = (is_bright[y%4] == is_bright[(y+1)%4]) ? -1 : 1;    /* points to the closest row having different exposure */
            for (int x = 5; x < w-5; x ++)
            {
                int e_best = INT_MAX;
                int d_best = d0;
                int dmin = 0;
                int dmax = COUNT(edge_directions)-1;
                int search_area = 5;
               
                /* only use high accuracy on the dark exposure where the bright ISO is overexposed */
                if (!BRIGHT_ROW)
                {
                    /* interpolating bright exposure */
                    if (fullres_curve[raw_get_pixel32(x, y)] > fullres_thr)
                    {
                        /* no high accuracy needed, just interpolate vertically */
                        not_shadow++;
                        dmin = d0;
                        dmax = d0;
                    }
                    else
                    {
                        /* deep shadows, unlikely to use fullres, so we need a good interpolation */
                        deep_shadow++;
                    }
                }
                else if (raw_get_pixel32(x, y) < white_darkened)
                {
                    /* interpolating dark exposure, but we also have good data from the bright one */
                    not_overexposed++;
                    dmin = d0;
                    dmax = d0;
                }
                else
                {
                    /* interpolating dark exposure, but the bright one is clipped */
                    semi_overexposed++;
                }
               
                if (dmin == dmax)
                {
                    d_best = dmin;
                }
                else
                {
                    for (int d = dmin; d <= dmax; d++)
                    {
                        int e = 0;
                        for (int j = -search_area; j <= search_area; j++)
                        {
                            int dx1 = edge_directions[d].ack.x + j;
                            int dy1 = edge_directions[d].ack.y * s;
                            int p1 = raw2ev[gray[x+dx1 + (y+dy1)*w]];
                            int dx2 = edge_directions[d].a.x + j;
                            int dy2 = edge_directions[d].a.y * s;
                            int p2 = raw2ev[gray[x+dx2 + (y+dy2)*w]];
                            int dx3 = edge_directions[d].b.x + j;
                            int dy3 = edge_directions[d].b.y * s;
                            int p3 = raw2ev[gray[x+dx3 + (y+dy3)*w]];
                            int dx4 = edge_directions[d].bck.x + j;
                            int dy4 = edge_directions[d].bck.y * s;
                            int p4 = raw2ev[gray[x+dx4 + (y+dy4)*w]];
                            e += ABS(p1-p2) + ABS(p2-p3) + ABS(p3-p4);
                        }
                       
                        /* add a small penalty for diagonal directions */
                        /* (the improvement should be significant in order to choose one of these) */
                        e += ABS(d - d0) * EV_RESOLUTION/8;
                       
                        if (e < e_best)
                        {
                            e_best = e;
                            d_best = d;
                        }
                    }
                }
               
                edge_direction[x + y*w] = d_best;
            }
        }

My dual ISO preview code only takes about 10ms per frame :P The cr2hdr stuff I didn't write, nor do I totally understand everything it does.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: andy kh on January 17, 2015, 09:10:58 AM

This happens only with my new MLV file (https://www.dropbox.com/s/pac7egqarj34kko/M16-0963.MLV?dl=0) (recorded today). If I import an older MLV file (https://www.dropbox.com/s/k1ut4s3oc4da2ny/M18-1424.MLV?dl=0), which was recorded with a nightly build from previous year (I don't know the date), then everything works fine.

i also have problem with new mlv files recorded @ 23.97 fps which i recorded recently. i am unable to import xml in resolve. older mlv files which i recorded last year wirth t2i  works fine.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 17, 2015, 08:55:57 PM
Updates:
fixed issues causing MLVFS to not work with full res silent pictures MLVs
enabled SSE2 extensions for AMaZE (AMaZE itself is faster, but the cr2hdr stuff overall isn't really any faster)
ability to override the framerate
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 18, 2015, 02:01:53 AM
Thanks for the update @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on January 18, 2015, 09:47:00 AM
@dmilligan: Thanks for checking the files and for the explanation.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on January 18, 2015, 09:37:09 PM
great. thx dmilligan.
so we now only need silentpic fullres working for canon 5d mk3 123 :))))
5 month no progression.. hope alex is back soon..
greets.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on January 19, 2015, 08:33:36 AM
Quote
great. thx dmilligan.
so we now only need silentpic fullres working for canon 5d mk3 123 (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fwww.magiclantern.fm%2Fforum%2FSmileys%2Faaron%2Flaugh.gif&hash=c87e4ecafb8b6276a833f475fd5299e9)))
5 month no progression.. hope alex is back soon..

And the 7D.....  ;)

Cheers
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 21, 2015, 09:04:41 AM
@dmilligan -- any idea why the MLV files that are produced from adv_intervalometer during time-lapse in the newly updated FRSP module  (http://www.magiclantern.fm/forum/index.php?topic=12523.msg137957#msg137957module) that I used on 7D 2.0.3 w latest nightly (Jan 15) aren't corresponding with latest MLVFS ... it just keeps crashing on me few mins after I enable '20-bit cr2hdr' and sometimes it would just become unresponsive. (Same even if I don't select any of the options and leave all as default)

Like it would start with a few MLV's (shows DNG's) and then the rest of MLV's (w empty folder -- no dog's) when I thought it may be a long process.... waited for a quite while and still no avail?

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7531%2F16331757545_46d6d49186_n.jpg&hash=86163537881639136b14abc0704f82f6) (https://flic.kr/p/qTbwwD) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7526%2F16145875207_8e6a0d94c2_n.jpg&hash=54dfd5581c61f76ea445e0c01e82931c) (https://flic.kr/p/qAKQbK) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7472%2F16145875167_e25c8173fb_n.jpg&hash=3c2c687135c93202f2f77e38354d96e5) (https://flic.kr/p/qAKQb4)

Heck sometimes the mount folder would just disappear on its own after a while and then I would have to manually restart my Mac Mini to get the 'Mount Folder' back... so strange!?

I even deleted the IDX files just to be sure.

This only applies to MLV's from time-lapses (both non & Dual-ISO) with FRSP enabled -- regular RAW/MLV work fine atm.

This was done on OS X Yosemite 10.10.1 in case you were wondering.

Thoughts?

Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: poromaa on January 21, 2015, 10:10:32 AM
Hi and thanks for yet another great utility for us film makers!

I guess everybody hate requests, so don't see it as one, but I am filming raw with 5D2 and editing on the fly with my macbook air (!) yes, it works. And MLVFS is really what makes it possible.

Just one thought, would it be possible to have a setting for virtual dngs image resolution? I'm thinking it would work like a proxy and be faster than debayering the whole thing? Or is it not working that way at all?

If possible that would enabling having the real footage in low resolution while editing, and then just change settings before render/grading.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dubzeebass on January 21, 2015, 01:49:37 PM
That's a good idea!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 21, 2015, 06:49:58 PM
Good thinking @poromaa!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on January 21, 2015, 07:48:52 PM
Expanding on poromaa's suggestion, maybe have an option to have a full-size embedded jpeg preview, and use it as the proxy? Low rez proxies would work too but might get complicated when we have to edit our project settings for different resolutions.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 21, 2015, 08:28:26 PM
Point taken @N/A...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 21, 2015, 09:04:56 PM
Just one thought, would it be possible to have a setting for virtual dngs image resolution? I'm thinking it would work like a proxy and be faster than debayering the whole thing? Or is it not working that way at all?
Do you not already get real-time playback?

Premiere, SpeedGrade, and Resolve can all playback this stuff in real-time with GPU acceleration (and MLRawViewer too). The main limiting factor is usually hard drive speed. MLVFS would still have to read the entire image data from the MLV file anyway. And also this is a problem:
might get complicated when we have to edit our project settings for different resolutions.

Expanding on poromaa's suggestion, maybe have an option to have a full-size embedded jpeg preview, and use it as the proxy?
CDNG doesn't allow embedded previews (only 1 main IFD is allowed). Though, you could make it an option where MLVFS just outputs JPEGs or TIFFs, and I've thought about this, but it opens up a whole new can of worms that I really have no desire to get into, since you're no longer dealing with raw data: you've got to debayer (and deal with GPU acceleration needed for that), white balance, colorspace transformations, highlight recovery, etc. There are already professional tools out there that do this stuff, better/faster than I ever could. MLVFS does what it needs to do: provide these tools with the raw data in a format they can understand.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 21, 2015, 11:28:34 PM
it just keeps crashing on me few mins after I enable '20-bit cr2hdr' and sometimes it would just become unresponsive. (Same even if I don't select any of the options and leave all as default)
Do you see any mlvfs crash logs in ~/Library/Logs/DiagnosticReports

Heck sometimes the mount folder would just disappear on its own after a while and then I would have to manually restart my Mac Mini to get the 'Mount Folder' back... so strange!?
If that happens you can type 'umount path/to/mount/folder' in the terminal (note the lack of an 'n' in umount, that is not a typo)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mufanito on January 22, 2015, 12:11:40 PM
Hi there,

I´m having trouble getting a timeline in Resolve with dual-iso and regular iso clips. I cannot select the dual-iso option for the dual-iso files on the webserver without afecting the non dual-iso files. So, when mixing the 2 kind of files on a timeline i either get a correct image on the dual-iso files and a weird looking image on the non dual, or if i don´t select the dual-iso option on the webserver, the dual-iso files have that interlaced look and the non dual are correct. I´ve tried to put the two files types on separate folders and mount each one, but as soon as i mount the second one the first webserver tab gives place to a new one and it´s settings override the ones i set previously.

So, any tips on how to get dual-iso and non dual-iso files on the same timeline?

Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 22, 2015, 04:10:06 PM
You can start the webserver for the 2nd instance with a different port number with the --port=[port#] option. Then you can have two separate webguis. If you used for example 8000 and 8001 for port numbers, you'll use http://localhost:8000 and http://localhost:8001 to control the different instances.

However, if you have a non-dual ISO shot even if cr2hdr is turned on, it should detect that is not dual ISO and leave it alone (this check will hurt performance somewhat though). What we really need is dual ISO metadata provided in camera, so we know for sure whether to skip or apply dual ISO processing. I believe the MLV format has a provision for this, it's just not implemented. I'll look into implementing this, it shouldn't be too hard.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mufanito on January 22, 2015, 08:31:01 PM
Thanks!

"However, if you have a non-dual ISO shot even if cr2hdr is turned on, it should detect that is not dual ISO and leave it alone (this check will hurt performance somewhat though)." - i´ve tried this but to no avail.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on January 22, 2015, 09:33:23 PM
Ive tried to search this thread for the answer but I can't seem to find it.

MLVFS does not show up in the Davinci Resolve library for me (11 lite).  In OSX I can mount the files fine and they appear in the sidebar of finder. I have selected 'naming scheme Davinci resolve'.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on January 22, 2015, 10:05:26 PM
Nevermind I figured it out somewhat. It doesn't look like other peoples screen shots but I can access the mounted folders.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 22, 2015, 11:00:55 PM
"However, if you have a non-dual ISO shot even if cr2hdr is turned on, it should detect that is not dual ISO and leave it alone (this check will hurt performance somewhat though)." - i´ve tried this but to no avail.
I found the problem. The issue was that the metadata was being written as if the file had been dual ISO processed even if it had been skipped. The image data was correct, but the black level was wrong (cr2hdr processing changes the blacklevel). I'll post a new build shortly.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 23, 2015, 12:11:32 AM
okay, I've uploaded a build with the fix
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 23, 2015, 12:16:26 AM
@dmilligan -- sorry I've been on the road and will be back home tonight to check for MLVFS crash logs per your request.

Also would you think the latest update from today would solve my issue?

Ether way I'll still hope to locate the crash logs for you.

Thanks again for the update, David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mufanito on January 23, 2015, 08:12:32 PM
Thanks dmilligan! Gonna try the new build now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 25, 2015, 07:59:10 AM
Excuse me for the delay...

Here are the recent crash logs per your request from earlier. (I included a brief-test FRSP made MLV file as well just for you to check out)

'For @dmilligan' (http://https://www.dropbox.com/sh/euxlizop58cwz9r/AADwkMslhGNfzuPDfjtmqXOHa?dl=0'For dmilligan') <-- fixed this link (sorry)

currently updating Davinci Resolve and will let you know how it goes.

Thanks, again!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 25, 2015, 03:08:54 PM
http://www.dropbox.com/home/Public/For_dmilligan (http://www.dropbox.com/home/Public/For_dmilligan)
That's not the right link. You need to get the public link from dropbox (only you can access this link you gave me).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 25, 2015, 06:58:03 PM
Sorry dmilligan -- guess I was half asleep last night...

here's the updated link...

'For dmilligan' (http://https://www.dropbox.com/sh/euxlizop58cwz9r/AADwkMslhGNfzuPDfjtmqXOHa?dl=0'For @dmilligan')
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 25, 2015, 08:24:51 PM
The crashes are because you had some MLV files that either didn't have the 'bits per pixel' metadata correct, or MLVFS didn't read it right (my guess would be the former).

The problem you are having with the FRSP dual ISO is actually due to the FUSE library timing out. A FUSE client app (MLVFS) has only so long to respond to a request before the FUSE library will timeout and kill the client and the mount. By default this is set to I think 30 seconds. It's taking the cr2hdr code longer than this to process a single frame, so it times out. You can override the timeout by supplying -o daemon_timeout=0

I would probably recommend against using MLVFS for processing full res dual ISO shots since it takes so long. It's better to use a more tradition conversion approach where you permanently process the DNG frames (so you don't have to do it again). The other reason is that the MLVFS mount is going to appear incredibly unresponsive, because it's have to work so hard to process frames.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on January 27, 2015, 07:00:32 AM
hi,

@dmilligan:
mlvfs worked so fine the last couple of month, its a great tool.
today i opened my working folder to remount some of the older mlvfs dirs, but suddenly safari can not link to the webhost, i get the message that the page could not be loaded..?
what happened. the files i try to mount are working normally..?
i run the latest fuse 2.7.4 drivers and the latest mlvfs

update: puh, i dont know why, but i had to "ignore the rights of the owner of the hdd" to make it work again. so if others have the same problems, just check the box of the information window of your drive.

best wishes, sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 27, 2015, 07:07:53 AM
I would probably recommend against using MLVFS for processing full res dual ISO shots since it takes so long. It's better to use a more tradition conversion approach where you permanently process the DNG frames (so you don't have to do it again). The other reason is that the MLVFS mount is going to appear incredibly unresponsive, because it's have to work so hard to process frames.

I second that now... From here on out I will try to refrain from using Dual-ISO's during Timelapse FRSP (unless if I need to) in order to use your app more often when it comes to TL-FRSP within MLV files.

Thanks again for your time on this!


update: puh, i dont know why, but i had to "ignore the rights of the owner of the hdd" to make it work again. so if others have the same problems, just check the box of the information window of your drive.

was this done on a Mac?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 27, 2015, 08:43:21 AM
I've been prompted to update my Nvidia CUDA and went ahead with it.

Now whenever I try to open Davinci Resolve 11 (with DR enabled within MLVFS) and once I start to scroll through REGULAR MLV's (not Dual-ISO's)... it would then become unresponsive and crash.

I would then get this window...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7313%2F15756352154_4f5b46426f_n.jpg&hash=1db69387871b905e6a08e99ef94a4c0c) (https://flic.kr/p/q1kqy3)

Not sure if this is related to MLVFS because the local web host is still active and this custom-modified Mac Mini sure seems to be up to standards for this type of stuff... (& Yes AE [w Smartimport2] as well as CC PP 2014 works just fine w MLVFS in case you were wondering)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm9.staticflickr.com%2F8605%2F16191198878_a26c95a134_n.jpg&hash=b0b559afd72fa319137cfb9e02c4fbe1) (https://flic.kr/p/qEL8km) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7368%2F16352856826_0f6c42b02a_n.jpg&hash=e00dca18e8d5259964284a6bff65001f) (https://flic.kr/p/qV3EBm) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7292%2F16377933842_782e4bf287_n.jpg&hash=b42c8ce1d3389896ed2378a7f3d22041) (https://flic.kr/p/qXgc8G) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7355%2F16377933782_2a9ae90fe3_n.jpg&hash=5225b6e0d1f7d72447f89d8541978ef1) (https://flic.kr/p/qXgc7E)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 27, 2015, 12:52:38 PM
Sounds like a problem with Resolve or the graphics driver. MLVFS does not use the GPU in any way.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 27, 2015, 09:01:26 PM
Okay thanks -- just thought it would because of the CUDA update that I just did.

Anyone else out here have a similar system OS X with the graphic card that I have?

Anyone experiencing in lagging during playback with MLVFS mounted files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 27, 2015, 09:05:49 PM
Sorry, that was supposed to say GPU, MLVFS does not use the GPU or CUDA, so it's not anything related to MLVFS.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 27, 2015, 09:57:20 PM
Okay gotcha -- so would the graphic card be a factor then?

Strange it worked fine before the CUDA update.

Otherwise should I just downgrade DR back to 10 or 9 -- would this be even worth to try? I think not.

Hopefully I'll get to the bottom of this sooner rather than later.

Thanks again @dmilligan for your efforts!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 27, 2015, 11:13:06 PM
CUDA is a driver for the graphics card that allows programs like Resolve to take advantage of some special types of parallel processing that the GPU is good at. This is typically used for debayering. It sounds like there's some kind of issue between the CUDA driver update you installed and Resolve (google is your friend!). This would have nothing to do with MLVFS.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: niklas.seehausen on January 31, 2015, 11:40:18 AM
The question is pretty dump. But is audio "glued" to the video or do I get 1 audio file and a folder with  cdngs ? :-\
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Lars Steenhoff on January 31, 2015, 06:18:10 PM
Is there a way to change the default settings?

for example I would like to have bad pixel correction and stripe correction always on
What would I need to change to make this the default setting when starting MLVFS?

Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 31, 2015, 08:23:42 PM
The question is pretty dump. But is audio "glued" to the video or do I get 1 audio file and a folder with  cdngs ? :-\
I'm not sure what you mean by "glued", they are not in the same file, but in when viewed in media browsers for most apps (like Premiere of Resolve) they show up and import as one single clip with audio.

Is there a way to change the default settings?
Yes. You can edit the MLVFS service in Automator. One of the "steps" is a shell script that actually invokes MLVFS. You can specify what ever additional command line options you would like there. MLVFS has its own command line options that control the various settings you talk about (see near the bottom of main.c in the source code for a list), there are also a lot of OSXFUSE built in options that control how the mount works. See the OSXFUSE documentation for more info about those.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Lars Steenhoff on January 31, 2015, 08:59:49 PM
Thanks dmilligan

I found the ones that I wanted:
--stripes
--bad-pix
--resolve-naming


Saves me a few clicks every time  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DanHaag on February 01, 2015, 11:30:01 AM
Hi Lars, can you copy&paste the shell script with your adjustments in it so I can see what it looks like?  :) Need those changes, too. But I know nothing about using Automator or writing code.  :-[
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on February 01, 2015, 12:01:32 PM
Any work being done on the ported "Pismo" version for Windows? I find it preferable to extracting the DNGs as I did before and it's really easy to use. It's very fast and responsive when you have mounted the mlv's.  However the mounting time takes quite a while for large files. Mounting like 256gb of mlv's from the harddrive must be somewhere  around 30mins or so. As I understand it its almost instant on the original fuse mac version?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Lars Steenhoff on February 01, 2015, 05:56:07 PM
@ DANHAAG


nohup ~/Library/Services/MLVFS.workflow/Contents/mlvfs "$2" --stripes --bad-pix --resolve-naming --mlv_dir="$1" -f -o volname="$(basename $2) (MLVFS Mount)" -o local -o allow_other > /dev/null 2>&1 &


Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DanHaag on February 01, 2015, 07:08:09 PM
Thank you, Lars! Would never have figured that out myself, helped a lot!  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 01, 2015, 07:14:01 PM
Any work being done on the ported "Pismo" version for Windows?
There is no one working on it ATM.

However the mounting time takes quite a while for large files.
That's because MLVFS needs to build an index file (.IDX file). Basically it searches through the MLV file and builds a table of where all the data is, that way it can access the data quickly. This may take some time for extremely large files. It does this on Mac too, but only "on demand", so the Mac mount will show up instantly, but you may have to wait for the index files to get built before you can actually browse or use the mount. You only have to build the index files once though. So the next time you mount those files, it should be much faster (as long as you don't delete the .IDX file). If you playback the video in camera, the camera does this too, and it's the same format (actually much the same code), that builds that file, and MLVFS can actually use those index files. So if you playback in camera, make sure to copy the .IDX files, and MLVFS should mount faster.

Having "random access" storage (SSD, SD, CF, etc.) will likely make building the index files a lot faster as well. To build the table we don't have to read very much data, but we do have read a little data from all over then entire file. Spinning disks are not so good at this because of "seek time" required to move the head around to different areas. With solid state, we can read from anywhere in the same amount of time (hence "random access").
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on February 01, 2015, 07:36:16 PM
Thanks for your great reply! I guessed it had to do with indexing. I can see I have the Idx files saved next to mlv's I have mounted.  I'll have to test remounting some of them and  see if it's faster.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 02, 2015, 07:03:28 PM
Once the indexing files (.IDX files) are made for each MLV files -- then the next time you mount them with MLVFS it basically would be on the fly.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on February 03, 2015, 12:34:29 AM
HI,

I got the MLVFS to work on my iMac and imported the DMG to Resolve 11. The problem I have is that all the raw clips turned way too blue and I have to adjust the white balance quite a bit. i don't know why as if I converted the MLV with MIRawViewer the white balance is normal. Could you help?

Thanks!

John
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 03, 2015, 01:26:44 AM
Well, it's not particularly important, typically you'll want to make adjustments to the WB anyway, that is one of the advantages to shooting raw.

What did you have the WB set to in camera? Does that not match what MLVFS is producing? I think MLRawViewer ignores the WB metadata in the file, so perhaps the default WB MLRawViewer is using just happens to be correct, while you actually had the WB from the camera set incorrectly and that is the WB MLVFS is using.

If you still think it's a problem with MLVFS, can you upload a sample MLV file that exhibits the problem?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on February 03, 2015, 04:17:44 AM
Thanks for the reply. I used the Canon 5D Mark III and I set the WB as Auto. The WB should be around 5900K but the file from MLVFS has 3200K when it is opened in Resolve. I can adjust it for sure but just feel something wrong in the setting of the MLVFS. I would love to upload the MLV file for you to check and just tell me how to do so.

John
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on February 03, 2015, 07:12:03 AM
@johnwangphoto Dropbox would do?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on February 03, 2015, 01:37:49 PM
Hey!

Thanks for this app! - it has certainly saved me many hours of conversions and wondering wether to keep originals etc.

Can anyone recommend a work flow for deleting unused clips with in Davinci Resolve? I used to update usage data then delete one by one from my folders, but now they are virtual folders so the files arn't real :D

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 05, 2015, 11:46:05 PM
Compressed (LZMA) MLV file support added. You probably won't see realtime playback speeds for compressed MLV files, I would recommend simply lowering the bit depth of the MLV file if you want compression (to 12 or 10 bits), with that method you still get speedy playback. You'll probably see almost no IQ loss converting to 12 bit if you want to save some space with your MLV files. The LZMA compressed MLV files are not much smaller than 12 bit ones, so I'd drop the bit depth before I tried compression.

There is one issue if you use mlv_dump to change the bit depth of the MLV file and that is: it doesn't also update the black and white levels appropriately (if we drop bits off the image data values, we also need to drop bits off of the black and white levels). I have a fix (https://bitbucket.org/dmilligan/magic-lantern/commits/a22302855d05c6cb587336943c8b5fb623a96bb4) for this, until it gets merged, you can try this mlv_dump (https://bitbucket.org/dmilligan/magic-lantern/downloads/mlv_dump).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 06, 2015, 08:14:12 AM
I've given up after the fact that I learned about how I didn't actually need NVIDIA/CUDA installed in my Mac Mini and now that I've uninstalled it... It still LAGS playback!

Doubtful that I had done it completely. Tried in both Terminal and through Library but couldn't find "Framework" folder in Yosemite. Used to be able to in Mavericks.

I miss using MLVFS on the fly ever since my stupidity in updating the CUDA/NVIDIA unknowing that Ill get myself into this pit.

Not sure what's to do at this point other than trying to find a older version of CUDA/NVIDIA but after learning that I don't need it with the graphic card that I'm using...

Feeling  ???  as of lately...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on February 06, 2015, 07:31:10 PM
I just found out if I set the 5DIII WB other than Auto the MLVFS can read the set WB correctly but when I set it to Auto WB for some reason the DNG file from MLVFS always has the WB 3150K (very blue) no matter what the actual WB is. Can MLVFS understand Auto WB or I miss something?

Thanks for any comments.

John
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 06, 2015, 07:35:39 PM
can you upload a sample MLV file that exhibits the problem?

or at least the output of mlv_dump -v
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on February 06, 2015, 09:06:45 PM
OK. Here is the Dropbox link for the original MLV clip and a dng file from MLVFS. The clips was shot on a sunny day with Auto WB. Let me know what can you figure out.

https://www.dropbox.com/sh/lrtgf2xnej7vpjk/AABGgmmVvKY472grMEBeGhu-a?dl=0

Thanks!

John
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 06, 2015, 10:56:28 PM
Code: [Select]
Block: WBAL
  Offset: 0x00000340
    Size: 48
    Time: 405.718000 ms
     Mode:   0
     Kelvin:   3100
     Gain R:   491
     Gain G:   1024
     Gain B:   638
     Shift GM:   0
     Shift BA:   0

Code: [Select]
        unsigned                wb_mode;  // see property.h for possible values
        unsigned                kelvin;   // wb temperature; only used when wb_mode = WB_KELVIN
        unsigned                WBGain_R; // only used when wb_mode = WB_CUSTOM
        unsigned                WBGain_G; // only used when wb_mode = WB_CUSTOM
        unsigned                WBGain_B; // only used when wb_mode = WB_CUSTOM

This is actually sort of an ML issue. If you have the WB set to "auto", then ML doesn't put any information about the actual WB in the metadata. MLVFS uses the kelvin value in the MLV file even when the mode is "auto", but this value doesn't actually contain the actual auto WB (but there's nothing else MLVFS can do!). In your case it was set to 3100. Don't ask me why. AFAIK, we may not even know how to read the Canon auto WB.

I suppose MLVFS could compute an auto WB itself, but this would be quite complicated, and I don't really see the point. WB is something your NLE/CC software should handle.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on February 06, 2015, 11:48:07 PM
Thanks for the effort to figure it out and well like you said that is the beauty of the raw shooting: I can adjust to whatever I want. Maybe shooting a color chart and do the match in Resolve will be the best solution.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: poromaa on February 08, 2015, 03:15:26 PM
If I pull the DNGs directly into Premiere, I get a-lot of aliasing problems at sharp edges. However, if I import the DNG trough ACR (Lightroom/Photoshop) I get a totally different result. Even turning off every single setting (noise reduction, color noise etc) I get a clearly better result from the “ACR”-version.

What am I doing wrong here?

Below is a screenshot from within Premiere, where i have stacked the two versions on top of each other.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs17.postimg.org%2Faw643780f%2FScreen_Shot_2015_02_08_at_15_10_24.png&hash=7896c3632a9a2563a51504958a8cdfd3)
As one can see the aliasing gets really horrible on the file loaded in Premiere (to the right)

I have checked for field order etc but can’t find any problems with it. There does not seem to be any debayering-settings in Premiere either, so I don't know what it's doing. I have also played around with the chroma smoothing, but this does not fix it.

Any ideas?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dubzeebass on February 08, 2015, 03:38:21 PM
Is that dualiso and what camera?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 08, 2015, 03:57:19 PM
If I pull the DNGs directly into Premiere, I get a-lot of aliasing problems at sharp edges. However, if I import the DNG trough ACR (Lightroom/Photoshop) I get a totally different result. Even turning off every single setting (noise reduction, color noise etc) I get a clearly better result from the “ACR”-version.
Well this is to be expected. Premiere does not use ACR. ACR is a very good, custom, CPU based demosaicing algorithm. It's really slow as a result, making it not so good for video. Premiere uses a more generic algorithm that can be hardware accelerated, so that playback is realtime. Therefore it is not as good at removing defects like this.

This really doesn't have anything to do with MLVFS. It's an issue with Adobe (and also the design of your camera, which uses line skipping in video mode, which causes the issue in the first place)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: poromaa on February 08, 2015, 04:41:26 PM
Yeah, the camera (5Dmk2) has some awful aliasing and does not even support progressive dual-iso in video-mode, but I guess I can live with it if using ACR.

My workaround is simply to use daVinci Resolve instead, since that debayering currently looks more like ACR, but gives realtime playback. Drawback with that is that Resolve does not have any free denoise and noisy footage tend to break if color-correcting in Resolve (for me, I guess I have some to learn here).

Well, thanks for the info.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dubzeebass on February 08, 2015, 08:32:37 PM
I wish I could get realtime playback! I have an early 2013 iMac loaded to the max and I have to use node rendering to work quickly.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 08, 2015, 09:55:13 PM
I wish I could get realtime playback! I have an early 2013 iMac loaded to the max and I have to use node rendering to work quickly.

@dubzeebass -- is this currently one of the only few solutions?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dubzeebass on February 09, 2015, 03:51:17 AM
@dubzeebass -- is this currently one of the only few solutions?

That or basic white balance and export as Cinelog-C then re-import the files and apply the LUTs. I often find myself skipping Cinelog-C when it's only a few clips and going right from BMD Film to Hunter's LUT. Not terribly fast but this is a good product... well, both are.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 16, 2015, 09:01:31 PM
I'm currently going to use MLVFS via PP (DR11 stills crashes) just for basic dailies since I have no time for grading plus client likes how I shot it as it is in RAW/MLV (it was well exposed for the most part) so after playing around with @Danne's cr2hdr-r (betas and non betas) which gave me some real decent footage but today I've decided to give it a short run with your app on the "fly" -- it's currently .IDX'ing right now so hoping for good things to come.

Is it normal for it take awhile on a spanned MLV file? (up to 8 spanned files for a single MLV) it took about 3-4 minutes for to spit out an .IDX) lol...

Thanks again @dmilligan!
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 16, 2015, 09:27:30 PM
Also here's what I've BEEN dealing with regarding DR11 for some time now whenever I use MLVFS (I know you mentioned this is solely DR's problem and not with your app) but thought it was worth sharing...

https://vimeo.com/119791222 (https://vimeo.com/119791222)

Another reason why it can be rather difficult to use @Andy600's Cinelogs LUTS in conjunction w MLVFS if DR11 cannot hold up it's performance on my supposedly POS late 2012 modified MacMini!

Perhaps, any one out there have similar experiences?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 17, 2015, 07:05:12 AM
Hi David! How doyou find kelvin values in the mlv files? Are they not to be shown in exiftool efter export with mlv_dump? All I can see is asShotneutral values. Also looking for time code values which alse seems to show somewhere?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 17, 2015, 08:47:11 AM
Ah, found the metadata through mlv_dump.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 17, 2015, 12:50:15 PM
DNG metadata only stores white balance as RGB multipliers (AsShotNeutral). So if you have kelvin in the MLV file, you have to convert it to RGB multipliers for the DNG, and converting to/from kelvin is rather complicated and involves using the cam->XYZ matrices and doing a bunch of matrix transformations.

http://www.magiclantern.fm/forum/index.php?topic=13152.msg128304#msg128304
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 17, 2015, 11:33:40 PM
I have implemented some experimental GIF previews. They are 1/4 res, black and white (from the green pixels so we don't have to debayer; and also because GIF is limited to a 256 color palette, which is further limited to 128 colors b/c of the simplistic way I needed to implement LZW, so it's easiest to just have the color palette be 127 shades of gray), and loop through 10 frames of the MLV file.

These will show up as _PREVIEW.gif in the mount and also in the file tables in the webgui. This is somewhat experimental and needs further testing (you will very likely run into bugs), so I'm not updating the .dmg download in the first post, instead use this link if you would like to test it out: https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_GIFPreviews.dmg

Let me know what you think.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 18, 2015, 12:04:40 AM
Nice work.
I click the name and got them bigger. Good. Press the back button and you,re back in thumbnail mode. Very handy.

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs27.postimg.org%2F8in99sqr3%2FScreen_Shot_2015_02_18_at_00_14_33.png&hash=26d7b0a06b85bb4b082e4c2fbfcceece) (http://postimg.org/image/8in99sqr3/)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs27.postimg.org%2Fofhfsrtrj%2FScreen_Shot_2015_02_18_at_00_14_45.png&hash=8998823bf399a5881437c55bcda78ed4) (http://postimg.org/image/ofhfsrtrj/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chainick on February 19, 2015, 07:23:55 AM
Can I get auto mount mlv-files in PFMAP (http://www.pismotechnic.com/pfm/ap/) on startup Windows?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Pedr on February 19, 2015, 10:35:57 AM
Am I right in thinking that you need to mount each directory separately?

For example if my directory structure is:

- shoots
  - 15_01_07
  - 16_01_07
  - 19_01_07

If I run the MLVFS service on shoots, I just get it's children as empty dirs in the mounted dir. The only way to get CDNGs from the contents of these dirs is to mount them individually. Is there any way to mount a whole directory structure so that any mlvs within child directories are mounted as CDNGs? I Need to use shots from a large number of directories in a project, so individually mounting each directory every time is a bit of an issue.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 19, 2015, 11:26:00 AM
Love the thumbnails preview @dmilligan... looking forward to its progress!

Possible to have the backspace button work as a "Back" as well as oppose to just coming from right-click only?

Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 19, 2015, 01:24:51 PM
Am I right in thinking that you need to mount each directory separately?
No. You shouldn't need to. You can mount an entire directory structure, it should work just fine.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 19, 2015, 01:28:23 PM
Possible to have the backspace button work as a "Back" as well as oppose to just coming from right-click only?
Are you talking about in the web browser? I can't do anything about the way your web browser works.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Pedr on February 19, 2015, 06:33:31 PM
@dmilligan That's definitely not working for me. If I mount a directory that contains sub directories, the web UI does show the subdirs, but then locks up when I try and navigate into one. DR is the same - it shows the top-level dirs, then locks up (spinning beachball) for a while, then the directories disappear and the mounted MLVFS drive unmounts.

I can mount individual dirs by themselves though.

If I can help you debug - send logs etc just ask.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 19, 2015, 07:12:42 PM
If I can help you debug - send logs etc just ask.
you will find crash logs here: ~/Library/Logs/DiagnosticReports
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Pedr on February 19, 2015, 10:21:56 PM
@dmilligan Got plenty of mlrawviewer .crash files, but nothing relating to MLVFS.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 20, 2015, 01:44:24 AM
If the mount disappears that means MLFVS crashed, and if it crashed there should be a crash report.

I've never seen or had any issues with subdirectories, I use them all the time, so my guess is it's something else besides the fact that you are using subdirectories that is the real problem. Make sure you don't have any strange characters in any file or folder names. Try different combinations of MLV files and folder structures. For example, if you have a directory that mounts correctly by itself, put it in another directory and mount that.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 20, 2015, 07:49:47 AM
@dmilligan:

I was referring to when you use mouse and right-click then scroll down to 'back' to get back into thumbnail mode.

Regardless of which web browser -- Is this the only way?

And not the backspace button - correct?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 20, 2015, 01:17:14 PM
I was referring to when you use mouse and right-click then scroll down to 'back' to get back into thumbnail mode.
I don't really know what you mean by "thumbnail mode". If you are right clicking on something and bringing up a context menu, then that has absolutely nothing to do with MLVFS, because there are no context menus or "back" or anything like that in MLVFS. If you click on a hyperlink in the webgui and it takes you to another page and you want to get back to the previous page, then how you do that depends on the browser you are using and the various ways it lets you go back to the previous page. The button in the top left corner of the window with an icon of a back arrow is the back button on almost all browsers, and there are numerous other ways that are browser specific. Here's a list of keyboard shortcuts for Chrome: https://support.google.com/chrome/answer/157179?hl=en&ref_topic=25799 I'm sure you can find a similar list for whatever browser you are using.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Pedr on February 20, 2015, 07:16:41 PM
@dmilligan That makes sense. I'll try and narrow down the issue later today. Anyway. It's a great approach. Thanks for all your work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 20, 2015, 08:25:42 PM
@dmilligan -- you were right it had nothing to do with MLVFS so my fault on that.

However, I did figured it out by going through the advance settings within Safari's.

Thank you for the clarification again.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 21, 2015, 09:08:07 PM
I had some time to test and clear up some bugs with the gif previews, it seems to be pretty stable now, so I have updated the main MLVFS.dmg download with the new gif previews feature. (if you were testing the gif previews, I recommend d/l the latest, lots of bugs/crashes have been fixed).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chainick on February 22, 2015, 09:16:40 AM
On Windows 7 for recursive mount mlv-files I use item in context menu on folder

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fimg-fotki.yandex.ru%2Fget%2F16156%2F6697399.0%2F0_f12d0_4a5d5316_orig.png&hash=f264e5e0a0ae1995029001408e7b448c)


Create AddItemToMenu.reg file and run it.

Code: [Select]
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\PismoRecursiveMount]
"MUIVerb"="Pismo recursive mount"
"Position"="Top"
"Icon"="C:\\Program Files\\Pismo File Mount Audit Package\\pfmcontrol.exe"

[HKEY_CLASSES_ROOT\Directory\shell\PismoRecursiveMount\command]
@="cmd.exe /c for /r \"%1\" %%i in (*.mlv) do pfm mount \"%%i\""

[HKEY_CLASSES_ROOT\Directory\shell\PismoRecursiveUnmount]
"MUIVerb"="Pismo recursive unmount"
"Position"="Top"
"Icon"="C:\\Program Files\\Pismo File Mount Audit Package\\pfmcontrol.exe"

[HKEY_CLASSES_ROOT\Directory\shell\PismoRecursiveUnmount\command]
@="cmd.exe /c for /r \"%1\" /d %%i in (*.mlv) do pfm unmount \"%%i\""
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on February 22, 2015, 02:25:27 PM
On Windows 7 for recursive mount mlv-files I use item in context menu

Whats the difference between recursive mount and quickmount? I already have quick mount option to mount mlvs when right-clicking files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chainick on February 22, 2015, 04:43:23 PM
What will you do if folders nested? For example see next structure of folders:
Code: [Select]
MyProject
  Day1
    CamA
    СamB
  Day2
    CamA
    CamB

I propose on MyProject folder call to context menu  and run recursive mount.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: sgofferj on February 24, 2015, 03:16:12 PM
Could we have a userspace umount option please? :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 24, 2015, 06:17:46 PM
does:
Code: [Select]
umount /path/to/mlvfs/mount/point/
not work?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: sgofferj on February 25, 2015, 12:29:09 PM
With sudo, yes. But that kinda defeats the idea of userspace tools, doesn't it? :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 25, 2015, 12:49:25 PM
no, userspace vs. kernel is not the same thing as root vs. standard user, there are plenty of userspace programs that must be run as root.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on February 27, 2015, 11:02:15 AM
Any news for a Windows update? I can't import new MLVs (DNG sequences) into AE... the interesting things is, that old MLVs (DNG sequences) are working properly. Is this because of the version differenc of MLs? What can cause this thing?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on February 27, 2015, 04:04:11 PM
surami,

Using Pismo, I assume? (as opposed to g3gg0's MLVS Windows Client)
Which ML version?
What symptoms when try to import into AE?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on February 27, 2015, 10:37:32 PM
@Frank7D

Yes, I'm using Pismo File Mount + the latest mlvfs.dll (by ayshih). As I remember the old MLV (https://www.dropbox.com/s/k1ut4s3oc4da2ny/M18-1424.MLV?dl=0) was created with a nightly build from previous year, somewhere in april and the new MLV (https://www.dropbox.com/s/pac7egqarj34kko/M16-0963.MLV?dl=0) is created with a build from this year, january. Are you using Pismo too? If yes, could you try the files.

I get this error in AE "the file format module could not parse the file" 45::35, when i try to import the new MLV's DNG sequence. I can adjust the settings in ACR, but after click on OK, the error window comes into my face.

In theory the mlv_rec modul should be the problematic thing or not? I think this because everything works perfectly with the old MLV.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on February 27, 2015, 11:16:53 PM
surami,

Using your mlv file, I reproduced this error.
Actually, this had happened to me as well, using "magiclantern-v2.3.NEXT.2014Feb27.7D203" build.
It only happened the first time I used the build (for several clips). Since then, no problem.

The problem only affects the first frame of the clip; if you advance to the second frame there is no error.

Not sure the best way to handle the problem clips you already have. In my case, I set the work area to start at the second frame and rendered starting there.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on February 28, 2015, 09:21:50 AM
@Frank7D

Thanks for your time to check the MLVs.

This mentioned trick isn't working for me. Another converter like raw2cdng produces good DNG sequence, what I can import into AE, but it would be better not to store unnecessarily the files.
Ayshih, please come back for an update :) or where lies the rub?

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 28, 2015, 05:05:26 PM
the files work fine in the current mac version
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on February 28, 2015, 09:06:18 PM
@dmilligan:

hi,

may i report a bug within mlvfs which i found today:

today i had a shoot with some slowmo takes. 50fps/5d mk3
i tried to span it over 2 cards while recording to squeeze everything out of the slots :) (CF as main and SD as secondary recording card.)

now i am at home and when i mount those files with mlvfs i get some missing frames within the take. it looks like those sd card frames are missing.
when i convert it with mlrawviewer instead, the frames are all there and it looks ok.

 
i stumbled upon another problem.
when the take was longer ml splitted the files on the cf cards. so i have a mlv and a m00 files.
on the sd card i also have a m00 file from the same take. when i copy them together onto the hdd the sd card files got renamed (cause they are the same m00 as on the cf card.)

on the hdd the files are: mlv, m00 and m00 2.
when mounted with mlvfs those takes are also weired.

those files are also weired (frames skipping) with mlrawviewer.

can you please take a quick look if there is something to tune within mlvfs, or how i should name my files to convert them correctly with mlvfs.

i also noticed that mlrawiewer converted dngs are bout 1.3mb and mlvfs dngs are 2.5mb

havent you implemented 2 compressing feature formats?
where can i find that, cause in the webinterface there is no compressing feature.

thank you very much. and hope to hear from you soon,

swin
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 28, 2015, 11:36:08 PM
For the spanning issue, can you send me example files? I have no way to test spanning myself.

Rename spanned files so that you have .mlv, .m00, .m01, .m02, .m03, etc. it doesn't matter which files get which number, you just need to have every number sequential.

I did not implement compressed output, nor do I ever plan on that, there's no point. I implemented MLVFS being able read MLV files that have been compressed (you can use mlv_dump to compress MLV files).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on March 01, 2015, 09:06:54 AM
@dmilligan:

thx for your help.

i tried some files with mlrawviewer with success - not all..
within mlvfs there are missing frames (i think these frames are from the sd card which do not find the way back to the source)

i am uploading the files now:
https://copy.com/ulsYvGWuzA80BCIz

in the directory 1 for example mlrawviewer works fine, but with mlvfs there are frames missing.
in the dir 2 the recording stopped and ml created many files from mlv to m08 on the cf card. on the sd card there are also many files from the same take m00,.. m08. i renamed the second charge to m09-m16 and within mlrawviewer this is also working well.


Thank you so much, swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 01, 2015, 08:39:52 PM
Delete your IDX files and let MLVFS re-create them and it works just fine (same with mlv_dump).

My guess what happened is that you tried to playback in camera or you tried to process the files without them named correctly first, which triggered the IDX file to be created, but this IDX files does not contain info about the missing frames in the spanned file. Then when you copied the spanned file and gave it the correct name, the index did not get rebuilt with the frames from the spanned file (because the IDX was already created).

Also, for the situation where you had a whole bunch of  m0X files because recording stopped, you can just delete all of those extra files (as long as the file is not larger than a couple MB, there's nothing it). Those are just created by mlv_rec when recording starts and the unused ones are normally cleaned up, but they don't get cleaned up if recording stops prematurely.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on March 01, 2015, 08:44:18 PM
oh. thank you a lot.
i will try that and will get back as soon as possilbe.

best wishes. swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on March 02, 2015, 10:34:23 PM
@dmilligan:

can you please give me an example of an mlv_dump compression terminal command which is working with mlvfs, cause i tried to compress a mlv but with no success.. (the file got smaller, nearly 45% but mlvfs can not convert it anymore.)
my command was:

mlv_dump -o Compressed.mlv -v -e -c -b 16 -l 9 -x <filenamexyz.mlv>

thank you very much.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 03, 2015, 12:20:18 AM
delta encode mode (-e switch) is not supported by MLVFS, also mlv_dump doesn't handle white and black level properly when changing bit depth (IMO), so if you use -b 16, the output black (and white) level from MLVFS will be wrong. mlv_dump will actually output the correct black level if you change bit depth, but that is because it assumes the black level is 14 bit (IOW, it assumes the original data is 14 bit). I don't like to make that assumption.

Here's a fix: https://bitbucket.org/dmilligan/magic-lantern/commits/a22302855d05c6cb587336943c8b5fb623a96bb4

The other thing is that LZMA is quite slow, and you'll never get realtime playback with it. That's why I also implemented the new compression scheme from here: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on March 03, 2015, 10:00:34 AM
hello dmilligan

thank you for the explanation.
so how can i compress them with the new option?
is there a simple batch possibility to compress - lets say a complete directory..

thank you very much.
sw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 03, 2015, 11:14:14 PM
so how can i compress them with the new option?
use this mlv_dump (https://bitbucket.org/dmilligan/mlvfs/downloads/mlv_dump)
like this
Code: [Select]
./mlv_dump -o <output.mlv> -c <input.mlv>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on March 04, 2015, 10:02:34 AM
Finally managed to record a timelapse with fullres silent pics (nothing spectacular, just something to try if it works), and I installed MLVFS to postprocess it. Wanted to try it for a long time, and this was the perfect occasion (since I don't really record raw videos anyway).

Already did some minor tweaks, to address the small installation difficulties I had:
https://bitbucket.org/dmilligan/mlvfs/pull-request/5/minor-tweaks-for-command-line-interface/diff

Problems found:
- an invalid MLV crashes MLVFS right away SOLVED
- the web interface doesn't work, browser gets stuck at "Request sent" SOLVED
- running cr2hdr in the virtual FS crashes after processing a few files (will try to fix, but I think I need more time)
- copying DNGs from the virtual FS to the real FS (same HDD drive, not SSD) is about twice as slow as a straight file copy (~20MB/s vs ~40MB/s, will look into it)
- exiftool can't change black level of the DNGs created by MLVFS (minor, probably some bad header)
- with spanned files, there is a black frame for each chunk (!), most likely the last frame, need to double-check (mlv_dump handles these frames properly)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Pyriphlegethon on March 06, 2015, 12:01:02 AM
null
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 08, 2015, 08:59:34 PM
- the web interface doesn't work, browser gets stuck at "Request sent", not sure why (tried to change port to 80000, no luck)
I think this might have to do with background mode and the threading model. Try running with -f. (I always use -f so I can debug and watch console output, so I've never noticed this issue). When I tried without -f on OSX, the webgui didn't work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on March 08, 2015, 11:10:59 PM
Correct. When I run it with -f, I get "Segmentation fault" right when I connect to a web browser on 127.0.0.1:8000.

That's the only message I get printed on the console.

Valgrind log:
Code: [Select]
==26190== Memcheck, a memory error detector
==26190== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==26190== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==26190== Command: ./mlvfs MLVFS -f --mlv-dir=/path/to/mlvs
==26190==
webgui start
webgui run
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
webgui poll
==26190== Thread 2:
==26190== Invalid read of size 8
==26190==    at 0x409059: xref_sort (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4098A7: make_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409B0C: build_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E23: force_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E79: get_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409F15: mlv_get_frame_count (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4375D5: webgui_generate_mlv_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x437F1E: webgui_generate_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4385A0: webgui_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42AE5A: call_user (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42FD78: call_request_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4309DF: call_request_handler_if_data_is_buffered (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==26190==
==26190==
==26190== Process terminating with default action of signal 11 (SIGSEGV)
==26190==  Access not within mapped region at address 0x0
==26190==    at 0x409059: xref_sort (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4098A7: make_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409B0C: build_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E23: force_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409E79: get_index (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x409F15: mlv_get_frame_count (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4375D5: webgui_generate_mlv_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x437F1E: webgui_generate_html (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4385A0: webgui_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42AE5A: call_user (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x42FD78: call_request_handler (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==    by 0x4309DF: call_request_handler_if_data_is_buffered (in /home/alex/src/ml/mlvfs/mlvfs/mlvfs/mlvfs)
==26190==  If you believe this happened as a result of a stack
==26190==  overflow in your program's main thread (unlikely but
==26190==  possible), you can try to increase the size of the
==26190==  main thread stack using the --main-stacksize= flag.
==26190==  The main thread stack size used in this run was 8388608.
==26190==
==26190== HEAP SUMMARY:
==26190==     in use at exit: 868,724 bytes in 105 blocks
==26190==   total heap usage: 1,096 allocs, 991 frees, 1,604,352 bytes allocated
==26190==
==26190== LEAK SUMMARY:
==26190==    definitely lost: 917 bytes in 34 blocks
==26190==    indirectly lost: 0 bytes in 0 blocks
==26190==      possibly lost: 1,088 bytes in 4 blocks
==26190==    still reachable: 866,719 bytes in 67 blocks
==26190==         suppressed: 0 bytes in 0 blocks
==26190== Rerun with --leak-check=full to see details of leaked memory
==26190==
==26190== For counts of detected and suppressed errors, rerun with: -v
==26190== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2)

edit: it was caused by an invalid MLV (created by latest fullres silent code). Deleting that MLV solved the crash.

edit2: fix commited, thanks dmilligan for write access.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on March 09, 2015, 08:46:06 PM
@dmilligan

would it be possible to compile a new mac version with the latest code changes at the bitbucket download section :)

thank you very much swinxx
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 10, 2015, 01:42:33 AM
done
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 10, 2015, 08:56:21 AM
Thanks for the update regarding the latest cr2hdr_20-bit from @a1ex into your app @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 10, 2015, 11:54:09 AM
huh? I have not updated the cr2hdr code. It is based on, but slightly different from the main cr2hdr code. The main differences are:
it doesn't use dcraw, the data comes internally straight from the MLV file
it doesn't use exiftool, dng header data is written by mlvfs itself
it's thread safe
some gcc specific language features were removed so it would compile with llvm
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 10, 2015, 06:33:16 PM
Okay gotcha -- my fault for assuming that you got the updated cr2hdr compiled for Mac by N/A that he released just few few days ago...

Will give it another test run w MLVFS with a quick Dual-ISO MLV footage (hopefully it won't come out pinkish again) although it still comes out in different kind of pink cast for non Dual-ISO MLV's as well.

I also updated the Adobe DNG to the latest firmware in case you were wondering.

Thanks again!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on March 10, 2015, 08:56:41 PM
I think dmilligan knows how to compile cr2hdr as well  ;)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on March 10, 2015, 10:34:06 PM
Ofc, I would assume so.

Doesn't hurt to ask.

Thx.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on March 13, 2015, 04:37:30 AM
Getting crazy vertical stripes and green tint occasionally with  the latest mac mlvfs. Doesn't happen all the time and not every mlv either, seems to happen when I first log on and mount my folder. Ejecting the drive, closing my browser and re-mounting it seems to fix it. Toggling vertical stripe correction and bad pixel fix doesn't fix it. Going to try to replicate the issue and see if its random or if certain mlv's and settings cause it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on March 13, 2015, 02:21:10 PM
Just want to share my "MLVFS server based converter"


So now, from my W7 PC i can put any MLV file in the mlv-in share and get the DNGs in the mlv-out folder share

Perfect !  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mario1000 on March 13, 2015, 04:34:43 PM
Just want to share my "MLVFS server based converter"

  • Server is clearos-community-6.6-x86_64
  • Installed FUSE (from clearos-core repo)
  • Compiled MLVFS from BitBucket
  • Created 2 shares (Flexshare) mlv-in and mlv-out
  • Created the file /etc/fuse.conf with user_allow_other argument
  • Finely lauched  mlvfs -f /var/flexshare/shares/magiclantern/mlv-out --mlv_dir=/var/flexshare/shares/magiclantern/mlv-in -o allow_other &

So now, from my W7 PC i can put any MLV file in the mlv-in share and get the DNGs in the mlv-out folder share

Perfect !  :)

Do I understand it right that this is a solution for windows users to use the latest update of dmilligan? If this is the case could you please provide a more detailed step by step guidance for users. What I mean is: does your solution needs linux or just windows... How do I setup Flexshare?

Sorry if this question seems to be a little stupid but I am a user and not a very skilled programmer.

Thanks in advance
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 13, 2015, 05:02:58 PM
You need a Linux box to do this (or a VM running Linux).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on March 13, 2015, 09:26:04 PM

....almost !!! grrrrrrrr ...for windows users!!!   :'(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on March 13, 2015, 09:35:50 PM
Arrinkiii: Running a VM is not that complicated. Try VMware Player.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 13, 2015, 09:46:17 PM
I've ran VMware on PC @ work since they don't have Mac's and MLVFS works quite well as per @dmilligan's suggestion.

However, I prefer to do it solely Mac's system alone.

*feedbacks*:

@dmilligan -- Based on my recent quick tests from earlier it seems I'm still getting strange pink frames from dual-ISO MLV files. Looks like it doesn't complete process during render before exporting from PP) Am I to export from the mount itself (with dual ISO and cs 2x2 enabled ON) or do I need to drag files onto system before I export it from PP?

Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 14, 2015, 02:22:35 PM
I'm not completely sure what you are asking. When I have time, I'll look into the dual ISO issue and see if I can duplicate it (I don't use dual ISO very much)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on March 14, 2015, 04:31:34 PM

Thank you guys, i will try the VMware in my PC =)) 

Two days a go i bought one MackBook Pro, second hand, dual-core processor, do you guys think is ok for the MLVFS  or is to old/slow ?? 

Thanks for all  :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 14, 2015, 04:43:31 PM
The speed of your storage media is much more important than CPU specs, when it comes to MLVFS. If it's a SSD then you should be good. I get realtime performance on a 2012 macbook air as long as I use a SSD. (MLVFS typically only uses around 30-70% of one core).

A separate consideration is the ability of the computer in regards to your NLE. Typically you'll need a decent GPU for realtime playback of raw material. You didn't specify how old your MBP is, so no way to know the specs. Your NLE will be much more demanding than MLVFS (MLVFS's job is pretty easy, all the real "work" it does is bit-unpacking).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on March 14, 2015, 04:53:24 PM

Thanks dmilligan, its the first time that i got mac... don't no sh** abou this but im learning. I must say that i bought this because i want to try the MLVFS. It don't have SSD but i will get one and change it.

My MBP is:

-HDD -> will change for SSD
-4 GB 1067 MHz DDR3 -> will change for 8GB
-NVIDIA GeForce 320M 256 MB
-2,4 GHz Intel Core 2 Duo   (will change for i5 or i7 one day lol )

I use Premiere, do you think with the extra memory (8GB total) plus the SSD i will get realtime playback in premiere?




Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dubzeebass on March 14, 2015, 09:11:03 PM
I've had to stop using this program because the drive disappears when I try to read the files in Resolve, though the mount points still shows up on my desktop. When I try to open it nothing is there, if I restart my iMac and then reload MLVFS it will work for a bit but then I'll try and view the files in the folder again and it will stop working. The only thing that works for a while is uninstalling OS X fuse and reinstalling it and then it stops working again after some short time. What logs do you need to troubleshoot?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 14, 2015, 11:57:54 PM
If you open up the "Console" app, on the left side there's a section "User Diagnostic Reports", there should be mlfvs crash reports there. Also, you can just go to ~/Library/Logs/DiagnosticReports
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 15, 2015, 02:18:54 AM
Congrats to @arrinkiiii for jumping onto the Mac Wagon. You'll never regret it!

As for the SSD -- it was rather easy to DIY on swapping HDD for SSD by watching YouTube videos online as I did with my 2006 MBP (upgraded to 360GB SSD as well 8GB memory ram) in which definitely made a difference and worth the effort! It also played a role in using MLVFS because I couldn't get smooth playback with the original specs in this particular MBP.

SSD have also gotten cheaper in prices lately respectively. Just make sure you have the proper tools as I had to get these in order to swap them properly. Most usually come with Clone-adapter Cable Kit.

Have fun! [emoji106]
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on March 15, 2015, 01:20:01 PM

Thanks  DeafEyeJedi =DD Yes, i saw it in youtube, also you can have 2 disks, removing the cd-rom and use a caddy for other disk, it will be more slow than the main one but it's good for data/stuff. Need money  :P

-Clone adapter cable? For cloning the disk? i Think will install all again =))

With your BMP (ssd + 8gb) can't you have real play in Premiere?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on March 16, 2015, 03:54:37 PM
Quote
Do I understand it right that this is a solution for windows users to use the latest update of dmilligan? If this is the case could you please provide a more detailed step by step guidance for users. What I mean is: does your solution needs linux or just windows... How do I setup Flexshare?

Sorry if this question seems to be a little stupid but I am a user and not a very skilled programmer.

Thanks in advance

Sorry to be late to answer,

ClearOS is a SOHO linux server. I use it for my backup, dlna, vpn server, and so on.
With this distro, it's possible to install fuse and to compile mlvfs without problems.
Flexshare is a file sharing system included in ClearOS (http://www.clearcenter.com/support/documentation/user_guide/flexshare)

You can install it on a small pc or a VM like VirtualBox. I have tried the same (without Flexshare) in Xubuntu 14.04 with success.

So learn to install linux in a VM (it's easy !) and i can help you with mlvfs.

Tips : VM : https://www.virtualbox.org/ Xubuntu : http://xubuntu.org/getxubuntu/

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mario1000 on March 16, 2015, 06:40:43 PM
Sorry to be late to answer,

ClearOS is a SOHO linux server. I use it for my backup, dlna, vpn server, and so on.
With this distro, it's possible to install fuse and to compile mlvfs without problems.
Flexshare is a file sharing system included in ClearOS (http://www.clearcenter.com/support/documentation/user_guide/flexshare)

You can install it on a small pc or a VM like VirtualBox. I have tried the same (without Flexshare) in Xubuntu 14.04 with success.

So learn to install linux in a VM (it's easy !) and i can help you with mlvfs.

Tips : VM : https://www.virtualbox.org/ Xubuntu : http://xubuntu.org/getxubuntu/



Hi vertigopix,

thank you very much up to this point. I downloaded and installed VM and Xubuntu as you have suggested and everthing runs without any problem. What are the next steps to install mlvfs?

Thanks in advance!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 16, 2015, 10:55:01 PM
I'm not completely sure what you are asking. When I have time, I'll look into the dual ISO issue and see if I can duplicate it (I don't use dual ISO very much)

Here's to what I was referring to earlier regarding Dual-ISO (and regular MLV as well) showing pinkish effect after being exported.

MLVFS:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7607%2F16837641825_bdf23b40b2_n.jpg&hash=dbdd3ef042c19dc85a8ba828ca6f1c0d) (https://flic.kr/p/rDTjip)

Dual-ISO MLV (Dual-ISO + cxs 2x2 enabled ON) via MLVFS > PP:
https://vimeo.com/122361687 (https://vimeo.com/122361687)

Dual-ISO MLV (@Danne's cr2hdr-r (http://www.magiclantern.fm/forum/index.php?topic=13512.msg130562#msg130562cr2hdr-r) ):
https://vimeo.com/122219543 (https://vimeo.com/122219543)

Non-Dual-ISO MLVFS > PP:
https://vimeo.com/122361688 (https://vimeo.com/122361688)

Non-Dual-ISO MLVFS > AE (graded w ACR):
https://vimeo.com/122363848 (https://vimeo.com/122363848)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2015, 02:05:33 AM
I can't reproduce. My dual ISO clips look fine in Premiere. Does it only happen when you export? (Do they look correct just viewing them?) Can you share the original MLV file for a short clip that exhibits the problem.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: marekk on March 17, 2015, 02:07:59 PM
@dmilligan - is there a possibility to add option to preload some number of frames not only from current file but also from next available file ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2015, 02:16:32 PM
how would mlvfs be able to figure out what the "next" file is?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: marekk on March 17, 2015, 04:08:41 PM
In current version there is an option to preload some frames. When I add all MLV files to Resolve timeline for grading and exporting to prores current CDNG is played smoothly. At the beginning of next file there is a 1-2 seconds period of interruption. So.. when MLVFS is currently working with FILE_1.MLV, we can preload some frames from FILE_2.MLV. Switch from FILE_1.MLV to FILE_2.MLV can be done smoothly. It will work only if files in timeline are sorted by name.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2015, 04:44:14 PM
there is a 1-2 seconds period of interruption
There must be something else going on, and I don't think what you are asking for will help the situation. Processing a frame should be on the order of 40ms (or less for realtime playback), so if it's taking 1-2 seconds, there's something else going on, that pre-fetching is not going to help. The solution is to figure out what is actually causing the delay, and fixing that.

The pre-fetch is really only useful for when heavy processing options (read: dual ISO) are enabled so that more CPU cores can be utilized. It can actually hurt performance, esp. if you use too high of a number of frames. If you're not at 100% CPU utilization for MLVFS for a single core, then I wouldn't use it at all, it's likely to only make things worse.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on March 17, 2015, 09:41:11 PM
Any way to add an option to webgui to disable gif previews? It's not too bad when there are less than 50 GB of MLV's to preview, but working with 100+ GB of MLV's and a not-so-powerful gpu can slow things down pretty significantly.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 18, 2015, 02:28:23 AM
@N/A: That's a Great idea regarding disabling gif previews!

@dmilligan: Here's the Dual-ISO MLV file from 5D3 per your request.

MLV (https://www.dropbox.com/s/x9muqqodosmz25b/17980007.MLV?dl=0MLV)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on March 18, 2015, 10:29:12 AM
@mario1000

Open Terminal Emulator and type :

Code: [Select]
sudo apt-get install git
sudo apt-get install libfuse-dev

git clone https://bitbucket.org/dmilligan/mlvfs

cd mlvfs
cd mlvfs   (yes, 2 times !)

make clean
make

sudo cp mlvfs /usr/bin

Create the fuse.conf file and add the 'user_allow_other' value into it

Code: [Select]
echo "user_allow_other" | sudo tee -a /etc/fuse.conf

If everything is ok, you can create 2 folders (eg. mlv-in and mlv-out)

Code: [Select]
mkdir /home/<YOUR_NAME_HERE>/mlv-in
mkdir /home/<YOUR_NAME_HERE>/mlv-out

and now, you can launch mlvfs !

Code: [Select]
mlvfs -f /home/<YOUR_NAME_HERE>/mlv-out --mlv_dir=/home/<YOUR_NAME_HERE>/mlv-in -o allow_other
Don't close the terminal, open a browser and type : 'http://127.0.0.1:8000'

Magic as a lantern !  :)

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mario1000 on March 18, 2015, 12:16:53 PM
Hurra! It works! Thanks a lot for your help!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on March 18, 2015, 02:23:31 PM
and when a new version exist, go to mlvfs base directory and type
Code: [Select]
git pull
after this
Code: [Select]
make clean
make

And you can copy the new version to /usr/bin (you need to kill the process before updating)

How to kill  ;)
Code: [Select]
ps -e | grep mlvfs
sudo kill <process number>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 19, 2015, 02:06:45 AM
@dmilligan: Here's the Dual-ISO MLV file from 5D3 per your request
Thanks, I do see some issues, not sure what the cause is yet, will have to investigate further.

How to kill  ;)
Code: [Select]
ps -e | grep mlvfs
sudo kill <process number>
A better way would be
Code: [Select]
sudo fusermount -u /home/<YOUR_NAME_HERE>/mlv-out
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: vertigopix on March 19, 2015, 07:34:45 AM
@dmilligan,

You're right, my mistake... ;-)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: arrinkiiii on March 20, 2015, 01:20:32 PM

I must write this... One week ago i bought 2nd hand MacBook Pro  dual-core, five days ago i have change for a 2nd hand MBP i5, from my girlfriend lol, since she just use it for internet, email, etc... 

I thought that being an i5 and a Mac it would have a disk that work at 7200rpm but no, it come with a 5400rpm, cant understand why they didn't do that... for a small extra of money you could have a better disk, at least 7200rpm.

I format the disk that have, 500GB and made a partition, so 2 x 250GB. Install Yosemite OS OXFUSE and MLVFS, also Adobe CC and an CUDA 5.0 drive for work good with adobe.

Put some ML files in a folder on the disk, Ctrl+Click and chose MLVFS for that folder. In that moment it prompt to chose one folder where he will release/convert the ML files to DNG's. Immediately he open a web page with the settings that you want to change.The end!!!  So simple as that. Im amazing!!! 

I threw some DNG's to Premiere and it works good. In my MBP it skip some, a few... frames. For sure that is because of the 5400rpm disk.

Just my thoughts: Wend im in the web page it would be better if the previews didn't star automatically. It's possible to have bigger view of the preview windows? Yes, you can. Just click in the file name of the MLV and you got all the meta data regarding that file and with bigger preview window. Neat =)

I think this app is just for sidemont/convert the ML files and use it with Premiere, Resolve, etc... but would be good if could convert for ProRes and with a LUT of chose. I use MlViewer and i love it, i think each one of them have there one way/purpose of working with MLV's. I like to work with DNG's but for some works ProRes is more then fine.

Saying this, i need to get one SSD and more memory to my MBP lol  Going to read this threat all again for see if i miss something.

Resume: It works good with Premiere. Thans for this =))

 
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on March 20, 2015, 02:55:16 PM
ProRes + LUTS = cr2hdr-r
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johnwangphoto on March 20, 2015, 11:05:00 PM
After I converted MLV files with MLVFS the dng files are only available in the MLVFS Mount folder for a little while then they all disappear so i have to do the conversion again. Is it normal?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 20, 2015, 11:50:54 PM
If the mount disappears, that means MLVFS crashed. There should be a log in ~/Library/Logs/DiagnosticReports. Or you can look in the 'Console.app' (located in Applications/Utilities).

Please send the log.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 21, 2015, 08:28:45 PM
Any way to add an option to webgui to disable gif previews?
Done. Previews don't load until you click "Load" (or unless you click on a specific MLV file).
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 21, 2015, 08:36:55 PM
Thanks for the Great feature modification regarding previews @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on March 22, 2015, 12:49:24 AM
Done. Previews don't load until you click "Load" (or unless you click on a specific MLV file).
Awesome, thank you. Got a shoot next weekend 5 hours away from home, having a fast, stable and reliable program like MLVFS is a lifesaver. Much appreciated.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on March 22, 2015, 08:13:47 PM
Hey David. Been researching a little on color matrix stuff. Question. I see forward matrix 1 and 2 are static for 5d mark 3. Where did you get these numbers? Also, how can numbers for other cams be found?
ah, see them now, part of the dng.c structure. Thanks!

WOuld you know if the model tags are the same in metadata as for "Canon 5d mark 3"? "Canon Eos 7D" and so on? Plan on implementing with exiftool but of course don,t have all the cameras in my possession. Need to differ them accordingly with camera name.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs9.postimg.org%2Fijimocf17%2FScreen_Shot_2015_03_22_at_20_17_13.jpg&hash=a473f654dbbb8382d7e2d2417f53fe77) (http://postimg.org/image/ijimocf17/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 22, 2015, 10:05:17 PM
Andy600 pulled them from Adobe DNG converter. All you need to do is run a DNG through Adobe DNG converter without those tags and it will add them. Though, you really don't even need to do that, they're all hard-coded in the MLVFS source code.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on March 22, 2015, 10:17:51 PM
Yeah, I got them from looking at your file. Your workflow is benchmark, still I got my own little cr2hdr-r project going on so would like to test some stuff out.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 27, 2015, 12:32:51 AM
FYI: http://www.magiclantern.fm/forum/index.php?topic=14803.msg143311#msg143311

(I've updated this OP with that utility)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: TKF Productions on March 28, 2015, 07:07:13 PM
I'm trying to convert the .mlv file to something useable.  I downloaded the FUSE MLV to cDNG converter and everything worked fine and I got the DNG files, however, when I tried to import them into After Effects I get the following error:

"After Effects error: the file format module could not parse the file. ( 45 :: 35 )

Anybody know how to fix this or how to get the .MLV files properly converted?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 28, 2015, 10:45:08 PM
Just downloaded and updated to the latest MLVFS -- after selecting a mount folder on desktop and once the web host site opens up it says "Local Host not found" I've seen this issue before but don't remember how to get around it? (I've tried both Safari and Firefox)

I tried restarting MBP and reinstalled MLVFS to no avail as well as deleting .IDX files... any suggestions?

*edit*

Tried again after hard reboot (not sure if I needed to do this on MBP) but it kind of helped a little... once the web host shows up with proper details but the mount folder doesn't get 'mounted' for some reason?

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7633%2F16959037581_f7c336a7b3_n.jpg&hash=657f05368b3511300491915d55eb6269) (https://flic.kr/p/rQBv2F)

hence the 'Mount Folder' highlighted on the desktop screen behind it... strange?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on March 29, 2015, 10:36:06 AM
*update* MLVFS works fine on my Mac Mini but not MBP.

Will look into it some more tmw!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on March 29, 2015, 08:14:39 PM
@TKF Productions,
Can you send me a sample file?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 30, 2015, 08:57:33 PM
Regarding AE (SmartImport2) with MLVFS... (I set the project as 25fps but then when I try to render them it says 30fps and I have to manually custom the FPS to 25 for each before I export in AE -- is this normal?)

https://vimeo.com/123651925 (https://vimeo.com/123651925)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on March 30, 2015, 09:35:38 PM
DeafEyeJedi,

Did you set AE to default to 25 fps for image sequences?
See: http://www.magiclantern.fm/forum/index.php?topic=13292.msg128368#msg128368 (http://www.magiclantern.fm/forum/index.php?topic=13292.msg128368#msg128368)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 30, 2015, 10:02:56 PM
I'm not familiar with that script, but seems to me like it would be an issue with the script not MLVFS.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 30, 2015, 10:17:01 PM
@Frank7D:

Thanks for the inputs. Will look into it.

@dmilligan:

so you wouldn't recommend using MLVFS and AE together then? (even without the SmartImport2 -- MLVFS alone won't bring Audio in AE, right?)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on March 30, 2015, 10:30:13 PM
MLVFS can be used with AE and will have audio; it's easy if you use the script. That's what I do.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 30, 2015, 10:46:19 PM
You speaking of the SmartImport2 script?

Then how come the audio seems strange in this footage I put together after rendering them in AE within MLVFS?

Holden's Key (http://www.magiclantern.fm/forum/index.php?topic=14840.msg143620#msg143620Holden's Key)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: TKF Productions on March 30, 2015, 10:55:13 PM
@TKF Productions,
Can you send me a sample file?

Thanks for offering to take a look.  I will send you a sample file tonight.  What would be the best way to send to you (doesn't looks like I can attach a file through PM)?  Also, do you just want one of the DNG files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on March 30, 2015, 10:57:37 PM
DeafEyeJedi,

I'm not at a place where I can listen to the audio right now. Can you describe the issue? Actually, maybe we should have this discussion here: http://www.magiclantern.fm/forum/index.php?topic=12422.msg119832#msg119832 (http://www.magiclantern.fm/forum/index.php?topic=12422.msg119832#msg119832)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on March 31, 2015, 12:31:38 AM
@TKF Productions,
Any file sharing service like dropbox, google drive, etc. I might could diagnose the issue from just a DNG, but I might need to have an actual MLV, and it will make things faster anyway. It can a short/small as you like as long as it exhibits the issue.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: anderSF on April 02, 2015, 06:00:05 PM
Hi!

I'm having a problem with MLVFS where footage shot with a shutter speed of 1/100 (at night, with tungsten lighting) shows up in ACR as 1/110. This seems to cause flicker in the shot, weirdly enough. One should think as long as the footage was actually shot at 1/100, the metadata shouldn't matter.... But that does not seem to be the case, because when I conform the shots through MlRawViewer (which reads the shutter speed correctly), they come out flicker free.

Any ideas?

PS: I'm not doing highlight-adjustments or anything like that in ACR. Nothing that could cause flicker.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 02, 2015, 08:48:30 PM
ACR can cause flicker on it's own, without any adjustments. I really doubt it's the metadata. If you want to test it, you can copy the DNGs out of the mount and change the metadata with exiftool.

I'd also be interested to see the output of exiftool -v for DNGs from MLVFS vs. MlRawViewer, to figure out why it's 1/110 instead of 1/100. The output of mlv_dump -v would be helpful too.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: anderSF on April 02, 2015, 09:37:01 PM
Thanks for the quick reply! But I haven't really used any non-GUI software before, so I would have to look into that first....
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 02, 2015, 10:26:25 PM
okay, then just send me DNGs from both, and/or the mlv file
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 03, 2015, 12:38:32 AM
*FEEDBACKS*

From my previous post
*update* MLVFS works fine on my Mac Mini but not MBP.

Will look into it some more tmw!

After all, I still don't understand how come MLVFS all of a sudden doesn't work on this MBP (when it previously has with no problems in the past!) still even after deleting all .IDX files...
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7618%2F16827741379_b7861db4f7_n.jpg&hash=621fb91badab8b7f94e66d023353cc78) (https://flic.kr/p/rD1zf4)

Strange that even with WebHost not working properly it still spits out 2-3 .IDX files in the Mount Folder but not for all... not sure if noting this would help?

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7609%2F17012741162_4cc5b6a125_n.jpg&hash=ff23fb33c59743e1a24b3a7ee0de1909) (https://flic.kr/p/rVmKeu)

MBP Specs:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm8.staticflickr.com%2F7585%2F17012517382_ed19db6180_n.jpg&hash=3096a2faf2598e605d44b01c30863bc1) (https://flic.kr/p/rVkAHd)


Hoping there's something that I may have miss out on solving this issue...

Thanks!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on April 03, 2015, 01:24:43 AM
So, does the mount still work okay? Just the webgui doesn't come up? Can you try running MLVFS from the command line? You can also try using a different port.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 03, 2015, 02:42:48 AM
No, the mount does NOT show up on desktop at all on MBP as well as the webgui -- I'll try running MLVFS from command line.

I think you have the directions on how to use Command Line on your first post, correct?

Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 03, 2015, 10:05:41 PM
Hi all! I have the strange problem... I made some changes with cDNG sequence in ACR After Effects, BUT! I cant see any changes with any frames in Virtual folder (mlv folder). I want to make Color Correction in AE, then unmount and edit corrected MLV video through direct import (with Drastic help) in Premiere with color corrected frames. What im doing wrong? Why i cant see any ACR changes (1st frame) in windows explorer (or when i use ACDSEE). Adobe dng codec 2.0 is installed...

Windows 8.1 64. Thanks a lot!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on April 04, 2015, 12:57:38 AM
ACR tries to write it's settings into the DNGs as metadata or into sidecar xmp files. The mount is going to be read only, so that won't work. I've added support for allowing things to write files in the mount in the Mac/Linux version, so ACR can create and save its xmp sidecar files. But the Windows version is way behind and it may not even be technically possible to implement something like that.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 04, 2015, 01:54:16 AM
But i thought ACR uses database or xmp. I switched to DB. It must be not write any data to dng
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on April 04, 2015, 03:43:03 AM
The windows/pismo file mount version of MLVFS does allow xmp sidecar files to be saved (in .MLD folders), so your ACR changes will be saved.
However, programs other than ACR may not be able to read these files.
Here's a thread with an overview of one possible workflow: http://www.magiclantern.fm/forum/index.php?topic=13354.0 (http://www.magiclantern.fm/forum/index.php?topic=13354.0)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 04, 2015, 04:25:10 AM
I dont know how, but previously (before reinstall os) all changes were in 1st dng file. With xmp... Its good, but for why i need it? I cant apply xmp to mlv without mounting. I just want to colorize mlv with ACR and work without recoding to Intermediate format (mxf, prores e.t.c). 1. Edit with arc (wb). 2. Edit corrected mlv direct in premiere. Is it possible?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on April 04, 2015, 04:36:45 AM
I don't know how you can colorize the mlv with ACR and then edit it without mounting it. I think if you're using ACR you need to transcode to something else in order to keep the color. You can edit mlv (mounted to virtual CDNGs) directly in Premiere CC but then you can't use ACR (I believe).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 04, 2015, 06:19:31 AM
Yeah, sorry I misread the original question. You cannot use ACR settings with anything other than ACR. Premiere will not apply ACR settings, because Premiere uses a different raw engine that is not ACR. And MLVFS is certainly not capable of applying ACR settings back into a MLV file (which is impossible anyway). If you want to use ACR, you must render out to an intermediate format from either AE or photoshop (something that uses ACR).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 04, 2015, 06:27:10 AM
With a help of Drastic software i can direct edit mlv without transcoding. But the question is - how to colorize dng inside mlv and get the corrected video even after unmount? I mean mlv video must be colorized after manipulations with dng inside. All we need - to have abilities to write dngs. This workflow can save 80% of time...
Edited: somehow couple month ago i got working changes in mlv mounted folder (i changed color and exposure). First frame was exactly like i made in ACR. ...Confused....
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on April 04, 2015, 06:45:38 AM
I don't use Adobe CC so not sure if this would work, but have you tried:

1. Mount MLV with MLVFS.
2. Edit (virtual) CDNG sequences in Premiere CC, then save project.
3. Open Premiere project in After Effects CC, then do a "replace footage" for each clip so that they open in ACR, where you can color them.

This would mean coloring the footage after it is edited, rather than before.
And once you're in After Effects, you can do a lot of other cool stuff too, of course.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 04, 2015, 06:50:58 AM
... For each clip... Its about 9500 clips... (7 lovestorys and 2 wedding videos + 2 music videos)... Thats why i need more easy and faster workflow... Or buy raw camcorder :(

Ok, anyway, thank you for the tips!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on April 04, 2015, 06:52:50 AM
But...you still have to color every individual clip if you use ACR, no matter how you do it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: eriklieders on April 04, 2015, 06:57:50 AM
Yes. But without transcoding. It saves a lot of time.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on April 04, 2015, 07:22:12 AM
If my 3-step proposed workflow works (and I don't know for sure that it does) then there is no transcoding involved.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 04, 2015, 07:16:10 PM
But the question is - how to colorize dng inside mlv and get the corrected video even after unmount?
Impossible. As I have already stated. You cannot apply ACR settings back into an MLV file. MLVFS gives the impression that an MLV file is just composed of a bunch of DNGs. That is not the case. MLVFS transcodes from the MLV format to DNG on the fly, making it appear like there are just a bunch of DNGs inside. There are not actually a bunch of DNGs inside. The MLV format does not support any sort of method of storing ACR metadata inside.

I don't really know anything about how exactly drastic's software works, but I don't think it uses ACR, so even if you could put ACR metadata inside an MLV file (which you can't), your plugin wouldn't be able to use it anyway.

Like I said previously, the only software capable of applying ACR settings is ACR itself. So if you want to use ACR, you HAVE to transcode out of AE. There is no way around this.

If this is too slow, then I suggest dropping ACR from your workflow. It is not intended for video anyway, and there are much better tools out there for video CC, that are actually intended for that purpose.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 04, 2015, 08:50:04 PM
MLVFS > AE (ACR) > NLE (FCPX, PP, etc) does your body good!

@eriklieders: you can try saving a preset (.xmp file) with your first file and then just paste them for the remaining files within ACR via AE.

(if you keep the exposure/WB the same during a shoot which you normally should when possible) then this workflow should suit you well enough.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on April 05, 2015, 08:11:41 AM
For those of us using the PISMO file mount utility in Windows for our MLVS files:  It works fine for normal MLVS videos but won't mount/open an MLVS file created via FRSP Silent_Pics.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on April 05, 2015, 01:56:14 PM
@dmilligan
i updated/added the 500D with DNG-Data. line 77 io.cs raw2cdng (https://bitbucket.org/chmee/raw2cdng/src/b67ee70dc91aa7143df66f00bb770156b6bae8a2/csharp2010_code/raw2cdng_v2/io.cs?at=master)

regards chmee
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on April 05, 2015, 10:52:17 PM
Thanks chmee.

@Ottoga,
Yeah the Windows port is pretty far behind because there's no developer with Windows that is keeping it up to date. The Mac/Linux version was fixed to work with FRSP a while ago.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on April 06, 2015, 05:35:58 AM
@dmilligan

I have an old Pentium 3 at home. Might have to dust it off and make a Linux box out of it. (slightly off topic.... any recommended distro?)

The good news for now is that I managed to extract the DNG's using MLV_Dump.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 06, 2015, 06:02:48 AM
I sort of doubt a pentium 3 could handle MLVFS, esp. the full res stuff (just 1080p24 on my 2ghz core i5 uses 60-70% cpu). You would have much better performance with a VM. There are some descriptions of setting this up a couple pages back in this thread.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on April 06, 2015, 11:51:21 AM
@dmilligan,

Thanks.  I'll look into the vm option. never set one up before, so a first time for everything.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: janoschsimon on April 22, 2015, 11:27:55 PM
will the 70D matrix coefficients be added in dng.c? https://bitbucket.org/dmilligan/mlvfs/src/d961f752f326856081a42f294e22ffca301df875/mlvfs/dng.c?at=master

cheers and thx janosch :-)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on April 22, 2015, 11:46:44 PM
If somebody tells me what the coefficients are, or sends me a DNG from a 70D with the matrices in it, I'll do it.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 23, 2015, 02:01:10 AM
Also got some 70D footage to test out with MLVFS -- first I need to know if you already have a copy of DNG from 70D?

If not, I can send you one... Dual-ISO or non Dual-ISO or Cdng?

Just let me know...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on April 23, 2015, 07:48:52 AM
Here,s a dng
https://drive.google.com/file/d/0B4tCJMlOYfirbndxRkt0cy0tX0U/view?usp=sharing

 Got it from here. They share test cr2 files for most if not all canon cameras.
http://www.photographyblog.com/previews/canon_eos_70d_photos/
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: janoschsimon on April 23, 2015, 08:06:55 AM
ah you need a photo dng or one from the video cdng? :-)

cheers janosch
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on April 23, 2015, 08:13:26 PM
I have started to experience a problem with MLVFS.

MLVFS has worked great for a few months. Now every so often when I go to load a large folder (500GB) of MLV for us in Davinci it just won't load. The MLFVS folder keeps ejecting its self, or Davinci doesn't see that the files are in davinci mode and instead see's empty folders. I end up restarting my computer about 10 times, reloading the folder over and over and eventually the problem is solved. It's then permanently solved and I can load and unload the footage repeatedly no problem. It seems to happen when have round tripped back to Davinci having not loaded the MLFVS for a while.

Does anyone else have this problem? Does anyone know how to overcome it in a timely manner?

Cheers,

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on April 23, 2015, 08:40:31 PM
are there any MLVFS logs in ~/Library/Logs/DiagnosticReports?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: chmee on April 24, 2015, 03:17:21 PM
@dmilligan here (https://bitbucket.org/chmee/raw2cdng/src/ef55e16df51fc66249f2215d0fffe269f0807766/csharp2013_code/raw2cdng_v2/io.cs?at=master) you go. 70D is in. (oops, i'm wondering why they re as 6D, have to doublecheck)

regards
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 25, 2015, 10:56:41 PM
@dmilligan:

Any progress towards an update for MLVFS to be able to read 70D files yet?

I have a bunch of small few seconds test clips (MLV/RAW) in ISO 800 and ISO 1600 (where you get to see a lot of hot pixels I suppose if that would help you, just let me know!

Thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on April 28, 2015, 04:53:23 PM
Quote
@dmilligan here you go. 70D is in. (oops, i'm wondering why they re as 6D, have to doublecheck)

regards

Viewing exif on two different 70D raw-dng files seems 6D and 70D share the exact same matrix specification so the numbers seems right.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on April 28, 2015, 05:56:01 PM
Is that to be expected especially coming from Canon?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on April 28, 2015, 08:30:46 PM
On further notice seems relying on dcraw color matrix 1 is recommended for now. The matrix reported as from 6D to th 70D are wrong and will give greenish pics. Don,t think adobe software is updated with this cam yet.

*updated greenish prores files. Not the dng files. This needs more investigation.

*update. Seems I could be wrong on the prores transcoding. I put the wrong numbers in. The chmee matrices might be right.

*update- Now also ProRes transcoding in cr2hdr-r seems to work. Conclusion. Chmee 70D/6D matrices seems right. Only did brief tests.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on April 29, 2015, 09:50:48 AM
I can confirm that regarding 70D/6D matrices seems similar since the ProRes files came out well from @Danne's latest cr2hdr-r for both MLV & RAW.

Looking forward to using these files for MLVFS as well...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on May 03, 2015, 04:57:43 AM
I have updated MLVFS with the 70D matricies (they are the same as 6D)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on May 03, 2015, 06:46:19 AM
Sweeeeeet can't wait to test it out -- Thanks, David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: janoschsimon on May 03, 2015, 09:56:17 AM
thx david will give it a try later :-)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: masc on May 05, 2015, 06:34:18 PM
I have started to experience a problem with MLVFS.

MLVFS has worked great for a few months. Now every so often when I go to load a large folder (500GB) of MLV for us in Davinci it just won't load. The MLFVS folder keeps ejecting its self, or Davinci doesn't see that the files are in davinci mode and instead see's empty folders. I end up restarting my computer about 10 times, reloading the folder over and over and eventually the problem is solved. It's then permanently solved and I can load and unload the footage repeatedly no problem. It seems to happen when have round tripped back to Davinci having not loaded the MLFVS for a while.

Does anyone else have this problem? Does anyone know how to overcome it in a timely manner?

I think I have the same problem. For smaller projects around 50GB it works great.

Now I have a bigger project with around 400GB (around 350 mlv files). Mostly the folder (mlvfs drive) mounts first, but in the browser I can't see a connection to the page which should pop up. If I click on the folder, it will be away up to the next system restart - no way to eject it or to access.

If I take only a few files of the project, around 150GB (100 mlv files) all seems to work fine. With only some more files the problem begins: the page will first be shown in the browser and I can access the mounted folder, after some minutes the page is away and the folder too.

/Library/Logs/DiagnosticReports is empty.

Thanks for your help in advance!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on May 05, 2015, 07:08:18 PM
It's probably the index files.

MLVFS creates indexes of all your MLV files (if they don't already exist). These files are named the same as your MLV file, but end in .IDX. This indexing has to happen before MLVFS can give any information about a particular MLV file (this is because metadata and/or frames can appear anywhere in an MLV file, and the MLV format does not require that a frame count be in the first header).

If you put 500GB of files all in the same directory and then mount that directory with MLVFS, when the webgui comes up it wants to display information about all those files (same thing if you browse to the folder). So it tries to index all of them. For 500GB this will take a while. My guess is your issue has something to do with this (maybe you just didn't wait long enough, or there was some kind of timeout that cause MLVFS to crash).

Watch the folder where your original MLV files are located when you launch MLVFS. You should see these files starting to get created (if they are not already existing). Also keep an eye on disk, CPU, and memory usage of MLVFS. Then let me know what is going on.

The issue also could be related to corrupt index files. I would suggest you delete all the .IDX files before trying this. Then watch what happens as MLVFS creates them back.

One solution is to split up your files into separate subdirectories, that way MLVFS doesn't try to index all of them immediately all at once (it will only try to index files in a particular sub-directory if you browse to it).

Once all the index files are created, you won't have to wait anymore. My guess is this is why @silenttapes eventually got it to work. Each time he tried, MLVFS created a couple more index files, and eventually all of them got created.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: masc on May 05, 2015, 08:53:33 PM
Thank you for your fast answer!
What I did now: I deleted all IDX files. Then I mounted again. As you said it builds the IDX files again. And you are right - I was to fast - that takes for this amount of data on my hardware around 15min (the drive is slow, I know...)! If I click in this time on any option on the webgui, mlvfs stops immediatelly creating IDX files. So I closed my browser when it pops up to not disturb mlvfs. RAM und CPU consumption is very low. Only <20MB RAM and <25% CPU. After all IDX are created I tried to start the browser watching the webgui... the gui is not there (server not found). Clicking on the mounted drive makes it away... mlvfs is listed no longer in the OSX activity window then.
After a system restart I tried mounting again. I waited until the CPU usage of mlvfs is listed with 0%. Opening the browser and going onto the webgui brings mlvfs to 100% CPU usage (on one core) and around 110MB RAM. There seems to be no access on the hard drive where the MLV files are located, the disk LED is not blinking. I can access the mounted folder and I see the DNGs, but I can't access the webgui. I am waiting now another 20min and nothing changes... the browser says "waiting for localhost...".
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: masc on May 05, 2015, 11:55:57 PM
Now I tried your second tip - subfolders. This seems to work. Thanks for your help! With that little workaround I can live. :-)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on May 06, 2015, 12:33:42 AM
I have updated MLVFS with the 70D matricies (they are the same as 6D)

Thanks again for the quick update for 70D color materices.

However, on my MBP it opens up web gui just fine and shows info.

But it the mount folder itself won't show up (even after spitting out the .IDX files)

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm9.staticflickr.com%2F8806%2F16761752664_7c7f1ecc8c_n.jpg&hash=fe6e7ddad673677156e625aa92290225) (https://flic.kr/p/rxbn6A)

This was done on my 2006 MBP... any ideas?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on May 13, 2015, 10:21:03 AM
Hi dmilligan! Tried out your mlv_dump for archiving purposes. Very nice indeed. However. When extracting dng files both compressed and non compressed files it doesn,t add white and black level info? Could this be fixed? What I really like is that you rewrote the dng tiff tags completely for mlv to dng files.

Here is a file developed straight from an mlv file with your mlv_dump from first post.

https://drive.google.com/file/d/0B4tCJMlOYfirbXMxaVRCV0RiT1k/view?usp=sharing

*tried adding white/black level with exiftool. Doesn,t work. Is exiftool not working/writing to cdng files?

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on May 19, 2015, 09:32:09 AM
*feedbacks*

Some few days ago, I decided to run your app with an old footage I shot from 3-4 months ago during a BTS shoot...

I enabled 20-bit Dual-ISO in web host config (although not all files were shot in this format) so is it problematic for me to have both normal and DI's files together in the mix? Do they need to be run seprately?

Also I noticed while rendering in AE CC 12.1, seems to be really slow (even for regular MLV's) is this normal?

It's literally almost like over an hour for each 13-15 second clips?

https://vimeo.com/128229218 (https://vimeo.com/128229218)

https://vimeo.com/128229544 (https://vimeo.com/128229544)

Thanks again, @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on May 19, 2015, 12:50:20 PM
Also I noticed while rendering in AE CC 12.1, seems to be really slow (even for regular MLV's) is this normal?
There is no metadata in the MLV that indicates whether or not the file is dual ISO (IMO, there should be). So even on the non-dual ISO files we have to actually try to do a dual-ISO conversion and fail to figure it out, b/c we don't know if they are dual-ISO or not and there is no other way to figure it out.

I suppose there are some things I could do to speed things up a bit for the non-dual-ISO case, perhaps keep track of if frames in an MLV file fail to convert and don't try again once one or two frames in a file fail.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on May 19, 2015, 03:37:47 PM
Can you use an if condition? I use it when differing dual iso from non dual iso searching for the line "'Edge-directed interpolation..." when doing a test conversion. That way it will only try to (test)convert one file instead of testing all files for dual iso. Sending out the process of the first file to a log file with tee command.


Code: [Select]
find . -name ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -print0 | xargs -0 /usr/bin/cr2hdr20bit | tee /tmp/magic_l/cr2hdr20bit_MLV.log ;
if grep -q 'Edge-directed interpolation...' tee /tmp/magic_l/cr2hdr20bit_MLV.log
then
echo yes
else
something else
fi

*
Quote
I suppose there are some things I could do to speed things up a bit for the non-dual-ISO case, perhaps keep track of if frames in an MLV file fail to convert and don't try again once one or two frames in a file fail.

You,re already on it I see :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ddelreal on May 19, 2015, 06:12:32 PM
Thanks so much for this, really like using it for my 6D files. Question though: with regular raw files using raw2dng (on Mac) ACR opens the converted files very flat looking and I can adjust accordingly or I can open in Resolve and set to BMD and do the same thing. Works pretty much the same in Resolve when using MLV converted to CDNG using MLVFS but if I use ACR, it looks like REC 709 already. Is that correct?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on May 20, 2015, 08:19:31 AM
Just got home from work and I find that MLVFS somehow froze up after spending the last few days rendering the shit outta my Mac Mini (all good the usual stuff) and apparently it thinks it ran out of diskspace but I looked up and see that I have 36 GB of free space.

Also Dual-ISO footage still comes out a bit darker than it normally would with other converters for some reason (I think we discussed this before) but not sure if it has been resolved or do i just need to boost up the exposure by a few stops just for these certain files only while using MLVFS?

Anyway, no big deal since I was't too pleased with the results of my footage. I'll have to work on some more with my color grading skills (if any) lol -- guess everything happens for a reason. Good thing I didn't waste a whole bunch of ProRes files for nothing.  :-X

Looking forward to your next update in regards to MLV/Dual-ISO files "edge-directed interpolation" system... it'll definitely be worth having this feature to speed up the workflow (even if its just a little bit) but perhaps not enough for you to redo all the codings?   :D

Thanks again, Dave!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on May 20, 2015, 06:27:37 PM
*updates*

Decided to try MLV spanning files again from a 5D3. Here's what I've encountered... One of the MLV's were recorded longer than 49 seconds (as stated in MLVFS web host) and when viewing playback the same original MLV's in MLRV (which playbacks fine the entire clip whereas one of the clip actually goes over 4 minutes long) but not after exporting from MLVFS. Strange?

https://vimeo.com/128391858 (https://vimeo.com/128391858)

Ah, so I'm not sure where to look into this so called spanning culprit? Do I need to rename the spanning files to make it recognizable? (Will have to try these spanning files again with another converter such as cr2hdr-r to see if problem still persist and if so then it may not be your app's issue) so I'll get back to you on that shortly.

Have you ran into similar situation or anyone else out there?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ddelreal on May 24, 2015, 11:03:29 PM
Quote
Thanks so much for this, really like using it for my 6D files. Question though: with regular raw files using raw2dng (on Mac) ACR opens the converted files very flat looking and I can adjust accordingly or I can open in Resolve and set to BMD and do the same thing. Works pretty much the same in Resolve when using MLV converted to CDNG using MLVFS but if I use ACR, it looks like REC 709 already. Is that correct?

Sorry to quote my own post but was just wondering about my question...
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on May 25, 2015, 03:57:55 AM
MLVFS outputs RAW sensor data in the form of DNGs. How various software handles and interprets that data is up to the particular software. There may be slight differences in the metadata output between raw2dng and MLVFS, causing various softwares to interpret them slightly differently by default. However, the RAW data is the same, so once you make adjustments, you should see no difference.

I can't really comment on what ACR specifically is doing because I don't use it for video (and I recommend against using it for video), but I highly doubt it's outputting REC 709 by default as that is a video standard and ACR is intended mainly for photography and print and was never really intended for video.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 06, 2015, 10:33:51 PM
@dmilligan:

Any suggestions on how to prevent MLVFS from disappearing from desktop during rendering within AE? It has happened to me three times in the last 48 hours on this Mac Mini running Yosemite...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F376%2F18537058731_3d29bf4d23_n.jpg&hash=861acd969377a32ef0c835fbd46ed477) (https://flic.kr/p/uf4gJT) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F9%2F8846%2F17914435473_5df64a4375_n.jpg&hash=7c2801f0e34cceeed641884cec14a2ab) (https://flic.kr/p/ti3ax8)

It's connected to an external drive via USB 3.0 -- tried this routine again with destination export on OS system instead & issue still persist. Is this normal when working with more than 50 files at once?

Thanks, David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: goran on June 07, 2015, 01:59:34 PM
Just a quick question since I couldn't find an answer, how do I remove the MLVFS from services should I wish to do so?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on June 07, 2015, 02:15:14 PM
Username/Library/services

You might have to unhide your library folder via terminal.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: goran on June 07, 2015, 03:01:51 PM
Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DanHaag on June 08, 2015, 05:56:50 PM
Noticed MLV audio playing as it should in DaVinci Resolve when using your awesome MLVFS. Got some old CDNGs converted with RawMagic, their video and audio are badly out of sync (audio starting way after the video). Is there some metadata MLVS is adding that I might apply to my old files as well using ExifTool to fix this? I think I saw somebody writing about this a while ago but can't find those postings anymore.  :(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on June 08, 2015, 06:03:33 PM
DanHaag,

If you have cDNGs then I assume you got a WAV file for the audio also. Can you just trim the WAV so it is correct? I don't think metadata in the cDNGs (if any) is going to fix your problem.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DanHaag on June 08, 2015, 06:38:59 PM
I've done that for parts of the footage in FCPX. Can't do it directly in Resolve since it would not consider this kind of edit when outputting single clips i.e. as proxy files for editing (tried that as well, audio just goes back where it originally was after export). Manually moving the audio track in my NLE messes up my workflow since what I usually do is take the proxies from Resolve, sync them with external audio recordings in Plural Eyes and send this timeline to FCPX for editing without having to worry about that part of the process anymore. There's a whole lot of footage involved in that project I've started working on before MLVFS was around (wasn't able to keep the original MLVs, crazy amount of TB...) - Long story short: I can work around the problem but it's a huge effort and if there was any way to fix the problem with correcting metadata I'd be more than happy to know.  :) It just looks like DaVinci reads incorrect information on which timecode to put beginning of each audio track. Maybe there's no solution for this, other than fixing it the hard way.  :'(
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on June 08, 2015, 06:47:09 PM
It's more than just trimming the WAV file. There is special time code metadata that MLVFS puts in both the DNG and the WAV file. I don't think it would really be possible to add this with exiftool to the DNGs and there's no way at all, that I know of, to add it to the WAV file. You would have to write a program to do it. For more info see dng.c and wav.c in the MLVFS source, or have a look at raw2cdng code which also adds this metadata.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DanHaag on June 08, 2015, 06:56:36 PM
Thanks for your clarification.  :) If there was a way to fix the files, it's totally out of my reach.  :'( But now at least I can focus on working around the audio sync manually without worrying if there's an easier way for me to do it. After that project is done I'll stick with MLVFS for sure.  ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on June 08, 2015, 07:23:28 PM
Try bwf meta edit. I use it to add the black magic cam info from dmilligan wav info in cr2hdr-r.(thanks dmilligan)
 I trim audio with sox and add meta time code info with exiftool. You can pm me a file with out of sync audio. Might be able to help.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: alexidoia on June 09, 2015, 03:21:04 PM
Hello,

I did not know where to post this, but I have discovered what I think is a bug from MLVFS and I wanted to report it.
I first used mlrawviewer to convert my mlvs to dngs but as I use the cinelog LUT and that they advise to use MLVFS because of its respect of metadata blablabla, I did another convertion from mlv to dng using mlvfs. And that point is important because I could have never found out about the bug if I had not kept the first export from mlrawviewer.

As I were editing, I found a shot of only one frame. I found that strange so I decided to compare that dng (produced by MLVFS) to the dng produced from mlrawviewer. I found out that the shot was actualy a minute long. I did another export from MLVFS just in case and it did produced the same one-shot DNG. A few hours laters similar problem with a shoot a few second long whereas I remembered having at least 2 minutes of ongoing action. I exported the DNG again from MLVFS and had the same shot with the same short length. I compared with the dng from mlrawviewer and found out that my clip was -as I thought it would be- 2minutes long.

My conclusion is that for some reason (that have nothing to do with the computer performance while exporting) MLVFS produced shorted DNGs. This looks clearly like a bug because if I repeat the export, the produce DNG is exactly the same, it is not a random stop.

I thought some people would know about it. This is very bad since, you really don't know if your shot is shorted or not.

Thanks.
Alex
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on June 09, 2015, 03:45:23 PM
So you have only one file in the mlvfs output folder when you navigate there? No GIF? No WAV? Screenshot of the folder contents?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: alexidoia on June 09, 2015, 06:22:43 PM
Quote
So you have only one file in the mlvfs output folder when you navigate there? No GIF? No WAV? Screenshot of the folder contents?
heu... that was not for me right ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on June 09, 2015, 07:24:13 PM
Yes it is...
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on June 09, 2015, 07:51:41 PM
@alexidoia

Take a screenshot of the web interface for those shortened clips, upload an example MLV file somewhere that exhibits the problem or at least send me the output of mlv_dump -v
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 09, 2015, 07:56:54 PM
I've reported this somewhat similar issue (http://www.magiclantern.fm/forum/index.php?topic=13152.msg147987#msg147987issue) but no one got back to me about it as of yet...

@alexidoia:  was yours done with spanning files or no?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on June 09, 2015, 11:14:23 PM
For those using the Pismo File Mount version of MLVFS on Windows, I know that it hasn't been updated in eight(!) months, and it's far behind @dmilligan's Mac/Linux version.  I took a very long break from ML or MLVFS development, and I'm starting to get back into it.  I still have to figure out the best way to "catch up", not least of which is figuring out how to implement the equivalent to the webgui.  Let me know if there are specific features that should be the highest priority (e.g., FRSP support).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on June 10, 2015, 07:52:32 AM
ayshih: Super! I use it all the time despite lack of Web gui. I just mark drowes files in total commander select quick mount and it mounts all the files I selected at once.
I've been wondering if pismo can take advantage of existing idx files. It seems to take about the same time to mount with or without existing idx.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: alexidoia on June 12, 2015, 03:30:45 PM
I'm sorry,
I did not get back to you, as I have a movie to edit and a baaad deadline.
I don't have time to put the files online nor to test anyfurther, but basicaly,

I have a directory of .mlv
I right click on that directory to choose service>MVLFS et choose an empty directory. For 265 .mlv I have 265 .MLV directories which contains
1 _PREVIEW.GIF file
X .dng files
1 .log file
1 .wav file

That is to answer part of your previous questions.
The problem is that for one .mlv that last about a minute in mlrawviewer, I have only 1 .dng produced
and for some others I have the wrong amount of .dng files.

I did move the .mlv files that had not been converted correctly in a separate directory. and I mounted the directory via the services again. To my surprise, the file that used to output only one .dng had about the right amount of .dng this time. I am saying about because when comparing with the output of mlrawviewer, some .dng where still missing.

So I don't know. It changed something to isolate them into a different directory but it didn't really solve the problem.
I am not running mvlfs through the terminal so I am not really sure how to ouput the verbose version of it.

I'll get back to that when I have more time but right now I can't be diverted too much from my editing.
Alex
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 12, 2015, 08:53:51 PM
Seriously... I can't take this anymore... I REALLY cannot fathom with this any longer in allowing myself to use MLVFS again with BIG projects (regardless of how nice and awesome it is) it just totally won't render/export from AE for some reason (At first it says its rendering and I check the destination folder that's on the same external drive where I can see the file being processed) and leave my MacMini on all overnight (with screensaver and disk sleep enabled 'NEVER' within setting in OS X -- because I feel these can corrupt the rendering if it goes to sleep overnight):

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F495%2F18748713141_1ff533bc54_n.jpg&hash=f1a298742d229b5d631e600cafb4f4c0) (https://flic.kr/p/uyL4dR)

It's driving me INSANE, this supposedly only happens if I use an external drive plugged in via USB 3.0 (big projects are usually over 160-200GB worth of MLV's) so I can't obviously use my Mac Mini's disk space which is limited to about 120gb of free disk space.

However, if I run small projects with only 10-20 MLV's then I can run MLVFS within my system on Mac Mini with export destination set on my desktop for faster rendering (Mac Mini has built in SSD) which works GREAT! I mean really fast and gets the job done well!

BUT for some reason if I use my external drives (which aren't SSD) seems to be running slower (which is to be expected) but then once I go to bed and wake up the next morning I find myself encountering with these nightmares on several occasions as previously reported:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F508%2F18740826902_6cdb7f07a3_n.jpg&hash=de46a780318b8e2d67011456176e2554) (https://flic.kr/p/uy4CVd) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F445%2F18748178761_1cd5116147_n.jpg&hash=b313e5ccb4eba50d7e942423562c6005) (https://flic.kr/p/uyHjnp) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F285%2F18123102844_869bb0e80f_n.jpg&hash=c982c8de96a9f29e3201ac1c5d6176b9) (https://flic.kr/p/tBtD5q)

Also worth noting is the fact that it appears to me that I may need to purchase a SSD external drive with USB 3.0 in order to prevent this from happening again? What I mean is what can I do (if any) to prevent from having MLVFS mount files disappear in the middle of the night thus I believe is what causing these errors in AE during rendering?

I don't have this problem if I run everything solely on this Mac Mini (small projects) but if I want to work with big projects then I need to be able to use my external drives without hesitation of any kind. I thought I used to be able to do this with no problems in the past but as of lately is has been and I'm not sure why or what has changed?

The two things that I think could be the culprit for me is the fact that I have shot both in Non-Dual ISO MLV as well as in Normal MLV and there are all mixed up in order -- so is it not a good idea to have CS2x2 & Dual-ISO (Full 20-bit) both enabled on in webGUI?

I don't mind the slow process but am I allowed to do it this way?

Or do I need to separate them like batch all the Dual-ISO MLV's into one Folder and Regular MLV into another Folder?

Forcing myself to run TWO separate Projects to possibility prevent from this so called culprit occurring again that I've been encountering as of lately - hmm?

Or would it be ideal to try and run two external drives (one 3.0 and one Firewire 800) with one being used to provide BIG PROJECTS and one being used for the destination point for ProRes exports?

Sorry to bring this to you, David but I feel that I've tried all possibilities other than considering myself to purchase a SSD external drive w thunderbolt support. Thoughts on that?

Thanks for reading this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on June 14, 2015, 02:52:24 AM
Issues like this are going to be very difficult for me to duplicate and fix, I would suggest trying to narrow it down as much as you can and figure out how to reliably duplicate the issue. I just don't have time to try and replicate issues involving unpredictability and vast amounts data/processing time.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 14, 2015, 05:41:30 AM
Okay no problem! I'll see what I can do over the weekend and will do the best I can.

FYI - I managed to get MLVFS to finally work again on my 2006 MBP by updating the Fuse to the latest and all is fine and dandy -- so please disregard this particular issue!

Thanks for your time, David!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on June 18, 2015, 05:23:53 PM
I anyone of windows users who recently installed PFMap from the link on the first page have problems with MLVFS and importing files to premiere - pay attention on PFMap build - now is 177, i had some serious problems with it (couldnt import to premiere) and figured out that older build 171 is the one to go. Seems that 177 has some changes and because of that it wasn't working for me.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on June 19, 2015, 05:43:26 AM
Good to know!  I still haven't been able to put back real time into MLVFS again, but it looks like there have been quite a few recent changes to Pismo File Mount.  Here is the link to build 171:

http://www.pismotechnic.com/download/archive/pfmap-171-win.exe
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on June 19, 2015, 12:54:11 PM
updated the download link in the OP with that one
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on June 21, 2015, 11:22:01 AM
Hi, Dmilligan, I'm a mlv_dump long time user. Some feedback on my attempt to try MLVFS under windows with pismo(1.7.1) driver with ACR and AE.
1. It works, fast and flawlessly.
2. WB temp "as shot" always 2550, mlv_dump produces dng sequence with correct WB from camera.
3. I find no way to use command line options.
4. DLL file seems not updated from october 2014. Both mlv_dump and cr2hdr has updates/bug fixes since then.
5. Something wrong with MLVFS CDNG files format itself - Capture One, for example, didn't recognize it. DNG from mlv_dump and CDNG from raw2cdng are OK, but CDNG from MLVFS - not.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on June 21, 2015, 11:50:04 AM
Curious.
Are you sure mlv_dump is translating to correct wb to as shot neutral values? The mac binary doesn, t seem to at least from what I can see.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on June 21, 2015, 01:04:34 PM
Yes, mlv_dump generates DNGs which looks absolutely the same as CR2 if I shot the same scene with same camera settings. Actually, wrong WB is not a big problem.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on June 21, 2015, 01:32:15 PM
Oh, gotta check for mac. Actually I personally prefer to nail wb before actually shooting. Saves a lot time.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ayshih on June 21, 2015, 04:07:15 PM
@mothaibaphoto I wrote the PFM version of MLVFS, and as you've already noted, it is out of date because I took a long personal break from development.  I hope to fix it up with @dmilligan's improvements, but it will take some time.

2. WB temp "as shot" always 2550, mlv_dump produces dng sequence with correct WB from camera.
Good to know.  What camera?  And are you shooting normal images or dual-ISO images?

3. I find no way to use command line options.
Given how the PFM version works – mounting a specific MLV file in Explorer, typically via the right-click menu – it's non-obvious what the best approach is to feed in options.  I'm open to suggestions!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on June 21, 2015, 06:58:28 PM
Hi, ayshih!!! It's so great that you have plans to update MLVFS on windows! It seems MLVFS becomes very popular among Mac and Linux users, and its simplicity tempting us, windows users.

Good to know.  What camera?  And are you shooting normal images or dual-ISO images?
5D MarkIII, normal video
it's non-obvious what the best approach is to feed in options.  I'm open to suggestions!
It could be some predefined ini-file in folder with *.MLV files. When any MLV file mounts, DLL looks for parameters in this file in same folder. Ideally, this file could contain path to *.Mxx files. This makes possible not to copy files from cards to one folder first when shoot in spanned mode.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on June 21, 2015, 10:43:06 PM
mlv_dump does not write correct WB in DNG files. It uses a hard coded AsShotNeutral for Daylight (5500). There is a PR of mine to replace the old CHDK DNG writing code in mlv_dump with the DNG code I wrote for MLVFS that does handle WB correctly (with some kelvin to RGB multipliers code from UFRAW). Are you using that version or something? Otherwise I don't see how you're getting correct WB from mlv_dump unless you always shoot in daylight and never noticed it was wrong for other lighting situations.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on June 22, 2015, 06:14:10 AM
Sorry, dmilligan and everyone confused with my wrong sentences about mlv_dump. It really use 5500 K Temp. I didn't notice it, i prefer to shoot in good light indeed :) Just checked again with file shot at 4500 K in camera:
MLRawViewer 1.4.3 - 5500 K
mlv_dump (last one, by the way, how to check its version?) - 5500 K
raw2cdng.1.6.5 - 4500 K (wow, the winner !!!)
mlvfs.dll 0.4.0.0. - 2500 K :(
Maybe the obsolete dll is the only problem?
And, please, pay attention: MLVFS output file - the only not recognised by Capture One.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: morrisonmike on June 25, 2015, 02:01:04 AM
Sorry, I'm new to Linux. Can you give me the command line entries for this?

"Install FUSE via the normal way for your distribution (it may already be included)
Download the code with git and compile using make"

I get the following error:

mike@GP532UC-ABA:~/mlvfs/mlvfs$ make
gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 main.c dng.o index.o wav.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose/mongoose.o webgui.o resource_manager.o lj92.o gif.o LZMA/7zAlloc.o LZMA/7zBuf.o LZMA/7zBuf2.o LZMA/7zCrc.o LZMA/7zCrcOpt.o LZMA/7zDec.o LZMA/7zFile.o LZMA/7zIn.o LZMA/7zStream.o LZMA/Alloc.o LZMA/Bcj2.o LZMA/Bra.o LZMA/Bra86.o LZMA/BraIA64.o LZMA/CpuArch.o LZMA/Delta.o LZMA/LzFind.o LZMA/Lzma2Dec.o LZMA/Lzma2Enc.o LZMA/Lzma86Dec.o LZMA/Lzma86Enc.o LZMA/LzmaDec.o LZMA/LzmaEnc.o LZMA/LzmaLib.o LZMA/Ppmd7.o LZMA/Ppmd7Dec.o LZMA/Ppmd7Enc.o LZMA/Sha256.o LZMA/Xz.o LZMA/XzCrc64.o -pthread -lfuse -lm -o mlvfs
main.c:33:18: fatal error: fuse.h: No such file or directory
 #include <fuse.h>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on June 25, 2015, 02:10:33 AM
Use your distro packaging tools to find what package provides fuse.h, some thing like libfuse-dev  on Debian.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: morrisonmike on June 25, 2015, 02:30:31 AM
Thanks, but "Use your distro packaging tools to find what package provides fuse.h, some thing like libfuse-dev  on Debian" doesn’t mean anything to me... Isn't there just some commands I can cut-n-paste into the command prompt?
-MM
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on June 25, 2015, 02:34:25 AM
No, because without telling which distribution you are using we can't tell.
Google
install fuse <insert name of your Linux distribution>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on June 25, 2015, 03:53:39 AM
assuming you are using a debian flavored distribution (like ubuntu) something like
Code: [Select]
sudo apt-get install libfuse-dev
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: morrisonmike on June 25, 2015, 09:25:04 PM
Thanks everyone. I'm up and running now.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 27, 2015, 08:19:42 PM
Just tried running MLVFS this morning on a test run and it randomly disappeared from my services list on MBP running Yosemite.

I should note that this happened after I tried to create a shortcut button for MLVFS -- could this be the culprit? (though I turned it off and it still doesn't show up on the services list after right click) and I also notice now it just opens Automator every time I want to reinstall MLVFS -- I must have pushed some buttons somewhere not realizing it within Automator?


How do I fix this and make it install and overwrite it as well as running MLVFS like normal again?

However, it still shows under settings preferences. See attachments below.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc4.staticflickr.com%2F4%2F3751%2F19022560018_b3b0756d9f_n.jpg&hash=074d3c688bb72704d837f6a835e212a6) (https://flic.kr/p/uYXAmJ) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fc1.staticflickr.com%2F1%2F406%2F19022817280_3641ae708f_n.jpg&hash=e0a5c14a7092830e0c6b8beca40334f9) (https://flic.kr/p/uYYUQh)

Anyone else out see a similar situation? I've also tried uninstalling/reinstalling both MLVFS and FUSE to no avail.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on June 27, 2015, 10:13:24 PM
You have to right click on a folder. You can only mount folders, not individual files.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on June 28, 2015, 05:18:39 AM
Actually I apologize for the false alarm this morning. Must have been either really tired or a bit hangover from being in the post world for quite a bit!

FYI regarding previous post (http://www.magiclantern.fm/forum/index.php?topic=13152.msg149498#msg149498post) that I've noticed MLVFS hasn't disappeared or any problems when rendering overnight as of lately is most likely due to the fact that I had updated to the latest FUSE.

Also is it a good idea to use their beta version of FUSE or best to stick with non-beta's?

Thanks again, @dmilligan!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: benyonker on July 30, 2015, 04:55:26 PM
I just found this a few days ago, and I've been using it re-process some MLVs that were full of bad pixels. The "bad pixel fix" feature alone is going to save me hours of post on this music video, so thank you!! I have two questions that I haven't been able to find in the forums:


Neither of these is a big deal; I'm obviously color-correcting anyway, and the dates aren't critical. I'm just curious.

Thanks again!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on July 30, 2015, 05:59:23 PM
1. A DNG is a DNG, the raw data is going to be the same between the two. "Darker" is just a result of how the raw development application (ACR, Pemiere, DaVinci, etc.) chooses to treat the file (you can't actually 'see' true raw data or levels, it must be processed, demosaiced, white balanced, and converted to some real colorspace to display on your monitor for you to even view it, there are going to be arbitrarily chosen default values for parameters used in these processes).

Simply increase the exposure in your post processing app and they will look exactly the same. The difference is likely due to some different metatdata in the DNG that causes the raw development program to interpret the raw data differently by default. There is a DNG metadata field for 'exposure offset' or 'exposure' or something like that. There have been a lot of people who complain about the DNGs not matching the camera's LCD screen and 'appearing darker'  (this complaint is sort of silly for the reason given above), my guess is that RAWMagic developer utilized this field to make the DNGs appear brighter to match the LCD screen (to appease his paying customers). If you'd like to find out for sure, try comparing the metadata between the two with exiftool.

2. The struct tm has a field tm_mon, on some systems this is [0,11] on others it is [1,12]. That is probably the issue.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: benyonker on July 30, 2015, 11:48:26 PM
Thank you for the quick reply. Your answer to #1 makes perfect sense. I agree that the complaint is sort of silly given your explanation. As for #2, I don't even know what to make of that or how I can alter the tm_mon field. I'm on a Mac. Is this something I would have to do in Xcode? Terminal?

One final question: I shot this footage at 29.97, but I want it to play back at 23.98. In the MLVFS window, I see an option to override the framerate. When I change it to 23.98 (or 23.976), PPro interprets the footage to be 23.000fps. If I use 24 in MLVFS, PPro interprets it at 24. That could work, but I'd still rather have it be 23.98. Am I better off leaving it at 0 and doing Interpret Footage in PPro? Will that cause issues in Resolve?

Thanks so much, and again I apologize if these are basic questions.

**Correction to above: Resolve sees the footage as 23.98. When I render proxies, PPro also sees them as 23.98. Looks like we're OK here; just a glitch with the way PPro sees the original DNGs.**
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on July 31, 2015, 01:15:16 AM
For the month stuff, I will have to look at the code and fix it
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Slasheal on August 01, 2015, 10:19:50 PM
Some info:
I tried to compare MLV MYSTIC / MIRAWVIEWVER / MLVFS

Size:
MLV MYSTIC : 3mo
MIRAWVIEWVER : 1.8mo
MLVFS : 3.4mo


White balance:
MLV MYSTIC : 5100K (even if manual WB)
MIRAWVIEWVER : 5000K (even if manual WB)
MLVFS: in AWB mode, it will take the value stored in manual WB.
If you set to a WB preset (ex: tungsten light), it will take the WB preset.


In short: DO NOT USE AWB with MLVFS.
Still, it's a good thing to be able to change WB before grading.
Most grading softwares don't have "photo" WB control. I don't want to waste my time on camera raw.

So yep, actually for me, MLVFS is THE best solution on OSX to manage raw video :).
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on August 02, 2015, 02:45:22 AM
I'm not 100% sure, but this may actually be an ML issue (AWB is not captured correctly in camera to the MLV format). Need to check. I'm pretty sure MLVFS handles WB as correctly as possible, at least according to the MLV format specification. It was a while ago that I implemented all the WB stuff, so I'm a bit fuzzy.

You can check yourself by using mlv_dump -v to see all the metadata from the MLV file.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on August 02, 2015, 08:09:01 AM
Some new observations regarding WB and different convertors. I shot Dual ISO silent timelapse in MLV yesterday. WB in camera was set to AWB. I usually process this kind of files with mlv_dump+cr2hdr routinely. This time I decided to try different converters while keeping eye on WB. First surprise - mlv_dump+cr2hdr produces DNG with correct WB - 4100 "As shot" in ACR. Second - I can't try anything else - raw2cdng.1.6.5 just crashes, mlvfs.dll mounts empty folders and crashes when I unmount such empty folder.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 02, 2015, 09:55:13 AM
Do you mean Mlvfs+cr2hdr? Mlv_dump hardcode wb settings.
Mlvfs delivers correct wb for me.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on August 02, 2015, 10:23:21 AM
Do you mean Mlvfs+cr2hdr?
I mean Mlvfs doesn't work at all with this kind of files, at least on Windows.

Mlv_dump hardcode wb settings.
I know it. But this setup wonderfully produces correct WB :)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on August 02, 2015, 07:14:40 PM
The Windows version is way behind the Mac/Linux version and does not handle WB correctly. I believe cr2hdr recomputes the WB, b/c the processing it does has an effect on WB (I could be wrong on this point).

I would not consider hardcoded WB that just so happens to be the same as the scene you shot "correct". More like "lucky".
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Marsu42 on August 04, 2015, 12:04:24 AM
The Windows version is way behind the Mac/Linux version and does not handle WB correctly.

Doh - any chance to get the win version updated as the webdav solution isn't available anymore (and less straight forward than mounting) ... or are the code bases so different the win version will lag behind?

I believe cr2hdr recomputes the WB, b/c the processing it does has an effect on WB (I could be wrong on this point).

I dunno if it's supposed to recompute wb, but it definitely has an effect on wb at least with my cams (60d, 6d) so with the adobe raw converter I usually have to do some manual tint/temperature adjustments.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on August 04, 2015, 06:20:12 AM
Maybe cr2hdr will produce more accurate result dealing with input DNG files with actual, not "hardcoded" WB?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 04, 2015, 01:55:03 PM
Doh - any chance to get the win version updated as the webdav solution isn't available anymore (and less straight forward than mounting) ... or are the code bases so different the win version will lag behind?
Main issue is I don't own a Windows box.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: sggvideo on August 06, 2015, 10:09:51 AM
Hi. I was very excited to discover and use MLVFS with my mac, and it has been a charm!!

I recently noticed when I try to mount a folder with my entire 63.98gb of .MLV data I sometimes lose frames in the .DNG's. In the browser window that pops up it also says there are less frames than I recorded. BUT, If I put one .MLV file (and all of its parts .M01, .M02..) into a folder alone and run MLVFS it gives me all the frames!

Anyone else had this happen?

If it's been answered I apologize, I read some of the discussion then skipped to the end  :o
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Slasheal on August 08, 2015, 06:28:41 PM
Another problem I have, I will test later if it's related to MLVFS or not:


The DNGs created with MLVFS, when opened with ACR, doesn't allow me tu properly use lens correction:

https://forums.adobe.com/message/7823545#7823545
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 09, 2015, 02:29:32 AM
I replied to the Adobe thread. Basically, it's going to be a challenge figuring out what the metadata needs to be for ACR to interpret the FOV correctly, so it can apply the lens corrections appropriately. Feel free to grab exiftool and the DNG spec (http://www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf) and start fiddling with the metadata. ACR is a bit of a black box, so figuring this stuff out requires some trial and error (something, I don't really have time to do myself).

When I was looking into this, I stumbled upon a DNG tag I had apparently previously overlooked, it's called DefaultScale. Basically it's pixel aspect ratio (the name makes no sense, which is why I had never found it before). Filling in the correct value here should allow us to re-scale the squeezed 50/60fps automatically! ACR seems to respect this value, I haven't tried others (Premiere, Resolve, etc.). I'll need some sample MLV files of squeezed footage from various cameras for testing, so please upload it (post it here: http://www.magiclantern.fm/forum/index.php?topic=11899.0).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 09, 2015, 10:28:54 PM
Updates:

I have implemented the automatic de-squeezing for 50/60 fps (and EOSM) 720p footage. Please test (and make sure also that it doesn't try to de-squeeze any sort of normal footage). Also, I have only tested ACR (which works correctly), other programs may or may not ignore this tag, so also let us know what programs work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 10, 2015, 12:25:54 AM
Gonna test this out tomorrow with my 5d mark III. Tried messing with exiftool yesterday but couldn, t get the tags in there. Thanks for sharing dng spec and really good find I must say.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on August 10, 2015, 02:42:43 AM
Very nice update, looks like it works correctly with squashed 7d footage. This seems to give us better scaling quality than what AE includes, presumably since it's the raw data that's being scaled. Do you have any insight on this, dmilligan? AE's detail-preserving upscale works well but only allows for the footage to be scaled to match the comp width or height, making this a great time-saver as well.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on August 10, 2015, 03:00:49 AM
It makes sense the quality might be slightly better since it allows the ACR debayer algorithm to be aware of the "non-square" pixels, and so it can take that into account when interpolating, resulting in slightly better interpolation.

Do you mind posting some 100% crop comparisons between the two methods for the pixel-peepers?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 10, 2015, 01:35:53 PM
Did a test with three mlv clips filmed in 48, 50 and 60 fps with my 5d mark III. ACR interpretes the default scale value accordingly but couldn,t get the reading in Davinci resolve nor Premiere pro CC 2014.
Nice to use in an ACR workflow. Good one.

ACR
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Fll46ql11z%2FScreen_Shot_2015_08_10_at_13_15_55.png&hash=893e24d96fbeae3859e81de6166644c5) (http://postimg.org/image/ll46ql11z/)

Davinci resolve
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2F4yroemefr%2FScreen_Shot_2015_08_10_at_13_22_17.png&hash=044b789078aa61a21d5a945ece743a4a) (http://postimg.org/image/4yroemefr/)

Premiere pro CC
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs15.postimg.org%2Frji00tk0n%2FScreen_Shot_2015_08_10_at_13_31_46.png&hash=72d9e219a65a6cdec4ee929411562a0f) (http://postimg.org/image/rji00tk0n/)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on August 10, 2015, 02:44:45 PM
Premier and Resolve likely ignore the tag as it's not in CinemaDNG specs.

In Premier you can add custom pixel aspect ratios by editing a text file called 'Interpretation Rules.txt' (found in the Premier/Plugins directory - likely in a region specific folder i.e. en_US etc)

The aspect ratios must be integer values so if the unscaled image requires vertical scaling of say 1.67 you would scale by 100 and enter 100/167 in the text file - I haven't tried this BTW so you will have to experiment with the numbers.

Resolve is more limited as it has preset pixel aspect ratios so you would scale the frame size instead.

I'm not sure if either of the options above have any direct affect on the debayer algorithm itself but set Resolve's debayer to highest quality (and I suggest using the 'smoother' debayer option for these type of clips).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 10, 2015, 02:55:55 PM
Premiere state you can edit interpretation rules in the txt file. Me and several others have tried with no sucess. In Davinci resolve it,s very easy to unstretch the dng files. I think it,s "stretch footage to corners" or something like that.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on August 10, 2015, 03:11:02 PM
re: Resolve - that will only work correctly if the unstretched pixel aspect ratio correctly matches the output frame dimensions (i.e. 16:9 etc even if the actual stretched image is letterboxed) - a lot of users shoot odd frame sizes. It's better to choose 'Scale entire image to fit" in the main settings then scale the vertical dimension using the scale/framing controls in the color panel.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 10, 2015, 03:14:34 PM
Scale image to fit. That was what I meant :). Too bad however that the default scale tag isn, t read in cdng spec though...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 10, 2015, 03:52:23 PM
Premier and Resolve likely ignore the tag as it's not in CinemaDNG specs.
Well technically, it is in the CinemaDNG spec, since the cDNG spec is basically: reference to the DNG spec + some extra tags + file/folder naming convention. The real reason it's ignored is b/c these programs don't really try to conform to the spec, they simply implement it for the specific 'flavors' of cDNG produced by specific cameras, and there probably aren't any cameras they support with non-square pixels (though if using anamorphic lenses, any camera can have non-square pixels). But it's not like anybody even fully supports the DNG spec either (and Adobe, who wrote the spec, is often the worst offender).

The aspect ratios must be integer values so if the unscaled image requires vertical scaling of say 1.67 you would scale by 100 and enter 100/167 in the text file - I haven't tried this BTW so you will have to experiment with the numbers.
The exact aspect ratio is 3:5, so wouldn't it be better to use 3/5?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: N/A on August 10, 2015, 04:47:34 PM

Do you mind posting some 100% crop comparisons between the two methods for the pixel-peepers?
Sure, I have a couple good examples. It may take a while though, my internet connection is pretty spotty right now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Andy600 on August 10, 2015, 04:53:24 PM
Yes, it's all down to what the devs choose to support beyond the mandatory tags.

I totally agree about Adobe. Their logic is more understandable when dealing with raw stills/print etc via ACR, Lightroom, Photoshop but I don't understand why they restrict Premier CDNG debayering to a display referred colorspace. It takes away some fundamental control of the raw image. Hopefully Resolve 12 has shown them how better to deal with colorspace management. I hope they move to something similar and also drop their reliance on ICC profiles - but I doubt it.

re: Pixel aspect ratio - of course, if that's the exact ratio it should be used.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on August 11, 2015, 12:24:00 AM
So far, I have been able to confirm with slo-mo files to be working fine with the latest updated MLVFS for both EOSM and 70D... it scales up nicely with ACR (SmartImport2) workflow in AE.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm1.staticflickr.com%2F522%2F20307155489_0dc320f90f_n.jpg&hash=3005c8b12be978f4baa9897d3c01b70e) (https://flic.kr/p/wWttQv) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm1.staticflickr.com%2F549%2F20485042732_ea030b4a37_n.jpg&hash=269a441748d9673198c6f7d2c5eb7384) (https://flic.kr/p/xdccvW) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm1.staticflickr.com%2F325%2F20305784328_0c19aa479e_n.jpg&hash=24e536f9216119cd2d0d75a2a0c0a630) (https://flic.kr/p/wWmseN) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm1.staticflickr.com%2F332%2F20307156969_792a71e2f4_n.jpg&hash=85404c6bc7cd24d5a0e886049546299c) (https://flic.kr/p/wWtuh2)


I can't speak for DR but can confirm that PP does NOT read the tags for proper scaling.

Up next is the 5D3 slo-mo files.

Thanks, @dmilligan for the kind update!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Slasheal on August 15, 2015, 04:27:50 PM
Hello.

MIRAWviewver allow lens correction
MLVFS doesn't yet
MLVMystic doesn't yet too

Maybe you should ask for help.
You've got and answer on adobe forum:

Quote
"Camera Raw / Lightroom needs to know the sensor area (or crop factor) to map the pixels of the source image to the physical model of the lens profile. ACR calculates this from the EXIF (not DNG) metadata. You can do this in one of two ways:
 
1. Use the FocalPlaneXResolution, FocalPlaneYResolution, and FocalPlaneResolutionUnit tags.
 
2. Use the FocalLength and FocalLengthIn35mm tags.
 
Of these two options, I recommend #1 because you'll have more precision. (For #2, FocalLength has tag type rational, which is fine, but FocalLengthIn35mm is an integer. That's an issue for short focal lengths.)"

Hope it will help.
If any more samples needed, don't hesitate to specify.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on August 15, 2015, 05:07:05 PM
I know, I saw that, it's the information I needed. When I have time I will work on it, it's a lot of work (figuring out what the focal plane resolution actually is, is not trivial, there are 3 different video resolution modes, and many supported ML cameras that all have different pixel pitches). Anybody is welcome to contribute code at any time.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: gmfotografie on August 15, 2015, 11:19:30 PM
after convertingi get the dng files in the specified folder.
 my browser then opens and gives me a error message ... localhost:8000 -> could not opend.

using osx system
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 17, 2015, 03:49:28 AM
I can't get this to work at all  :-\ @deafeyejedi has been trying to help me troubleshoot the problem, but still no luck. I keep getting safari can't connect to server "localhost". error

As far as I can tell, I've followed all of the correct steps, but I'm afraid I'm missing something

Running
OSX 10.7.5
OSXFuse 2.8.0

Any help would be great
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on August 17, 2015, 04:38:32 AM
I keep getting safari can't connect to server "localhost". error
As far as I can tell, I've followed all of the correct steps,
but I'm afraid I'm missing something
Running
OSX 10.7.5
OSXFuse 2.8.0
Any help would be great
Not to sure about mlvfs running on "Lion" OSX 10.7.5 , I know it runs on Maverick(10.9.x) & Yosemite (10.10.x)
You may have to up-grade your OSX.

Even thou you don't have a web interface working  do get a mounted mlvfs folder ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 17, 2015, 07:06:43 AM
@reddeercity nope, I don't get a mounted folder either. Anyone else think I might need to upgrade? I'd like to avoid this if possible
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on August 17, 2015, 07:20:05 AM
Another thing I just thought of , I'm using a older version(2.7.3) of osxfuse and a older build of MLVFS back from Nov/2014
Maybe try a older version of OSXFuse & or see if there is older build of mlvfs still available .

If I have some time in the next few days I have a old MacPro 1.1 that's on "Lion" OSX10.7.5 I see if I can get it working on it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 17, 2015, 07:37:34 AM
If you could check that out, that'd be great. I tried older versions of OSXFuse, but didn't have any luck. I didn't think an older mlvfs would make any difference since it's a set of Automator actions. I'm thinking there's something I'm missing with OSXFuse or some deeper issue
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 17, 2015, 02:55:14 PM
MLVFS is not set of automator actions. The automator actions are there just make it easier for users who don't know much about command line to run it. There is an executable named 'mlvfs' embedded inside the automator package. That is what does all the work. The automator stuff just fills in the correct command line parameters and calls mlvfs for you, and then opens up the web browser to the correct address for the webgui.

If you want to figure out what is going on, you need to run that executable from the command line and see the output (the automator actions just throw away the standard output). If MLVFS is crashing, there should also be logs in ~/Logs/DiagnosticReports.

I would not recommend using an older version of MLVFS, it's not going to make a difference, other than probably being less stable and having fewer features (also, please do not share older builds here, I delete them for a reason). I would highly recommend upgrading to the latest version of the OS if you can; I'm definitely not going to be able to support old OS versions. If you really want to use an old OS version, you may have to compile MLVFS yourself.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 17, 2015, 11:00:08 PM
@dmilligan sorry about that. I was half asleep when I wrote that response and had forgotten that I did open the Automator package and find the executable [facepalm] I don't know how to run it from the command line though :\
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on August 17, 2015, 11:58:56 PM
Run it with no parameters and it should tell you what the options are.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 18, 2015, 04:42:34 AM
Not sure I'm doing it right, but I get:

Illegal instruction: 4
logout

[Process completed]
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 18, 2015, 12:42:02 PM
http://stackoverflow.com/questions/14268887/what-is-the-illegal-instruction-4-error-and-why-does-mmacosx-version-min-10

You will have to compile MLVFS yourself or update to a more recent version of OSX.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on August 19, 2015, 01:58:42 AM
Update to at least OS X 10.8.x and upward...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: cmccullum on August 20, 2015, 03:54:10 AM
Updated my system, and everything is working now. Thanks everyone for your patience and help!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 23, 2015, 02:10:33 AM
RE: automatic lens corrections

I have implemented (https://bitbucket.org/dmilligan/mlvfs/commits/f026ffedf411ce5613daa694808d6fe913ed30e6) the tags necessary for this (FocalPlaneResolutionX, FocalPlaneResolutionY, FocalPlaneResolutionUnit), but only for my camera (60D).
I need the default values for these for all other cameras.

Here's how to get them:
1. Download and install exiftool (http://www.sno.phy.queensu.ca/~phil/exiftool/)
2. Take a normal raw still with your camera (CR2)
3. Run this command:
Code: [Select]
exiftool -v <image_filename.CR2>
(where <image_filename.CR2> is the actual filename of your CR2)
4. In the output look for the FocalPlaneResolutionX, FocalPlaneResolutionY, FocalPlaneResolutionUnit tags. It should look something like this:
Code: [Select]
  | | 18) FocalPlaneXResolution = 5728.176796 (5184000/905)
  | | 19) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 20) FocalPlaneResolutionUnit = 2

I need the two ratios (ex: 5184000/905, 3456000/595) and the unit.
Once I get these values for all the cameras, I will update the code and release a new build.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on August 23, 2015, 02:46:37 AM
5D3:
Code: [Select]
  | | 26) FocalPlaneXResolution = 3942.505133 (5760000/1461)
  | | 27) FocalPlaneYResolution = 3950.617284 (3840000/972)
  | | 28) FocalPlaneResolutionUnit = 2

70D:
Code: [Select]
  | | 26) FocalPlaneXResolution = 6086.76307 (5472000/899)
  | | 27) FocalPlaneYResolution = 6090.15025 (3648000/599)
  | | 28) FocalPlaneResolutionUnit = 2

EOSM:
Code: [Select]
  | | 26) FocalPlaneXResolution = 5798.657718 (5184000/894)
  | | 27) FocalPlaneYResolution = 5788.944724 (3456000/597)
  | | 28) FocalPlaneResolutionUnit = 2
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Slasheal on August 26, 2015, 01:04:24 AM
7D:

Quote
  | | 26) FocalPlaneXResolution = 5715.545755 (5184000/907)
  | | 27) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 28) FocalPlaneResolutionUnit = 2


Great!

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Mehmet Kozal on August 26, 2015, 07:23:12 AM
650D:

  | | 26) FocalPlaneXResolution = 5798.657718 (5184000/894)
  | | 27) FocalPlaneYResolution = 5788.944724 (3456000/597)
  | | 28) FocalPlaneResolutionUnit = 2
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on August 26, 2015, 08:28:19 AM
6D :

Code: [Select]
| | 26) FocalPlaneXResolution = 3810.584958 (5472000/1436)
| | 27) FocalPlaneYResolution = 3815.899582 (3648000/956)
| | 28) FocalPlaneResolutionUnit = 2

Thank you !
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on August 26, 2015, 09:05:40 AM
Should be for the rest of the bunch. Otherwise please tell.
*updated with a complete list of cameras.


5D mark 2

Code: [Select]
| | 24) FocalPlaneXResolution = 3849.211789 (5616000/1459)
  | | 25) FocalPlaneYResolution = 3908.141962 (3744000/958)
  | | 26) FocalPlaneResolutionUnit = 2

50D

Code: [Select]
| | 24) FocalPlaneXResolution = 5315.436242 (4752000/894)
  | | 25) FocalPlaneYResolution = 5306.532663 (3168000/597)
  | | 26) FocalPlaneResolutionUnit = 2

500D

Code: [Select]
| | 24) FocalPlaneXResolution = 5315.436242 (4752000/894)
  | | 25) FocalPlaneYResolution = 5342.32715 (3168000/593)
  | | 26) FocalPlaneResolutionUnit = 2

550D

Code: [Select]
| | 24) FocalPlaneXResolution = 5728.176796 (5184000/905)
  | | 25) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 26) FocalPlaneResolutionUnit = 2

600D

Code: [Select]
| | 26) FocalPlaneXResolution = 5728.176796 (5184000/905)
  | | 27) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 28) FocalPlaneResolutionUnit = 2

650D

Code: [Select]
| | 26) FocalPlaneXResolution = 5798.657718 (5184000/894)
  | | 27) FocalPlaneYResolution = 5788.944724 (3456000/597)
  | | 28) FocalPlaneResolutionUnit = 2

700D

Code: [Select]
| | 26) FocalPlaneXResolution = 5798.657718 (5184000/894)
  | | 27) FocalPlaneYResolution = 5788.944724 (3456000/597)
  | | 28) FocalPlaneResolutionUnit = 2

1100D

Code: [Select]
| | 26) FocalPlaneXResolution = 4720.441989 (4272000/905)
  | | 27) FocalPlaneYResolution = 4786.554622 (2848000/595)
  | | 28) FocalPlaneResolutionUnit = 2

5d mark 3

 
Code: [Select]
| | 26) FocalPlaneXResolution = 3942.505133 (5760000/1461)
  | | 27) FocalPlaneYResolution = 3950.617284 (3840000/972)
  | | 28) FocalPlaneResolutionUnit = 2

6D

Code: [Select]
| | 26) FocalPlaneXResolution = 3810.584958 (5472000/1436)
  | | 27) FocalPlaneYResolution = 3815.899582 (3648000/956)
  | | 28) FocalPlaneResolutionUnit = 2

7D

Code: [Select]
| | 24) FocalPlaneXResolution = 5715.545755 (5184000/907)
  | | 25) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 26) FocalPlaneResolutionUnit = 2

60D

Code: [Select]
| | 26) FocalPlaneXResolution = 5728.176796 (5184000/905)
  | | 27) FocalPlaneYResolution = 5808.403361 (3456000/595)
  | | 28) FocalPlaneResolutionUnit = 2

70D

Code: [Select]
| | 26) FocalPlaneXResolution = 6086.76307 (5472000/899)
  | | 27) FocalPlaneYResolution = 6090.15025 (3648000/599)
  | | 28) FocalPlaneResolutionUnit = 2

eos m

Code: [Select]
| | 26) FocalPlaneXResolution = 5798.657718 (5184000/894)
  | | 27) FocalPlaneYResolution = 5788.944724 (3456000/597)
  | | 28) FocalPlaneResolutionUnit = 2
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: vertigopix on August 26, 2015, 09:06:23 AM
5D2 :

Code: [Select]
| | 24) FocalPlaneXResolution = 3849.211789 (5616000/1459)
| | 25) FocalPlaneYResolution = 3908.141962 (3744000/958)
| | 26) FocalPlaneResolutionUnit = 2
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on August 26, 2015, 10:55:53 PM
Thanks everyone. New build is available.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on August 26, 2015, 11:42:40 PM
Thank You!

*edit*

Just downloaded and installed... Ran a few test files and kept getting the same errors that I've never seen before. I just ran this same file last night and worked fine but that was before I updated to the latest MLVFS.

I also deleted .IDX files and tried again with no luck.


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm6.staticflickr.com%2F5718%2F20722743888_f42e89ab34_n.jpg&hash=d9af5ddbc311c9396ad4a60696abe887) (https://flic.kr/p/xzctMJ)

Anyone else encountering similar errors?

*edit #2*

I just downgraded to an earlier version of MLVFS and ran into the same error -- now that most likely tells me that it's definitely not due your app and probably has to do with the pathway of the codec that I just installed DNxHR recently (although it worked fine last night) but I'll confirm this with you all once I get to the bottom of this.

*edit #3*

Got home, downloaded latest MLVFS on Mac Mini, Ran test file in AE and exported all Codec's like normal. Confirmed it has nothing to do with your app, once again I apologize for the false alarm. It definitely has to do with the pathway somehow the codecs got misplaced.

Thanks again, David for your great work so my question now is does the latest updates with automatic lens corrections affect the quality a little similar to how it does if we were to do the same in ACR? If so, will there be a way to disable it? (not that I wanted to but just a thought worth asking).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on September 01, 2015, 06:34:28 AM
Hello, couple things.

I've done this aspect ratio editing before and was successful in getting Ppro to recognize the new AR in stretched slo-mo footage from a 5D3. Unfortunately I can't remember exactly how I did it. I could have sworn I opened up a DNG in TextWrangler but maybe it was something else. However this is what I do remember:

-I used the DefaultScale option and calculated the PAR needed to counteract the "stretch"
-It was only one value I changed
-I only needed to edit the first file in the sequence. It seems Ppro gets its information from that first file and ignores the metadata in the rest. maybe this is common knowledge just thought I would mention

Also for those of you having trouble getting the localhost:8000 site to work, make sure you wait long enough. The webpage won't load until all IDX files have been created next to every MLV file you're trying to mount, and for me this took upwards of 10-20 minutes for all my files. I was pulling my hair out until I realized I just had to be patient.

Many thanks to the author. This functionality is primo, top shelf, keep up the great work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 01, 2015, 08:17:56 AM
How could you edit defaultscale tag when it hasn, t been introduced since recently in mlvfs by dmilligan? Or is the tag included in mlv_dump, raw2dng?
Would be cool if you remember what you did exactly :).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on September 01, 2015, 06:30:35 PM
It's part of the DNG standard and has a default value of 1. I really thought I opened it up in TextWrangler and it was all there in XML format. Now when I open up a file all I see is gibberish. I can remember everything I did except how I edited the value in the first place! How are you all editing EXIF values anyway? Exif editor doesn't show the advanced metadata categories, defaultscale included.

Edit: So, I think the DefaultScale tag is misunderstood here. I'm not sure if MLVFS is doing this but I took a look at my 60p footage and the tag has values of H=1 V=1.6667. Not sure where these numbers are coming from. What we want to do is change this value to counter the "stretch" of 1728x606 footage, which is an aspect ratio of 2.85. We want an aspect ratio of 1.7778 (16:9) so the pixels will need to be taller than they are wide, does this make sense? That means the correct DefaultScale values should be H=1 V=0.624.

Problem is, exiftool returns "1 image files unchanged" every time I try to write the file, can somebody help?

Code: [Select]
exiftool -DefaultScale='1 0.624' test.DNG
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on September 01, 2015, 08:11:19 PM
Can you edit this file and test it in adobe premiere pro.
https://drive.google.com/file/d/0B4tCJMlOYfirOUlSdDR4Q043NHc/view?usp=sharing

 I didn,t get it to work in Ppro but it worked in adobe camera raw.

I suggest you write
Code: [Select]
exiftool -DefaultScale='1 1.6667' test.DNG to get correct values.


By the way here is the file with inserted defaultscale value set at
Code: [Select]
DefaultScale='1 1.6667'https://drive.google.com/file/d/0B4tCJMlOYfirdlJicS0yX1l3Z1U/view?usp=sharing
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on September 01, 2015, 08:48:50 PM
Danne not sure if you misread my post but it's editing files in general that I'm having trouble with.

This is my command
Code: [Select]
exiftool -DefaultScale='1 0.624' test.DNG
and I'm getting
Code: [Select]
Nothing changed in test.DNG
    0 image files updated
    1 image files unchanged

Any idea why this is?

Again, the 1 1.6667 are the wrong values to use to "de-stretch," that's why we want to change it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 01, 2015, 09:09:51 PM
Again, the 1 1.6667 are the wrong values to use to "de-stretch," that's why we want to change it.
Nope, those are the correct values. The stretch is due to pixel skipping. The pixel skipping is every 3rd column and every 5th row. You can only skip odd numbers of rows or columns or you'd end up with missing colors (either blue or red) due to the bayer array. This means the ratio is exactly 3:5.

I think you have your horizontal and vertical mixed up. Horizontal (x) axis is always listed first. That means 1, 1.6667 is the correct value (3:5 = 1:1.6667). The equivalent of that would be 0.6:1, which would also work, and I think it's close to what you are going for (0.624 is wrong though, should be 0.6 exactly b/c like I explain above, we know the exact stretch ratio is 3:5), but I figure it's better to stretch things larger than to compress them smaller.

I have tested this and it de-stretches correctly in ACR (but not PPro, which just ignores it).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on September 01, 2015, 10:35:59 PM
Ah, my mistake, thanks for the correction dmilligan, you are right. However I'm not sure I understand the 3rd column/5th row skipping reasoning for the PAR of 0.6. Regardless of what pixels the camera skips, the output (at least for my 5D3) is 1728x606 which is a 2.85 AR not 3.5. A PAR of .6:1 or 1:1.667 would give a 1.71 AR, not quite the 1.7778 we want, no?

Also, does anyone have any idea why I can't change any exif info using exif tool? I keep getting "1 files unchanged" returned. Verbose mode just says "nothing changed in test.DNG"
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on September 01, 2015, 11:12:45 PM
The amount the image is stretched is exact (it's 3:5 and I just proved why that is the case). Just because you are getting 1728x606 out doesn't mean that image is supposed to be 16:9 when "de-stretched" (which is the assumption you are making). The original recorded image area before stretch is not exactly 16:9 equivalent.

To get to a specific aspect ratio, you should "de-stretch" by the exact amount and then crop to get to the exact aspect ratio you want, otherwise you are slightly distorting the image either vertically or horizontally.

If you still don't believe me that the exact PAR is 3:5, then feel free to shoot a resolution chart (or something with a perfect square on it) and measure it exactly for yourself.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on September 02, 2015, 12:57:25 AM
Ah! That makes sense. so I'm losing a tiny bit more resolution at these slo-mo rates. Bummer. Anyway I can recall that I also changed either a crop size or an image size tag to "1920x1080" have you tried that? It's so bizarre because Ppro is clearly reading some kind of PAR info from the file as evidenced by the Interpret footage dialog. I wish I could remember what I did to make it work.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: poromaa on September 04, 2015, 04:47:32 PM
Hi, Is there anyway to enable saving a file in the virtual folder? Maybe a setting in the FUSE-thing? This would have been great because then you could grade first frame of the footage in lightroom and save the metadata in the virtual folder. Then load it in after effects and have the grading done.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on September 04, 2015, 05:52:34 PM
I have implemented it so that you can save separate files into the virtual folder such as XMP sidecar files (when you do this, they will appear in a ".MLD" directory next to the original MLV file). Overwriting the DNGs themselves is impossible.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on September 04, 2015, 06:14:56 PM
Hi, Is there anyway to enable saving a file in the virtual folder? Maybe a setting in the FUSE-thing? This would have been great because then you could grade first frame of the footage in lightroom and save the metadata in the virtual folder. Then load it in after effects and have the grading done.
Perhaps why not just do it all solely in AE with SmartImport2 while running MLVFS?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: festr on September 10, 2015, 09:50:50 AM
Hi Im experiencing vertical banding on my 5d mk3 but it is not corrected by the latest mlv fuse. I can get rid of them.only when I shoot to raw and use raw2dng utility which detects them in debug output and dng is clear. I even see vertical bands in mlvviewer. Is this known issue? What I can do?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on September 10, 2015, 10:27:20 AM
Did you forget to "refresh" web browser after clicking on Vertical stripes fix in order for MLVFS to take effect prior to rendering otherwise?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: festr on September 10, 2015, 11:49:19 AM
Oh I totally forgot that there is web settings and so as you suggested it is not enabled by default probably. I will check it once I get to my pc again. Thank you for the hint.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: iheartbmx on September 29, 2015, 09:07:24 PM
I apologize if this has been asked, but its hard to track down - Right-clicking my folders brings up nothing for services. My MLV's are located on an external SSD if that makes any difference.

- Macbook Pro / 10.9.5
- I've installed the latest versions of FUSE and MLVFS (Also deleted previous versions)
- Attempted directly from CF and also from external SSD

Thanks in advance!

*Edit - Services is now working, however - WebGUI opens, navigate to the footage directory, WebGUI page does not exist. I attempt again, WebGUI fails entirely. Double clicking the mounted folder does not work either, but it shows up on the desktop.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on September 29, 2015, 11:25:43 PM
How many GB of MLV in total? If lots of footage then give it some time for MLVFS to generate all the required .IDX files for them to show up on the WebGUI.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: iheartbmx on September 30, 2015, 08:08:51 AM
How many GB of MLV in total? If lots of footage then give it some time for MLVFS to generate all the required .IDX files for them to show up on the WebGUI.

64GB's; Ill try again and give it some time. Thx for the help.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on October 06, 2015, 06:39:50 AM
Is anyone else having issues using the files from MLVFS in Adobe Premiere? It takes several seconds for the image to update in the source/preview monitors and I get the "media pending" screen often. When I convert using another method, RAWMagic for instance, playback and workflow is smooth.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: budafilms on October 07, 2015, 09:12:42 AM
- I have installed MLFVS in OSX Yosemite. I upgrade to EL Capitan, and everything is working with out resin tall again the scripts.

- The only confusing thing for me using this tool is the steps and pop up windows from navigator. I choose the option that I think (Resolve, Vertical Corrections) and I reload the page. After that I drag and drop the files to a local disk. I supposed that settings is applied in my DNGs footage.

- The other question is if exist a DNG compress, to save space.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on October 07, 2015, 11:50:48 AM
Please accept my apologies if this has already been asked, I have attempted to search but to no avail.

Is there any chance of MLFVS being able to reference a pool of dark frames (1 for each ISO), and then subtract it on the fly?

I understand I can do it per clip with MLV_dump but would rather use a more efficient method. I am having a tough time bringing in FPN removal to my work flow. My subtract experiments work much better in AE (Davinci Resolve seems to massively darken image instead, maybe i'm doing it wrong). AE isn't so great for my .xml workflow and I would much rather be working Davinci.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 07, 2015, 01:04:17 PM
- The other question is if exist a DNG compress, to save space.
The DNGs are virtual, so they already take up 0 space. "Compressing" them would only slow down MLVFS and they would still take up 0 space. However, it sounds like you are using this tool to permanently convert the DNGs ("After that I drag and drop the files to a local disk"). This sort of defeats the whole intended purpose of MLVFS, therefore I see no reason to do all the work to implement something like this. If you are permanently converting the DNGs anyway, there are plenty of tools out there than can compress them, and there are other MLV converters that provide compressed DNGs.

Is there any chance of MLFVS being able to reference a pool of dark frames (1 for each ISO), and then subtract it on the fly?
Yes it's possible, and something I'd like to see added. Can't say when I might have time to implement it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: budafilms on October 09, 2015, 07:25:23 AM
@dmilligan

It's true what you said. I'm permantently converting and dragging DNG. I tried all the converters under OSX, and the differences in quality from this to other are big. Not only in the size, I just tried to export with the same setting and color post processsiong one DNG from MLVS and one from other converter and the difference it's huge in color, in noise, even in death pixels from one to other.

The way to check and clean unusual shots in my case it's after copy DNG to my local disk, I open Resolve and see the take. I can leave or permanently delete with a tool in Resolve - with the other click of the mouse over the shot. I know lot of people use MLViewer to do this but I get repeat freeze from that software and lot of problems.

My question to compress is in the local disk. As I understood, DNG could be compress and the quality loss it's not percieve at all. Other sofware do this, but as I said, this is the more stable for me.

Thanks for your tme and sorry for my english, I talk italian or spanish.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on October 17, 2015, 10:40:08 PM
My default browser is Chrome and after installing El Capitan I noticed that instead of the _PREVIEW.gif I'm getting a broken image icon. Don't know if it has to do with the system but it used to work fine.

Anyone else experiencing this?

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 18, 2015, 01:58:41 AM
Click the 'Load' button. Previews no longer load by default.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on October 18, 2015, 07:32:11 AM
Doh!

Works fine. Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: templar on October 19, 2015, 05:17:28 PM
@dmilligan: great work and cool idea to mount and use the stuff on-the-fly

Do you know how far behind the windows version is?
Has someone actually tested this on windows with Davinci Resolve?
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 19, 2015, 05:32:26 PM
@templar:

http://www.magiclantern.fm/forum/index.php?topic=13268.0
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 19, 2015, 06:58:31 PM
There are two different windows implementations, one written by g3gg0 that uses WebDAV, that is somewhat different from the FUSE version in terms of implementation and features, and one written by Ayshih that uses PFMAP that is similar and has a lot of code in common with the FUSE version. However, they are both rather far behind the FUSE implementation (Mac/Linux version). I think in both cases it's probably been a year since they've been updated.

There is another alternative that is rather complex, and involves running the Linux version of MLVFS in a virtual machine:
http://magiclantern.fm/forum/index.php?topic=13152.msg142308#msg142308
http://magiclantern.fm/forum/index.php?topic=13152.msg142641#msg142641
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on October 20, 2015, 03:56:22 AM
Is anyone having trouble with importing with MLVFS? Programs are hanging indefinitely when I try the open file dialog box for browsing the filesystem when an MLVFS disk is mounted. As soon as I eject the image the problem goes away. I've reinstalled MLVFS as well as FUSE. I'm on OSX.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 20, 2015, 04:51:28 AM
Which OS X are you running?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: templar on October 20, 2015, 01:41:58 PM
@dmilligan thanks for your reply: HM, ok I see.
I guess the bottom line is - since I am now running win10 and the win version of the mounting scripts are more than a year old it's not an option anymore for us windows users.

have you heard of any plans to update the mounting sw?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 20, 2015, 07:45:50 PM
I have no plans to update the Windows version, as I don't own a Windows computer (and don't plan to). This project is open source, so anyone is more than welcome to update it themselves. Ayshih was the one who did the PFMAP version originally. We haven't heard from him in a while...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on October 21, 2015, 01:35:28 AM
@Templar , Why don't you use The PC version of mlvfs ?

Install PFMAP (http://www.pismotechnic.com/download/archive/pfmap-171-win.exe)
Download mlvfs.dll (https://bitbucket.org/dmilligan/mlvfs/downloads/mlvfs.dll)
Then register the MLV formatter DLL:
Code: [Select]
pfm register mlvfs.dllUsage: right-click on an MLV file and select "Quick Mount".

Work great , just doesn't have the fancy web GUI in the Mac version.
I use MLVFS on MAC & PC , mind you if you need chroma smoothing , bad pixel , vertical stripes fix & dual ISO
not sure if PC version as it as I have a 5d2 and don't need any of those extra setting
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on October 21, 2015, 07:09:19 AM
I was testing out MLVFS with a camera that has the focus pixel issue with raw video. No problem dealing with this on straight MLV by simply applying Chroma Smoothing 2x2 and turning on Bad Pixel Fix but when I got into dual_iso footage I just couldn't seem to get rid of all the focus pixels. I'm going to resurrect a discussion from a year ago starting here (http://www.magiclantern.fm/forum/index.php?topic=13152.msg131584#msg131584) and hopefully someone could shed some light on the current situation.

Quote from: dmilligan
@DeafEyeJedi, those look like focus pixels, you need to turn on chroma smoothing to get rid of them. Right now you might get some weird artifacts with chroma smoothing, I need to incorporate it into the correct sequence within cr2hdr stuff. There are also several cr2hdr options that I have not implemented yet. These options, when implemented, should improve image quality greatly, and match the result of the standard cr2hdr.
Quote from: a1ex
Note that focus pixels with dual ISO are still unsolved, at least in my implementation (didn't try MLVFS yet, but will definitely do it). So try not using both at the same time - I know you can, from your signature :P
Quote from: dmilligan
Okay I implemented the chroma smoothing the same way it's in cr2hdr (chroma smooth fullres and halfres before final blending). It helps with the focus pixels, but doesn't remove them completely. If you do the chroma smoothing after cr2hdr is completely finished, the focus pixels are completely gone, but there are strange green blob artifacts.

I haven't updated the binaries yet (I'll wait till some more updates)

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on October 21, 2015, 07:44:20 AM
Thanks for bringing this up again @dfort!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Mehmet Kozal on October 21, 2015, 09:10:18 AM
Not about dual-iso but ,my post about this issue with regular MLV files and MLVFS | Mystic http://www.magiclantern.fm/forum/index.php?topic=15808.msg153823#msg153823
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: templar on October 21, 2015, 10:22:46 AM
@reddeercity thanks for the hint, did you test this on win10?

I assume, BSOD mean Blue Screen Of Death...  ;)
http://www.magiclantern.fm/forum/index.php?topic=13268.125
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on October 21, 2015, 03:10:12 PM
@dfort,
One possibility is mapping the focus pixels for each camera and then interpolating out those pixels (rather than just chroma-smoothing the entire image). This might fix the dual ISO / focus pixel issue, IDK. It should be faster than chroma-smoothing, and not have any artifacts. It's a lot of work though, you need to create a focus pixel map for each camera and video mode.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on October 21, 2015, 06:04:33 PM
Sounds like a good plan regardless of how long this will take to make. I'm happy to help, @dfort!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on October 21, 2015, 06:42:54 PM
I think that was the method used by the PinkDotRemover (http://www.magiclantern.fm/forum/index.php?topic=6658.0) tool. Never could get that tool to work for me.

Danne is also looking into some sort of focus pixel masking or mapping solution.

This is an "interesting" problem because it looks like chroma smoothing removes the focus pixels on non dual_iso raw footage but it doesn't seem to work with dual_iso. I was able to remove the focus pixels in MLVFS on dual_iso footage with the "Bad Pixel Fix" set to "Aggressive" but it also pretty much destroyed the image. The following frame was processed using the cr2hdr Lightroom plugin but the results were the same in MLVFS.

Focus Pixels showing up on EOSM MLV Dual_ISO cropped mode 1280x720
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm6.staticflickr.com%2F5765%2F22373110971_36823003be_z.jpg&hash=0f8b4934659bb94adf758ee185684d23)

Same frame using --really-bad-pix option ("Bad Pixel Fix" set to "Aggressive" in MLVFS)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm1.staticflickr.com%2F671%2F22362420585_229bb60771_z.jpg&hash=8e9ef65d906d9ff80f45f749ba19d67f)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on October 21, 2015, 11:51:31 PM
@reddeercity thanks for the hint, did you test this on win10?
Your welcome,
Yes , and to sure I just Tested it now and no problems.
(Windows 10 Pro on SSD Hyper-X, AMD FX8350-4.9Ghz , 8GB Ram 1600MHz, 2xGTX580-3GB Vram
On Large Files it may take a few minutes so be patient, make sure you are on the fastest drive e.g. SSD or raid etc.....
I assume, BSOD mean Blue Screen Of Death...  ;)
Yes ,
MLVFS windows client is not really working well right now , I would just stick with the simple "PFMAP" Version
for Windows 10 .
I wrote a post for Windows 10 + Magic Lantern Raw Video Tools (http://www.magiclantern.fm/forum/index.php?topic=15654.msg152335#msg152335) have a look at that so see what supported   
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: templar on October 22, 2015, 09:12:44 AM
cool, sounds great!

However, AMD processor tz tz tz tz  :)

yeah I do have a SSD. I will test this on my win10 machine with *wonder wonder* including a Intel CPU  8)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on October 22, 2015, 11:27:26 PM
@dfort,
One possibility is mapping the focus pixels for each camera and then interpolating out those pixels (rather than just chroma-smoothing the entire image). This might fix the dual ISO / focus pixel issue, IDK. It should be faster than chroma-smoothing, and not have any artifacts. It's a lot of work though, you need to create a focus pixel map for each camera and video mode.

Thanks again dmilligan - I'm taking off with that suggestion!

http://www.magiclantern.fm/forum/index.php?topic=16054.msg155923#msg155923
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: templar on November 04, 2015, 05:58:57 PM
Ok tested on win10 with davinci resolve. Works well.

Thank you! This is brilliant!!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: silenttapes on November 04, 2015, 07:46:30 PM
Hey!

I've been searching for information regarding CompressMLV.dmg and its integration into MLVFS:
-it says testing on the 1st post of this page, whats your experience of using it? (i do want to compress for archival and general space saving during project use)
-how much slower is it?
-would you recommend it?

A final unrelated question: has anyone worked out a way to keep your project MLVs only (auto delete unused ones, preferably via davinci)?

Cheers,
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: iheartbmx on November 07, 2015, 10:12:15 PM
Where could I download older versions? Thanks
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 08, 2015, 12:36:42 AM
Nowhere. Why?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: iheartbmx on November 08, 2015, 01:40:05 AM
Nowhere. Why?

Well I'm having issues with MLVFS:
-The web GUI fails when I navigate to the folder containing the MLV's
-If I try running MLVFS directly on the folder with MLV's it fails
-The mount on my desktop is acting like an icon and thats about it (clicking it does not bring up a folder)

I thought an older version may fix this issue. I made sure the directory is a blank folder, I tried the previous version of OSXFuse, and I've just upgraded to El Capitan.. but still no luck.

Any help would be much appreciated, Thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on November 08, 2015, 02:34:08 AM
Things you can try:

- figure out if it is a specific MLV file that is causing the problem (try mounting various combinations of the MLV files you have, a binary search would narrow it down to a single file the quickest)
- check for crash logs in ~/Library/Logs/DiagnosticReports
- run mlvfs from the command line and send me any output

Also you could be a little more specific about what you mean by 'it fails'. Do you get an error message, does it crash, does it just sit there, what exactly happens. If you have lots of MLV files, or really large ones, it make take a while for mlvfs to index them, so it may sit there for a while until the indexing is finished. You can check the folder with the MLV files in it for .IDX files being created.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: iheartbmx on November 08, 2015, 08:29:46 AM
Things you can try:

- figure out if it is a specific MLV file that is causing the problem (try mounting various combinations of the MLV files you have, a binary search would narrow it down to a single file the quickest)
- check for crash logs in ~/Library/Logs/DiagnosticReports
- run mlvfs from the command line and send me any output

Also you could be a little more specific about what you mean by 'it fails'. Do you get an error message, does it crash, does it just sit there, what exactly happens. If you have lots of MLV files, or really large ones, it make take a while for mlvfs to index them, so it may sit there for a while until the indexing is finished. You can check the folder with the MLV files in it for .IDX files being created.

Thanks, The web GUI works as normal until I click the folder name containing the MLV's, It brings me to a blank page like I'm not connected to the web.

 I have not tried running it from the command line as I'm not familiar with it. I pulled up the log but I'm not sure what to look for (It did say that thread#2 crashed). I copied one MLV to an external drive and ran MLVFS, and that worked fine. My assumption is that I filled the 32gb cf card to the point that MLVFS is not able to write anything else like the index files - is that even possible?.. My second 64gb card (that I didn't fill) worked fine with MLVFS. I should also mention the 32gb card was acting odd and has also done so in the past, and I though it was corrupt. I tried moving all the contents of the card to an external drive and was getting multiple error codes (-36, -43, -1407). Maybe my card is the issue here..? ugh..

I moved all the files to an SSD and then ran it through MLVFS and it worked fine.

Thanks again for all the help!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on November 08, 2015, 02:28:42 PM
Definitely sounds like the card is the problem. Can you please send me the crash log(s).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Fer5275 on November 30, 2015, 06:58:25 PM
Thank You!!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: monky on December 05, 2015, 11:11:26 PM
Hey Guys, im using mlvfs and its working like a charme for me only 1 thing seems to be strange.
The White Balance is not like it was in the shot its really blueish.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Ffs5.directupload.net%2Fimages%2F151205%2Ftemp%2Fmefnjacp.jpg&hash=07eba3a29fc2465bc3387f0212561b2a)
Am i doing something wrong? Or better is there an easy Fix for that? 
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on December 06, 2015, 12:34:47 AM
Could you have been shooting in awb? I think you can use all modes except awb, and if you use awb it will rely on whatever kelvin your camera i set on. A really good way to nail white balance is via the Auto adjust Kelvin + G/M in ML menu. http://www.magiclantern.fm/forum/index.php?topic=16283.msg158136#msg158136
Unfortunately GM and BA numbers aren,t computed in any converter that I am aware of at the moment.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: monky on December 06, 2015, 11:59:01 AM
yes i was shooting in awb. But it seems linke is just a problem in mlvfs the footage is showen correctly in mlvrawviewer rawmagic and so on.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on December 06, 2015, 01:16:44 PM
It,s not correct. It goes to default 5500k in those converters.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 06, 2015, 07:32:06 PM
Some updates:

Loading pages in the webgui is now asynchronous, so if the index files have not generated yet, the page will come up immediately, and the rows will say "loading" until MLVFS processes the corresponding file.

If you have a large set of files, help test the feature by deleting existing .IDX files and letting MLVFS regenerate them. The webgui should come up immediately, and the rows should load asynchronously.

Also I've added an experimental "deflickering" feature that will match the exposure of each frame by equalizing the median pixel values of each frame to some specified value. This hasn't been extensively tested, so please let me know if/how well it works. It also depends on your raw processor actually respecting the BaselineExposure tag, so if it doesn't work, this may be the reason. AFAIK, ACR seems to respect this tag, but it typically applies its own non-linear processing to images, making deflickering more unpredictable.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: monky on December 06, 2015, 09:32:51 PM
The new Version is a little problematic for me. At the Gui the Videoinformation didnt show up. In Davinci i get an error message of a full gpu drive and wrong previews
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 06, 2015, 10:05:16 PM
Could you be a little more specific, provide more detailed information and perhaps post some screenshots?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on December 06, 2015, 11:16:18 PM
At the Gui the Videoinformation didnt show up.
ok, think I fixed this part, try new download
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: monky on December 07, 2015, 07:02:45 PM
Loading inside the Gui is now okay for me.

But here are my Davinci Errors:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Ffs5.directupload.net%2Fimages%2F151207%2Fj2csfozj.jpg&hash=fb7f7d7fe1da2401f4de9ba6c50bdd47) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Ffs5.directupload.net%2Fimages%2F151207%2Fyhieenis.png&hash=eca7b4666acd52b1750190a12ee03cd8)
I run it on a Macbook Pro 2013 with an Geforce 650 Videocard.

*Edit*


Seems like its working again now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on December 07, 2015, 09:17:01 PM
Very cool stuff, deflickering. Just tried with a HDR file to see the reaction(dforts suggestion) value set to 5000 . ACR evens out the exposures quite good actually between iso 100-1600. I can see how good this will be with timelapses and frsp. Too bad dcraw doesn,t read these dng values.
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fs12.postimg.org%2F6cq12aaft%2FScreen_Shot_2015_12_07_at_21_16_24.jpg&hash=fba7ec46aaaf4654bb5bfdc706149e98) (http://postimg.org/image/6cq12aaft/)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 07, 2015, 10:17:10 PM
@monky,
I don't see how resolve running out of GPU memory could have anything to do with MLVFS.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: monky on December 07, 2015, 11:54:49 PM
hm dont know. But its happaning since i updated mlvfs and there are glitchy thumbnails of the video itself.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: anzecokl on December 12, 2015, 12:45:22 PM
I am having same problems with white balance as were described some time ago using Premiere Pro CC. Playing real time no problem, sound is in sync, everything just works, EXCEPT white balance, which is completely off no matter how i set it (awb, any of the presets or manual kelvin input). What am i missing?
Using MLVFS for me at least beats any other workflow!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on December 12, 2015, 01:25:32 PM
Post an MLV file that exhibits the problem
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johannsebastianbach on December 24, 2015, 03:29:13 AM
I'm sorry if this has been asked here before, but is there any chance to open .RAW files with it?
It's the only way for me to shoot 60p (MLV gives out pink frames) and MLVFS is essential if you don't want or can't have double gb for projects  ::) :)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on December 24, 2015, 11:26:03 PM
Perhaps give @Danne's MLP a chance on your Mac if you really want RAW instead of MLV? [emoji6]
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: lanternman on December 25, 2015, 04:32:22 AM
hi guys,

Is there a way to batch import all the MLV clips into PP CC without selecting the first DNG file of every MLV clip? I'm using 6D ML and can only record 6s per clip, hence it would be extremely tedious if I have to import each clip individually.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johannsebastianbach on December 31, 2015, 11:15:27 AM
There is a AE Script to do so. You probably would need to convert your cDNG footage in a editable format like ProRes or dnxhd first anyway: http://www.magiclantern.fm/forum/index.php?topic=12422.msg119832#msg119832
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 03, 2016, 07:40:42 PM
Updates:

Implemented (https://bitbucket.org/dmilligan/mlvfs/commits/753d7ea1ccf269ecd000f04b3e1fad0849d5de1a) a1ex's pattern noise reduction algo: http://www.magiclantern.fm/forum/index.php?topic=11787.msg158766#msg158766
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on January 03, 2016, 08:13:13 PM
Cool, looks like it worked with minimal changes.

Samples welcome, especially if the algorithm struggles (either if it fails to fix the pattern noise in some images, or if it actually introduces more stripes).
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 03, 2016, 08:51:13 PM
Hell Yeah ... Thanks for the update and Prosperous New Years to y'all!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 05, 2016, 02:26:31 AM
Cool, looks like it worked with minimal changes.
All I really had to do was switch the assumed bayer order, but it would be fairly easy to just read the bayer order metadata and adjust accordingly.

Samples welcome, especially if the algorithm struggles (either if it fails to fix the pattern noise in some images, or if it actually introduces more stripes).
Didn't try to throw anything tough at it, just something simple with really apparent pattern noise so I could just make sure it was actually doing something.

https://bitbucket.org/dmilligan/mlvfs/downloads/52020000_000000_orig.dng
https://bitbucket.org/dmilligan/mlvfs/downloads/52020000_000000.dng

100% crop, ACR, exposure +5 ev
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fbitbucket.org%2Fdmilligan%2Fmlvfs%2Fdownloads%2Fpatternnoise.jpg&hash=66ef874fa114b4ababb6ee998db4511e)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: baldavenger on January 05, 2016, 02:54:01 AM
I had a go with some noisy high ISO footage. In this case it introduced horizontal anomalies.

Original:

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FAfDhg4fh.png&hash=b2d1cabe2530a9d48dcded403149217c)

With Fix Pattern Noise selected:

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2Fao5b4QOh.png&hash=11825a2fae7f98c5a02527febe9c5fb6)

I tested on the same footage that had a dark frame mlv subtracted via mlv_dump. The dark frame subtraction worked well, but the horizontal anomalies where the same as before once Fix Pattern Noise was selected.

This download contains the 4 dngs:

https://www.dropbox.com/s/ai343f6xo7a6zpq/pattern%20noise%20test.zip?dl=0

I hope this can be of some help to you. As always, I really appreciate the great work you're doing :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: hjfilmspeed on January 05, 2016, 05:33:55 AM
Whoa!!! Is there anyway to test this on windows!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on January 05, 2016, 06:17:46 AM
http://magiclantern.fm/forum/index.php?topic=16422.msg159671#msg159671
Quote
Plus I show how to install MLVFS for windows including register the MLV formatter .dll

Or try running OS X with standard PC hardware. AKA Hackintosh. 2 machines running here.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 05, 2016, 08:37:54 AM
Or you can get a Linux VM (try a lighter one like Lubuntu) started in VirtualBox, extract the DNGs with MLVFS and put them in the shared folder.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Lars Steenhoff on January 05, 2016, 11:12:36 AM
Great progress !
The example from dmilligan looks super clean !
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: swinxx on January 06, 2016, 03:56:06 PM
hello!

i have a problem with the new version of mlvfs.. when i try to mount a dir with mlv files, i select the mountpoint (same hdd as ever) and then the files start to load (the dir is without idx files) then suddenly the mlvfs mounting points disappear suddenly? is my system broken?
thx
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 07, 2016, 01:04:52 AM
logs? example files that cause the problem? any sort of clues at all you can give me?

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on January 08, 2016, 04:12:03 AM
@baldavenger,
I think those artifacts are due to a bug on my part, not a result of a1ex's algorithm. I think I fixed it, will compile a new build soon. Working on some optimizations to make it faster.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: budafilms on January 08, 2016, 09:09:11 AM
I don't know if this helps to Devs and research: in the open source soft called HandBreak, in settings, are a couple of new function about Noise Reduction and works very well, results very similar as the noise reduction from Davinci Resolve Studio.

 ;)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on January 08, 2016, 09:36:31 AM
You mean for NR to be implemented into MLVFS, @budafilms?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: MortenSJ on January 19, 2016, 04:19:50 AM
Quick question. Am i supposed to go into the mounted folder/drive and copy the MLV folders to another folder?

I get the mounted drive and when i click it, i see my folders, but i'm not really sure what to do after that?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 19, 2016, 11:03:39 AM

Quick question. Am i supposed to go into the mounted folder/drive and copy the MLV folders to another folder?
You do what you want, you can copy the DNG files somewhere else and use those (keep MLV or not), or use them straight from that folder (realtime decoding FTW!).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ArthurX on January 19, 2016, 03:32:48 PM
Hi All,

Thanks for the great work you are all doing. I use MagicLantern for a long time. But I am new to Raw ML video. I use Debian and try to install mlvfs,
but get this error message:
Main point can not  find fuse.h and
hdiutil

Code: [Select]
main.c:33:18: fatal error: fuse.h: No such file or directory
 #include <fuse.h>
                  ^
compilation terminated.
Makefile:35: recipe for target 'mlvfs' failed
make: *** [mlvfs] Error 1
make: Leaving directory '/usr/local/src/mlvfs/mlvfs'
cp: cannot stat ‘mlvfs/mlvfs’: No such file or directory
/usr/local/src/mlvfs/build_installer.sh: 22: /usr/local/src/mlvfs/build_installer.sh: hdiutil: not found
/usr/local/src/mlvfs/build_installer.sh: 23: /usr/local/src/mlvfs/build_installer.sh: hdiutil: not found
/usr/local/src/mlvfs/build_installer.sh: 28: /usr/local/src/mlvfs/build_installer.sh: hdiutil: not found
/usr/local/src/mlvfs/build_installer.sh: 29: /usr/local/src/mlvfs/build_installer.sh: hdiutil: not found

I looked around to find the hdiutil package, couldnt find it myself.

thanks in advance,
Greetings Arthur
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 19, 2016, 04:00:53 PM

Code: [Select]
main.c:33:18: fatal error: fuse.h: No such file or directory
-> https://packages.debian.org/search?searchon=contents&keywords=fuse.h&mode=path&suite=stable&arch=any (https://packages.debian.org/search?searchon=contents&keywords=fuse.h&mode=path&suite=stable&arch=any) -> apt-get install libfuse-dev

Code: [Select]
/usr/local/src/mlvfs/build_installer.sh: 22: /usr/local/src/mlvfs/build_installer.sh: hdiutil: not found
No you don't run the .SH, is for Mac (I guess), anyway, after a successful compile, put mlvfs in /usr/local/bin/ and that's it
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 19, 2016, 04:03:37 PM
Code: [Select]
apt-get install libfuse-devOr something like that.

The build_installer.sh is only for Mac. It creates a Automator workflow and Mac dmg using a Mac utility called hdiutil.

To "install" MLVFS on Linux, just 'make' and then copy the executable somewhere on your path.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ArthurX on January 19, 2016, 06:47:56 PM
Milligan and Licaon,
Thanks a lot. It works like a charm. :) :D
Have to start using it now. The video is playing in the localhost:8000
Arthur
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ibrahim on January 19, 2016, 07:27:39 PM
I've installed the files pfm-178-win-exe and mlvf.dll and put them in one and the same folder. From the thread I learned that the v.1.7.8 is more stable for my win 7.

Might be a stupid question, but how exactly do I carry out the following step:

Quote
Then register the MLV formatter DLL:
Code: [Select]
pfm register mlvfs.dll
Unlike the FUSE approach on the other platforms, you mount each MLV file individually: right-click on an MLV file and select "Quick Mount".
If you wish, you can build the MLV formatter DLL yourself by installing the Windows SDK and running nmake using the SDK command prompt in the mlvfs/win directory.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on January 19, 2016, 08:28:27 PM
I've installed the files pfm-178-win-exe and mlvf.dll and put them in one and the same folder. From the thread I learned that the v.1.7.8 is more stable for my win 7.
Might be a stupid question, but how exactly do I carry out the following step:

Read this post , should help New PC Tools to replace ACR & A.E. with same or better results (Without Resolve) (http://www.magiclantern.fm/forum/index.php?topic=16422.msg159671#msg159671)

FYI , I use pfm-171-win-exe on my Win7 pro Laptop & Desktop Workstation works just fine.
If you have your MLV files on a SSD then mount it , it will work a lot better then off the memory card .
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ibrahim on January 19, 2016, 11:15:28 PM
Thanks.
I have installed pfm-178-win-exe and followed the first part of the video tutorial (installed the DNG codec and registered the dll).

When I mount the .MLV RAW footage the DNGs are exported into c drive > volume. However, I still see horizontal lines (dual ISO); it hasn't been processed. What's the missing step?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 19, 2016, 11:21:59 PM
Windows version is way behind and does not support dual ISO conversion.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 19, 2016, 11:55:09 PM

Windows version is way behind and does not support dual ISO conversion.
Virtualbox (https://www.virtualbox.org/wiki/Downloads) -> Lubuntu (https://help.ubuntu.com/community/Lubuntu/GetLubuntu) -> MLVFS -> Shared folder (http://virtuatopia.com/index.php/VirtualBox_Shared_Folders) -> better?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ibrahim on January 20, 2016, 12:07:20 AM
Virtualbox (https://www.virtualbox.org/wiki/Downloads) -> Lubuntu (https://help.ubuntu.com/community/Lubuntu/GetLubuntu) -> MLVFS -> Shared folder (http://virtuatopia.com/index.php/VirtualBox_Shared_Folders) -> better?

Is this the workflow to handle dual iso in windows or in linux?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 20, 2016, 12:22:22 AM
Is this the workflow to handle dual iso in windows or in linux?
You will export the processed dual-iso files in Linux and you can copy them to Windows.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ibrahim on January 20, 2016, 12:27:13 AM
Using linux on my windows and then return to windows?... :D Sounds confusing. :(
Is there another way to remove noise from my footages without using dual iso+ETTR? That is why I am trying to use dual iso+ETTR.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on January 20, 2016, 01:04:47 AM

Using linux on my windows and then return to windows?
It gets the job done.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on January 20, 2016, 07:17:25 AM
Hello everyone,

     I'm having a difficult time with MLVFS. I have vertical banding in my shots, and even with the vertical stripes fix on, the dng files are not changing. I am shooting on a 7D. Is there a step I'm missing? It seems when the directory pops up on Safari everything is fine, but even with Chroma Smoothing 2x2 I get weird purple and green artifacts, almost like dead pixels. In MlRawviewer (I know it's not developed anymore) I don't get vertical stripes. I'm guessing since there is less detail in this program and because the vertical stripe fix works in MLRawviewer. Can anyone help me? Thanks.

- Paul Harwood
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 20, 2016, 03:57:10 PM
AFAIK, the 7D's vertical stripes and the 5D3's are slightly different in nature. MLVFS incorporates the vertical stripes fix created by a1ex for 5D3. MLRV has a different algorithm developed by baldand who was a 7D user and created it mainly for 7D.

People with 5D3s who use MLRV actually have the opposite problem you're having.

Chroma smoothing will not fix vertical stripes (a luminance issue).

It might be a good idea to post and example file.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on January 20, 2016, 06:24:01 PM
Here are some DNG's from 7D recently...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1528%2F23876614673_c46996cff2_n.jpg&hash=5e040ef7733a5e7b0041dda4a0ac7b08) (https://flic.kr/p/CnTTGe) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1646%2F24503424065_d2ac307289_n.jpg&hash=ea4779e81985256335b20c11b03b476c) (https://flic.kr/p/DkhshH)

https://mega.nz/#F!6okHxJIZ!Mu8084Iw31wxf9Cut9ryEw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johannsebastianbach on January 20, 2016, 07:20:25 PM
What do you guy's think about a resizing option? I already love that MLVFS is automatically stretching the DNG's from 1280-mode in the right aspect ratio and it could be useful in conforming the size already before cutting (e.g. for multicam shots or if you need to downsize the framing to get more fps or so), right?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 20, 2016, 08:29:20 PM
You can't really resize raw data until after you demosaic it (you could downsample it, but that would potentially introduce aliasing, and why do that anyway?) and then it's not raw data anymore. Pixel aspect ratio is just metadata. We are telling the raw processing software that it needs to do the stretch, but we can't actually do the stretch ourselves (we don't demosaic raw data, and I don't plan on adding any such feature).

Only additional thing that would be possible is specifying a custom pixel aspect ratio for anamorphic shooters.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johannsebastianbach on January 20, 2016, 10:12:16 PM
You can't really resize raw data until after you demosaic it (you could downsample it, but that would potentially introduce aliasing, and why do that anyway?) and then it's not raw data anymore. Pixel aspect ratio is just metadata. We are telling the raw processing software that it needs to do the stretch, but we can't actually do the stretch ourselves (we don't demosaic raw data, and I don't plan on adding any such feature).

Only additional thing that would be possible is specifying a custom pixel aspect ratio for anamorphic shooters.

That makes sense, learned something new, thanks :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on January 21, 2016, 01:48:18 AM
Hey dmilligan and DeafEyeJedi,

     I think I found a solution in MLVFS. I turned off Chroma Smoothing, kept Vertical Stripes fix on, and I switch Bad Pixel Fix to Aggressive. This seemed to eliminate most of the vertical stripes. I could use MLRawviewer to export cdngs, but MLVFS is so fast and convenient. Do you know what ISO settings I should try to stay at? And is there a way to set the EV and Black Levels in the camera to avoid vertical stripes all together? I read this online, but I don't quite understand it. Any more help would be greatly appreciated, and thank you for all that you have already done.

- Paul Harwood
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on January 21, 2016, 02:43:12 AM
You need to be more specific about the type of vertical stripes you are talking about because it is not clear (which is why I asked you to upload a sample).

I'm not sure what you mean by "EV" which is typically used as a unit of measurement based on a logarithmic scale. Perhaps you mean "EC" (exposure compensation)?

Black level is not something you can set or adjust either. It is piece of metadata need by the raw processing software (e.g. ACR, Resolve, etc.) to correctly interpret the raw data. It is either correct (matches the actual physical black level of the sensor) or incorrect (does not match). If it is wrong, your image will look totally screwy and have extreme color casts. This is something that you shouldn't need to fool with and should be totally transparent to you as an end user.

There are posts in this forum about how to "fix the black level". This is only necessary if you have footage with incorrect black level metadata (as a result of an ML bug). But this does not sound like your issue.

Discussion about choosing ISO or optimizing exposure is really outside the scope of this thread.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on January 21, 2016, 09:09:49 PM
Hello dmilligan,

     Thanks for your reply. Here is a link to the vertical stripes issue I'm having with the 7D: https://drive.google.com/open?id=0B7-vwCkJCer5TmRFVnBiOWJsX0E

If you could take a look and provide any input that would be awesome. Thanks.

- Paul Harwood
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on January 21, 2016, 09:39:31 PM
Sorry everyone, here is the link, but now it is shareable to all of you: https://drive.google.com/file/d/0B7-vwCkJCer5TmRFVnBiOWJsX0E/view?usp=sharing
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bwinter88 on January 29, 2016, 01:43:09 AM
I'm having trouble loading After Effects projects with MLVFS image sequences in them. I've been sitting here for 10 minutes and the Open Project progress bar is at 32%. mlvfs process in activity monitor shows 99% CPU. In fact, anytime I try to open sequences in Premiere or AE, the program hangs and MLVFS process revs up the CPU for 1-2 minutes before the program resumes. Is this normal behavior? I'm on a Mac Pro 2010 with OS X 10.11.2.

I updated mlvfs and FUSE to the latest version a couple days ago. I think something changed? Although performance has always been a little slow. For instance every time I want to modify source settings of an MLVFS image sequence (open up ACR dialog in AE for instance) I have to wait about 60 seconds until the window appears.

I know it's not a hard drive latency issue because it's on an external RAID that blackmagic disk speed test gives a 240MB/sec read speed. I've also accessed these files before from an external USB 3 drive and performance was actually better then. (!)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on January 29, 2016, 06:21:38 PM
Hey bwinter88,

     This might seem silly, but I think it's worth trying. Try uninstalling Fuse and MLVFS, and reinstalling. But, use the versions that worked before you experienced crashes. To be honest, I've had to downgrade from El Capitan to Mavericks because I was experiencing so many crashes. It works a lot better. But, uninstalling and reinstalling the previous versions of FUSE and MLVFS might just do the trick. I hope this helps!

- Paul Harwood
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: aschille84 on February 02, 2016, 10:04:09 PM
Is MLVFS resizing 1280 footage in DR12? Seems like I have to stretch the footage or is there any other way to do it?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: aschille84 on February 02, 2016, 10:20:24 PM
Hi johannsebastianbach, maybe you know something about this?

Anyway, found a way. In "Input scaling presets" in project settings, chose "stretch to fit corners", and voila! It stretches to the timeline resolution.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: johannsebastianbach on February 03, 2016, 01:15:18 AM
Interesting, thanks!!


At this point, I would like to ask you something guys:

I read quite often in this forum (also here: http://www.magiclantern.fm/forum/index.php?topic=16542.msg161275#msg161275) and also have myself the problem, that even with global draw off, the maximum performance I can do with MLV's is 1920x490 on 50p. Everything above is making me ocasional (60p) and extremely often (GD or higher res.) pink frames which are ruining the clips. RAW Module on the other hand works flawlessly even with 67fps!

While I shoot slow motion most of the time and without sound anyway I don't need MLV, but there is a huge advantage - it can be opened by MLVFS for my so beloved super space-saving workflow. It really makes a difference, if you have a 1 Terabyte project going on and don't have to copy your footage to CDNG's before formatting to ProRes for example. I even tried renaming footage.RAW to footage.MLV and opening it with MLVFS hehehe Would it somehow be possible to do so :)?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 03, 2016, 02:56:29 PM
I won't support the RAW format as is. I don't believe it should be used in its current state for anything other than development purposes. However, MLVFS is open source, so you are free to modify it as you see fit and add support for whatever format yourself.

My goals and intentions for working on this project are probably at lot less pragmatic than most general users (who simply want something that works and gets the job done). My goal is to help drive the project forward: improving the quality of the code, increasing my understanding of ML and the internals of the underlying cameras (and helping others do the same). By not supporting the RAW format, I hope to encourage users not to take the easy way out, and actually try and solve the problem and move the project forward.

Faced with your problem, which of the two following paths to a solution do you think helps to move the project forward?
1. Just use the old raw_rec
2. Investigate and fix the pink frame issue with mlv_rec or propose better, alternate solutions



Since it does seem that there is a real resource constraint when using mlv_rec at high FPS or while using CPU intensive ML overlays, perhaps there is something that can be done without incurring any extra overhead that solves the following issues I have with the old RAW format:
1. The file extension should not be .RAW (this is way too generic, and users think they can open it with something like photoshop), it should be something unique
2. File header information and metadata should be stored at the beginning of the file, not at the end
3. There should be more metadata than is currently provided (at the very least, camera model, ML version, etc.)
4. There should be header information in every file if it is split into chunks

I see two possible solutions to address these issues:
1. Create a whole new file format
2. Create a special extension to the MLV format that allows us to write the extra metadata and whatnot we need at the beginning of the file (using existing defined MLV block structure), and then after that simply dump frames sequentially into the file like the raw format (maybe we need a different file extension if we do this?).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 03, 2016, 04:12:47 PM
Quote
2. Create a special extension to the MLV format that allows us to write the extra metadata and whatnot we need at the beginning of the file (using existing defined MLV block structure), and then after that simply dump frames sequentially into the file like the raw format (maybe we need a different file extension if we do this?).

+1
And if this could be used straight in MLVFS it would be great.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 06, 2016, 05:13:09 AM
Exciting news for Windows users: https://bitbucket.org/dmilligan/mlvfs/issues/7/dokany-support

We're also close to having the focus pixel issue solved. New builds soon.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on February 06, 2016, 06:31:07 AM
Hello everyone,

     I have had some issues today with handling my .mlv files with MLVFS. I am running OS X Fuse 2.8.3.

      For some reason, After Effects CC 2015 (Version 13.7 - most recent) is not communicating well with my Macbook Pro, Late 2013, with a Nvidia GeForce GT 750M Graphics Card, and MLVFS.

     I removed OS X Fuse in System Preferences, and reinstalled, but I still had the same issue.

     (A few weeks ago I had downgraded from OS X El Capitan 10.11.3 to OS X Mavericks 10.9.5. The main reasons were because of many crashes, force quits, and Nvidia Cuda wasn't communicating with Premiere Pro CC 2015 when editing. - wasn't showing effects/dissolves, and I was forced to use OpenCL by OS X El Capitan. So I had downgraded to Mavericks since when I bought my Macbook Pro that was the operating system that was installed. Apparently the operating system the computer is initially installed with works the best, from what I've read.

    So after still having After Effects freeze with MLVFS, I called Adobe, and they told me the recent After Effects updates require OS X Yosemite 10.10.5. (This is odd since last week they told me Mavericks 10.9.5 was the minimal required version.)

     This afternoon I did a clone of my internal drive, made a bootable OS X Yosemite drive with Disk Maker X, did a clean install of 10.10.5, and installed my programs and files from the clone I made of my internal drive.

     After all of this, I still had the same exact issue, now with OS X Yosemite 10.10.5.

     So I called Adobe, and they told me I might need to convert the dng files to TIFs, and then import those. This completely disregards the point of MLVFS and an online directory's convenience. I have done this in the past, and it is quite time consuming, and takes up a lot of hard drive space.

    I asked Adobe if a clean install of Adobe Creative Cloud and the programs would help. They said it was worth a try.

    So, I uninstalled all Adobe applications and the Creative Cloud Desktop Manager. I couldn't get the desktop manager to reinstall, so I had to reinstall OS X Yosemite 10.10.5.

    I now have all my Adobe programs working and Creative Cloud Desktop Manager as well. However, I am still having After Effects give the spinning beach ball with MLVFS.

   
    The only other possible reasons I can think of that may have caused MLVFS to act odd are:

    Using OnyX software to clean my computer, possibly deleting important files (prior to this morning).

    The folder's name was changed and moving it back and forth between the internal drive and a G Drive Thunderbolt/USB 3.0, possibly confused files.

    But what's weird is, MLVFS loads the files rather quickly. Google Chrome and Safari are pretty equal in speed, so using one over the other wasn't the issue. But I cannot copy the MLVFS files and put them on another hard drive now, which I could do before (I can but it says it will take 18 hours for 64 gb, which makes no sense).

    I'm using Bad Pixel Fix at Aggressive, Vertical Lines Fix On, and Fix Noise Pattern On.

    Bottom line, something's not right. If anyone has any input on what I can do to make MLVFS and After Effects run smoother I would greatly appreciate it. I apologize if what I typed is a bit confusing, I tried to make it as clear as possible. I just wanted to make sure I included everything I tried today in order to pinpoint what might be causing the problem. Thanks so much.

- Paul Harwood
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 06, 2016, 01:49:34 PM
Pattern noise fix is still somewhat experimental and VERY slow. The bad pixel fix will slow things down a little too.

Bad pixel fix and focus pixel fix are two different things. Focus pixel fix is pretty fast and is default to on for cameras that need it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: PaulHarwood856 on February 06, 2016, 03:38:33 PM
Hey dmilligan,

     I changed the settings on MLVFS to just Vertical Stripes fix, and everything is so much faster now! Thank you so much, I'm so relieved!

- Paul Harwood
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: andy kh on February 06, 2016, 11:28:58 PM
very exciting as a window user
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: hjfilmspeed on February 07, 2016, 01:41:58 AM
I've been using on windows without a hitch. I had some issues with ACR but that's it. I have been mounting the MLVs right on the card, then compressing them with slimraw raw right from the mounted DNGs! So basically I now turn my transfer stage into a convert to DNG, compress, and transfer to hard drive all it one step! Saves soooooo much time!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 07, 2016, 04:38:55 AM
The latest MLVFS is now available for Windows!

Thanks to g3gg0 and the dokany (https://github.com/dokan-dev/dokany) project, MLVFS now works on Windows just like the Mac/Linux FUSE version.

Download: MLVFS_x86.zip (https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip)
You'll need to install Dokany (https://github.com/dokan-dev/dokany/releases/download/v0.8.0/DokanInstall_0.8.0.exe)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: andy kh on February 07, 2016, 05:21:03 AM
thank you so much@dmilligan & g3ggo for this wonderful mlvfs. love it
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on February 07, 2016, 08:32:29 AM
Cool!  :D Need to check this out. Hot pixel fix plus pattern noise reduction avaliable for windows now?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 08:42:13 AM
Am I the first one who really tried?
Dokany installed. I see mounter.exe process launched from Dokany install dir.
Here is my attempts to launch mlvfs:
Code: [Select]
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>D:\_soft\4\MLVFS\MLVFS_x86\mlvfs.exe
MLVFS: no mlv path specified


MLVFS options:
    --mlv-dir=%s
    --port=%s
    --resolve-naming
    --cs2x2
    --cs3x3
    --cs5x5
    --bad-pix
    --really-bad-pix
    --stripes
    --dual-iso-preview
    --dual-iso
    --amaze-edge
    --mean23
    --no-alias-map
    --alias-map
    --prefetch=%d
    --fps=%f
    --deflicker=%d
    --fix-pattern-noise
    --version


C:\Users\VOLD>D:\_soft\4\MLVFS\MLVFS_x86\mlvfs.exe Z:\MLVFS --mlv-dir=Z:\out

C:\Users\VOLD>D:\_soft\4\MLVFS\MLVFS_x86\mlvfs.exe Z:\MLVFS
MLVFS: no mlv path specified


MLVFS options:
    --mlv-dir=%s
    --port=%s
    --resolve-naming
    --cs2x2
    --cs3x3
    --cs5x5
    --bad-pix
    --really-bad-p --dual-iso-preview
    --dual-iso
    --amaze-edge
    --mean23
    --no-alias-map
    --alias-map
    --prefetch=%d
    --fps=%f
    --deflicker=%d
    --fix-pattern-noise
    --version


C:\Users\VOLD>D:\_soft\4\MLVFS\MLVFS_x86\mlvfs.exe Z:\MLVFS --mlv-dir=Z:\out

C:\Users\VOLD>
If I try to start the exe with all requered params it just exits silently:(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on February 07, 2016, 09:00:11 AM
Is the MLVFS folder at the path and environment variables (http://www.computerhope.com/issues/ch000549.htm)?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 09:14:38 AM
@surami: It makes me wonder, why, but I tried with no difference. Did you get it working this way? If no, please, don't advice just guessing.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: surami on February 07, 2016, 09:58:30 AM
@mothaibaphoto: Sorry, I just guessed. I can't get it working till now, let's wait for a pro...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 10:07:48 AM
mlvfs <mount point> --mlv_dir=<directory with MLV files>
for windows there is only the possibility to mount as drive, so the path must be a drive letter

e.g.
mlvfs x: --mlv-dir=d:\raw
to mount on X:


edit:
clarified my post, didnt see that you've tried to mount into a dir.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 10:53:12 AM
Great, it works!!! Thanks to g3gg0, MLVFS for us, windowsers just around the corner !!!
Issues:
1. I can browse with explorer, but Bridge prompts that there is no free space on disc. If enter to folder with Bridge anyway, mlvfs.exe crashes:
Code: [Select]
Problem signature:
  Problem Event Name: APPCRASH
  Application Name: mlvfs.exe
  Application Version: 0.0.0.0
  Application Timestamp: 56b6ad0b
  Fault Module Name: VCRUNTIME140.dll
  Fault Module Version: 14.0.23026.0
  Fault Module Timestamp: 558ce3d9
  Exception Code: c0000005
  Exception Offset: 0000d960
  OS Version: 6.3.9600.2.0.0.256.4
  Locale ID: 1049
  Additional Information 1: 5861
  Additional Information 2: 5861822e1919d7c014bbb064c64908b2
  Additional Information 3: a10f
  Additional Information 4: a10ff7d2bb2516fdc753f9c34fc3b069
And Resolve crashes mlvfs.exe too, without promps.

2. If I try to open http://localhost:8000/ I get messages in cmd window with started mlvfs:
Code: [Select]
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>D:\_soft\4\MLVFS\MLVFS_x86\mlvfs.exe X: --mlv-dir=Z:\out
webgui: scanning Z:\out/...
load_resource: fopen error
webgui: scanning Z:\out/favicon.ico...
load_resource: fopen error
webgui: scanning Z:\out/favicon.ico...
load_resource: fopen error
Page in browser stays empty.
I tried AE too: it imports and renders without problems...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 07, 2016, 12:03:55 PM
For windows version Web based GUI works but strangely all fields in "Configuration Options" are empty or not checked. If you try to check any radio button the "Configuration Options" table looses it's last three lines (e.g. "Interpolation" and below, same in firefox, internet explorer and microsoft edge). Refreshing page gives all blank fields again. MLV files info is working, previews are working as well.

So the paratemets can not be changed from the web I guess. Did not try from CLI switches though.

Thanx for the great solution for ml raw editing :)
bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 12:10:54 PM
@mothaibaphoto:
i am using it on a folder with a dozen of faulty, old, new and weird MLVs without any crash or (unexpected) error.
but i am only using the windows explorer and picture viewer.

can you nail it down to either a specific MLV file that causes trouble or the tool that raises that errors?
maybe its a combination of two things, idk

you can add the option -d to mlvfs to show a debug log, please post/upload one when it's crashing.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 12:36:37 PM
It's a fresh and correct MLVs shot with last MKIII build, non-crop mode, nothing special. Windows explorer and even Adobe AE works flawlessly for me too. Here is what happens with Bridge:
Code: [Select]
        Attributes: 0 (0x0)
        Options: 96 (0x60)
CreateFile status = 0 - lastError = 2
###GetFileInfo 13838
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileStandardInformation
###Read 13838
ReadFile : \M05-1619.MLV\M05-1619_000005.dng from 4210688 len 2048
###Cleanup 13838
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Close 13838
Close: \M05-1619.MLV\M05-1619_000005.dng

###Read 13834
ReadFile : \M05-1619.MLV\M05-1619_000005.dng from 65536 len 1040384
###Read 13834
ReadFile : \M05-1619.MLV\M05-1619_000005.dng from 1105920 len 1040384
###Read 13834
ReadFile : \M05-1619.MLV\M05-1619_000005.dng from 2146304 len 1040384
###Read 13834
ReadFile : \M05-1619.MLV\M05-1619_000005.dng from 3186688 len 1026048
###Create 13839
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 18
###GetFileInfo 13839
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileBasicInformation
###Cleanup 13839
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Close 13839
Close: \M05-1619.MLV\M05-1619_000005.dng

###Create 13840
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 3
###GetFileInfo 13840
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileNetworkOpenInformation
###Cleanup 13840
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Close 13840
Close: \M05-1619.MLV\M05-1619_000005.dng

###Create 13841
CreateFile : \M05-1619.MLV\M05-1619_000005.xmp
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 1073741772 - lastError = 3
###GetFileInfo 13841
GetFileInfo : \M05-1619.MLV\M05-1619_000005.xmp
        result =  3221225524
###Cleanup 13841
Cleanup: \M05-1619.MLV\M05-1619_000005.xmp

###Close 13841
Close: \M05-1619.MLV\M05-1619_000005.xmp

###Create 13842
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 3
###GetFileInfo 13842
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileBasicInformation
###Cleanup 13842
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Close 13842
Close: \M05-1619.MLV\M05-1619_000005.dng

###Create 13843
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 2
###GetFileInfo 13843
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileBasicInformation
###Cleanup 13843
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Close 13843
Close: \M05-1619.MLV\M05-1619_000005.dng

###Create 13844
CreateFile : \
        DesiredAccess: SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 32 (0x20)
CreateFile status = 0 - lastError = 0
###GetFileInfo 13844
GetFileInfo : \
        result =  0
  unknown type:55
###FindFiles 13844
FindFiles :\
index from 0
FileMatch? : . (M05-1619.MLV,0,0)
FileMatch? : .. (M05-1619.MLV,0,0)
FileMatch? : M05-1617.MLV (M05-1619.MLV,0,0)
FileMatch? : M05-1619.MLV (M05-1619.MLV,0,0)
  =>return single entry
index to 1
###Cleanup 13844
Cleanup: \

###Close 13844
Close: \

###Create 13845
CreateFile : \M05-1619.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 32 (0x20)
CreateFile status = 0 - lastError = 2
###GetFileInfo 13845
GetFileInfo : \M05-1619.MLV
        result =  0
  unknown type:55
###FindFiles 13845
FindFiles :\M05-1619.MLV
index from 0
FileMatch? : . (M05-1619_000006.dng,0,0)
FileMatch? : .. (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619.wav (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619.log (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000000.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000001.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000002.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000003.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000004.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000005.dng (M05-1619_000006.dng,0,0)
FileMatch? : M05-1619_000006.dng (M05-1619_000006.dng,0,0)
  =>return single entry
index to 1
###Cleanup 13845
Cleanup: \M05-1619.MLV

###Close 13845
Close: \M05-1619.MLV

###Create 13846
CreateFile : \
        DesiredAccess: SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 32 (0x20)
CreateFile status = 0 - lastError = 0
###GetFileInfo 13846
GetFileInfo : \
        result =  0
  unknown type:55
###FindFiles 13846
FindFiles :\
index from 0
FileMatch? : . (M05-1619.MLV,0,0)
FileMatch? : .. (M05-1619.MLV,0,0)
FileMatch? : M05-1617.MLV (M05-1619.MLV,0,0)
FileMatch? : M05-1619.MLV (M05-1619.MLV,0,0)
  =>return single entry
index to 1
###Cleanup 13846
Cleanup: \

###Close 13846
Close: \

###Create 13847
CreateFile : \M05-1619.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_ATTRIBUTES|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 32 (0x20)
CreateFile status = 0 - lastError = 2
###GetFileInfo 13847
GetFileInfo : \M05-1619.MLV
        result =  0
  unknown type:55
###FindFiles 13847
FindFiles :\M05-1619.MLV
build_index: fopen error: Too many open files
###Create 13848
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 3
###GetFileInfo 13848
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileBasicInformation
###Cleanup 13848
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Create 13849
CreateFile : \M05-1619.MLV\M05-1619_000005.dng
###Close 13848
        DesiredAccess: Close: \M05-1619.MLV\M05-1619_000005.dng

FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 0 - lastError = 18
###GetFileInfo 13849
GetFileInfo : \M05-1619.MLV\M05-1619_000005.dng
        result =  0
        FileNetworkOpenInformation
###Cleanup 13849
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

###Create 13850
###Close 13849
CreateFile : \M05-1619.MLV\M05-1619_000005.xmp
Close: \M05-1619.MLV\M05-1619_000005.dng

        DesiredAccess: FILE_READ_ATTRIBUTES
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 2097152 (0x200000)
CreateFile status = 1073741772 - lastError = 3
###GetFileInfo 13850
GetFileInfo : \M05-1619.MLV\M05-1619_000005.xmp
        result =  3221225524
###Cleanup 13850
Cleanup: \M05-1619.MLV\M05-1619_000005.xmp

###Close 13850
Close: \M05-1619.MLV\M05-1619_000005.xmp

ReturnedLength 0
ReturnedLength 0
ReturnedLength 0
ReturnedLength 0
ReturnedLength 0
###Cleanup 13834
Cleanup: \M05-1619.MLV\M05-1619_000005.dng

ReturnedLength 0
ReturnedLength 0
Ioctl failed for wait with code 55.
Ioctl failed for wait with code 55.
Ioctl failed for wait with code 55.
Thread will be terminated
Ioctl failed for wait with code 55.
Thread will be terminated
Dokan Error: Ioctl failed with code 55
Ioctl failed for wait with code 55.
Thread will be terminated
Thread will be terminated
Thread will be terminated
Ioctl failed for wait with code 55.
ReturnedLength 0
Ioctl failed for wait with code 55.
Thread will be terminated
Thread will be terminated
Ioctl failed for wait with code 55.
Ioctl failed for wait with code 55.
Thread will be terminated
Thread will be terminated

C:\Users\VOLD>

And with Resolve:
Code: [Select]
FileMatch? : M05-1617_000572.dng (_PREVIEW.gif,0,0)
FileMatch? : M05-1617_000573.dng (_PREVIEW.gif,0,0)
FileMatch? : M05-1617_000574.dng (_PREVIEW.gif,0,0)
FileMatch? : M05-1617_000575.dng (_PREVIEW.gif,0,0)
FileMatch? : M05-1617_000576.dng (_PREVIEW.gif,0,0)
FileMatch? : M05-1617_000577.dng (_PREVIEW.gif,0,0)
FileMatch? : _PREVIEW.gif (_PREVIEW.gif,0,0)
  =>return single entry
index to 1
###Cleanup 0173
Cleanup: \M05-1617.MLV

###Close 0173
Close: \M05-1617.MLV

###Create 0174
CreateFile : \M05-1617.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 33 (0x21)
CreateFile status = 0 - lastError = 2
###FindFiles 0174
FindFiles :\M05-1617.MLV
index from 0
FileMatch? : . (M05-1617_000000.dng,0,0)
FileMatch? : .. (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.wav (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.log (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617_000000.dng (M05-1617_000000.dng,0,0)
  =>return single entry
index to 1
###Cleanup 0174
Cleanup: \M05-1617.MLV

###Close 0174
Close: \M05-1617.MLV

###Create 0175
CreateFile : \M05-1617.MLV\M05-1617_000000.dng
        DesiredAccess: FILE_GENERIC_READ
        ShareAccess: FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 128 (0x80)
        Options: 96 (0x60)
CreateFile status = 0 - lastError = 2
###Read 0175
ReadFile : \M05-1617.MLV\M05-1617_000000.dng from 0 len 4212736
###Cleanup 0175
Cleanup: \M05-1617.MLV\M05-1617_000000.dng

###Close 0175
###Create 0176
CreateFile : \M05-1617.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 33 (0x21)
Close: \M05-1617.MLV\M05-1617_000000.dng

CreateFile status = 0 - lastError = 2
###FindFiles 0176
FindFiles :\M05-1617.MLV
index from 0
FileMatch? : . (M05-1617_000000.dng,0,0)
FileMatch? : .. (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.wav (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.log (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617_000000.dng (M05-1617_000000.dng,0,0)
  =>return single entry
index to 1
###Cleanup 0176
Cleanup: \M05-1617.MLV

###Close 0176
Close: \M05-1617.MLV

###Create 0177
CreateFile : \M05-1617.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 33 (0x21)
CreateFile status = 0 - lastError = 2
###FindFiles 0177
FindFiles :\M05-1617.MLV
index from 0
FileMatch? : . (M05-1617_000000.dng,0,0)
FileMatch? : .. (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.wav (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.log (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617_000000.dng (M05-1617_000000.dng,0,0)
  =>return single entry
index to 1
###Cleanup 0177
Cleanup: \M05-1617.MLV

###Create 0178
CreateFile : \M05-1617.MLV\M05-1617_000000.dng
        DesiredAccess: FILE_GENERIC_READ
        ShareAccess: FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 128 (0x80)
        Options: 96 (0x60)
###Close 0177
Close: \M05-1617.MLV

CreateFile status = 0 - lastError = 2
###Read 0178
ReadFile : \M05-1617.MLV\M05-1617_000000.dng from 0 len 4212736
###Cleanup 0178
Cleanup: \M05-1617.MLV\M05-1617_000000.dng

###Close 0178
###Create 0179
CreateFile : \M05-1617.MLV
        DesiredAccess: SYNCHRONIZE|FILE_READ_DATA
        ShareAccess: FILE_SHARE_DELETE|FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 0 (0x0)
        Options: 33 (0x21)
Close: \M05-1617.MLV\M05-1617_000000.dng

CreateFile status = 0 - lastError = 2
###FindFiles 0179
FindFiles :\M05-1617.MLV
index from 0
FileMatch? : . (M05-1617_000000.dng,0,0)
FileMatch? : .. (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.wav (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617.log (M05-1617_000000.dng,0,0)
FileMatch? : M05-1617_000000.dng (M05-1617_000000.dng,0,0)
  =>return single entry
index to 1
###Cleanup 0179
Cleanup: \M05-1617.MLV

###Close 0179
Close: \M05-1617.MLV

###Create 0180
CreateFile : \M05-1617.MLV\M05-1617.wav
        DesiredAccess: FILE_GENERIC_READ
        ShareAccess: FILE_SHARE_WRITE|FILE_SHARE_READ
        Disposition: FILE_OPEN (1)
        Attributes: 128 (0x80)
        Options: 96 (0x60)
CreateFile status = 0 - lastError = 0
###GetFileInfo 0180
GetFileInfo : \M05-1617.MLV\M05-1617.wav
        result =  0
        FileStandardInformation
###Read 0180
ReadFile : \M05-1617.MLV\M05-1617.wav from 0 len 4096
###Read 0180
ReadFile : \M05-1617.MLV\M05-1617.wav from 630 len 512

C:\Users\VOLD>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 07, 2016, 02:14:11 PM
@bouncyball

What browser? IE? I noticed this too, there's something wrong with the jquery that reads the initial values, it still works to set the values of options though. The last three options are actually supposed to be hidden unless dual ISO is turned on, since they only apply to dual ISO.

Does it work with other browsers?

@g3gg0
I actually managed to mount into an empty directory (Win7 x86 VM)

@mothaibaphoto
You need to start MLVFS from the directory it's in (cd to mlvfs.exe dir before launching it) so it can find it's support files. That's why the webgui is blank.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 03:20:05 PM
@mothaibaphoto
You need to start MLVFS from the directory it's in (cd to mlvfs.exe dir before launching it) so it can find it's support files. That's why the webgui is blank.
I suspect I did something dumb :)
Ok, now webgui behaves exactly as @bouncyball described with Firefox and MSIE.
Command-line parameters seems works - I successfully tested --stripes on file with vertical stripes.
It's possible to open files one by one.
Attempt to select all files and copy to another disc - crashes mlvfs.exe, such as Bridge and Resolve.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 03:40:35 PM
@dmilligan:
repeated copying several times, worked properly for a while. selectively copying files is fine.

after maybe about 2k DNGs i got:
build_index: fopen error: Too many open files

so there is something troublesome indeed.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 03:54:26 PM
ok think i've found the reason.
it doesnt always close chunks. prepared a patch.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 05:59:19 PM
@dmilligan:
https://bitbucket.org/g3gg0/mlvfs/commits/48aa2f6f7434df7bdf05b33b33d1e6fe5621b8fb
https://bitbucket.org/g3gg0/mlvfs/commits/a6602a4b9cf649cfa58db3bc1db6e86946430e9c

those two address some issues that cause the mlvfs.exe to crash.
e.g. the .gif reading crashed in a reproducable manner when offsets were beyond end and debugging was enabled.
so i also sanitized parameters in .dng reading.

it also adds a lot of checks to malloc'ed areas to play safe.
i hope it will fix the reported problems.


still i am wondering how it will free allocated image buffers.
the memory footprint raises slowly but continuously when copying files.
memory is allocated via get_or_create_image_buffer and never free'd.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 05:59:59 PM
ah shiiit. VS doesnt take the indentation the project uses by default...
will fix.

thats the fixed version:
https://bitbucket.org/g3gg0/mlvfs/commits/594552cf4727b1f6acc291189565062de1b59b0d
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 06:24:41 PM
for those who want to try: mlvfs.exe (http://magiclantern.fm/modules/modules/mlvfs.exe/mlvfs.exe)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 07, 2016, 06:26:09 PM
@dmilligan
the problem shows up in every browser: firefox, msie, chrome, microsoft edge.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 07, 2016, 06:31:44 PM
@g3gg0

Image buffers should be getting marked when the file is closed and then disposed when new image buffers are needed. I recently changed some of this behavior so there may be some issues. All this code is rather convoluted b/c we try to be efficient as possible. For example if some program opens a file, reads part of it, closes it and then opens it again, we don't want to have to reprocess the frame, so we try to to be smart about keeping processed frames around for a bit after being closed. MLVFS was much simpler and efficient when we could just process the requested section of the file, but now we have processing options that need to process the whole frame at once (decompression, dual ISO, etc).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 06:34:27 PM
yeah, makes sense and fits to what ive seen.

proposal:
check if image buffers that are already allocated > x MiB and free the oldest ones to get < y MiB
every buffer has an age field (timestamp of the last access using time()? or some counter?)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 07, 2016, 06:48:57 PM
With last exe browsed some folders with Bridge(!!!), than run Resolve ... and get smiley bsod never seen before...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 07, 2016, 10:03:42 PM
no troubles here with LR :(

but when stumbling through the code, i found another thing that can be troublesome.
can you redownload the mlvfs.exe from the last link and try if its better?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 08, 2016, 12:46:14 AM
proposal:
check if image buffers that are already allocated > x MiB and free the oldest ones to get < y MiB
every buffer has an age field (timestamp of the last access using time()? or some counter?)
I've cleaned up and simplified that code quite a bit. Since its a linked list we just delete from the beginning of the list and add on to the end, so the oldest buffers are always at the beginning. Haven't verified that there aren't any leaks, so there could still be some, or leaks from other places.

Just curious, which version of dokany are you compiling against? I was using 0.8.0 and provided links for that version. Just want to make sure we're on the same page.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 01:04:56 AM
yeah it's the oldest one, but is it also the LRU one? hehe.

right, i am using 0.8.0
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 08, 2016, 01:46:05 AM
Yeah I know, but vast majority of the time programs access frame sequentially. I think KISS applies.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 01:46:18 AM
ok ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 08, 2016, 05:08:12 AM
new builds uploaded
* fixed the webgui issue on windows (windows file paths contain "\" and json doesn't like that)
* streamlined buffer management (and eliminated 'prefetch' option)
* lots of stability improvements from g3gg0


still todo: embed git version info on the Windows version like the Mac/Linux version
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 08, 2016, 12:54:55 PM
Last mlvfs.exe with Resolve:
Code: [Select]
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>cd /d D:\_soft\4\MLVFS\MLVFS_x86

D:\_soft\4\MLVFS\MLVFS_x86>mlvfs.exe X: --mlv-dir=Z:\out
mlvfs_wrap_read(1426): caught exception 0xC0000005

D:\_soft\4\MLVFS\MLVFS_x86>
Resolve can be downloaded here (https://www.blackmagicdesign.com/products/davinciresolve)
Start with any new project, Media Storage - try to navigate to any MLVFS folder - immediate crash of mlvfs.exe :(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 08, 2016, 01:47:43 PM
Hi Again,

Problem 1
E:\MLVFS>mlvfs E:\DNG --mlv-dir=E:\MLV
mlvfs_wrap_read(1440): caught exception 0xC0000005
mlvfs_wrap_read(1440): caught exception 0xC0000005

no difference if you mount it to drive letter or empty dir.
exceptions mostly caught when accessing WAV file. Wave shows correct size but I can not copy it. However I could copy it with the previous mlvfs version.

Problem 2
It seams that MLVs bigger than 2048mb (also MLVs + M00...) can not be mounted at all?! they are not shown on dir list. Checked all available versions of mlvfs.

OS: Windows 10 Pro x64, Dokan 0.8 (checked Dokan 1.0-beta and it shows the same behavior)

@dmilligan: Webui works. BTW it lists every MLV even the >2gb ones which are not mounted (seen) in file explorer.

regards
bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 03:10:20 PM
please redownload mlvfs.exe (http://magiclantern.fm/modules/modules/mlvfs.exe/mlvfs.exe) and try.
it will still crash but show an address that might help to find the exact crash position.

meanwhile i am downloading resolve.
guess it's some off-by-one-error or such with the wave decoder that causes the troubles.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 08, 2016, 03:20:18 PM
No output to CMD window, problem details from popup:
Code: [Select]
Problem signature:
  Problem Event Name: APPCRASH
  Application Name: mlvfs.exe
  Application Version: 0.0.0.0
  Application Timestamp: 56b89cfa
  Fault Module Name: StackHash_8e87
  Fault Module Version: 6.3.9600.17415
  Fault Module Timestamp: 54504b0d
  Exception Code: c0000374
  Exception Offset: PCH_80_FROM_ntdll+0x0003CEEC
  OS Version: 6.3.9600.2.0.0.256.4
  Locale ID: 1049
  Additional Information 1: 8e87
  Additional Information 2: 8e87aeb06e784aea107beb1d064656d5
  Additional Information 3: dc1c
  Additional Information 4: dc1c75fae54762c904db111a92126c62
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 08, 2016, 04:13:33 PM
Just accessing dir or accesing wav:
E:\MLVFS>mlvfs_debug E:\DNG --mlv-dir=E:\MLV
mlvfs_wrap_read(1440): caught exception 0xC0000005 at address 0x739AE21C

Sometimes (very rearly) DokanMounter service stops and needs to be started manually.
E:\MLVFS>mlvfs_debug E:\DNG --mlv-dir=E:\MLV
Unmount

!!! Checked on 3 PCs MLVs >2GB are not mountig at all and can not be seen in file explorer (win10prox64, unfortunatelly have no win7 or 8.1 to test it on) !!!

@mothaibaphoto: What ver of win do you use and do you have the problem with MLVs bigger than 2GB

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 08, 2016, 05:53:38 PM
@mothaibaphoto: What ver of win do you use and do you have the problem with MLVs bigger than 2GB
1. win 8.1x64
2. Yes.
Code: [Select]
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Users\VOLD>dir X:\
 Volume in drive X is DOKAN

 Directory of X:\

05.02.2016  16:18    <DIR>          M05-1617.MLV
05.02.2016  16:19    <DIR>          M05-1619.MLV
05.02.2016  16:20    <DIR>          M05-1620.MLV
05.02.2016  16:21    <DIR>          M05-1621.MLV
05.02.2016  16:29    <DIR>          M05-1629.MLV
05.02.2016  16:30    <DIR>          M05-1630.MLV
05.02.2016  16:31    <DIR>          M05-1631.MLV
05.02.2016  16:36    <DIR>          M05-1636.MLV
               0 File(s)              0 bytes
               8 Dir(s)               0 bytes free

C:\Users\VOLD>dir Z:\out
 Volume in drive Z is EVO850
 Volume Serial Number is F873-BE21

 Directory of Z:\out

08.02.2016  00:30    <DIR>          .
08.02.2016  00:30    <DIR>          ..
08.02.2016  00:30            11 836 M05-1617.IDX
05.02.2016  16:18       396 013 620 M05-1617.M00
05.02.2016  16:18     1 714 848 352 M05-1617.MLV
07.02.2016  20:54            31 216 M05-1618.IDX
05.02.2016  16:19     1 137 176 628 M05-1618.M00
05.02.2016  16:19     4 599 571 040 M05-1618.MLV
07.02.2016  17:55             1 684 M05-1619.IDX
05.02.2016  16:19        61 763 636 M05-1619.M00
05.02.2016  16:19       207 090 272 M05-1619.MLV
07.02.2016  17:55            12 088 M05-1620.IDX
05.02.2016  16:20       421 445 684 M05-1620.M00
05.02.2016  16:20     1 729 380 960 M05-1620.MLV
07.02.2016  20:54            14 248 M05-1621.IDX
05.02.2016  16:21       486 842 420 M05-1621.M00
05.02.2016  16:21     2 070 897 248 M05-1621.MLV
07.02.2016  20:54            31 384 M05-1622.IDX
05.02.2016  16:22     1 104 478 260 M05-1622.M00
05.02.2016  16:22     4 588 671 584 M05-1622.MLV
07.02.2016  20:54            23 056 M05-1623.IDX
05.02.2016  16:24       802 926 644 M05-1623.M00
05.02.2016  16:24     3 436 962 400 M05-1623.MLV
07.02.2016  20:54            60 280 M05-1624.IDX
05.02.2016  16:27     2 132 660 276 M05-1624.M00
05.02.2016  16:27     8 875 790 944 M05-1624.MLV
07.02.2016  20:54             3 568 M05-1629.IDX
05.02.2016  16:29       153 260 084 M05-1629.M00
05.02.2016  16:29       672 367 200 M05-1629.MLV
07.02.2016  20:54             2 464 M05-1630.IDX
05.02.2016  16:30        98 877 492 M05-1630.M00
05.02.2016  16:30       449 892 960 M05-1630.MLV
07.02.2016  20:54             2 944 M05-1631.IDX
05.02.2016  16:31       168 091 700 M05-1631.M00
05.02.2016  16:31       726 749 792 M05-1631.MLV
07.02.2016  20:54             2 740 M05-1636.IDX
05.02.2016  16:36        79 102 004 M05-1636.M00
05.02.2016  16:36       543 826 528 M05-1636.MLV
07.02.2016  20:54            15 832 M05-1639.IDX
05.02.2016  16:40       821 092 404 M05-1639.M00
05.02.2016  16:40     2 395 668 480 M05-1639.MLV
              39 File(s) 39 875 661 952 bytes
               2 Dir(s)  24 641 163 264 bytes free

C:\Users\VOLD>
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 08:22:44 PM
WAV: nailed it down and fixing it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 08:51:27 PM
please redownload mlvfs.exe (http://magiclantern.fm/modules/modules/mlvfs.exe/mlvfs.exe) and try.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 08, 2016, 09:42:55 PM
4GiB files also fixed: please redownload mlvfs.exe (http://magiclantern.fm/modules/modules/mlvfs.exe/mlvfs.exe) and try.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 09, 2016, 05:52:36 AM
Good news:
Resolve imports and plays great, with sound.
Not so good:
Big files still ignored. The directory list I posted recently looks the same.
Several consecutive refreshing of webGUI with changing parameters causes smiley BSOD. Reproduced several times - comp. restart - webGUI - attempt to save anything - BSOD.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 09, 2016, 01:39:32 PM
@g3gg0: Great! Wav plays nicely :). However, as mothaibaphoto mentioned above MLVs >2GB ignored again. Can you compile the test binary against the Dokan version 1.0-beta1? The new version gets rid of a DokanMounter service and adds windows mount manager support.

Several consecutive refreshing of webGUI with changing parameters causes smiley BSOD. Reproduced several times - comp. restart - webGUI - attempt to save anything - BSOD.

Think I nailed exact reason of it. Switching "Naming Scheme" from "Deafult" to "DaVinci Resolve" and after that accessing folders, every time, ends up with exception. Sometimes restarting DokanMounter service helps and you can remount the folder, but sometimes OS freezes and only hard reset is a solution.
Other parameters in webgui are working just fine and only "Fix Pattern Noise" slows the process down a little bit. I think It's a normal behavior.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 09, 2016, 03:53:00 PM
@bouncyball:
tried with a forged 19GiB file and that works well.
will have to do some tests what may cause that.

will try 1.0
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 09, 2016, 05:12:41 PM
tried with a forged 19GiB file and that works well.

hmm... will try to record some huge mlvs on exfat. From ~50 test files of mine worked none which is bigger than 2048mb. They just do not show up in mounted folder.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 09, 2016, 05:21:35 PM
@dmilligan:
What does exacty switching "Naming Scheme" from "Default" to "DaVinci Resolve"? Changes some cdng metadata in favor of resolve? Or something else?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 09, 2016, 06:26:30 PM
It changes how the DNGs that show up in the mount folder are named. If you use a particular naming convention that Resolve likes (one that mimics the naming convention used by BM cameras), then the parent folders of cDNGs show up as video clips in media browser.

It makes sense that it would be the source of issues on Windows since file naming rules are different (case insensitive, different path separator, etc.), and it's adding a layer of complexity on top of the path translations mlvfs is doing (FUSE sends mlvfs a file path, mlvfs has to figure out what data to send back).

forged 19GiB file
Heh, this made me think of using FUSE to generate fake, huge MLV files for testing. It could just take an existing small MLV file and basically just repeat the frame chunks over and over. Then we can have huge testing MLV files that don't waste a lot of hard drive space.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 09, 2016, 06:46:12 PM
Not trying to be the party pooper in here for Windows users ... I'm very excited for all of you and it's been a long wait!

However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work and how can I help troubleshoot this or at least get a log for you?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on February 09, 2016, 07:01:14 PM
...If you use a particular naming convention that Resolve likes (one that mimics the naming convention used by BM cameras)...
Speaking of DaVinci Resolve Naming Scheme and Black Magic camera file names--

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1653%2F24291908904_546bd19854_z.jpg&hash=5351a22a5fe0bb6be7511d46d3a8045a)

I shot this test today and the filename has a date for last month. The date is correct in the browser window.

More on DaVinci Resolve Naming Scheme--I found this information about how the file name is formatted:

Quote
File Naming Convention
The Blackmagic Cinema Camera uses the following file naming convention when recording video.
[Camera ID]_[Reel Number]_[yyyy-mm--dd]_[hhmm]_C[Clip number].mov
This is an example of how this would appear on the touch screen of the camera:
BMC01_1_2012-08-08_1631_C0002.mov QuickTime Movie Filename
BMC01_1_2012-08-08_1631_C0002.mov Camera ID
BMC01_1_2012-08-08_1631_C0002.mov Reel Number
BMC01_1_2012-08-08_1631_C0002.mov Date (2012 Aug 08)
BMC01_1_2012-08-08_1631_C0002.mov Time (16:31pm - 24hrs)
BMC01_1_2012-08-08_1631_C0002.mov Clip Number
For CinemaDNG files, the folder of the image sequence will also be named the same way.

If you check discussions from people that are using these files in post you'll find a lot of dissatisfaction. Besides the long filename the information that editors need to see, the clip number, is at the end of the name and depending on the editing system it might be cut off.

Still, the date should be correct and it would be an improvement to add the time.

If we take an MLV file as it comes off the camera:
M02-0828.MLV

The Black Magic naming would probably go something like this:
M_02_2012-08-08_1631_C0828

I'm not suggesting such a radical change because it isn't a very practical name for most editing systems and the "M" is way too generic for a camera name but at least the fields in the name have some meaning instead of being padded out with zeros. Just something to think about.

Maybe something like this?
M02-0828_02_2012-08-08_1631_C0828
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 09, 2016, 07:08:43 PM
How is folder/audio embed working with the dng files in the case of shortening the folder name but not the dng files? The longer extension of zeroes are there to give each frame a sequential number.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 09, 2016, 07:10:32 PM
I shot this test today and the filename has a date for last month. The date is correct in the browser window.
Ah yes. You found a bug :) The tm.month field is zero based (0-11), so we need to add 1 before we print it out. This is already corrected in some places, but you found a place it was wrong.

However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work
I think I know what's wrong: forgot to include some resource files in the Mac build. Will fix.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 09, 2016, 08:08:31 PM
@4Gib:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fpicload.org%2Fthumbnail%2Fwoccpwd%2Flargemlv.jpg&hash=e71da934018575ed76865ba14c1df902) (https://picload.org/view/woccpwd/largemlv.png.html)
for me it works :-S
no clue what the problem really is.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 09, 2016, 09:45:33 PM
However, I just wanted to share that I downloaded the latest MLVFS for Mac and now the web GUI won't work and how can I help troubleshoot this or at least get a log for you?
I think I know what's wrong: forgot to include some resource files in the Mac build. Will fix.
Actually, everything seems fine. Web GUI works just fine here.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 09, 2016, 09:52:15 PM
@dmilligan:
get_mlv_filename needs to be reworked.
currently it's not possible to create files in directories within a MLV file.
and we have to use dokan v1 since file creation doesnt work with 0.8.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 10, 2016, 03:23:38 AM
and we have to use dokan v1 since file creation doesnt work with 0.8.
v1 doesn't seem to work on my win7 VM, fuse_main() just returns 1 immediately

get_mlv_filename needs to be reworked.
I think all the path handling should be much better now
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 10, 2016, 09:34:50 AM
oh thats odd. works fine on my win 10.

wow you did a lot of improvements. nice.

the problem with virtual folder is still persistent.
for a requested file:

/path/to/some/file.mlv/with/some/subfolder/leads/to.txt

we have to iterate over every element and check if it is a (existing!) MLV.

/path
/path/to
/path/to/some
/path/to/some/file.mlv


and cut the remaining part
/with/some/subfolder/leads/to.txt

and recombine it again with the MLD
/path/to/some/file.MLD/with/some/subfolder/leads/to.txt


Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 10:11:20 AM
@dmilligan, g3gg0
You really did some magic to the source code :)

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 10, 2016, 10:12:48 AM
oh i am glad to read this.
then we are on a good path ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 02:06:28 PM
@dmilligan, g3gg0
You really did some magic to the source code :)

In the last uploaded mlvfs (which btw is linked againts debug versions of the vclibs)
works:
1. files >2gb!!! (strange but) - YES!  I have not changed anything in my system, same Dokan v0.8
2. webgui "Naming Scheme" - YES
3. webgui all other params - YES
4. wav - YES

not working:
1. webgui gif previews in "DaVinci Resolve" naming mode
I guess something related to win paths and naming again

bb
Redownloaded the exe after reading this and everything the same (:
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 04:38:56 PM
Redownloaded the exe after reading this and everything the same (:

Have you downloaded this file?
https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip (https://bitbucket.org/dmilligan/mlvfs/downloads/MLVFS_x86.zip)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 05:06:40 PM
Did some test copy of 4,918,485,826 bytes (1168 files) from one SSD to second SSD

result (average of 3 runs):
Dokan 1.0 - 58mb/sec, mlvfs.exe consumed ~13% of CPU (Intel Xeon E3 @ 3.4Ghz)
Dokan 0.8 - 52mb/sec, mlvfs.exe consumed ~12% of CPU

A little bit subjective and depends on hardware but v1.0 has some performance boost (maybe new fs driver optimizations?)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 05:10:11 PM
No, This one, as posted by g3gg0 (http://magiclantern.fm/modules/modules/mlvfs.exe/mlvfs.exe)
I tried this one too: it asks for vcruntime140d.dll, I find it and put to program dir, than ucrtbased.dll, than...
it just show the message "The application was umable to start correctly (0x000007b)." Click OK to close.
:(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 05:16:36 PM
yes it's linked against debug version of dlls.

get 2 files below and drop to the saem dir with mlvfs.exe:
http://www.dll-files.com/dllindex/dll-files.shtml?vcruntime140d (http://www.dll-files.com/dllindex/dll-files.shtml?vcruntime140d)
http://www.dll-files.com/dllindex/dll-files.shtml?ucrtbased (http://www.dll-files.com/dllindex/dll-files.shtml?ucrtbased)

oh forgot to mention get 32bit versions
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 05:19:13 PM
I find these files myself on that site, but it anyway don't run with message from my previous post.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 05:22:28 PM
If you got 32bit versions and still have an error message then something messed up in your system with ms c++ redistributables.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 05:29:54 PM
Ups, it seems I mixed versions, now it works and I see all the files !!!
Great thanks !!!
Edit:
I think it's meaningless to measure performance of the debug version: it definitely less. With that version Resolve plays with 24 FPS(real time) with this - 14.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 10, 2016, 06:03:56 PM
Good point. We'll see... happy bright future is almost over here! Not only for MAC users ;)

Just kidding, have 2 old hackintoshes myself. But love doing/testing video/audio things under win or linux.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 10, 2016, 06:33:15 PM
125MiB/s for 19GiB with plain old HDD and my x64 build (http://magiclantern.fm/modules/modules/mlvfs_x64_rc.exe/mlvfs_x64_rc.exe) for dokan v1.0-rc ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 06:47:16 PM
Wow...
And where did you get that dokan v1.0-rc?
here:
https://github.com/dokan-dev/dokany/releases
only 1.0.0-BETA1
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 10, 2016, 06:49:27 PM
yeah sorry, its the beta1, you are right.
thats the correct one.

here some more optimized builds (http://magiclantern.fm/modules/modules/mlvfs_optimized.zip/mlvfs_optimized.zip)
choose the one that works for your CPU.

AVX2 is the best of course, but not all CPUs support it.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 10, 2016, 07:44:06 PM
Congratulations, you did a great job,
Great thanks to everyone involved!!!
Now it would be perfect if some more people try this asap to localize possible issues.
I tried for now only 64 build without avx(as i have old bloomfield) -
Resolve plays 30 fps FHD project in full screen, cpu load 90%(but 75% by Resolve itself), video - 40%,
I can't estimate disk load as Task manager shows 0%(probably due to indirect access).
Edit:
And regarding that Naming Scheme - i can't reproduce anymore with 12.3 version of Resolve that "show but not play" behavior. Maybe they relaxed naming requirements.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 10, 2016, 09:19:06 PM
@bouncyball:
can you do a another comparison of the MLVFS speed with the optimized builds?
as you seem to have rea-world footage and a high end machine i think thats a good basis for comparisons

i have those variants:

x86 vs. x64  [CPU architecture]
nopar vs. par [multicore parallelization]
noSSE, SSE, SSE2 on x86 [vectoring engine]
noAVX, AVX, AVX2 on x64 [vectoring engine]

i am the most curious about the nopar/par performance boost (if any).
so if you want to do a minimum setup, it would be the best you choose x64/avx2 and compare nopar vs. par.
if the tool crashes, your CPU probably doesnt support avx2, try avx then.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 11, 2016, 06:05:14 AM
@g3gg0

I have ben watching this thread with interest for some time now and decided to try mlvfs.

My Gear:
---------------------------------------
CPU:            Intel i5-3317U quad core
Ram:           6GB
Graphics:     Intel HD 4000
OS:             w10 Pro 64bit updated with the most recent patches.

I did have Pismo file mount for Windows installed but uninstalled it to avoid any potential conflicts with DOKAN.

The initial installation of DOKAN (v1) displayed the following message;

     http://1drv.ms/1WfsIs3
    (note the screen shot was from v0.800 but apart from the version number in the message, the message was the same).

I clicked "YES" then proceeded to download and install MS Visual C++ - Redistributable (x64). It installed without error.

DOKAN v1 appears to have installed correctly as verified via the cmd line screenshot below.

     http://1drv.ms/1KGNvnA
     Note: version was checked before and after manually starting the Driver and network service. Given the absence of error messages I
              assume that this is working correctly.

When I try to launch MLVFS.exe I initially had errors re: vcruntime140.dll and msvcp140.dll not found yet both of these files were in the C:\Windows\System32 directory. Based on some google research I moved copies of these files into the C:\Windows\SysWOW64 directory.

Subsequent to this the "dll not Found" error messages no longer appears but the following one does.
     http://1drv.ms/1KGO0xG

I acknowledge the error and the application fails to launch. Note that I have tried to launch it at the cmd line level with the same results.

Any thoughts, ideas? Is there an architecture dependency (e.g.: graphics card) that I'm missing?

Edit add;  If I run dokanctl.exe from the C:\Program Files\Dokan\DokanLibrary directory it appears to work correctly. However,
                 If I run it from the C:\Program Files (x86)\Dokan\DokanLibrary directory, it fails with the same error as mlvfs.exe.


Might be a clue to what is going on.


Cheers... Otto
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 11, 2016, 07:55:09 AM
@Ottoga:
1. Docan install always ask to install VC Redist, whether it installed, or not. No problem to reinstall it.
2. The message you got after manually adding dll files in respond to  "dll not Found" meaning you mix versions, ie, run 32 bit exe with 64 bit dll or vice versa.
To fix everything try:
1. Discard all the dll files you copied manually.
2. Install/reinstall Docan & VC redist. No need to run anything manually.
3. Unpack Optimized builds (http://magiclantern.fm/modules/modules/mlvfs_optimized.zip/mlvfs_optimized.zip)
to directory where you previosly unpacked mlvfs.zip.
4. As you on 64 win, run 64-bit exe, mlvfs_x64_rc.exe or even mlvfs_x64_rc_avx2.exe as your processor seems support that.
Make text file with 2 next rows and save it with .bat extension:
cd /d path_to_your_mlvfs_x64_rc.exe
mlvfs_x64_rc X: --mlv-dir=path_to_your_MLV_dir
where X - the Letter for disk not yet present in your computer.
Then just run that bat file and configure MLVFS with your web browser pointing it to http://localhost:8000/.
For example, my working bat file:
cd /d D:\PortableSoft\video\bin\mlvfs_optimized
mlvfs_x64_rc X: --mlv-dir=Z:\out
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 08:31:17 AM
@Ottoga:
as mothaibaphoto already wrote - MSVC runtime is a bit bitchy. never just download the DLLs.
always search for "vc_redist" packages when you miss the DLLs.
the optimized builds (which are still experimental of course) do not require the debug versions of the DLLs.

@dmilligan:
can you upload the "release" build of the package?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: squig on February 11, 2016, 08:38:55 AM
I'm having some issues with the new Mac build. MLV previews aren't visible in the Resolve media window. Folders are visible where there used to be previews. When I go into the folders most of them look empty. I can't load any MLV files into Resolve at all.  :'(

 Also when I try to refresh the browser I get a "can't connect to the server" error message.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: squig on February 11, 2016, 08:40:42 AM
I'm running Yosemite 10.10.3, Davinci Resolve 12.0, Safari 8.06, and Fuse 2.8.2

I went back to the Feb 2015 build: all good. One thing I noticed: my MLV folder takes a few seconds to load (spinning wheel of death). There's no spinning wheel of death showing up when I try to load the MLV folder in Resolve with the new build.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on February 11, 2016, 11:32:08 AM
VC Redist issues, maybe fix it with the pack: http://forums.guru3d.com/showthread.php?t=393405
(right click run as Admin)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 11, 2016, 12:33:21 PM
@g3gg0

Machine: cpu Xeon E3-1245 @ 3.4ghz, mem 16gb, 2x 120 gig kingston ssds <- they are no highend ones but not so bad anyway :)
Folder with MLVs reside on one ssd, mounted to drive Z: -> copied to second ssd.
Copied 19,852,533,609 bytes, 4717 files, 5 folders

Without runing mlvfs:
Result of sequential copy of a large file from 1st to second gives ~220mb/sec.
Result of sequential copy of a dir with a real DNGs of the same size gives ~175mb/sec.

results are below (orange line - mlvfs cpu usage):
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_avx2_nopar.jpg&hash=d4f778df768ce1424a1da138c4f5ed7b)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_avx2_par.jpg&hash=d5ef0c0c45c541111d07465b8cdc73f7)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_avx_nopar.jpg&hash=ce5875eb41d7ea662228acee175102f5)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_avx_par.jpg&hash=3c521aff8e4d2cac851c2727be65bc68)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_noavx_nopar.jpg&hash=344874fdf4d8601c0df8dbeb146b3924)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_noavx_par.jpg&hash=fd4e37abd85561f4115749a0bb3e5ee0)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_nosse_nopar.jpg&hash=62237e35fc12f3383df367de44bf247a)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_nosse_par.jpg&hash=3b39f83dbb790b4dc3e3951ef9782ec2)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_sse2_nopar.jpg&hash=2f0543a7ccf02cade71351bc85377f36)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_sse2_par.jpg&hash=56174db714119993376236106a7d2f9f)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_sse_nopar.jpg&hash=4e14518c48c6cd0a918db3795610cdfc)

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx86_sse_par.jpg&hash=fa5d7dcd0f5601b488c1d7ea4604e180)

regards
bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 11, 2016, 12:50:22 PM
@g3gg0

I think every binary performance is about the same. Par versions are just CPU hogs (would be nice to test them in random or parralel DNG file access mode). Anyway it is nice to have native 64bit binary and avoiding WOW64 :)
Forgot to mention, test was performed with mlvfs default parameters.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 03:48:25 PM
@bouncyball:
wow, nice. thanks a lot!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 11, 2016, 03:56:48 PM
@mothaibaphoto  @g3gg0

Thanks for your replies. I've got it working albeit I'm still driving it manually via the cmd line. I'll set-up the batch file and figure out how to point the application to my web browser after I've had some zzz's

One thought/suggestion: I've had to manually run dokanctl to unmount the mapped drive (i.e.: X) which in turn automatically closes the MLVFS application. However, if I exit the MLVFS application via say ctl/c then I'm unable to unmount the mapped drive letter via Dokanctl.

Does MLVFS monitor for a hidden/undocumented "quit" command that would send the unmount command to DOKAN and then gracefully exit MLVFS? or is that part of the web GUI that I've yet to figure out how to set up?

Other than that, fantastic job porting this to windows to all involved. I've successfully tested it with normal and FRSP mlvs. I use Cyberlink photo and video editing software and both happily imported the dngs from the mounted mlv's. one less step in the workflow. Who could complain about that.

Will play some more with the optimized builds tomorrow. mlvfs_x64_rc_avx2.exe crashes but mlvfs_x64_rc_avx.exe seems to work.

Tomorrows another day.  Thanks again gentlemen.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 04:00:46 PM
ok my conclusion:
parallelization for the 12bit->16bit conversion is pointless due to probably thread start overhead.
too much CPU usage and no visible performance boost.

the vectorization. hmm also not convincing.
does anyone have a "clearly visible" performance boost with some version?
guess only x64 gives a proper speedup.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 04:03:07 PM
@Ottoga:
you should be able to just run mlvfs and it will automatically mount it at the drive letter (or folder) you specified.
to unmount you simply close/ctrl-c mlvfs.exe and thats it.
30 seconds later the drive letter should disappear.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 11, 2016, 04:23:38 PM
@g3gg0

Thanks. Absolutely correct,  I just didn't wait long enough to observe this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 11, 2016, 04:36:57 PM
@g3gg0: u r welcome

Just did copy same files with microsoft tool "robocopy /MT:8" 8 threads turned on. mlvfs_x64_avx2_nopar vs mlvfs_x64_avx2_par -  2:17min vs 2:14min accordingly.
Result is 3 milisecond difference ;)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 05:38:43 PM
i guess seconds? ;)
nah thats not worth it. hoped there is a lightweight multithreading implementation involved, but seems not.

can you do another try with mlvfs_x64_avx2_par.exe ?
i modified it to use OpenMP. perhaps ist a bit better.
just redownload, the file date is 11.02.2016 17:37
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 11, 2016, 07:23:58 PM
Yup, my bad 3 seconds ;)

Windows 10 standard copy:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_avx2_par_openmp.jpg&hash=66f0a1940c79aa1950b1ee404548676b)

robocopy time was 2:18. In both cases max rate was at 158-162mb/sec and min rate 75-85. Rate rise at the beginning ~30sec I guess is a cache effect then it normalizes with some spikes. There are 5 mlvs (under 1gb, under 2gb, under 3gb, under 4gb and more than 4gb). Disk usage in resource monitor shows that 3 or 4 MLV files are open simultaneously for reading by mlvfs process.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 11, 2016, 08:05:37 PM
okay thanks again.
so no parallelization method helps speeding up RAW->DNG pixel operations.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 12, 2016, 09:00:49 AM
btw. with dokan v0.8 and win10:

if you copy DNGs from the MLVFS and cancel copying (no matter when) does it crash?

or is this only a dokoan v1.0 issue?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 12, 2016, 11:54:50 AM
u r welcome again :)

Returned to Dokan v0.8, did tests with x64_avx2_par and x64_avx2_nopar versions ubraptly terminating copy process with ESC. Par version crashed more often.
Once printed "mlvfs_wrap_read(1497): caught exception 0xC0000005 at address 0x7F64C85B" but still running!
Once OS rebooted?!

Installed Dokan 1.0 again, did same tests (also copying to the not cleaned dest dir) par version crashed several times.
Once printed "smlvfs_wrap_read(1497): caught exception 0xC0000005 at address 0x2627C85B" but still running. 
Nopar never crashed.

Switched to x64_noavx_nopar version. Several times ran 3 copy processes simultaneously from the same source to the same destination (file overwrite on) and terminated with ESC or mouse (X) click. No crashes.

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fnic.caucasus.net%2Fmlvfs%2F%3Fdir%3D%26amp%3Bdownload%3Dx64_noavx_nopar_3xcopy.jpg&hash=2fda1300bd0a6ca0d1d2253811c7f4fe)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 12, 2016, 05:51:33 PM
ok another speedup.
reduced the instruction count for RAW->DNG pixel conversion from 18-20 to 14 instructions on x64.
x86 should also benefit, its now 15 instructions.

here the optimized builds (http://magiclantern.fm/modules/modules/mlvfs_optimized.zip/mlvfs_optimized.zip)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 12, 2016, 08:39:15 PM
robocopy no threads

new builds:
2:35 mlvfs_x64.exe
2:36 mlvfs_x64_avx.exe
2:36 mlvfs_x64_avx2.exe

old builds:
2:37 mlvfs_x64_noavx_nopar.exe
2:38 mlvfs_x64_avx_nopar.exe
2:37 mlvfs_x64_avx2_nopar.exe

Don't even know what to say :) I guess 4717 files are not sufficient to measure the good difference and I'm too lazy to test it with 15K ;)
In real usage prog is quite snappy and rearly crashes. Used with resolve and it's fine.

Weekend is coming! :) Will test on my haswell 4790k 4.5ghz in premiere and after effects at home tomorrow.

Thanx g3gg0
bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on February 12, 2016, 10:53:08 PM
Those pesky kids and their Windows builds...


Linux now fails :( 


Somewhat expected: https://bitbucket.org/dmilligan/mlvfs/commits/097ffc1508d3daf7b827d4bfb1b39247debefc9b





Code: [Select]

gcc -Wall -std=gnu99 -D_FILE_OFFSET_BITS=64 -DVERSION="\"584941c\"" -DBUILD_DATE="\"2016-02-12 21:51:31\"" -DFUSE_USE_VERSION=26 main.c dng.o index.o wav.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose/mongoose.o webgui.o resource_manager.o lj92.o gif.o patternnoise.o LZMA/7zAlloc.o LZMA/7zBuf.o LZMA/7zBuf2.o LZMA/7zCrc.o LZMA/7zCrcOpt.o LZMA/7zDec.o LZMA/7zFile.o LZMA/7zIn.o LZMA/7zStream.o LZMA/Alloc.o LZMA/Bcj2.o LZMA/Bra.o LZMA/Bra86.o LZMA/BraIA64.o LZMA/CpuArch.o LZMA/Delta.o LZMA/LzFind.o LZMA/Lzma2Dec.o LZMA/Lzma2Enc.o LZMA/Lzma86Dec.o LZMA/Lzma86Enc.o LZMA/LzmaDec.o LZMA/LzmaEnc.o LZMA/LzmaLib.o LZMA/Ppmd7.o LZMA/Ppmd7Dec.o LZMA/Ppmd7Enc.o LZMA/Sha256.o LZMA/Xz.o LZMA/XzCrc64.o -pthread -lfuse -lm -o mlvfs
main.c: In function ‘mlvfs_getattr’:
main.c:1017:23: error: storage size of ‘mlv_stat’ isn’t known
         struct stat64 mlv_stat;
                       ^
main.c:1019:26: warning: implicit declaration of function ‘stat64’ [-Wimplicit-function-declaration]
         int mlv_status = stat64(mlv_filename, &mlv_stat);
                          ^
main.c:1017:23: warning: unused variable ‘mlv_stat’ [-Wunused-variable]
         struct stat64 mlv_stat;
                       ^
Makefile:35: recipe for target 'mlvfs' failed
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 12, 2016, 11:01:56 PM
well, things develop step by step.

add a
#define stat64 stat
and try again
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Licaon_Kter on February 12, 2016, 11:09:15 PM
Looks like that was that... 10q


The mlvfs file is now ~40kb smaller ( 466kb vs 504kb ), is that expected?


Should I bother with a PR or ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 13, 2016, 01:03:34 AM
A possible bug and some questions please.

Today I have launched MLVFS_x64_avx to a local directory containing a single MLV file mapped as drive X: All is well as confirmed via the localhost:8000 view in my browser.

     http://1drv.ms/1o7Mtqk

While I was randomly viewing individual frames (by double clicking on them) I noticed in the MLVFS cmd screen a number of loadchunk error messages. These related to mlv files tested yesterday that reside in a different (external HD) location. This suggests that MLVFS is storing previously analysed MLV file names/details somewhere. In itself, these error messages are not an issue however from a performance perspective MLVFS probably shouldn't be trying to open MLV files other than those in the mapped directory.

    http://1drv.ms/1o7OHpy

Edit:
  The generated errors messages may have been my fault. I had originally launched:
      mlvfs_x64_avx2.exe X: --mlv-dir=H:\Video\20150825_Banff_Surrouds\MLVs (where the 4  MLVs related to the error messages reside - it crashed),  I then launched

      mlvfs_x64_avx.exe X: --mlv-dir=D:\MLVs (where the single MLV resided)

  As there was no appreciable delay between the crashing of the mlvfs_x64_avx2 instance vs the launching of the mlvfs_x64_avx instance it suggests that maybe I should
  have waited a bit longer to allow the first instance to clean up after itself.


Questions

Where does MLVFS store details of historical MLVs processed? (Maybe irrelevant).
Configuration options set in the localhost view don't appear to be persistent. Is this the expected behaviour?
Having set the configuration options in localhost at what point will they apply e.g: on the fly upon opening, viewing  or extracting the frames?
MLVS caters for a number of parameters:
     - what are the defaults settings?
     - what the purpose of --port and is its sub-parameter a directory name or port number (or both)?

Possibly the above questions could be answered by adding a little more commentary to the options list provided when running MLVFS without parameters.
     
      http://1drv.ms/1TgUrdK

Thanks in advance.... Otto
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 13, 2016, 04:50:41 AM
--port is the port number for the webserver, default is 8000, it's so that you can have multiple instances running at the same time

default values are the ones you see in the webgui initially
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 14, 2016, 02:06:09 AM
@dimilligan.... Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jogodedentro on February 14, 2016, 03:55:37 AM
just reached this forum after having no joy converting mlv files for use with davinci resolve, there seems to be a solution in these threads, but my understanding of coding and command line use isn't great, but please can someone send me a command line sequence that has worked for them in a similar set-up to me, I have windows 10 64bit, I have already downloaded Dokany and mlvfs_86 and the optimized mlvfs zip, I also have oracle virtual box. How do I get from having all this to successfully converting mlv files for use with davinci resolve....thanks to all for working on this.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 14, 2016, 04:30:36 AM
@jogodedentro

I had problems with a flaky Visual studio Redistribution 2015 runtime library install that was causing the application to fail.

The problem was fixed by re-installing all of the MS redistributable runtimes via the package advised to me by Licaon_Kter in the following post:
http://www.magiclantern.fm/forum/index.php?topic=13152.msg162133#msg162133

The following post contains guidance by Mothaibaphoto on how to run it.
http://www.magiclantern.fm/forum/index.php?topic=13152.msg162127#msg162127

There is a wealth of information within this topic that will help with your understanding. My suggestion, if you haven't already done so is, read the 1st couple of posts and then as the Windows developments are more recent go to the last post and work your way backwards.

A small tip, If you have already installed dokany and MLVFS open a cmd line window. Then:

for Dokany - cd /d C:\Program Files\Dokan\DokanLibrary
                   dokanctl.exe [enter]
and

for MLVFS - CD to wherever you have unpacked MLVFS to
                  mlvfs.exe [enter]

Both applications will present you with a list of applicable parameters.

Hope this helps.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jogodedentro on February 14, 2016, 05:52:38 AM
thanks for your reply, yes I can see the options for DokanyLibrary using command prompt, so I assume I need to create a mount point but I can't tell how you do that from the options listed, what option in the DokanyLibrary is used to create a mount point?
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: Ottoga on February 14, 2016, 06:50:42 AM
@jogodedentro

2nd link in my original post to you.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 14, 2016, 12:29:11 PM
SemiOffTopic info :)

Bad news for windows users using SlySoft's great image mounter "Virtual CloneDrive". Dokan 1.0-BETA driver conflicts and renders useless that mounter. after dokanctl /r d (removing driver) and rebooting things are sorting out. No problem with Dokan 0.8 though (0.8 version has it's own issues, and with latest version of mlvfs I'd recommend 1.0)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on February 14, 2016, 12:34:19 PM
That's a pity for Windows 7 users! Are other image mounters affected as well?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 14, 2016, 12:39:40 PM
Don't know. Have not tried any for a while.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 14, 2016, 12:44:41 PM
can you report that issue to the dokany devs?

edit: okay you already did :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 14, 2016, 12:48:22 PM
:)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jogodedentro on February 14, 2016, 12:49:13 PM
@ottoga

Thanks for pointing me in the right direction I created the .bat and have localhost running in the web browser.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 14, 2016, 01:40:57 PM
Oh! ;) meanwhile Dokan 1.0.0-RC1 has arrived ~20 hours ago! (didn't know it, dev have just told me)

No more experiencing problems with CloneDrive! but .... :)

@g3gg0: They changed/fixed lots of things and also related to dokan_fuse and dokanfuse.dll default location. Think you have to take a look at it.

edit: all dll and lib names changed so none of the versions of mlvfs work. Situation definitely needs your attention ;)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: michaelfilm on February 14, 2016, 08:24:49 PM
Hi everybody,

I´am new here, maybe I have an question that has already been answered...
- I have an Issue with MLVFS using After Effects CC (current version):

After adjusting the DNG in the Camera Raw Window, I get a sequence in After Effects, but the Raw-Adjustments only affects the first frame...

For any help, I would be very grateful.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 14, 2016, 09:20:19 PM
and this doesnt happen when all files are in one directory on your HDD?
any files being created in that directory?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: liryna on February 15, 2016, 01:32:20 PM
Hi everyone,

I am one of the dokan maintainers.

I just wanted to say that I added the MLVFS project to our home page http://dokan-dev.github.io/  ;)
https://github.com/dokan-dev/dokan-dev.github.io/commit/951b0a9d33cab3301566adc5d3222336cae526bf

If you need any help or remark about dokan, feel free to open an issue on github !

(I would specially be interested about your feelings in the dokan_fuse wrapper)

Thank your for using Dokan !
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 15, 2016, 05:20:26 PM
and this doesnt happen when all files are in one directory on your HDD?
Yes. Correct

any files being created in that directory?
Nope

After Effects tries to change the dng file metadata :) but alas...

Z:\M28-1253.MLV>fc /b M28-1253_000000.dng D:\DEST\M28-1253_000000.dng
Comparing files M28-1253_000000.dng and D:\DEST\M28-1253_000000.DNG
00000004: 08 00
00000005: 00 48
00000006: 00 40
FC: D:\DEST\M28-1253_000000.DNG longer than M28-1253_000000.dng

1st real file on HDD changes, others not

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 15, 2016, 09:25:01 PM
hi liryna .

welcome to our board. and thanks for maintaining and pushing dokany!
i really appreciate your work :)

as of now we are very happy with dokan_fuse and only have minor, not yet closer analyzed glitches that ive seen on 1.0-beta.
e.g. when copying from MLVFS with explorer and cancelling that copy operation, dokan will crash somewhere in nowhere or call fuse cbr's with "ddddddddddddddddddddddddddddddddddddddddddddddd....." as filename.

but i have to
a) install the 1.0-rc and
b) use the debug versions plus
c) create logs for it :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 15, 2016, 10:15:57 PM
just tried dokan-1.0-rc debug with MLVFS and it just doesnt work anymore.
didnt change the code but fuse_main returns 1
no matter if i specify -f, -d, -o debug or all together.
no log messages, nothing.

anything that has changed?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 15, 2016, 11:41:12 PM
After Effects tries to change the dng file metadata :) but alas...

So MLVFS tells FUSE that the DNGs are writeable so they show up in the mount as writeable, but they aren't really (which would sort of impossible anyway). The reason it's like this is that ACR/AE refuses to open readonly DNGs, even though it doesn't actually need to write to them. On my Mac ACR doesn't actually try to put metadata in the DNGs, instead it creates XMP sidecar files and puts all its metadata there (there might be a setting for this somewhere for you). MLVFS allows a "passthrough" so programs can create and modify sidecar files like this inside the mount directory. It stores the actual files next to the MLV files in the real file system in an .MLD directory. You can put anything you want except .DNG and .WAV files into the mount. At least, this is how it is supposed to work, and how it works on Mac.

So you to get ACR to use sidecars and then make sure that MLVFS's passthrough works (just try to copy some file not ending in DNG or WAV into the mount)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 16, 2016, 08:40:45 AM
@dmilligan

Thanx for your attention. Yes on my hakintosh AE works well. Have to look to the mounted folder.

just try to copy some file not ending in DNG or WAV into the mount

That's exactly what I'm trying to do. Passtrough does not work under win. (I think g3gg0 already mentioned this)
In file explorer: copying file - no error but file is not there (no any .MLD). Making new dir or new file results in error message "file/folder name you are creating is too long" and I have not even input the file name yet.

edit: I'm kinda big fan of the "FAR manager" and using it many years and  just checked copying with it. Can not copy/create file, BUT... can create DIR and MLD appears in mounted folder (can not be deleted after that though, mkdir command does the same). Hmmm... strange and unpredictable behavior. (Bugs! Bugs are everywhere ;) )

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: liryna on February 16, 2016, 07:58:02 PM
@g3gg0

Did you copy dokan1.dll in your runtime folder ?
dokan1.dll is no longer in system32 (for compatibility reason for past / next release).

If this doesn't work, could you open an issue on dokan repository with logs ?

Thank is advance
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 16, 2016, 10:08:01 PM
issue already resolved and created a ticket: https://github.com/dokan-dev/dokany/issues/175 :)

BR,
g3gg0
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 17, 2016, 12:53:11 PM
just symlinked:

MLVFS-DIR\dokan.dll -> DOKAN-RC1-DIR\dokan1.dll
MLVFS-DIR\dokanfuse.dll -> DOKAN-RC1-DIR\dokanfuse1.dll

and "mlvfs_x64" works again
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 17, 2016, 05:18:30 PM
@dmilligan

What the log file in the virtual mlv folder does? As I see it's always empty.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 17, 2016, 05:52:20 PM
The MLV file format spec (https://docs.google.com/spreadsheet/ccc?key=0AgQ2MOkAZTFHdHJraTVTOEpmNEIwTVlKd0dHVi1ULUE#gid=0) supports embedding plain text blocks. If there is any in the MLV file, it will appear in the log. I've thought about including other diagnostic information in this file too, like a dump of the header info.

I think currently the only thing that will write these text blocks in the MLV file is if you turn on the debug tracing option (I think you have to load trace.mo too).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 17, 2016, 06:52:28 PM
@dmilligan: Thanx for comprehensive explanation :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 17, 2016, 08:48:52 PM
@dmilligan:
whenever mlv_rec gets a keypress signalled, it will create a "MARK" block.
do you think there is a good use case for it? e.g. insert a text file for this frame number.

same for blocks like LENS which will update sporadically and will reflect aperture and focal length, focal distance etc.
this data could be used on a per-frame basis.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 17, 2016, 09:45:07 PM
well, i dont get any crashes anymore when aborting copy operations with dokan 1.0-rc

MLVFS_x86 (http://www.magiclantern.fm/modules/modules/MLVFS_x86.zip/MLVFS_x86.zip) or MLVFS_x64 (http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip) to be used with dokan 1.0-rc -> DokanSetup.exe (https://github.com/dokan-dev/dokany/releases)

make sure you uninstall the old version first.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 17, 2016, 11:02:00 PM
There might be something in the CDNG spec for markers, I'll have to take a look. Another possibility is an option that breaks them up into separate sequences, so restart the numbering of the DNGs with a different prefix or something like that.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 18, 2016, 09:53:40 AM
MLVFS_x86 (http://www.magiclantern.fm/modules/modules/MLVFS_x86.zip/MLVFS_x86.zip) or MLVFS_x64 (http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip) to be used with dokan 1.0-rc -> DokanSetup.exe (https://github.com/dokan-dev/dokany/releases)

@g3gg0: Great, Thanx! :)
Can you look at the creating .MLD folder bug? e.g. make passthrough mode normally work on win.

edit: Exceptions (at 2 addresses) below caught while maniputating DNGs in Premiere Pro, e. g. stoping starting, scrubing back and forward and changing "lumetri color" parameters. MLVFS does not stop entirely just hickups and running while prevew freezes in premiere. It seems MLVFS does not like random file access very much :)

E:\MLVFS_old>mlvfs_x64.exe Z: --mlv-dir=E:\MLV
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xC476FD74
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xC476FD74
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877
mlvfs_wrap_read(1512): caught exception 0xC0000005 at address 0xBED5C877

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 18, 2016, 05:49:19 PM
@g3gg0, dmilligan

This exeption 0xC0000005 - generated as far as I understand on "segmentation fault"/"access violation" (accesssing virtual/physical memory - the state of wich is not allowing that access) - occurs when you are stopping and starting playing dng sequence in premiere or resolve. It is similar to the situation when you are copying files and then suddenly canceling it. As g3gg0 mentioned above this randomly was ending by a crash (after printing this exeption as I've seen).

x86 version gives a little bit different numbers:
mlvfs_wrap_read(1505 <- is it line number or what?) and surely another virtual address.

E:\MLVFS_x86>mlvfs_x86.exe Z: --mlv-dir=E:\MLV
mlvfs_wrap_read(1505): caught exception 0xC0000005 at address 0x6A70D72E

In short mlvfs completely crashes very rearly, but exeption is generated almost on every play/stop while prog goes on runnig-hickup-running.

Unfortunatelly I'm not a c/c++ dev in any way.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 19, 2016, 02:05:19 AM
can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

the things not working:

 - editing *.dng, *.gif, *.log, *.wav of a virtual MLV folder (subdirs are okay)
 - placing *.dng, *.gif, *.log, *.wav into a virtual MLV folder (subdirs are okay)
 - copying/moving *.dng, *.gif, *.log, *.wav of a virtual MLV folder into a subdirectory
 - deleting folders (https://github.com/dokan-dev/dokany/pull/166)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Lasting Image on February 19, 2016, 07:38:48 AM
At least, this is how it is supposed to work, and how it works on Mac.

I'm using a Mac and have the same problem where only the first dng in the After Effects sequence has the Camera Raw corrections. There's a video I saw where Reddeercity uses Camera Raw in After Effects on the first dng and each subsequent dng has the corrections - not just the first. I'm not noticing any difference in what I'm attempting from the settings in reddeercity's video (I think). Any ideas on what I'm doing wrong? In other threads I recall people complaining about later versions of After Effects. Could it be an issue with the latest version of AE CC?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 19, 2016, 08:16:20 AM
@Lasting Image:
I'm using a Mac and have the same problem where only the first dng in the After Effects sequence has the Camera Raw corrections.
This is how AE interprets DNG sequences: It opens ACR window with first file, you store settings in either file itself or sidecar xmp according to your settings.
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
This has some benefits (no flickering, for example), and drawbacks (you can't ramp exposure, or wb, for example), but this is absolute offtopic to MLVFS.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 19, 2016, 10:25:59 AM
can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

With previous no debug version of mlvfs binaries:
Under the root dir (e.g. mount point Z:\ or whatever) where MLV virtual folders are - no file/dir operations possible
Under virtual MLV folders - only directory creation is possible (no subdirs), .MLD created in the orig mlv folder

With last posted debug version of mlvfs binaries:
Under the root DIR - possible: creating dir/file, copying dir/file, not possible: delete created dir/file (as expected)
Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

BTW Am I supposed to see the .IDX .M01 .M02 files in the root dir along with the virtual .MLV folders? They are there :-\ in last debug version.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 19, 2016, 02:02:31 PM
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
Actually, this is wrong. It does not disregard xmp files or embedded metadata for other images in the sequence if they are there. That's how this script (http://www.magiclantern.fm/forum/index.php?topic=8850.0) is even able to work. I have used it many times with AE.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 19, 2016, 02:53:20 PM
@dmilligan:
I'm a big fan and very power user of your script :)
Take this opportunity to say Great Thanks for it one more time :)
But ... my AE works as I described. And it's generally good for me. Just specially tried to save different settings and open sequence. Maybe some settings I don't know affect this behavior.
After applying your script I always convert right in Bridge - CtrlA->CtrlR
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: Ottoga on February 19, 2016, 03:49:27 PM
@Dmilligan

A big thumbs up from me as well for your Windows version.

I've been giving it a solid try out during the mlv lite testing and love the on the fly the processing.

Took me a while to figure out that that I had to set the parameters before opening the Dokan mounted file. [Facepalm].
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 20, 2016, 10:28:19 AM
@bouncyball:
can you check it?

can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

the things not working:

 - editing *.dng, *.gif, *.log, *.wav of a virtual MLV folder (subdirs are okay)
 - placing *.dng, *.gif, *.log, *.wav into a virtual MLV folder (subdirs are okay)
 - copying/moving *.dng, *.gif, *.log, *.wav of a virtual MLV folder into a subdirectory
 - deleting folders (https://github.com/dokan-dev/dokany/pull/166)

well, i dont get any crashes anymore when aborting copy operations with dokan 1.0-rc

MLVFS_x86 (http://www.magiclantern.fm/modules/modules/MLVFS_x86.zip/MLVFS_x86.zip) or MLVFS_x64 (http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip) to be used with dokan 1.0-rc -> DokanSetup.exe (https://github.com/dokan-dev/dokany/releases)

make sure you uninstall the old version first.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 20, 2016, 01:41:09 PM
@g3gg0

I already did :)

http://www.magiclantern.fm/forum/index.php?topic=13152.msg162572#msg162572 (http://www.magiclantern.fm/forum/index.php?topic=13152.msg162572#msg162572)

Has anything changed since then?

It's not crashing but, as you said, unusable in real situations because of the standard output log slowing it down so much.
Could not manage to stress it so to see some exceptions.

Passthrough related things are in my message above.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 20, 2016, 02:53:11 PM
redownloaded binaries. yeah they dated 02/20/16 :). after checking, can admit, that applies exactly the same what I've been talking about in 2 earlier messages.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 20, 2016, 09:38:08 PM
sorry, too much noise so i didnt see your post.
thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 21, 2016, 08:08:14 PM
@dmilligan

I have one question about bad pixel removal. Can we do this with pixel maps (like focus pixel maps by dfort). Just having an option to remove them by the map or current algorithm and then anyone can make pixel maps for his/her needs: sensor and resolution (crop, non crop etc). It would be lot more faster.

Besides I have just one pixel and that's all ;)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on February 21, 2016, 11:26:05 PM
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Of course this would require some effort on the user but it shouldn't be any more difficult than the instructions in dcraw:

Code: [Select]
FILES
       ./.badpixels, ../.badpixels, ../../.badpixels, ...
              List of your camera's dead pixels, so that dcraw can interpolate
              around them.  Each line specifies the column, row, and UNIX time
              of death for one pixel.  For example:

               962   91 1028350000  # died between August 1 and 4, 2002
              1285 1067 0           # don't know when this pixel died

              These coordinates are before any stretching or rotation, so  use
              dcraw -j -t 0 to locate dead pixels.

I don't see any real need to keep track of the date.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 22, 2016, 12:52:54 AM
that unix time idea is quite funny t.b.h :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on February 22, 2016, 01:35:09 AM
You mean the "UNIX time of death for one pixel" idea?  :P I never use it but dcraw "badpixels" lists require something in that field. I just fill it in with zeros.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on February 22, 2016, 04:39:18 AM
Excellent concept @bouncyball and great suggestion by @dfort!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 22, 2016, 08:38:28 AM
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Nice idea! Let's wait what master David will say :)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 22, 2016, 09:05:50 AM
Actually, is there dead pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Etc) :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 22, 2016, 09:51:23 AM
Quote
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 22, 2016, 10:37:23 AM
Ah, my bad :(
Bad pixels always there.
Cold/hot pixels can be relatively stable. So,
Is there dead pixel = Fn(Camera Serial, Time)
Is there Cold/hot pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Time, Etc)
Ultimately, it would be great to have sensor profiling tool, similar to dot_tune.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 22, 2016, 01:22:26 PM
Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

I think you are right :). Appearance of that kind of pixels depends on several things:
AFAIK canon software has factory callibration built in and maps pixels itself (however you can kinda alter this remapping with several second manual sensor cleaning procedure. That's how I cleaned up some hot pixels - not bads - on my former 7D) if you got chance read raw buffer before that then you have vanilla information with all off uncorercted pixels etc and I think this is a case when we get croped resolutions of raw data from the sensor (some buffers in memory I guess). Also line skipping/bining affects coordinates and appearence of those pixels. In short, whether they appear in output raw or not, depends on the camera mode and what canon firmware does in that moment and also what we are doing with great help of ML software ;).

But... the pixel correction map is still for a dedicated resolution and mode, hence valid in most cases.

This is my understanding... correct me please if i'm wrong.


bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 22, 2016, 01:28:05 PM
Would have been smooth with a dot picker in MLVFS which creates the map on the fly when clicking your dots on an dng file :P (Wishful thinking)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 22, 2016, 01:43:04 PM
Saying Clicking do you mean automatic removal algo? Sometimes it misses some dots ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on February 22, 2016, 01:46:40 PM
Since it often happens you only have one or two annoying pixel dots you could find its coordinates on pixel level(zoomed in around 400%). Since Davids web gui works in realtime maybe this could be achieved by grabbing neighbouring pixels like in dcraw and the previewing dng would instantly show the result? As I said, wishful thinking  8)
There,s also opcode lists which could work with the dng format. But how I don,t know.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on February 22, 2016, 01:48:59 PM
If I may use a term that dmilligan used previously, K-I-S-S (https://en.wikipedia.org/wiki/KISS_principle).

Focus pixels and bad/dead pixels are always there. Maybe they are not always visible or pose a problem but the location of these pixels on the sensor for a particular camera never change. These are the pixels that can be relatively easily mapped and eliminated.

Pixels that change according to the temperature of the sensor or other conditions are something else. Astrophotographers go to great lengths to separate celestial objects from this noise. Google astrophotography bias frames and you'll see what I mean.

Then there is fixed pattern noise (FPN) (https://en.wikipedia.org/wiki/Fixed-pattern_noise) which is something that can also change and thus cannot be mapped out.

Maybe I'm wrong but it seems that sensor profiling cannot be done just once and it will be good for the life of that camera. Focus pixels and bad/dead pixels fixing is permanent.

As far as a dot picker--unless MLVFS gets a graphical user interface that can zoom in far enough to work on individual pixels it is easy enough to map out dead pixels with Photoshop or Gimp.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 22, 2016, 08:49:18 PM
@bouncyball:
> Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

can create dirs and files without any issues.
delete/copy not possible. trying to work around that dokan issue (https://github.com/dokan-dev/dokany/issues/178)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jogodedentro on February 23, 2016, 05:30:19 AM
Hi There

With help from this forum I was happily colour grading MLV files in davinci resolve. But now after a reinstall of windows 10, I have installed Dokany and despite entering the path to DokanyLibrary the dokanctl.exe command brings up a 'not recognized internal or external command...' error message. This command was previously working on the same Windows 10 machine, but now after a re-install of windows I have this issue. I have installed the latest version of Dokany. Any ideas how I can get this to work again, advice will be much appreciated...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: jogodedentro on February 23, 2016, 06:36:59 AM
ok solved problem re-install Dokany but don't re-install microsoft redistritbutable on prompt, dokany installation will then complete.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 23, 2016, 11:01:23 AM
path to dokanktl.exe (Dokan 1.0.0-RC1): C:\Program Files\Dokan\Dokan Library-1.0.0\dokanctl.exe

edit: ah ok. nice to hear that :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 23, 2016, 11:54:39 AM
@g3gg0

> can create dirs and files without any issues.
> delete/copy not possible. trying to work around that dokan issue

Post your latest compiled binaries please and I will check them with pleasure :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 23, 2016, 12:37:01 PM
@g3gg0

with the last debug version that I have

root dir looks like this. mlv folder plus IDX and Mxx files and here I can create dirs and files
------------------------------------------------------------------------
Z:\>dir
 Volume in drive Z is DOKAN

 Directory of Z:\

02/11/2016  12:40 PM                  5 ,572 M28-1253.IDX
05/28/2015  12:53 AM    <DIR>               M28-1253.MLV
02/11/2016  12:40 PM                   4,204 M28-1255.IDX
05/28/2015  12:56 AM    <DIR>               M28-1255.MLV
02/11/2016  12:39 PM                 10,144 M28-1313.IDX
05/28/2015  01:14 AM    <DIR>               M28-1313.MLV
02/11/2016  12:39 PM                 15,460 M28-1316.IDX
05/28/2015  01:16 AM    <DIR>               M28-1316.MLV
02/11/2016  12:39 PM                 35,512 M28-1322.IDX
05/28/2015  01:24 AM     4,294,385,716 M28-1322.M00
05/28/2015  01:24 AM          94,462,004 M28-1322.M01
05/28/2015  01:23 AM    <DIR>               M28-1322.MLV
               7 File(s)  4,388,918,612 bytes
               5 Dir(s)               0 bytes free
-------------------------------------------------------------------------

in mlv folders:

trying creating dir: no output but in reality folder did not created!
Z:\M28-1253.MLV>mkdir folder1

trying creating file: error message
Z:\M28-1253.MLV>echo 123 > file.txt
The system cannot find the file specified.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 23, 2016, 01:15:20 PM
Can someone please help - i am a noob with this - i have followed the steps in the OP and included the following code in the C:/ but getting a syntax error. What is the 'mount point'?

I am on windows 10.

Need to learn how to use this program to decode MLV Lite files. At the moment MLViewer is showing pitch black screens of video recorded with the MLV Lite module.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 23, 2016, 10:23:01 PM
updated the MLVFS_x64.zip, please retry
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 24, 2016, 12:54:05 AM
Sorry guys but for me I find this hard to believe that no other Mac users is experiencing what I am going through for the past week or so...


It's rather quite annoying and I've also trying deleting .IDX and .MRX files just to be sure as well as updating/downgrading OS X FUSE to no avail.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 24, 2016, 01:27:38 AM
Anything in ~/mlvfs.log? Any crash logs?
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 24, 2016, 02:30:18 AM
I'm afraid I may be doing this incorrectly via Terminal...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1687%2F25106404052_71d25f21e1_n.jpg&hash=b318199278f09192c8efe3e5b60a8cd2) (https://flic.kr/p/EfyTeo)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 24, 2016, 10:22:48 AM
@g3gg0

> updated the MLVFS_x64.zip, please retry

with new one - copy/creation of file/dir is the same (no avail) as described above.
one minor change to root dir - .IDX files disappeared (good), .Mxx files are still there (bad) :)

bb
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: koenkooi on February 24, 2016, 12:10:46 PM

I'm afraid I may be doing this incorrectly via Terminal...

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1687%2F25106404052_71d25f21e1_n.jpg&hash=b318199278f09192c8efe3e5b60a8cd2) (https://flic.kr/p/EfyTeo)

Try 'cat ~/mlvfs.log' to show the contents in the terminal window or 'open ~/mlvfs.log' to try opening it with the osx log reader. Or point Finder to your home directory, probably /Users/DeafEyeJedi.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 24, 2016, 12:47:43 PM
@DeafEyeJedi can you please describe textually what problem you are experiencing?

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 24, 2016, 10:19:59 PM
@DeafEyeJedi can you please describe textually what problem you are experiencing?
Well I believe this whole 'culprit' of mine started around the time when the Windows version of MLVFS was finally released by @dmilligan (which I am happy about since we want more users to come on board with us) however it is my understanding that this anomaly shouldn't be occurring with different types of MLV files from 5D3, 7D, EOSM or even SL1 which used to work without the infamous web gui error (hence in my video above) which shows some 5D3 wont work and some would along with 7D files working just fine. It's just so random to me.

I then was suggested by @dfort to go ahead and downgrade to an older version of MLVFS (Version: 1bc6f7f 2016-02-04 03:51:18) which seems to be working fine on my end with the same exact files that weren't working earlier with the latest version of MLVFS. Not sure why though and I can't seem to find any crash logs probably because it hasn't crashed yet.


Try 'cat ~/mlvfs.log' to show the contents in the terminal window or 'open ~/mlvfs.log' to try opening it with the osx log reader. Or point Finder to your home directory, probably /Users/DeafEyeJedi.
Tried all these to no avail. Thanks for the courtesy tho and much appreciated. So I would like to help further down by narrowing down this so called 'culprit' since I can now open all the files that I wouldn't be able to run with your latest MLVFS as oppose to the Feb 3rd version.

Any ideas/suggestions is welcome...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 24, 2016, 10:49:13 PM
as i am on a slow network i cannot watch this video and cannot understand what the problem in general is.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 24, 2016, 11:00:21 PM
With all the MLV Files that I am trying to work on my Mac (mostly 5D3 with 7D and few SL1) which doesn't work with latest MLVFS (Version: a853aac 2016-02-10 02:05:13) but then once I downgrade to an older version of MLVFS (Version: 1bc6f7f 2016-02-04 03:51:18) which works fine like normal and can work on them in Post with no problems.

So AFAIK something happened between those two dates somewhere in the source code of some kind.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 24, 2016, 11:06:33 PM
what does "does not work" mean?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 24, 2016, 11:18:01 PM
@bouncyball:
thanks, i found the reason. didnt create the .MLD properly
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 24, 2016, 11:49:43 PM
what does "does not work" mean?

•Not working (Version: a853aac 2016-02-10 02:05:13) = (screenshot below)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1676%2F25242774385_b21acacedf_n.jpg&hash=dd9e7f0dd57ceb4ae34c6ea7a5627790) (https://flic.kr/p/EsBPo2)

•Working (Version: 1bc6f7f 2016-02-04 03:51:18) = (screenshot below)
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm2.staticflickr.com%2F1491%2F25124486682_058863e7ab_n.jpg&hash=d0c16f993c86acff3991fa26b7027e3d) (https://flic.kr/p/EhayyS)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 25, 2016, 12:08:12 AM
Sorry, there's a dot.

Code: [Select]
cat ~/.mlvfs.log
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 25, 2016, 12:50:54 AM
Okay so now this brings me to another question ... Do I run this command while running the older version of MLVFS that works or with the latest one that's supposedly causing problems on my end?

I would assume to go with the latest one since that's the one that you guys want to know, correct?

Thanks, D!
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 25, 2016, 01:55:00 AM
That is just a log file. When you run MLVFS using the Automator workflow, the standard output is written to that file. The cat command simply displays the file, alternatively you can open it with a text editor or whatever.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: squig on February 25, 2016, 02:12:06 AM
I posted about the new Mac build being DOA a while back http://www.magiclantern.fm/forum/index.php?topic=13152.msg162129#msg162129
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on February 25, 2016, 01:38:34 PM
Put MLVFS through its paces again today with a variety of MLV_Lite generated files that included single/multi-chunk mlvs, auto stopped mlvs due to frame skipping and one crash mlv when it tried to switch from 5x to 10 recording. All good, no bad frames that I could find and no error messages in the mlvfs cmd window.

Camera is a 7D.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 25, 2016, 04:52:24 PM
@g3gg0

i found the reason. didnt create the .MLD properly

Hey, great! Can you post new binaries? :)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 25, 2016, 09:24:59 PM
yep, uploaded
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 26, 2016, 02:59:14 AM
Hey everyone, I must admit that I am completely lost with what to do. Have reached a point (after endless hours of sitting on the computer and a bad back with a few 10's of $'s spent on buying programs that will let me buy .dll files and all) where I can go into the cmd and request the computer to execute the mlfvs executable and all that. But do not understand what to do next ... for example where do the converted DNG's surface. Cannot see these anywhere .. and since there is no error on the cmd screen, I reckon all must be good.

My workflow:

I first download the mlvfs_x64.exe zip and then unzip (naturally  ;))
Then copy the unzipped mlvfs_x64 folder to the my f:/ drive
Mount the f:/ drive to my z:/ drive - so now all that is on f:/ shows up on z:/ (every thing and the mlvfs_x64 folder too)
I also copy the mlv that I want to convert to the f:/ root folder
Then, I just go to cmd where if first bring the computer to f:/ (then) it is f:/mlvfs_x64 (then) f:/mlvfs_x64/mlfvs_x64.exe f:/ --mlv-dir=z:/out (where 'out' is my intended output folder on z:/ where I want my DNG's to go

Not sure what to do with the dokey elements. When I download it, it is not coming on but there is an error in the .exe

I am on Windows 10 at the moment. Will try using these executables (FUSE and all) on the my Macbook pro to see if I come up with different results.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 26, 2016, 03:53:41 AM
Once you start mlvfs like this:
Code: [Select]
mlvfs_x64.exe Z:\ --mlv-dir=F:\
That is all you have to do.

Now if you copy an MLV file to the root of F:\, look in Z:\, you should see a folder with the same name as the MLV file, inside it are your DNGs.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 26, 2016, 08:21:55 AM
do not mount the drive somewhere with windows methods.

just extract the files to f:\mlvfs_x64\
open a commandline

Code: [Select]
f:
cd \mlvfs_x64
mlvfs_x64.exe Z:\ --mlv-dir=F:\

if it doesnt work, show us screenshots of what you typed
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on February 26, 2016, 08:53:12 AM
That is just a log file. When you run MLVFS using the Automator workflow, the standard output is written to that file. The cat command simply displays the file, alternatively you can open it with a text editor or whatever.

Thanks anyway I had to take parts of the log out into context since it was too large to paste the entire textfile on this post so here it is...

Code: [Select]
Last login: Thu Feb 25 23:41:04 on ttys000
Apples-Macintosh-10:~ DeafEyeJedi$ cat ~/.mlvfs.log
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/favicon.ico...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/favicon.ico...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 26, 2016, 09:00:39 AM
@g3gg0

yep, uploaded

Thanx! Great improvement over the last one !!!

1. Can create dirs and files everywhere including root dir
2. Can delete files! but not folders (guess related to the dokan issue)
3. One annoying thing: .M00, .M01 ... files are visible in root dir. Definitely we don't need this, because it's possible to delete them and mlvfs crashes its a..s off :) tested.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 26, 2016, 09:06:42 AM
After some pause I finally find time to continue with MLVFS testing.
Setup: Dokan 1.0.0-RC1, MLVFS_x64.zip 25-Feb-2016 20:24, Resolve 12.3.1.001
Issues:
1. Any file with non-MLV and non-IDX extension present in the root of the virtual disk. All the chunks, wavs, and so on. Actually, its not a problem, just wonder why.
2. Performance is greatly redused. I have constant lags in playback. Without any option enbled(everythig "OFF" in "Configuration Options") mlvfs_x64 consumes 52-56% of CPU all the time and this looks as main reason for that.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 26, 2016, 09:29:39 AM
Right. On my system plain file copy shows that, CPU is higher 5-7% vs 10-14%, but speed is about the same 100-125mb/sec.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 26, 2016, 10:01:15 AM
@g3gg0: (OT) what the snd_viz.mo does?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 26, 2016, 05:31:01 PM
computer is looking for dokanfuse1.dll on my windows 10 machine after i press the command on cmd

now downloaded the latest mlvfs_x86 file version 2/11/2016 and when i execute cmd i get the message 'unmount' but no files.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 26, 2016, 05:51:06 PM
No success. I finally give up :-(

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on February 26, 2016, 06:52:29 PM
Just installed it for the first time on W7. Dokany 0.8.0. Worked.
May I offer a Teamviewer session?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 26, 2016, 07:05:52 PM
what is a team viewer session?

after giving up on windows 10, i tried the mac but am getting a permission denied for the mlvfs executable file - what do I do from here?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 26, 2016, 08:54:54 PM
@Canon eos m:

win10:
please uninstall dokan from your control panel
install dokan v1.0.0-rc1 (https://github.com/dokan-dev/dokany/releases/download/v1.0.0-RC1/DokanSetup.exe)
unpack mlvfs_x64 (http://www.magiclantern.fm/modules/modules/MLVFS_x64.zip/MLVFS_x64.zip) into your folder f:\mlvfs_x64\
open a commandline
Code: [Select]
f:
cd \mlvfs_x64
mlvfs_x64.exe Z:\ --mlv-dir=F:\

osx:
did you do a chmod +x FILENAME ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 26, 2016, 10:38:04 PM
@g3gg0

you are right too much noise did you read my messages above?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 26, 2016, 11:20:04 PM
yeah, the higher load -> need to examine
Mxx files -> todo

my goal, after breaking some stuff, is to make it usable again and these are nonfunctional requirements.
had to rework a lot of stuff due to the win32 specialities and added new features like proper .MLD support with nesting folders etc.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 26, 2016, 11:42:29 PM
yeah take your time, thanks for doing all this stuff.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 27, 2016, 02:45:40 AM
ok getting closer to the end goal .. on my mac have everything lineup but now there is localhost8000 error: safari cannot open page localhost8000 because safari cannot connect to server 'localhost'

tried to execute chmod + x but get the error the file does not exist (at least now know what chmod does  :-)



success on mac .. finally .. will try on Windows10 later
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on February 27, 2016, 03:20:23 AM
Now that makes the two of us reporting these so called web GUI errors on OS X ... Anyone else experiencing this as well?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 27, 2016, 05:32:52 AM
delete ~/.mlvfs.log and then run it one time where it fails and then send me the log (I want the log of a single failing run)

is there anything mlvfs related in ~/Library/Logs/DiagnosticReports ?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 27, 2016, 06:19:37 AM
i finally cracked it. downloaded the latest FUSE version that g3gg0 shared yesterday (see a few postings above the last one). It was an easy ride thereafter:

1. the FUSE automatically stations in the desktop folder
2. then, i created a blank folder in the desktop folder (mine was called MLV) - remember this should be blank
3. transferred, the camera recorded mlv file to a new folder on my desktop (not the blank MLV folder mentioned above - this should stay blank for till the FUSE is run on it)
4. then went to this folder with the mlv recorded file and right clicked. Created an association with the mlvfs format - this is the second last menu 5. 5. option on the drop down that shows up when you right click the mlv recorded folder
6. the moment you click the mlvfs tab it will show a folder for you to look for the place where you want to mount and transfer your extracted mlv file. 7. here i just directed the mac to the previously created blank MLV folder and the rest is history
8. opened the extracted files as a sequence in AE .. all showed up fine

Hope this helps everyone struggling to make this work - i have literally died trying to set this up - cannot still work out how to do this on Windows (that would be better since i have all my recorded mlv's in there) - but the donkey folder shows up some error (shared the error with g3gg0) - lets see

I am still trying to figure out how to covert several mlv files in one go or at different times using the same folder the I first created (at the moment have to create a new one each time). Plus, the FUSE GUI that opens up continues to show the first file i covered and now the new ones

Lastly, i covered the files and imported the sequence in AE. This took me to the ACR GUI. I changed the frame but the changes stay with the first frame - anyone know how i can transfer the effects and changes to other frames in the sequence

Thanks
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Frank7D on February 27, 2016, 06:42:16 AM
"Lastly, i covered the files and imported the sequence in AE. This took me to the ACR GUI. I changed the frame but the changes stay with the first frame - anyone know how i can transfer the effects and changes to other frames in the sequence"

When you import an image sequence in AE, the changes to the first frame should be applied to all frames in the sequence.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 27, 2016, 08:22:29 AM
Hi Frank, they aren't. Have been googling this since morning. when i log out of ACR the xmp sidecar isn't created either. even forcing ACR is not working. So, when I import the first frame is the corrected one but other don't carry the change. still looking will share if i do find what the problem was.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 27, 2016, 09:57:32 AM
@Canon eos m:
did you uninstall all dokan versions from your control panel before?

> but the donkey folder shows up some error (shared the error with g3gg0) - lets see
you mean "dokan". you sent me the *installer* logfile.

for the installer: are you sure you have all windows updates installed on your system?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 27, 2016, 11:04:37 AM
@bouncyball:
if you redownload, it might be a bit faster. i am caching internal structures now and skip checking for existense of the MLV file internally.

getting inconstistent performance measurement results when using resolve.
the most CPU time seems to be used in the locking mechanism.

guess resolve is loading the files in different threads and the locking is a bit coarse.
but i am trying to find that out.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 27, 2016, 03:18:28 PM
@g3gg0, I did uninstall all versions that the I had and checked. Not sure about the latest windows updates - Wil check later tonight and report.

Dokan is clearly the challenge. Have the Mlvfs working on the Mac for the moment and am satisfied with the speed of conversion bit struggle to understand the GUi interface and related. Will work to master in the coming days and report progress. I wanted to work with mlvfs since I want to work with mlv lite. Hope to test that too this week and next.

But for now the more pressing issue is the dng and xmp side car problem I am facing. Need to work out how to transfer modifications made with ACR on a single frame to all others. Mac not creating sidecars but understand that is not possible with dngs anyway since the xmp data is baked into the dng. So, the problem maybe with AE - not sure yet.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on February 27, 2016, 03:27:46 PM
XMP sidecars work fine on Mac
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 27, 2016, 04:19:59 PM
@g3gg0

if you redownload, it might be a bit faster. i am caching internal structures now and skip checking for existense of the MLV file internally.

Nice :D you are on a good direction!

Plain copy mlvfs CPU usage: (tested on my home pc which is more powerful (1ghz diff) hence smaller numbers than in previous posts)
25feb build - average ~7-8%, peak ~18%
27feb build - average ~4-4.5% peak ~15%
Throughput about the same ~115mb/sec
Results are collected from 10 runs each, hence no placebo :)

In resolve 27feb build definitely snappier! almost no lug on play/pause/rewind/fast forward/scrub, however, not very-very-very sure 'bout it, could be a lil'bit placebo effect ;)

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 27, 2016, 05:07:03 PM
Hopefully fixed fuse pass through issue on Mac (XMP sidecars not getting created), new build uploaded.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on February 27, 2016, 05:07:59 PM
if you redownload, it might be a bit faster.
Code: [Select]
mlvfs_wrap_getattr(1746): caught exception 0xC000001D at address 0xB664A3F3
As soon as i try to open virtual disk :(
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 27, 2016, 05:32:04 PM
mlvfs_wrap_getattr(1746): caught exception 0xC000001D at address 0xB664A3F3

Strange. None of exceptions here. And this is not the exception (mlvfs_wrap_read: 0xC0000005) I've been getting  in earlier builds, but when it was happening, I'm sure, cause of it was antivirus real time protection. Do you run mlvfs with administrator rights? Happens with all mlvs?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: lanternman on February 27, 2016, 10:43:33 PM
Hi guys,

I'm on Windows 10 x64 and using the new port with Dokan and mlvfs x86.

Tried mounting my MLV video folder on a random drive letter (X or Z) and there weren't errors, but the drive did not appear in my windows file explorer. any idea how I can get it to show up on my Premiere Pro CC?
http://imgur.com/aPaKGJo
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FaPaKGJo.png&hash=2eb95f9eee8482e31a8b8c4063e64c3d)
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fi.imgur.com%2FloDyJq3.png&hash=8d770e55c62dff5c4eb113f44a5334c7)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on February 27, 2016, 10:49:54 PM
Are you able to access localhost:8000 in your web browser?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 27, 2016, 11:41:30 PM
@dmilligan:
i created a new branch (https://bitbucket.org/dmilligan/mlvfs/branch/path_rework) which is working already well on windows.
it features a proper subdir handling and rename/delete support without errors.

can you test if it does it compile on osx?
thanks :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 27, 2016, 11:59:48 PM
no, O_BINARY is not defined, nor is _unlink, and there are some implicit function declarations

in addition theres a bunch of const warnings for modifying the 'path' parameter, which is declared const, perhaps its best to make a copy
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 28, 2016, 12:04:24 AM
ah it is complaining about the implicit const to non-const conversion in e.g. mlvfs_resolve_path?
how picky clang is :)


thanks, found it. indeed modified const string.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 28, 2016, 12:26:40 AM
seems to be working though

OSX Finder has issues copying files in and such but it always has. Command line cp works fine, except it reports: fchmod failed: Function not implemented (which has also been the case previously)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 28, 2016, 12:34:00 AM
okay fixed these issues in repository hopefully. yay cool, looks good then.

i also added unlink and stafs (which returns fake data), it wasn't implemented before.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: lanternman on February 28, 2016, 03:56:43 PM
No I am unable to. localhost = 404 not found.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Walter Schulz on February 28, 2016, 04:11:38 PM
Check:
Do you have VC++ Runtime installed? Install both, shouldn't bother.
Which Dokany version installed?
Do you have started CMD as Administrator?
Running
Code: [Select]
net start|find /I "DokanMounter"should report
Code: [Select]
E:\MLVFS_x86\MLVFS_x86>net start|find /I "DokanMounter"
   DokanMounter
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 28, 2016, 04:15:16 PM
404 means that there is something missing. did you run it from the directory where it is installed?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 28, 2016, 05:32:07 PM
g3gg0, checked widows is uptodate but dokan still has installation error 0x80091007 - hash value is not correct
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Canon eos m on February 28, 2016, 05:39:02 PM
g3gg0, if i use the old dokan on the opening page then the error is - the program cant start because dokanfuse.dll is missing from your computer. try installing the program to fix this problem
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 28, 2016, 07:45:33 PM
@g3gg0, dmilligan

compiled path_rework branch, win version of mlvfs in VS
commit e7a8e68 - with lot of warnings but ok
following commits (3 more) - failed with errors

mlvfs.h(137): error C2010: '.': unexpected in macro formal parameter list
mlvfs.h(139): error C2010: '.': unexpected in macro formal parameter list

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 28, 2016, 08:12:32 PM
ah, boo microsoft

variadic macros work fine in gcc and clang
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on February 28, 2016, 09:06:29 PM
ok, maybe fixed, try again
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 28, 2016, 10:02:07 PM
> Error   C2039   '_unlink': is not a member of 'fuse_operations'   mlvfs   D:\cygwin\home\g3gg0\git\mlvfs\mlvfs\main.c   1808      Build      
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on February 28, 2016, 10:04:21 PM
you can use _unlink on windows too btw
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on February 29, 2016, 11:32:56 AM
Compile latest path_rework branch -> OK! (with lots of warnings as usual)

Tested plain copy average cpu/speed: 3.5-4%/105-110mb/sec
Passtrough is working exept deletion as expected. No nasty exceptions so far.

Nice :)

Congrats!
bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on March 01, 2016, 06:01:40 AM
So, I need to compile myself? I'm not sure if it will work with cygwin i just managed to compile ML itself...
Binaries from mlvfs_optimized.zip are the last one worked fast not furious for me with Resolve.
But now they claims for dokanfuse.dll while last MLVFS has dokanfuse1.dll.
I just renamed dokanfuse1.dll to dokanfuse.dll and happy until developers finally polish this great tool.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 01, 2016, 08:02:10 AM
If you wanna test latest bin pm me. Can't just post it here. Would be interesting to see because you got some exception recently.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on March 01, 2016, 08:42:46 AM
Thanks, I appreciate your willing to help but I already converted footage i had with described setup and now have no more records and spare time, so lets things move as it could.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ruber on March 01, 2016, 12:46:07 PM
Hi guys,

I am in windows 10

I actually use Pismo and convert mlv files directly in davinci resolve. Is really smooth.

But Resolve did not recognized audio and video linked so I spent hours relinking audio and video together

Is there a solution? (newer version, or a different software than Pismo)?

Thank you. Cheers.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: lanternman on March 01, 2016, 12:48:38 PM
hi guys, MLVFS finally worked in my other desktop.

1 question: How do I quickly add ALL clips in 1 folder to Premiere Pro CC, instead of having to select the 1st DNG of EVERY short clip manually?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 01, 2016, 12:58:19 PM
Is there a solution? (newer version, or a different software than Pismo)?
Yes, start reading here: http://www.magiclantern.fm/forum/index.php?topic=13152.msg161902#msg161902
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ruber on March 01, 2016, 07:36:18 PM
thanks dmilligan

kind as usual  ;)

I'll download it and install
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 01, 2016, 09:40:29 PM
Shot some test MLV_Lite files on 5D3 running 123 today earlier and just tried using the latest MLVFS (Version: 93684e7 2016-02-27) to check out the files.

The WebGUI works like normal but then the Mount folders are empty. Both of them.

Here's part of the log from the beginning since the entire code is too long for this post.

Code: [Select]
Last login: Mon Feb 29 14:55:41 on console
Apples-Macintosh-10:~ DeafEyeJedi$ cat ~/.mlvfs.log
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/favicon.ico...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon-precomposed.png...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/favicon.ico...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/apple-touch-icon.png...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2141.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2142.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2143.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2147.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2144.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2148.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2145.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2151.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2146.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2153.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2152.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2156.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2159.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2248.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2246.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2249.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2250.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2251.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2258.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2301.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2257.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV...
/Volumes/KB_128GB/DCIM/100EOS5D/M16-2256.MLV: Error reading frame headers: vidf block for frame 0 was not found
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2302.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2303.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2304.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2305.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2306.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2308.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2307.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2310.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2309.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2311.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2314.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2313.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2312.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2315.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2316.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2320.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2319.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2318.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2317.MLV...
webgui: analyzing /Volumes/KB_128GB/DCIM/100EOS5D/M16-2321.MLV...
webgui: scanning /Volumes/KB_128GB/DCIM/100EOS5D/...

Should I consider downgrading to previous versions of MLVFS? I'm also running these same files with MLP atm and all seems fine and running like normal though I still have to wait for the final results.

Will report back and Thanks, D!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 01, 2016, 10:14:19 PM
The beginning of the log tells me nothing, it's old (same as the one you posted before). Send the end of the log, upload the whole log somewhere, or preferably: delete the log before running MLVFS, then run MLVFS with the problem files and then send the log that is generated so that it's just the info the run that failed.

Code: [Select]
mv ~/.mlvfs.log ~/old.mlvfs.log
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 01, 2016, 10:28:54 PM
Okay something odd is going on. I just randomly decided to try AE and they show up fine in there. I can see the files in the Mount folder. However, when clicking on DAvinci Resolve within Web GUI -- thats when the files disappear inside the mount folder and thus I can't see the files in DR12.

Here's log from while in DR mode:

Code: [Select]
Last login: Tue Mar  1 13:24:29 on ttys000
Apples-Macintosh-10:~ DeafEyeJedi$ cat ~/.mlvfs.log
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
Apples-Macintosh-10:~ DeafEyeJedi$


Here's the log from while in Default mode:

Code: [Select]
Last login: Tue Mar  1 13:24:31 on ttys000
Apples-Macintosh-10:~ DeafEyeJedi$ Last login: Tue Mar  1 13:24:29 on ttys000
Apples-Macintosh-10:~ DeafEyeJedi$ cat ~/.mlvfs.log
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MDeafEyeJedi  ttys000                   Tue Mar  1 13:27   still logged in
DeafEyeJedi  ttys000                   Tue Mar  1 13:24 - 13:26  (00:02)
DeafEyeJedi  ttys000                   Tue Mar  1 13:24 - 13:24  (00:00)
DeafEyeJedi  ttys000                   Tue Mar  1 13:23 - 13:24  (00:00)
DeafEyeJedi  ttys000                   Tue Mar  1 13:22 - 13:23  (00:01)
DeafEyeJedi  ttys000                   Tue Mar  1 12:36 - 12:41  (00:04)
DeafEyeJedi  ttys000                   Sat Feb 27 14:52 - 15:20  (00:28)

wtmp begins Fri Feb 26 19:16
Apples-Macintosh-10:~ DeafEyeJedi$ Apples-Macintosh-10:~ DeafEyeJedi$ cat ~/.mlvfs.log
-bash: Apples-Macintosh-10:~: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1246.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1247.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1251.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: analyzing /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/M01-1252.MLV...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ webgui: scanning /Users/appleuser/Desktop/SHOGUN vs MLV_Lite/MLVLite/...
-bash: webgui:: command not found
Apples-Macintosh-10:~ DeafEyeJedi$ Apples-Macintosh-10:~ DeafEyeJedi$
-bash: Apples-Macintosh-10:~: command not found
Apples-Macintosh-10:~ DeafEyeJedi$
Apples-Macintosh-10:~ DeafEyeJedi$

Now I am baffled at why the log is coming out differently even tho it's using the same exact set of MLV Lite files?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 01, 2016, 10:32:23 PM
The naming scheme option changes the file/folder names, so if you are going to do it, then you should do it first, immediately before doing anything else like browsing files. Otherwise you are ending up in folders that don't exist any more (b/c you changed the naming scheme for them).
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: squig on March 02, 2016, 12:53:41 AM
It's not a user error issue. It's an issue with the new Mac build. I'm using MLVS and Resolve in the same manner I've been using them for almost a year. Choosing the Resolve naming scheme at launch prior to launching Resolve results in the MLVS folders appearing empty in the Resolve media window. In previous builds a thumbnail image appeared in place of a folder. Leaving the default naming scheme in place at launch results in a single file appearing within the MLVS folders in the Resolve media window (appearing as a thumbnail image). I can't open any of my Resolve MLVS projects because they were all created with the Reslove naming scheme.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 02, 2016, 04:44:42 AM
There's been some rework to the path handling and the resolve naming scheme had not yet been implemented (I didn't realize this). I've just implemented it and uploaded a new build. The actual naming scheme had to change slightly b/c of the new way we are doing things, sorry if this breaks your existing projects.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: newsense on March 02, 2016, 07:43:45 AM
I'm using MLVFS to convert my MLV files and it works great. I want to use a camera profile in ACR but none show up. the stock camera profiles include profiles for the 650D but in ACR my camera is listed as Rebel T4i, would that info be stored in the DNG's produced by MLVFS?
Title: Please, white and black levels in web gui
Post by: mothaibaphoto on March 02, 2016, 08:56:26 AM
Some time ago I realised that MLVP produce dng that looks something different in Resolve.
Difference among other convertors was minor, while with MLVP - huge.
Same time ACR gets confused with predefined camera profiles, only "Embedded" looks good.
Author responded that its due to different white level - MLVP use 16383.
Moreover, it's possible to change this value.
Need to say my camera is always 16383 in video mode(okay, while not tricked with adtg_gui :) ).
MLVFS seems use 15000 hardcoded.
Today I tried to tune MLVFS dng with exiftool.
Need to say my exiftool is very choosy regarding changing these tags, white level works only with -IFD0:
Code: [Select]
exiftool -IFD0:WhiteLevel=16383 *.dng -overwrite_originalAnd ... I didn't find the way to change black level(from 2047 to 2044)
so I don't know if it matter or not but changing white level alone makes difference.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: newsense on March 02, 2016, 09:17:18 AM
Some time ago I realised that MLVP produce dng that looks something different in Resolve.
Difference among other convertors was minor, while with MLVP - huge.
Same time ACR gets confused with predefined camera profiles, only "Embedded" looks good.
Author responded that its due to different white level - MLVP use 16383.
Moreover, it's possible to change this value.
Need to say my camera is always 16383 in video mode(okay, while not tricked with adtg_gui :) ).
MLVFS seems use 15000 hardcoded.
Today I tried to tune MLVFS dng with exiftool.
Need to say my exiftool is very choosy regarding changing these tags, white level works only with -IFD0:
Code: [Select]
exiftool -IFD0:WhiteLevel=16383 *.dng -overwrite_originalAnd ... I didn't find the way to change black level(from 2047 to 2044)
so I don't know if it matter or not but changing white level alone makes difference.
Thanks but I think I'll wait to see if anyone has a proper fix or try another app but I do appreciate the help
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on March 02, 2016, 09:41:33 AM
Thanks but I think I'll wait to see if anyone has a proper fix or try another app but I do appreciate the help
Sorry, my post was not addressed to your problems.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 02, 2016, 09:42:13 AM
Compiling under VS fails again:

>main.obj : error LNK2001: unresolved external symbol slre_match
>mlvfs.exe : fatal error LNK1120: 1 unresolved externals
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 02, 2016, 01:09:36 PM
Compiling under VS fails again:
yep, I know, see the commit messages

MLVFS seems use 15000 hardcoded.
Nope. It uses the value in the MLV file, same with black level. If it's wrong, then it's wrong in the MLV file => therefore you need to go to the source of the MLV file to fix the issue.

And ... I didn't find the way to change black level(from 2047 to 2044)
so I don't know if it matter or not but changing white level alone makes difference.
White level and black level are objective values that are either correct or incorrect. There is no "tuning" you should do to these values to try and adjust the "look" of your image.

the stock camera profiles include profiles for the 650D but in ACR my camera is listed as Rebel T4i, would that info be stored in the DNG's produced by MLVFS?
Yep, and again this is something that comes out of the MLV file.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: newsense on March 02, 2016, 02:11:16 PM
yep, I know, see the commit messages
Nope. It uses the value in the MLV file, same with black level. If it's wrong, then it's wrong in the MLV file => therefore you need to go to the source of the MLV file to fix the issue.
White level and black level are objective values that are either correct or incorrect. There is no "tuning" you should do to these values to try and adjust the "look" of your image.
Yep, and again this is something that comes out of the MLV file.
Just to make sure I understand the UniqueCameraId is stored in the MLV file so it's ML reporting my camera as rebel t4i instead of 650d so this happens before I even convert to DNG?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Danne on March 02, 2016, 02:26:21 PM
You are double posting http://www.magiclantern.fm/forum/index.php?topic=16738.msg163431#msg163431.
Your camera can either register the t4i or with 650D tag, it,s nothing wrong with either one. If acr plugins or other applications can,t read that tag you can change the tag to whatever you want with exiftool.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: squig on March 02, 2016, 02:50:38 PM
There's been some rework to the path handling and the resolve naming scheme had not yet been implemented (I didn't realize this). I've just implemented it and uploaded a new build. The actual naming scheme had to change slightly b/c of the new way we are doing things, sorry if this breaks your existing projects.

Okay, it appears we're back in business. MLVs are loading in Resolve and the thumbnails are visible. No problem re the naming scheme change, I can run an older version for existing projects.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on March 02, 2016, 04:32:20 PM
Tried to reproduce this issue (http://www.magiclantern.fm/forum/index.php?topic=13152.msg159718#msg159718), but couldn't (probably it was fixed meanwhile).

Instead, I've found an issue with webgui on Linux (which used to work): it gives a blank page, and this message in console:
Code: [Select]
load_resource: fopen error: html_template.html

It doesn't seem to look in the right path - getcwd() returns the directory where I've started mlvfs from.

The old version where the webgui used to work didn't use an external HTML template - it was hardcoded in webgui.c.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 02, 2016, 05:07:47 PM
yep, I know, see the commit messages

@dmilligan: Ah, my bad. Added slre source and header to the project and it compiled flawlessly :)
testing...
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 02, 2016, 06:00:15 PM
It doesn't seem to look in the right path - getcwd() returns the directory where I've started mlvfs from.
That's odd, it just sends a relative path to fopen. Maybe it doesn't like the mode option "rb"?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mothaibaphoto on March 02, 2016, 06:07:51 PM
Nope. It uses the value in the MLV file, same with black level. If it's wrong, then it's wrong in the MLV file => therefore you need to go to the source of the MLV file to fix the issue.
So, it's hardcoded in raw.c:
Code: [Select]
#define WHITE_LEVEL 15000Actually, raw.c has routine to define white level - autodetect_white_level, but it's not called,
at least not in video mode: i uncommented debug message inside it and that "White..."
didn't appears on screen. Is that a bug?
White level and black level are objective values that are either correct or incorrect. There is no "tuning" you should do to these values to try and adjust the "look" of your image.
It's up to you, whether to implement or not the feature I suggest to your project.
Just let me decide myself what and how I can "tune".
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: a1ex on March 02, 2016, 11:50:43 PM
That's odd, it just sends a relative path to fopen.

That relative path is just "html_template.html", and I guess it should include "data" (maybe also find somehow the directory of the executable, or expect some specific path where it should be installed).

So, I started mlvfs from the data directory (../mlvfs [options]) and the webgui works again.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 03, 2016, 01:55:23 AM
I intended to just have everything in the same directory to avoid cross platform discrepancies (like different path separators and whatnot). I realize it's confusing for those things to be in the "data" directory in the source tree. I'm open to suggestions.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: QuickHitRecord on March 04, 2016, 09:39:08 AM
After updating to the latest nightly build today, I also installed the newest version of MLVFS and decided to do another body cap ISO test (see the video description for more details):


I'm not convinced that changing the boxes in the web interface is doing anything. The clips with "Bad Pixel Fix" checked don't seem to be any different than those without it. What am I doing wrong? I restarted Resolve after making the changes to the web interface so that they would take.
Title: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: DeafEyeJedi on March 04, 2016, 09:56:31 AM
I'm not convinced that changing the boxes in the web interface is doing anything. The clips with "Bad Pixel Fix" checked don't seem to be any different than those without it. What am I doing wrong? I restarted Resolve after making the changes to the web interface so that they would take.

Maybe you forgot to 'refresh' the Web interface after making your adjustments checked? I've had this happened to me occasionally whenever I forget. [emoji6]

@dmilligan - I ran a few test MLV_Lite files with latest MLVFS (Thanks for the update) and it works like normal with these files.

However, when I try running a folder with regular MLV files which then shows me the web gui error. Strange?

Link to MLVFS Log ... https://mega.nz/#F!alU0DTJa!9Z-kho7OPZPg7msyltc9-g

*edit*

Ah, Ha ... Looks like I managed to figured it out on my end (not sure why this wasn't an issue in the past) but I decided to rename my Folders that didn't have any (spaces or ' , _ etc) and reran latest MLVFS with all of my old MLV files from varies of cameras and all seems to be working like normal again. Whew!

Sorry for the False alarm even tho I could swear that Ive used Spaces or Punctuation Marks with no problems (according to my memory) or no?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: QuickHitRecord on March 04, 2016, 09:17:28 PM
Maybe you forgot to 'refresh' the Web interface after making your adjustments checked? I've had this happened to me occasionally whenever I forget. [emoji6]

You were right. Thanks for the tip. I re-did the test, and expanded on it:


Some takeaways from this exercise:
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 04, 2016, 10:27:40 PM
Sorry for the False alarm even tho I could swear that Ive used Spaces or Punctuation Marks with no problems (according to my memory) or no?
Spaces, quotes, and all sorts of manner of strange characters work just fine in the file system. It appears that some characters cause files not to show up in the webgui, this is probably b/c the json/url requests aren't being escaped properly.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 05, 2016, 02:02:52 AM
Think it should be fixed now.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: voblaunsane on March 05, 2016, 09:49:24 PM
Hi all,

I love the concept, but I cant get Premiere CC to properly deal with the files. After importing the sequence, scrubbing works nowhere (media bowser/bins/timeline), hence I initialy thought it does not work at all, BUT it plays back with no problem after a pausing a moment. Mind you that I have no such problems with dng extracted using raw2cdng 1.7.4

Alas, the biggest issue remains rendering, where I get "error compiling movie. unknown error" and therefore being unable to export at all! Again, this does not occur with dng's extracted with raw2cdng 1.7.4.. Here's what I get in console after export attempt:

Code: [Select]
H:\DCIM\100EOS5D/M04-1511.MLV: Error reading frame headers: vidf block for frame 0 was not found
H:\DCIM\100EOS5D/M04-1511.MLV: Error reading frame headers: vidf block for frame 0 was not found
H:\DCIM\100EOS5D/M04-1511.MLV: Error reading frame headers: vidf block for frame 0 was not found
H:\DCIM\100EOS5D/M04-1511.MLV: Error reading frame headers: vidf block for frame 0 was not found
mlvfs_wrap_read(1497): caught exception 0xC0000005 at address 0x7061D72E

I'm running this on Win7 SP1


Context Menu for Win

I would also like to make a small contribution to windows users adding a context menu option "Mount with MLVFS" to your folders. Technically registry entry only should suffice, but I found that webGUI loads blank and console outputs fopen errors without setting command path to mlvfs folder of residence before executing, hence the bat file..
Obviously, you'll have to change paths to suit your requirements.

mlvfs.reg (execute this file to add context menu option for your folders):
Code: [Select]
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\MLVFS]
@="Mount with MLVFS"

[HKEY_CLASSES_ROOT\Directory\shell\MLVFS\command]
@="\"C:\\Program Files (x86)\\MLVFS_x86\\mlvfs.bat\" \"%1\""

mlvfs.bat (place this next to mlvfs.exe)
Code: [Select]
@echo off
c:
cd "c:\Program Files (x86)\MLVFS_x86"
mlvfs.exe x:\ --mlv_dir=%1
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 06, 2016, 02:52:20 PM
Thanks. Context menu is nice.

I've tried this method some time ago and mlvfs path was a problem for webgui as you described and I ended up with .bat command file as you did :). Besides I always run mlvfs with admin rights b/c dokan sometimes could not properly unmount the drive. I've tested with mlvfs in compatibility mode as admin (opens 2 console windows :P if executed from bat) and with "runas.exe /user:Administrator ..." command and it sorta worked the way I'd like but it complicated things a bit (enter password etc) and I gave up and just use the shortcut with "run as administrator" checked on or just run it from FarManager console session.

I think the best way to solve this problem for all platforms is to hardcode the html and js resource into mlvfs binary itself (early versions were this way if I'm not mistaken). However, I guess, David had his own reasons for current implementation.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 06, 2016, 07:02:50 PM
Constantly recompiling to fix html, css, and ajax is annoying. Writing all the html as string literals is also quite cumbersome, and I don't know of a good, simple, reliable, cross-platform way of inserting text documents into binaries. Some particular method might work well for gcc/make but not LLVM or VS and vice versa. Much simpler to just load them. With the whole focus pixel thing we have the need for additional resource files as well, so it was starting to get a little out of hand to keep everything in the binary. Plus it means if/when new focus pixel cameras are added, pixel maps can be added without recompiling, and people can make their own maps to fix things like troublesome hot/dead pixels.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on March 06, 2016, 07:53:32 PM
the only proper solution is making some kind of "resource" (i.e. zipping all files) that is either linked (depends on which linker you use) or converted to a .c with some  "char resourceData[] = { 0xaa, 0xbb, ... }".
but thats a bit annoying as you need additional helpers.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: voblaunsane on March 07, 2016, 03:37:00 PM
any idea about the unhandheld (haha, sorry/not sorry) exceptions and errors reading frame headers, which I am guessing are the causes of problematic handling by premiere?...

I've described those few posts above..
http://www.magiclantern.fm/forum/index.php?topic=13152.msg163728#msg163728

Please let me know if there is any troubleshooting or testing I could do in helping to improve it!

Thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Roberto Mena on March 08, 2016, 10:34:45 AM
Hi Guys. Thanks for all the amazing work. I downloaded FUSE and MLVFS but I am not getting the "Services" option when I right click a folder containing mlv files, showing at the bottom of the window near the "Tags" section. What am I doing wrong? Thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 08, 2016, 01:45:49 PM
any idea about the unhandheld (haha, sorry/not sorry) exceptions and errors reading frame headers, which I am guessing are the causes of problematic handling by premiere?...

I never experienced frame header error with premiere or anything else. You should double check that particular mlv file in other tools.

mlvfs_wrap_read(1497): caught exception 0xC0000005 - was very common (in early windows mlvfs builds) until I disabled antivirus real time protection.

bb
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on March 09, 2016, 12:55:14 AM
any idea about the unhandheld (haha, sorry/not sorry) exceptions and errors reading frame headers, which I am guessing are the causes of problematic handling by premiere?...

can you check if the file with vidf errors reads fine using mlv_dump?
can you run mlvfs.exe with parameter -d and upload the whole log? (append "> log.txt" to log into a file)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: USADavid on March 10, 2016, 01:29:38 AM
Hey guys,

long time creep, but never had a reason to post anything. Great work everyone is doing here. Love the simplicity and cross-platform capability of MLVS.
Unfortunately, I've had issues with the sound for a while now, blamed it on me first, but after trying a lot of different things never seem to be able to fix it.

I'm running the latest MLVFS on my Mac (2016-03-04) and all my MLVs which are around > 2GB have an empty wav file. The wav files still show up even with the alleged size, but there's no sound. If I open up the MLVs in MlRawViewer, it plays just fine. I'm on the latest Fuse and even tried the Beta, also the ~./mlvfs.log file doesn't offer any clues on why this occurs.

Any help would be greatly appreciated.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: voblaunsane on March 10, 2016, 05:48:46 AM
can you check if the file with vidf errors reads fine using mlv_dump?
can you run mlvfs.exe with parameter -d and upload the whole log? (append "> log.txt" to log into a file)

Thanks!

I tried all combinations of commands I could've possibly ran, alas was unable to reproduce, very strange because I received those all first day of use and then next day it was gone like nothing ever happened! Also deleted the files before reading you post, aww..

Yet main issue being unable to render persists! So I ran mlvfs with -d 2>log.txt mounting a folder with one sequence (3 files mlv/m00/idx), created new premiere project, located sequence in media browser, dropped it straight on the timeline and hit render.  This produced "error compiling movie. unknown error" and 54.6MB logfile here: https://drive.google.com/file/d/0Bzfm11RAqkBzaXgzOTUwQ1dIYUU/view?usp=sharing (https://drive.google.com/file/d/0Bzfm11RAqkBzaXgzOTUwQ1dIYUU/view?usp=sharing). Not knowing what I'm looking for tried to do some detective work and the following seems to be all over the place:
Code: [Select]
CreateFile status = 0 - lastError = 2
This mlv was recorded with no sound, but I see the following in report alot.
Code: [Select]
CreateFile : \M04-1513.MLV\M04-1513.wav
I left an instance of mlvfs with -d running while typing this post, new log file is nearly 2GB - I'm scared :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: g3gg0 on March 10, 2016, 08:19:15 PM
if you extract the files using mlvfs and let premiere work on the extracted files, does that work?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: voblaunsane on March 10, 2016, 08:44:05 PM
Found a few things here:

1. Finally I managed to render mlvfs on premiere! I found that 25fps material works flawlessly at a lighting speed, when 23.976fps is impossible to scrub and it doesn't render "error compiling movie. unknown error"... when that same 23.976fps mlv, extracted with raw2cdng works just fine. Any ideas?

I logged the activity with separately mounted differing fps files here: https://drive.google.com/file/d/0Bzfm11RAqkBzN0dnV05kMGpmSms/view?usp=sharing

2. Mounting a full CF results in empty mlv folders. localhost:8000 reports 0 frames in files found. Freeing some space resolves the issue. It could be nice have mlvfs reporting "not enough disk space".. Otherwise it gives the impression that files are damaged..

Thanks!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: voblaunsane on March 10, 2016, 08:49:47 PM
if you extract the files using mlvfs and let premiere work on the extracted files, does that work?

no. same issues with 23.976. Works fine, same as on the fly with 25fps..

Using mounted 23.976 material renders always fail at a particular stage, until I drag the clip to another sequence, then it gets a new fail point. More often then not fail point is 99%.. The extract from mlvfs, even in new sequence, always fails at the similar point.. but same material extracted again, gets a new fail point .. makes sense?

Also caught the exception, actually all 6 of them, but was running many tests and can not reproduce, here's the log: https://drive.google.com/file/d/0Bzfm11RAqkBzb0xwbHpTT3JqSEE/view?usp=sharing
Title: Dokany trouble
Post by: Markus on March 12, 2016, 07:00:53 PM
I've installed dokan to try it out on win10 64bit but can't get it to function properly. I can mount the files and se the dng-files inside but if i try to open anything i just get: "load_chunks: fopen error: No such file or directory"

Anyone have any idea of what could be the problem?

v. 0.8.0 of dokan and MLVFSx86 from first post link
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on March 13, 2016, 05:35:33 AM
@Markus

First try using the latest supported versions of Dokan and MLVFS.

Links to these can be found here:  http://www.magiclantern.fm/forum/index.php?topic=13152.msg163041#msg163041

Note that you will need to uninstall Dokan first before reinstalling the latest version. The MLVFS download totally replaces your existing install instance.

Also, do the paths to your MLV files contain spaces?  If so, create a path without spaces and try again.

If it still fails after that, report back with some screen shots demonstrating successful Loading of Dokan, File mounts (A localhost one) and the error message when trying to access a specific DNG.

What application are you using to access the DNG's. The issue may be with the application and not MLVFS. what happens if you just double click on a DNG to view the image with windows default viewer.

@DMilligan - Might be worthwhile to update the links in the 1st post to reflect the latest supported versions.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on March 13, 2016, 05:50:19 AM
@Developers

Just a note to advise that Dokan v1.0.0-RC2 is available and, at least on my W10 system is working error free. Tested with single and multi-chunk MLV's.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: bouncyball on March 13, 2016, 12:04:04 PM
@Ottoga: Thanks for pointing it out!

@g3gg0, dmilligan: delete/copy file/dir in passthrough mode is OK! for RC2 :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on March 14, 2016, 12:17:22 PM
@Ottoga Thanks for reply! I'll try it out as soon as possible. Excited of getting this to work instead of pismo. Fix pattern noise seems extremely useful.
Does anyone know if it is possible to process dng - sequences for fixed pattern noise if you haven't saved the mlv's?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on March 14, 2016, 12:30:03 PM
@Markus

Quote
Does anyone know if it is possible to process dng - sequences for fixed pattern noise if you haven't saved the mlv's?

From memory, you might be able to use mlv_dump to recreate an MLV file from a DNG series. One of the Devs may be able to confirm that.

If this is confirmed, then the answer to your question is likely to be "yes".

Oh, I also uninstalled Pismo and rebooted my PC before installing Dokan. I didn't see any point having two services running in the background that are essentially trying to do the same thing. Just something else to consider.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 14, 2016, 09:22:59 PM
Not sure if this a issue use the new dual iso video for 5d2 but i'm have problems with the latest mlvfs for mac.
sometimes when i toggle on dual iso processing with default setting it doesn't always process  the dual iso .
Tried deleting the .idx file re-mounting the file , seems find on a older version i have on my hackintoss .
I'm working off a SSD , i see if there's any log files .
Also Is dual iso processing normally very slow ? I using AMaZe for Interpolation , Alias Map & Fullres Blending enable.
First time working with dual iso , with non dual iso there hardly any delay and can but edited etc.. in realtime in Resolve
but this is not possible with dual iso it seems , i play around with some of the setting and see if i can improve on this .
Just curious , when the web page opens and read the files etc.. shouldn't it say it's "dual iso" ? of give the two iso setting ?
all i see is the lower number e.g. 100 out of 100/800
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on March 14, 2016, 10:16:15 PM
The dual ISO algorithm is just very complex and slow, there's no way around it. There is a fast preview mode, but it is very low quality. So the usefulness of dual ISO in relation to the MLVFS "on the fly" paradigm is rather limited. Personally, I think dual ISO video is just not worth the effort and drawbacks (aliasing, loss in resolution, etc) and most people want to use simply because it sounds cool and somewhat of a placebo effect.

As for the webgui, there is no dual ISO related metadata in the MLV file (IMO there probably should be). So the only way to figure out if something is dual ISO or not is to actually do some complex analysis on a frame. Doing this would bog down the gui for everyone, not just dual ISO users.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 14, 2016, 10:58:56 PM
Sound good , i understand now thanks.



Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on March 16, 2016, 07:24:06 PM
There seems to be an issue with the timecode on 23.98fps material.

Using exiftool to look at the timecode from a DNG frame numbered 0 and the timecode is 00:00:00.00 as expected. All is fine until you get to frame 24 and the timecode jumps to 00:00:01.01. This +1-frame is cumulative adding a frame to the timecode every second.

I don't know if this is intentional or if it affects any other framerate.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on March 16, 2016, 07:43:17 PM
Still can't get MLVFS to work. (Win10 64bit)
I installed the new version of mlvfs with x64 bit support and dokan 1RC2 (tryed RC1 as vell) Uninstalled before reinstalling with boot in between.
MLVFS version from this post http://www.magiclantern.fm/forum/index.php?topic=13152.msg163041#msg163041  (http://www.magiclantern.fm/forum/index.php?topic=13152.msg163041#msg163041)
Used the following string to mount: "mlvfs_x64.exe C:\MLV --mlv_dir=F:\DCIM\100EOS5D\"
The MLV-file I tryed to mount was about 4.5 gig in size
This time I can not even enter the mounted catalog and keep getting the following error message when I try to:
mlvfs_wrap_getattr(1746): caught exception 0xC000001D at address 0x5AF8A3F3
 
even tryed going offline and deactivating avast anti-virus program...
Also uninstalled Pismo mounting.

So now I'm at a loss what more to try.
Anyone got any suggestions?
Any help would be greatly appreciated.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 16, 2016, 09:16:33 PM
There seems to be an issue with the timecode on 23.98fps material.
Feel free to correct this math, it came from chmee's tool. I suspect it doesn't handle rounding correctly.
Code: [Select]
    //from raw2cdng, credits: chmee
    int hours = (int)floor((double)frame / framerate / 3600);
    frame = frame - (hours * 60 * 60 * (int)framerate);
    int minutes = (int)floor((double)frame / framerate / 60);
    frame = frame - (minutes * 60 * (int)framerate);
    int seconds = (int)floor((double)frame / framerate) % 60;
    frame = frame - (seconds * (int)framerate);
    int frames = frame % MAX(1,(int)round(framerate));
   
    buffer[*data_offset] = to_tc_byte(frames) & 0x3F;
    if(drop_frame) buffer[*data_offset] = buffer[*data_offset] | (1 << 7); //set the drop frame bit
    buffer[*data_offset + 1] = to_tc_byte(seconds) & 0x7F;
    buffer[*data_offset + 2] = to_tc_byte(minutes) & 0x7F;
    buffer[*data_offset + 3] = to_tc_byte(hours) & 0x3F;

Here's the full output for a 23.976 file
Code: [Select]
$ exiftool -TimeCodes *.dng
======== M14-1233.MLV-C_000000.dng
Time Codes                      : 00:00:00.00
======== M14-1233.MLV-C_000001.dng
Time Codes                      : 00:00:00.01
======== M14-1233.MLV-C_000002.dng
Time Codes                      : 00:00:00.02
======== M14-1233.MLV-C_000003.dng
Time Codes                      : 00:00:00.03
======== M14-1233.MLV-C_000004.dng
Time Codes                      : 00:00:00.04
======== M14-1233.MLV-C_000005.dng
Time Codes                      : 00:00:00.05
======== M14-1233.MLV-C_000006.dng
Time Codes                      : 00:00:00.06
======== M14-1233.MLV-C_000007.dng
Time Codes                      : 00:00:00.07
======== M14-1233.MLV-C_000008.dng
Time Codes                      : 00:00:00.08
======== M14-1233.MLV-C_000009.dng
Time Codes                      : 00:00:00.09
======== M14-1233.MLV-C_000010.dng
Time Codes                      : 00:00:00.10
======== M14-1233.MLV-C_000011.dng
Time Codes                      : 00:00:00.11
======== M14-1233.MLV-C_000012.dng
Time Codes                      : 00:00:00.12
======== M14-1233.MLV-C_000013.dng
Time Codes                      : 00:00:00.13
======== M14-1233.MLV-C_000014.dng
Time Codes                      : 00:00:00.14
======== M14-1233.MLV-C_000015.dng
Time Codes                      : 00:00:00.15
======== M14-1233.MLV-C_000016.dng
Time Codes                      : 00:00:00.16
======== M14-1233.MLV-C_000017.dng
Time Codes                      : 00:00:00.17
======== M14-1233.MLV-C_000018.dng
Time Codes                      : 00:00:00.18
======== M14-1233.MLV-C_000019.dng
Time Codes                      : 00:00:00.19
======== M14-1233.MLV-C_000020.dng
Time Codes                      : 00:00:00.20
======== M14-1233.MLV-C_000021.dng
Time Codes                      : 00:00:00.21
======== M14-1233.MLV-C_000022.dng
Time Codes                      : 00:00:00.22
======== M14-1233.MLV-C_000023.dng
Time Codes                      : 00:00:00.23
======== M14-1233.MLV-C_000024.dng
Time Codes                      : 00:00:01.01
======== M14-1233.MLV-C_000025.dng
Time Codes                      : 00:00:01.02
======== M14-1233.MLV-C_000026.dng
Time Codes                      : 00:00:01.03
======== M14-1233.MLV-C_000027.dng
Time Codes                      : 00:00:01.04
======== M14-1233.MLV-C_000028.dng
Time Codes                      : 00:00:01.05
======== M14-1233.MLV-C_000029.dng
Time Codes                      : 00:00:01.06
======== M14-1233.MLV-C_000030.dng
Time Codes                      : 00:00:01.07
======== M14-1233.MLV-C_000031.dng
Time Codes                      : 00:00:01.08
======== M14-1233.MLV-C_000032.dng
Time Codes                      : 00:00:01.09
======== M14-1233.MLV-C_000033.dng
Time Codes                      : 00:00:01.10
======== M14-1233.MLV-C_000034.dng
Time Codes                      : 00:00:01.11
======== M14-1233.MLV-C_000035.dng
Time Codes                      : 00:00:01.12
======== M14-1233.MLV-C_000036.dng
Time Codes                      : 00:00:01.13
======== M14-1233.MLV-C_000037.dng
Time Codes                      : 00:00:01.14
======== M14-1233.MLV-C_000038.dng
Time Codes                      : 00:00:01.15
======== M14-1233.MLV-C_000039.dng
Time Codes                      : 00:00:01.16
======== M14-1233.MLV-C_000040.dng
Time Codes                      : 00:00:01.17
======== M14-1233.MLV-C_000041.dng
Time Codes                      : 00:00:01.18
======== M14-1233.MLV-C_000042.dng
Time Codes                      : 00:00:01.19
======== M14-1233.MLV-C_000043.dng
Time Codes                      : 00:00:01.20
======== M14-1233.MLV-C_000044.dng
Time Codes                      : 00:00:01.21
======== M14-1233.MLV-C_000045.dng
Time Codes                      : 00:00:01.22
======== M14-1233.MLV-C_000046.dng
Time Codes                      : 00:00:01.23
======== M14-1233.MLV-C_000047.dng
Time Codes                      : 00:00:01.00
======== M14-1233.MLV-C_000048.dng
Time Codes                      : 00:00:02.02
======== M14-1233.MLV-C_000049.dng
Time Codes                      : 00:00:02.03
======== M14-1233.MLV-C_000050.dng
Time Codes                      : 00:00:02.04
======== M14-1233.MLV-C_000051.dng
Time Codes                      : 00:00:02.05
======== M14-1233.MLV-C_000052.dng
Time Codes                      : 00:00:02.06
======== M14-1233.MLV-C_000053.dng
Time Codes                      : 00:00:02.07
======== M14-1233.MLV-C_000054.dng
Time Codes                      : 00:00:02.08
======== M14-1233.MLV-C_000055.dng
Time Codes                      : 00:00:02.09
======== M14-1233.MLV-C_000056.dng
Time Codes                      : 00:00:02.10
======== M14-1233.MLV-C_000057.dng
Time Codes                      : 00:00:02.11
======== M14-1233.MLV-C_000058.dng
Time Codes                      : 00:00:02.12
======== M14-1233.MLV-C_000059.dng
Time Codes                      : 00:00:02.13
======== M14-1233.MLV-C_000060.dng
Time Codes                      : 00:00:02.14
======== M14-1233.MLV-C_000061.dng
Time Codes                      : 00:00:02.15
======== M14-1233.MLV-C_000062.dng
Time Codes                      : 00:00:02.16
======== M14-1233.MLV-C_000063.dng
Time Codes                      : 00:00:02.17
======== M14-1233.MLV-C_000064.dng
Time Codes                      : 00:00:02.18
======== M14-1233.MLV-C_000065.dng
Time Codes                      : 00:00:02.19
======== M14-1233.MLV-C_000066.dng
Time Codes                      : 00:00:02.20
======== M14-1233.MLV-C_000067.dng
Time Codes                      : 00:00:02.21
======== M14-1233.MLV-C_000068.dng
Time Codes                      : 00:00:02.22
======== M14-1233.MLV-C_000069.dng
Time Codes                      : 00:00:02.23
======== M14-1233.MLV-C_000070.dng
Time Codes                      : 00:00:02.00
======== M14-1233.MLV-C_000071.dng
Time Codes                      : 00:00:02.01
======== M14-1233.MLV-C_000072.dng
Time Codes                      : 00:00:03.03
======== M14-1233.MLV-C_000073.dng
Time Codes                      : 00:00:03.04
======== M14-1233.MLV-C_000074.dng
Time Codes                      : 00:00:03.05
======== M14-1233.MLV-C_000075.dng
Time Codes                      : 00:00:03.06
======== M14-1233.MLV-C_000076.dng
Time Codes                      : 00:00:03.07
======== M14-1233.MLV-C_000077.dng
Time Codes                      : 00:00:03.08
======== M14-1233.MLV-C_000078.dng
Time Codes                      : 00:00:03.09
======== M14-1233.MLV-C_000079.dng
Time Codes                      : 00:00:03.10
======== M14-1233.MLV-C_000080.dng
Time Codes                      : 00:00:03.11
======== M14-1233.MLV-C_000081.dng
Time Codes                      : 00:00:03.12
======== M14-1233.MLV-C_000082.dng
Time Codes                      : 00:00:03.13
======== M14-1233.MLV-C_000083.dng
Time Codes                      : 00:00:03.14
======== M14-1233.MLV-C_000084.dng
Time Codes                      : 00:00:03.15
======== M14-1233.MLV-C_000085.dng
Time Codes                      : 00:00:03.16
======== M14-1233.MLV-C_000086.dng
Time Codes                      : 00:00:03.17
======== M14-1233.MLV-C_000087.dng
Time Codes                      : 00:00:03.18
======== M14-1233.MLV-C_000088.dng
Time Codes                      : 00:00:03.19
======== M14-1233.MLV-C_000089.dng
Time Codes                      : 00:00:03.20
======== M14-1233.MLV-C_000090.dng
Time Codes                      : 00:00:03.21
======== M14-1233.MLV-C_000091.dng
Time Codes                      : 00:00:03.22
======== M14-1233.MLV-C_000092.dng
Time Codes                      : 00:00:03.23
======== M14-1233.MLV-C_000093.dng
Time Codes                      : 00:00:03.00
======== M14-1233.MLV-C_000094.dng
Time Codes                      : 00:00:03.01
======== M14-1233.MLV-C_000095.dng
Time Codes                      : 00:00:03.02
======== M14-1233.MLV-C_000096.dng
Time Codes                      : 00:00:04.04
======== M14-1233.MLV-C_000097.dng
Time Codes                      : 00:00:04.05
======== M14-1233.MLV-C_000098.dng
Time Codes                      : 00:00:04.06
======== M14-1233.MLV-C_000099.dng
Time Codes                      : 00:00:04.07
======== M14-1233.MLV-C_000100.dng
Time Codes                      : 00:00:04.08
======== M14-1233.MLV-C_000101.dng
Time Codes                      : 00:00:04.09
======== M14-1233.MLV-C_000102.dng
Time Codes                      : 00:00:04.10
======== M14-1233.MLV-C_000103.dng
Time Codes                      : 00:00:04.11
======== M14-1233.MLV-C_000104.dng
Time Codes                      : 00:00:04.12
======== M14-1233.MLV-C_000105.dng
Time Codes                      : 00:00:04.13
======== M14-1233.MLV-C_000106.dng
Time Codes                      : 00:00:04.14
======== M14-1233.MLV-C_000107.dng
Time Codes                      : 00:00:04.15
======== M14-1233.MLV-C_000108.dng
Time Codes                      : 00:00:04.16
======== M14-1233.MLV-C_000109.dng
Time Codes                      : 00:00:04.17
======== M14-1233.MLV-C_000110.dng
Time Codes                      : 00:00:04.18
======== M14-1233.MLV-C_000111.dng
Time Codes                      : 00:00:04.19
======== M14-1233.MLV-C_000112.dng
Time Codes                      : 00:00:04.20
======== M14-1233.MLV-C_000113.dng
Time Codes                      : 00:00:04.21
======== M14-1233.MLV-C_000114.dng
Time Codes                      : 00:00:04.22
======== M14-1233.MLV-C_000115.dng
Time Codes                      : 00:00:04.23
======== M14-1233.MLV-C_000116.dng
Time Codes                      : 00:00:04.00
======== M14-1233.MLV-C_000117.dng
Time Codes                      : 00:00:04.01
======== M14-1233.MLV-C_000118.dng
Time Codes                      : 00:00:04.02
======== M14-1233.MLV-C_000119.dng
Time Codes                      : 00:00:04.03
======== M14-1233.MLV-C_000120.dng
Time Codes                      : 00:00:05.05
======== M14-1233.MLV-C_000121.dng
Time Codes                      : 00:00:05.06
======== M14-1233.MLV-C_000122.dng
Time Codes                      : 00:00:05.07
======== M14-1233.MLV-C_000123.dng
Time Codes                      : 00:00:05.08
======== M14-1233.MLV-C_000124.dng
Time Codes                      : 00:00:05.09
======== M14-1233.MLV-C_000125.dng
Time Codes                      : 00:00:05.10
======== M14-1233.MLV-C_000126.dng
Time Codes                      : 00:00:05.11
======== M14-1233.MLV-C_000127.dng
Time Codes                      : 00:00:05.12
======== M14-1233.MLV-C_000128.dng
Time Codes                      : 00:00:05.13
======== M14-1233.MLV-C_000129.dng
Time Codes                      : 00:00:05.14
======== M14-1233.MLV-C_000130.dng
Time Codes                      : 00:00:05.15
======== M14-1233.MLV-C_000131.dng
Time Codes                      : 00:00:05.16
======== M14-1233.MLV-C_000132.dng
Time Codes                      : 00:00:05.17
======== M14-1233.MLV-C_000133.dng
Time Codes                      : 00:00:05.18
======== M14-1233.MLV-C_000134.dng
Time Codes                      : 00:00:05.19
======== M14-1233.MLV-C_000135.dng
Time Codes                      : 00:00:05.20
======== M14-1233.MLV-C_000136.dng
Time Codes                      : 00:00:05.21
======== M14-1233.MLV-C_000137.dng
Time Codes                      : 00:00:05.22
======== M14-1233.MLV-C_000138.dng
Time Codes                      : 00:00:05.23
======== M14-1233.MLV-C_000139.dng
Time Codes                      : 00:00:05.00
======== M14-1233.MLV-C_000140.dng
Time Codes                      : 00:00:05.01
======== M14-1233.MLV-C_000141.dng
Time Codes                      : 00:00:05.02
======== M14-1233.MLV-C_000142.dng
Time Codes                      : 00:00:05.03
======== M14-1233.MLV-C_000143.dng
Time Codes                      : 00:00:05.04
======== M14-1233.MLV-C_000144.dng
Time Codes                      : 00:00:06.06
======== M14-1233.MLV-C_000145.dng
Time Codes                      : 00:00:06.07
The seconds are always correct, but the frame numbers are off.
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: DeafEyeJedi on March 17, 2016, 12:28:31 AM
@dfort -- I wonder if your Genius Brother can help solve this formula for the TimeCode bug?
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2016, 01:55:56 AM
So this gets close:
Code: [Select]
    double time = frame / framerate;
    int hours = (int)floor(time / 3600);
    int minutes = ((int)floor(time / 60)) % 60;
    int seconds = ((int)floor(time)) % 60;
    int frames = (int)floor((time - floor(time)) * framerate);
   
    buffer[*data_offset] = to_tc_byte(frames) & 0x3F;
    if(drop_frame) buffer[*data_offset] = buffer[*data_offset] | (1 << 7); //set the drop frame bit
    buffer[*data_offset + 1] = to_tc_byte(seconds) & 0x7F;
    buffer[*data_offset + 2] = to_tc_byte(minutes) & 0x7F;
    buffer[*data_offset + 3] = to_tc_byte(hours) & 0x3F;

But it doesn't "drop" frames at the standardized amount for NTSC (30/1.001 fps) which is: two frames every minute except each tenth minute. It just rounds the exact time to the nearest frame.

I don't know if the same convention applies for 23.976.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on March 17, 2016, 02:55:50 AM
I thought it was because of these lines:

Code: [Select]
    frame = frame - (seconds * (int)framerate);
    int frames = frame % MAX(1,(int)round(framerate));

The first line isn't rounding the framerate so it is changing 23.976 to 23. I tried changing it but ran into compiling problems that I didn't have before.

As for NTSC drop frame timecode, it only applies to 29.97 and 59.94 frame rates. As far as I know there is no drop frame timecode for 23.976 video, at least I've never encountered it. Someone can do a fact check on that. In any case, the only times I had to deal with drop-frame timecode was on NTSC master tapes that had to be delivered for broadcast. All the editing and VFX workflows that I've encountered used non-drop timecode. In addition, the math is so much easier with non-drop frame timecode when converting from frame numbers to timecode and back.

In other words, if you strip out all the drop-frame code nobody should miss it.

Here's that compiling issue I mentioned. The build script finishes and I get a binary but it won't create a mounted volume or webpage.

Code: [Select]
./build_installer.sh
rm -f mlvfs dng.o index.o wav.o stripes.o cs.o amaze_demosaic_RT.o hdr.o histogram.o mongoose/mongoose.o webgui.o resource_manager.o lj92.o gif.o patternnoise.o LZMA/7zAlloc.o LZMA/7zBuf.o LZMA/7zBuf2.o LZMA/7zCrc.o LZMA/7zCrcOpt.o LZMA/7zDec.o LZMA/7zFile.o LZMA/7zIn.o LZMA/7zStream.o LZMA/Alloc.o LZMA/Bcj2.o LZMA/Bra.o LZMA/Bra86.o LZMA/BraIA64.o LZMA/CpuArch.o LZMA/Delta.o LZMA/LzFind.o LZMA/Lzma2Dec.o LZMA/Lzma2Enc.o LZMA/Lzma86Dec.o LZMA/Lzma86Enc.o LZMA/LzmaDec.o LZMA/LzmaEnc.o LZMA/LzmaLib.o LZMA/Ppmd7.o LZMA/Ppmd7Dec.o LZMA/Ppmd7Enc.o LZMA/Sha256.o LZMA/Xz.o LZMA/XzCrc64.o
gcc -c -Wall -D__DARWIN_64_BIT_INO_T=1 -D__FreeBSD__=10 -D_FILE_OFFSET_BITS=64 -I/usr/local/include/osxfuse -std=gnu99 -DVERSION="\"30aea14\"" -DBUILD_DATE="\"2016-03-17 01:18:13\"" -DFUSE_USE_VERSION=26 dng.c -o dng.o
In file included from dng.h:27:0,
                 from dng.c:28:
mlvfs.h:107:22: error: unknown type name '__always_inline'
 #define FORCE_INLINE __always_inline
                      ^
dng.c:793:8: note: in expansion of macro 'FORCE_INLINE'
 static FORCE_INLINE size_t dng_get_image_data_inline(struct frame_headers * frame_headers, uint16_t * packed_bits, uint8_t * output_buffer, off_t offset, size_t max_size, int32_t bpp)
        ^
dng.c:793:28: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'dng_get_image_data_inline'
 static FORCE_INLINE size_t dng_get_image_data_inline(struct frame_headers * frame_headers, uint16_t * packed_bits, uint8_t * output_buffer, off_t offset, size_t max_size, int32_t bpp)
                            ^
dng.c: In function 'dng_get_image_data':
dng.c:841:20: warning: implicit declaration of function 'dng_get_image_data_inline' [-Wimplicit-function-declaration]
             return dng_get_image_data_inline(frame_headers, packed_bits, output_buffer, offset, max_size, 8);
                    ^
make: *** [dng.o] Error 1
cp: mlvfs/mlvfs: No such file or directory
...
...
created: /Users/rosiefort/mlvfs/pack.temp.dmg
chmod: /Volumes/MLVFS/.Trashes: Permission denied
"disk7" unmounted.
"disk7" ejected.
Preparing imaging engine…
Reading Protective Master Boot Record (MBR : 0)…
   (CRC32 $9A1DC65A: Protective Master Boot Record (MBR : 0))
Reading GPT Header (Primary GPT Header : 1)…
   (CRC32 $9598AAEB: GPT Header (Primary GPT Header : 1))
Reading GPT Partition Data (Primary GPT Table : 2)…
   (CRC32 $7D08CF8A: GPT Partition Data (Primary GPT Table : 2))
Reading  (Apple_Free : 3)…
   (CRC32 $00000000:  (Apple_Free : 3))
Reading disk image (Apple_HFS : 4)…
...
   (CRC32 $C1163E58: disk image (Apple_HFS : 4))
Reading  (Apple_Free : 5)…
...
   (CRC32 $00000000:  (Apple_Free : 5))
Reading GPT Partition Data (Backup GPT Table : 6)…
...
   (CRC32 $7D08CF8A: GPT Partition Data (Backup GPT Table : 6))
Reading GPT Header (Backup GPT Header : 7)…
...
   (CRC32 $B05F628B: GPT Header (Backup GPT Header : 7))
Adding resources…
...
Elapsed Time:  2.070s
File size: 1500593 bytes, Checksum: CRC32 $B1F95B1F
Sectors processed: 16384, 13964 compressed
Speed: 3.3Mbytes/sec
Savings: 82.1%
created: /Users/rosiefort/mlvfs/MLVFS.dmg
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2016, 03:33:54 AM
try using clang to compile instead of gcc, I'll try and fix the issue soon

how about this:
Code: [Select]
    //use drop frame if the framerate is close to 30 / 1.001
    int drop_frame = round((1.0 - framerate / ceil(framerate)) * 1000) == 1 && ceil(framerate) == 30;
    int hours, minutes, seconds, frames;
   
    if (drop_frame)
    {
        int d = frame / 17982;
        int m = frame % 17982;
        int f = frame + 18 * d + 2 * (MAX(0, m - 2) / 1798);
       
        hours = ((f / 30) / 60) / 60;
        minutes = ((f / 30) / 60) % 60;
        seconds = (f / 30) % 60;
        frames = f % 30;
    }
    else
    {
        //round the framerate to the next largest integral framerate
        //tc will not match real world time if framerate is not an integer
        double time = framerate == 0 ? 0 : frame / (framerate > 1 ? round(framerate) : framerate);
        hours = (int)floor(time / 3600);
        minutes = ((int)floor(time / 60)) % 60;
        seconds = ((int)floor(time)) % 60;
        frames = framerate > 1 ? (frame % ((int)round(framerate))) : 0;
    }
   
    buffer[*data_offset] = to_tc_byte(frames) & 0x3F;
    if(drop_frame) buffer[*data_offset] = buffer[*data_offset] | (1 << 7); //set the drop frame bit
    buffer[*data_offset + 1] = to_tc_byte(seconds) & 0x7F;
    buffer[*data_offset + 2] = to_tc_byte(minutes) & 0x7F;
    buffer[*data_offset + 3] = to_tc_byte(hours) & 0x3F;
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on March 17, 2016, 03:42:55 AM
@MARKUS

Quote
Used the following string to mount: "mlvfs_x64.exe C:\MLV --mlv_dir=F:\DCIM\100EOS5D\"

It looks like the format of your MLVFS launch command is incorrect. 'C:\MLV' is a path to a folder not a drive letter. Try the following:

launch a command prompt (in administrator mode) then:

cd /d [path_to_your_mlvfs_x64_rc.exe]

mlvfs_x64.exe X: --mlv-dir=[path_to_your_MLV_dir]

Where X: - the logical drive letter that will be assigned by the system for mounting the MLV file/s. Note that it doesn't have to be X: but it needs to be a drive letter that is not already in use on your computer.

Then type the url http://localhost:8000/ into your default browser that will bring up the GUI and let you configure the MLVFS correction/adjustment parameters.

Also, once you have done the above, if you launch Windows File Explorer you should see your chosen drive letter listed and the individual MLV files listed as sub-directories.

A small tip: set your preferred correction/adjustment setting in the GUI before accessing a mounted mlv folder in your logical drive or the settings won't be applied.

The MLVFS process will automatically build index files for each of the MLV files in your --mlv-dir location.  I see from your mount string that you are going directly to the CF card. under this scenario the index files will be created on your CF card. There might be a performance hit from this approach. Also, this is where your original footage is. If there is a hiccup with your PC that causes the card to corrupt then you have potentially lost everything.

It would be safer and I dare say faster if you copied your MLVs to a directory on your local hard drive.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on March 17, 2016, 04:36:29 AM
try using clang to compile instead of gcc, I'll try and fix the issue soon

That fixed the compiling issue--looks like I got to fix that hack in the compiling tutorial I posted a while back. Been procrastinating busy!

Anyway, at first I got this error:

Code: [Select]
dng.c:545:9: error: redefinition of 'drop_frame'
    int drop_frame = round((1.0 - framerate / ceil(framerate)) * 1000) == 1 && ceil(framerate) == 30;
        ^
dng.c:539:52: note: previous definition is here
static uint32_t add_timecode(double framerate, int drop_frame, uint64_t frame, uint8_t * buffer, uint32_t * data_offset)

and it wouldn't create a dng.o so I did this to your code:

Code: [Select]
-    int drop_frame = round((1.0 - framerate / ceil(framerate)) * 1000) == 1 && ceil(framerate) == 30;
+    drop_frame = round((1.0 - framerate / ceil(framerate)) * 1000) == 1 && ceil(framerate) == 30;

But as you can see there's still issues.

Code: [Select]
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000000.dng
Time Codes                      : 00:00:00.00
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000001.dng
Time Codes                      : 00:00:00.00
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000002.dng
Time Codes                      : 00:00:00.01
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000003.dng
Time Codes                      : 00:00:00.02
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000004.dng
Time Codes                      : 00:00:00.03
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000005.dng
Time Codes                      : 00:00:00.04
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000006.dng
Time Codes                      : 00:00:00.05
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000007.dng
Time Codes                      : 00:00:00.06
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000008.dng
Time Codes                      : 00:00:00.07
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000009.dng
Time Codes                      : 00:00:00.08
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000010.dng
Time Codes                      : 00:00:00.09
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000011.dng
Time Codes                      : 00:00:00.10
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000012.dng
Time Codes                      : 00:00:00.11
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000013.dng
Time Codes                      : 00:00:00.12
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000014.dng
Time Codes                      : 00:00:00.13
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000015.dng
Time Codes                      : 00:00:00.14
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000016.dng
Time Codes                      : 00:00:00.15
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000017.dng
Time Codes                      : 00:00:00.16
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000018.dng
Time Codes                      : 00:00:00.17
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000019.dng
Time Codes                      : 00:00:00.18
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000020.dng
Time Codes                      : 00:00:00.19
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000021.dng
Time Codes                      : 00:00:00.20
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000022.dng
Time Codes                      : 00:00:00.21
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000023.dng
Time Codes                      : 00:00:00.22
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000024.dng
Time Codes                      : 00:00:01.00
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000025.dng
Time Codes                      : 00:00:01.00
======== /Users/rosiefort/Desktop/mount/M27-1302.MLV/M27-1302_000026.dng
Time Codes                      : 00:00:01.01

I'm a rank beginner in C so I'm a bit lost with what's going on here.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dmilligan on March 17, 2016, 11:49:48 AM
Sorry, I've made other changes elsewhere. Easiest way to test is to just put that in a function in separate program, call it from a loop, and print out the time code.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on March 17, 2016, 01:32:04 PM
@Ottoga This is as far as I get. Everyting seems fine in browser but I cant access X:\. If I type sub-paths in --mlv_dir command I can't see them in browser.
Had the same problem trying it out on win 7...
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Flh4.googleusercontent.com%2F-BYHw0WuenIs%2FVuqh_kN6XLI%2FAAAAAAAACc0%2FvKkweGplmQEpZsWn8shI83NR2zGLii8rA%2Fw888-h897-no%2FMLVFS%252BTrouble.jpg&hash=bd88caff70d8ff84c8da30756c8091a5)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: mindfull on March 17, 2016, 05:43:11 PM
is this normal?

El capitan OSX 10.11

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fdl.dropboxusercontent.com%2Fu%2F65174281%2FScreen%252520Shot%2525202016-03-17%252520at%25252011.04.04%252520AM.png&hash=3a73c2ba27e3dc6635c696024c419507)
Title: Re: MLVFS - a FUSE based, &quot;on the fly&quot; MLV to CDNG converter
Post by: dmilligan on March 17, 2016, 08:59:45 PM
There's a folder selection dialog waiting for you to select a folder
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: dfort on March 17, 2016, 10:03:46 PM
Sorry, I've made other changes elsewhere. Easiest way to test is to just put that in a function in separate program, call it from a loop, and print out the time code.

Uh, yeah. That's a bit beyond my current C coding abilities. However, I do have lots of experience in post production (IMDB link (http://www.imdb.com/name/nm0287123/)) and am wondering if there's a reason why so much effort is going into drop frame timecode? Is someone asking for it? I see no point in automatically forcing drop-frame timecode on 29.97fps DNG files. The productions that I have worked on have specified not to use drop-frame timecode on anything with a few exceptions for NTSC videotapes that were delivered to a broadcasting facility.

In any case, something is broken with the timecode and I'd like to help so I guess I need to hit the C programing tutorials.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Ottoga on March 18, 2016, 01:40:30 AM
@Markus

Quote
Your mlvfs launch command:

c:\mlvfs_x64\mlvfs_x64.exe X:\ --mlv-dir=c:\

The '\' in your X:\ parameter is not required.

The parameter --mlv-dir=c:\  as per your launch command will result in MLVFS (or more likely the background dokan service) passing through, trying to map and interpret all files and sub-directories under the root of C: to drive X:

I'd suggest that under this scenario it is probably hitting a file that it can't deal with elegantly and as such causing the error. It also looks like it is trying to process the same MLV file twice which may be causing a conflict.

the --mlv-dir= parameter should contain the path to the folder where the MLV files that you want to process actually reside.

So, based on your screen shots I suggest that your mlvfs launch command should be:

c:\mlvfs_x64\mlvfs_x64.exe X: --mlv-dir=c:\HEj

MLVFS as a process appears to be working as it has successfully analysed M17-1143.mlv given that it is listed with all details in your localhost view. So hopefully with a corrected launch command you will be successful.

If you are still getting the error after that, then I'm at a loss and will need to hand your issue up to DMilligan. just provide fresh screen shots including some from File manager showing drive X: and any of its sub directories as well as one with a sub-directory contents showing.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 18, 2016, 05:51:49 AM
Testing MLVFS for Windows7 , everything when as per instructions for x86 .
Web GUI work without any problems , did have same crashes  with dual ISO from the
5d2 , when I try to copy the dng's from the DOKAN\Z: fold to a real folder it will crash after
about 500 frame out of 1500 , stop responding etc... then lost the mounted folder
tried to remount but had problems , rebooted Windows everything back to normal
but I haven't tried to copy the file . Will try a different mount drive .
besides that works just like the Mac version .
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on March 18, 2016, 06:11:25 AM
@reddeercity - could you please make a video tutorial how to get the newest method of MLVFS on PC working? I am still using the old version, it is a little bit hard to follow some new developments in this area though, i really enjoyed your previous tutorials. I bet it would be great for many people here if someone could sum it up to this point. Thanks.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: Markus on March 18, 2016, 02:21:53 PM
After trying anything and everything. I got an idea to try the x86 version of mlvfs instead and got it working right away. So it seems x64 version will not run on win 10 (at least not for me). Something goes wrong when it tries to mount the files to X:

I noticed that Pismo is much faster so I'll be sticking with that for a while. I mount my files from the card directly and then use slimraw to copy/compress them to harddrive. In that way I only get compressed footage on the harddrive. With Pismo and Slimraw this is just a little bit slower than copying the mlvs to the harddrive directly.

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 18, 2016, 10:59:01 PM
@reddeercity - could you please make a video tutorial how to get the newest method of MLVFS on PC working? I am still using the old version, it is a little bit hard to follow some new developments in this area though, i really enjoyed your previous tutorials. I bet it would be great for many people here if someone could sum it up to this point. Thanks.

Sure , I was thinking that myself the other day  ;)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 19, 2016, 05:12:18 PM
@kgv5 here you go as requested ,  Video Tutorial for MLVFS for Windows .
I mainly talk about windows7 but the same will hold true for Win8 & Win10

Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: kgv5 on March 19, 2016, 07:07:53 PM
@reddeercity
Great, thank you very much. Very informative and helpful tutorial, as usual :) I needed such an explanation, it was little bit hard to follow this thread, now everything is clear!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: axelcine on March 19, 2016, 08:13:38 PM
@reddeercity - Thank you very much. This was exactly what I needed.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: reddeercity on March 19, 2016, 11:44:08 PM
Thanks Guys !  Just doing my Part : I like doing ML Tutorial , it let me take a brake from my
paid project as I'm in the really boring part  "grading" indie movie .
It can be exciting sometimes ! when a image come to life though hours & hours of works  ::)
then you sit back , look at it and go " I  Love "F"ing Raw"  :D 

Only 520 clips to go  , that what keep telling myself 519 to go ,  518 to go ,  517 to go .......
Done 500 already   :)
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: USADavid on March 20, 2016, 02:26:15 AM
Is anyone else having empty .wav files with .MLV files > 2GB on Mac?
Plays fine with MlRawViewer!
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: ruber on March 21, 2016, 03:11:25 PM
Hi Guys,

thx to your good tips, I am working on a project with mlvfs and Dokan for the first time

I am in windows 10 pro, working with Davinci Resolve right now.

Clips are 1080p 25fps (NO dual iso)
My camera is the 5dmk3

I have succesfully imported all the clips (about 50 files) into davinci but the problem now is that playback is really slow, it stutters and it's not possible to work this way

Surprinsingly, when I deliver (render) one of these clips, rendering speed is quite good (40fps minimum with DNxHD)

EDIT: now playback in timeline is smooth and that's the most important thing. The problem is when I play the clips in the "media" section.

Would you have some tips? :)

Thank you. Cheers.
Title: Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
Post by: markodarko on March 22, 2016, 04:41:08 PM
Firstly, thank you @dmilligan for developing this solution for the community. It really is most excellent. I wonder if it would be possible for you to answer these questions for me as my search on this forum didn't bring up any:

1. What is the difference between "On" and "Aggressive" for bad pixel correction in terms of use and speed differences (if any)?
2. What are the differences in speed between opening the cDNGs using MLVFS or by first copying out the cDNG files to a folder and opening them there?

I ask because rendering 1080p clips from my 5D III in DaVinci Resolve 12 Lite is painfully slow for me. I have a MBP i7 8Gb RAM / 512Mb Nvidia with 512Gb SSD system drive and 2TB SSD media drive installed and rendering a 2 minute clip to ProRes 4444 with "Aggressive" bad pixel correction (as "on" didn't get rid of them) as well as colour grading in Resolve took 6 hours. I'm currently rendering another timeline with various clips and a total duration of 4 minutes and the estimated render time is SIXTEEN hours - it's 11 hours in right now so I'm not going to stop it to do some tests at this point.

I'd just like to get to the bottom of where the bottleneck is (aside from my mediocre Mac system compared to today's standards!) as I've read that others are getting 40fps render times. Is it the "Aggressive" mode or is it opening the cDNG files directly instead of first copying them?

Thanks for your help,

Mark.

EDIT: I just stopped the render as it was ridiculous so I'm going to do some tests. Incidentally, MLVFS i