Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 957791 times)

DeafEyeJedi · « **Reply #625 on:** May 20, 2015, 08:19:31 AM »

Just got home from work and I find that MLVFS somehow froze up after spending the last few days rendering the shit outta my Mac Mini (all good the usual stuff) and apparently it thinks it ran out of diskspace but I looked up and see that I have 36 GB of free space.

Also Dual-ISO footage still comes out a bit darker than it normally would with other converters for some reason (I think we discussed this before) but not sure if it has been resolved or do i just need to boost up the exposure by a few stops just for these certain files only while using MLVFS?

Anyway, no big deal since I was't too pleased with the results of my footage. I'll have to work on some more with my color grading skills (if any) lol -- guess everything happens for a reason. Good thing I didn't waste a whole bunch of ProRes files for nothing.

Looking forward to your next update in regards to MLV/Dual-ISO files "edge-directed interpolation" system... it'll definitely be worth having this feature to speed up the workflow (even if its just a little bit) but perhaps not enough for you to redo all the codings?

Thanks again, Dave!

DeafEyeJedi · « **Reply #626 on:** May 20, 2015, 06:27:37 PM »

*updates*

Decided to try MLV spanning files again from a 5D3. Here's what I've encountered... One of the MLV's were recorded longer than 49 seconds (as stated in MLVFS web host) and when viewing playback the same original MLV's in MLRV (which playbacks fine the entire clip whereas one of the clip actually goes over 4 minutes long) but not after exporting from MLVFS. Strange?

https://vimeo.com/128391858

Ah, so I'm not sure where to look into this so called spanning culprit? Do I need to rename the spanning files to make it recognizable? (Will have to try these spanning files again with another converter such as cr2hdr-r to see if problem still persist and if so then it may not be your app's issue) so I'll get back to you on that shortly.

Have you ran into similar situation or anyone else out there?

ddelreal · « **Reply #627 on:** May 24, 2015, 11:03:29 PM »

Quote

Thanks so much for this, really like using it for my 6D files. Question though: with regular raw files using raw2dng (on Mac) ACR opens the converted files very flat looking and I can adjust accordingly or I can open in Resolve and set to BMD and do the same thing. Works pretty much the same in Resolve when using MLV converted to CDNG using MLVFS but if I use ACR, it looks like REC 709 already. Is that correct?

Sorry to quote my own post but was just wondering about my question...

dmilligan · « **Reply #628 on:** May 25, 2015, 03:57:55 AM »

MLVFS outputs RAW sensor data in the form of DNGs. How various software handles and interprets that data is up to the particular software. There may be slight differences in the metadata output between raw2dng and MLVFS, causing various softwares to interpret them slightly differently by default. However, the RAW data is the same, so once you make adjustments, you should see no difference.

I can't really comment on what ACR specifically is doing because I don't use it for video (and I recommend against using it for video), but I highly doubt it's outputting REC 709 by default as that is a video standard and ACR is intended mainly for photography and print and was never really intended for video.

DeafEyeJedi · « **Reply #629 on:** June 06, 2015, 10:33:51 PM »

@dmilligan:

Any suggestions on how to prevent MLVFS from disappearing from desktop during rendering within AE? It has happened to me three times in the last 48 hours on this Mac Mini running Yosemite...

It's connected to an external drive via USB 3.0 -- tried this routine again with destination export on OS system instead & issue still persist. Is this normal when working with more than 50 files at once?

Thanks, David!

goran · « **Reply #630 on:** June 07, 2015, 01:59:34 PM »

Just a quick question since I couldn't find an answer, how do I remove the MLVFS from services should I wish to do so?

Danne · « **Reply #631 on:** June 07, 2015, 02:15:14 PM »

Username/Library/services

You might have to unhide your library folder via terminal.

goran · « **Reply #632 on:** June 07, 2015, 03:01:51 PM »

Thanks!

DanHaag · « **Reply #633 on:** June 08, 2015, 05:56:50 PM »

Noticed MLV audio playing as it should in DaVinci Resolve when using your awesome MLVFS. Got some old CDNGs converted with RawMagic, their video and audio are badly out of sync (audio starting way after the video). Is there some metadata MLVS is adding that I might apply to my old files as well using ExifTool to fix this? I think I saw somebody writing about this a while ago but can't find those postings anymore.

Frank7D · « **Reply #634 on:** June 08, 2015, 06:03:33 PM »

DanHaag,

If you have cDNGs then I assume you got a WAV file for the audio also. Can you just trim the WAV so it is correct? I don't think metadata in the cDNGs (if any) is going to fix your problem.

DanHaag · « **Reply #635 on:** June 08, 2015, 06:38:59 PM »

I've done that for parts of the footage in FCPX. Can't do it directly in Resolve since it would not consider this kind of edit when outputting single clips i.e. as proxy files for editing (tried that as well, audio just goes back where it originally was after export). Manually moving the audio track in my NLE messes up my workflow since what I usually do is take the proxies from Resolve, sync them with external audio recordings in Plural Eyes and send this timeline to FCPX for editing without having to worry about that part of the process anymore. There's a whole lot of footage involved in that project I've started working on before MLVFS was around (wasn't able to keep the original MLVs, crazy amount of TB...) - Long story short: I can work around the problem but it's a huge effort and if there was any way to fix the problem with correcting metadata I'd be more than happy to know.

It just looks like DaVinci reads incorrect information on which timecode to put beginning of each audio track. Maybe there's no solution for this, other than fixing it the hard way.

dmilligan · « **Reply #636 on:** June 08, 2015, 06:47:09 PM »

It's more than just trimming the WAV file. There is special time code metadata that MLVFS puts in both the DNG and the WAV file. I don't think it would really be possible to add this with exiftool to the DNGs and there's no way at all, that I know of, to add it to the WAV file. You would have to write a program to do it. For more info see dng.c and wav.c in the MLVFS source, or have a look at raw2cdng code which also adds this metadata.

DanHaag · « **Reply #637 on:** June 08, 2015, 06:56:36 PM »

Thanks for your clarification.

If there was a way to fix the files, it's totally out of my reach.

But now at least I can focus on working around the audio sync manually without worrying if there's an easier way for me to do it. After that project is done I'll stick with MLVFS for sure.

Danne · « **Reply #638 on:** June 08, 2015, 07:23:28 PM »

Try bwf meta edit. I use it to add the black magic cam info from dmilligan wav info in cr2hdr-r.(thanks dmilligan)
I trim audio with sox and add meta time code info with exiftool. You can pm me a file with out of sync audio. Might be able to help.

alexidoia · « **Reply #639 on:** June 09, 2015, 03:21:04 PM »

Hello,

I did not know where to post this, but I have discovered what I think is a bug from MLVFS and I wanted to report it.
I first used mlrawviewer to convert my mlvs to dngs but as I use the cinelog LUT and that they advise to use MLVFS because of its respect of metadata blablabla, I did another convertion from mlv to dng using mlvfs. And that point is important because I could have never found out about the bug if I had not kept the first export from mlrawviewer.

As I were editing, I found a shot of only one frame. I found that strange so I decided to compare that dng (produced by MLVFS) to the dng produced from mlrawviewer. I found out that the shot was actualy a minute long. I did another export from MLVFS just in case and it did produced the same one-shot DNG. A few hours laters similar problem with a shoot a few second long whereas I remembered having at least 2 minutes of ongoing action. I exported the DNG again from MLVFS and had the same shot with the same short length. I compared with the dng from mlrawviewer and found out that my clip was -as I thought it would be- 2minutes long.

My conclusion is that for some reason (that have nothing to do with the computer performance while exporting) MLVFS produced shorted DNGs. This looks clearly like a bug because if I repeat the export, the produce DNG is exactly the same, it is not a random stop.

I thought some people would know about it. This is very bad since, you really don't know if your shot is shorted or not.

Thanks.
Alex

Licaon_Kter · « **Reply #640 on:** June 09, 2015, 03:45:23 PM »

So you have only one file in the mlvfs output folder when you navigate there? No GIF? No WAV? Screenshot of the folder contents?

alexidoia · « **Reply #641 on:** June 09, 2015, 06:22:43 PM »

Quote

So you have only one file in the mlvfs output folder when you navigate there? No GIF? No WAV? Screenshot of the folder contents?

heu... that was not for me right ?

Licaon_Kter · « **Reply #642 on:** June 09, 2015, 07:24:13 PM »

Yes it is...

dmilligan · « **Reply #643 on:** June 09, 2015, 07:51:41 PM »

@alexidoia

Take a screenshot of the web interface for those shortened clips, upload an example MLV file somewhere that exhibits the problem or at least send me the output of mlv_dump -v

DeafEyeJedi · « **Reply #644 on:** June 09, 2015, 07:56:54 PM »

I've reported this somewhat similar issue but no one got back to me about it as of yet...

@alexidoia: was yours done with spanning files or no?

ayshih · « **Reply #645 on:** June 09, 2015, 11:14:23 PM »

For those using the Pismo File Mount version of MLVFS on Windows, I know that it hasn't been updated in eight(!) months, and it's far behind @dmilligan's Mac/Linux version. I took a very long break from ML or MLVFS development, and I'm starting to get back into it. I still have to figure out the best way to "catch up", not least of which is figuring out how to implement the equivalent to the webgui. Let me know if there are specific features that should be the highest priority (e.g., FRSP support).

Markus · « **Reply #646 on:** June 10, 2015, 07:52:32 AM »

ayshih: Super! I use it all the time despite lack of Web gui. I just mark drowes files in total commander select quick mount and it mounts all the files I selected at once.
I've been wondering if pismo can take advantage of existing idx files. It seems to take about the same time to mount with or without existing idx.

alexidoia · « **Reply #647 on:** June 12, 2015, 03:30:45 PM »

I'm sorry,
I did not get back to you, as I have a movie to edit and a baaad deadline.
I don't have time to put the files online nor to test anyfurther, but basicaly,

I have a directory of .mlv
I right click on that directory to choose service>MVLFS et choose an empty directory. For 265 .mlv I have 265 .MLV directories which contains
1 _PREVIEW.GIF file
X .dng files
1 .log file
1 .wav file

That is to answer part of your previous questions.
The problem is that for one .mlv that last about a minute in mlrawviewer, I have only 1 .dng produced
and for some others I have the wrong amount of .dng files.

I did move the .mlv files that had not been converted correctly in a separate directory. and I mounted the directory via the services again. To my surprise, the file that used to output only one .dng had about the right amount of .dng this time. I am saying about because when comparing with the output of mlrawviewer, some .dng where still missing.

So I don't know. It changed something to isolate them into a different directory but it didn't really solve the problem.
I am not running mvlfs through the terminal so I am not really sure how to ouput the verbose version of it.

I'll get back to that when I have more time but right now I can't be diverted too much from my editing.
Alex

DeafEyeJedi · « **Reply #648 on:** June 12, 2015, 08:53:51 PM »

Seriously... I can't take this anymore... I REALLY cannot fathom with this any longer in allowing myself to use MLVFS again with BIG projects (regardless of how nice and awesome it is) it just totally won't render/export from AE for some reason (At first it says its rendering and I check the destination folder that's on the same external drive where I can see the file being processed) and leave my MacMini on all overnight (with screensaver and disk sleep enabled 'NEVER' within setting in OS X -- because I feel these can corrupt the rendering if it goes to sleep overnight):

It's driving me INSANE, this supposedly only happens if I use an external drive plugged in via USB 3.0 (big projects are usually over 160-200GB worth of MLV's) so I can't obviously use my Mac Mini's disk space which is limited to about 120gb of free disk space.

However, if I run small projects with only 10-20 MLV's then I can run MLVFS within my system on Mac Mini with export destination set on my desktop for faster rendering (Mac Mini has built in SSD) which works GREAT! I mean really fast and gets the job done well!

BUT for some reason if I use my external drives (which aren't SSD) seems to be running slower (which is to be expected) but then once I go to bed and wake up the next morning I find myself encountering with these nightmares on several occasions as previously reported:

Also worth noting is the fact that it appears to me that I may need to purchase a SSD external drive with USB 3.0 in order to prevent this from happening again? What I mean is what can I do (if any) to prevent from having MLVFS mount files disappear in the middle of the night thus I believe is what causing these errors in AE during rendering?

I don't have this problem if I run everything solely on this Mac Mini (small projects) but if I want to work with big projects then I need to be able to use my external drives without hesitation of any kind. I thought I used to be able to do this with no problems in the past but as of lately is has been and I'm not sure why or what has changed?

The two things that I think could be the culprit for me is the fact that I have shot both in Non-Dual ISO MLV as well as in Normal MLV and there are all mixed up in order -- so is it not a good idea to have CS2x2 & Dual-ISO (Full 20-bit) both enabled on in webGUI?

I don't mind the slow process but am I allowed to do it this way?

Or do I need to separate them like batch all the Dual-ISO MLV's into one Folder and Regular MLV into another Folder?

Forcing myself to run TWO separate Projects to possibility prevent from this so called culprit occurring again that I've been encountering as of lately - hmm?

Or would it be ideal to try and run two external drives (one 3.0 and one Firewire 800) with one being used to provide BIG PROJECTS and one being used for the destination point for ProRes exports?

Sorry to bring this to you, David but I feel that I've tried all possibilities other than considering myself to purchase a SSD external drive w thunderbolt support. Thoughts on that?

Thanks for reading this.

dmilligan · « **Reply #649 on:** June 14, 2015, 02:52:24 AM »

Issues like this are going to be very difficult for me to duplicate and fix, I would suggest trying to narrow it down as much as you can and figure out how to reliably duplicate the issue. I just don't have time to try and replicate issues involving unpredictability and vast amounts data/processing time.

News:

Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 957791 times)