Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 960177 times)

dmilligan · « **Reply #600 on:** April 22, 2015, 11:46:44 PM »

If somebody tells me what the coefficients are, or sends me a DNG from a 70D with the matrices in it, I'll do it.

DeafEyeJedi · « **Reply #601 on:** April 23, 2015, 02:01:10 AM »

Also got some 70D footage to test out with MLVFS -- first I need to know if you already have a copy of DNG from 70D?

If not, I can send you one... Dual-ISO or non Dual-ISO or Cdng?

Just let me know...

Danne · « **Reply #602 on:** April 23, 2015, 07:48:52 AM »

Here,s a dng
https://drive.google.com/file/d/0B4tCJMlOYfirbndxRkt0cy0tX0U/view?usp=sharing

Got it from here. They share test cr2 files for most if not all canon cameras.
http://www.photographyblog.com/previews/canon_eos_70d_photos/

janoschsimon · « **Reply #603 on:** April 23, 2015, 08:06:55 AM »

ah you need a photo dng or one from the video cdng? :-)

cheers janosch

silenttapes · « **Reply #604 on:** April 23, 2015, 08:13:26 PM »

I have started to experience a problem with MLVFS.

MLVFS has worked great for a few months. Now every so often when I go to load a large folder (500GB) of MLV for us in Davinci it just won't load. The MLFVS folder keeps ejecting its self, or Davinci doesn't see that the files are in davinci mode and instead see's empty folders. I end up restarting my computer about 10 times, reloading the folder over and over and eventually the problem is solved. It's then permanently solved and I can load and unload the footage repeatedly no problem. It seems to happen when have round tripped back to Davinci having not loaded the MLFVS for a while.

Does anyone else have this problem? Does anyone know how to overcome it in a timely manner?

Cheers,

dmilligan · « **Reply #605 on:** April 23, 2015, 08:40:31 PM »

are there any MLVFS logs in ~/Library/Logs/DiagnosticReports?

chmee · « **Reply #606 on:** April 24, 2015, 03:17:21 PM »

@dmilligan here you go. 70D is in. (oops, i'm wondering why they re as 6D, have to doublecheck)

regards

DeafEyeJedi · « **Reply #607 on:** April 25, 2015, 10:56:41 PM »

@dmilligan:

Any progress towards an update for MLVFS to be able to read 70D files yet?

I have a bunch of small few seconds test clips (MLV/RAW) in ISO 800 and ISO 1600 (where you get to see a lot of hot pixels I suppose if that would help you, just let me know!

Thanks.

Danne · « **Reply #608 on:** April 28, 2015, 04:53:23 PM »

Quote

@dmilligan here you go. 70D is in. (oops, i'm wondering why they re as 6D, have to doublecheck)

regards

Viewing exif on two different 70D raw-dng files seems 6D and 70D share the exact same matrix specification so the numbers seems right.

DeafEyeJedi · « **Reply #609 on:** April 28, 2015, 05:56:01 PM »

Is that to be expected especially coming from Canon?

Danne · « **Reply #610 on:** April 28, 2015, 08:30:46 PM »

On further notice seems relying on dcraw color matrix 1 is recommended for now. The matrix reported as from 6D to th 70D are wrong and will give greenish pics. Don,t think adobe software is updated with this cam yet.

*updated greenish prores files. Not the dng files. This needs more investigation.

*update. Seems I could be wrong on the prores transcoding. I put the wrong numbers in. The chmee matrices might be right.

*update- Now also ProRes transcoding in cr2hdr-r seems to work. Conclusion. Chmee 70D/6D matrices seems right. Only did brief tests.

DeafEyeJedi · « **Reply #611 on:** April 29, 2015, 09:50:48 AM »

I can confirm that regarding 70D/6D matrices seems similar since the ProRes files came out well from @Danne's latest cr2hdr-r for both MLV & RAW.

Looking forward to using these files for MLVFS as well...

dmilligan · « **Reply #612 on:** May 03, 2015, 04:57:43 AM »

I have updated MLVFS with the 70D matricies (they are the same as 6D)

DeafEyeJedi · « **Reply #613 on:** May 03, 2015, 06:46:19 AM »

Sweeeeeet can't wait to test it out -- Thanks, David!

janoschsimon · « **Reply #614 on:** May 03, 2015, 09:56:17 AM »

thx david will give it a try later :-)

masc · « **Reply #615 on:** May 05, 2015, 06:34:18 PM »

Quote from: silenttapes on April 23, 2015, 08:13:26 PM

I have started to experience a problem with MLVFS.

MLVFS has worked great for a few months. Now every so often when I go to load a large folder (500GB) of MLV for us in Davinci it just won't load. The MLFVS folder keeps ejecting its self, or Davinci doesn't see that the files are in davinci mode and instead see's empty folders. I end up restarting my computer about 10 times, reloading the folder over and over and eventually the problem is solved. It's then permanently solved and I can load and unload the footage repeatedly no problem. It seems to happen when have round tripped back to Davinci having not loaded the MLFVS for a while.

Does anyone else have this problem? Does anyone know how to overcome it in a timely manner?

I think I have the same problem. For smaller projects around 50GB it works great.

Now I have a bigger project with around 400GB (around 350 mlv files). Mostly the folder (mlvfs drive) mounts first, but in the browser I can't see a connection to the page which should pop up. If I click on the folder, it will be away up to the next system restart - no way to eject it or to access.

If I take only a few files of the project, around 150GB (100 mlv files) all seems to work fine. With only some more files the problem begins: the page will first be shown in the browser and I can access the mounted folder, after some minutes the page is away and the folder too.

/Library/Logs/DiagnosticReports is empty.

Thanks for your help in advance!

dmilligan · « **Reply #616 on:** May 05, 2015, 07:08:18 PM »

It's probably the index files.

MLVFS creates indexes of all your MLV files (if they don't already exist). These files are named the same as your MLV file, but end in .IDX. This indexing has to happen before MLVFS can give any information about a particular MLV file (this is because metadata and/or frames can appear anywhere in an MLV file, and the MLV format does not require that a frame count be in the first header).

If you put 500GB of files all in the same directory and then mount that directory with MLVFS, when the webgui comes up it wants to display information about all those files (same thing if you browse to the folder). So it tries to index all of them. For 500GB this will take a while. My guess is your issue has something to do with this (maybe you just didn't wait long enough, or there was some kind of timeout that cause MLVFS to crash).

Watch the folder where your original MLV files are located when you launch MLVFS. You should see these files starting to get created (if they are not already existing). Also keep an eye on disk, CPU, and memory usage of MLVFS. Then let me know what is going on.

The issue also could be related to corrupt index files. I would suggest you delete all the .IDX files before trying this. Then watch what happens as MLVFS creates them back.

One solution is to split up your files into separate subdirectories, that way MLVFS doesn't try to index all of them immediately all at once (it will only try to index files in a particular sub-directory if you browse to it).

Once all the index files are created, you won't have to wait anymore. My guess is this is why @silenttapes eventually got it to work. Each time he tried, MLVFS created a couple more index files, and eventually all of them got created.

masc · « **Reply #617 on:** May 05, 2015, 08:53:33 PM »

Thank you for your fast answer!
What I did now: I deleted all IDX files. Then I mounted again. As you said it builds the IDX files again. And you are right - I was to fast - that takes for this amount of data on my hardware around 15min (the drive is slow, I know...)! If I click in this time on any option on the webgui, mlvfs stops immediatelly creating IDX files. So I closed my browser when it pops up to not disturb mlvfs. RAM und CPU consumption is very low. Only <20MB RAM and <25% CPU. After all IDX are created I tried to start the browser watching the webgui... the gui is not there (server not found). Clicking on the mounted drive makes it away... mlvfs is listed no longer in the OSX activity window then.
After a system restart I tried mounting again. I waited until the CPU usage of mlvfs is listed with 0%. Opening the browser and going onto the webgui brings mlvfs to 100% CPU usage (on one core) and around 110MB RAM. There seems to be no access on the hard drive where the MLV files are located, the disk LED is not blinking. I can access the mounted folder and I see the DNGs, but I can't access the webgui. I am waiting now another 20min and nothing changes... the browser says "waiting for localhost...".

masc · « **Reply #618 on:** May 05, 2015, 11:55:57 PM »

Now I tried your second tip - subfolders. This seems to work. Thanks for your help! With that little workaround I can live. :-)

DeafEyeJedi · « **Reply #619 on:** May 06, 2015, 12:33:42 AM »

Quote from: dmilligan on May 03, 2015, 04:57:43 AM

I have updated MLVFS with the 70D matricies (they are the same as 6D)

Thanks again for the quick update for 70D color materices.

However, on my MBP it opens up web gui just fine and shows info.

But it the mount folder itself won't show up (even after spitting out the .IDX files)

This was done on my 2006 MBP... any ideas?

Danne · « **Reply #620 on:** May 13, 2015, 10:21:03 AM »

Hi dmilligan! Tried out your mlv_dump for archiving purposes. Very nice indeed. However. When extracting dng files both compressed and non compressed files it doesn,t add white and black level info? Could this be fixed? What I really like is that you rewrote the dng tiff tags completely for mlv to dng files.

Here is a file developed straight from an mlv file with your mlv_dump from first post.

https://drive.google.com/file/d/0B4tCJMlOYfirbXMxaVRCV0RiT1k/view?usp=sharing

*tried adding white/black level with exiftool. Doesn,t work. Is exiftool not working/writing to cdng files?

DeafEyeJedi · « **Reply #621 on:** May 19, 2015, 09:32:09 AM »

*feedbacks*

Some few days ago, I decided to run your app with an old footage I shot from 3-4 months ago during a BTS shoot...

I enabled 20-bit Dual-ISO in web host config (although not all files were shot in this format) so is it problematic for me to have both normal and DI's files together in the mix? Do they need to be run seprately?

Also I noticed while rendering in AE CC 12.1, seems to be really slow (even for regular MLV's) is this normal?

It's literally almost like over an hour for each 13-15 second clips?

https://vimeo.com/128229218

https://vimeo.com/128229544

Thanks again, @dmilligan!

dmilligan · « **Reply #622 on:** May 19, 2015, 12:50:20 PM »

Quote from: DeafEyeJedi on May 19, 2015, 09:32:09 AM

Also I noticed while rendering in AE CC 12.1, seems to be really slow (even for regular MLV's) is this normal?

There is no metadata in the MLV that indicates whether or not the file is dual ISO (IMO, there should be). So even on the non-dual ISO files we have to actually try to do a dual-ISO conversion and fail to figure it out, b/c we don't know if they are dual-ISO or not and there is no other way to figure it out.

I suppose there are some things I could do to speed things up a bit for the non-dual-ISO case, perhaps keep track of if frames in an MLV file fail to convert and don't try again once one or two frames in a file fail.

Danne · « **Reply #623 on:** May 19, 2015, 03:37:47 PM »

Can you use an if condition? I use it when differing dual iso from non dual iso searching for the line "'Edge-directed interpolation..." when doing a test conversion. That way it will only try to (test)convert one file instead of testing all files for dual iso. Sending out the process of the first file to a log file with tee command.

Code: [Select]

find . -name ${BASE}_1_$(date +%Y-%m-%d)_0001_C0000_000000.dng -print0 | xargs -0 /usr/bin/cr2hdr20bit | tee /tmp/magic_l/cr2hdr20bit_MLV.log ;
if grep -q 'Edge-directed interpolation...' tee /tmp/magic_l/cr2hdr20bit_MLV.log
then
echo yes
else
something else
fi

*

Quote

I suppose there are some things I could do to speed things up a bit for the non-dual-ISO case, perhaps keep track of if frames in an MLV file fail to convert and don't try again once one or two frames in a file fail.

You,re already on it I see

ddelreal · « **Reply #624 on:** May 19, 2015, 06:12:32 PM »

Thanks so much for this, really like using it for my 6D files. Question though: with regular raw files using raw2dng (on Mac) ACR opens the converted files very flat looking and I can adjust accordingly or I can open in Resolve and set to BMD and do the same thing. Works pretty much the same in Resolve when using MLV converted to CDNG using MLVFS but if I use ACR, it looks like REC 709 already. Is that correct?

News:

Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter (Read 960177 times)