Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter  (Read 695624 times)

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1475 on: March 30, 2017, 08:01:57 PM »
You can't specify an arbitrary scaling in the metadata, it's a pixel aspect ratio. You can only specify a change in the ratio of width to height, not some absolute change in both width and height.

EDIT: there does actually appear to be "BestQualityScale" tag that might work for this purpose. Anyone is welcome to fire up exiftool and try and see if it actually works. I don't plan on implementing this specific request, but rather have been meaning to for a while provide a generic way to override/add any particular DNG metadata. Just haven't had the time (probably won't for a while, if ever).

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6936
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1476 on: March 30, 2017, 10:58:41 PM »
Quote
...provide a generic way to override/add any particular DNG metadata
Cool.

I tried some but don,t really know if it change anything. A bit short on time but I added and changed the tag with exiv2 and then checked metadata with exiftool. Couldn,t remember how to alter scale numbers with exiftool but exiv2 works.

So if anyone wants to experiment here,s a little howto. Install both exiv2 and exiftool with homebrew . First install homebrew.
In terminal do:
Code: [Select]
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" then do:
Code: [Select]
brew install exiv2
Code: [Select]
brew install exiftool
To simply add the "BestQualityScale" tag to a dng file do:
Code: [Select]
exiv2 -M"set Exif.Image.BestQualityScale Rational 1/1" INPUT.dngThen maybe try:
Code: [Select]
exiv2 -M"set Exif.Image.BestQualityScale Rational 5/3" INPUT.dngor maybe
Code: [Select]
exiv2 -M"set Exif.Image.BestQualityScale Rational 7/3" INPUT.dng
Read more on the tag here:
http://wwwimages.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf



COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1477 on: April 04, 2017, 12:03:36 PM »
Hello. MLVFS is a very convenient thing, but for some reason it does not load the correction profiles of phase AF points when opening files written by "new" versions of ML.

To files from 3.07.2016 the profiles are still loaded, but to files written with "new" versions of ML - no.

Camera 650D
OS Windows 8.1
Canon 650D, EOSM 2.02

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1478 on: April 04, 2017, 01:30:33 PM »
You're going to have to be more precise than just saying "new" and probably provide samples.

COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1479 on: April 04, 2017, 07:38:53 PM »
So, at the opening sequence MLV-file (shot 07.03.2016, the firmware was up-to-date at the time) mounted in MLVFS console appears "Loading focus pixel map '80000301_1872x1060.fpm' ..." and no AF points.

When opening files with firmware from 6.01.2017 (and later) to 10/14 bit, no information is displayed in the console, and the files open in Davinci with AF points.
Canon 650D, EOSM 2.02

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1480 on: April 04, 2017, 08:35:34 PM »
There was a July 9 build and a June 13 build. You can find them here:

https://builds.magiclantern.fm/650D-104-all-builds-changes.html

Run a test with these two builds and report back whether or not you can reproduce the issue. (Good luck--I doubt that's the problem.)

It would help if you could post a short MLV file that MLVFS can't remove the focus pixels. You said that it is happening with the later builds so it would be best if you use the newest nightly (unified) build if you want to record a new MLV file. You mentioned 10/14 bit so maybe you've been using an experimental build?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1481 on: April 04, 2017, 11:11:14 PM »
My results:

• Build on Jun13:
ok, no AF
• Build on Jul09:
ok, no AF
• Build on Mar30-2017:
bad, AF

By the link: in the build folder: jpeg file + dng file (from MLVFS) and original MLV
Canon 650D, EOSM 2.02

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1482 on: April 05, 2017, 06:47:29 PM »
It would really help if you can find the exact build when this first started happening. Start with the build right after July 9.

Is there another 650D user reading this that can reproduce the issue?


Sent from my iPhone using Tapatalk
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1483 on: April 05, 2017, 07:48:18 PM »
As I suspected, the problem is the raw buffer size changed (from 1808x1190 to 1808x1189). MLVFS detects the map to use based on the camera id and the raw buffer size. All you need to do is duplicate the file 80000301_1808x1190.fpm and rename it to 80000301_1808x1189.fpm.

With the new RAWC block and crop_rec stuff we can (and will have to) come up with a little better system for getting the correct map.

COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1484 on: April 05, 2017, 08:08:17 PM »
Thank you so much. And what about a crop mode/others resolutions?
Canon 650D, EOSM 2.02

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1485 on: April 05, 2017, 11:47:36 PM »
You tell me. I don't own a 650D.

FYI when I say resolution, I'm not talking about the resolution you seleceted in the menu, rather the full (max) resolution of the current video mode. Since there is not (yet) metadata about what the video mode and crop settings are, we have to guess the video mode based on the full raw buffer resolution (each video mode has a unique raw buffer resolution currently, though that is no longer true with crop_rec and the new "4K" stuff)

COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1486 on: April 05, 2017, 11:57:07 PM »
Based on personal experience I would say that the other profiles are used for crop mode.

But I do not know the buffer-size to rename files.
Canon 650D, EOSM 2.02

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1487 on: April 06, 2017, 12:03:49 AM »
mlv_dump -v

COMMANDES

  • New to the forum
  • *
  • Posts: 22
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1488 on: April 06, 2017, 02:04:42 AM »
Thank you, I've fixed everything I needed modes.

Based on this block:
Code: [Select]
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 209.197000 ms
    Res:  1920x640
    raw_info:
      api_version      0x00000001
      height           1107
      width            2592
      pitch            3240
      frame_size       0x004C9EA8
      bits_per_pixel   10
      black_level      127
      white_level      937
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1101
      active_area.x2   2592
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Canon 650D, EOSM 2.02

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1489 on: April 06, 2017, 02:04:55 AM »
As I suspected, the problem is the raw buffer size changed (from 1808x1190 to 1808x1189). MLVFS detects the map to use based on the camera id and the raw buffer size. All you need to do is duplicate the file 80000301_1808x1190.fpm and rename it to 80000301_1808x1189.fpm.

With the new RAWC block and crop_rec stuff we can (and will have to) come up with a little better system for getting the correct map.

Right, Danne ran into that too so he did some adjustments that allowed for a little slop in the vertical resolution. Fortunately is isn't shifting the coordinates of the focus pixels so it still works.

Code: [Select]
  mv1080)
    raw_buffer=1808x11**
  mv720)
    raw_buffer=1808x72*
  mv1080crop)
    raw_buffer=1872x10**
  zoom)
    raw_buffer=2592x110*

Not sure how that would apply in MLVFS but there's also the crop_rec module that requires a different FPM file that's the same size as the mv720 raw buffer. I experimented with combining the map files but it didn't work so we'll have to deal with this at some point. Isn't there new metadata for crop_rec that can be used?
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

festr

  • New to the forum
  • *
  • Posts: 31
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1490 on: April 24, 2017, 09:04:37 PM »
Hi Dmiligan, do you plan to integrate the new lossless raw compression?

https://www.magiclantern.fm/forum/index.php?topic=18443.msg183653#msg183653

ludzik3d

  • New to the forum
  • *
  • Posts: 7
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1491 on: May 02, 2017, 03:05:53 AM »
Hi
I want to try MLVFS but I get an error message "dokanfuse.dll is missing, please reinstall dokany.
I use Windows 10 64bit.
I tried everything what I found on the forum and in the manuals but it still doesn't work :(

TrEK

  • New to the forum
  • *
  • Posts: 20
  • Canon 5d mk2 , CF Sandisk Extreme PRO 64gb 160MB/s
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1492 on: May 02, 2017, 04:20:59 PM »


Hello.
I was make MLVSF





But when i try put one folder on the timeline Adobe Premiere cc 2015 i receive such result:





CDNG sequence duplicate such times as how available quanity of dng-pictures....

and i need put first dng-picture on timeline for avoid it:





in such way its all correct...

what i can do for ability put all folder from BIN to timeline in one time ?


dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1493 on: May 06, 2017, 01:00:12 AM »
Merged @bouncyball's lj92 stuff, and updated the Mac build.

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1494 on: May 06, 2017, 07:56:16 AM »
Only problem is, at 7.7MB per frame, I can't get real time 3K playback in Resolve with MLVFS (all options off) even on a relatively fast Mac (3.33 GHz 6 Core 48GB ram SSD/raid GTX-980. Tested on Yosemite Resolve 12.5 and El Capitan Resolve 14.

The same MLV file converted with MLV_dump (6.6MB per frame) plays back smoothly in Resolve.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 806
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1495 on: May 06, 2017, 07:04:39 PM »
The same MLV file converted with MLV_dump (6.6MB per frame) plays back smoothly in Resolve.
Have you tried playback, with thouse 7.7MB MLVFS DNGs copied to HDD? If playback is OK than definitely (actually I'm quite confident about it) decompression overhead is an issue. If Canon HW compressor would produce multi slice (not the same as multi component) RAW data, decompression could be multi threaded per slice and hence be faster.

Unfortunatelly I'm quite far from this kind of programming (MLVFS actually is multi threaded app in its core) and can't say anything specific about making this portion of code multi threaded.

P.S. What is the CPU load during playback? (is it hackintosh?)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12461
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1496 on: May 06, 2017, 10:01:54 PM »
If Canon HW compressor would produce multi slice (not the same as multi component) RAW data, decompression could be multi threaded per slice and hence be faster.

It actually does (2 slices).

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1497 on: May 07, 2017, 08:54:11 AM »
Have you tried playback, with thouse 7.7MB MLVFS DNGs copied to HDD? If playback is OK than definitely (actually I'm quite confident about it) decompression overhead is an issue.

P.S. What is the CPU load during playback? (is it hackintosh?)

Mac Pro 2010. CPU load with MLVFS running is around 27%. 14-19% with the copied file. The copied file plays back smoothly.



bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 806
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1498 on: May 07, 2017, 10:42:14 AM »
@squig

Can you uncompress that high resolution MLV (mlv_dump -o uncompressed_dst.mlv compressed_src.mlv) and try to play in MLVFS?

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #1499 on: May 07, 2017, 01:31:36 PM »
@squig

Can you uncompress that high resolution MLV (mlv_dump -o uncompressed_dst.mlv compressed_src.mlv) and try to play in MLVFS?

Sure, but you're gonna have to give me the exact command. The file name is M01-2316.MLV