Author Topic: Updated MLVFS supporting Canon HW compressed MLVs  (Read 10954 times)

bouncyball

  • Senior
  • ****
  • Posts: 362
Updated MLVFS supporting Canon HW compressed MLVs
« on: April 25, 2017, 08:39:43 PM »
Hi guys,

Last two weeks I've been investigating lossles JPEG (92) specs, LJ92Lib from Andrew Baldwin and MLV files produced by both a1ex's lovely crop_rec_4k/mlv_lite and by mlv_dump from lj92 branch, also DNGs produced by MLRAWviewer.

It seems that Andrew Baldwin's implementation is somewhat weird. It encodes raw data as two images each with its own headers, one scan and one component in it. That means that encoded data has two SOI (0xFFD8, Start of image) and EOI (0xFFD9, End of image) markers and each image has its own SOF3 (0xFFC3, Start of frame header), DHT (0xFFC4, Huffman table) and SOS (0xFFDA, Start of scan header) with its own one image component with full width and half height. Also baldand chose the prediction method 6 in his encoder because of reasons explained here. If someone interested the original document is here.

Meanwile the raw data encoded (as configured by a1ex) by Canon HW compressor fully follows the specs and has one SOI/EOI, which accordingly have DHT, SOF3 and SOS in it. SOS consists of two components with full width and half height. Predictor used is number 1.

This is the reason why LJ92lib can not decode ML files out of the box despite it supports all 7 predictors while decoding (not only number 6). The component number should be correctly taken into account.

While studying all these I came across Syoyo Fujita's github account with his modified header file based on baldand's LJ92lib. ( which also was pointed out by @martinherring :) )

In short Syoyo correctly modded the main loop in parseScan function to support components and changed decoder struct and added 'components' field to it. Also he added multiple Huffmann table support to the lib which is really unnecessary at least for ML compressed data. So I edited lj92.c accordingly. Unfortunatelly encoder still needs some work. However for MLVFS encoder is unnecessary :)

Using all this info I pathched MLVFS and it works nicely with all its bells and whistles (on the fly raw processing). I want to thank Danne as usual :) for testing/support and MAC binary.

David Milligan merged changes to MLVFS repo.
Download latest MAC binary here and Windows x64 binary here.
The latest 'DokanSetup_redist.exe' installer is here.

I also modified mlv_dump-on-steroids but unfortunatelly it's not ready for prime time yet. Needs more time. Always time :P. Now supports all latest stuff plus some features that none of the mlv_dump versions have. Binaries uploaded.

regards
bb

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3045
  • 5D3 / M1 / 7D / 70D / SL1
Updated MLVFS supporting Canon HW compressed MLVs
« Reply #1 on: April 25, 2017, 08:46:55 PM »
Dude... am I dreaming? :D
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #2 on: April 25, 2017, 08:51:29 PM »
@DeafEyeJedi: Hope you gonna test it asap ;)

Danne

  • Hero Member
  • *****
  • Posts: 3478
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #3 on: April 25, 2017, 08:52:30 PM »
Bravo Bouncyball. Thanks for the insightful explanations. Lj92 code very much needed around all bleeding edge compressed raw coming from a1ex. Been running a bunch of test compressed raw files from Deafeyejedi on both MLVFS and mlv_dump flavours(ml-dng, steroid) code and from my testing it works really well. Did try to find some vertical stripes to fix but cannot say I found any stripes to check yet so that needs to be tested. Darkframe averaging was still not working but all the rest seems to be in there which is so great to have. Will post a version of cr2hdr.app soon. And also, so cool to see Baldand´s code coming alive again.

togg

  • Senior
  • ****
  • Posts: 279
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #4 on: April 26, 2017, 01:02:24 AM »
I've not followed the developments with compressed MLV but it really interests me a lot since 3TB drives piles up pretty fast. I'll take whatever comes next!


erikbaldwinson

  • Freshman
  • **
  • Posts: 70
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #6 on: April 26, 2017, 08:47:12 AM »
I replaced the mlvfs.exe on windows764bit and now get an error message "dokanfuse.dll is missing, please reinstall dokany".

What am I missing?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3045
  • 5D3 / M1 / 7D / 70D / SL1
Updated MLVFS supporting Canon HW compressed MLVs
« Reply #7 on: April 26, 2017, 09:23:07 AM »
Been sleepwalking ever since Dr. B has given us this opportunity to use MLVFS just like the good old days...

It's a breeze to be able to quickly view files virtually especially w UHD/3K/4K/Full Res lossless stuff. Happy times!  8)



Thanks a bunch for this and will definitely test out thoroughly with several samples that I've collected from other users complaining of vertical stripes not working and such, etc.. Fun Fun!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #8 on: April 26, 2017, 09:57:53 AM »
I replaced the mlvfs.exe on windows764bit and now get an error message "dokanfuse.dll is missing, please reinstall dokany".
What am I missing?
Well, what comes to my mind first is: do you use 32 bit dokany/mlvfs on 64bit Win7? I compiled native x64 binary, so dokan must be x64 too. Install this package: https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe and then copy from C:\Program Files\Dokan\Dokan Library-1.0.3 to mlvfs dir 2 files: dokan1.dll and dokanfuse1.dll.

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #9 on: April 26, 2017, 10:10:54 AM »
Thanks a bunch for this and will definitely test out thoroughly with several samples that I've collected from other users complaining of vertical stripes not working and such, etc.. Fun Fun!
Good to hear :) thanks.

Decompressing takes place before raw data processing. So there is no reason for stripes corection not to work. However this algo is not 100% perfect and there are some situations when it's not getting rid of all stripes regardless data compressed or not.

regards
bb

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #10 on: April 26, 2017, 10:20:41 AM »
@lorenzo353: thank you for posting links.

mageye

  • Senior
  • ****
  • Posts: 296
  • Atheist
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #11 on: April 26, 2017, 12:51:28 PM »
Oh my, oh my. Thanking you Mr Bouncy Balls ;D. I have been lost without MLVFS. In fact - thank you everybody who makes this possible! :D
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

festr

  • New to the forum
  • *
  • Posts: 30
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #12 on: April 26, 2017, 03:06:26 PM »
I'm getting 100% usage on CPU (all cores) when I try to copy files from mounted mlv which has compressed lossless raw. Is this known issue on macos version? (I even do not see the usage in "top" command - (system 100% cpu)

edit: this was probably because I selected bad pixel checkbox. Now without it it is blazing fast.

Another question - is there any mlv_dump version which can correct (and force it) vertical stripes?

Markus

  • Senior
  • ****
  • Posts: 313
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #13 on: April 26, 2017, 03:51:24 PM »
I just keep getting mlvfs stopped working when i try to test this new verison of mlvfs.
I installed the newer Dokany 1.0.3 and I'm using the mlvfs.exe from the link in this thread. (https://bitbucket.org/bouncyball/mlvfs/downloads/?tab=downloads)
I had standard mlvfs that worked previous to trying this build. And I uninstalled the older version of Dokany before trying out the new version.
Avast and windows defender went bananas when I first tried to execute it but since then I added exceptions for that library.
In the old version of mlvfs there were a bunch of other files besides the mlvfs.exe in the same folder so I also tried to add these to the new versions location but got the same stopped working error.

Any Idea what I'm doing wrong?
Any help appreciated.

Using Win 10 x64 (Havn't upgraded to creators update jet)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3052
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

martinhering

  • Member
  • ***
  • Posts: 100
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #15 on: April 26, 2017, 06:39:51 PM »
A idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.
5D Mark III, EOS M, 700D

Danne

  • Hero Member
  • *****
  • Posts: 3478
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #16 on: April 26, 2017, 06:55:09 PM »
Check out dmilligan's work on mlv compression.
https://bitbucket.org/dmilligan/magic-lantern/branch/ml_dng_lj92

How about losslessly compressed dng files like in MlRawViewer?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3045
  • 5D3 / M1 / 7D / 70D / SL1
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #17 on: April 26, 2017, 06:59:39 PM »
An idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.

+1

How about losslessly compressed dng files?

Ah, that's right. Especially with @cpc's slimRAW stuff. Great reminder @Danne!
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

justinbacle

  • New to the forum
  • *
  • Posts: 39
  • DP
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #18 on: April 26, 2017, 07:13:50 PM »
This update is very nice indeed :)
Could you update the Binary for MLVFS_x64 ? I'd like to try it, but only MLVFS_x64 works on my PC :s

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3052
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #19 on: April 26, 2017, 07:51:34 PM »
A idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.

not wanna totally hijack this thread.
compression is already implemented.
though not exactly the same method as canon uses, but close to.

but further mlv_dump discussion please in some other thread :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #20 on: April 26, 2017, 08:44:10 PM »
Could you update the Binary for MLVFS_x64 ? I'd like to try it, but only MLVFS_x64 works on my PC :s
Binary in download area is indeed native x64 exe not the 32bit version.

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #21 on: April 26, 2017, 08:47:27 PM »
@g3gg0: Thank you I saw those changes. It's just the version of the lj92.c you used has tons of unnecessary stuff from tiny_dng_loader author :)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3052
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #22 on: April 26, 2017, 10:38:34 PM »
i am happy to take over a cleaner one.
does your version support multi component compression?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

D_Odell

  • Member
  • ***
  • Posts: 180
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #23 on: April 26, 2017, 11:48:20 PM »
Hi!

Very happy to read this! I wonder thou I have previous dmg of MLVFS installed on OS X, do I need to uninstall/remove pref or just install latest?

Best,
David
5D3 (OLPF removed) :: 1.1.3 :: Canon FD L Serie

Danne

  • Hero Member
  • *****
  • Posts: 3478
Re: Updated MLVFS supporting Canon HW compressed MLVs
« Reply #24 on: April 26, 2017, 11:57:58 PM »
Just double click to overwrite the old one.