Updated MLVFS supporting Canon HW compressed MLVs

Started by bouncyball, April 25, 2017, 08:39:43 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

EuAcelasi


EuAcelasi

Quote from: hindra on May 05, 2017, 04:33:31 AM
I have it working on Windows 10, just got it up tonight. I had to install the x64 installer and it was good to go. Thank you everyone!!!

EDIT: The only bug I'm noticing is I cant close MLVFS unless I go into the task manager and end task. My raw files come out really cool/blue. Other than that everything looks great.
Installed Visual Studio... Then in Dokan installed the.msi file, all .exe files... executed the mlvfs.exe ... doesn`t work...

EuAcelasi

Quote from: bouncyball on April 25, 2017, 08:39:43 PM
Hi guys,

Last two weeks I've been investigating lossles JPEG (92) specs, LJ92Lib from Andrew Baldwin and MLV files produced by both a1ex's lovely crop_rec_4k/mlv_lite and by mlv_dump from lj92 branch, also DNGs produced by MLRAWviewer.

It seems that Andrew Baldwin's implementation is somewhat weird. It encodes raw data as two images each with its own headers, one scan and one component in it. That means that encoded data has two SOI (0xFFD8, Start of image) and EOI (0xFFD9, End of image) markers and each image has its own SOF3 (0xFFC3, Start of frame header), DHT (0xFFC4, Huffman table) and SOS (0xFFDA, Start of scan header) with its own one image component with full width and half height. Also baldand chose the prediction method 6 in his encoder because of reasons explained here. If someone interested the original document is here.

Meanwile the raw data encoded (as configured by a1ex) by Canon HW compressor fully follows the specs and has one SOI/EOI, which accordingly have DHT, SOF3 and SOS in it. SOS consists of two components with full width and half height. Predictor used is number 1.

This is the reason why LJ92lib can not decode ML files out of the box despite it supports all 7 predictors while decoding (not only number 6). The component number should be correctly taken into account.

While studying all these I came across Syoyo Fujita's github account with his modified header file based on baldand's LJ92lib. ( which also was pointed out by @martinherring :) )

In short Syoyo correctly modded the main loop in parseScan function to support components and changed decoder struct and added 'components' field to it. Also he added multiple Huffmann table support to the lib which is really unnecessary at least for ML compressed data. So I edited lj92.c accordingly. Unfortunatelly encoder still needs some work. However for MLVFS encoder is unnecessary :)

Using all this info I pathched MLVFS and it works nicely with all its bells and whistles (on the fly raw processing). I want to thank Danne as usual :) for testing/support and MAC binary.

David Milligan merged changes to MLVFS repo.
Download latest MAC binary here and Windows x64 binary here.
The latest 'DokanSetup_redist.exe' installer is here.

I also modified mlv_dump-on-steroids but unfortunatelly it's not ready for prime time yet. Needs more time. Always time :P. Now supports all latest stuff plus some features that none of the mlv_dump versions have. Binaries uploaded.

regards
bb
Is it a x64 binary in the link? It is a mlvfs.exe rather then a mlvfs_64.exe and it I can`t mount the drive with it... in the Command Prompt it gives an error...

COMMANDES

Hello, what about the removal of AF points (EOSM)?
I have in my folder the required * .fpm profiles, but mlvfs does not use them.

Canon 650D, EOSM 2.02, M50 1.1.0

IDA_ML

Bouncyball,

First of all, I would like to thank you for all your time and efforts that you dedicate to improving and making ML tools more efficient for all of us.  I appreciate that very much and am sure that many other people on this forum feel the same way.

I have been using the 32-bit version of MLVFS very successfully for years now and  wanted very much to upgrade to the x64 bit version to use it with losslessly compressed MLVs.  I have been trying this for several months now, unfortunately without success.  I have been following this thread as well as the original one:

https://www.magiclantern.fm/forum/index.php?topic=13152.0

very carefully.  I also repeated the procedure described in your post #47 at least 20 times, no luck.  As I described in another thread, (the one about the MLVApp), I am on Win7 x64 Ultimate, SP1 with all Windows updates installed and NO antivirus software.   Every time I start the batch file as usual, I get the message:

"MLVFS.exe has stopped working".

And this happens on at least 3 different computers that I tested.

When I reinstall the 0.8.0 version of Dokan and the very old 32 bit MLVFS.exe file from the OP in the above thread, everything works fine again but only with 10/12/14-bit uncompressed MLV files.

As you see, there are also other people experiencing exactly the same problem meaning that there is some kind of a compatibility issue between your x64 bit software and some Win x64 computers that noone of us has experienced with other software .  Noone knows what the problem is and believe us, we did everything stricktly following your instructions - no messing around with 32/64 bit versions or anything else.  Obviously, you are the only one who can find a solution, maybe with some help from the other developers.  This would make us more than happy and your continued efforts in this direction are very welcome.

I have a question for you and the other developers.  Is there somewhere a 32-bit version of MLVFS.exe that works with Dokan 0.8.0 and can process 8 to 14 bits losslessly compressed MLV files?  A download link should be greatly appreciated.

bouncyball

Well I have not tried MLVFS with dokany version later than 1.0.3 (2.0.0-BETA1 is the latest). Maybe this is the reason IDK. Gotta test it myself. However with v1.0.3 MLVFS never, I repeat, NEVER crashed on start for me.

Edit: I have to say I'm sitting under Linux nearly all the time and maybe some later windows updates also have some impact. Have to take a look...
Edit2: I don't have windows 7 x64 anywhere around to test with ;)

IDA_ML

Thanks a lot for your willing to help, Bouncyball.  I hope, a solution will eventually be found.

I would like to inform you that I tried all dokany versions starting with 1.0.3 and ending with 2.0.0. Beta-1.  Of course, I added the corresponding versions of the dokan.dll and dokanfuse.dll to the directory containing the mlvfs.exe package.  Exactly the same behavior on all of them - crash upon mlvfs.exe start.  I was wondering if the latest mlvfs.exe version could be compiled to 32 bit.  In that case it should work with the old 0.8.0 dokan version.  Is this too much work to do?  If not, this may be the working solution.  Don't you think?

bouncyball

Sure. I'm gonna compile 32bit version as soon as I get to my windows environment at work.
Actually I remember that all crashes and hangups for me occurred with version 0.8.

bouncyball

Meanwhile I tested latest x64 binary on my home PC. Here is Win10 installed. All works as expected.

With dokany up to 1.0.5 even dll files are not required in the mlvfs folder.
With dokany 2.0.0-BETA1 only dokanfuse2.dll have to be renamed to dokanfuse1.dll and copied to the mlvfs folder.

Also tried to install Dokany on some old Win7pro x64 virtual machine and it complained that some update is not present on this windows system and it could not be installed.

So... I've come up to the conclusion that this is most likely could be Windows 7 + Dokany issue (as I remember others experiencing this on Win7 as well)

I'll compile 32bit version tomorrow.

regards
bb

IDA_ML

Quote from: bouncyball on October 22, 2017, 10:31:16 AM
I'll compile 32bit version tomorrow.
bb

Wow, that sounds exciting, Bouncyball!  Thank you so much.  I can't wait to test!

IDA_ML

Quote from: bouncyball on October 22, 2017, 10:31:16 AM
So... I've come up to the conclusion that this is most likely could be Windows 7 + Dokany issue (as I remember others experiencing this on Win7 as well)

I doubt it, although I have no idea about software and computer programming.  The fact that MLVApp shows exactly the same behavior even without Dokany being installed at all, (MLVApp crashes upon an attempt to import a MLV file), indicates that Dokany has nothing to do with the issue.  I am very much inclined to think that these crashes have something to do with 64 vs. 32 bit versions of your software on Win 7x64.  The only version of MLVApp that worked on all of my Win7 x64 computers was v.3.0.  As far as I remember, this was a 32-bit version.  The very first MLVFS that works with Dokany 0.8.0 is also a 32 version.  Conclusion:  Win 7x64  likes 32-bit versions but does not like 64-bit versions of your software.

bouncyball

@IDA_ML

Sorry about late post. Here is the x86 (32bit) version of latest mlvfs.

Try it with dokany 1.0.5 x86 under Win 7 or if you like with your beloved v0.8 ;). Use both files from zip.


bouncyball

Nope. Mac version is at official (dmilligan's) MLVFS repository - here.

IDA_ML

Thank you so much, Bouncyball.  I will test as soon as I get back home. 

IDA_ML

Bouncyball,

You are my hero !!!  It works !!!  I can now use your mlvfZ.exe for losslessly compressed MLV files in the MLVFS system.  It even kills the focus pixels frm my 100D files using the old .fpm files from the MLVFS_x86 directory.

For those MLVFS users on Win7 x64 who wish to try it, here are a few instructions:

1) Uninstall the Dokany 0.8.0 library and restart your PC

2) Install the DokanSetup_redist-1.0.5.1000 package

3) Copy the two files from Bouncyball's ZIP file into your MLVFS_x86 directory

4) Rename the old mlvfs.exe to mlvfs_old.exe in case you wish to revert and rename mlvfz.exe to mlvfs.exe

5) Enjoy!

------------------------

Thank you so much, Bouncyball!  For me, there is no doubt now that Win7 x64 works well with the 32-bit versions of your software but crashes with the 64-bit ones.  If you could also compile a 32-bit version of the latest MLVApp, I would finally be able to test it. 

bouncyball

@IDA_ML

Nice to hear! :D

Quote from: IDA_ML on October 25, 2017, 08:16:39 AM
For me, there is no doubt now that Win7 x64 works well with the 32-bit versions of your software but crashes with the 64-bit ones.  If you could also compile a 32-bit version of the latest MLVApp, I would finally be able to test it. 
I would post 32bit mlvapp but it crashes when importing MLV. We did not figure out why yet.

p.s. mlvfZ, btw was typo ;)

IDA_ML

It works so well, Bouncyball, unbelievable !  I am so happy, I am back in the good old days of using MLVFS all the time.  Now, it works with losslessly compressed MLVs with sound for me.  What a relief!  Thank you again.  May I suggest that you upload the 32-bit version of MLVFS to the download area?  I am sure, people on Win7 x64 will be very happy about that.

BTW, if you may need lossless MLVs with sound from the 100D for your tests, please let me know.  I plan to post one on the MLV_Lite thread later today anyway.  I do hope, you will be able to fix the crash issue on the 32-bit version of MLVApp too.  I am very excited about it!


bouncyball

@IDA_ML

I'm glad it work good for you :)

Quote from: IDA_ML on October 25, 2017, 11:59:36 AM
May I suggest that you upload the 32-bit version of MLVFS to the download area?  I am sure, people on Win7 x64 will be very happy about that.
It is already. You downloaded it from there :)

The_bald_guy

Hi guys,

First thank you so much for your hard work, I believe in ML community.
So here is my question, I am colour grading the short film that I made with the 5D mkIII.

This shot will be the example. Specs: 14bit lossless 1920x1080 23.976fps 800 iso 50th/sec. There seems to be a fix pattern on the footage. I tried a darkframe automation, Switch with all sorts of export configs and MLVFS. None of them gets rid of the pattern. When I apply the fix pattern it gives me a weird cross hatched image. Look on the door and you will see the issue.

Here is the original DNG: https://www.dropbox.com/s/an80tp2sd41kpp7/1Z7A2572_000422.dng?dl=0









bouncyball

Applying dark/flat frame should be the best option here, image is very noisy at ISO 800, strange. Is there also vertical stripes correction turned on?

With dark/flat frame ask Danne to help you, he's got quite an experience in this field :)

a1ex

The image is underexposed by more than 3 stops.

Based on http://www.magiclantern.fm/forum/index.php?topic=10111.msg118553#msg118553, my advice would be to use a slightly higher exposure time (maybe 1/33 - that one shouldn't cause trouble with artificial lights, just like 1/50) and increase ISO to 3200 or even 6400. On 5D3, the benefits of increasing ISO are much higher in regular 1080p or 720p video modes, compared to photo mode or 1:1 crop, because of... pixel binning.

1080p/720p: by increasing ISO from 3200 to 6400, you lose 1 stop of highlights to gain about 0.5 stops in shadows. Above ISO 6400, there's nothing more to gain.

1:1 crop / photo mode: by increasing ISO from 1600 to 6400, you lose 2 stops of highlights to gain about 0.5 stops (0.36+0.24) in shadows. You'll probably think twice before increasing ISO above 1600 or 3200.


# compute shadow improvement when increasing ISO (100->200, 200->400, ..., 6400->12800)
# in other words: how cleaner your shadows will be after increasing ISO by 1 stop (thus clipping 1 stop of highlights)
# measurements done on 5D3, http://www.magiclantern.fm/forum/index.php?topic=10111.msg118553#msg118553
#         isos    =   [ 100    200    400    800    1600   3200   6400  12800 ];
octave:1> dr_1080 =   [ 11.10  11.05  11.02  10.93  10.73  10.39  9.88  8.88  ];
octave:2> dr_crop =   [ 11.22  11.13  11.00  10.82  10.32   9.68  8.92  7.70  ];
octave:3> 1 + diff(dr_1080)
                           0.950  0.970  0.910  0.800  0.660  0.490  0.00
octave:4> 1 + diff(dr_crop)
                           0.910  0.870  0.820  0.500  0.360  0.240 -0.22
# note: output aligned manually for easier reading


In your case: 1/50 -> 1/33 means more photons captured (0.6 EV more), and ISO 800 -> 6400 will improve the shadows by about 2 stops (0.8+0.66+0.5), while clipping 3 stops of highlights. In your test image, clipping 3 + 0.6 EV of highlights would affect a grand total of... 72 pixels (0.0035% of the image area).

Math:

dcraw -4 -E 1Z7A2572_000422.dng
octave:1> a = imread('1Z7A2572_000422.pgm');
octave:2> prctile(a(:), [50 90 99 99.9 99.99 99.999 100])'
ans =
   2094.0   2184.0   2306.0   2402.0   3037.0   3335.5   3493.0

# overexposure amount (using default white level from MLV metadata)
octave:3> log2(16200-2048) - log2(3493-2048)
ans =  3.2919

# how many pixels would be clipped after increasing the exposure by 3.6 stops?
octave:4> 2 ^ (log2(16200-2048) - 3.6) + 2048
ans =  3215.1
octave:5> sum(a(:) > 3215)
ans =  72


The algorithm used for fixed pattern noise has a major weakness: it gets easily confused by horizontal or vertical lines in the actual image. I have some ideas to rework the algorithm, by looking at past and future frames and applying some sort of optical flow, but it's not easy.

A dark frame may improve things, but my previous attempt in a similar case was unsuccessful (because the pattern is not exactly repeatable with a dark frame - instead, my understanding is that Canon code fine-tunes it continuously, with a time constant of a few seconds - see e.g. the vertical artifacts in centered x10 zoom with crop_rec_4k builds).

I'd like to take a closer look at this pattern change, but cannot promise a timely fix. Do you mind uploading a sample MLV?

Danne

QuoteThe image is underexposed by more than 3 stops.
+1
Darkframes probably won't bite on this.

The_bald_guy

Great to have some feedback on this issue. I know that I have a problematic exposure. I do think that in other shots where exposure is well to the right it would not be as apparent.

Here is the original MLV and a dark frame at 800iso.
https://wetransfer.com/downloads/6622b5b15ba8ca07d5d5ff12d882fa6520171118150345/bafaaff4cebaf896c59e92ce36ba665920171118150345/08c1cf
2 files 1.4gig
1Z7A2572.MLV
800iso.MLV

Thank you so much for your support guys!

a1ex

Hm, the first frame from the MLV is dark - how did you get that?!

H.264 proxy getting slightly out of sync?

edit: no improvements using this dark frame...