Hi guys,
Last two weeks I've been investigating lossles JPEG (92) specs, LJ92Lib from Andrew Baldwin and MLV files produced by both a1ex's lovely crop_rec_4k/mlv_lite and by mlv_dump from lj92 branch, also DNGs produced by MLRAWviewer.
It seems that Andrew Baldwin's implementation is somewhat weird. It encodes raw data as two images each with its own headers, one scan and one component in it. That means that encoded data has two SOI (0xFFD8, Start of image) and EOI (0xFFD9, End of image) markers and each image has its own SOF3 (0xFFC3, Start of frame header), DHT (0xFFC4, Huffman table) and SOS (0xFFDA, Start of scan header) with its own one image component with full width and half height. Also baldand chose the prediction method 6 in his encoder because of reasons explained here (http://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html). If someone interested the original document is here (http://www.w3.org/Graphics/JPEG/itu-t81.pdf).
Meanwile the raw data encoded (as configured by a1ex) by Canon HW compressor fully follows the specs and has one SOI/EOI, which accordingly have DHT, SOF3 and SOS in it. SOS consists of two components with full width and half height. Predictor used is number 1.
This is the reason why LJ92lib can not decode ML files out of the box despite it supports all 7 predictors while decoding (not only number 6). The component number should be correctly taken into account.
While studying all these I came across Syoyo Fujita's github account (https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h) with his modified header file based on baldand's LJ92lib. ( which also was pointed out by @martinherring :) )
In short Syoyo correctly modded the main loop in parseScan function to support components and changed decoder struct and added 'components' field to it. Also he added multiple Huffmann table support to the lib which is really unnecessary at least for ML compressed data. So I edited lj92.c accordingly. Unfortunatelly encoder still needs some work. However for MLVFS encoder is unnecessary :)
Using all this info I pathched MLVFS and it works nicely with all its bells and whistles (on the fly raw processing). I want to thank Danne as usual :) for testing/support and MAC binary.
David Milligan merged changes to MLVFS repo (https://bitbucket.org/dmilligan/mlvfs/commits/all).
Download latest MAC binary here (https://bitbucket.org/dmilligan/mlvfs/downloads/) and Windows x64 binary here (https://bitbucket.org/bouncyball/mlv-tools/downloads/).
The latest 'DokanSetup_redist.exe' installer is here (https://github.com/dokan-dev/dokany/releases).
I also modified mlv_dump-on-steroids (http://www.magiclantern.fm/forum/index.php?topic=18975.0) but unfortunatelly it's not ready for prime time yet. Needs more time. Always time :P. Now supports all latest stuff plus some features that none of the mlv_dump versions have. Binaries uploaded.
Note: mlv_dump-on-steroids merged in ML repository to crop_rec_4K branch.
regards
bb
Dude... am I dreaming? :D
@DeafEyeJedi: Hope you gonna test it asap ;)
Bravo Bouncyball. Thanks for the insightful explanations. Lj92 code very much needed around all bleeding edge compressed raw coming from a1ex. Been running a bunch of test compressed raw files from Deafeyejedi on both MLVFS and mlv_dump flavours(ml-dng, steroid) code and from my testing it works really well. Did try to find some vertical stripes to fix but cannot say I found any stripes to check yet so that needs to be tested. Darkframe averaging was still not working but all the rest seems to be in there which is so great to have. Will post a version of cr2hdr.app soon. And also, so cool to see Baldand´s code coming alive again.
I've not followed the developments with compressed MLV but it really interests me a lot since 3TB drives piles up pretty fast. I'll take whatever comes next!
Hi,
You've got information here too:
http://lclevy.free.fr/cr2/
https://github.com/lclevy/libcraw2/blob/master/docs/cr2_lossless.pdf?raw=true
https://github.com/lclevy/libcraw2
Laurent
I replaced the mlvfs.exe on windows764bit and now get an error message "dokanfuse.dll is missing, please reinstall dokany".
What am I missing?
Been sleepwalking ever since Dr. B has given us this opportunity to use MLVFS just like the good old days...
It's a breeze to be able to quickly view files virtually especially w UHD/3K/4K/Full Res lossless stuff. Happy times! 8)
(https://c1.staticflickr.com/5/4157/34277560275_99a6fe0344.jpg) (https://flic.kr/p/UdZssx) (https://c1.staticflickr.com/3/2851/34237161706_301f1b3647.jpg) (https://flic.kr/p/Uaqpnu)
Thanks a bunch for this and will definitely test out thoroughly with several samples that I've collected from other users complaining of vertical stripes not working and such, etc.. Fun Fun!
Quote from: erikbaldwinson on April 26, 2017, 08:47:12 AM
I replaced the mlvfs.exe on windows764bit and now get an error message "dokanfuse.dll is missing, please reinstall dokany".
What am I missing?
Well, what comes to my mind first is: do you use 32 bit dokany/mlvfs on 64bit Win7? I compiled native x64 binary, so dokan must be x64 too. Install this package: https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe (https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe) and then copy from C:\Program Files\Dokan\Dokan Library-1.0.3 to mlvfs dir 2 files: dokan1.dll and dokanfuse1.dll.
Quote from: DeafEyeJedi on April 26, 2017, 09:23:07 AM
Thanks a bunch for this and will definitely test out thoroughly with several samples that I've collected from other users complaining of vertical stripes not working and such, etc.. Fun Fun!
Good to hear :) thanks.
Decompressing takes place before raw data processing. So there is no reason for stripes corection not to work. However this algo is not 100% perfect and there are some situations when it's not getting rid of all stripes regardless data compressed or not.
regards
bb
@lorenzo353: thank you for posting links.
Oh my, oh my. Thanking you Mr Bouncy Balls ;D. I have been lost without MLVFS. In fact - thank you everybody who makes this possible! :D
I'm getting 100% usage on CPU (all cores) when I try to copy files from mounted mlv which has compressed lossless raw. Is this known issue on macos version? (I even do not see the usage in "top" command - (system 100% cpu)
edit: this was probably because I selected bad pixel checkbox. Now without it it is blazing fast.
Another question - is there any mlv_dump version which can correct (and force it) vertical stripes?
I just keep getting mlvfs stopped working when i try to test this new verison of mlvfs.
I installed the newer Dokany 1.0.3 and I'm using the mlvfs.exe from the link in this thread. (https://bitbucket.org/bouncyball/mlvfs/downloads/?tab=downloads)
I had standard mlvfs that worked previous to trying this build. And I uninstalled the older version of Dokany before trying out the new version.
Avast and windows defender went bananas when I first tried to execute it but since then I added exceptions for that library.
In the old version of mlvfs there were a bunch of other files besides the mlvfs.exe in the same folder so I also tried to add these to the new versions location but got the same stopped working error.
Any Idea what I'm doing wrong?
Any help appreciated.
Using Win 10 x64 (Havn't upgraded to creators update jet)
hi, good work.
added it to mlv_dump two days ago :-)
https://www.magiclantern.fm/forum/index.php?topic=18443.msg183690#msg183690
https://bitbucket.org/hudson/magic-lantern/commits/45796d811c2c21c259b92951ab255d4bef217316
br,
g3gg0
A idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.
Check out dmilligan's work on mlv compression.
https://bitbucket.org/dmilligan/magic-lantern/branch/ml_dng_lj92
How about losslessly compressed dng files like in MlRawViewer?
Quote from: martinhering on April 26, 2017, 06:39:51 PM
An idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.
+1
Quote from: Danne on April 26, 2017, 06:55:09 PM
How about losslessly compressed dng files?
Ah, that's right. Especially with @cpc's slimRAW stuff. Great reminder @Danne!
This update is very nice indeed :)
Could you update the Binary for MLVFS_x64 ? I'd like to try it, but only MLVFS_x64 works on my PC :s
Quote from: martinhering on April 26, 2017, 06:39:51 PM
A idea I had was an option in mlv_dump to recompress existing mlv files. Might be useful for some, who like to archive MLV instead of CinemaDNG. It would also be useful to have an option for compressing the raw image when converting to CinemaDNG.
not wanna totally hijack this thread.
compression is already implemented.
though not exactly the same method as canon uses, but close to (https://bitbucket.org/hudson/magic-lantern/commits/7732a13b494aefe17686aa1170a2562168b634c5#Lmodules/mlv_rec/mlv_dump.cT2890).
but further mlv_dump discussion please in some other thread :)
Quote from: justinbacle on April 26, 2017, 07:13:50 PM
Could you update the Binary for MLVFS_x64 ? I'd like to try it, but only MLVFS_x64 works on my PC :s
Binary in download area is indeed native x64 exe not the 32bit version.
@g3gg0: Thank you I saw those changes. It's just the version of the lj92.c you used has tons of unnecessary stuff from tiny_dng_loader author :)
i am happy to take over a cleaner one.
does your version support multi component compression?
Hi!
Very happy to read this! I wonder thou I have previous dmg of MLVFS installed on OS X, do I need to uninstall/remove pref or just install latest?
Best,
David
Just double click to overwrite the old one.
I just tried the new version. I'm new to it. Still don't know how to activate the command line controls BUT.... grayscale previews on 3k 12bit vids. I imported a DNG sequence from the mounted drive straight into AE CC and output a MOV just fine if not a bit slower. Vertical Stripes were gone after selecting them in the interface. Huzzah! on this release.
Craig
QuoteWell, what comes to my mind first is: do you use 32 bit dokany/mlvfs on 64bit Win7? I compiled native x64 binary, so dokan must be x64 too. Install this package: https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe and then copy from C:\Program Files\Dokan\Dokan Library-1.0.3 to mlvfs dir 2 files: dokan1.dll and dokanfuse1.dll.
Thank you, working now! One more thing tho, http://localhost:8000/ is now an empty white page?
Also, in resolve this is happening..
(http://i.imgur.com/HibiBwe.png)
Quote from: erikbaldwinson on April 27, 2017, 01:05:21 AM
Thank you, working now! One more thing tho, http://localhost:8000/ is now an empty white page?
Also, in resolve this is happening..
Don't know about resolve issue but for correctly display settings web page you absolutly have to change dir (cd) to MLVFS folder when running it from the command line. Otherwise mlvfs can not find needed files to display page corectly. Also it can not find focus pixel maps.
@g3gg0:
Quote from: g3gg0 on April 26, 2017, 10:38:34 PM
i am happy to take over a cleaner one.
does your version support multi component compression?
Unfotunately, as I said in the first post encoder needs some work (correct component support). However MLVFS decodes it without problem along with hw encoded files.
I made a pull request with a small fix absolutely necessary to decode compressed files produced by mlv_dump in crop_rec_4k branch. The main header been overwritten at the end by /* fix frame count */ code, hence videoClass was wrong 0x01 istead of 0x21.
regards
bb
thanks for the fix.
i think it leaves LZMA flags (which still could appear) set now instead if LJ92 :D
QuoteDon't know about resolve issue but for correctly display settings web page you absolutly have to change dir (cd) to MLVFS folder when running it from the command line. Otherwise mlvfs can not find needed files to display page corectly. Also it can not find focus pixel maps.
You're the man! thx!
@g3gg0: Thank you for merging it so fast :). LZMA flag needs attention, correct, almost forgot about it.
I am interested in trying out MLVFS, I understand it works for Windows now aswell?
Can someone point me to a thread on setting it up ?
I run Win 8.1.
Tutorial:
https://youtu.be/6v4EWj0zXDs (https://youtu.be/6v4EWj0zXDs)
I didn't get this new version working on win 10 though...
I get instant "this program stopped working" when executing mlvfs.exe.
Let me know if you have any luck on win 8
Quote from: Markus on April 30, 2017, 12:14:09 PM
Tutorial:
https://youtu.be/6v4EWj0zXDs (https://youtu.be/6v4EWj0zXDs)
I didn't get this new version working on win 10 though...
I get instant "this program stopped working" when executing mlvfs.exe.
Let me know if you have any luck on win 8
I have it working on Windows 10, just got it up tonight. I had to install the x64 installer and it was good to go. Thank you everyone!!!
EDIT: The only bug I'm noticing is I cant close MLVFS unless I go into the task manager and end task. My raw files come out really cool/blue. Other than that everything looks great.
Quote from: hindra on May 05, 2017, 04:33:31 AM
The only bug I'm noticing is I cant close MLVFS unless I go into the task manager and end task.
Use 'dokanctl.exe' to unmount the drive correctly: dokanctl.exe /u Z (or whatever drive you use)
Here it's location (x64 ver) and usage:
C:\Program Files\Dokan\Dokan Library-1.0.0>dokanctl.exe
Driver path: 'C:\WINDOWS\system32\drivers\dokan1.sys'
dokanctl /u MountPoint (/f)
dokanctl /m
dokanctl /i [d||n]
dokanctl /r [d||n]
dokanctl /v
Example:
/u M : Unmount M: drive
/u C:\mount\dokan : Unmount mount point C:\mount\dokan
/u 1 : Unmount mount point 1
/u M /f : Force unmount M: drive
/i d : Install driver
/i n : Install network provider
/r d : Remove driver
/r n : Remove network provider
/d [0-9] : Enable Kernel Debug output
/v : Print Dokan version
Quote from: hindra on May 05, 2017, 04:33:31 AM
My raw files come out really cool/blue. Other than that everything looks great.
Let me guess - you used auto or aperture/shutter priority exposure mode.
Quote from: bouncyball on May 05, 2017, 09:49:03 AM
Use 'dokanctl.exe' to unmount the drive correctly: dokanctl.exe /u Z (or whatever drive you use)
Here it's location (x64 ver) and usage:
C:\Program Files\Dokan\Dokan Library-1.0.0>dokanctl.exe
Driver path: 'C:\WINDOWS\system32\drivers\dokan1.sys'
dokanctl /u MountPoint (/f)
dokanctl /m
dokanctl /i [d||n]
dokanctl /r [d||n]
dokanctl /v
Example:
/u M : Unmount M: drive
/u C:\mount\dokan : Unmount mount point C:\mount\dokan
/u 1 : Unmount mount point 1
/u M /f : Force unmount M: drive
/i d : Install driver
/i n : Install network provider
/r d : Remove driver
/r n : Remove network provider
/d [0-9] : Enable Kernel Debug output
/v : Print Dokan version
Let me guess - you used auto or aperture/shutter priority exposure mode.
Thank you for the instruction on unmounting the drive. I did not use any auto exposure mode, it was manual at f4, iso 200 1/60th but I believe I was on auto WB.
QuoteI just keep getting mlvfs stopped working when i try to test this new verison of mlvfs.
I installed the newer Dokany 1.0.3 and I'm using the mlvfs.exe from the link in this thread. (https://bitbucket.org/bouncyball/mlvfs/downloads/?tab=downloads)
I had standard mlvfs that worked previous to trying this build. And I uninstalled the older version of Dokany before trying out the new version.
Avast and windows defender went bananas when I first tried to execute it but since then I added exceptions for that library.
In the old version of mlvfs there were a bunch of other files besides the mlvfs.exe in the same folder so I also tried to add these to the new versions location but got the same stopped working error.
After installing this version of MLVFS on a win 10 computer at home without Avast (firewall/antivirus prog.) I got it working without problems. So somehow the initial crash that occurred when Avast virus search kicked in have permanently made mlvfs.exe crash on that computer.
I tried adding exceptions for mlvfs in both Avast and windows defender, even uninstalling Avast, Reg edit cleaning tools, re-downloaded and reinstalled mlvfs and dokany. Nothing has worked so far.
Also recreated the same problem on a win7 computer that's also using avast. After mlvfs crashes due to Avast virus check It keeps crashing every time you try to start it after that.
Does anyone have any Idea how one crash can make that file crash forever on that system even though you reinstall dokany and download the mlvfs-file once again?
Quote from: Markus on May 07, 2017, 12:18:59 AM
Does anyone have any Idea how one crash can make that file crash forever on that system even though you reinstall dokany and download the mlvfs-file once again?
Yes I also experienced this with Nod32. MLVFS crashed all the time randomly. Stopping only file monitoring service helped but not fully. After uninstalling antivirus _entirely_ issue has gone away. Well that kind of protection software hooks all file IO (and not only IO) under its control so there is no surprise that MLVFS, also very intense on file system multi threaded IO, does not like this.
My rule is: when using MLVFS - antivirus is an enemy :). Under Windows it definitely causes performance/crashing issues.
bb
Hi! Using Mac OS 10.12.5 with latest of MLVFS. Things have been working quite good. But today after shooting Full-resolution LiveView: 5120 x 2768 at 7.4 fps (14bit lossless) MLVFS keeps crashing Finder. Unmounting and force robot Finder only hangs the computer. Reboot to make it work again. Any clues?
After trying everything in this thread, I must say that mlvfs on steroids is broken on Win7_x64, as reported earlier here.
The executable keeps crashing, and the Esset AV activity has no effect on this.
The previous version works like a charm :(
First of all there is no any "MLVFS on steroids" in the nature.
Second please describe the environment you use and provide information and some MLVs to test and reproduce crashes.
bb
Win7_x64, and following procedure described in the first post. Carefully restarting after each step in addition. No matter what arguments I supply, I get APPCRASh shortly after calling it.
What environment did you use for compiling it?
For people developing this software, Zstandardt is the new next thing on lossless data compression, maybe it could help save some space (although I don't know where to use it in this case... BTRFS is already adding support for it):
http://www.zstd.net/
http://www.phoronix.com/scan.php?page=news_item&px=Facebook-Linux-Zstd
For people using "anti-virus" on Windows: stop. It will not make your computer more secure. For explanation, just look on how these "anti-virus" works. If you want more security, install uBlock on browser and don't open random documents while surfing or from email.
@milank
Quote from: milank on August 07, 2017, 04:58:23 PM
Win7_x64, and following procedure described in the first post. Carefully restarting after each step in addition. No matter what arguments I supply, I get APPCRASh shortly after calling it.
Well if you carefully read 1st post of this _exact_ thread you should notice that there are none of the instructions of using MLVFS. Just download links. Maybe you meant this thread http://www.magiclantern.fm/forum/index.php?topic=13152.0 (http://www.magiclantern.fm/forum/index.php?topic=13152.0)
Quote from: milank on August 07, 2017, 04:58:23 PM
What environment did you use for compiling it?
Windows 10 pro x64, Visual Studio 2015, compiled native x64 binary.
Btw you mentioned that previous version worked without issues. This updated version is exactly the same as previous for uncomperssed MLVs (it just additionally supports compressed ones).
So... if you used uncompressed MLVs the result must be the same. But if you use compressed ones you can't even compare because previous version simply does not support them.
bb
ok, let's start again.
I meant this thread, starting from working previous version of mlvfs.exe.
Downloading new mlvfs.exe, installing DokanSetup_redist, rebooting.
While previous version (11.2.2016) still working, new version results in APPCRASH when executing without arguments. Therefore no link to whatever MLV file.
Uninstalling 0.8 (win32) dokany does not help. Digging in ProcMon, I see no significant difference comparing both versions.
MVC++2015_redist_x64 installed. Disabling AV does not help.
Have experienced the same problem. Couldn't fix it on the computer that crashed on first launch due to the virus protection kicking in. Got it working on my gaming pc though. Made sure that virus program did not interfere there. No reinstall/uninstall measures could get it working on my work pc though :-(
@milank
OK then :).
If:
1. you uninstalled Dokany ver 0.8
2. installed Dokany version 1.0.3 (x64)
3. downloaded latest mlvfs.exe (x64)
4. created new folder and put this binary and two needed files: "jquery-1.12.0.min.js" and "html_template.html" to it (also some .fpm focus pixel maps if needed)
5. copied 2 DLLs (dokan1.dll, dokanfuse1.dll) from "\program files\dokan\dokan library\" to this newly created folder
6. disabled antivirus entirely!
and MLVFS crashes right after starting the binary
then
either you messed up with Dokan DLL versions (x86 instead of x64)
or
It's really weird and I guess, unfortunately, something is wrong with your system
Please try those steps on some clean windows system and report back.
regards
bb
I confirm the steps you described, but the problem persists.
Seems like the only new mlvfs.exe is suffering, as dokanctl.exe is running fine as well.
So giving up for now, will survive with mlvdump for the time being.
Thank you, anyway.
Quote from: bouncyball on April 26, 2017, 09:57:53 AM
Well, what comes to my mind first is: do you use 32 bit dokany/mlvfs on 64bit Win7? I compiled native x64 binary, so dokan must be x64 too. Install this package: https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe (https://github.com/dokan-dev/dokany/releases/download/v1.0.3/DokanSetup_redist.exe) and then copy from C:\Program Files\Dokan\Dokan Library-1.0.3 to mlvfs dir 2 files: dokan1.dll and dokanfuse1.dll.
Downloaded the package to the created MLVFS_x64 directory; Installed all the dokanSetup exe, copied the 2 files into the MLVFS directory; then what?.... How do I get to mount the new Z drive? when I try executing the mlvfs.exe it stops working... I downloaded the x64 version...
I`m installing on Windows 10 x64
Quote from: hindra on May 05, 2017, 04:33:31 AM
I have it working on Windows 10, just got it up tonight. I had to install the x64 installer and it was good to go. Thank you everyone!!!
EDIT: The only bug I'm noticing is I cant close MLVFS unless I go into the task manager and end task. My raw files come out really cool/blue. Other than that everything looks great.
Installed Visual Studio... Then in Dokan installed the.msi file, all .exe files... executed the mlvfs.exe ... doesn`t work...
Quote from: bouncyball on April 25, 2017, 08:39:43 PM
Hi guys,
Last two weeks I've been investigating lossles JPEG (92) specs, LJ92Lib from Andrew Baldwin and MLV files produced by both a1ex's lovely crop_rec_4k/mlv_lite and by mlv_dump from lj92 branch, also DNGs produced by MLRAWviewer.
It seems that Andrew Baldwin's implementation is somewhat weird. It encodes raw data as two images each with its own headers, one scan and one component in it. That means that encoded data has two SOI (0xFFD8, Start of image) and EOI (0xFFD9, End of image) markers and each image has its own SOF3 (0xFFC3, Start of frame header), DHT (0xFFC4, Huffman table) and SOS (0xFFDA, Start of scan header) with its own one image component with full width and half height. Also baldand chose the prediction method 6 in his encoder because of reasons explained here (http://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html). If someone interested the original document is here (http://www.w3.org/Graphics/JPEG/itu-t81.pdf).
Meanwile the raw data encoded (as configured by a1ex) by Canon HW compressor fully follows the specs and has one SOI/EOI, which accordingly have DHT, SOF3 and SOS in it. SOS consists of two components with full width and half height. Predictor used is number 1.
This is the reason why LJ92lib can not decode ML files out of the box despite it supports all 7 predictors while decoding (not only number 6). The component number should be correctly taken into account.
While studying all these I came across Syoyo Fujita's github account (https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h) with his modified header file based on baldand's LJ92lib. ( which also was pointed out by @martinherring :) )
In short Syoyo correctly modded the main loop in parseScan function to support components and changed decoder struct and added 'components' field to it. Also he added multiple Huffmann table support to the lib which is really unnecessary at least for ML compressed data. So I edited lj92.c accordingly. Unfortunatelly encoder still needs some work. However for MLVFS encoder is unnecessary :)
Using all this info I pathched MLVFS and it works nicely with all its bells and whistles (on the fly raw processing). I want to thank Danne as usual :) for testing/support and MAC binary.
David Milligan merged changes to MLVFS repo (https://bitbucket.org/dmilligan/mlvfs/commits/all).
Download latest MAC binary here (https://bitbucket.org/dmilligan/mlvfs/downloads/) and Windows x64 binary here (https://bitbucket.org/bouncyball/mlv-tools/downloads/).
The latest 'DokanSetup_redist.exe' installer is here (https://github.com/dokan-dev/dokany/releases).
I also modified mlv_dump-on-steroids (http://www.magiclantern.fm/forum/index.php?topic=18975.0) but unfortunatelly it's not ready for prime time yet. Needs more time. Always time :P. Now supports all latest stuff plus some features that none of the mlv_dump versions have. Binaries uploaded.
regards
bb
Is it a x64 binary in the link? It is a mlvfs.exe rather then a mlvfs_64.exe and it I can`t mount the drive with it... in the Command Prompt it gives an error...
Hello, what about the removal of AF points (EOSM)?
I have in my folder the required * .fpm profiles, but mlvfs does not use them.
(https://thumb.ibb.co/jmT1jQ/M15_2130_000000.png) (https://ibb.co/jmT1jQ)
Bouncyball,
First of all, I would like to thank you for all your time and efforts that you dedicate to improving and making ML tools more efficient for all of us. I appreciate that very much and am sure that many other people on this forum feel the same way.
I have been using the 32-bit version of MLVFS very successfully for years now and wanted very much to upgrade to the x64 bit version to use it with losslessly compressed MLVs. I have been trying this for several months now, unfortunately without success. I have been following this thread as well as the original one:
https://www.magiclantern.fm/forum/index.php?topic=13152.0
very carefully. I also repeated the procedure described in your post #47 at least 20 times, no luck. As I described in another thread, (the one about the MLVApp), I am on Win7 x64 Ultimate, SP1 with all Windows updates installed and NO antivirus software. Every time I start the batch file as usual, I get the message:
"MLVFS.exe has stopped working".
And this happens on at least 3 different computers that I tested.
When I reinstall the 0.8.0 version of Dokan and the very old 32 bit MLVFS.exe file from the OP in the above thread, everything works fine again but only with 10/12/14-bit uncompressed MLV files.
As you see, there are also other people experiencing exactly the same problem meaning that there is some kind of a compatibility issue between your x64 bit software and some Win x64 computers that noone of us has experienced with other software . Noone knows what the problem is and believe us, we did everything stricktly following your instructions - no messing around with 32/64 bit versions or anything else. Obviously, you are the only one who can find a solution, maybe with some help from the other developers. This would make us more than happy and your continued efforts in this direction are very welcome.
I have a question for you and the other developers. Is there somewhere a 32-bit version of MLVFS.exe that works with Dokan 0.8.0 and can process 8 to 14 bits losslessly compressed MLV files? A download link should be greatly appreciated.
Well I have not tried MLVFS with dokany version later than 1.0.3 (2.0.0-BETA1 is the latest). Maybe this is the reason IDK. Gotta test it myself. However with v1.0.3 MLVFS never, I repeat, NEVER crashed on start for me.
Edit: I have to say I'm sitting under Linux nearly all the time and maybe some later windows updates also have some impact. Have to take a look...
Edit2: I don't have windows 7 x64 anywhere around to test with ;)
Thanks a lot for your willing to help, Bouncyball. I hope, a solution will eventually be found.
I would like to inform you that I tried all dokany versions starting with 1.0.3 and ending with 2.0.0. Beta-1. Of course, I added the corresponding versions of the dokan.dll and dokanfuse.dll to the directory containing the mlvfs.exe package. Exactly the same behavior on all of them - crash upon mlvfs.exe start. I was wondering if the latest mlvfs.exe version could be compiled to 32 bit. In that case it should work with the old 0.8.0 dokan version. Is this too much work to do? If not, this may be the working solution. Don't you think?
Sure. I'm gonna compile 32bit version as soon as I get to my windows environment at work.
Actually I remember that all crashes and hangups for me occurred with version 0.8.
Meanwhile I tested latest x64 binary on my home PC. Here is Win10 installed. All works as expected.
With dokany up to 1.0.5 even dll files are not required in the mlvfs folder.
With dokany 2.0.0-BETA1 only dokanfuse2.dll have to be renamed to dokanfuse1.dll and copied to the mlvfs folder.
Also tried to install Dokany on some old Win7pro x64 virtual machine and it complained that some update is not present on this windows system and it could not be installed.
So... I've come up to the conclusion that this is most likely could be Windows 7 + Dokany issue (as I remember others experiencing this on Win7 as well)
I'll compile 32bit version tomorrow.
regards
bb
Quote from: bouncyball on October 22, 2017, 10:31:16 AM
I'll compile 32bit version tomorrow.
bb
Wow, that sounds exciting, Bouncyball! Thank you so much. I can't wait to test!
Quote from: bouncyball on October 22, 2017, 10:31:16 AM
So... I've come up to the conclusion that this is most likely could be Windows 7 + Dokany issue (as I remember others experiencing this on Win7 as well)
I doubt it, although I have no idea about software and computer programming. The fact that MLVApp shows exactly the same behavior even without Dokany being installed at all, (MLVApp crashes upon an attempt to import a MLV file), indicates that Dokany has nothing to do with the issue. I am very much inclined to think that these crashes have something to do with 64 vs. 32 bit versions of your software on Win 7x64. The only version of MLVApp that worked on all of my Win7 x64 computers was v.3.0. As far as I remember, this was a 32-bit version. The very first MLVFS that works with Dokany 0.8.0 is also a 32 version. Conclusion: Win 7x64 likes 32-bit versions but does not like 64-bit versions of your software.
@IDA_ML
Sorry about late post. Here is the x86 (32bit) version of latest mlvfs (https://bitbucket.org/bouncyball/mlv-tools/downloads/mlvfs_winx86.zip).
Try it with dokany 1.0.5 x86 under Win 7 or if you like with your beloved v0.8 ;). Use both files from zip.
is there a mac version?
Nope. Mac version is at official (dmilligan's) MLVFS repository - here (https://bitbucket.org/dmilligan/mlvfs/downloads/).
Thank you so much, Bouncyball. I will test as soon as I get back home.
Bouncyball,
You are my hero !!! It works !!! I can now use your mlvfZ.exe for losslessly compressed MLV files in the MLVFS system. It even kills the focus pixels frm my 100D files using the old .fpm files from the MLVFS_x86 directory.
For those MLVFS users on Win7 x64 who wish to try it, here are a few instructions:
1) Uninstall the Dokany 0.8.0 library and restart your PC
2) Install the DokanSetup_redist-1.0.5.1000 package
3) Copy the two files from Bouncyball's ZIP file into your MLVFS_x86 directory
4) Rename the old mlvfs.exe to mlvfs_old.exe in case you wish to revert and rename mlvfz.exe to mlvfs.exe
5) Enjoy!
------------------------
Thank you so much, Bouncyball! For me, there is no doubt now that Win7 x64 works well with the 32-bit versions of your software but crashes with the 64-bit ones. If you could also compile a 32-bit version of the latest MLVApp, I would finally be able to test it.
@IDA_ML
Nice to hear! :D
Quote from: IDA_ML on October 25, 2017, 08:16:39 AM
For me, there is no doubt now that Win7 x64 works well with the 32-bit versions of your software but crashes with the 64-bit ones. If you could also compile a 32-bit version of the latest MLVApp, I would finally be able to test it.
I would post 32bit mlvapp but it crashes when importing MLV. We did not figure out why yet.
p.s. mlvfZ, btw was typo ;)
It works so well, Bouncyball, unbelievable ! I am so happy, I am back in the good old days of using MLVFS all the time. Now, it works with losslessly compressed MLVs with sound for me. What a relief! Thank you again. May I suggest that you upload the 32-bit version of MLVFS to the download area? I am sure, people on Win7 x64 will be very happy about that.
BTW, if you may need lossless MLVs with sound from the 100D for your tests, please let me know. I plan to post one on the MLV_Lite thread later today anyway. I do hope, you will be able to fix the crash issue on the 32-bit version of MLVApp too. I am very excited about it!
@IDA_ML
I'm glad it work good for you :)
Quote from: IDA_ML on October 25, 2017, 11:59:36 AM
May I suggest that you upload the 32-bit version of MLVFS to the download area? I am sure, people on Win7 x64 will be very happy about that.
It is already. You downloaded it from there :)
Hi guys,
First thank you so much for your hard work, I believe in ML community.
So here is my question, I am colour grading the short film that I made with the 5D mkIII.
This shot will be the example. Specs: 14bit lossless 1920x1080 23.976fps 800 iso 50th/sec. There seems to be a fix pattern on the footage. I tried a darkframe automation, Switch with all sorts of export configs and MLVFS. None of them gets rid of the pattern. When I apply the fix pattern it gives me a weird cross hatched image. Look on the door and you will see the issue.
Here is the original DNG: https://www.dropbox.com/s/an80tp2sd41kpp7/1Z7A2572_000422.dng?dl=0
(http://thumb.ibb.co/d6eVv6/1_Z7_A2572_000422_original.jpg) (http://ibb.co/d6eVv6)
(http://thumb.ibb.co/b3yOF6/1_Z7_A2572_000422_fix_stripe_on.jpg) (http://ibb.co/b3yOF6)
(http://thumb.ibb.co/cCaaTR/1_Z7_A2573_000422_Fix_Patern.jpg) (http://ibb.co/cCaaTR)
Applying dark/flat frame should be the best option here, image is very noisy at ISO 800, strange. Is there also vertical stripes correction turned on?
With dark/flat frame ask Danne to help you, he's got quite an experience in this field :)
The image is underexposed by more than 3 stops.
Based on http://www.magiclantern.fm/forum/index.php?topic=10111.msg118553#msg118553, my advice would be to use a slightly higher exposure time (maybe 1/33 - that one shouldn't cause trouble with artificial lights, just like 1/50) and increase ISO to 3200 or even 6400. On 5D3, the benefits of increasing ISO are much higher in regular 1080p or 720p video modes, compared to photo mode or 1:1 crop, because of... pixel binning.
1080p/720p: by increasing ISO from 3200 to 6400, you lose 1 stop of highlights to gain about 0.5 stops in shadows. Above ISO 6400, there's nothing more to gain.
1:1 crop / photo mode: by increasing ISO from 1600 to 6400, you lose 2 stops of highlights to gain about 0.5 stops (0.36+0.24) in shadows. You'll probably think twice before increasing ISO above 1600 or 3200.
# compute shadow improvement when increasing ISO (100->200, 200->400, ..., 6400->12800)
# in other words: how cleaner your shadows will be after increasing ISO by 1 stop (thus clipping 1 stop of highlights)
# measurements done on 5D3, http://www.magiclantern.fm/forum/index.php?topic=10111.msg118553#msg118553
# isos = [ 100 200 400 800 1600 3200 6400 12800 ];
octave:1> dr_1080 = [ 11.10 11.05 11.02 10.93 10.73 10.39 9.88 8.88 ];
octave:2> dr_crop = [ 11.22 11.13 11.00 10.82 10.32 9.68 8.92 7.70 ];
octave:3> 1 + diff(dr_1080)
0.950 0.970 0.910 0.800 0.660 0.490 0.00
octave:4> 1 + diff(dr_crop)
0.910 0.870 0.820 0.500 0.360 0.240 -0.22
# note: output aligned manually for easier reading
In your case: 1/50 -> 1/33 means more photons captured (0.6 EV more), and ISO 800 -> 6400 will improve the shadows by about 2 stops (0.8+0.66+0.5), while clipping 3 stops of highlights. In your test image, clipping 3 + 0.6 EV of highlights would affect a grand total of... 72 pixels (0.0035% of the image area).
Math:
dcraw -4 -E 1Z7A2572_000422.dng
octave:1> a = imread('1Z7A2572_000422.pgm');
octave:2> prctile(a(:), [50 90 99 99.9 99.99 99.999 100])'
ans =
2094.0 2184.0 2306.0 2402.0 3037.0 3335.5 3493.0
# overexposure amount (using default white level from MLV metadata)
octave:3> log2(16200-2048) - log2(3493-2048)
ans = 3.2919
# how many pixels would be clipped after increasing the exposure by 3.6 stops?
octave:4> 2 ^ (log2(16200-2048) - 3.6) + 2048
ans = 3215.1
octave:5> sum(a(:) > 3215)
ans = 72
The algorithm used for fixed pattern noise has a major weakness: it gets easily confused by horizontal or vertical lines in the actual image. I have some ideas to rework the algorithm, by looking at past and future frames and applying some sort of optical flow, but it's not easy.
A dark frame may improve things, but my previous attempt (http://www.magiclantern.fm/forum/index.php?topic=18975.msg186766#msg186766) in a similar case was unsuccessful (because the pattern is not exactly repeatable with a dark frame - instead, my understanding is that Canon code fine-tunes it continuously, with a time constant of a few seconds - see e.g. the vertical artifacts in centered x10 zoom with crop_rec_4k builds).
I'd like to take a closer look at this pattern change, but cannot promise a timely fix. Do you mind uploading a sample MLV?
QuoteThe image is underexposed by more than 3 stops.
+1
Darkframes probably won't bite on this.
Great to have some feedback on this issue. I know that I have a problematic exposure. I do think that in other shots where exposure is well to the right it would not be as apparent.
Here is the original MLV and a dark frame at 800iso.
https://wetransfer.com/downloads/6622b5b15ba8ca07d5d5ff12d882fa6520171118150345/bafaaff4cebaf896c59e92ce36ba665920171118150345/08c1cf
2 files 1.4gig
1Z7A2572.MLV
800iso.MLV
Thank you so much for your support guys!
Hm, the first frame from the MLV is dark - how did you get that?!
H.264 proxy getting slightly out of sync?
edit: no improvements using this dark frame...
I used the latest 2017-10-16magiclantern-crop_rec_4k firmware in 14bit lossless at 23.976fps 50th/sec on Komputerbay 128gig cards. I would get some crash on audio recording from the canon UI every once in a while, but we made it thought the shoot.
Well, a1ex's comprehensive answer is a top notch as usual :)
Got some half-decent results with this: https://github.com/pathak22/pyflow
For each frame:
- selected a few nearby frames (choice not critical, e.g. i-3 ... i+3 is fine, but some difficult frames will need more)
- computed a warped version from the nearby frames, using their demo script (only changed the file names)
- averaged the current frame and its warped versions => a temporally denoised frame
- from the difference image (denoised - original), kept only the vertical pattern (column medians) and discarded the rest
Seems to work as long as the camera is moved horizontally (otherwise it's hard to tell the difference between pattern noise and vertical objects in the image). When the camera motion stops, the pattern reappears (need to pick different frames).
Can you upload a few frames before and after your test image (http://www.magiclantern.fm/forum/index.php?topic=19429.msg193315#msg193315), graded in the same way? 15 JPEGs before and 15 after should be fine, just to run a quick test and have something to compare with. I've got this (http://a1ex.magiclantern.fm/bleeding-edge/raw/pattern-noise/1Z7A2572_000422-a5-cm.jpg), but it's hard to judge when they don't have the same grading.
Hello! This is my first post, although i'm not really that new to ML. In the first place, I wanna' thank you all for what you are doing for me, for Canon, for everyone :). On the second place, I bumped into a problem. I recorded some compressed mlvs, about 150 with my 700d, spread on 3 sd cards, in 600d's crop mode. The problem is that i know how to work with mlvfs, the old one, but this one gives me dngs for just one or two mlvs, and the rest of the folders from the virtual drive are empty. The webgui sees their names, but with no info, no preview. The resolution used is 1600x900. The Mlvs are not corrupt, because i can work with them in mlvp, but for such number of files I would prefeer the mlvfs workflow. I could use your help :). Thanks!
Coud you upload somewhere smaller file which is not showing in preview of webgui?
Thank you and sorry for wasting your time, I eventually found out what was causing the problem. It was the file "fseventsd" which was on the card. It took me several hours to become smarter than a rock, but I did it :D . Now it works just like before, maybe a tad slower. Thank you again. Have a great day!
Hm... what is that file?
Directory. A gift from macOS/OS X.
Ah :) understood.
I have a problem with both the x64 and x86 version of this version of mlvfs. I downloaded and installed dokany 1.0.5 64Bit and installed it. Then i downloaded the file "mlvfs.exe" from bouncyballs repo. I placed it into "C:\mlvfs" and copied the fpm files into the folder. When i open the command prompt and type in "mlvfs.exe x: --mlv_dir=C:\mlv\in" mlvfs just stops working. When I do the same thing with the x86 version of mlvfs, it creates the new drive and shows my files but i cant open localhost:8000 or rather it is just a white page. My system is Windows 10 x64 with newest updates installed (Fall creators update).
What am I missing here? Would be nice to use the x64 version and be able to test out the fpm files dfort created for the 100D and be able to set options inside the browser.
Did you install Dokany 64bit with 'DokanSetup_redist.exe'? this is essential.
I now did, but still mlvfs crashes when i try to open it. It works fine on my Laptop, which is pretty much the same software, Windows 10 x64 Home. Any other runtime library I have to install?
IDK, sorry could be anything (any lib incompatibility) you have to track it down yourself sorry, I personally had no issues long time now.
I'll try, thanks for your help :) If I'll find what's causing the problems I will report it here in case someone else has it too. And the x86 version works fine. I deleted the html files while testing out different versions of mlvfs and forgot to add them again... So I can at least test it now.
Quote from: canneloni on January 22, 2018, 09:25:45 AM
I'll try, thanks for your help :) If I'll find what's causing the problems I will report it here in case someone else has it too. And the x86 version works fine. I deleted the html files while testing out different versions of mlvfs and forgot to add them again... So I can at least test it now.
I might have had the same ptoblem. In one of my office computers my antivirus program somehow made mlvfs crash the fist time I installed it, after that no amount of reinstallation could resolve this problem. Installing mlvfs on my gaming PC avoiding the antivirus program to interfere I got it working there. It's like something permanently broke in the system I first tried to install it on. Haven't tried reinstalling win due to the pain of it.
Skickat från min SM-G900F via Tapatalk
Quote from: canneloni on January 22, 2018, 09:25:45 AM
And the x86 version works fine.
This is one of the indication that some libs for 64bit are incompatible.