Author Topic: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter  (Read 855301 times)

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #975 on: February 19, 2016, 08:16:20 AM »
@Lasting Image:
I'm using a Mac and have the same problem where only the first dng in the After Effects sequence has the Camera Raw corrections.
This is how AE interprets DNG sequences: It opens ACR window with first file, you store settings in either file itself or sidecar xmp according to your settings.
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
This has some benefits (no flickering, for example), and drawbacks (you can't ramp exposure, or wb, for example), but this is absolute offtopic to MLVFS.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #976 on: February 19, 2016, 10:25:59 AM »
can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

With previous no debug version of mlvfs binaries:
Under the root dir (e.g. mount point Z:\ or whatever) where MLV virtual folders are - no file/dir operations possible
Under virtual MLV folders - only directory creation is possible (no subdirs), .MLD created in the orig mlv folder

With last posted debug version of mlvfs binaries:
Under the root DIR - possible: creating dir/file, copying dir/file, not possible: delete created dir/file (as expected)
Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

BTW Am I supposed to see the .IDX .M01 .M02 files in the root dir along with the virtual .MLV folders? They are there :-\ in last debug version.

bb

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #977 on: February 19, 2016, 02:02:31 PM »
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
Actually, this is wrong. It does not disregard xmp files or embedded metadata for other images in the sequence if they are there. That's how this script is even able to work. I have used it many times with AE.

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #978 on: February 19, 2016, 02:53:20 PM »
@dmilligan:
I'm a big fan and very power user of your script :)
Take this opportunity to say Great Thanks for it one more time :)
But ... my AE works as I described. And it's generally good for me. Just specially tried to save different settings and open sequence. Maybe some settings I don't know affect this behavior.
After applying your script I always convert right in Bridge - CtrlA->CtrlR

Ottoga

  • Member
  • ***
  • Posts: 203
  • The Ox is Slow but the Earth is Patient
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #979 on: February 19, 2016, 03:49:27 PM »
@Dmilligan

A big thumbs up from me as well for your Windows version.

I've been giving it a solid try out during the mlv lite testing and love the on the fly the processing.

Took me a while to figure out that that I had to set the parameters before opening the Dokan mounted file. [Facepalm].
EOS 7D.203, EFS 55-250mm, EF 75-300 III, Tamron 16-300 DiII VC PZD Macro, SpeedLite 580EX II.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3182
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #980 on: February 20, 2016, 10:28:19 AM »
@bouncyball:
can you check it?

can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

the things not working:

 - editing *.dng, *.gif, *.log, *.wav of a virtual MLV folder (subdirs are okay)
 - placing *.dng, *.gif, *.log, *.wav into a virtual MLV folder (subdirs are okay)
 - copying/moving *.dng, *.gif, *.log, *.wav of a virtual MLV folder into a subdirectory
 - deleting folders (https://github.com/dokan-dev/dokany/pull/166)

well, i dont get any crashes anymore when aborting copy operations with dokan 1.0-rc

MLVFS_x86 or MLVFS_x64 to be used with dokan 1.0-rc -> DokanSetup.exe

make sure you uninstall the old version first.

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #981 on: February 20, 2016, 01:41:09 PM »
@g3gg0

I already did :)

http://www.magiclantern.fm/forum/index.php?topic=13152.msg162572#msg162572

Has anything changed since then?

It's not crashing but, as you said, unusable in real situations because of the standard output log slowing it down so much.
Could not manage to stress it so to see some exceptions.

Passthrough related things are in my message above.

bb

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #982 on: February 20, 2016, 02:53:11 PM »
redownloaded binaries. yeah they dated 02/20/16 :). after checking, can admit, that applies exactly the same what I've been talking about in 2 earlier messages.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3182
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #983 on: February 20, 2016, 09:38:08 PM »
sorry, too much noise so i didnt see your post.
thanks.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #984 on: February 21, 2016, 08:08:14 PM »
@dmilligan

I have one question about bad pixel removal. Can we do this with pixel maps (like focus pixel maps by dfort). Just having an option to remove them by the map or current algorithm and then anyone can make pixel maps for his/her needs: sensor and resolution (crop, non crop etc). It would be lot more faster.

Besides I have just one pixel and that's all ;)

bb

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #985 on: February 21, 2016, 11:26:05 PM »
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Of course this would require some effort on the user but it shouldn't be any more difficult than the instructions in dcraw:

Code: [Select]
FILES
       ./.badpixels, ../.badpixels, ../../.badpixels, ...
              List of your camera's dead pixels, so that dcraw can interpolate
              around them.  Each line specifies the column, row, and UNIX time
              of death for one pixel.  For example:

               962   91 1028350000  # died between August 1 and 4, 2002
              1285 1067 0           # don't know when this pixel died

              These coordinates are before any stretching or rotation, so  use
              dcraw -j -t 0 to locate dead pixels.

I don't see any real need to keep track of the date.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3182
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #986 on: February 22, 2016, 12:52:54 AM »
that unix time idea is quite funny t.b.h :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #987 on: February 22, 2016, 01:35:09 AM »
You mean the "UNIX time of death for one pixel" idea?  :P I never use it but dcraw "badpixels" lists require something in that field. I just fill it in with zeros.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3401
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #988 on: February 22, 2016, 04:39:18 AM »
Excellent concept @bouncyball and great suggestion by @dfort!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #989 on: February 22, 2016, 08:38:28 AM »
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Nice idea! Let's wait what master David will say :)

bb

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #990 on: February 22, 2016, 09:05:50 AM »
Actually, is there dead pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Etc) :)

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7240
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #991 on: February 22, 2016, 09:51:23 AM »
Quote
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

mothaibaphoto

  • Senior
  • ****
  • Posts: 393
  • pesky kid
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #992 on: February 22, 2016, 10:37:23 AM »
Ah, my bad :(
Bad pixels always there.
Cold/hot pixels can be relatively stable. So,
Is there dead pixel = Fn(Camera Serial, Time)
Is there Cold/hot pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Time, Etc)
Ultimately, it would be great to have sensor profiling tool, similar to dot_tune.

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #993 on: February 22, 2016, 01:22:26 PM »
Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

I think you are right :). Appearance of that kind of pixels depends on several things:
AFAIK canon software has factory callibration built in and maps pixels itself (however you can kinda alter this remapping with several second manual sensor cleaning procedure. That's how I cleaned up some hot pixels - not bads - on my former 7D) if you got chance read raw buffer before that then you have vanilla information with all off uncorercted pixels etc and I think this is a case when we get croped resolutions of raw data from the sensor (some buffers in memory I guess). Also line skipping/bining affects coordinates and appearence of those pixels. In short, whether they appear in output raw or not, depends on the camera mode and what canon firmware does in that moment and also what we are doing with great help of ML software ;).

But... the pixel correction map is still for a dedicated resolution and mode, hence valid in most cases.

This is my understanding... correct me please if i'm wrong.


bb

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7240
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #994 on: February 22, 2016, 01:28:05 PM »
Would have been smooth with a dot picker in MLVFS which creates the map on the fly when clicking your dots on an dng file :P (Wishful thinking)

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 840
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #995 on: February 22, 2016, 01:43:04 PM »
Saying Clicking do you mean automatic removal algo? Sometimes it misses some dots ;)

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7240
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #996 on: February 22, 2016, 01:46:40 PM »
Since it often happens you only have one or two annoying pixel dots you could find its coordinates on pixel level(zoomed in around 400%). Since Davids web gui works in realtime maybe this could be achieved by grabbing neighbouring pixels like in dcraw and the previewing dng would instantly show the result? As I said, wishful thinking  8)
There,s also opcode lists which could work with the dng format. But how I don,t know.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #997 on: February 22, 2016, 01:48:59 PM »
If I may use a term that dmilligan used previously, K-I-S-S.

Focus pixels and bad/dead pixels are always there. Maybe they are not always visible or pose a problem but the location of these pixels on the sensor for a particular camera never change. These are the pixels that can be relatively easily mapped and eliminated.

Pixels that change according to the temperature of the sensor or other conditions are something else. Astrophotographers go to great lengths to separate celestial objects from this noise. Google astrophotography bias frames and you'll see what I mean.

Then there is fixed pattern noise (FPN) which is something that can also change and thus cannot be mapped out.

Maybe I'm wrong but it seems that sensor profiling cannot be done just once and it will be good for the life of that camera. Focus pixels and bad/dead pixels fixing is permanent.

As far as a dot picker--unless MLVFS gets a graphical user interface that can zoom in far enough to work on individual pixels it is easy enough to map out dead pixels with Photoshop or Gimp.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3182
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #998 on: February 22, 2016, 08:49:18 PM »
@bouncyball:
> Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

can create dirs and files without any issues.
delete/copy not possible. trying to work around that dokan issue
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

jogodedentro

  • New to the forum
  • *
  • Posts: 5
Re: MLVFS - a FUSE based, "on the fly" MLV to CDNG converter
« Reply #999 on: February 23, 2016, 05:30:19 AM »
Hi There

With help from this forum I was happily colour grading MLV files in davinci resolve. But now after a reinstall of windows 10, I have installed Dokany and despite entering the path to DokanyLibrary the dokanctl.exe command brings up a 'not recognized internal or external command...' error message. This command was previously working on the same Windows 10 machine, but now after a re-install of windows I have this issue. I have installed the latest version of Dokany. Any ideas how I can get this to work again, advice will be much appreciated...