MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

[mothaibaphoto]

@Lasting Image:

I'm using a Mac and have the same problem where only the first dng in the After Effects sequence has the Camera Raw corrections.

This is how AE interprets DNG sequences: It opens ACR window with first file, you store settings in either file itself or sidecar xmp according to your settings.
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
This has some benefits (no flickering, for example), and drawbacks (you can't ramp exposure, or wb, for example), but this is absolute offtopic to MLVFS.

[bouncyball]

can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

With previous no debug version of mlvfs binaries:
Under the root dir (e.g. mount point Z:\ or whatever) where MLV virtual folders are - no file/dir operations possible
Under virtual MLV folders - only directory creation is possible (no subdirs), .MLD created in the orig mlv folder

With last posted debug version of mlvfs binaries:
Under the root DIR - possible: creating dir/file, copying dir/file, not possible: delete created dir/file (as expected)
Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

BTW Am I supposed to see the .IDX .M01 .M02 files in the root dir along with the virtual .MLV folders? They are there in last debug version.

bb

[dmilligan]

From this point it converts any other file in sequence disregarding has it his own metadata, or not.

Actually, this is wrong. It does not disregard xmp files or embedded metadata for other images in the sequence if they are there. That's how this script is even able to work. I have used it many times with AE.

[mothaibaphoto]

@dmilligan:
I'm a big fan and very power user of your script
Take this opportunity to say Great Thanks for it one more time
But ... my AE works as I described. And it's generally good for me. Just specially tried to save different settings and open sequence. Maybe some settings I don't know affect this behavior.
After applying your script I always convert right in Bridge - CtrlA->CtrlR

[Ottoga]

@Dmilligan

A big thumbs up from me as well for your Windows version.

I've been giving it a solid try out during the mlv lite testing and love the on the fly the processing.

Took me a while to figure out that that I had to set the parameters before opening the Dokan mounted file. [Facepalm].

[g3gg0]

@bouncyball:
can you check it?

can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

the things not working:

 - editing *.dng, *.gif, *.log, *.wav of a virtual MLV folder (subdirs are okay)
 - placing *.dng, *.gif, *.log, *.wav into a virtual MLV folder (subdirs are okay)
 - copying/moving *.dng, *.gif, *.log, *.wav of a virtual MLV folder into a subdirectory
 - deleting folders (https://github.com/dokan-dev/dokany/pull/166)

well, i dont get any crashes anymore when aborting copy operations with dokan 1.0-rc

MLVFS_x86 or MLVFS_x64 to be used with dokan 1.0-rc -> DokanSetup.exe

make sure you uninstall the old version first.

[bouncyball]

@g3gg0

I already did

http://www.magiclantern.fm/forum/index.php?topic=13152.msg162572#msg162572

Has anything changed since then?

It's not crashing but, as you said, unusable in real situations because of the standard output log slowing it down so much.
Could not manage to stress it so to see some exceptions.

Passthrough related things are in my message above.

bb

[bouncyball]

redownloaded binaries. yeah they dated 02/20/16 . after checking, can admit, that applies exactly the same what I've been talking about in 2 earlier messages.

[g3gg0]

sorry, too much noise so i didnt see your post.
thanks.

[bouncyball]

@dmilligan

I have one question about bad pixel removal. Can we do this with pixel maps (like focus pixel maps by dfort). Just having an option to remove them by the map or current algorithm and then anyone can make pixel maps for his/her needs: sensor and resolution (crop, non crop etc). It would be lot more faster.

Besides I have just one pixel and that's all

bb

[dfort]

Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Of course this would require some effort on the user but it shouldn't be any more difficult than the instructions in dcraw:

Code: [Select]

FILES
       ./.badpixels, ../.badpixels, ../../.badpixels, ...
              List of your camera's dead pixels, so that dcraw can interpolate
              around them.  Each line specifies the column, row, and UNIX time
              of death for one pixel.  For example:

               962   91 1028350000  # died between August 1 and 4, 2002
              1285 1067 0           # don't know when this pixel died

              These coordinates are before any stretching or rotation, so  use
              dcraw -j -t 0 to locate dead pixels.

I don't see any real need to keep track of the date.

[g3gg0]

that unix time idea is quite funny t.b.h

[dfort]

You mean the "UNIX time of death for one pixel" idea?  I never use it but dcraw "badpixels" lists require something in that field. I just fill it in with zeros.

[DeafEyeJedi]

Excellent concept @bouncyball and great suggestion by @dfort!

[bouncyball]

Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Nice idea! Let's wait what master David will say

bb

[mothaibaphoto]

Actually, is there dead pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Etc)

[Danne]

Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

[mothaibaphoto]

Ah, my bad
Bad pixels always there.
Cold/hot pixels can be relatively stable. So,
Is there dead pixel = Fn(Camera Serial, Time)
Is there Cold/hot pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Time, Etc)
Ultimately, it would be great to have sensor profiling tool, similar to dot_tune.

[bouncyball]

Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

I think you are right . Appearance of that kind of pixels depends on several things:
AFAIK canon software has factory callibration built in and maps pixels itself (however you can kinda alter this remapping with several second manual sensor cleaning procedure. That's how I cleaned up some hot pixels - not bads - on my former 7D) if you got chance read raw buffer before that then you have vanilla information with all off uncorercted pixels etc and I think this is a case when we get croped resolutions of raw data from the sensor (some buffers in memory I guess). Also line skipping/bining affects coordinates and appearence of those pixels. In short, whether they appear in output raw or not, depends on the camera mode and what canon firmware does in that moment and also what we are doing with great help of ML software .

But... the pixel correction map is still for a dedicated resolution and mode, hence valid in most cases.

This is my understanding... correct me please if i'm wrong.


bb

[Danne]

Would have been smooth with a dot picker in MLVFS which creates the map on the fly when clicking your dots on an dng file (Wishful thinking)

[bouncyball]

Saying Clicking do you mean automatic removal algo? Sometimes it misses some dots

[Danne]

Since it often happens you only have one or two annoying pixel dots you could find its coordinates on pixel level(zoomed in around 400%). Since Davids web gui works in realtime maybe this could be achieved by grabbing neighbouring pixels like in dcraw and the previewing dng would instantly show the result? As I said, wishful thinking 
There,s also opcode lists which could work with the dng format. But how I don,t know.

[dfort]

If I may use a term that dmilligan used previously, K-I-S-S.

Focus pixels and bad/dead pixels are always there. Maybe they are not always visible or pose a problem but the location of these pixels on the sensor for a particular camera never change. These are the pixels that can be relatively easily mapped and eliminated.

Pixels that change according to the temperature of the sensor or other conditions are something else. Astrophotographers go to great lengths to separate celestial objects from this noise. Google astrophotography bias frames and you'll see what I mean.

Then there is fixed pattern noise (FPN) which is something that can also change and thus cannot be mapped out.

Maybe I'm wrong but it seems that sensor profiling cannot be done just once and it will be good for the life of that camera. Focus pixels and bad/dead pixels fixing is permanent.

As far as a dot picker--unless MLVFS gets a graphical user interface that can zoom in far enough to work on individual pixels it is easy enough to map out dead pixels with Photoshop or Gimp.

[g3gg0]

@bouncyball:
> Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

can create dirs and files without any issues.
delete/copy not possible. trying to work around that dokan issue

[jogodedentro]

Hi There

With help from this forum I was happily colour grading MLV files in davinci resolve. But now after a reinstall of windows 10, I have installed Dokany and despite entering the path to DokanyLibrary the dokanctl.exe command brings up a 'not recognized internal or external command...' error message. This command was previously working on the same Windows 10 machine, but now after a re-install of windows I have this issue. I have installed the latest version of Dokany. Any ideas how I can get this to work again, advice will be much appreciated...