MLVFS - a FUSE based, "on the fly" MLV to CDNG converter

Started by dmilligan, August 31, 2014, 02:01:24 AM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

mothaibaphoto

@Lasting Image:
Quote from: Lasting Image on February 19, 2016, 07:38:48 AM
I'm using a Mac and have the same problem where only the first dng in the After Effects sequence has the Camera Raw corrections.
This is how AE interprets DNG sequences: It opens ACR window with first file, you store settings in either file itself or sidecar xmp according to your settings.
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
This has some benefits (no flickering, for example), and drawbacks (you can't ramp exposure, or wb, for example), but this is absolute offtopic to MLVFS.

bouncyball

Quote from: g3gg0 on February 19, 2016, 02:05:19 AM
can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

With previous no debug version of mlvfs binaries:
Under the root dir (e.g. mount point Z:\ or whatever) where MLV virtual folders are - no file/dir operations possible
Under virtual MLV folders - only directory creation is possible (no subdirs), .MLD created in the orig mlv folder

With last posted debug version of mlvfs binaries:
Under the root DIR - possible: creating dir/file, copying dir/file, not possible: delete created dir/file (as expected)
Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

BTW Am I supposed to see the .IDX .M01 .M02 files in the root dir along with the virtual .MLV folders? They are there :-\ in last debug version.

bb

dmilligan

Quote from: mothaibaphoto on February 19, 2016, 08:16:20 AM
From this point it converts any other file in sequence disregarding has it his own metadata, or not.
Actually, this is wrong. It does not disregard xmp files or embedded metadata for other images in the sequence if they are there. That's how this script is even able to work. I have used it many times with AE.

mothaibaphoto

@dmilligan:
I'm a big fan and very power user of your script :)
Take this opportunity to say Great Thanks for it one more time :)
But ... my AE works as I described. And it's generally good for me. Just specially tried to save different settings and open sequence. Maybe some settings I don't know affect this behavior.
After applying your script I always convert right in Bridge - CtrlA->CtrlR

Ottoga

@Dmilligan

A big thumbs up from me as well for your Windows version.

I've been giving it a solid try out during the mlv lite testing and love the on the fly the processing.

Took me a while to figure out that that I had to set the parameters before opening the Dokan mounted file. [Facepalm].
EOS 7D.203, EFS 55-250mm, EF 75-300 III, Tamron 16-300 DiII VC PZD Macro, SpeedLite 580EX II.

g3gg0

@bouncyball:
can you check it?

Quote from: g3gg0 on February 19, 2016, 02:05:19 AM
can you redownload from the same URL and test again?
it also contains debug info that will slow down the tool.

the things not working:

- editing *.dng, *.gif, *.log, *.wav of a virtual MLV folder (subdirs are okay)
- placing *.dng, *.gif, *.log, *.wav into a virtual MLV folder (subdirs are okay)
- copying/moving *.dng, *.gif, *.log, *.wav of a virtual MLV folder into a subdirectory
- deleting folders (https://github.com/dokan-dev/dokany/pull/166)

Quote from: g3gg0 on February 17, 2016, 09:45:07 PM
well, i dont get any crashes anymore when aborting copy operations with dokan 1.0-rc

MLVFS_x86 or MLVFS_x64 to be used with dokan 1.0-rc -> DokanSetup.exe

make sure you uninstall the old version first.

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

@g3gg0

I already did :)

http://www.magiclantern.fm/forum/index.php?topic=13152.msg162572#msg162572

Has anything changed since then?

It's not crashing but, as you said, unusable in real situations because of the standard output log slowing it down so much.
Could not manage to stress it so to see some exceptions.

Passthrough related things are in my message above.

bb

bouncyball

redownloaded binaries. yeah they dated 02/20/16 :). after checking, can admit, that applies exactly the same what I've been talking about in 2 earlier messages.

g3gg0

sorry, too much noise so i didnt see your post.
thanks.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

@dmilligan

I have one question about bad pixel removal. Can we do this with pixel maps (like focus pixel maps by dfort). Just having an option to remove them by the map or current algorithm and then anyone can make pixel maps for his/her needs: sensor and resolution (crop, non crop etc). It would be lot more faster.

Besides I have just one pixel and that's all ;)

bb

dfort

Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Of course this would require some effort on the user but it shouldn't be any more difficult than the instructions in dcraw:

FILES
       ./.badpixels, ../.badpixels, ../../.badpixels, ...
              List of your camera's dead pixels, so that dcraw can interpolate
              around them.  Each line specifies the column, row, and UNIX time
              of death for one pixel.  For example:

               962   91 1028350000  # died between August 1 and 4, 2002
              1285 1067 0           # don't know when this pixel died

              These coordinates are before any stretching or rotation, so  use
              dcraw -j -t 0 to locate dead pixels.


I don't see any real need to keep track of the date.

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dfort

You mean the "UNIX time of death for one pixel" idea?  :P I never use it but dcraw "badpixels" lists require something in that field. I just fill it in with zeros.

DeafEyeJedi

Excellent concept @bouncyball and great suggestion by @dfort!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

bouncyball

Quote from: dfort on February 21, 2016, 11:26:05 PM
Here's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Nice idea! Let's wait what master David will say :)

bb

mothaibaphoto

Actually, is there dead pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Etc) :)

Danne

QuoteHere's an idea. How about if there is a pixel map file with the camera serial number in the MLVFS Contents? That way it will only fix the dead pixels on that specific camera. This would be in addition to the focus pixel map files because these cameras can also have dead pixels.

Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

mothaibaphoto

Ah, my bad :(
Bad pixels always there.
Cold/hot pixels can be relatively stable. So,
Is there dead pixel = Fn(Camera Serial, Time)
Is there Cold/hot pixel = Fn(Camera Serial, ISO, Shutter, Sensor temp, Time, Etc)
Ultimately, it would be great to have sensor profiling tool, similar to dot_tune.

bouncyball

Quote from: Danne on February 22, 2016, 09:51:23 AM
Are dead/hot pixels always on the same place? My understanding is that the will vary from time to time. Of course a dead pixel is a dead pixel but a hot pixel could pop up anywhere? Am I wrong here?

I think you are right :). Appearance of that kind of pixels depends on several things:
AFAIK canon software has factory callibration built in and maps pixels itself (however you can kinda alter this remapping with several second manual sensor cleaning procedure. That's how I cleaned up some hot pixels - not bads - on my former 7D) if you got chance read raw buffer before that then you have vanilla information with all off uncorercted pixels etc and I think this is a case when we get croped resolutions of raw data from the sensor (some buffers in memory I guess). Also line skipping/bining affects coordinates and appearence of those pixels. In short, whether they appear in output raw or not, depends on the camera mode and what canon firmware does in that moment and also what we are doing with great help of ML software ;).

But... the pixel correction map is still for a dedicated resolution and mode, hence valid in most cases.

This is my understanding... correct me please if i'm wrong.


bb

Danne

Would have been smooth with a dot picker in MLVFS which creates the map on the fly when clicking your dots on an dng file :P (Wishful thinking)

bouncyball

Saying Clicking do you mean automatic removal algo? Sometimes it misses some dots ;)

Danne

Since it often happens you only have one or two annoying pixel dots you could find its coordinates on pixel level(zoomed in around 400%). Since Davids web gui works in realtime maybe this could be achieved by grabbing neighbouring pixels like in dcraw and the previewing dng would instantly show the result? As I said, wishful thinking  8)
There,s also opcode lists which could work with the dng format. But how I don,t know.

dfort

If I may use a term that dmilligan used previously, K-I-S-S.

Focus pixels and bad/dead pixels are always there. Maybe they are not always visible or pose a problem but the location of these pixels on the sensor for a particular camera never change. These are the pixels that can be relatively easily mapped and eliminated.

Pixels that change according to the temperature of the sensor or other conditions are something else. Astrophotographers go to great lengths to separate celestial objects from this noise. Google astrophotography bias frames and you'll see what I mean.

Then there is fixed pattern noise (FPN) which is something that can also change and thus cannot be mapped out.

Maybe I'm wrong but it seems that sensor profiling cannot be done just once and it will be good for the life of that camera. Focus pixels and bad/dead pixels fixing is permanent.

As far as a dot picker--unless MLVFS gets a graphical user interface that can zoom in far enough to work on individual pixels it is easy enough to map out dead pixels with Photoshop or Gimp.

g3gg0

@bouncyball:
> Under any virtual MLV folder - nothing's possible! no file/dir creation/copying/moving/deleting, hence no MLD directories appear in the orig mlv destination

can create dirs and files without any issues.
delete/copy not possible. trying to work around that dokan issue
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

jogodedentro

Hi There

With help from this forum I was happily colour grading MLV files in davinci resolve. But now after a reinstall of windows 10, I have installed Dokany and despite entering the path to DokanyLibrary the dokanctl.exe command brings up a 'not recognized internal or external command...' error message. This command was previously working on the same Windows 10 machine, but now after a re-install of windows I have this issue. I have installed the latest version of Dokany. Any ideas how I can get this to work again, advice will be much appreciated...