Dealing with Focus Pixels in raw video

[DeafEyeJedi]

Absolutely stunning work @dfort and I am just in awe by staring at these patterned Black/White alienated-alike checker board especially on the last few ( seriously wth?!  ) which is rather surprising to me!

... but I'm not sure that would solve this puzzle. The focus pixels on the 100D are much easier to see so maybe giving that platform the crop_rec module might shed some light on this?

That's true -- I notice that the SL1/100D does show the Focus Pixels in a certain way that makes it easier to see than the other affected ones.

@nikfreak -- is there a branch that I can use to test this with the new 10/12-bit options or is that already on your 'to-do list'?

[dfort]

@nikfreak -- is there a branch that I can use to test this with the new 10/12-bit options or is that already on your 'to-do list'?

Actually what we want is the crop_rec module for the 100D.

BTW--(off topic) what's the holdup with the 70D and 100D merge? Looks like those cameras are about as stable as the rest of the bunch.

[andy kh]

I have been following this forum though i dont use 650d anymore. Btw waiting for the 70d merge

[dfort]

Let's try to make sense of the new information we got from looking at the 3x3 and 3x5 patterns from a1ex's octave script.

If we start with the 100D's 1x1 focus pixels by looking at the pixel maps for mv1080crop or zoom modes then run that through all the possible 3x3 combinations we get nothing that resembles what the focus pixel map that we know is working for that camera. In fact the mv1080 focus pixel maps for all cameras have the same pattern. If we overlay the map3x3+1+1 from the 650D/700D (and theoretically EOSM) mv1080 video mode with the same map3x3+1+1 from the 100D and look at the pixels that are common to all cameras something interesting shows up:



So here is a new hypothesis, not all focus pixels are activated when the shooting in mv1080 mode. Does this hold true for mv720 mode? The pattern that matches the working focus pixel map in this case turned out to be the map5x3+1+3 and overlaying the 100D with the other cameras' mv720 focus pixel map files shows that once again not all focus pixels are active.



Ok--so what's my hypothesis on the weird map I pulled for the EOSM crop_rec?



At first I suspected a rounding error on the frame size because we're using the same full raw buffer size for the crop_rec 3x3 mv720 as the original 3x5 mv720 so something might have been going on with the vertical line skipping. Nice try but not likely.

I believe that somehow I got some 3x5 frames mixed up with 3x3 when I was searching for all the focus pixels that I missed. Here's what it looks like if you overlay the 3x3 with the 3x5 focus pixel maps:



The mirror image might be a different combination of pixel maps but it does prove a point which is basically--my bad!

[dfort]

Was it my bad or perhaps an accidental discovery?

Overlaying both the focus pixel map file for the EOSM crop_rec and the "normal" mv720 full raw buffers gave an interesting pattern that was close to what I came up with when I was trying to map all of the focus pixels form a group of crop_rec MLV files. There must have been a regular mv720 file mixed in there because when I finally got the fpm file working it looked a lot like the pattern from laying the two map files over each other. I tried it again, simply by merging the two fpm files and it worked on all the files I tested. This is what the pattern looks like:


80000331_1808x727

You can click on that link if you want to take a closer look or even download a full sized version. Here's a section of that image:



This map file can handle both crop_rec and mlv_720 raw video files. Sure, it hits a lot of extra pixels for each video mode but it doesn't seem to affect the image much--at least not on the tests that I have done. In addition, it doesn't hit any pixels that aren't on either crop_rec or mv_720 modes. Since crop_rec and mv_720 have the same full raw buffer sizes and that's what we're using to identify the various video modes, this saves us from having to switch between two different focus pixel map files.

I actually got this idea as I was writing that previous post but I was having problems because I was converting between focus pixel map files and portable bit map files using the octave script that a1ex provided. It does work fast but the pixels are off by one in both the vertical and horizontal axis which of course renders the final fpm file useless. I wasn't sure how to fix the script but it didn't matter because all I really had to do was to merge the two working fpm files which are simply tab or whitespace delimited text files that look like this:

Code: [Select]

79  219
87  219
95  219
103  219
111  219
119  219
127  219
135  219
143  219
...
Technically you should be able to map a pixel that is located at 0 0 but it doesn't work in the octave script. Anyone have a solution for that? It would also be nice to have a script that converts from pbm to fpm that runs as fast as a2ex's octave script but honestly I think we've mapped all of the focus pixels for all the cameras with this issue. The newer cameras that are being ported to ML aren't showing their focus pixels on raw video files.

crop_rec is not in the nightly downloads yet but users are experimenting with it so it might be a good time to give the post processing applications developers a heads up that this is coming and that we've already got a focus pixel map file that can deal with it.

By the way, all the files I've been referring to are available on bitbucket:

https://bitbucket.org/daniel_fort/ml-focus-pixels

[dfort]

5x crop zoom mode on the 5D3 wasn't centered until this commit that was announced in this forum post. Zoom mode is quite useful on the 5D3 because it can record at 2.5k resolution. @Danne even modified a lens so it would be centered when using this mode.

That got me thinking about the zoom mode issues with focus pixels. The EOSM/100D/650D/700D cameras have the crop hack that was adapted from the 600D so zoom mode is like the ugly step child for these cameras but it does have a larger raw buffer so it can theoretically record larger image sizes. Of course you need to keep in mind the limits to writing to the SD card and the new 10bit 12bit option along with slowing down the frame rate helps with lowering the data rate. In any case, the problem is that in zoom mode you can pan around the sensor and if the image isn't perfectly centered when recording in raw 1.0 the focus pixels probably won't be removed. The reason for this is because raw 1.0 doesn't have the Crop metadata that MLV has in order to line up the focus pixel map files. The zoom mode map files are already a compromise because they are a combination of 21 possible raw buffer locations. So here's the idea, what if we make a new focus pixel map that has all the possible pan positions too? Panning moves aren't in pixel increments so the combined map file should still have enough "holes" in it to allow the image to come through. Now the question is if it will affect the image quality? This would only be used for raw 1.0 which is probably going to be replaced by raw 1.1 (MLV Lite) so it is more an exercise of how far can we go with these focus pixel maps rather than anything users would adapt. Still, if you come across a raw 1.0 file that was shot in zoom mode that wasn't centered exactly--this should still remove the focus pixels.

[Danne]

Let,s try it out. Gonna be a pretty big amount of coordinates in there but like you said. There should be enough neighbouring pixels to go by.

[dfort]

The purpose of this post is to hopefully grab the attention of anyone who is willing to help add automatic focus pixel removal to mlv_dump.

In order to make the work already done on focus pixel removal more accessible there will be some major changes in the ML Focus Pixels bitbucket repository in the next few days. There are several reasons for the reorganization. The scripts are maturing and doing more than their original functions, there are new focus pixel map files and combinations of old and new map files that we should keep organized as new video modes emerge and the code needs to be documented clearly so that the focus pixel cleanup algorithms can be implimented in other applications.

The goal is to make focus pixels a non-issue for users. The raw video processing applications should shield users from having to take any special steps to deal with it. MLVFS, MLVProducer, MLP and cr2hdr.app are using the focus pixel maps and scripts that were developed from the comments posted in this topic, most notably by dmilligan - Thanks David!

In order to deal with the focus pixels the first thing that needs to be done is to find them. This isn't as simple as it may seem at first because the focus pixels are only visible under certain conditions. One method that has proven effective for finding focus pixels is to shoot a "black frame" -- a common practice in astrophotograhy to sort out "hot pixels" from stars. Once enough coordinates are mapped a pattern becomes apparent and the location of the harder to find pixels can be predicted.

Once all the focus pixels are mapped the pattern created by the focus pixels can be turned into a formula. I experimented with this and implimented it in a bash script that Danne is using in his Mac Automator apps. It takes an extra step to update a formula for a new raw video mode and takes more effort to maintain but the advantage is that you don't need to deal with a library of external map files.

[Danne]

I really like that focus pixel script of yours. Very nice to hear you are planning on improvements.

[dfort]

I just found out that the xxd trick to find the camera model isn't working with MLV Lite files. Everything else seems to be working. Maybe just go back to using exiftool mlv_dump and not worry about it?

[dfort]

I put aside this topic for a while to concentrate on the 10bit 12bit raw video development when dmilligan pointed out that there might be a raw video stream that doesn't show focus pixels. It took some effort but at one point thought I found it.

First of all -- thanks to dmilligan for having the patience to work with me on this.

NO MORE FOCUS PIXELS!



Here's what does the trick--only working on the EOSM at this time:

Code: [Select]

#ifdef CONFIG_EOSM
#define PREFERRED_RAW_TYPE 18
#endif

This is a before shot (using PREFERRED_RAW_TYPE 34)



and here is what it looks like with PREFERRED_RAW_TYPE 18



I'm still running some tests and haven't gotten it working with the crop_rec module yet but I wanted to share the news.

Alas, I wasn't the first one to try this out and the hot pixels that appeared in this video stream was even more difficult to deal with. The stream choices were limited to 64 but it seemed that the default for the 700D was 78 so I increased the limit to 128 and kept looking. A funny thing happened on a few of the video streams. There were more focus pixels. In fact at some of the settings the EOSM files looked identical to the 100D which if you've been following this topic has a different pattern than the 650D, 700D and EOSM which share the same pattern--or so I thought.



Here's a closer look:



Normally the EOSM/700D/650D shows focus pixels over a horizontal strip while the 100D shows focus pixels over a much larger area of the sensor. Seems like there may be focus pixels over a larger portion that we first thought.

Weird part is that using the 100D focus pixel map file on this it took care of most of the pixels. Sure, it isn't perfect and it is extending beyond where the "normal" 100D focus pixels stop but it is an interesting discovery.



Movie crop mode also shows an interesting pattern that we haven't seen before:



Ok, this is all very entertaining but is there any practical value in this? Maybe. When using the video stream that doesn't show focus pixels some hot pixels appear around high contrast boundaries.



The "normal" focus pixel map file in MLVFS, MLP, cr2hdr.app and probably MLV Producer won't get rid of these pixels and chroma smoothing doesn't do it either but when using the focus pixel map file from the 100D they were gone! I haven't found a video stream that is 100% clear of focus pixels but perhaps there are video streams that can use a one size fits all focus pixel map file?

[nikfreak]

Just a reminder as it now could apply to a unifying scenario. I have seen you also discovered that those bunch of cameras use the same sizes for edmac raw slurp, too:

http://www.magiclantern.fm/forum/index.php?topic=16608.msg174241#msg174241

[dfort]

I'm good with a unifying scenario even if it means coming up with all new focus pixel map files.

The octave script a1ex made is super fast and I'd love to have another octave script that can take turn the pbm back to a fpm file but the script is off by 1 pixel on both the x and y axis. Can't figure out how to fix it.

The fpm file has this:

Code: [Select]

78 290
77 291
In Photoshop the pbm file has the dots at:

Code: [Select]

77 289
76 290



There isn't anything to adjust in the script but maybe there is a preference setting somewhere? It seems that instead of starting at 0,0 it starts at 1,1. If I put in a coordinate at 0,0 which should be a valid location, this error comes up:

Code: [Select]

error: index (0,_): subscripts must be either integers 1 to (2^31)-1 or logicals
error: unhandled execution exception -- eval failed

Here's the octave script again:

Code: [Select]

function M = fpm2pbm(image_size, filename)
  # fpm2pbm.m
  #
  # Create a Portable Bit Map file from an
  # MLVFS .fpm (focus pixel map file)
  #
  # Based on fpm2pbm.sh, http://www.magiclantern.fm/forum/index.php?topic=16054.msg161309#msg161309
  # Requires GNU octave.
  #
  # Usage fpm2pbm(image_size, filename)
  # example:
  # octave> fpm2pbm([1108 2592], '80000331_2592x1108.fpm');
  # shell$ octave-cli --eval "fpm2pbm([1108 2592], '80000331_2592x1108.fpm');"
  #
  # This is free and unencumbered software released into the public domain.
   
  M = ones(image_size);
  d = dlmread(filename);
  x = d(:,1); y = d(:,2);
  i = sub2ind(image_size, y, x);
  M(i) = 0;
  imwrite(M, [filename '.pbm']);


[dfort]

Just a reminder as it now could apply to a unifying scenario. I have seen you also discovered that those bunch of cameras use the same sizes for edmac raw slurp, too:

http://www.magiclantern.fm/forum/index.php?topic=16608.msg174241#msg174241

Lots of stuff going on at the same time. I'm suspecting that some of the recent discoveries the focus pixels that appear in ML raw video might have to do with a focus peaking feature that maybe Canon was looking into but wasn't implimented on the EOSM/100D/650D/700D. From my research focus peaking first came out on the M3. In any case this is all just speculation.

[maxotics]

I laugh to think of all the time I spent on the EOS-M a couple of years ago.  I blew many brain cells on the focus pixel problem.  In case you haven't seen this, this is what I came up with at the end, a windows program that takes a known pattern (crop mode) and removes the pixels through debayering.  The "parallelism" idea didn't work as either Alex or g3ggo predicted (frames get out of order).  Anyway, there may be code in here that's of some use to you.  I don't remember what I did, so probably couldn't answer any questions.  Sor far, I find the stripe removal in mlrawviewer to be good enough for me.

https://bitbucket.org/maxotics/focuspixelfixer/src

I just looked at the code again.  I think blood just came out of my ear   

[DeafEyeJedi]

Ha, It's so great to hear from you again @maxotics!

[a1ex]

[...]but the script is off by 1 pixel on both the x and y axis.

Right, in octave, array indexing starts from 1. This should fix it:

Code: [Select]

  i = sub2ind(image_size, y+1, x+1);


[dfort]

...there may be code in here that's of some use to you..

Oh yeah -- I looked at your code and even rendered your pixel map file in this post:

http://www.magiclantern.fm/forum/index.php?topic=16054.msg159826#msg159826

Right, in octave, array indexing starts from 1.

Figured it was that. Thanks for the fix!

[Danne]

Just discovered a nice program which seems useful to detect and coordinate focus pixels pretty quick. It,s called imageJ. I found it while searching for a dead pixel picker which would create coordinates when picking a dead pixel. This also works with the program with a plugin called point picker plugin. It depends on java runtime 6 but just follow the link when asked during installation and it should work after this.

ImageJ
https://imagej.nih.gov/ij/index.html

Installation instructions
https://imagej.nih.gov/ij/docs/install/osx.html

Point picker plugin
http://bigwww.epfl.ch/thevenaz/pointpicker/index.html#Download


I started playing with a dng file from my eosm, exported to tiff with dcraw. Changing some parameters in something called find Maxima seems to be effective. Once sweet spot is found you can send the coordinates out to a list and go from there. I couldn,t really get good use of it yet cause I don,t know quite buffer correlations and so on but will experiment some more when I can.

[maxotics]

ImageJ is pretty cool.  I've used it for photo stitching.  I'm curious why EOS-M Focus Pixels is still an issue when MLVProducer seems to deal with them very well, with additional filters to soften aliasing (moire).  It even does 10bit.  Is it because MLVProduder is PC, or is there something else I'm missing?  Also, MLRawViewer removes focus pixels through "stripe" removal.  Finally, I want to take very opportunity to rave about MLVProducer 

[Danne]

It,s not an issue maybe cause of the work of dfort but it might be useful for fast isolating hot pixels with the point picker. And it seems way faster than imagemagick to find the focus pixels.
By the way. Mlvproducer is using Dfort,s very pixel lists in his program if I,m not mistaken. As well as MLP, cr2hdr.app and MLVFS 

[maxotics]

Thanks Danne. Yes, I knew that but you're SO RIGHT.  I need to rave about dfort too! DFORT, consider yourself an ML hero in my book up there with Alex and g3ggo!  (Whatever happened to 1%?).  Wait, getting off track.  I've been using and loving my EOS-M and I know I have DFORT to thank!

[dfort]

If you are going to be giving credit to anyone it should be dmilligan. He guided me through this whole process.

Interesting find there Danne. There are lots of ways to deal with focus pixels. The focus pixel map is just one way. It may be overkill at times because we're always filling in those coordinates even when the conditions that the frame was shot in doesn't trigger the focus pixel to appear.

By the way, I first got into compiling the Magic Lantern code just so I could turn on Chroma Smoothing in raw2dng to deal with focus pixels on the EOSM. I still turn on Chroma Smoothing when using MLVFS to get rid of the few remaining focus pixels that sneak in at times.

[Danne]

I tried applying the added coordinates by making a list and running it through dcraw. I didn,t seem to do much. I went with the crop_rec.mo on my eosm. Maybe a bad choice?
Anyway. I went with dcraw -T -j -4 -W to create the tiff. Not with -E(where did you find that setting anyway? Not documented) cause it wouldn,t be effective finding the pixels.
In my thinking the very coordinates created from the tiff would work with dcraw when put into a list? Is raw buffer to be considered as well?

[maxotics]

I tried using hot pixel fixes with dcraw and never got anywhere.  As dfort pointed out, the pixels aren't predictable (at least I couldn't) and the ones that can fire take up almost the entire frame. if I remember correctly.  For me, the problem is solved because our sensitivity (or lack of it) for color allows a significant amount of color-information removal to take place before we notice it--the whole idea behind 4:2:0 compression.  At first, that answer I'm giving you was not good enough for me.  I hated the idea of losing, or doubling up on data points like video compression.   So even when Alex or G3ggo came out with chroma smoothing I resisted it.

Now it doesn't bother because my primary love of RAW video is the fact that it allows me to set each pixels 8-bit value (even if smoothed) from 14-bit sources.  One of my biggest curiosities is why I can never get 8-bit video to have the same DR feel as RAW.  Even with the EOS-M at sub 720p, the RAW has a more nuanced looked than, say, 4K from a GH4 or A7 (I've tried just about every consumer camera you can think of).  How can that be?  The only cameras that seem to approach the RAW "nuance" starts with the Canon C100 or Sony FS5, both close to $5,000.  My guess, until someone with more knowledge can explain it to me,  is that in-camera exposure is always a tiny bit off and once it writes to an 8-bit (or 24bit full color) space, even if 4K, there is no way to re-adjust the exposure without seriously degrading the DR feel. 

If the above is right, everyone's effort in 10bit is game-changing to me.  Initial tests I've done, and looking at other videos' shows that even 10bit RAW is vastly superior to any in camera video out there, even 4K.  10bit not only allows higher resolutions, it can also reduce file size.  A focus pixel removed through crude chroma smoothing will probably not be visible to anyone who put a priority on DR.  Video shot in 10bit RAW, I wager, will provide a feel that even 8K will not provide.  Thoughts?