Dealing with Focus Pixels in raw video

[a1ex]

I'm wondering if there is any possible way to get mv1080 on the EOSM but I take it that has been tried before.

It wasn't tried too hard

Looking a bit in forum history, it seems like in the very first implementations of raw video, there was a (undocumented) way to do it: http://www.magiclantern.fm/forum/index.php?topic=5860 and http://www.magiclantern.fm/forum/index.php?topic=12022

Maybe I should revisit it, but that would be a different topic.

[Danne]

That is one cleaned up test chart dfort. Amazing what neighbouring pixels can do.

[dfort]

...I'm wondering if there is any possible way to get mv1080 on the EOSM but I take it that has been tried before.

It wasn't tried too hard
...
Maybe I should revisit it, but that would be a different topic.

Getting the EOSM MLV performance on par with its DSLR cousins would be awesome. That's a topic I'd like to see! 

That is one cleaned up test chart dfort. Amazing what neighbouring pixels can do.

Speaking of cleaning up, I did some cleanup on the mv720 focus pixel map files and updated them on my bitbucket download area. It looks like a couple of people are playing around with the files so whoever you are you might want to get the latest update. I also just discovered that the mv1080 needs some cleanup work too.

Also some progress on the SL1/100D. Looks like the focus pixels make up a rather interesting pattern.

SL1 / 100D mv1080crop mode


SL1 / 100D mv1080 mode

Maybe the pattern is a clue to how line skipping works on this camera?

[Steve_FR]

@dfort
I'm happy to help test Eos M focus pixels, but sadly I've hit a brick wall trying to output an image file from cygwin.
I seem unable to figure out the proper syntax for executing the script in cygwin. I already formatted it for linux line returns with notepad++, but after running:

$ /usr/bin/fpm2pbm.sh 2592x1108 80000326_2592x1108.fpm
sed: can't read 80000326_2592x1108.fpm: No such file or directory

I have both my fpm2pbm.sh & 8000326_2592x1008.fpm files located in the usr/bin/ folder. Pretty sure I'm doing something wrong, but I'm inexperienced with cygwin. Can you instruct me on how to execute the script properly?

[dfort]

@Steve_FR

Ok, let's work this out one step at a time.

I'm happy to help test Eos M focus pixels... 80000326_2592x1108.fpm

The files you need for the EOSM start with 80000331 though it doesn't really matter in this case because you're looking at the 5x zoom pixel map for the T5i/700D which happens to share the same pattern as the EOSM. What is unique about the EOSM is that it doesn't have mv1080 mode. Maybe someday but not now.

I already formatted it for linux line returns with notepad++

I take it you're referring to the fpm2pbm.sh script. That step shouldn't be necessary for the .fpm files because they were created on OS-X which uses the POSIX-compliant Line Feed (LF) character for line endings. POSIX stands for Portable Operating System Interface and that's what Cygwin gives you for Windows. BTW--I'm pretty sure you don't need to change line endings for the scripts or the map files to work.

...
$ /usr/bin/fpm2pbm.sh 2592x1108 80000326_2592x1108.fpm
sed: can't read 80000326_2592x1108.fpm: No such file or directory

I have both my fpm2pbm.sh & 8000326_2592x1008.fpm files located in the usr/bin/ folder....

You were probably in a different directory when you ran that command. In your case this should work:

Code: [Select]

/usr/bin/fpm2pbm.sh 2592x1108 /usr/bin/80000326_2592x1108.fpm
Let's do even better than that. You have the fpm2pbm.sh script in your default path (which includes /usr/bin) but it might not be executable. Run this command so you won't have to type in the path to the script.

Code: [Select]

chmod +x /usr/bin/fpm2pbm.sh
Also, it is better if you don't put the focus pixel map file in /usr/bin/ - that directory is reserved for program files. You can run it from whatever directory you happen to be in, most likely your home directory which is where you're at when you launch Cygwin. So assuming you want the EOSM pixel map the command is simply:

Code: [Select]

fpm2pbm.sh 2592x1108 80000331_2592x1108.fpm
Make sure you install ImageMagick via Cygwin for the script to run. Simply run the Cygwin setup program to install the ImageMagick package.

fpm2pbm.sh is a simple bash shell script and it runs very slowly. The script that goes the other direction, pbm2_MLVFS_pixel_map.sh, runs even slower--it is best to run a 2592x1108 file overnight, it takes that long. BTW--here's a tip when you have a long name to type in the terminal, simply type the first few characters and hit "tab" to auto complete the name as far as it can. If it doesn't auto complete hit the "tab" twice and it should show you all the possible files in the path.

One last tip, this is a work in progress so you should check to see if there is a newer version of the test files on my bitbucket download area before you get started or we might be duplicating our efforts to refine these focus pixel maps.

Hope this helps.

[Steve_FR]

@dfort

Thank you for the detailed instructions! They worked perfectly and the script executed. The .pbm file is building right now. Success!
I'm using the focus pixel file from your newest bitbucket 2016-01-31_Focus_Pixel_Map_Files_testing.zip archive.

I also corrected my other mistake (I am now using **331 instead of **326.)

[dfort]

@Steve_FR

Great. You may want to examine your camera's MLV files with mlv_dump. If you run "mlv_dump -m -v *.DNG | less" you can scroll through the metadata using the arrow keys on your keyboard. The "Crop:" field tells you where to line up your image on the focus pixel map file. Make sure you're using the latest nightly build. There were a few recent updates that fixed a problem with the Crop metadata.

Speaking of updates, was an issue with the 700D (and possibly the 650D) that didn't save the camera metadata properly. It looks like the update hasn't made it into the nightly builds yet so I posted ML testing builds for those cameras in my bitbucket download area.

@Everybody

Focus Pixel Map files for all affected cameras and all MLV modes are now posted on my bitbucket download area. That includes the testing version of MLVFS for OS-X with the .fpm files already installed.

Here are some of the things you might find interesting.

The .fpm with the least amount of mapped focus pixels is for the mlv720 on the EOSM / 650D / 700D (they all share the same focus pixel pattern) at 2,940 while the largest file is for the 100D  zoom mode at 151,122 mapped pixels. The 100D has a very aggressive focus pixel pattern on the sensor that fills most of the mlv modes.

Here's a list from what I found out about all the mlv modes for the cameras that show focus pixels on raw/mlv video files:

Code: [Select]

Mode          Buffer Size   Notes
mv1080      - 1808x1190  -  not available on the EOSM.
mv720       - 1808x727   -  needs to be stretched by 1.67x - up to 50/60 fps except on EOSM.
mv640       - 1808x1190  -  basically the same as mv1080 mode.
mv1080crop  - 1872x1060  -  highest resolution and largest sustainable frame sizes on all cameras.
mv640crop   -            -  not possible.
zoom        - 2592x1108  -  same 5x magnification as mv1080crop but can pan around sensor. Performance not as good as mv1080crop.
Some settings may seem impressive but just because you can set ML to record mlv/raw at frame sizes up to 1792x1026 and up to 60fps doesn't mean it is practical with these cameras -- you might get 1 sec. recording times at best. Scaling down the frame size and frame rate helps for sustained recording. Of course that also means you're not using the entire sensor. The highest quality and performance is mv1080crop. You can get the same image sizes along with the ability to pan around the sensor using zoom mode but recording times are limited in that mode probably because of the larger raw frame buffer size.

The only issue that I can see at the moment is that focus pixels still show up on vertical lines with high contrast.

@dmilligan

In the comments for mlvfs/mlvfs/cs.c:

Code: [Select]

    //simply average the two horizontally neighboring pixels of the same color
    //simple and fast, do we need something better?
    //horizontal direction only b/c could be dual ISO

I also made that assumption but a while back I had good results using dcraw on a Dual ISO. My understanding is that dcraw is also including the neighboring vertical pixels when interpolating the "badpixels" values.

Here's what I posted back on Reply #22:

I've been testing Danne's new MLPP workflow and he discovered that it works with the focus pixel map file on dual iso mlv files. Quite a surprise. I thought dual iso was causing issues with this.

Compare this with the same shot run without the focus pixel map file.

Yes, I know that the color and contrast don't match between these clips. I used different processes to produce the videos.

Maybe it is because 3 out of the 4 pixels are close enough in value or maybe dcraw is throwing out the outlier pixel?

[dmilligan]

So it basically sounds like the simple linear interpolation isn't going to be good enough. There are certainly numerous, more complex and robust ways of interpolation, I was just hoping to avoid needing to use them (main reason: speed). I will need some good example footage that clearly shows where the current method struggles (and some dual ISO too).

[Steve_FR]

Compared the Eos M focus pixels from the 80000331_2592x1108 pbm file to the 5x silent image. Coverage looks excellent, and I haven't noticed any out of place pixels yet.

Looked at 80000331_1808x727 in full screen mode, coverage looks excellent as well.

After taking some sample silent pics in full screen mode (1808x727) on my Eos M, I'm noticing some strange extra pixels that appear to be vertical rows of focus pixels. The vertical pixel rows aren't on the 1808x727 pixel map, any idea if these are actual focus pixels or fixed pattern noise? I tried many different color backgrounds, but I have been unable to produce a completely clean example file yet.
It took a lot of contrast and color channel manipulation to spot them.

Small crop preview:


Full size tiff uncropped (7mb, dirty example): https://drive.google.com/file/d/0B73VZPdpR0_hcnVobUhVb2Z0a0k/view?usp=sharing

[dfort]

I will need some good example footage that clearly shows where the current method struggles (and some dual ISO too).

Here are SL1/100D regular mv1080 and Dual ISO MLV. Sorry for the large size of the Dual ISO shot. I'll shoot some more samples when I get a chance. (The electricity was out since yesterday afternoon due to a big wind storm.)

https://www.dropbox.com/sh/un75uk8tym22fnn/AADyxww_M3PYCZLlQqI8AxV-a?dl=0

I'm noticing some strange extra pixels that appear to be vertical rows of focus pixels. The vertical pixel rows aren't on the 1808x727 pixel map, any idea if these are actual focus pixels or fixed pattern noise?

It sure looks like some of those are focus pixels. Could you upload the DNG?

[dfort]

Shot a test chart and uploaded it to that same dropbox shared folder.

I was hoping to be done shooting test charts for a while but they are good for showing problems. Note that the only focus pixels visible are on high contrast vertical lines.

100D mv1080

100D mv1080 Dual ISO

[AWPStar]

@dfort
Can you send me this(not dual iso) mlv file? I want to compare interpolations.

[Steve_FR]

@dfort

I took the simple silent image using simple .mlv, here is a link to the original .mlv file:
 https://drive.google.com/file/d/0B73VZPdpR0_hMHJ6S0VOV0t2UE0/view?usp=sharing

I tried to reproduce the same lighting situation with a simple silent .dng. Link to here:
https://drive.google.com/file/d/0B73VZPdpR0_hdU81ODVRR3BpQlk/view?usp=sharing

The vertical columns on the left and right sides are visible in both files with some fiddling. Let me know if you need any other files.

[dfort]

@dfort
Can you send me this(not dual iso) mlv file? I want to compare interpolations.

I uploaded the files in this dropbox folder. The Dual ISO shots are clearly marked. (Keep reading, praise for your work is coming up.)

https://www.dropbox.com/sh/un75uk8tym22fnn/AADyxww_M3PYCZLlQqI8AxV-a?dl=0

The vertical columns on the left and right sides are visible in both files with some fiddling. Let me know if you need any other files.

Wow, your finding is very significant. Thanks for this!

First of all, here's what I was able to pull out of your MLV:

Now the same frame with the focus pixel map file in place.

The pattern on your shot looks very much like a 700D sample that I had from a user when I first started this topic.

So much for trying to re-invent the wheel. Looks like foorgol (PinkDotRemover tool 650D)  and AWPStar (MLVProducer) got it right. AWPStar's maps are by far the cleanest and most accurate. Mind sharing how you did it?

I've got some more observations to share about the focus pixel patterns but first I need to get back to work updating the .fpm files for the EOSM / 650D / 700D. Thanks again Steve_FR!

[AWPStar]

i meant resolution measuring picture.

Mind sharing how you did it?

I didn't take pixels from the images. I generated them with some pattern.
Little bit later i will try to rebuild your maps.

[dfort]

i meant resolution measuring picture.

They are in that dropbox folder. If you want the chart itself, it is here:

http://www.graphics.cornell.edu/~westin/misc/res-chart.html

Little bit later i will try to rebuild your maps.

If you wait until I'm done rendering new focus pixel map files you'll find that they probably match yours very closely. Though as you can see I didn't plagiarize them.

[dmilligan]

MLVFS updated...

[AWPStar]

@dfort
I missed them. Thx!

@dmilligan
I think it can be better. i need more tests.
I'll let you know if I'll get the better results.

[dfort]

@AWPStar
Please let all of know if you can come up with better results, especially for Dual ISO.

@dmilligan
Huge improvement! I need to do some more Dual ISO testing but here are the results I'm getting with yesterday's update, commit 319d6e3.

SL1/100D mv1080


SL1/100D mv1080 Dual ISO


SL1/100D mv1080 - before yesterday's update


SL1/100D mv1080 - with yesterdays update - commit 319d6e3

There are still some focus pixels showing up on Dual ISO.

SL1/100D mv1080 Dual ISO - commit 319d6e3

I was able to reproduce the issue with the focus pixels showing up across the entire frame. They start showing up in underexposed frames that have the exposure stetting stretched in Adobe Camera Raw. Here's that image Steve_FR made with the focus pixels fixed. Compare it to the same shot posted in Reply #163. The image is stretched to the max so there's lots of noise but no dots.

Steve_FR discovered that the best way to get these pixels to show is to shoot a dark frame. Here's a nice shot of the back of Steve_FR's lens cap.

EOSM black frame

The revised fpm file seems to be working on those focus (?) pixels.

EOSM black frame with focus pixel map file

I'll post the updates for anyone wanting to run their own tests to my bitbucket download area. I'll also leave the previous versions up for a while in case anyone wants to do some regression tests. (a.k.a. before and after.)

[EDIT: I was wondering if maybe the SL1/100D has the same issue so I shot a black mlv and ran it with the focus pixel map in place and it looks like I've got more work to do on those focus pixel maps.]

100D mv1080 black frame with focus pixel fix

[AWPStar]

@dfort
Do you need some tool to work with maps? i can make standalone or integrate it in mvp.

[dfort]

Do you need some tool to work with maps?

Actually yes, there are several tools that I could use but I'm not sure how to describe them yet. Up until now all the progress has been on MLV but apparently there are still some situations where the original ML raw video may be preferred. I think that with the possible exception of zoom mode, it should be feasible to support raw with a little bit of user input. For example the user supplies the video mode (mv1080, mv720, mv1080crop, etc.) and by determining the frame size and looking up a list of crop values the focus pixel maps can be used with raw video. The thing is that there are more image size choices in raw than in MLV.

Hopefully the stuff I've been posting is helping developers improve their tools.

Part of the reason I started this topic was because of the problem removing focus pixels from dual iso footage and it is still an issue. I looked up various bad pixel interpolation techniques but the best material came from astronomy and it is way over my head. Way, way over. Kidding aside it seems to me that something that might work is taking the adjacent pixels in the x,y direction and averaging only the two closest matches. That means that it might take the two pixels to the immediate left and right or immediately above and below or if it turns out to be the best match, one +- horizontally and the second +- vertically. That just might work. One of the papers I looked at included the diagonal pixels but that might not be necessary.

Maybe processing that same area of the chart in various ways will help? All of these were from the same mv1080 Dual ISO MLV.

dual iso with fpm fix ACR



unprocessed dual iso without fpm fix dcraw -D



unprocessed dual iso with fpm fix dcraw -D



unprocessed dual iso with fpm fix dcraw -T

That green blip on top of the number 5 would not be there if the two pixels on x axis of the focus pixel were used for interpolation and the magenta in the number 7 caused by a diagonal pair of focus pixels should have been handled by one +x and one -y adjacent pixel on the lower left focus pixel and the horizontal pair on the upper right focus pixel. Is this making sense or have I been staring at focus pixels for too long?

[AWPStar]

As i understand, the problem is that at the contrast transitions, left and right pixels are different and algorithm should choose one of them.
Solution is:
1. Use threshold to not blend pixels.
2. Do something to make a chose.

I think we can compare difference at pixels above or below or both.


Or

OurPixel =((A1+A2)*B3)/(B1+B2)

[dmilligan]

A quick google search turned this up: http://harvestimaging.com/pubdocs/072_2003_SPIE_AdaptivePixelCor.pdf

As pointed out in that paper, the main issue is that red and blue pixels' neighbors of the same color are so far away (and if we are in a line skipping mode, they are even farther away).

This method sounds pretty robust, but it uses a rather large kernel (7x7), so it might be rather slow (esp. if the 100D really does have 100k focus pixels). It would also probably take me a while to understand their maths enough to actually implement it.

(NOTE: this method also has a means of correcting column defects, not just individual pixels, which might work as an alternate for the stripes correction, IDK)

A simpler possibility is to model the green channel (with a spline or something else) and then use that to interpolate the red or blue focus pixel. I should think the 8 nearest green pixels and the 4 nearest same color pixels would be enough for this (so we only need to read 12 pixels).

Dual ISO is still going to be a problem though. We might have to measure the exposure difference between line pairs and then offset the neighboring pixels that are at a different ISO by this much to allow them to still be used in the interpolation (problem with this: the pixels of the other ISO could be clipped or mostly noise).

One final possibility is to try and understand what the values of the focus pixels means, and then actually use their value in some way to help inform the interpolation. This could be helpful or not, IDK.

[dfort]

I also came across that paper but didn't bring it up because they were using a 7x7 area.

Something that got me thinking is how the Dual ISO issue seemed solved way back on Reply #22 so I thought I'd take another look. It was done with Danne's MLP so I created a dcraw badpixels file for the 100D, ran it on the chart image and it looked good. The end product was a ProRes 444 QuickTime file so I tried it again with just dcraw and here is what a TIFF looks like.

A closer look shows that the problem with the focus pixels is still there but much softer so it doesn't jump out. Perhaps this is because the default setting for cr2hdr that MLP is using has chroma smoothing turned on?

For comparison here is what the DNG file created with MLP looks like before going through dcraw -P dead_pixel_list.txt.

BTW--new updates on the bitbucket download area. New 100D focus pixel map files.

[dmilligan]

new interpolation method, works better, I think, in most cases, esp dual ISO

https://bitbucket.org/dmilligan/mlvfs/commits/1bc6f7f