Dealing with Focus Pixels in raw video

[dfort]

@2blackbar - I got into compiling Magic Lantern some years ago in order to turn on the chroma smoothing feature in raw2dng to remove the focus pixels. Back then the only way to turn on that feature was to compile it with that feature enabled. Chroma smoothing works -- sort of. It doesn't get rid of all of the focus pixels and some people say that it degrades the image though I can't see much of a problem there. However, with the focus pixel map files you are only working on the specific area that need to be processed so it should work without having to turn on chroma smoothing. Ok--sometimes there are some left over dots and chroma smoothing can iron those out.

As far as the map files not installing on MLV App running under Windows 7 -- I can't help you with that. I only have an old PC laptop running Windows 10 (barely running) and it is working fine. I'd suggest posting on the MLV App topic for help getting the map files working on Windows 7.

@Rewind - Wow, great post! Should be required reading for all developers working on MLV processing applications.

is this in mlv app yet

Seems like MLV App is using the same or very similar method to MLVFS. The cross method does look much better. Here is more information on Rewind's modified Adaptive interpolation algorithm and how Rewind applied it to PinkDotRemover tool (PDR).

[masc]

MLVApp has two different methods, select between them with the interpolation method buttons (1,2). Maybe @bouncyball can explain what mode uses what algorithm... around and median?! No exact idea

[masc]

In order to evaluate how focus dots removal works in popular raw processors (MLVFS, MLV App, MLV Producer) I'm using this improvised test chart:


14 bit raw video (1736 x 976) shot on Canon 650D with mlv_lite module in non-crop 1080p mode.
PDR (Pink Dot Remover tool) with updated dot data and altered interpolation algorithm used as a reference guide.

A couple of observations and suggestons:

1. When shooting 14 bit raw in non-crop mode, focus pixels are located only in center part of the image.
No matter what AF points pattern used, what AF method selected in Canon menu or AF disabled, all the focus pixels are concentrated in central 290 pixels height area:


Therefore, assuming focus pixels all over the frame (like actual fpm tables does) we introduce color artifacts in areas which were clean originally. This behavior should be avoided:


Suggestion: let's update fpm tables the way they don't affect the top and bottm parts of the image.
Focus pixels should be treated only in 290 pix height central area in order not to ruin the originally clean image.
Above is applicable to 650D and may vary by camera models. If this is the case, may be we should detect camera model and limit the FP affected area accordingly, or even let user decide where to remove those pixels by dragging some selection area.

2. Interpolation algorithm works better if takes into account only horizontal and vertical neighbour pixels, avoiding diagonals.
Let's call it “cross” method for now. I've already mentioned this back in 2013, and this idea may sound strange at first, but here are some fresh examples. Judge for yourself:



While this is obviously exaggerated extreme test (although shooting let's say a book page in movies is not so rare), almost all real-world scenarios became a bit cleaner and seems more calm, when this “cross” method used.
So my second suggestion is: Let's introduce this interpolation method as an option in UI, so the user may decide which one is better in a given situation.
This applies to MLVFS and MLV App. MLV Producer uses its own method which is better and very close to what PDR does (is it the same?)

DNG's used for examples:
Original
Treated by MLV App / MLVFS
Treated by PDR

Modified interpolation algorithm explanation and code (java)

I tried out your code in MLVApp now, but I can't get it to work. When thinking about it... does this work on RAW data? I guess not... This is for debayered image data, right?


I converted your code to this C function, which is executed for each pixel in the map:

Code: [Select]

static inline void interpolate_rewind(uint16_t * image_data, int x, int y, int w, int h)
{
    if ((x < 3) || (x > (w - 4)) || (y < 3) || (y > (h - 4))) return;

    // 1. Retrieve vectors from 7x7 kernel
                // d[0] — vertical vector
                // d[1] — horizontal vector
                // index reference:
                //        paper     -3 -2 -1 0 +1 +2 +3
                //        actual     0  1  2    3  4  5
    int d[2][6] = {
        {image_data[x+((y-3)*w)], image_data[x+((y-2)*w)], image_data[x+((y-1)*w)], image_data[x+((y+1)*w)], image_data[x+((y+2)*w)], image_data[x+((y+3)*w)]},
        {image_data[x-3+(y*w)],   image_data[x-2+(y*w)],   image_data[x-1+(y*w)],   image_data[x+1+(y*w)],   image_data[x+2+(y*w)],   image_data[x+3+(y*w)]}
                    };

    // 2,3 — We don't need these stepse because of diagonal af dots arrangement

    // 4. Normalizing vectors
    // vertical norm.
    d[0][2] = d[0][1]+((d[0][2]-d[0][0])/2);
    d[0][3] = d[0][4]+((d[0][3]-d[0][5])/2);
    // horizontal norm.
    d[1][2] = d[1][1]+((d[1][2]-d[1][0])/2);
    d[1][3] = d[1][4]+((d[1][3]-d[1][5])/2);

    // 5. Deltas and Weights
    int dVert = ABS(d[0][2]-d[0][3]);
    int dHoriz = ABS(d[1][2]-d[1][3]);
    int Delta = dVert + dHoriz;

    float wVert = 1-((float) dVert / Delta);
    float wHoriz = 1-((float) dHoriz / Delta);

    // 6. Calculating new pixel value
    float newVal = wVert*((d[0][2]+d[0][3])/2) + wHoriz*((d[1][2]+d[1][3])/2);
    if( newVal > 65535.0f ) newVal = 65535.0f;
    if( newVal < 0.0f ) newVal = 0.0f;
    image_data[x+(y*w)] = (uint16_t)newVal;
}
Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?

[Rewind]

When thinking about it... does this work on RAW data? I guess not...

This method works on RAW data of course:
The Paper

[masc]

Thank you @Rewind. I commited the code to MLVApp repos and added 2 more buttons (interpolation method "3") for bad and focus pixels. It seems to work fine for 14bit footage. For some 10bit clips it looks wrong. No idea why.

Edit: okay... maybe has nothing to do with bitdepth. Does not work with most of my EOS M clips. Some testclips from other cams work. Strange.

[masc]

I found a quick temp fix for the interpolation method from @Rewind. See here. Whyever this helps, because after line 185 those values should already be 0..1. Maybe someone understand this and has a better idea, how to fix. But all artifacts are gone with this for my test clips. Would be cool, if someone else could compile and test. The result is worth it! Looks indeed better than the other two methods.

[Danne]

Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.

[masc]

Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.

This is just another (third) interpolation method to use the pixel maps. Especially in tree's it looks much better...
Methods in MLVApp: 1,2,3. 3 is the new one.

[Danne]

Impressive. Should be default if working all cases.

[masc]

...if working all cases.

That is the question to find out now

[dfort]

Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?

I had a similar issue when I first started seeing 10/12bit lossless compression samples. Focus pixels started showing up in places where I never saw them before on 14bit files. This is why the new map files cover the entire image area and in some cases we need multiple pass map files because the pixel patterns are combinations of 3x3, 1x1 and (especially on the EOSM) 3X5 sampling patterns.

It probably doesn't have anything to do with what you are doing but it is worth mentioning that weird things happen with reduced bit depth lossless compression.

[kayman1021]

I've implemented vertical and horizontal spline interpolation in my monster program. Seems to work on dualiso dng too. My next step would be make it able to write back(or into copy) changes into dng.

Below picture is a difference between corrected and original values. No change is 0, on focus fixel coordinates it's something else.
Playing with exposure to make it visible to humans:

This was from 10bit anamorphic footage. Gradients everywhere. Maybe interpolation is the only sensible way to correct this.

[dfort]

Yes, it is visible.



Though it looks like you're missing some of the focus pixels. Note how they seem to fade out here and there--a.k.a. gradient.

[masc]

Just because it is a eFAQ (extremely frequently asked question) I decided to bring the few clicks into a tutorial video: Howto install dforts focus pixel maps into MLVApp. I hope it helps.

[yourboylloyd]

Just because it is a eFAQ (extremely frequently asked question) I decided to bring the few clicks into a tutorial video: Howto install dforts focus pixel maps into MLVApp. I hope it helps.

It would be helpful if dfort could post this Youtube video on the FIRST POST of this thread. I think it wouldn't be so frequently asked if that was on the first post.

[whitelight]

Thanks masc, just what I was looking for 

[masc]

Hej dfort! Danne tuned the 4.5K anamorphic mode on EOS M - resolution is now slightly smaller and it should work with less corrupted frames. Unfortunatelly the old focus pixel map seems not to work with the new preset. Do you have a EOS M yourself, or do you need a MLV for updating the corresponding focus pixel map?

[dfort]

It would be helpful if dfort could post this Youtube video on the FIRST POST of this thread.

Done!

Unfortunatelly the old focus pixel map seems not to work with the new preset. Do you have a EOS M yourself, or do you need a MLV for updating the corresponding focus pixel map?

I've got several EOSM's but to make sure I'm working with the correct settings please upload and link to a short MLV file that clearly shows the focus pixels. I'll do my best to come up with a map file that works with it.

Like others on the forum it seems that my free time is rather limited lately so if anyone wants to help out -- please do so. All the instructions for creating focus pixel map files are in this topic and I do welcome pull requests on the Bitbucket project.

[2blackbar]

-delete

[masc]

Sorry for the late reply. It all changed a bit now. When I wrote the last post here, the 4.5K non rewire ana. mode had a little bug, that's why the focus pixel maps did not work for it. Danne fixed that and it works again. Now I work more with the 4K ana. rewire mode and it seems there is no focus pixel map yet. (right?) Here a sample:
https://www.dropbox.com/s/4h2qi606fsyhwnc/M11-1942.MLV?dl=0

[dfort]

Looks like the current maps are working well. I'd like to come up with a way to generate maps for every possible setting but it is like trying to herd cats.

[Danne]

Looks like the current maps are working well. I'd like to come up with a way to generate maps for every possible setting but it is like trying to herd cats.

Any ideas to how? How about:
- automated detection of pattern then apply accordingly. Maybe dig in to chroma smooth code and see what calculations are doing in there.
- understand how raw stream works and what happens when reg 6804 is changed. Look into crop_re.c and raw.c code and try to match changes with focus pixel code automated..
- refine tools and automation building map files itself. I tried but failed every time even following your provided steps.

[masc]

Looks like the current maps are working well. I'd like to come up with a way to generate maps for every possible setting but it is like trying to herd cats.

Can't completely agree... I mean the maps are working well, yes... but for this MLV file there is no map. At least I installed all maps from your repos and MVLApp won't select one of them. So the focus pixels won't be removed (completely) - it just looks as if the old coded pixel removal removes some of the focus pixels. In latest revision (to be compiled) there is a small dialog showing which maps are installed and which map is used (if a map is used).

For the MLV I uploaded, no "green line" is shown (-> no map used).

[dfort]

It was rather difficult so see the focus pixels on that sample but after some saturation, vibrance, contrast, etc. tweaking they showed up just enough to see that this is very similar to another setting that already has a map file.

What is happening is that we already had a setting with an 1872x1951 full raw buffer as @masc pointed out. Well this one is 1872x1920 so even though this is somewhat close the app isn't accepting it because it is outside the size tolerance. I'm not really sure what the tolerance is on MLV App but on on MLVFS there is no tolerance--it has to be an exact match.

I made a new map file for this latest sample. Please check it out and report back:

https://bitbucket.org/daniel_fort/ml-focus-pixels/commits/58b7aea0d9f5bd7b66bf31ee75ec8c28a7830924

[Danne]

Great dfort. I recently decreased height in raw.c and that seemed to help to the extent this mode became corruption free. At least I couldn't reproduce any pink frames yet...