Author Topic: Dealing with Focus Pixels in raw video  (Read 162986 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3683
Re: Dealing with Focus Pixels in raw video
« Reply #675 on: May 30, 2019, 07:10:53 PM »
@2blackbar - I got into compiling Magic Lantern some years ago in order to turn on the chroma smoothing feature in raw2dng to remove the focus pixels. Back then the only way to turn on that feature was to compile it with that feature enabled. Chroma smoothing works -- sort of. It doesn't get rid of all of the focus pixels and some people say that it degrades the image though I can't see much of a problem there. However, with the focus pixel map files you are only working on the specific area that need to be processed so it should work without having to turn on chroma smoothing. Ok--sometimes there are some left over dots and chroma smoothing can iron those out.

As far as the map files not installing on MLV App running under Windows 7 -- I can't help you with that. I only have an old PC laptop running Windows 10 (barely running) and it is working fine. I'd suggest posting on the MLV App topic for help getting the map files working on Windows 7.

@Rewind - Wow, great post! Should be required reading for all developers working on MLV processing applications.

is this in mlv app yet

Seems like MLV App is using the same or very similar method to MLVFS. The cross method does look much better. Here is more information on Rewind's modified Adaptive interpolation algorithm and how Rewind applied it to PinkDotRemover tool (PDR).
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #676 on: June 11, 2019, 04:43:09 PM »
MLVApp has two different methods, select between them with the interpolation method buttons (1,2). Maybe @bouncyball can explain what mode uses what algorithm... around and median?! No exact idea ;)
5D2.212 | EOSM.202

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #677 on: June 11, 2019, 08:08:58 PM »
In order to evaluate how focus dots removal works in popular raw processors (MLVFS, MLV App, MLV Producer) I'm using this improvised test chart:


14 bit raw video (1736 x 976) shot on Canon 650D with mlv_lite module in non-crop 1080p mode.
PDR (Pink Dot Remover tool) with updated dot data and altered interpolation algorithm used as a reference guide.


A couple of observations and suggestons:

1. When shooting 14 bit raw in non-crop mode, focus pixels are located only in center part of the image.
No matter what AF points pattern used, what AF method selected in Canon menu or AF disabled, all the focus pixels are concentrated in central 290 pixels height area:


Therefore, assuming focus pixels all over the frame (like actual fpm tables does) we introduce color artifacts in areas which were clean originally. This behavior should be avoided:


Suggestion: let's update fpm tables the way they don't affect the top and bottm parts of the image.
Focus pixels should be treated only in 290 pix height central area in order not to ruin the originally clean image.
Above is applicable to 650D and may vary by camera models. If this is the case, may be we should detect camera model and limit the FP affected area accordingly, or even let user decide where to remove those pixels by dragging some selection area.

2. Interpolation algorithm works better if takes into account only horizontal and vertical neighbour pixels, avoiding diagonals.
Let's call it “cross” method for now. I've already mentioned this back in 2013, and this idea may sound strange at first, but here are some fresh examples. Judge for yourself:



While this is obviously exaggerated extreme test (although shooting let's say a book page in movies is not so rare), almost all real-world scenarios became a bit cleaner and seems more calm, when this “cross” method used.
So my second suggestion is: Let's introduce this interpolation method as an option in UI, so the user may decide which one is better in a given situation.
This applies to MLVFS and MLV App. MLV Producer uses its own method which is better and very close to what PDR does (is it the same?)

DNG's used for examples:
Original
Treated by MLV App / MLVFS
Treated by PDR

Modified interpolation algorithm explanation and code (java)

I tried out your code in MLVApp now, but I can't get it to work. When thinking about it... does this work on RAW data? I guess not... This is for debayered image data, right?


I converted your code to this C function, which is executed for each pixel in the map:
Code: [Select]
static inline void interpolate_rewind(uint16_t * image_data, int x, int y, int w, int h)
{
    if ((x < 3) || (x > (w - 4)) || (y < 3) || (y > (h - 4))) return;

    // 1. Retrieve vectors from 7x7 kernel
                // d[0] — vertical vector
                // d[1] — horizontal vector
                // index reference:
                //        paper     -3 -2 -1 0 +1 +2 +3
                //        actual     0  1  2    3  4  5
    int d[2][6] = {
        {image_data[x+((y-3)*w)], image_data[x+((y-2)*w)], image_data[x+((y-1)*w)], image_data[x+((y+1)*w)], image_data[x+((y+2)*w)], image_data[x+((y+3)*w)]},
        {image_data[x-3+(y*w)],   image_data[x-2+(y*w)],   image_data[x-1+(y*w)],   image_data[x+1+(y*w)],   image_data[x+2+(y*w)],   image_data[x+3+(y*w)]}
                    };

    // 2,3 — We don't need these stepse because of diagonal af dots arrangement

    // 4. Normalizing vectors
    // vertical norm.
    d[0][2] = d[0][1]+((d[0][2]-d[0][0])/2);
    d[0][3] = d[0][4]+((d[0][3]-d[0][5])/2);
    // horizontal norm.
    d[1][2] = d[1][1]+((d[1][2]-d[1][0])/2);
    d[1][3] = d[1][4]+((d[1][3]-d[1][5])/2);

    // 5. Deltas and Weights
    int dVert = ABS(d[0][2]-d[0][3]);
    int dHoriz = ABS(d[1][2]-d[1][3]);
    int Delta = dVert + dHoriz;

    float wVert = 1-((float) dVert / Delta);
    float wHoriz = 1-((float) dHoriz / Delta);

    // 6. Calculating new pixel value
    float newVal = wVert*((d[0][2]+d[0][3])/2) + wHoriz*((d[1][2]+d[1][3])/2);
    if( newVal > 65535.0f ) newVal = 65535.0f;
    if( newVal < 0.0f ) newVal = 0.0f;
    image_data[x+(y*w)] = (uint16_t)newVal;
}

Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?
5D2.212 | EOSM.202

Rewind

  • Senior
  • ****
  • Posts: 353
Re: Dealing with Focus Pixels in raw video
« Reply #678 on: June 11, 2019, 08:53:32 PM »
Quote
When thinking about it... does this work on RAW data? I guess not...
This method works on RAW data of course:
The Paper

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #679 on: June 11, 2019, 09:19:57 PM »
Thank you @Rewind. I commited the code to MLVApp repos and added 2 more buttons (interpolation method "3") for bad and focus pixels. It seems to work fine for 14bit footage. For some 10bit clips it looks wrong. No idea why.

Edit: okay... maybe has nothing to do with bitdepth. Does not work with most of my EOS M clips. Some testclips from other cams work. Strange.
5D2.212 | EOSM.202

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #680 on: June 12, 2019, 06:14:38 PM »
I found a quick temp fix for the interpolation method from @Rewind. See here. Whyever this helps, because after line 185 those values should already be 0..1. Maybe someone understand this and has a better idea, how to fix. But all artifacts are gone with this for my test clips. Would be cool, if someone else could compile and test. The result is worth it! Looks indeed better than the other two methods.
5D2.212 | EOSM.202

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5373
Re: Dealing with Focus Pixels in raw video
« Reply #681 on: June 12, 2019, 06:24:16 PM »
Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #682 on: June 12, 2019, 06:32:48 PM »
Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.
This is just another (third) interpolation method to use the pixel maps. Especially in tree's it looks much better...
Methods in MLVApp: 1,2,3. 3 is the new one.
5D2.212 | EOSM.202

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5373
Re: Dealing with Focus Pixels in raw video
« Reply #683 on: June 12, 2019, 06:43:31 PM »
Impressive. Should be default if working all cases.

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1067
Re: Dealing with Focus Pixels in raw video
« Reply #684 on: June 12, 2019, 07:46:36 PM »
...if working all cases.
That is the question to find out now ;)
5D2.212 | EOSM.202

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3683
Re: Dealing with Focus Pixels in raw video
« Reply #685 on: June 13, 2019, 07:15:55 PM »
Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?

I had a similar issue when I first started seeing 10/12bit lossless compression samples. Focus pixels started showing up in places where I never saw them before on 14bit files. This is why the new map files cover the entire image area and in some cases we need multiple pass map files because the pixel patterns are combinations of 3x3, 1x1 and (especially on the EOSM) 3X5 sampling patterns.

It probably doesn't have anything to do with what you are doing but it is worth mentioning that weird things happen with reduced bit depth lossless compression.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102