Author Topic: Dealing with Focus Pixels in raw video  (Read 168522 times)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3715
Re: Dealing with Focus Pixels in raw video
« Reply #675 on: May 30, 2019, 07:10:53 PM »
@2blackbar - I got into compiling Magic Lantern some years ago in order to turn on the chroma smoothing feature in raw2dng to remove the focus pixels. Back then the only way to turn on that feature was to compile it with that feature enabled. Chroma smoothing works -- sort of. It doesn't get rid of all of the focus pixels and some people say that it degrades the image though I can't see much of a problem there. However, with the focus pixel map files you are only working on the specific area that need to be processed so it should work without having to turn on chroma smoothing. Ok--sometimes there are some left over dots and chroma smoothing can iron those out.

As far as the map files not installing on MLV App running under Windows 7 -- I can't help you with that. I only have an old PC laptop running Windows 10 (barely running) and it is working fine. I'd suggest posting on the MLV App topic for help getting the map files working on Windows 7.

@Rewind - Wow, great post! Should be required reading for all developers working on MLV processing applications.

is this in mlv app yet

Seems like MLV App is using the same or very similar method to MLVFS. The cross method does look much better. Here is more information on Rewind's modified Adaptive interpolation algorithm and how Rewind applied it to PinkDotRemover tool (PDR).
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #676 on: June 11, 2019, 04:43:09 PM »
MLVApp has two different methods, select between them with the interpolation method buttons (1,2). Maybe @bouncyball can explain what mode uses what algorithm... around and median?! No exact idea ;)
5D2.212 | EOSM.202

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #677 on: June 11, 2019, 08:08:58 PM »
In order to evaluate how focus dots removal works in popular raw processors (MLVFS, MLV App, MLV Producer) I'm using this improvised test chart:


14 bit raw video (1736 x 976) shot on Canon 650D with mlv_lite module in non-crop 1080p mode.
PDR (Pink Dot Remover tool) with updated dot data and altered interpolation algorithm used as a reference guide.


A couple of observations and suggestons:

1. When shooting 14 bit raw in non-crop mode, focus pixels are located only in center part of the image.
No matter what AF points pattern used, what AF method selected in Canon menu or AF disabled, all the focus pixels are concentrated in central 290 pixels height area:


Therefore, assuming focus pixels all over the frame (like actual fpm tables does) we introduce color artifacts in areas which were clean originally. This behavior should be avoided:


Suggestion: let's update fpm tables the way they don't affect the top and bottm parts of the image.
Focus pixels should be treated only in 290 pix height central area in order not to ruin the originally clean image.
Above is applicable to 650D and may vary by camera models. If this is the case, may be we should detect camera model and limit the FP affected area accordingly, or even let user decide where to remove those pixels by dragging some selection area.

2. Interpolation algorithm works better if takes into account only horizontal and vertical neighbour pixels, avoiding diagonals.
Let's call it “cross” method for now. I've already mentioned this back in 2013, and this idea may sound strange at first, but here are some fresh examples. Judge for yourself:



While this is obviously exaggerated extreme test (although shooting let's say a book page in movies is not so rare), almost all real-world scenarios became a bit cleaner and seems more calm, when this “cross” method used.
So my second suggestion is: Let's introduce this interpolation method as an option in UI, so the user may decide which one is better in a given situation.
This applies to MLVFS and MLV App. MLV Producer uses its own method which is better and very close to what PDR does (is it the same?)

DNG's used for examples:
Original
Treated by MLV App / MLVFS
Treated by PDR

Modified interpolation algorithm explanation and code (java)

I tried out your code in MLVApp now, but I can't get it to work. When thinking about it... does this work on RAW data? I guess not... This is for debayered image data, right?


I converted your code to this C function, which is executed for each pixel in the map:
Code: [Select]
static inline void interpolate_rewind(uint16_t * image_data, int x, int y, int w, int h)
{
    if ((x < 3) || (x > (w - 4)) || (y < 3) || (y > (h - 4))) return;

    // 1. Retrieve vectors from 7x7 kernel
                // d[0] — vertical vector
                // d[1] — horizontal vector
                // index reference:
                //        paper     -3 -2 -1 0 +1 +2 +3
                //        actual     0  1  2    3  4  5
    int d[2][6] = {
        {image_data[x+((y-3)*w)], image_data[x+((y-2)*w)], image_data[x+((y-1)*w)], image_data[x+((y+1)*w)], image_data[x+((y+2)*w)], image_data[x+((y+3)*w)]},
        {image_data[x-3+(y*w)],   image_data[x-2+(y*w)],   image_data[x-1+(y*w)],   image_data[x+1+(y*w)],   image_data[x+2+(y*w)],   image_data[x+3+(y*w)]}
                    };

    // 2,3 — We don't need these stepse because of diagonal af dots arrangement

    // 4. Normalizing vectors
    // vertical norm.
    d[0][2] = d[0][1]+((d[0][2]-d[0][0])/2);
    d[0][3] = d[0][4]+((d[0][3]-d[0][5])/2);
    // horizontal norm.
    d[1][2] = d[1][1]+((d[1][2]-d[1][0])/2);
    d[1][3] = d[1][4]+((d[1][3]-d[1][5])/2);

    // 5. Deltas and Weights
    int dVert = ABS(d[0][2]-d[0][3]);
    int dHoriz = ABS(d[1][2]-d[1][3]);
    int Delta = dVert + dHoriz;

    float wVert = 1-((float) dVert / Delta);
    float wHoriz = 1-((float) dHoriz / Delta);

    // 6. Calculating new pixel value
    float newVal = wVert*((d[0][2]+d[0][3])/2) + wHoriz*((d[1][2]+d[1][3])/2);
    if( newVal > 65535.0f ) newVal = 65535.0f;
    if( newVal < 0.0f ) newVal = 0.0f;
    image_data[x+(y*w)] = (uint16_t)newVal;
}

Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?
5D2.212 | EOSM.202

Rewind

  • Senior
  • ****
  • Posts: 356
Re: Dealing with Focus Pixels in raw video
« Reply #678 on: June 11, 2019, 08:53:32 PM »
Quote
When thinking about it... does this work on RAW data? I guess not...
This method works on RAW data of course:
The Paper

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #679 on: June 11, 2019, 09:19:57 PM »
Thank you @Rewind. I commited the code to MLVApp repos and added 2 more buttons (interpolation method "3") for bad and focus pixels. It seems to work fine for 14bit footage. For some 10bit clips it looks wrong. No idea why.

Edit: okay... maybe has nothing to do with bitdepth. Does not work with most of my EOS M clips. Some testclips from other cams work. Strange.
5D2.212 | EOSM.202

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #680 on: June 12, 2019, 06:14:38 PM »
I found a quick temp fix for the interpolation method from @Rewind. See here. Whyever this helps, because after line 185 those values should already be 0..1. Maybe someone understand this and has a better idea, how to fix. But all artifacts are gone with this for my test clips. Would be cool, if someone else could compile and test. The result is worth it! Looks indeed better than the other two methods.
5D2.212 | EOSM.202

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5639
Re: Dealing with Focus Pixels in raw video
« Reply #681 on: June 12, 2019, 06:24:16 PM »
Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #682 on: June 12, 2019, 06:32:48 PM »
Ok, is this a boost to already created pixel maps? Or could it be used on any file containing focus pixels? Will check some more tonight.
This is just another (third) interpolation method to use the pixel maps. Especially in tree's it looks much better...
Methods in MLVApp: 1,2,3. 3 is the new one.
5D2.212 | EOSM.202

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5639
Re: Dealing with Focus Pixels in raw video
« Reply #683 on: June 12, 2019, 06:43:31 PM »
Impressive. Should be default if working all cases.

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #684 on: June 12, 2019, 07:46:36 PM »
...if working all cases.
That is the question to find out now ;)
5D2.212 | EOSM.202

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3715
Re: Dealing with Focus Pixels in raw video
« Reply #685 on: June 13, 2019, 07:15:55 PM »
Edit...: I see this code works nicely on 14bit RAW data... those dots happen for 10bit. Someone has a guess why?

I had a similar issue when I first started seeing 10/12bit lossless compression samples. Focus pixels started showing up in places where I never saw them before on 14bit files. This is why the new map files cover the entire image area and in some cases we need multiple pass map files because the pixel patterns are combinations of 3x3, 1x1 and (especially on the EOSM) 3X5 sampling patterns.

It probably doesn't have anything to do with what you are doing but it is worth mentioning that weird things happen with reduced bit depth lossless compression.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

kayman1021

  • New to the forum
  • *
  • Posts: 13
Re: Dealing with Focus Pixels in raw video
« Reply #686 on: June 26, 2019, 10:50:06 PM »
I've implemented vertical and horizontal spline interpolation in my monster program. Seems to work on dualiso dng too. My next step would be make it able to write back(or into copy) changes into dng.

Below picture is a difference between corrected and original values. No change is 0, on focus fixel coordinates it's something else.
Playing with exposure to make it visible to humans:

This was from 10bit anamorphic footage. Gradients everywhere. Maybe interpolation is the only sensible way to correct this.
EOS 100D, Xiaomi A1

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3715
Re: Dealing with Focus Pixels in raw video
« Reply #687 on: June 27, 2019, 01:52:32 AM »
Yes, it is visible.



Though it looks like you're missing some of the focus pixels. Note how they seem to fade out here and there--a.k.a. gradient.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

masc

  • Contributor
  • Hero Member
  • *****
  • Posts: 1217
Re: Dealing with Focus Pixels in raw video
« Reply #688 on: July 23, 2019, 04:31:09 PM »
Just because it is a eFAQ (extremely frequently asked question) I decided to bring the few clicks into a tutorial video: Howto install dforts focus pixel maps into MLVApp. I hope it helps.
5D2.212 | EOSM.202

yourboylloyd

  • Freshman
  • **
  • Posts: 56
  • Sigma 35 Art = best lens hands down. Fight me
Re: Dealing with Focus Pixels in raw video
« Reply #689 on: July 23, 2019, 07:03:32 PM »
Just because it is a eFAQ (extremely frequently asked question) I decided to bring the few clicks into a tutorial video: Howto install dforts focus pixel maps into MLVApp. I hope it helps.

It would be helpful if dfort could post this Youtube video on the FIRST POST of this thread. I think it wouldn't be so frequently asked if that was on the first post.

whitelight

  • Freshman
  • **
  • Posts: 50
Re: Dealing with Focus Pixels in raw video
« Reply #690 on: July 23, 2019, 10:51:44 PM »
Thanks masc, just what I was looking for