Dealing with Focus Pixels in raw video

[dfort]

It also works on the one @papasapien posted on the MLVFS topic.

Here's one with the map file that is in MLVFS -- well, it doesn't even use the map file. It is stretched like crazy so the focus pixels pop:



and here's the new map that covers the entire sensor.



By the way, on this MLV file the full raw buffer size is also 1808x1189 and mlv_dump (crop_rec_4k) reports it as 1808x1188 for the reasons explained a few posts back.

[theBilalFakhouri]

@dfort
Really nice work man! it worked!
But unfortunately there are more pixels we couldn't see it. look like this:

[/s][/s]

Why we couldn't see them?
Because the pixels dots were behind the scenes in the white and green lines in the pattern
the edge only visible for us we can't see in the white/dark.

So I made the pattern flickering and I moved the camera slowly around the pattern to catch them but still too hard to see I am trying another patterns to make the view easier.

Is this really matter? I mean the most pixels was cleaned perfectly why we care for few Matrix Pixels
Because they were in my face and it will be in your face next time and in the people's faces that time so let's work on it.

Maybe it takes some time for me!

EDIT: I found it! Uploading the samples!
EDIT 2 : I was wrong this cold/hot pixels. SO PERFECT JOB @dfort thank you!

[dfort]

Here's a tip for posting YouTube videos. Change the https: to http: and enclose in youtube tags like this:

Code: [Select]

[youtube]http://youtu.be/7N5NgRBUWU8[/youtube]

Those remaining dots don't look like focus pixels. They may be hot/cold/dead pixels or even just dust on the sensor. Try Googling for help on how to deal with those kind of problems. If nothing works, you can add the coordinates of those stubborn pixels to the focus pixel map file. You need to take into account the Pan information.

mlv_dump -v Contrast\ Edge.MLV | less

Code: [Select]

Block: VIDF
  Offset: 0x00000600
  Number: 13
    Size: 1621504
    Time: 133.415000 ms
   Frame: #0000
    Crop: 72x118
     Pan: 72x118
   Space: 32

So if you find a problem pixel at say x=100, y=100 put "172 218" in the map file. It shouldn't matter if you put it at the start, end or middle of the file.

[bouncyball]

@dfort

Have you updated your script to generate this map?

[dfort]

Have you updated your script to generate this map?

Not yet -- I need to consult with my brain trust first, you and @Danne, about this. Of course anyone else who has an opinion about this should chime in.

The more I think about this stuff the more I'm leaning towards modifying the map files and script to cover the entire active full raw buffer area with the focus pixel pattern. One reason is because it will cover the 12-bit issue that @theBilalFakhouri brought to our attention. Another reason, and why I don't think it should just apply to 12-bit files, is because if only the center of the image has the pattern applied it might cause issues if we start to compare the areas that received the pixel pattern fix to those areas that didn't.

We're already doing that on the 100D and it doesn't seem to be a problem so how about doing the same for the other cameras?

One thing that I'm not too sure about is the 2-pass crop_rec map file for the EOSM but even there, covering the entire sensor in two passes shouldn't be a problem.

Your thoughts?

[Danne]

I'm all for your modification thinking. It will not affect the overall picture it seems.

[dfort]

Well if it does affect the image it will be the overall image not just the middle section of it.

[Danne]

Since you get more than decent results out of 100D focus pixel footage I don't think neighbouring pixels will be visible(distorting) on the image? Let's do som tests?

[dfort]

@Danne -- Let's see if I have this figured out, the "pattern A" cameras EOSM/650D/700D are the ones that sometimes report a different full raw buffer vertical resolution while the "pattern B" cameras, 100D and possibly EOSM2, always report the same height. Though I've got an mv720 map file that is 1808x727 and a crop_rec map file that's 1808x726, go figure.

The "pattern A" cameras are a problem on MLVFS because it means having to double up on some of the map files. On the new map files I'm taking into account the top and left out of bounds areas but am running the focus pixel pattern all the way to the right and bottom edges of the full raw buffer. That means that there is a difference between the 1808x1189 and 1808x1190 map files. Well, maybe it doesn't need to be this precise but better err on the side of being too precise.

I want the script to be able to create that precise full raw buffer map file that can be used in MLVFS. While we're at it, we should be able to verify these values:

[EDIT] I'm already seeing some issues on the EOSM. skip_left should be 72 and skip bottom default should be 0. I do see an unused area on the bottom of mv720 but it is 3 white pixels, not black as in the other out of bounds areas. One more problem--simple silent isn't working with the crop_rec_4k branch so I can't look at crop_rec.

src/raw.c

Code: [Select]

        #if defined(CONFIG_650D) || defined(CONFIG_EOSM) || defined(CONFIG_100D)
        #warning FIXME: are these values correct for 720p and crop modes?
        skip_top    = 28;
        skip_left   = 74;
        skip_right  = 0;
        skip_bottom = 6;
        #endif

        #ifdef CONFIG_700D
        skip_top    = 28;
        skip_left   = 72;
        skip_right  = 0;
        skip_bottom = zoom ? 0 : mv1080crop ? 0 : 4;
        #endif

I'm not too concerned about skip_right and skip_bottom for the focus pixel map files and script. Also, your heuristic code that takes into account differences in the raw buffer height is working fine with everything including the crop_rec_4k mlv_dump that rounds the height to an even number.

@bouncyball - I take it that all this also applies to fpmutil, right?

[bouncyball]

@bouncyball - I take it that all this also applies to fpmutil, right?

Yes it tolerates some vertical range and when exporting map with default name inserts correct/exact height of original MLV to this name.

Regarding interpolation of the unaffected areas of the image, I think the less we alter the image is better and this belief based on proof that in some very contrasty situations interpolation produces even more artifacts than there were in the beginning. Remember example with cage pattern MLV which been posted here.

Even 100D example from you: View from the window with pool, trees and buildings. On the left building there is a contrasty pattern on the windows and only MLVFS newer interpolation method handles it more or less correctly.

Another example 2pass maps (needed to prevent introducing post interpolation artifacts). They of course doing better job and not screwing up the whole image but who knows in what video mode and pattern and frequency of repeating this pattern could be introduced issues when adding support for all other now unsupported cameras and video modes.

regards
bb

Edit: quite a hard journey isn't it?, well... you started all this

[Danne]

No easy way around these pixels ey  . Yet.

[theBilalFakhouri]

Hey Magic Lantern world
This samples for each mod with another pattern: I called it PIXEL SCANNER pattern
https://drive.google.com/drive/folders/1X47f9foy6CdOiifndJSdFHyEcGWFDO2V

Because it's PERFECTLY shows hot/cold/dead Pixels maybe it will help for cameras have this issue heavily.
by doing script take this pixels out and save the pattern for each camera sensor for users then applying it to the raw video.
(Every camera will have it own hot/cold pixels and even if this the same cameras. from 700d to 700d will be difference.So maybe every user will have his own cold/pixel pattern if I am correct)

Tip: for mapping 12\11-8bit lossless Pixel Dots easily or to make sure all the pixels dots was mapped:
Shoot a raw video sample for each mod with new PIXEL SCANNER pattern (I played the pattern on 42 inch screen and I smoothed the focus a bit for decreases moire and show pixels dots a bit clearly). Zoom in the pattern if the screen was small.
Then process mlv file once for new map applied and once with no map applied. Import dngs sequences and put them above each other in any editing software now disable\enable the video on the top layer and see the difference before\after.

For red box only: I don't know if this artifacts or remaining pixels dots see below.

For artifacts problem here what I got: (This is 12bit lossless raw video)
New map applied:

I zoomed some areas:

Note: In the right part of sky this was overexposed so no artifacts on blown whites. And on the left-bottom corner in the horizontal white lines I don't know if this artifacts or remaining pixels dots. Here the original mlv make a tests:
https://drive.google.com/open?id=1hGJ6b-NH66mVFqtFIhnYzeTne9f83dmW

*A question: what is the process does MLVFS doing to remove this pixels dots?
By restoring the original pixels or changing the pixels dots to the safe pixels around pixels dots?

Affected cameras: EOSM, 100D, 600D, 650D, 700D, (and more?)
Method 1 - In Camera

I am somehow looking for this method.
I have question:
Is the pixels dots removing via Canon h.264 encoder or .cr2  .jpeg compression ? or the mlv_lite\silent_pic re enable somehow these pixels but the canon code disable it?
Maybe the encoder, reason: Enable proxy with mlv_lite you will have Clean H.264 and raw video with dots pixels in the same live view the same settings.
How does Framing mode in mlv_lite preview settings shows the dots pixels?
I am trying to find answer. If we, maybe we will see something like this:

Affected cameras: EOSM, 100D, 600D, 650D, 700D, (and more?)

[dfort]

@theBilalFakhouri - Those 12-bit files that you posted are proving to be quite useful. I was a bit skeptical of your moving PIXEL SCANNER pattern at first but it does a good job of bringing out those focus pixels.

Like I said before some of what you are seeing may be hot/dead/cold pixels. I started this topic because I wasn't satisfied with the dot removal tools that were available so I experimented with the "badpixels" feature in dcraw. As the name implies, it is a way to deal with those "bad" pixels.

In the old days we would check the camera's film gate for dust and hair before moving onto the next shot. Well dust and hair can still land on the sensor but we aren't checking for it any more because it is easy to fix it in post. I often thought it would be a nice to be able to easily map out problem pixels.

Before we start adding new features let's deal with the issues you brought up with those 12-bit files. I tried a few times to come up with a map file that will work for both mv720 and crop_rec when using MLVFS. Looks like it is finally working thanks to the 2pass map file @bouncyball came up with--and on those 12-bit files no less!



I've got some more experimenting to do with the map files but I'll keep pushing them to my ml-focus-pixels repository if anyone wants to try them out. I'm also make changes to the fpm.sh script which can be used to create the MLVFS map files.

For red box only: I don't know if this artifacts or remaining pixels dots...

What area are you zoomed in on on that red box? It is very difficult to make it out on YouTube. Your 12-bit MLV's are starting to look pretty clean over here with the exception of a few "bad" pixels. Tests on my 700D aren't showing anything like that so maybe you need to check your sensor?

I zoomed some areas:

Note: In the right part of sky this was overexposed so no artifacts on blown whites. And on the left-bottom corner in the horizontal white lines I don't know if this artifacts or remaining pixels dots. Here the original mlv make a tests:
(Uploading)

The way you edited that image is a bit disorienting but I can see some focus pixels in the sky that are a problem. Do you have a link to download it yet? If the MLV is too large you know that you can cut it down to just a few frames, right? [EDIT] Got it!

[theBilalFakhouri]

Did anyone notice something weird?
By changing bit-depth from 14bit lossless to 12bit lossless the whole pixels dots disappeared? (In whites & darks) and we get in very magic situation another new pattern in the edges only.
I don't know what the parameters had been changed to get this clean image, But what is the fu***** thing makes these pixels disappeared this way? White level? I don't know.
I really believe we are so close to forget pixels dots !
WHAT IS YOUR THOUGHTS MEN?

[dfort]

...we get in very magic situation another new pattern in the edges only....

You mean like this?



Nothing magical here, look back through the various posts on this topic and you'll see several examples like this. The above detail was done in MLVFS without the focus pixel map and processed in dcraw. The image needs some help with the blown out pink sky.

Here is what it looks like with the new pixel map in MLVFS, 2x2 chroma smooth and some playing around with Adobe Camera Raw.



Go ahead, click on the image and pixel peep. It is about as clean as any other camera in mv1080 mode except the 5D3 which is still the best because of the pixel binning methods used on that camera.

[EDIT] Pixel peeping:



I'm not seeing the artifacts you pointed out. How are you processing your DNG's?

Hey Magic Lantern world
*A question: what is the process does MLVFS doing to remove this pixels dots?
By restoring the original pixels or changing the pixels dots to the safe pixels around pixels dots?

That is best answered by the developer, @dmilligan. However, MLVFS is open source and you can go through the code to see how it works. You can find the fix_focus_pixels function in cs.c.

[bouncyball]

By changing bit-depth from 14bit lossless to 12bit lossless the whole pixels dots disappeared? (In whites & darks)

What do you mean by that? Changing 14bit->12bit? By camera settings or resample raw data manually? And how this related to disappearing of focus pixels?

Edit: In camera, changing lossless 14bits to 12/10bits just amplifies signal and white level lowers, the actual raw data remains 14bit. Manual resampling is a real bit depth conversion.

[a1ex]

@bb: in ML settings, see #424 and follow the links.

Short answer: it's Canon's imaging pipeline fixing these pixels; this only happens in 8...12-bit lossless modes (they use raw type 0x12 DEFCORRE, other modes use 0x10 CCD aka the first step in the pipeline).

[bouncyball]

@bb: in ML settings, see #424 and follow the links.

ok

[bouncyball]

@a1ex: Ah, yes I know that, thank you

[bouncyball]

I'm lost because all this talking is around 12bit lossless images he posted earlier and they have tons of focus pixels and magically in latest message they disappeared

[dfort]

@theBilalFakhouri and anyone else who want's a somewhat less technical explanation, check out Reply #285 of the Dealing with Focus Pixels in raw video topic and this pull request that I submitted and later declined.

Bascially, there are several choices of raw types. The one used in 8...12-bit lossless modes has at least a part of code that hides the focus pixels but they are still showing up on contrasty boundaries. We haven't been able to figure out how to eliminate the focus pixels completely in-camera but it does make sense that something like this is showing up because these camera models use contrast-detection autofocus. It looks a lot like the "focus peaking" feature to help focus manual lenses.

What is interesting is that up until now the focus pixel maps that we've been using haven't been able to eliminate these focus pixels but these new maps are working, at least in MLVFS. I'm testing in MLVFS because it is easy to swap out the map files in this app.

Will it work on other apps? I tried it on dcraw and it doesn't work even though the map file is lining up perfectly. This means that the app developers might have to re-work their focus pixel fixing code if they want to support 12-bit lossless on these cameras.

[dfort]

I'm lost because all this talking is around 12bit lossless images he posted earlier and they have tons of focus pixels and magically in latest message they disappeared

They didn't disappear, they just don't show up under certain conditions. We still need to map out their known locations.

I updated the script and most of the fpm files for MLVFS. Everything seems to be working properly for the 650D/700D/EOSM cameras even in 12-bit lossless except for the mv1080crop that @theBilalFakhouri supplied. Maybe it is one of those cases where everything has shifted?

Still needs testing on the 100D.

[theBilalFakhouri]

with the exception of a few "bad" pixels. Tests on my 700D aren't showing anything like that so maybe you need to check your sensor?

Yeah I checked the sensor today I saw some dots on it, So the pattern is show perfectly the dust on the sensor

I'm not seeing the artifacts you pointed out. How are you processing your DNG's?

I tried both mlv_dump_on_steriods and MLVFS with new map but without chroma smoothing. After applying 2x2 chroma smooth the image is perfect clean. I am using Davinci Resolve 14 also ACR for comparing results.
I don't like chroma smooth because it's produce some more aliasing as I know?

The image needs some help with the blown out pink sky.

Any solution for pink highlights?

 

Short answer: it's Canon's imaging pipeline fixing these pixels; this only happens in 8...12-bit lossless modes (they use raw type 0x12 DEFCORRE, other modes use 0x10 CCD aka the first step in the pipeline).

Looking Nice

[a1ex]

Re white level:

http://www.magiclantern.fm/forum/index.php?topic=16040.msg191131#msg191131
http://www.magiclantern.fm/forum/index.php?topic=16040.msg190644#msg190644
https://bitbucket.org/hudson/magic-lantern/commits/98539896ebd056e0a2128658aacd73b757a78b08
https://bitbucket.org/hudson/magic-lantern/commits/ab792b7c5137fceee4016bf9ef88767d031d27c5

Sample DNG: max = 4932 at ISO 100, 1080p24. However, that's just 1 data point out of nearly 100.

For existing 12-bit lossless footage from 700D at ISO 100 1080p(24?) with pink highlights (possibly other cameras/modes, no guarantees):

Code: [Select]

exiftool *.dng -WhiteLevel=4931


[dfort]

Played around with it and couldn't fix the pink highlights on the DNG's using exiftool but it cleaned up nicely with mlv_dump:

mlv_dump --dng


mlv_dump --dng --white-fix=4931


Processed through "dcraw -T" so no color correction, highlight recovery, etc. Now if we can only combine this with the focus pixel fixing that works so well in MLVFS.

Still to do - Figure out why the new maps aren't working with mv1080crop 12-bit lossless and need to run some tests on the 100D.