Dealing with Focus Pixels in raw video

[Danne]

Added latest mlv_dump on steroids along with the later fpm files into Switch for testing. Todo: add the actual switch into menu.
Is fpm.sh updated?
fpmutil status?

[bouncyball]

fpmutil status?

fpmutil is lagging, not updated with new pattern generators yet (so does mlv_dump os). Work is in progress, some other changes have to be done before adding latest generators.

[bouncyball]

Updated the script and the MLVFS fpm files to account for possible 8-pixel shifting in mv1080crop and zoom modes (1:1 sampling) on the EOSM/650D/700D cameras.

As I understand in zoom mode image area (hence current viewable raw buffer region) can be shifted by pressing some buttons on the camera (or because of digital dolly mode or because some lua script).

This can be done on purpose or accidentally in any direction by multiples of 8.

If this is the case, don't you think this is the reason of existance of those shifted buffer MLVs?

[dfort]

When you pan around in 5x zoom mode if you get close to the edge of the full raw buffer the buffer jumps to a different position on the sensor. I was able to find 21 possible positions and these are much larger jumps than 8-pixels. We can't determine when it makes the jump or the position of the raw buffer on the sensor so I included all the possible focus pixel locations in the zoom map file. I have referred to this image several times because it clearly shows how the buffer moves around the sensor:



@dmilligan and @a1ex posted some more technical explanations well worth reading. In fact now that I reviewed them I'm not sure why I included Crop in my script, only Pan is needed to line up the focus pixel maps. [EDIT] Strike that, reverse it -- Crop is needed.

There's another plot twist -- the 8-pixel offset is happening in both zoom and mv1080crop modes. Both the 14-bit lossless EOSM mv1080crop MLV file we looked at back in September and the 14-bit lossless 700D zoom MLV from @theBilalFakhouri have the 8-pixel offset.

I shot tests on my cameras in various settings, matching the image size and video modes of those sample files and came up with different crop and pan values. On mv1080crop mode they should match.

"Normal" EOSM mv1080crop 1600x640

Code: [Select]

    Crop: 176x222
     Pan: 172x223

"Broken" EOSM 14bit lossless mv1080crop 1600x640

Code: [Select]

    Crop: 176x220
     Pan: 170x220

I'm not sure what is going on but is it fixable? Here is the zoom mode test from @theBilalFakhouri:

Before


After


The mv1080crop example has more color variations so let's use that to see what is going on.



Closer view:


If we take the current focus pixel pattern and repeat it every 8-pixels on the x axis it will cover those shifted focus pixels:



But wait! There's more! We need to make a second pass sifted one more pixel to cover the possibility of yet another issue.



This is getting pretty extreme but it doesn't seem to have any adverse effects on the image. Note that the noise pretty much matches what we see between the focus pixels.



Ok--let's back up and see at how it looks after removing the focus pixels.




This is working fine in MLVFS but there are some left over focus pixels if we use the same map file with dcraw.



It is subtle so here's an even closer look:



I have no idea why only those few rows of pixels are still showing up. The map file is definitely covering them.

[bouncyball]

It seems like a bug in panning mode.

I have no idea why only those few rows of pixels are still showing up. The map file is definitely covering them.

They could be secondary artifacts after interpolation of adjacent or nearly adjacent pixels. That is why we are using 2pass mode for some flooded by focus dots maps. 'dcraw' does it differently.

[dfort]

It seems like a bug in panning mode.

The terminology could get confusing. As you know the panning information we're talking about here is the position of the upper left corner of the image to the full raw buffer. When I wrote the script I wanted to cover all the possible ways it could be used including legacy RAW and DNG which don't have that metadata. What the script does is to assume the image is centered on the buffer:

Code: [Select]

./fpm.sh -vm mv1080crop EOSM_14bit_losless_broken_raw_buffer_000000.dng

Camera:               EOSM
Camera Model:         80000331
Resolution:           1600x640
Video Mode:           mv1080crop
Full Raw Buffer Size: 1872x10**
Pan:                  170x221
Crop:                 176x220
Output File:          EOSM_mv1080crop_1600x640.txt
File Format:          dcraw

Not exactly the same as the original MLV but surprisingly it worked.

They could be secondary artifacts after interpolation of adjacent or nearly adjacent pixels. That is why we are using 2pass mode for some flooded by focus dots maps. 'dcraw' does it differently.

That's exactly what is going on. It was probably happening for a while but we haven't noticed it until now. Let's take a really close look at the problem pixels.

No focus pixel map


With focus pixel map


Left over focus pixel after processing in dcraw


What threw me off was that the leftover pixels were not on the original but they were being covered by the map file


My theory is that dcraw is interpolating all the surrounding pixels including that pixel that doesn't need fixing. We can't just remove them from the map file because focus pixels can show up in these locations. Why is it happening on just that row and not on other adjacent pixels? I don't really know but I'll bet that if that row is processed on a second pass it will work in dcraw.

Of course we're already doing two passes on this so maybe it needs third and fourth passes? It is just a matter of how the coordinates are sorted on the map file.

[dfort]

Quick update -- doing a second pass on the row of pixels that dcraw left behind worked.

Before


After


This isn't needed on the MLVFS maps but they should probably be updated so everything is consistant.

Of course the more testing the better, especially on the 100D.

[Danne]

Looking good!

[bouncyball]

'fpmutil' updated:

1. supports new (latest) pattern generators from @fdort. Activated by '-l' or '--lossless' option or detected automatically if MLV file specified.
2. supports exporting of separate '.pbm' images for each pass automatically. Can be overridden by '-1' or '--one-pass-pbm' option to save 1 pass all in one (old style) PBM.

The long help is here
Commits are here

For now, does not support loading back all separate pass PBM files to convert to one FPM again.

regards
bb

[Danne]

Nice B! Will take a look when I can. Too much work atm...

[Danne]

@Bouncyball.
Tried the latest fpmutil to see if I could get rid of the focus pixels on this file:
https://drive.google.com/open?id=1N7NOWzcbCA5Ltd_hD1ta8rCgFvCsNGez
Coming from here:
http://www.magiclantern.fm/forum/index.php?topic=16054.msg195331#msg195331

Result:
applied -l(lossless) and -m croprec -c 700D and get the following file:
#FPM 80000326 1808 726 1 2 -- fpmutil v0.7
I´m using the latest version of mlv_dump on steroids tested in Switch and outside. All pixels still there it seems.

Tested in MLVFS with dfort´s lates fpm files and all pixels are gone!

[bouncyball]

Thanks Danne, I'll take a look.

[bouncyball]

@Danne

Fixed. Now seems it's working as expected. Try please.

Additionally implemented new feature to specify more than one input '.pbm' files (passes) separated by whitespace on the command line. This will output the combined multipass '.fpm'.

bb

[Danne]

Works perfectly! Worked straight up in Switch without any further script tweaks. Tested on a bunch of files from *Bilal* and they were all cleaned:
1736x688.MLV
1736x688(Stretch1152).MLV
1736x1152.MLV
1800x1008.MLV
2520x1072.MLV
mv720.MLV

[bouncyball]

Cool

[dfort]

What is "outside"?

I'm using fpmutil to look at the multipass map files and I might be starting to understand a little bit of why these "phantom" focus pixels are showing up in dcraw. I'm not really familiar with the way dcraw works thought it is certainly no secret. It looks like the focus pixels (bad_pixels in dcraw) are fixed in the same order that we are creating the map files--starting from the top left (0,0 position) and reading left to right one line at a time. No big surprise there but are the fixes happening before or after debayering? I'm not entirely sure but isn't part of the process to blend surrounding pixels? If that's true then reading from left to right, top to bottom, the surrounding pixels are blended into this pixel:



Thus leaving behind a "phantom" pixel that wasn't on the original image:



This can be resolved using a multi-pass map file. That pixel is processed after the row below it so the area sampled is not contaminated with pink. Makes sense but what about this pixel?



Using a multi-pass map that top row is processed last so why isn't it picking up the pink? Is it because adjacent focus pixels are saved in complimentary colors that cancel each other out? IDK.



The mystery remains unsolved.

I keep seeing spots before my eyes.
Have you seen a doctor?
No, just spots.

Continuing with the dcraw tests, there was still one last row of those "phantom" pixels near the bottom of the frame. On MLVFS @dmilligan is doing a special process on the focus pixels near the edge of the frame. I have been recommending adding some chroma smoothing to get those last few stubborn focus pixels. I couldn't find a chroma smoothing option in dcraw but it does have this:

-m number_of_passes
        After interpolation, clean up color artifacts by repeatedly applying a 3x3 median filter to the R-G and B-G channels.

Running two passes (-m 2) cleaned up those phantom pixels. At one point we were running chroma smoothing to eliminate focus pixels so I was wondering if maybe this can be used the same way in dcraw. Just to see what would happen I ran it ten times (-m 10) and it worked! Well, almost. There were still focus pixels showing up on the edges of the frame. Some of those pixels weren't even in the original so let's just say they were very bright "phantom" pixels.

I'm getting a lot of mileage on this 14-bit lossless MLV with the shifted pixels. I've been throwing more and more coordinates into the map file wondering just how far we can take it before it breaks. This is what the 4-pass mv1080crop focus pixel map looks like:



It doesn't seem to be hurting dcraw:



MLVFS looks like it might have a problem with it.



So I thought I'd try the old single pass map file. Along the way I discovered something interesting--recall this:

"Normal" EOSM mv1080crop 1600x640

Code: [Select]

    Crop: 176x222
     Pan: 172x223

"Broken" EOSM 14bit lossless mv1080crop 1600x640

Code: [Select]

    Crop: 176x220
     Pan: 170x220


Shifting the old map file 2-pixels did the trick.



However, the results look pretty much the same as the mega map file. Maybe even a little worse? I'm pretty sure I haven't mixed them up.



It seems logical that we should be mapping only the fewest locations where focus pixels are showing up. However, it doesn't seem to be harming the image if we map all possible locations. This is good news for the map files we're making for MLVFS because we don't need to swap out map files--I believe Switch is doing this for the crop_rec map files.

While I was working on this post @bouncyball has been pushing updates so the testing continues!

[Danne]

This thread is the shit! Great posts @dfort.

[bouncyball]

I think that in order to avoid contamination with unusually bright/dark or different color, in the interpolation code, all the surrounding pixels must be double checked and suspects gotta be excluded from averaging. This will definitely help on very contasty edges too.

Those algos already do some checking (btw doing this on raw (bayer) data is a bigger PITA than doing on debayered RGB image). I've dug in this code long time ago. Have to revisit it if I have more time... no promises though

[dfort]

When I started this topic I thought it would be challenging but at some point the focus pixel issues would be completely resolved. There seems to be no end in sight.

I've been running acid tests on the new focus pixel maps with all the problem MLV files I could find and they seem to be working fine with MLVFS. Here's an oldie that @DeafEyeJedi shot a while back.

100D mv1080 Dual ISO before focus pixel fixing


No need to pixel peep to find those focus pixels! Looks pretty good after fixing the focus pixels.



However, I wouldn't recommend Dual ISO in movie files because of the aliasing issues. Check out the far bleachers and the grid lines on the field.

If you want HDR video check out the work @Danne is doing on the HDR script keep original framerate topic.

Hum--wonder if the Dual ISO aliasing can be smoothed out with the methods discussed in these topics:

Experiment - reducing aliasing in post using optical flow

Reducing aliasing in post using enfuse/hugin align_image_stack

[Danne]

I could need a little more infor about this @Bouncyball:

Additionally implemented new feature to specify more than one input '.pbm' files (passes) separated by whitespace on the command line. This will output the combined multipass '.fpm'.

How is fpmutil working atm? Multipasses as default? I see a command line option for selecting 1pass:

Code: [Select]

-1|--one-pass-pbm         export multi pass '.fpm' as one pass '.pbm'
One more thing. As I'm doing tests I often encounter some bit corruption like pink worm strings in highlights.. Solution is always to select -b 14 for upsizing to 14bit(mlv_dump on steroids). Question is if this should default like in original mlv_dump? Or else some heavy recoding of other functions like vs, cs etc to account for 10/12bit files?

[dfort]

Splitting multi-pass map files into separate image files is great for debugging (dedotting?) problem files. Put each of the map files on separate Photoshop layers and you can figure which focus pixels each pass is hitting.

I've got another idea, one that goes all the way back to the first post on this topic:

Here is what I ended up doing to get rid of just one focus pixel--and the diagonal secondary pixel.

Code: [Select]

1247 121 0
1248 120 0
1248 121 0
1248 125 0
1248 126 0
1248 122 0
1248 123 0
1248 124 0
1248 126 0
1248 122 0
1248 120 0
1248 124 0

Note that I hit the diagonal pixel just once but had to work top and bottom to middle and hit the same pixels several times in different sequences. Think of it as stippling a brush to spot a print--in photography school I spent many hours spotting prints. (By the way, this a map of all the focus pixels for the 650D that was used on the PinkDotRemover tool.) [EDIT] Broken Link

Here is the result. I circled the area where the hot pixel was located and yes, it looked just like those other ones before running the dng through dcraw.

Hang in there, this is going to be a little abstract. When we used to spot prints using Spotone it wasn't just one hit of the brush on the print and you're done. You need to hit the same spot repeatedly in order to blend it in to the surrounding area and match the grain pattern. In other words you need to stipple out the spots. Let's say that we have a hot pixel that is ruining all of our footage. How about making a "Spotone" pass by simply adding those coordinates at the end of the fpm file? That last pass shouldn't be sorted and in effect it is doing many passes over the same area until the problem pixel disappears. This is a little hard to do manually but in a GUI each time you hit the brush on a pixel, it should build the map file in the order the pixels were selected. That would simulate the process of spotting a print.

Does this make sense?

In effect it might be doing what the dcraw -m option does but only in the areas that are needed in a predetermined order.

[bouncyball]

How is fpmutil working atm? Multipasses as default?

It concerns only PBM image map output. In some cases, like crop_rec, the map represents combined coordinates of mv720 and mv1080, we call it 2pass map. If output is FPM, exported file is one, if PBM by default will be exported as many PBM files as passes are. When just converting FPM to PBM by default each pass is one PBM. If '-1' specified there will be exported only one combined PBM. In case PBM to FPM convertion you can supply several input PBM files on the command line and they will be combined in one multipass map and exported as FPM.

One more thing. As I'm doing tests I often encounter some bit corruption like pink worm strings in highlights.. Solution is always to select -b 14 for upsizing to 14bit(mlv_dump on steroids). Question is if this should default like in original mlv_dump? Or else some heavy recoding of other functions like vs, cs etc to account for 10/12bit files?

Not every 10/12 bit footage suffers, but yes all processing algorithms (written mostly by a1ex, and modified by dmilligan) been optimized or even specifically written for 14bit raw (black, white, histogram, raw2ev, ev2raw). I modified them as far as I understand the idea of this particular processing method. Now, on 10/12bit stuff, it works better then original code, but it is definitely not the best option. Would be better if author(s) could help to tweak the code further , or even write some newer, better one .

[bouncyball]

I've got another idea, one that goes all the way back to the first post on this topic:
...
wasn't just one hit of the brush on the print and you're done. You need to hit the same spot repeatedly in order to blend it in to the surrounding area and match the grain pattern.
...
area until the problem pixel disappears. This is a little hard to do manually but in a GUI each time you hit the brush on a pixel, it should build the map file in the order the pixels were selected. That would simulate the process of spotting a print.

Does this make sense?

If those annoying, only manually removable pixels are a few, then yes, this makes sense
Interpolating bad pixel and then adjacent pixels several times definitely will give pleasing result if some very bright/dark or very different color pixel will not contaminate the average.

bb

[dfort]

My first success at removing focus pixels was to use the chroma smoothing in raw2dng. In fact I setup my first ML development environment specifically to get this feature by uncommenting just this one line:

Code: [Select]

/* useful to clean pink dots, may also help with color aliasing, but it's best turned off if you don't have these problems */
//~ #define CHROMA_SMOOTH

I sometimes still use chroma smoothing to get rid of some stubborn focus pixels. I recently discovered a similar feature in dcraw with the '-m' option. So I thought I'd see how well these methods work without using the focus pixel map files and it looks like I found the same bug in mlv_dump, MLVFS and dcraw.

Let's start with dcraw. The you can look at the code on dcoffin's website -- search for the median_filter() function to see how it works.

I ran these tests with mlv_dump and MLVFS by removing the focus pixel map files and only using the chroma smoothing options with the same results.

This is what it looks like using mlv_dump on steroids with the --no-fixfp option -- thanks for this @bouncyball



Note that there's a row of focus pixels right on the bottom edge of the image. Now let's do two passes with the median filter '-m 2'



Much better but you can still see them. I found that '-m 4' was the limit, anything more didn't help and I tried it all the way to '-m 100'



Note that the focus pixels on the bottom just won't go away.

Now let's try it again only this time using mlv_dump with some chroma smoothing. I got the same results using cs2x2, cs3x3 and cs5x5:



With dcraw we could shave off one pixel from the bottom of the image but with mlv_dump we would have to remove the bottom five pixels. In addition, there is another problem with some focus pixels that were close to the side of the image.

cs2x2 and cs3x3 gave the same results:



cs5x5 also showed a green focus pixel:



Is there anything that can be done to improve how chroma smoothing works on the edges of the frame? The chroma smoothing code is a bit over my head.

[dfort]

Hey @theBilalFakhouri - @bouncyball updated mlv_dump on steroids with the latest focus pixel maps so going back to your shot, here it is processed through Adobe Camera Raw maxing out on the highlight recovery, clarity, vibrance and saturation. That should bring out any remaining focus pixels.

mlv_dump --dng --white-fix=4931


There is an area that still shows some focus pixel artifacts:



I doubt they would be a problem unless you stretch the image as far as I have. Adding chroma smoothing eliminates it and I don't see any other issues introduced by chroma smoothing.

mlv_dump --dng --white-fix=4931 --cs3x3


Click on the images to see them full size.

I take it you didn't use a VAF filter on your 700D. There are a few post production tricks you can use to get rid of the aliasing.

Let's see what happens.

Hum--is there a problem with the youtube tags on the forum? Here's the link:

Yep, some aliasing issues but that has nothing to do with the focus pixels.