Dealing with Focus Pixels in raw video

[dfort]

The way you get into the various modes are:

mv1080 - Canon menu at 1920x1080 -- raw buffer size is 1808x1190 - 24, 25, 29.97fps - EOSM is unique in that it won't go into this mode unless it is recording H.264 video.

mv720 - Canon menu at 1280x720 -- raw buffer size is 1808x727 - 50, 59.94fps - EOSM is unique because it defaults to 29.97fps.

mv1080crop - Canon menu at 1920x1080, ML Movie crop mode -- raw buffer size is 1872x1060 - 24, 25, 29.97fps

zoom - Canon menu at 1920x1080, press magnify button to get to 5x LiveView -- raw buffer size is 2592x1108 - 24, 25, 29.97fps. Tricky on the EOSM because the magnifying glass button is in one of the LiveView layouts. You can actually record at about 2496x1078 in this mode but limited by the SD card hardware.

crop_rec - Canon menu at 1280x720, crop_rec module (from experimental builds) set to 3x3 720p -- raw buffer size is 1808x727 - 50, 59.94fps - EOSM at 29.97fps.

I believe you wanted a blank white wall no clipping, right? Here are silent DNG's from the 700D:

https://www.dropbox.com/sh/fvjgwdy9yyeycic/AAAF44SlZ7S642WQGRZJX5Dya?dl=0

You might want to shoot a black frame on your EOSM and push it as far as you can in ACR. It will be noisy but you'll see a different focus pixel pattern.

I'm on a Mac and have taken a look at your program. I've been a bit too busy on several projects lately to run any tests on it.

[EDIT] I never had much luck with digital dolly on the EOSM. It seems to work best on the 5D3 though I haven't played around with it on that camera. As far as zoom mode and digital dolly, there are 21 raw buffer positions on the sensor and no indication of which one is active so it is a challenge figuring out where the focus pixels are located. What I did might be overkill but it seems to be working for the few people that figured out how to use zoom mode on the EOSM/100D/650D/700D.

[martinhering]

Thanks for the info. I could get some footage on the EOS-M in zoom mode and and made some new pixel maps:
http://fs.footageformac.com/index.php?dir=focus_pixels/pix_maps

For the mode auto-detection I did this to cope with the crop rect in zoom mode:

Code: [Select]

        FTRawImageFocusPixelsType type = kFTRawImageFocusPixelsTypeEOSM;

        NSInteger rawBufWidth = (block.cropPosX > 0) ? 80 + raw_info->width + (block.cropPosX-80)*2 : raw_info->width;
        NSInteger rawBufWidthZoomRecording = (block.cropPosX > 0) ? 128 + raw_info->width + (block.cropPosX)*2 : raw_info->width;
        if (rawBufWidth == 1808) {
            type |= kFTRawImageFocusPixelsType1808x728;
        }
        else if (rawBufWidth == 1872) {
            type |= kFTRawImageFocusPixelsType1872x1060;
        }
        else if (rawBufWidth == 2592 || rawBufWidthZoomRecording == 2592) {
            type |= kFTRawImageFocusPixelsType2592x1108;
        }

Seems to work.

I could not check your silent pictures, since I don't have a DNG decoder yet. I always take the silent pictures in MLV format. Can you do some MLV silent pictures with your 700D in all modes? Thank you.

I could not get mv1080 mode working on EOS-M. Recording in H.264 doesn't make sense obviously. And silent pictures either return mv720 or mv1080crop mode buffers.

[dfort]

Those new pixel maps are not covering all the possible focus pixels. Believe me, I've been there.

I could not check your silent pictures, since I don't have a DNG decoder yet. I always take the silent pictures in MLV format. Can you do some MLV silent pictures with your 700D in all modes? Thank you.

Sure -- uploaded to that same dropbox folder. Get dcraw -- we're big on open source software here.

I could not get mv1080 mode working on EOS-M. Recording in H.264 doesn't make sense obviously. And silent pictures either return mv720 or mv1080crop mode buffers.

Who says this makes sense? You can shoot silent stills while recording H.264. It might help if you use this branch:

https://bitbucket.org/hudson/magic-lantern/branch/raw-h264-proxy

[martinhering]

Those new pixel maps are not covering all the possible focus pixels. Believe me, I've been there.

OK, since it is hard to get a good shot that covers all focus pixels, I deal with that once I have footage that needs better pixel maps. So far, I am satisfied with the result.

Sure -- uploaded to that same dropbox folder.

These helped a lot. Thank you. I am using the same pixel maps for 650D, 700D and EOS-M now. Seems to work.
http://fs.footageformac.com/index.php?dir=focus_pixels/pix_maps

The crop-rec module gives the same raw buffer size than mv720 mode, I think, so it's kinda hard to come up with an auto-detection. But since the module is currently experimental, I am postponing this issue as well.

[dfort]

OK, since it is hard to get a good shot that covers all focus pixels, I deal with that once I have footage that needs better pixel maps. So far, I am satisfied with the result.

At some point that is going to bite you. Here is what happened to me--I too was satisfied that I got all the focus pixels covered when @Steve_FR sent in this shot:



Applying the focus pixel map that I was satisfied with at the time resulted in this:



After adjusting my focus pixel map the focus pixels were gone:



Ok--this shot won't make it into the "Year in Pictures" special issue but note that it did remove the focus pixels without affecting the overall texture of the sensor noise. The brightness shift was because I was playing around with the settings to make sure I got the areas that weren't covered by the old map.

Separating focus pixels into red and blue maps is intriguing. Does it work in all cases? Reason I ask is because it seems that the pixels change colors depending on the lighting. Here's an example from the edge of a frame from the 100D. Note that this is also showing focus pixels outside of the area that you are covering with your maps.

I am using the same pixel maps for 650D, 700D and EOS-M now. Seems to work.

Yes, that's all covered in this topic and in my ML Focus Pixels bitbucket repository. Theoretically the 100D shares the same pixel map as the EOSM2 because they use the same sensor but the M2 hasn't been ported to ML and probably never will because it was not a very popular camera and distribution was mostly limited to the Asian market. Ok, maybe that was too much irrelevant information--there are only two main focus pixel patterns we need to deal with, one I call the A pattern for the 650D/700D/EOSM and and the other I call the B pattern for the 100D and possibly EOSM2. However, there have been reports of other cameras showing focus pixels in certain situations. Then again maybe there is a way to "fix" these cameras with focus pixel issues by adjusting the ADTG/CMOS registry as suggested by @Greg in this post:

http://www.magiclantern.fm/forum/index.php?topic=16054.msg167428#msg167428

The crop-rec module gives the same raw buffer size than mv720 mode, I think, so it's kinda hard to come up with an auto-detection.

Yes, that's another subject that was discussed. This is especially important with the mv720 raw buffer. As you know the "normal" mv720 uses 3x5 line skipping or binning and the aspect ratio needs to be adjusted. The crop_rec module also uses the mv720 raw buffer but uses 3x3 line skipping or binning so it doesn't need the aspect ratio adjustment. We probably need to ask @g3gg0 to add a new structure to the MLV format specifications.

https://docs.google.com/spreadsheets/d/1ItNuLj34JlK6bZgiAJLrmG7p3iyCmCju4XYyiwjqJOM/edit#gid=1

Then of course the ML source code needs to be modified to use the new structure. Note that there is another new structure proposed to support long lens names for the Non-CPU lens info branch. More about that project here:

http://www.magiclantern.fm/forum/index.php?topic=18083.0

One last note -- have you looked into Dual ISO support for your app? Fixing focus pixels on Dual ISO is especially tricky. Read through the Dealing with Focus Pixels in raw video topic to find out why.

[Danne]

Why not just use your fine maps dfort and all is fine straight away. GPL style of course. That's how they come to be in the first place anyway.

[dfort]

Why not just use your fine maps dfort and all is fine straight away.

Well it would be good if someone else validates my focus pixel map files. Separating them into blue and red map files is a novel idea and maybe it is a better way of dealing with focus pixels.

Of course the best solution of all is to eliminate them in camera and maybe there is some focus pixel free video stream somewhere that we haven't found yet. Funny thing is that we seem to have found a way to put focus pixels into H.264 via the crop_rec module so that might provide a clue?

[martinhering]

Separating focus pixels into red and blue maps is intriguing. Does it work in all cases? Reason I ask is because it seems that the pixels change colors depending on the lighting.

Well I did not find any downsides yet. I also think that separating the channels for correction yields better image quality, since you are only repairing 1 pixel and not its surroundings. I played a bit with the interpolation matrix and I think an average of the 8 surrounding pixels is good enough, but you need to make sure that the surrounding pixels are not dead pixels. I just leave them out and average the remaining pixels.

Here's an example from the edge of a frame from the 100D. Note that this is also showing focus pixels outside of the area that you are covering with your maps.

Do you still have the MLV? The reason, why I use the MLV is, that I can separate the components and save a grayscale image for every component (I could also do this with DNGs, but I dont have a decoder yet). I can then use the "Threshold" tool in Photoshop to extract the bad pixels. Very simple. I save the BW image as an 8bit png with 2 colors (2kb max). Then I read in the pixel map and create a table with x and y values for red and blue (there seem to be no focus pixels in green) and iterate over the table and fix the pixels. This way I can always simply open the BW png in PS and map out new pixels on the go and never need to change the code. Very fast turnaround. I can also use component images of new MLVs and overlay the existing pixel map and map out new pixels this way. Very easy.

One last note -- have you looked into Dual ISO support for your app? Fixing focus pixels on Dual ISO is especially tricky. Read through the Dealing with Focus Pixels in raw video topic to find out why.

Not yet. I am also postponing that. I first need to give my app some more useful features like histogram and RGB parade, thumbnail grid, etc.

[dfort]

Do you still have the MLV?

No, I did the 100D tests a while ago with a borrowed camera and deleted most of the files to make space on my drive for other projects. You can very easily find many of the focus pixels that are not the pixel map files that you created by shooting a frame with the lens cap on. Then adjust the contrast, brightness, saturation and whatever else you've got on that dark and noisy frame and you should see something like this:



Like I said before, I have never seen all of the focus pixels show up on a single shot so if you created your map files from a single exposure you will eventually have someone come to you with a shot where your app didn't removed all of the focus pixels.

[martinhering]

@dfort

You can very easily find many of the focus pixels that are not the pixel map files that you created by shooting a frame with the lens cap on.

I bought a used 700D of eBay and could verify this with some dark frames and I updated my pixel maps. Also had to update my auto-detection code for the 700D.

Code: [Select]

    NSInteger rawBufWidth = 0;
    NSInteger rawBufHeight = 0;
    NSInteger rawBufWidthZoomRecording = 0;

    switch (_idntInfo.cameraModel) {
        case kMLVCameraModelEOSM:
            type = kFTRawImageFocusPixelsTypeEOSM;

            rawBufWidth = (block.cropPosX > 0) ? 80 + raw_info.width + (block.cropPosX-80)*2 : raw_info.width;
            rawBufHeight = (block.cropPosY > 0) ? 10 + raw_info.height + (block.cropPosY-10)*2 : raw_info.height;
            rawBufWidthZoomRecording = (block.cropPosX > 0) ? 128 + raw_info.width + (block.cropPosX)*2 : raw_info.width;
            break;
           
        case kMLVCameraModel650D:
            type = kFTRawImageFocusPixelsType650D;

            rawBufWidth = (block.cropPosX > 0) ? 64 + raw_info.width + (block.cropPosX-64)*2 : raw_info.width;
            rawBufHeight = (block.cropPosY > 0) ? 26 + raw_info.height + (block.cropPosY-26)*2 : raw_info.height;
            break;

        case kMLVCameraModel700D:
            type = kFTRawImageFocusPixelsType700D;

            rawBufWidth = (block.cropPosX > 0) ? 64 + raw_info.width + (block.cropPosX-64)*2 : raw_info.width;
            rawBufHeight = (block.cropPosY > 0) ? 26 + raw_info.height + (block.cropPosY-26)*2 : raw_info.height;
            break;

        case kMLVCameraModel100D:
            type = kFTRawImageFocusPixelsType100D;

            rawBufWidth = (block.cropPosX > 0) ? 64 + raw_info.width + (block.cropPosX-64)*2 : raw_info.width;
            rawBufHeight = (block.cropPosY > 0) ? 26 + raw_info.height + (block.cropPosY-26)*2 : raw_info.height;

        default:
            break;
    }


    if (rawBufWidth == 1808) {
        if (rawBufHeight > 1000) {
            type |= kFTRawImageFocusPixelsType1808x1190;
        } else {
            type |= kFTRawImageFocusPixelsType1808x728;
        }
    }
    else if (rawBufWidth == 1872) {
        type |= kFTRawImageFocusPixelsType1872x1060;
    }
    else if (rawBufWidth == 2592 || rawBufWidthZoomRecording == 2592) {
        type |= kFTRawImageFocusPixelsType2592x1108;
    }

I could not verify yet, if it's really working on the 650D. I just hope so.

[dfort]

I could not verify yet, if it's really working on the 650D. I just hope so.

Looks like your testers are verifying it for you. I also had the same experience, the code for EOSM/650D/700D is the same.

I examined files shot on various cameras at different ISO and exposures, scene brightness, etc. and eventually came up with my maps. There were a few holes here and there that I discovered when I wrote a function that would create the focus pixel pattern so eventually all of the possible focus pixel coordinates were mapped. Well, let's just say I'm pretty sure they are all mapped. Looks like you're doing the same thing so I don't understand why the EOSM is different from the 650D/700D/100D -- that doesn't match what I came up with.

[martinhering]

Looks like you're doing the same thing so I don't understand why the EOSM is different from the 650D/700D/100D -- that doesn't match what I came up with.

The pixel maps as such are not different, but the crop rectangle logic is different. The various buffer modes are centered around a specific point and that point is differently offset. On the EOSM its 16px more to the right and 16 px more up. You only need this logic for reverse calculating the original raw buffer size for the auto-detection. You don't need it for the correction, as cropX/Y parameters are enough for that. Not saying that my maps are perfect yet. I am not confident yet that they are, but so far they are working. I'll update them as needed.

[dfort]

...You only need this logic for reverse calculating the original raw buffer size for the auto-detection...

Not sure I follow you on that. The original raw buffer size is included in the MLV RAWI block so why do you need to reverse calculate the size?

[dfort]

The focus pixel maps that we came up with on this topic were actually created by shooting several "simple silent" stills using the various video modes. The reason it can't be done with just one DNG is because there are a few different kinds of focus pixels and they become visible in specific lighting conditions. In other words, shooting a silent still with the lens cap on will show one pattern of focus pixels while shooting a medium grey target will show a different pattern. In addition, there are focus pixels that show up as different colors. @martinhering came up with a different method of dealing with these focus pixels using two different maps, one for red and a different one for blue.

[dfort]

Now how about that crop_rec focus pixel list for the 100D? Could it be added to fpm.sh?

I need some files. Turn on crop_rec, mlv_rec or mlv_lite and silent. Set the movie recording mode 1280x720 in the Canon menu and on the Magic Lantern menu to MLV and turn on crop_rec. Now leaving the camera on movie mode turn on silent and shoot some simple silent DNG frames. Check out the previous post for suggestions like shooting some varied scenes, dark, light, etc. Pick out a few frames that show lots of focus pixels and send them to me.

To make up the focus pixel maps I need the full raw buffer so sending me movie footage won't do it for me. The DNGs don't show the full raw buffer. For that I needed this tip:

To get the entire raw buffer from a DNG, including OB area: dcraw -4 -E
To get the active area only: dcraw -4 -D

Then comes the fun part, map all of the focus pixels and if a pattern emerges write an algorithm that can recreate that pattern.

[Danne]

Here are two files. Tell me if you need more.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_crop_rec_focus_pixel_DNG.zip

I specified:

Code: [Select]

dcraw -D -E -T INPUT.DNGSeems to work and we get a tiff instead of a pgm file. Output came out all black with -4 so skipped that.



Ant farm

[dfort]

I specified:

Code: [Select]

dcraw -D -E -T INPUT.DNGSeems to work and we get a tiff instead of a pgm file. Output came out all black with -4 so skipped that.

The problem with that is that your tiff file is a different size than a pgm file using dcraw -4 -E. This is what I need to work on the pixel map files:



This file is the size of the full raw buffer, 1808x727. The tiff you created trimmed off the out of bounds area and it is 1660x699. Yes, the pgm file looks black but if you adjust the brightness and contrast you can see the detail. Notice how it clearly shows the area that got cropped off.

The good news is that 100D crop_rec isn't showing two focus pixel patterns superimposed over each other like on the EOSM and 700D so this is going to be easier. Now all I need is more hours in a day to get through all my projects.

[DeafEyeJedi]

Ant farm

More like Fire Ants! 

[Danne]

I see dfort.

Code: [Select]

dcraw -T -E -4 -b 20 INPUT.DNG

[dfort]

Ok--there's more than one way to do this.

I just checked the focus pixel map files for the 100D and it looks like the one for mv720 is a perfect match. Check out the image file labeled pattern_B_mv720_1808x727. The extra pixels on the sides are there because there are some focus pixels in that area that pop up in certain lighting conditions.

So--isn't your Switch app already working with the 100D in crop_rec?

[Danne]

Not really. I get this file from your script:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/allbadpixels.map

Tested against this file(turned off cold pixel fix):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_crop_rec.MLV



In your script fpm.sh

Code: [Select]

    # These cameras don't have crop_rec yet. Remove from list once the crop_rec modules is available.
    if [ "$camera" == "100D" ] || [ "$camera" == "650D" ]; then video_mode=mv720; fi
    ;;
  1872x10**)
    video_mode=mv1080crop
*Seems to work after all. Think my mlv_dump disables also map files when noo cold pixel fix is selected. Are the map files the same size crop_rec and non crop_rec?

[dfort]

Are the map files the same size crop_rec and non crop_rec?

mv720 and crop_rec 3x3 use the same full raw buffer size and in the case of the 100D it appears that they also share the same focus pixel map file.

So it is working? Can you show a before and after? I'm at work right now and can't test it.

[Danne]

Here is a test. Files developed with mlv_dump_on_steroids(fpmutil) inside Switch and then dcraw to tiff. Also checked agains fpm.sh
Testfiles:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_mv1080p_M06-0739_samples.zip
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_crop_rec_M06-0733_samples.zip

At grannys place(mv1080p, reference)

Checking the lamp for focus pixel residues. Seems fine.


crop_rec(not sure what´s up with the black extra border. Didn´t show up in earlier build, another issue)

Check the texture residue

More texture


A little chroma smooth 2x2 solves the issue




If all this is to be expected then I say it works. If any of the texture handling can be fixed in the script then there´s more work here 

[dfort]

Those areas with residual focus pixels shouldn't be like that. The crop_rec 3x3 frames should be as clean as the mv1080. That black bar is an indicator that something isn't quite right:



You might also check out mv720 because it is using the same map file as crop_rec.

A while back you PM'd me a file from an EOSM that showed focus pixels. Turned out that it was shifted 8-pixels horizontally. In this case the focus pixel map is lining up so it will require some more digging.

Maybe something is going on in the crop_rec_4k branch? Lossless? Reduced bit rate? Something else?

[Danne]

Yes maybe. Could you check against this file if it works or not on alternate files and whatnot:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/100D_crop_rec_focus_pixel_DNG.zip

Too many variables for me to do any conclusions. You are the boss here.