Dealing with Focus Pixels in raw video

[Danne]

which is used in MLP right now.

cr2hdr.app

Digital dolly, anybody ever used that feature?

[bouncyball]

Digital dolly, anybody ever used that feature?

IDK, but it indeed sounds cool

[a1ex]

That feature would probably be a bit more interesting in crop mode with automatic subject tracking

It is possible to move the crop window on the fly, though a bit tricky, as it breaks Canon's pattern noise correction. It can probably be corrected in software, and the tracking algorithm will be likely slow, but the hardware can do this.

[Danne]

automatic subject tracking

Lock on to a bird or person for instance. Possible, how?

[a1ex]

For tracking: meanshift/camshift or something similar.

For moving the sensor area: alter the image position registers used in crop_rec (in rough increments) and adjust our raw recording area for fine increments.

From crop_rec.c:

Code: [Select]

            /* raw buffer centered in zoom mode */
            case CROP_PRESET_CENTER_Z:
                cmos_new[1] = PACK12(9+2,42+1); /* vertical (first|last) */
                cmos_new[2] = 0x09E;            /* horizontal offset (mask 0xFF0) */
                break;


[bouncyball]

Would be great!

Can this be used to stabilize the image according to camera level sensor data? Well as far as sensor precision and CPU speed can get.

bb

[a1ex]

Unfortunately, Canon's level sensor is not the best for this purpose.

http://www.magiclantern.fm/forum/index.php?topic=18473
http://www.magiclantern.fm/forum/index.php?topic=13229.0

[bouncyball]

Ah... understood, gyroscope's absent.

[dfort]

Since crop_rec came up, has anyone else noticed focus pixels on the EOSM (or my 700D crop_rec build) when shooting H.264 with the crop_rec module active?

[Danne]

Yes, always there on eosm. How will other raw streams affect H.264 crop_rec footage?
On a sidenote I noticed some focus pixels with the -n switch when using mlv_dump on steroids processing crop_rec material. Posted test pics in bouncyball,s thread.

[DeafEyeJedi]

...has anyone else noticed focus pixels on the EOSM (or my 700D crop_rec build) when shooting H.264 with the crop_rec module active?

Yes, I can confirm this w EOSM.

On a sidenote I noticed some focus pixels with the -n switch when using mlv_dump on steroids processing crop_rec material. Posted test pics in bouncyball,s thread.

Good catch, Danne!

[Danne]

Trying to figuring out the -n switch and why some focus pixels aren,t covered. Using the script without the -n switch I notice the first nine coordinates aren,t created like when running without it. Check these two examples.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/pixelmaps.zip

I try to understand by checking the script but couldn,t figure out what is going on.

[bouncyball]

@dfort: would you mind to upload to your bitbucket download area all original raw buffer .PBM files made/used by you to generate .fpm maps for MLVFS?

[Danne]

Referring to these files?
https://bitbucket.org/daniel_fort/ml-focus-pixels/downloads/2016-02-18_raw_buffer_fpm_files.zip

Or here

https://bitbucket.org/daniel_fort/ml-focus-pixels/src/4a431cfc9748af34056cca702bd4f421cd54fbcd/focus_pixel_map_files/?at=default

[bouncyball]

@Danne:

Nope, they are text maps. I need bitmaps in the PBM format which where used to generate these FPMs.
I can generate PBMs from the existing fpms with script but I'm interesting in original files saved from graphical software.

[dfort]

Got it. Different time zone here and about to start a busy day but managed to add graphic files in two formats to the repository. PBM P1 (plain text) and PBM P4 which is more common in graphics programs. Either format should open fine for you. I'll explain more about these files later.

focus_pixel_image_files

[dfort]

@dfort: would you mind to upload to your bitbucket download area all original raw buffer .PBM files made/used by you to generate .fpm maps for MLVFS?

You got it. Just some notes about this. In my repository you'll find a couple of bash scripts that will convert between fpm and pbm named fpm2pbm.sh and pbm2fpm.sh. These run very slow but do the job. a1ex wrote an octave script named fpm2pbm.m that will convert from fpm to pbm much faster but it requires setting up octave along with a graphics library. The pbm format used in these scripts is known as P1 or "Plain PBM" and is about the simplest graphics image format possible but also very inefficient because it is saved as a plain text file. Most image file editing programs that can work with pbm files will save in a P4 binary format which is also very simple but the file size is considerably smaller. It is easy to convert between these formats with ImageMagick. I saved pbm files in both formats for you. So as you can see these focus pixel map files can be represented by either a text file with the coordinates of every known focus pixel or as an graphic image file.

There are only two sensors that are causing these focus pixels. The EOSM/650D/700D all use a sensor that creates one pattern which I called the "A" pattern in my script and the 100D creates a different pattern which I named "B".

Here's a look at the crop_rec pattern A image file:



Note that the image is extracted from the full raw buffer and there is an area on the top and left of the full raw buffer that is always blank so I didn't map any focus pixels in that area.

One thing that I haven't experimented with yet is removing focus pixels on simple silent shots. There are no focus pixels with Full Resolution Silent Pictures but when shooting simple silent it captures the raw buffer used in movie capture and it does show focus pixels.

[martinhering]

Here are the focus pixel maps, I extracted so far:

https://www.dropbox.com/s/e42a015uoeabbv4/Focus-Pixels.zip?dl=0

Seem to work great with silent pictures and any video I came across yet. I am also trying to detect which pixel map to apply automatically, which seems to work so far:

Code: [Select]

    if (_idntInfo.cameraModel == kMLVCameraModelEOSM) {
        FTRawImageFocusPixelsType type = kFTRawImageFocusPixelsTypeEOSM;

        NSInteger rawBufWidth = (block.cropPosX > 0) ? 80 + rawInfo->width + (block.cropPosX-80)*2 : rawInfo->width;
        if (rawBufWidth == 1808) {
            type |= kFTRawImageFocusPixelsType1808x728;
        }
        else if (rawBufWidth == 1872) {
            type |= kFTRawImageFocusPixelsType1872x1060;
        }

        [_rawImage fixFocusPixelsWithType:type withCropX:block.cropPosX: block.cropPosY];
    }
    else if (_idntInfo.cameraModel == kMLVCameraModel100D) {
        FTRawImageFocusPixelsType type = kFTRawImageFocusPixelsType100D;

        NSInteger rawBufWidth = (block.cropPosX > 0) ? 64 + rawInfo->width + (block.cropPosX-64)*2 : rawInfo->width;
        if (rawBufWidth == 1808) {
            type |= kFTRawImageFocusPixelsType1808x1190;
        }
        else if (rawBufWidth == 1872) {
            type |= kFTRawImageFocusPixelsType1872x1060;
        }
        [_rawImage fixFocusPixelsWithType:type withCropX:block.cropPosX: block.cropPosY];
    }


I have yet to test the 650D and 700D though.

[bouncyball]

You got it. Just some notes about this. In my repository you'll find a couple of bash scripts that will convert between fpm and pbm named fpm2pbm.sh and pbm2fpm.sh. These run very slow but do the job. a1ex wrote an
There are only two sensors that are causing these focus pixels. The EOSM/650D/700D all use a sensor that creates one pattern which I called the "A" pattern in my script and the 100D creates a different pattern which I named "B".

Thank you so much for files.

Yes I know all about PBM (raw/ascii) formats. I need P4, P1 is a huge overkill. Yesterday, I also used all your scripts there and experienced lovely time wasting moments of my life . a1ex's fpm2bpm.m works, but octave is a real canon from the days of civil war vs pneumatic pistol we actually need to do the job .

I've found out about "A" and "B" patterns from your script as well. Love this script and the idea to generate the maps on the fly by known patterns.

Nice to have full frame buffer map images. GIMP is great to work with both formats of PBM.

Thanks again
bb

[bouncyball]

@martinhering: You are making real progress every day!

[martinhering]

Does anybody know what actually causes these Focus Pixels? Is it the sensor or the processor? And why do only the models with the DIGIC 5 processor show these?

[bouncyball]

Not a CPU issue. Sensor has them all phisically on the surface. Canon firmware, while reading raw data from the sensor, does processing and according to some register values gives number of so called (processed/unprocessed) raw streams. In one stream there may be maped out all bad pixels, in another, vertical stripe correction applied (or both) and so on. There can/should be at least one stream with all focus pixels maped out but no one knows how to make it working at the moment .

On affected cameras, for MLV/SilentPic recording ML uses LV raw buffer with the stream which contains those pixels in it, hence they have to be removed in post.

[dfort]

Right, what @bouncyball said.

For an illustration of this, check out this post from a while back. Thought I found the Holy Grail but it turned out that the cure turned to be worse than the disease. (I sometimes throw around these technical terms when communicating with developers.) Ok, kidding aside there might a video stream that doesn't show these focus pixels but we haven't found it yet.

I took another look at the @martinhering focus pixel files and I'm still scratching my head at how these work because some of the maps have pixels next to each other on the horizontal axis, others are missing sections and still others have focus pixels mapped in areas that we know there can't be any focus pixels because that it is an inactive area of the raw buffer. There are also some video modes that are missing, like zoom mode which has a full raw buffer size of 2592x1108 and can move around the sensor like this:



That caused all sorts of issues when I tried coming up with a focus pixel map file that would take care of all possible raw buffer locations. I ended up just stacking all 21 raw buffers in Photoshop. This might be overkill because if the raw buffer happens to be at the top or bottom of the sensor it might be in an area that doesn't have any focus pixels but since we don't know the position of the raw buffer on the sensor we need to account for all possible locations.

Another challenge is with the crop_rec module. On the EOSM (and my experimental version for the 700D) it uses the mv720 full raw buffer (1808x727) with a 3x3 pixel skipping pattern and the focus pixels that show up on both the mv720 and mv1080 video modes appear on the image. I thought that using a combo map would work for both the crop_rec and mv720 video modes but after extensive testing with Danne, we found out that it doesn't work that way and we always ended up with at least some focus pixels showing up on crop_rec footage like in this example:



Adding some Chroma Smoothing helps get rid of these in raw video but crop_rec H.264 footage also shows focus pixels and removing these in post production is even more challenging.

I have yet to test the 650D and 700D though.

I've got a 700D and can provide you with test footage for all the video modes. I believe I read somewhere that you needed a blank wall at a certain exposure, could you please post exactly how you would like the tests shot?

[martinhering]

I'm still scratching my head at how these work because some of the maps have pixels next to each other on the horizontal axis, others are missing sections and still others have focus pixels mapped in areas that we know there can't be any focus pixels because that it is an inactive area of the raw buffer. There are also some video modes that are missing, like zoom mode which has a full raw buffer size of 2592x1108 and can move around the sensor

I actually bought a used EOS M on ebay and did some silent pictures, and yes, I mapped out some pixels in the inactive area, I will correct that. You're right. I only checked fullframe and crop mode yet, since I could not find the other modes. The settings are kinda convoluted. I'll check the crop_rec module though and see if I can find other circumstances that produce focus pixels pattern I am not taking care of yet. Were can I set up "Zoom Mode"? I could not find the freakin setting. Also dolly mode on the EOS M is really unusable. It uses buttons that are normally used for other stuff. So I guess that this is not something that is in wide spread use.

I ended up just stacking all 21 raw buffers in Photoshop. This might be overkill because if the raw buffer happens to be at the top or bottom of the sensor it might be in an area that doesn't have any focus pixels but since we don't know the position of the raw buffer on the sensor we need to account for all possible locations.

This does not sound correct. Is this just a thing with dolly mode?

I've got a 700D and can provide you with test footage for all the video modes. I believe I read somewhere that you needed a blank wall at a certain exposure, could you please post exactly how you would like the tests shot?

I think you said that 700D, 650D and EOS M experience the same patterns, so I would just need one file in every mode to confirm that. Already thinking of buying more camera on ebay :-) If you could provide the silent pictures, that would be awesome. BTW do you have a Mac? Can you test my app?

[martinhering]

One more thing:

What would really help is a better way to blend the pixels. Your idea of changing the colors is interesting, are you interpolating just one color channel surrounding a focus pixel?
A problem with blending surrounding pixels comes up when there are high contrast sharp lines like in this shot from a 100D:
I'm not sure what will work for situations like this. What helps is to combine chroma smoothing along with the focus pixel map but even that isn't enough in some cases.

Yes, I am correcting focus pixels on the actually raw buffer itself and I am just interpolating surrounding pixels from the same channel. About the false colors in sharp lines, I read some stuff about it and I think that this is a problem with the demosaicing itself. Hence Amaze which reduces aliasing. Since I cant change the demosaicing method of the Apple RAW engine at the moment, I might be able to find a solution in post. Something like chroma smoothing in areas around edges. I think it should be possible to come up with some GPU code.