Dealing with Focus Pixels in raw video

[Greg]

We need to know why FRSP is no focus pixels.
Maybe a moire test, but I'm not sure.

[Welles]

@Frank7D, dfort, DeafEyeJedi

Thank you for your analysis. Crop mode does help a lot, but indeed this mode can't be used all the time. I'll test chroma smoothing on the white car this week.

A comparison from my first raw video (posted in the "Share Your Videos" thread) is provided below, normal first, crop mode below. DNGs from MLVFS are imported directly in PP, with a basic grading using a LUT, only with exposure, highlights, shadows, blacks and whites.

There also a disturbing irisation on the rocks (last pic). To my untrained eyes it does not look like focus pixels nor aliasing. What do you think?

Full shot here:
https://picload.org/image/rgcrgirr/shot0001.png

200% zoom on the house:


Full shot, crop mode:


Strange rocks

[Welles]

@dfort:

I've just tested, chroma smoothing do not do much about that artifacts on the white car it seems.

Nothing: https://picload.org/image/rgcaadrl/nofix.png
Chroma 3x3: https://picload.org/image/rgcaadgc/c33.png
Chroma 5x5: https://picload.org/image/rgcaadra/c55.png

[DeafEyeJedi]

@Welles-

Just to be sure ... You are actually 'Refreshing' the webpage of the Localhost while running MLVFS after each selective options has been changed, correct?

Because otherwise it won't take action if that's what you're referring to from earlier re: aliasing on the wires & artifacts on the white car.

Perhaps upload a original MLV (of that particular footage) for us to play with and see what we can get out of on our end.

[Welles]

@DeafEyeJedi

I did not refresh, because it seemed to me at first that changes are taking effect immediately, and apparently it's not 

So I did it again, here how:

I have a sequence opened in PP, direct import of the mounted MLV in PP, no transcoding.
I tested each setting (nothing, then chroma 3x3, then chroma 5x5) like this:

1. Closing PP,
2. Change the setting with refreshing the browser, loading the preview
3. Reopening PP, play a few seconds of video, export a PNG image (the same each time).

Here they are:

https://picload.org/image/rgcalapw/nofix.png
https://picload.org/image/rgcalapl/c33.png
https://picload.org/image/rgcalapi/c55.png

I'm trying to upload the MLV (do you the .IDX file too? idk if it is important), but I'm on a shitty connection right now.

[DeafEyeJedi]

No need .IDX as we'll obviously be able to generate that file on our own while running MLVFS.

[Welles]

OK, the MLV is uploaded here: http://dl.free.fr/en1sTPWzp

Oh, I've just noticed that they modified the site since the last time I put something there: you have to disable ad block and there's an annoying ad to watch before you can click the green button and download, I'm very sorry about that.

[Welles]

@DeafEyeJedi,

I uploaded the MLV here, it is hopefully more convenient:

https://filesender.renater.fr/?s=download&token=562051f3-665f-d279-e59f-d03c0c4a90a3

[DeafEyeJedi]

Thanks @Welles for the file. Based on my observations with a workflow that I use in DR12.5 Beta 3 which includes for me to use Cinelog-C as a flat log and then apply Cinelog-C_Rec709 FM on top along with a few color tweaks just to make it pop. Apparently you may or may not be right that it doesn't seem to help (or it may seem to) when applying varieties of fixes within MLVFS Localhost webpage. Here are the results in comparisons.

No Fix
https://vimeo.com/168440109

Bad Pixel Fix
https://vimeo.com/168440108

BPF + CS2x2
https://vimeo.com/168440113

BPF + CS3x3
https://vimeo.com/168440105

BPF + CS5x5
https://vimeo.com/168440268

Aggressive
https://vimeo.com/168440111

Agg + CS2x2
https://vimeo.com/168440112

Agg + CS3x3
https://vimeo.com/168440107

Agg + CS5x5
https://vimeo.com/168440110

Since it's also coming from a 100D user in myself as well ... I believe this is what we can expect from a such tiny camera in terms of quality (from the sensor) that delivers Raw, sure it's not a 5D3 but hey I wouldn't know which camera you used if you hadn't told me in the first place.

[Welles]

@DeafEyeJedi,

Thanks so much for having a look at this! I guess I'm gonna go with crop mode as much as I can then.

The only drawback I see is about the limited focal length on the wide side: by using the value provided by ML in the movie menu (5.83 instead of the regular 1.94 with a resolution of 1440×602) my 18-55mm lens is the equivalent  of a 105-320 in crop mode. Maybe someday I'll buy the 10-18mm IS STM.

On the good side, beside getting rid of the artifacts, being more in the center of the glass should provide better quality, or at least reduce drastically the vignette effect of the lens. And my 55-250 is the equivalent of a 320-1460mm!

[DeafEyeJedi]

Were you able to tell which one was slightly better than the other? I could.

Another trick is to try and shoot it as wide open as possible w the iris in order to try and defocus the wires or even should help with not showing as much aliasing and such.

Perhaps try scooping up a used wide angle lens as I've gotten myself a decent 6.5mm f3.5 from Opteka which was made for APS-C Sensors , however , it's not as sharp as I would like which then can help lessen the chances of aliasing/artifacts. Still a tad slow for me tho.

Also I've seen others get away with nice crop mode footage by using a 8mm f2.8 from either Samyang or Rokinon in which I plan on selling my crappy Opteka 6.5mm f3.5 for one of these.

Definitely could use the extra light for indoors and more quality in sharpness overall.

[Welles]

Were you able to tell which one was slightly better than the other? I could.

Yeah, from what I see, the more CS, the worst it is. I'd better not touch this. But I do not see any difference between NoFix, BadPixelFix and Agressive alone. Do you?

[Erik Krause]

About two weeks ago greg wrote:

Someone compared FRSP vs LV raw crop frame?

If the hybrid AF is like a dual pixel, one photodiode is composed of two :
http://www.imaging-resource.com/ee_uploads/news/4486/hybrid-cmos-af-iii.jpg

There is a possibility that in the photo mode, data of halves focus pixel are averaged. What is the result, focus pixel works as normal pixel.

So maybe just change the ADTG/CMOS registry to get a image without focus pixels.

Has someone investigated in this direction?

[AWPStar]

Thanks dfort, dmilligan, A1ex, g3gg0 and many others!

I did implement it to mlvp. But, still don't know what to to with .RAW and 3x5 skipping frames.

dfort, special thanks to you for your patterns!

[dfort]

Hi @AWPStar

Glad that you were able to use some of this stuff. For .RAW you need to supply the camera and video mode information because that isn't available in the metadata. There is a separate pattern for 3x5 skipping frames (also known as mv720).

Check out the source, written in bash:

https://bitbucket.org/daniel_fort/ml-focus-pixels/src/28fe6f4a5eb75a166bfecbca5a5fdcbdb7cd4666/mlv2badpixels.sh

[AWPStar]

Fresh release with your patterns  Thank you!
http://www.magiclantern.fm/forum/index.php?topic=15271.msg170632#msg170632

[dfort]

Looks like this is coming back to haunt us:

The video mode can be probably identified by (uncropped) raw buffer size - hopefully there aren't video modes with identical sizes, but different maps.

@rbrune has come up with a way to use the crop_rec module to record a sort of an mv720 mode with a 3x3 pattern instead of a 3x5 pattern. This of course is good news because the frame doesn't need to be adjusted to correct the aspect ratio but it is going to cause an issue having two conflicting focus pixel maps. The focus map affected is 80000331_1808x727.fpm. The 80000331 is the EOSM id and 1808x727 is the full raw buffer size. @a1ex has an idea to deal with this. In the meantime I'm going to see if I can create a focus pixel map file for this new video mode.

[DeafEyeJedi]

O0o0hhh this is so petrifying and yet feels so good in a way that I could've never imagined especially with an EOSM -- Thanks to everyone on their contributions in opening this wonderful can of worms especially to @rbrune which we shall probably name the so called ghost after?

The best of luck goes to @dfort and please do not hesitate to ask for help, if any.

[dfort]

Ok--here we go. A new focus pixel map file for the experimental EOSM crop_rec module. This is how it was done.

* Set up a focus target and put the camera on a sturdy tripod.
* Turn on the mlv_rec, crop_rec and silent modules.
* Shoot a short MLV file of the target and without moving anything shoot a Simple Silent picture--keep the camera in movie mode.
* Create dng files of the MLV file using your favorite converter, you only need to save one dng file.
* Run "dcraw -4 -E" on the Simple Silent shot, this will give you a .pgm file of the entire raw buffer.
* First put the raw buffer still into Photoshop (or Gimp) and raise the exposure so you can see some detail--it will be in black and white.
* In Photoshop I changed the Image mode from Grayscale to RGB.
* Throw in the dng file that you pulled from the MLV onto a second layer.
* Line it up, but don't do it by eye--even one pixel misalignment will cause big problems. Instead use "mlv_dump -v" and note the Crop numbers. In this case it came out to:

Code Select Expand

    Crop: 304x28
     Pan: 298x28


Like I said, use the Crop to line up the layer:



* Turn the layer on and off to check that things are lined up properly.



* Ok, cool. Now zoom in at gawk at all those focus pixels you need to hit.



* After a lot of trial and error I found out that making a new white layer and using the pencil tool set to 1-pixel black works great. I also set the layer to Multiply as I'm working. Obviously once you discover the pattern you can replicate it over and over until the whole focus pixel area is covered. By experimenting we discovered that not all focus pixels appear at all times. We also discovered that the pattern is repeated across the entire horizontal axis of the sensor but it doesn't extend all the way to the top and bottom. You don't want to map areas that don't have focus pixels because it might soften the image a bit, it makes for huge map files and it is too much unnecessary work.



* Change the layer back to Normal and export it as a Portable Bit Map file.



* I've written some bash scripts to convert the .pbm file to either a text file that can be used in dcraw (using the "dcraw -P deadpixels.txt" option) or a .fpm file that can be used in MLVFS. All of these tools and the map files I built are located in a bitbucket repository:

https://bitbucket.org/daniel_fort/ml-focus-pixels

After making map files for all the cameras that show focus pixels and all the raw video modes you would think that I could just knock out a new map file but it isn't that easy. Sometimes the focus pixels are subtle and sometimes they don't show up at all. Then there are situations where you're sure you hit all of them and then you shoot something that activates a whole slew of focus pixels that you missed. So if you find some that I missed--please let me know.

[dfort]

Some more focus pixel hunting on the EOSM crop_rec project turned up some new ones that I haven't seen before. Makes me wonder if the other fpm files have all the focus pixels mapped properly. Haven't heard any complaints so like they say, if it ain't broken don't fix it.

Here's what the crop_rec pattern looks like after hunting focus pixels on dark frames and user supplied MLV files.

Full raw buffer map

80000331_1808x727

Here's a closeup of the pattern


Somehow it doesn't seem right. More like some sort of alien hieroglyphics.

[a1ex]

I've tried to diagnose it by looking at the focus pixel map from 1:1 crop mode. We have some idea about the 3x3 binning filter, but we don't know the exact offset, so there must be 9 possibilities.

After losing patience waiting for fpm2pbm, I wrote an equivalent script in octave:

Code Select Expand


function M = fpm2pbm(image_size, filename)
    # fpm2pbm.m
    #
    # Create a Portable Bit Map file from an
    # MLVFS .fpm (focus pixel map file)
    #
    # Based on fpm2pbm.sh, http://www.magiclantern.fm/forum/index.php?topic=16054.msg161309#msg161309
    # Requires GNU octave.
    #
    # Usage fpm2pbm(image_size, filename)
    # example:
    # octave> fpm2pbm([1108 2592], '80000331_2592x1108.fpm');
    # shell$ octave-cli --eval "fpm2pbm([1108 2592], '80000331_2592x1108.fpm');"
    #
    # This is free and unencumbered software released into the public domain.
   
    M = ones(image_size);
    d = dlmread(filename);
    x = d(:,1); y = d(:,2);
    i = sub2ind(image_size, y+1, x+1);
    M(i) = 0;
    imwrite(M, [filename '.pbm']);
end


Now, to get the 9 possible focus maps after 3x3 column binning / line skipping:

Code Select Expand


pkg load image

# column binning filter
f = [ 1 0 1 0 1 ] / 3;

# focus pixel map from 1:1 crop mode, EOS M (assuming centered focus box)
M = fpm2pbm([1108 2592], '80000331_2592x1108.fpm');

# 3x3 binned image (full-res version containing all the possible offsets)
# actual 3x3 binning will result in one of those: MB(i:3:end, j:3:end) for i=1:3, j=1:3)
MB = imfilter(M, f);

# save all 9 possibilities for starting offset
for i = 1:3
  for j = 1:3
    Mb = MB(i:3:end, j:3:end);
    imwrite(double(Mb > 0.99), sprintf('map3x3+%d+%d.pbm', j, i));
  end
end


This gives 9 nice patterns, nowhere near the above example. Go figure...

On the good side, the above script correctly finds the patterns from 1808x727 (5x3) and 1808x1190 (3x3), starting from 2592x1108 (1:1).

To get the 5x3 maps:

Code Select Expand


# save all 15 possibilities for starting offset
for i = 1:5
  for j = 1:3
    Mb = MB(i:5:end, j:3:end);
    imwrite(double(Mb > 0.99), sprintf('map5x3+%d+%d.pbm', j, i));
  end
end


(edit: fixed off-by-one error)

[DeafEyeJedi]

Stunning work as usual @a1ex!

[dfort]

After losing patience waiting for fpm2pbm, I wrote an equivalent script in octave:

Yeah, my pbm2fpm.sh and fpm2pbm.sh bash scripts are painfully slow. I was thinking about trying to optimize them but thought I was the only one using them and even I don't use them very often.

Octave? Hum. Never heard of it. I installed octave along with the image package you're using and figured out the very basics but those bits of code don't seem to add up to a fully functional script. I can't get it working. Believe me I really want to.

You used the zoom mode map file which could be a problem. That file takes into consideration all the possible positions that the raw buffer can move around the sensor so it is a combination of 21 layers each representing a position of the raw buffer on the sensor. It would be better to use the mv1080crop map file--aka Movie crop mode which is 1872x1060. With that file you won't be looking at coordinates where no focus pixels have been sighted. Yeah, these focus pixels are elusive and I have never had a frame that showed all of the focus pixels at once.

One of my ideas when working on this project was to create a master map file of the entire sensor and pull the 3x3 and 3x5 cropped patterns like what you seem to be doing with your octave script. However, I don't have the coding chops for this and there was no way to figure out exactly what position the zoom mode raw buffer was at so I had to give up on that idea.

I wish I could see the patterns that you're looking at because it would help verify that we really do have all the focus pixels mapped. That last one for the crop_rec just doesn't look right. My hypothesis is that the 100D has the same pattern as the other cameras but it is covering a larger area of the sensor. It also seems to use those pixels differently, that's why they are more prominent than on the EOSM, 700D and 650D.

Having coordinates on the map files in places where there aren't any focus pixels might give us a hit in resolution but the trade off is possibly missing focus pixels. I'm thinking about combining the "normal" mv720 with rbrune's crop_rec mv720 like I did with the zoom map so that it will take care of both modes with the same fpm file. That is until there's a way differentiate these modes from the MLV metadata. No rush, EOSM crop_rec isn't exactly main stream at the moment.

[a1ex]

First script: save it as fpm2pbm.m in the current directory, then copy the example command (you can run it either from octave prompt, or from the shell).

Second script: either save it as maps.m (or whatever), then type its name at the octave prompt (without extension, so just "maps"), or paste everything into the octave prompt.

My hypothesis about the mv1080 crop map: it might be some mix between the 3x3 map (the real focus pixel coordinates) and the 5x3 one (where Canon's FPN handling code is expecting to find the focus pixels).

We might be able to get a focus pixel map from a full-res silent picture. That one shouldn't be ambiguous, but we need to patch Canon code to change the raw type. I can look into it if there is a good reason for doing so.

[dfort]

@a1ex

I found out that octave scripts that end with an .m extension are for MATLAB compatibility so it seems that your fpm2pbm.m needs it but it doesn't work with the maps script. Anyway, got it working.

Checking out the EOSM focus pixel map files it looks like everything except the latest crop_rec is working as expected. The 100D map files completely blew away my hypothesis.

With the EOSM the full (1x1) pattern seems to be this:




The possible 3x3 skip patterns according to your scripts are just two basic patterns. One looks like this:




And the other one is the one we're using for the mv1080crop and zoom raw video modes:




With 3x5 there are three basic patterns. This one:




I didn't make a mistake here, some possible 3x5 combinations actually miss all the focus pixels:




And finally this one which is used in the mv720 focus pixel map file:




So far everything seems to be falling into place. The EOSM patterns are also the same for the 650D and 700D.

There's only one more camera to check out, the 100D which has a 1x1 pattern that we believe looks like this. Note that if you overlay the EOSM/650D/700D pattern you'll see that it is a subset of this pattern:




Using the octave script the possible 3x3 patterns are:




And:




Hum--that doesn't match up with what we ended up using for the focus pixel map files. The mv1080 map file pattern we're using for the 100D matches the other cameras. Ok, so how about the 3x5 patterns? This should also match the other cameras:




And:




What the.....?

My hypothesis about the mv1080 crop map: it might be some mix between the 3x3 map (the real focus pixel coordinates) and the 5x3 one (where Canon's FPN handling code is expecting to find the focus pixels).

Looking at what I came up with for the EOSM crop_rec pattern you're probably right. I've most certainly got some stray coordinates in there so how to figure this out so we can get the "real" pattern?

We might be able to get a focus pixel map from a full-res silent picture. That one shouldn't be ambiguous, but we need to patch Canon code to change the raw type. I can look into it if there is a good reason for doing so.

That would be interesting but I'm not sure that would solve this puzzle. The focus pixels on the 100D are much easier to see so maybe giving that platform the crop_rec module might shed some light on this?