As some of you know I've been adding map files for the latest bleeding edge crop_rec module experiments. Most of the new resolutions have been coming from the EOSM experiments but there has also been experiments done on the 100D and 700D. At this point it looks like we've got all the current settings covered so I thought it would be a good time to take a step back and reflect on where we are at with this focus pixel project.
Where are these so-called focus pixels on my camera's sensor -- Everywhere!When I started this topic it seemed that the focus pixels were bunched up in the center of the sensor in a cross shaped pattern.
Next we found out that the patter stretched across the entire sensor.
This makes sense because it would be very difficult to manufacture a sensor with different types of receptors in a cross pattern. The main difference between the "older" 650D, 700D and EOSM sensor and the "newer" one used on the 100D and EOSM2 was the area covered by the focus pixels.
At first there also seemed to be a difference in the color and pattern of the focus pixels of the newer sensor but after seeing many sample files it looked more and more like these sensors all shared the same pattern and showed up in images in pretty much the same way. Just like the discovery that the focus pixels weren't confined to a small cross of the older sensor or large box of the newer sensor, focus pixels started appearing in areas outside of these boundaries.
There is an area we call the out of bounds area of the full raw buffer. That can be clearly seen using the raw_diag module from the ISO research tools branch.
Up until now I've been careful to keep from mapping the out of bounds areas but on this latest round of updates I started mapping the entire full raw buffer because some of the experimental crop_rec settings might have different OB zones and it made comparing the image files easier by lining up the upper right corners of the images and finally because it was easier. The savings in the number of mapped coordinates is rather insignificant. I mean we're already defining up to 154,440 coordinates on these map files, what's a few thousand extra dots between friends?
So to conclude this section--the map files are covering the entire area of the full raw buffer.
What is the focus pixel pattern? -- There's more than one.When I first started I wanted to eventually map out all of the focus pixels on the full sensor then calculate all the pattern for the various settings. However, getting a good look at focus pixels on the full sensor wasn't possible until recently when @theBilalFakhouri came up with a setting that gave us access to the full 5280x3508 raw buffer--and it showed focus pixels. This is the pattern:
Notice that the pixels on the horizontal axis repeat every 24 pixels. File that because we'll come back to it.
The first 1:1 sampling options we had with these cameras was the zoom mode which is activated with the magnifying glass button and the Movie crop mode setting which is something that is unique to these cameras that show focus pixels. What I noticed when testing zoom mode was that the pattern would shift 8 pixels left or right. Why this happens is probably due to the way the "crop" value is determined. If you run mlv_dump -v you will see that each frame has a crop and pan setting. We're using the crop value to line up the map file.
A while back there was a bug with the way crop was calculated and fairly recently
a similar issue came up with the lossless compression settings. When I wrote the fpm.sh script I took this into account and calculated the crop setting from the pan values like this:
##
# Allow for legacy MLV files with cropX, cropY bug
#
cropX=$((panPosX + 7 & ~7))
cropY=$((panPosY & ~1))
This should no longer be necessary so I'll be taking it out.
Alright, so where am I headed with this and why did I ask you to remember 24? Because in the zoom setting you can move the capture area around the sensor and the pixels will jump left right in 8 pixel increments as it switches between the 21 full raw buffers used in that mode.
To compensate for this I copied the map file so that the same map would still work even if the focus pixels moved left or right by 8 pixels.
Now I'm starting to question if this is necessary because on my new map files I didn't do this on the 100D yet it worked on all the samples that I received. Looks like more testing on the zoom settings are in order. My question is does this also affect the Movie crop mode setting? Yet more testing. Perhaps the code should be changed so that crop is forced into 24 pixel increments? (There's that number again.)
Maybe like this?
raw_info.width, (skip_x + 23) & ~23, skip_y & ~1,Moving on to the 3x3, 5x3 and the new 1x3 sampling settings -- at one point
@a1ex looked at the focus pixels as part of his
pixel binning research.
I played around with his octave scripts and using the 1x1 sampling pattern we were able to recreate the 3x3 and 5x3 patterns. Here they are:
3x3 sampling (mv1080) focus pixel pattern
5x3 sampling (mv720) focus pixel pattern
Here's the latest setting that is getting attention:
1x3 sampling focus pixel pattern
Ok now, those of us who are better at remembering images than numbers might have just gotten an "ah ha" moment. That's the same pattern as the one we're using on the 1x1 sampling adjusted to take into account the 8 pixel left/right shifting. For those of you better with numbers, 24/3=8. Alright, I may not be a math wiz but I did pass elementary school.
This brings up an off topic point -- the horizontal binning pattern probably should be evenly divisible by 3 but we are forcing the crop setting to be evenly divisible by 8. Maybe that's another reason to change the code so that it forces the crop value to be evenly divisible by 24? I don't know, just a shot in the dark on this.
But wait, there's more! (Is this starting to sound like an infomercial?)
Recently @Danne came up with a setting that showed a different focus pixel pattern. Not that surprising if you have ever played around with the various PREFERRED_RAW_TYPE:
raw.c
#undef RAW_DEBUG_TYPE /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */Here's the focus pixel pattern for his "reconfig_cat" setting or whatever he calls it:
What's with these "multi-pass" map files? -- When one pattern isn't enough.When we were experimenting around trying to get mv1080 (3x3 sampling) working on the EOSM we started getting
images where the map files weren't working and focus pixels were starting to make a strange pattern, like two patterns were layered on top of each other. We also had a situation where the same full raw buffer size was used for both 3x3 and 5x3 sampling so what to do?
At first I made a map file that combined the focus pixel patterns but that didn't work because the coordinates were too close to each other and they started picking up the color from adjacent focus pixels and it turned into quite a mess. The
solution was discovered by @bouncyball which was to make a separate pass, one for each focus pixel pattern.
This trick also worked for situations where the same sized full raw buffer was used for different sampling, like what happened on that crop_rec setting. So even though other cameras didn't show both focus pixel patterns superimposed over each other like the EOSM, they did benefit from multi-pass map files. Now we've got to ask, isn't all this extra processing affecting the image? Yes, though it is much better than having focus pixels on your video. For the most part the image doesn't seem to suffer unless you clipped the highlights and bring the exposure way down in post so they turn grey. At that point, the focus pixel map starts showing up on the image but at that point the image is quite a mess anyway. A little bit of chroma smoothing helps. I've suggested that several times on this topic.
Now with these latest crop_rec settings I looked at MLV files that had
the 1x1 sampling focus pixel pattern superimposed over the 3x3 pattern. Again, a multi-pass map file was able to deal with that.
Where do we go from here? A light touch or throw everything including the kitchen sink at it?Here's the problem, we will continue to find new resolutions that require one or more of the focus pixel patterns. Right now which one(s) can't be predicted. So do we try to find a way to figure out which pattern is needed? With this latest round of testing it looks like that would be very difficult to do. How about making a pass with each of the known patterns? That might work as long as we can figure out how to line up the map files for every possible setting though it does seem like overkill.
Another problem is that the inventory of map files is growing exponentially because for every new resolution there are five cameras that might need that new map file. The good news is that it appears that every camera seems to be using the same patterns. Yes, even the "old" and "new" sensors are using the same patterns. That means that we shouldn't need to make a separate map file for each camera.
Well this post turned out much longer than I anticipated but if you got to this point perhaps you have some idea you might want to share?
(NOTE: Please, oh please don't quote this entire post on your reply!)
