Dealing with Focus Pixels in raw video

[dfort]

Checked both of those files and the map file fits it perfectly.

[bouncyball]

Hi guys,

Based on your information and my experiments I've come up with the map which cleans Dann's samle MLV quite acceptably.

Here are sample maps.

All 3 maps are hybrid mv720+mv1080(shifted up by 30 pixels and cropped to mv720 size) maps.

1. 100D_crop_rec.pbm - is the graphical image to see how this map looks like.
2. 80000346_1808x726.fpm - is the standard map which made from this graphical image map.
3. 80000346_1808x726_dual_pass.fpm - this is more interesting one

3rd map coordinates are listed in this order: first mv720 and then mv1080, I called it dual pass map. Actually the interpolation process is linear but logically according to this map, interpolating code does mv720 interpolation first and then mv1080(shifted/cropped) interpolation.

Why I needed this: the focus pixels are so close to each other that interpolation of all pixels produces artifacts (artifacts which I've also seen on 700D Crop rec)
What does this mean: we have 2 available interpolation methods, older and simpler one from raw2dng code and later and more advanced, hence really better one, from cr2hdr/MLVFS. So when using raw2dng method there are some pinkish dot artifacts appear in the image. With MLVFS method there are none of these dots but if you interpolate all pixels in a raw according of their x,y, some horizontal very different color dot rows introduced all over the image.

With this dual pass map all artifacts are gone with better MLVFS interpolation method

All this can be tested with Danne's sample MLV.
Latest 'fpmutil' updated in the repository and can generate dual pass map for croprec 100D

regards
bb

Edit: This sample MLV from 100D has no shifted to any direction raw buffer. So all dfort's original calculations fit perfectly.

[Danne]

Very nice job Bouncyball. Will update and put it to real use in Switch. Will do some testing first and learn how to use it first.
This dual pass map should be used with eosm and 700D too right not only 100D?

[DeafEyeJedi]

Nice stuff, BB and Thanks for sharing your new dual pass map! 

[bouncyball]

Actually this map is very simple. The coordinates go for full mv720 pattern first and then for mv1080 cropped to mv720 size and shifted up by 30 pixels. No special usage needed.

This kind of map is generated for 100D crop rec only. Have to look if this approach will help to use MLVFS averaging for 700D without artifacts.

Would be very interesting your opinion @Danne and @dfort. Please use both interpolation methods and compare with both maps.

[Danne]

As soon as I get some energy I'm on it.

[bouncyball]

Cool, thank you

[dfort]

All this makes sense. The reason we were able to get away with a single map file for the EOSM/700D crop_rec was because the sensor has fewer focusing points. The 100D and probably EOSM2 is jam packed with focus pixels. Here is a "theoretical" map for the 100D I made a while back using what we learned from the EOSM/700D:



and a closer look:



Reminds me of Space Invaders.

Since some of the focus pixels are adjacent to each other the 100D will probably clean up better using two passes as bouncyball discovered.

[bouncyball]

Reminds me of Space Invaders.

Yes

This map also looks like you mentioned above.

[Danne]

Ok, 2 pass crop_rec mv720/mv1080 operation implemented into Switch. It works exactly as expected with perfect results. Great work Bouncyball .
Commit:
https://bitbucket.org/Dannephoto/switch/commits/09b441918cae37cd55e016b7383d290fa30c4c07

Works in conjunction with (ms) mlv_dump_on_steroids in Switch. Still can´t get clean results with fpm.sh script so still need to understand how that can be done to work with (m) mlv_dump in Switch.

Not the nicest footage but still shows the effectiveness of fpmutil.

[dfort]

This map also looks like you mentioned above.

Tried your map file in MLVFS and came up with this:



Using the plain mv720 map file it looks pretty good but it has something Danne called "texture residue" all over it (click to see full sized):



Finally, a little 2x2 chroma smooth and the MLVFS results look pretty good.



I've had to use a little bit of chroma smoothing in several images using MLVFS so this isn't something that's unique to the 100D crop_rec.

[Danne]

Strange. I use the map and it comes out like this(totally clean without chroma smooth added):

Code: [Select]

Loading focus pixel map '80000346_1808x726.fpm'...

crop:


Make sure you erase the top text row inside the file. Could be the problem. Noticed MLVFS(in Switch) was hanging otherwise.

Code: [Select]

#FPM 80000346 1808 726 1 -- fpmutil v0.5
78 86
86 86
...

[dfort]

I took bouncyball's image file and converted it to a focus pixel map file:



By the way, the "theoretical" map file I made a while back sort of works but why complicate things when the mv720 map is working fine--at least in MLVFS.

I'm still lost as to how you are doing that, "2 pass crop_rec mv720/mv1080 operation" - do you need a focus pixel map file that combines mv720 and mv1080 (which is how I made the "theoretical" map) or are you running a different pass with each map file?

Just for the record, it looks like crop_rec is recording a full raw buffer size of 1808x726 while regular mv720 shows a size of 1808x727. In MLVFS simply copy the map file that is in there and rename it to the crop_rec buffer size. Both files will be identical.

Code: [Select]

80000346_1808x726.fpm
80000346_1808x727.fpm
Yeah, not very efficient having two copies of the same file but that's the way MLVFS is set up. It does make it easier to experiment with new map files.

[Danne]

3rd map coordinates are listed in this order: first mv720 and then mv1080, I called it dual pass map. Actually the interpolation process is linear but logically according to this map, interpolating code does mv720 interpolation first and then mv1080(shifted/cropped) interpolation.

Seems all is done in one swoop but the it works the pixels in specified order from the file.

I took bouncyball's image file and converted it to a focus pixel map file:

Amazing there could even be a viewable picture in there.

[bouncyball]

Seems all is done in one swoop but the it works the pixels in specified order from the file.

Yes exactly. The map made from the graphical image will produce the ugly artifacts you see on the samples of dfort. But 2pass map has exactly the same coordinates but they are ordered differently in the fpm list and this works as expected. 1st goes all mv720 coordinated and then mv1080 (but cropped to mv720 size and shifted up by 30 pixels).

Honestly it's so simple! You could do this just by adding mv1080 coordinates at the end of the mv720 fpm (in this case you'll just end up with a lot of unused, out of the buffer, pixels because mv720 buffer is lot smaller then mv1080) and it would work except upper 30 pixel range.

The interpolating code takes one line at a time from fpm. It's only order of pixels matter and logically we get 2 passes.

example:

let's say mv720.fpm looks like this:

Code: [Select]

x    y
-------
100  1
150  2
200  3

and mv1080.fpm look like this:

Code: [Select]

x    y
-------
50   1
160  2
190  3

the map made from graphical image will be the fusion of both in this order:

Code: [Select]

x    y
-------
50   1
100  1
150  2
160  2
190  3
200  3

and 2 pass map will be the fusion of both in this order:

Code: [Select]

x    y
-------
100  1
150  2
200  3
50   1
160  2
190  3

This is the difference and it gives us logically 2 passes. Look at Y coordinates above.

regards
bb

[Danne]

Hehe, smart one.

[bouncyball]

And what's important - very simple and does not require any changes to the existing correcting algorithm.

[bouncyball]

Yeah, not very efficient having two copies of the same file but that's the way MLVFS is set up. It does make it easier to experiment with new map files.

Yes right and to bypass this in the 'MLV App' I integrated your pattern generators into the code . Works on the fly like a charm. Internal generator can be overridden by placing the map to the binary directory.

[bouncyball]

In the fpm.sh:

instead of this line:

Code: [Select]

if [ $pattern == "B" ]; then mv720; fi # crop_rec not yet available on the 100D

insert this code and it should work for 100D crop_rec:

Code: [Select]

if [ $pattern == "B" ]; then
   mv720
   fp_start=89; fp_end=724; x_rep=8; y_rep=10
   for j in $(seq $((fp_start)) $((fp_end)) ); do
      if   (( (($j +  0)) % y_rep == 0 )); then shift1=0; shift2=0
      elif (( (($j +  1)) % y_rep == 0 )); then shift1=1; shift2=1
      elif (( (($j +  5)) % y_rep == 0 )); then shift1=5; shift2=5
      elif (( (($j +  6)) % y_rep == 0 )); then shift1=4; shift2=4
      else continue
      fi
      output_row
   done
fi

regards
bb

[dfort]

Thanks!

Committed that change in fpm.sh and will experiment with the 2 pass map file in MLVFS.

[dfort]

Although the bb 2-pass map looks pretty good in MLVFS without having to throw in chroma smoothing:



There are some residual focus pixels only at the top of the frame:



I converted the map file for this image and it looks like this:



Danne and I worked hard on refining a crop_rec map for the EOSM and had problems getting it perfect. I have a feeling that using bouncyball's 2-pass map will help. Now getting back to those residual focus pixels, on the EOSM map the mv1080 focus pixels extends not only below but also above the mv720 focus pixels:



So it looks like if we add a few more rows at the top we'll be hitting those residual focus pixels at the top of the image.

[EDIT] After posting I noticed some people may be wondering about the blank area on the left. On a full raw buffer map it is "normal" to have a blank area on the top and left because this is out of bounds. In other words the image is never recorded to this area. Why? I don't know, maybe someone else can explain it.

[bouncyball]

There are some residual focus pixels only at the top of the frame:

Very strange because this map includes 30 pixel wide upper part of the image. What I mentioned In all my previous posts.

Here is the 16bit PNG sample file which is cleaned by this map.

[Danne]

No pixel residues on my end:




Are you using this map file? Erased top row?
http://nic.caucasus.net/fpm/?dir=&download=80000346_1808x726_dual_pass.fpm

[dfort]

this map includes 30 pixel wide upper part of the image. What I mentioned In all my previous posts.

Oops--I missed that when I tried to re-create your map. It does clean up nicely in MLVFS using your 2-pass map and doesn't need any chroma smoothing in MLVFS.



I'll commit this to the archives so I won't lose it. I'm having problems with some of my scripts that I use to convert fpm to pbm and back but once I get your map converted to an image file I'll save that too. Of course my scripts won't convert from an image file to a 2-pass map file so this will be a one way trip.

Just a few notes: Sometimes I get a crop_rec MLV that mlv_dump shows has a full raw buffer of 1808x726 and other times it is 1808x727. Danne took care of it by making my script accept a range of sizes but that isn't possible with MLVFS. Also, MLVFS only looks at the size of the full raw buffer to match up the map file so we can't have a separate maps for mv720 and crop_rec. I tried working around this by using the crop_rec map on mv720 but it caused problems (residual focus pixels) but with the 2-pass method maybe that will work? It isn't ideal to be hitting areas that don't have focus pixels but if it doesn't have an impact on the quality of the image or processing speed maybe that would be a way to get crop_rec working in MLVFS without having to swap out the map files?

[dfort]

Are you using this map file? Erased top row?
http://nic.caucasus.net/fpm/?dir=&download=80000346_1808x726_dual_pass.fpm

I was trying to make my own but missed the part about 30 pixels at the top. All good now.

Need some 100D mv720 clips to try my MLVFS theory--if you want to test just use bouncyball's crop_rec 2-pass map file on mv720 clips.