Dealing with Focus Pixels in raw video

Started by dfort, October 22, 2015, 11:09:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dfort

Thanks, I'll look into it this weekend.

dfort

Oh yeah, this is going to take a while.

1x3 focus pixels


1x1 focus pixels


The pattern is pretty much as expected. When I first started on figuring out focus pixels I thought the way to do it was to map the full sensor (1x1) and figure out the other resolutions by the binning/skipping patterns. It doesn't quite work out that way in practice so this is almost like starting all over again.

Of course you caught me right when I'm in the middle of several time consuming projects so any help I could get on this would be greatly appreciated.

dfort

Some more strangeness with these new 1x3 and 1x1 files.

In order to create the focus pixel map file I need to make an image file that shows the entire raw buffer. This is done with dcraw -4 -E. The resulting black and white image should be the same size as the full raw buffer reported in mlv_dump -v.

1x3
Image size = 1736x3478
Full Raw Buffer size = 1808x3508
Block: RAWI
...
    Res:  1736x3478
    raw_info:
...
      height           3508
      width            1808


1x1
Image size = 5208x3472
Full Raw Buffer size = 5280x3508
Block: RAWI
...
    Res:  5208x3472
    raw_info:
...
      height           3508
      width            5280


However, the dcraw trick doesn't work with these files. I can still figure out the focus pixel coordinates using the Crop and/or Pan information from mlv_dump so all is not lost but thought it was worth posting these findings.

dfort

@theBilalFakhouri - Got a 1x3 MLVFS focus pixel map file that works with your 700D MLV samples:

https://bitbucket.org/daniel_fort/ml-focus-pixels/src/default/focus_pixel_map_files/80000326_1808x3508.fpm

Test it out if possible--all your files cleaned up perfectly. However, there may be some focus pixels that show up in certain lighting conditions so give it a good workout.





Made this up from scratch. Still need to work on the 1x1 file then translate these to the fpm bash script and bouncyball's mlv-tools so the various mlv processing apps can use them.

If we can get these settings working on the 650D and EOSM this focus pixel map should work with those cameras. The 100D (and EOSM2) will need a different map file.

theBilalFakhouri

@dfort

Great work and well done! Now 1x3 files is cleaned perfectly even with Dual ISO files. The only small problem we have already knew it is the "height" value in "raw_info" or should I say the offsets. Using mlv_dump -v in 1736x2214 MLV file: (EDIT: of course the resolution was dropped in crop_rec not in mlv_lite, if it was from mlv_lite it will be no offsets)

Block: RAWI
...
    Res:  1736x2214
    raw_info:
...
      height           2244
      width            1808


By changing the 1x3 focus map name from 80000326_1808x3508.fpm to 80000326_1808x2244.fpm now the focus pixels will be cleaned nicely in 1736x2214 files. I have tried it in MLVApp by dropping the focus pixels map in the MLVApp.exe directory.

I think we should a trick for that like using the full focus pixels map (The original one) and makes MLVApp checking the height in raw_info and make a calculation for the offsets depending on MLV files info --> Hello @masc @bouncyball @Ilia3101 is this possible ? :D

Same thing with 1x1 Binning but I think this will affect the "width" value too.

Here are MLV files in 1736x2214 Normal and with Dual ISO: Download (The internet speed is bad, will upload ASAP Edit: Uploaded).

Quote from: dfort on October 15, 2018, 06:05:55 PM
However, there may be some focus pixels that show up in certain lighting conditions so give it a good workout.
Can you share a sample? I will make more tests in future.

Quote from: dfort on October 15, 2018, 06:05:55 PM
If we can get these settings working on the 650D and EOSM this focus pixel map should work with those cameras. The 100D (and EOSM2) will need a different map file.
Yes it should be the same in 650D and EOS M.

If I can help in anything please tell me how ;D .

Great work!

Danne

So cool dfort.
Man, Bilal, we need this thingy on the 5D mark III and the 100D. I am so irregular in my registry quests so it probably takes me 5 years to accomplish(if ever) what you do in a month  ;D.

theBilalFakhouri

@Danne

nikfreak has helped with 100D for first tries to increase resolution in x5 but he is offline for about 10 days. Can you help again @nikfreak  :D ? The only problem with me because I don't have these cameras and still I didn't know why it stuck okay let's have more tries to find out why :D

I will try to look into 5D3 too I think we should start from current crop_rec.c it has 1x3 preset. @a1ex is there bleeding edge version for crop_rec for 5D3? :P

dfort

Quote from: theBilalFakhouri on October 15, 2018, 08:23:01 PM
Can you share a sample? I will make more tests in future.

Not really, it is just that sometimes focus pixels pop up in places we haven't mapped. This map covers everything that shows up in the samples you provided.

As far as the 1808x3508 map working on the 1808x2244 raw buffer, that's lucky though not surprising. Looking forward to checking out that new file once you get it uploaded.

Got a few busy days coming up so don't hold your breath waiting for me to finish working on these new map files.

theBilalFakhouri

Okay I have updated the download link above. Take your time ;)

bouncyball



IDA_ML

Hello Dfort and Bouncyball,

As I promissed a while ago, here are a few shortened MLV files shot on the 100D at the high-resolution cropped modes in 14-bit lossless at the following resolutions:

2520x1304
3096x1320
4096x2280

https://we.tl/t-6PWhiEQK3i

The files, intended for the preparation of new focus pixel maps,  were shot with the 100D on a tripod at Normal and Dual ISO 100/800 modes - the latter of which providing very good results in term of low noise and high dynamic range on the 100D.  Therefore, Dual ISO is of great practical importance and I personally use it very often.

I used Danne's build from June 30, 2018.  From all 6 files in the above archive only the 2520x1304 Normal one seems to be free of focus pixels.  The other ones, especially the Dual ISO files, need cleaning, therefore new focus pixel maps for MLVFS, MLVDump, MLVProducer and MLVApp are necessary.  In the Dual ISO files, the focus pixels are most obvious on the vertical edges of high-contrast elements such as the cupboard doors.

Please let me know if the files are OK for you, if not I can shoot and provide additional ones.  Just let me know at what modes/resolutions do you need them.

The above link will be active for 7 days.

theBilalFakhouri

@dfort

It's working perfectly as expected, Thanks!

dfort

Quote from: IDA_ML on October 22, 2018, 12:21:54 PM
From all 6 files in the above archive only the 2520x1304 Normal one seems to be free of focus pixels.

Maybe the application you are using can already handle this file. Here's how I check for focus pixels and create a map file. Note that no programming skills are necessary just a basic understanding on how to use the terminal. You'll also need mlv_dump installed on your system along with some common command line tools -- bash, sed, file, dcraw and Imagemagick.

First use mlv_dump to make some dng files. Use the --no-fixfp option to make sure mlv_dump doesn't attempt to fix the focus pixels. (Click on the screenshots if you want to see them at full resolution.)



Tip: You can drag the mlv file into the terminal window so you don't need to enter the path and file name. I'm working in the same directory as the mlv files to keep the illustrations simple.

If you open one of these dng files in Resolve or ACR you might have a hard time seeing the focus pixels because these applications tend to smooth them out. In order to get them to pop we'll use dcraw to convert one of these dng files to tiff.



Open the tiff in Photoshop or Gimp. It should look fine when looking at the whole frame.



However, zoom in and punch up the saturation and vibrance.



It is rather easy to create a new layer and hit these focus pixels with the pencil tool but first we need to make some adjustments in order to line up the map file.

The way the apps work with the focus pixel map files is that they first look at the full image buffer size and determine where the image is located on that buffer using the crop and pan information. We can do the same thing in Photoshop.

We are going to adjust the canvas size (not the image size!) so we need to turn that background into a layer.



Next, make the canvas size the same size as the full raw buffer. Use this command to easily scroll through the mlv_dump -v output:

mlv_dump -v 100D_Normal_2520x1304.MLV | less

The full raw buffer size can be found in the RAWI block:



Adjust the canvas size so it is the same as the full raw buffer:



Next, the image needs to be moved around on the canvas so that it matches what is in the mlv file. Once again, using mlv_dump -v:



We're interested in the Crop. In many cases the Pan will have the same values but not always. For example, on the 700D full resolution 1x1 file it came out like this:

    Crop: 72x30
     Pan: 72x31


Ok--back to the one we're working on, move the image so that it is lined up according to the Crop data. Zoom all the way in on the upper left corner of the canvas:



Now we can make a new layer and start hitting those focus pixels with the pencil tool. Make sure the tool is adjusted to 1 pixel and black.



You'll quickly find out that there are lots of focus pixels and this is very tedious. You'll also discover that there is a pattern but if you try to copy this pattern to other areas of the image there might not be any focus pixels there or there may be focus pixels that aren't covered.



I have never seen an image that shows all of the possible focus pixels, however, by working over an area, looking for a pattern and extending it over the entire image area you will eventually cover all the focus pixels. The way I do this is by copying the layer then moving the copy to extend the pattern, merge the layers, make another copy -- rinse repeat until the entire image is covered.

You'll notice that with the settings that easily show focus pixels, like 14-bit lossless, the focus pixels are confined to a defined area. So why map out the entire sensor? We found out that settings that don't easily show focus pixels, like 8-bit lossless, focus pixels will appear in areas that are not confined to a defined area. They usually show up in high contrast boundaries. Since this is video and the image is constantly changing the focus pixels pop on and off and give a "sparkling" effect. Thus it is important to map out the entire image area.

Now make a new white opaque layer and place it under the layer with the focus pixels that you just made, zoom back a bit and you should see a definite pattern.



You'll note that the focus pixel layer extends to areas where the image ends. This isn't a problem though I prefer to remove the extra focus pixels that are in the out of bounds area of the full raw buffer.



Now save the focus pixels as a portable bit map file and name it with the camera model number and the full raw buffer size:



There are a few scripts and applications that can turn this pbm file into a fpm that can be used in MLVFS. I'm going to use my pbm2fpm.sh [EDIT] I renamed it to img2fpm.sh because if you have ImageMagick installed it can use any image file format -- not because it is the best or fastest but I wrote it and know how it works. Photoshop saves what are known as P4 portable bit map files which are binary files. This is a bit of a problem with bash so the script uses ImageMagick to convert it into a P1 ascii file. The script is rather slow though I have optimized it so it isn't as bad as it used to be. I prefer running it with the -v option to see the progress. [EDIT] The -v option is now the default. Use the -q (quiet mode) option if you are piping it through your own scripts.

Ok--here we go:



Now take that 80000346_2592x1332.fpm we just created, drop it in with the other map files in MLVFS and let's check to see if it is working.

Starting with a pixel peeping view of the normal file without fixing the focus pixels:



Here is how it looks like in MLVFS using the map we just created:



Looks good. How about the dual iso example?



Oh no! Did we somehow miss those focus pixels that are showing up? I put this frame into the Photoshop project and turned on the focus pixel map layer we just created:



Looks like those stray focus pixels that are showing up are mapped so what is going on? In order to fill in the focus pixels we need to average in the surrounding pixels. Because of the way dual iso works, the line above or below will probably be so different that averaging the area immediately above or below a focus pixel won't work so we can only average pixels on the horizontal axis. This means that when we hit a contrasty vertical line, there just isn't enough information to fill in the focus pixel. At least that's how I understand this works with MLVFS, other apps may be different.

The best we can do with MLVFS is to turn on some chroma smoothing:



Yes, a tiny bit is still there and some users say that they can see some reduced resolution with chroma smoothing on (I can't see it) but that's the limitation we've got to work with until someone figures out something better.

theBilalFakhouri

Quote from: dfort on October 22, 2018, 09:56:49 PM
Looks like those stray focus pixels that are showing up are mapped so what is going on? In order to fill in the focus pixels we need to average in the surrounding pixels. Because of the way dual iso works, the line above or below will probably be so different that averaging the area immediately above or below a focus pixel won't work so we can only average pixels on the horizontal axis. This means that when we hit a contrasty vertical line, there just isn't enough information to fill in the focus pixel. At least that's how I understand this works with MLVFS, other apps may be different.

I think in MLVApp the focus pixels cleared after applying Dual ISO in this case the image will look like normal MLV and it will be no problem. Not sure but depending in my tests @bouncyball is this right? So the focus pixels map is good but MLVFS way isn't that good.

dfort

I've got some more work to do in order to get this into the bash script so @bouncyball can update MLVApp. We might need to change from using the full raw buffer size to determine which map file to use to using the full raw buffer width along with the sampling. This should allow us to work with all the various new crop_rec resolutions.

IDA_ML

Thank you so much, Dfort, for this comprehensive review of the focus pixel issue.  I never thought it was so complicated.  Congratulations to the smart way you found to generate the focus pixel maps!

dfort

Quote from: theBilalFakhouri on October 22, 2018, 10:16:16 PM
I think in MLVApp the focus pixels cleared after applying Dual ISO in this case the image will look like normal MLV and it will be no problem.

It turns out that the example that I used already has a focus pixel map file that is in the latest MLV App so I thought I'd check that out.



I'd say that's a problem.

The download link should still be valid if anyone wants to try it out:

https://we.tl/t-6PWhiEQK3i

Quote from: IDA_ML on October 23, 2018, 08:56:55 AM
I never thought it was so complicated.

I tried to simplify it as much as possible. You don't need any coding skills to create a new focus pixel map file.

a1ex

Bilal's bug looks interesting:

Quote from: theBilalFakhouri on October 29, 2018, 12:17:23 AM
there is a bug in the generic code when using other PowerSaveTiming registers that are not necessary for 700D, keep ADTG 8172  8178  82b6 and comment out the others ADTG.
[...]
Also after commenting out these ADTG registers some new focus pixels showed in 1x3 Binning mode.

Look up the "others ADTG" on this graph, in particular, 8196 and 8197:

Quote from: a1ex on October 29, 2018, 05:38:24 AM


ADTG 8196 is set to 459 (0x12E NRZI).
ADTG 8197 is set to 759 (0x38C NRZI).
ADTG 82B6 is set to 1191 (0x6F4 NRZI).
ADTG 8172/87 are set to 1193 (0x6FD NRZI).

Notice the first two are roughly symmetrical around the middle of the frame. If you consider the 28px top bar, they become perfectly symmetrical:

(1192+28) / 2 - 460 = 150
(1192+28) / 2 - 760 = -150

Hypothesis: what if these are related to focus dots?

Test images (silent pictures from 1080p24, default LiveView configuration):

fp-dark.dng
fp-gray.dng

Setting ADTG 8197 to 596 (close to half of the frame), i.e. dialing 0x37E in adtg_gui, gives the following silent pictures:

fp-dark-8197-596-nrzi37e.dng
fp-gray-8197-596-nrzi37e.dng

To my untrained eye, the focus pixels are completely gone from the dark frame in the bottom half!

On the gray frame, they are still present, but apparently somewhat weaker...

Exact half of the frame should be at 610 (dial 0x353 in adtg_gui); not tested.

Somehow I've got a CR2 with focus pixels, too: fp.cr2

theBilalFakhouri

I have played with these two registers a while ago and found you can create patterns as you want  :P and other things related with focus pixels.

Danne

Tried tweaking ADTG 8196 and ADTG 8197 but seems it doesn´t do anything to the 100D image in regards to focus pixels. I view through "framing" mode which reveals all pixels in live view window.
Maybe the tweaking shouldn´t be randow so if any ideas/directions are very welcome.

dfort

Quote from: a1ex on October 29, 2018, 09:31:16 AM
Somehow I've got a CR2 with focus pixels, too: fp.cr2

Interesting:



Looks familiar.

I have found that the focus pixel pattern extends the full width of the sensor though they are visible only under certain lighting conditions. Some settings, like reduced bit depth lossless compression, will show focus pixels the full width and height of the sensor but they only show up on high contrast boundaries. That's why I've been extending my focus pixel maps over the full sensor.

If you can find a setting that makes focus pixels vanish, EOSM, EOSM2, 700D, 650D users will greatly appreciate it!

a1ex

So, these pixels have a slightly different response even in the CR2?

To capture that CR2:
- enable adtg_gui in photo mode
- take a picture
- lock all ADTG registers
- take another picture

Narrowed down to ADTG[8900]: setting it to 0 in photo mode reveals that pattern.

Another trick: in 1080p, CMOS[8] is set to 0x400 by default; clearing this bit makes the pattern less noticeable.

dfort

Quote from: a1ex on October 30, 2018, 09:22:45 AM
Another trick: in 1080p, CMOS[8] is set to 0x400 by default; clearing this bit makes the pattern less noticeable.

Wonder if something like that is happening with the reduced bit rate lossless compression settings. The focus pixels seem absent at first but upon closer inspection they show up on high contrast boundaries all over the image area--not just the areas where the focus pixels readily show up in 14-bit images.

I'm a bit behind on my homework but it seems that it should be possible to reduce the number of focus pixel map files if we allow the map file to cover the maximum possible height of the sensor. That's 3508 for 1x sampling and 1168 (theoretically, haven't tested) for 3x sampling. Remember these are full raw buffer sizes, the actual image size will be somewhat less.

a1ex

Quote from: dfort on October 31, 2018, 05:26:12 AM
Wonder if something like that is happening with the reduced bit rate lossless compression settings. The focus pixels seem absent at first but upon closer inspection they show up on high contrast boundaries all over the image area--not just the areas where the focus pixels readily show up in 14-bit images.

With reduced bit rate, we are using a different raw type (i.e. not the "minimally processed" Bayer stream we are used to). Just double-checked and had a little surprise - raw type 0x12 (what we use for reduced bit depth) is actually "lv_af_raw" (known to show this kind of dots even on the old 5D2). This is also labeled as DEFCORRE or DEFOE in some firmwares.

Best guess: the focus pixels must be corrected - to some extent - by Canon firmware. Are these dots always the same place? Maybe they are just the result of imperfect interpolation done by Canon?