Dealing with Focus Pixels in raw video

Started by dfort, October 22, 2015, 11:09:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dfort

Quite a bit has happened since starting this topic. If you want to cut to the chase and want the quickest way to eradicate focus pixels simply download MLV App and load in the latest focus pixel map files. This video shows how to install the map files into the app:

https://youtu.be/fYqSavtita0

The rest of this is post for historical (hysterical?) purposes:

Quote from: dmilligan on October 21, 2015, 03:10:12 PM
@dfort,
One possibility is mapping the focus pixels for each camera and then interpolating out those pixels (rather than just chroma-smoothing the entire image). This might fix the dual ISO / focus pixel issue, IDK. It should be faster than chroma-smoothing, and not have any artifacts. It's a lot of work though, you need to create a focus pixel map for each camera and video mode.

So here we go--

Affected cameras: EOSM, 100D, 600D, 650D, 700D, (and more?)
[EDIT: Pretty sure now that the 600D doesn't show focus pixels in raw video.]

Focus pixels, phase detectors, call them what you will when these dots show up on your raw video they are annoying. There have been attempts at clearing them up like the PinkDotRemoval tool that has apparently been abandoned. Chroma smoothing helps but it affects the overall image and it doesn't work for dual iso footage.

I tried to shoot something that shows as many of these pixels as possible by shooting an out of focus white paper with a couple different colors of lights and processing the image with a heavy hand to show the pixels as clearly and dramatically as possible.


Got your attention? This is a frame from an mlv file shot on an EOSM in crop movie mode. It doesn't look quite as bad when shooting regular subjects but it is a problem.


It is possible to get an even clearer view of these pixels by running a frame through dcraw using the -d option which, according to the documentation:
Quote-d
Show the raw data as a grayscale image with no interpolation.

Here is a closeup of that same frame at the top of this post.


Once we know the coordinates of these pixels they can be zapped.

Method 1 - In Camera

The ideal way to deal with them would be in the camera. In fact there is a way that "bad pixels" can be mapped out and apparently it is common practice to map out these faulty pixels when the camera is manufactured or repaired--saves on reject sensors. Assuming that we can't access the code that maps out pixels in the Canon firmware, the developers over at the chdk project have figured out more than one way to do it: http://chdk.wikia.com/wiki/Badpixel_removal

I'm not skilled enough to take what they have done and apply it to Magic Lantern but maybe someone who has the knowledge can drop a few hints?

Method 2 - In Post

Yeah, we can fix it in post. In fact there are several programs that can deal with pixel problems. Pixel Fixer is a dedicated tool, RawTherapee has an option to deal with hot and dead pixels and an interesting Python wrapper for the LibRaw library called rawpy that creates a ".badpixels" file dcraw can use to zap out pixels. This might be a way to create the database of focus pixel maps dmilligan suggested at the top of this post.

I tried using dcraw to get rid of one of the focus pixels on a dual iso frame and found that one pass wouldn't do it. Maybe it is because the process is breaking down by running the file through cr2hdr first? My understanding is that the debayering process is averaging neighboring pixels and in dual iso it seems to "smear" the pixel vertically. In addition, some of these focus pixels have a secondary pixel diagonally across that also needs to be mapped.

Here is what I ended up doing to get rid of just one focus pixel--and the diagonal secondary pixel.


1247 121 0
1248 120 0
1248 121 0
1248 125 0
1248 126 0
1248 122 0
1248 123 0
1248 124 0
1248 126 0
1248 122 0
1248 120 0
1248 124 0


Note that I hit the diagonal pixel just once but had to work top and bottom to middle and hit the same pixels several times in different sequences. Think of it as stippling a brush to spot a print--in photography school I spent many hours spotting prints. (By the way, this a map of all the focus pixels for the 650D that was used on the PinkDotRemover tool.)

Here is the result. I circled the area where the hot pixel was located and yes, it looked just like those other ones before running the dng through dcraw.


So if this is the solution then what's the problem?

The problem is that I'd like to get some feedback that this would actually be practical. It looks like there would have to be a different focus pixel map file for every camera and every crop factor (raw image size) and all of this would have to be assembled in a way that users could handle. Danne is the master of assembling scripts that can automate complex processes so it is possible to piece together something with dcraw--but then again is that the best solution? Maybe it would be better to adapt dcraw's "badpixels" code to mlv_dump or cr2hdr or MLVFS?

Just asking for a little guidance here.

DeafEyeJedi

Practical? This is a stepping stone in History. Probably by far from becoming one of the most sought-after script of the decade, if not the century. No kidding here, Buddy!

This means it would make some of us users with heavy dependent on PDR like our $200-$700 Toys feel like $1-2K, if not more!

There may have been a few run-in's with certain threads in the past on this Forum if I can recall correctly on top of my head but this is by far...

Probably the 2nd Method is the most ideal, BEST, most practical option there is and looking forward to jump on board with this yet another adventurous project of yours, Daniel!

Two-Thumbs up...  8)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Quote from: DeafEyeJedi on October 23, 2015, 04:12:54 AM
Practical? This is a stepping stone in History. Probably by far from becoming one of the most sought-after script of the decade, if not the century. No kidding here, Buddy!

I could use some cheering--things didn't work out that well today. I struggled with rawpy and managed to write out a list of pixels with this:
import rawpy.enhance
import numpy

paths = ['image_000000.dng', 'image_000001.dng', 'image_000002.dng']
bad_pixels = rawpy.enhance.find_bad_pixels(paths, find_hot=True, find_dead=True, confirm_ratio=0.9)
# rawpy.enhance.save_dcraw_bad_pixels(path, bad_pixels)
numpy.savetxt('pixels.txt', bad_pixels, '%5.0f')


Note that I never did get the "rawpy.enhance.save_dcraw_bad_pixels(path, bad_pixels)" to work. Lots of searching turned up that rawpy uses numpy arrays so I figured out how to save that to a file but the list was huge! Checking the dng's that I used it looked like there were lots of false positives. I probably could have mapped out the pixels manually in the time it took me to get this far today.

It seems that the first thing to do is to map out the focus pixels on a "regular" raw file and make sure that works and all the focus pixels get mapped before trying to figure out what is going on with dual iso. Right now I don't think that dcraw is the right tool for taking out focus pixels on dual iso files because the way I understand it the pixel is filled in with the average of the adjacent pixels. That would mean that it would be trying to average the value of pixels that were exposed at different iso values. The pixels to average need to be on the same x axis, in other words the pixels to the left and right of the focus pixel.

DeafEyeJedi

Quote from: dfort on October 23, 2015, 04:21:44 AM
Note that I never did get the "rawpy.enhance.save_dcraw_bad_pixels(path, bad_pixels)" to work. Lots of searching turned up that rawpy uses numpy arrays so I figured out how to save that to a file but the list was huge! Checking the dng's that I used it looked like there were lots of false positives. I probably could have mapped out the pixels manually in the time it took me to get this far today.

I second that and definitely should be worth the effort to do this manually. Any chance I could help? I miss spending time on Photoshop. lol

Quote from: dfort on October 23, 2015, 04:21:44 AM
It seems that the first thing to do is to map out the focus pixels on a "regular" raw file and make sure that works and all the focus pixels get mapped before trying to figure out what is going on with dual iso. Right now I don't think that dcraw is the right tool for taking out focus pixels on dual iso files because the way I understand it the pixel is filled in with the average of the adjacent pixels. That would mean that it would be trying to average the value of pixels that were exposed at different iso values. The pixels to average need to be on the same x axis, in other words the pixels to the left and right of the focus pixel.

Bummer about dcraw though it makes perfect sense. I still have a good feeling about this possibility one way or another.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

PaulHarwood856

     I'm not sure if I have the exact same issue, but I have had my 7D record hot pixels in raw video. This of course happens in H.264 as well if not using the sensor cleaning trick for a while (I believe this is a common problem with the 7D - occurs with the film school I go to with their 7Ds, and less of a problem with my T3i). The trick is doing manual sensor cleaning with just the body cap on and pointing down for thirty seconds. But what's nice about raw video is when I do get hot pixels, I am able to remove them with "Remove Chromatic Aberration" in Adobe Camera Raw (or Lightroom probably). I am completely aware not everyone can afford these programs, especially with Adobe Creative Cloud, but it is possible to buy Lightroom 5 on disc I believe for a reasonable price.

     I hope this is helpful. It seems you are trying to eliminate the problem in the first place, which is great, just figured I would inform you and the forum of how I remove the hot pixels. Hope all is well!

dfort

Quote from: PaulHarwood856 on October 23, 2015, 05:50:53 AM
     I'm not sure if I have the exact same issue, but I have had my 7D record hot pixels in raw video.

Not the same issues--you have "real" hot or dead pixels. We're talking about the focusing pixels that show up on certain models. That list doesn't include the 7D.

dfort

This morning I mapped the focus pixels for the EOSM 1280x720 crop mode MLV. I had to do it manually row by row and ended up with over 6,000 pixels mapped. It seems to be working for all the images that I have tried. Here's a frame from the camera before processing:


Looks ok until you take a closer look:


Running it through dcraw with the -P option pointing to focus_pixels_EOSM_1280X720_crop_mode.txt takes out the focus pixels.


Yeah, I lost some resolution when it was saved as a portable pixmap format (PPM) which is the default for dcraw.

So the next challenge is going to be removing pixels on dual iso.


This was shot at the same time but the focus pixels can't be removed either before or after processing. I'm not sure that doing the "stippling" method from my original post is feasible or even desirable. Looking at the dcraw code maybe (?) there is a way to average the pixels to the left and right of the focus pixels (x axis) and not use pixels that are on a different line (y axis) because that would be averaging in different iso values.

Snippet from dcraw.c:
/*
   Seach from the current directory up to the root looking for
   a ".badpixels" file, and fix those pixels now.
*/
void CLASS bad_pixels (const char *cfname)
{
  FILE *fp=0;
  char *fname, *cp, line[128];
  int len, time, row, col, r, c, rad, tot, n, fixed=0;

  if (!filters) return;
  if (cfname)
    fp = fopen (cfname, "r");
  else {
    for (len=32 ; ; len *= 2) {
      fname = (char *) malloc (len);
      if (!fname) return;
      if (getcwd (fname, len-16)) break;
      free (fname);
      if (errno != ERANGE) return;
    }
#if defined(WIN32) || defined(DJGPP)
    if (fname[1] == ':')
      memmove (fname, fname+2, len-2);
    for (cp=fname; *cp; cp++)
      if (*cp == '\\') *cp = '/';
#endif
    cp = fname + strlen(fname);
    if (cp[-1] == '/') cp--;
    while (*fname == '/') {
      strcpy (cp, "/.badpixels");
      if ((fp = fopen (fname, "r"))) break;
      if (cp == fname) break;
      while (*--cp != '/');
    }
    free (fname);
  }
  if (!fp) return;
  while (fgets (line, 128, fp)) {
    cp = strchr (line, '#');
    if (cp) *cp = 0;
    if (sscanf (line, "%d %d %d", &col, &row, &time) != 3) continue;
    if ((unsigned) col >= width || (unsigned) row >= height) continue;
    if (time > timestamp) continue;
    for (tot=n=0, rad=1; rad < 3 && n==0; rad++)
      for (r = row-rad; r <= row+rad; r++)
for (c = col-rad; c <= col+rad; c++)
  if ((unsigned) r < height && (unsigned) c < width &&
(r != row || c != col) && fcol(r,c) == fcol(row,col)) {
    tot += BAYER2(r,c);
    n++;
  }
    BAYER2(row,col) = tot/n;
    if (verbose) {
      if (!fixed++)
fprintf (stderr,_("Fixed dead pixels at:"));
      fprintf (stderr, " %d,%d", col, row);
    }
  }
  if (fixed) fputc ('\n', stderr);
  fclose (fp);
}

dmilligan

I would suggest mapping the focus pixels from a silent DNG taken in each video mode. That way the coordinates are relative to the full, entire raw buffer including OB areas. That way you can use the same map for anything taken in that video mode, and simply adjust it based on the resolution crop and horizontal and vertical offset (which are already recorded in the MLV file)

dfort

Thanks, I'll do that--once I figure out what you just said. Haven't done anything with silent DNG's yet.

The still photos don't show the extent of the problem so I thought I'd shoot something for dramatic effect. Believe me the video looked much worse before YouTube compressed it.



EOSM 1280x720 Dual ISO crop mode MLV. I opened and closed the aperture because the pixels change appearance depending on the color and brightness of the image. The pixels dance around on dual iso video so it looks like we're going to have to hit a moving target!

dfort

Looks like this is going to be a popular project!



I've got mixed emotions about that.  :) :'( ;) :-* :D :-\ ;D :-X >:( :-[ :( :-[ :o ::) ???

Ok, seriously--It looks like it will be possible to eliminate the focus pixels in post. One thing that needs to be done is to map the focus pixels on the various cameras that are affected. I was able to do it for the EOSM at one resolution but it was a very tedious process doing it manually and prone to error. Any suggestions on how to do this more reliably and less painfully?

By the way, I got called out of retirement to fill in for a couple of months so I won't have as much time as I'd like to work on this. It doesn't mean I'm going to disappear -- I'm still going to put in a few hours here and there trying to figure out how to make these pixels disappear but I could use some help.

DeafEyeJedi

Don't let your sense of humor run dry... EVER!

TBH I didn't even know that either Tapatalk or ML has a like button similar to FB? lol

I'm off this weekend and would like to get my hands tied down to remapping the Focus Pixels from my SL1 -- can this be done in Photoshop, right?

If so, I'll reply to your recent PM about this very matter.

Otherwise we can just stand by and said for others to chime in better "suggestions" on how to speed up the process.

Thanks, D! [emoji1]
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

I got you a testversion of folder_ProRes444_lut with an included dead_pixel list feature.

DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Whoo Hoo--Danne's folder_ProRes444_lut test version is working!

Well, almost. So far it only works for one camera (EOSM) at one resolution (1280x720) not on dual_iso yet and we got some kinks to work out but the results are encouraging.

DeafEyeJedi

5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

How did you manage to map all of those pixels? Amazing. Thanks for nailing down the correct path to the dcraw command. Works correct now so will fix so it works with previews and stuff. Of course you have to do manual pixel mapping but there won,t be 6000 dots on all sensors to map out ;).

dfort

Basically, I started with shooting a flat gray frame and use "dcraw -b [name of dng file]" to get the pixels to stand out. Then in Photoshop adjust the brightness and contrast and with the "Units & Rulers" preference set to Units, Rulers: Pixels and Type: Pixels start hunting for the focus pixels. You have to be zoomed in all the way to measure individual pixels. The pixels that repeat in a pattern are the focus pixels. They were always spaced 24 pixels apart on the EOSM so I made a spread sheet and worked it out row by row. I also found that there were some rows that went all the way across the frame and others that were only in the center so first I mapped the ones that went completely across the frame and made a "badpixel" file then mapped out the ones in the center. After I thought I had them all I tried running the "pixel fix" on various frames and found several rows that didn't show up on the file I was using. It is tricky to find all of them because their visibility depends on exposure, contrast, color, etc of the image. Some rows also seem to change intensity from one side of the frame to the other so it becomes sort of a game of hide and seek to hunt them all down. I might have mapped a few points that aren't really focus pixels, I'm not really sure but it doesn't seem to be hurting the final results.

Looks like another how-to article. First I want to make sure I map all of the pixels on the EOSM, right now I've only got the ones in the crop mode area.

Then using dmilligan's suggestion interpolate the focus pixels needed map the various image size options. That's still a lot of work to do.

Oh, and by the way--I'm calling them focus pixels but I'm not really sure what they are.

One more thing, even cameras that don't have the focus pixel issue might have hot, dead, whatever, pixels that need to be removed. Maybe even the old hair in the gate issue that we used to have in the film days might benefit from this.

dfort

@dmilligan -- it would be great if you could share some of your knowledge on the following. At least let me know if I'm going in the right direction.

Quote from: dmilligan on October 24, 2015, 01:01:13 AM
I would suggest mapping the focus pixels from a silent DNG taken in each video mode. That way the coordinates are relative to the full, entire raw buffer including OB areas. That way you can use the same map for anything taken in that video mode, and simply adjust it based on the resolution crop and horizontal and vertical offset (which are already recorded in the MLV file)

My first thought was to shoot a full-resolution silent DNG and try to map all the focus pixels for the full sensor but the focus pixels didn't show up on any of the full-resolution frames that I shot. Maybe the firmware mapped them out? The focus pixels did show up at different resolutions but the spacing between them varies. I think that might be because of the line (and row?) skipping at various resolutions.

With Danne's help I was able to extract just the metadata from an MLV file using mlv_dump:

mlv_dump -v -m [mlv file] > meta.txt

So now I've got to figure out what this means:
    Crop: 336x184
     Pan: 330x181
   Space: 4064


If I understood how line/row skipping works and was better at math maybe mapping all of the focus pixels then predicting where the focus pixels would land at various resolutions would be feasible. For now, it takes me less time shooting a few frames at each resolution then using Photoshop and a spreadsheet to map them out one row at a time for each resolution. The dcraw -d option also helps to hunt down the pixels.

The dancing focus pixels in dual_iso video has become a little less mysterious. Here's a closeup of some focus pixels before processing in cr2hdr.


The horizontal lines aren't static, they appear to be marching up the frame thus giving the appearance that the focus pixels are jumping around. After processing in cr2hdr or MLVFS the pixels appear to be "smeared" vertically. See one of my early posts for an example of this. I think that dcraw is averaging the surrounding pixels when the "badpixel" file is being processed but this isn't working with dual_iso shots. It seems to me that only the pixels to the immediate left and right of the focus pixel should be used to fill in the "bad" focus pixel and this should probably be done before processing in cr2hdr. Does that seem right? Maybe run it though a version of dcraw that only averages by columns and not rows first and then feed the output into cr2hdr? Is that possible?

I posted a snippet of the dcraw source code but on closer examination it looks like it is the section that reads the ".badpixels" file but not the section that fills in the "bad" pixels. I'm not very experienced with code and not really sure where to look but the dcraw source code is published on the Dave Coffin's dcraw website. Any hints on where to look?

DeafEyeJedi

Hello Folks-

Well where do I start with this? I was intrigued by the idea that @dfort shared this concept that there may be a possibility that SLR/100D uses the same Focus Pixel mappings on their sesnors as the EOS-M as well as other Rebels... At least they are supposed to be somewhat similar.

So off I go to dig into this wonderful world of Focus Pixelation which feels like a War of Terror that it never ends. Or at least it seems so.

I shot a test footage from SL1 in 24p 1280x720p MLV in both Non-Crop & Crop-Mode to see how much of the Focus Pixels disappear after converting with @Danne's incredible app (cr2hdr-r/ProRes_Folder service workflow) that included @Dfort's Pixel_Fix tags which supposedly does the magic for you. Let's see if it does with this test footage I shot.

(Mind you I shot several Dual-ISO takes just for the heck of it because I knew those Focus Pixels wouldn't disappear from Dual-ISO MLV at least not yet atm) Here it is:

https://vimeo.com/144091012

Here are the screenshots:

Pre:



Post:



Terror and Behold ... Where did those Focus Pixels go?

Simply Amazing.

It is definitely a step closer to perfection and I am happy with the progress!

Excellent work once again by both Daniel & Danne on their never-ending contributions ... Now I reckon that many of you are probably wondering if we could do the same for the other Rebels?

Well you can help us by providing us short samples of MLV's (1 second is plenty) in every possible resolution with both Non-crop and Crop-mode from your 600D, 650D, 700D, 750D, etc... It would definitely speed up the process!

Chime on in...
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Is that a row of pixels on the very bottom of the before frame? Looks like that needs to be mapped.

Also see a few showing up in the highlights. Need to see if those pixels are in the list of if maybe they need a little extra treatment in post.

DeafEyeJedi

Yep. You got that right. Also noticed some in highlights as well and I think this can be fixed in post but not sure if the mapping needs adjustments to get these killed as well?

This was shot on a 6.5mm 3.5 lens, in case you were wondering.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

I was hoping that all the cameras with the focus pixel issue could share the same map file but it looks like this isn't going to work. At least not for the 700D. Here's what I could pull out of a couple of different image settings:

700D non-crop mode 1728x1152


700D crop mode 1280x720

This is going to be challenging just trying to find the pattern. Quite a difference from the EOSM focus pixel pattern from my first post.

dfort

I've been testing Danne's new MLPP workflow and he discovered that it works with the focus pixel map file on dual iso mlv files. Quite a surprise. I thought dual iso was causing issues with this.



Compare this with the same shot run without the focus pixel map file.



Yes, I know that the color and contrast don't match between these clips. I used different processes to produce the videos.

nikfreak

[size=8pt]70D.112 & 100D.101[/size]

Danne

That is great stuff dfort. Greatful for adding the dual iso fix (--no stripe fix) To those challenging dual iso files, as well.