Dealing with Focus Pixels in raw video

[dmilligan]

Where do I stick the 80000331_1872x1060.fpm file?

Updated the automator workflow:
https://bitbucket.org/dmilligan/mlvfs/commits/c1cadee9ab6d9bb20dacbe5a63d4ed213df901ab

so now you can just put the .fpm file inside here (where the mlvfs executable is):
~/Library/Services/MLVFS.workflow/Contents
and run mlvfs via the automator workflow like normal.

[dfort]

Great--works on my end now.

I see some pixels on the very edges that didn't get zapped so I'll see if they are on the pixel map. Then of course I'll test various resolutions and aspect ratios on the crop hack setting then move on to other modes and other cameras.

Thanks again!

[dmilligan]

I see some pixels on the very edges that didn't get zapped

Yeah, it skips pixels on the edge because there's not enough adjacent pixels to interpolate from (I was just sort of hoping there wouldn't be any that fall on the edge). I suppose it could just only use the pixels available. I'll make an update.

Interpolation is also very simple and only in the horizontal direction. It just averages the next pixel of the same color immediately to the left and right of the current pixel (technically, this is also the median since there are only 2 values ). The reason we only do horizontal is for dual ISO (which I would be interested to see some examples of to see if it works now without artifacts, it should).

[AWPStar]

mlvp code for interpolation if needed.

                i = y * sWidth + x
               
                ' Horizontal difference
                rz1 = Abs(dstt[i - 1] - dstt[i + 1]) + Abs(dstt[i - 2] - dstt[i + 2]) + Abs(dstt[i - 1 - sWidth ] - dstt[i + 1 - sWidth]) + Abs(dstt[i - 1 + sWidth ] - dstt[i + 1 + sWidth])
                ' Vertical difference
                rz2 = Abs(dstt[i - sWidth] - dstt[i + sWidth]) + Abs(dstt[i - w2] - dstt[i + w2]) + Abs(dstt[i - sWidth - 1 ] - dstt[i + sWidth - 1]) + Abs(dstt[i - sWidth + 1 ] - dstt[i + sWidth + 1])
               
                If rz1 < rz2 Then
                    dstt = (dstt[i - 2] + dstt[i + 2]) shr 1 ' /2
                Else
                    dstt = (dstt[i - w2] + dstt[i + w2]) shr 1 ' /2
                End If

The reason we only do horizontal is for dual ISO

I think if count on difference in vertical and horizontal you'll not get artefacts.


Original
Pixel removed
Processed

[dfort]

Interesting. A while back I posted a snippet of dcraw code that shows how Dave Coffin handled it:

Code: [Select]

/*
   Seach from the current directory up to the root looking for
   a ".badpixels" file, and fix those pixels now.
 */
void CLASS bad_pixels (const char *cfname)
{
  FILE *fp=0;
  char *fname, *cp, line[128];
  int len, time, row, col, r, c, rad, tot, n, fixed=0;

  if (!filters) return;
  if (cfname)
    fp = fopen (cfname, "r");
  else {
    for (len=32 ; ; len *= 2) {
      fname = (char *) malloc (len);
      if (!fname) return;
      if (getcwd (fname, len-16)) break;
      free (fname);
      if (errno != ERANGE) return;
    }
#if defined(WIN32) || defined(DJGPP)
    if (fname[1] == ':')
      memmove (fname, fname+2, len-2);
    for (cp=fname; *cp; cp++)
      if (*cp == '\\') *cp = '/';
#endif
    cp = fname + strlen(fname);
    if (cp[-1] == '/') cp--;
    while (*fname == '/') {
      strcpy (cp, "/.badpixels");
      if ((fp = fopen (fname, "r"))) break;
      if (cp == fname) break;
      while (*--cp != '/');
    }
    free (fname);
  }
  if (!fp) return;
  while (fgets (line, 128, fp)) {
    cp = strchr (line, '#');
    if (cp) *cp = 0;
    if (sscanf (line, "%d %d %d", &col, &row, &time) != 3) continue;
    if ((unsigned) col >= width || (unsigned) row >= height) continue;
    if (time > timestamp) continue;
    for (tot=n=0, rad=1; rad < 3 && n==0; rad++)
      for (r = row-rad; r <= row+rad; r++)
 for (c = col-rad; c <= col+rad; c++)
   if ((unsigned) r < height && (unsigned) c < width &&
  (r != row || c != col) && fcol(r,c) == fcol(row,col)) {
     tot += BAYER2(r,c);
     n++;
   }
    BAYER2(row,col) = tot/n;
    if (verbose) {
      if (!fixed++)
 fprintf (stderr,_("Fixed dead pixels at:"));
      fprintf (stderr, " %d,%d", col, row);
    }
  }
  if (fixed) fputc ('\n', stderr);
  fclose (fp);
}
I was able to remove the focus pixels on dual_iso with dcraw as noted in this post: http://magiclantern.fm/forum/index.php?topic=16054.msg157589#msg157589

The issue with dcraw to remove focus pixels is that a different pixel map is needed for different image sizes, it doesn't track 5x crop video and as far as I know it doesn't save to DNG with the fixed pixels.

@AWPStar I hope this topic is benefitting your 'MLVProducer for Windows' development. As you can see I was able to make image files of your pixel maps and if you compare the one you have for crop mode on the EOSM and 650D you'll see there are a few stray pixels in your maps that don't hit any focus pixel and there are also a few focus pixels that you missed. I wanted to convert my master map to your format but it seems that if you want to use the crop information embedded in the mlv metadata perhaps you should consider changing your pixel maps so 0,0 is at the upper left instead of in the center. Maybe that way we can also share the same focus pixel map files.

Taking a closer look at this first test with MLVFS I put a DNG on a layer over the master focus pixel map file in Photoshop. Here is a closeup of the focus pixels that appear at the top edge.

Adjusting the transparency to show the pixel map reveals that the pixels that were not removed are on the focus pixel map file. (Is that what .fpm stands for?)

I don't understand why these didn't get fixed because they are not on the very edge and from my understanding the interpolation is happening to the pixels on the immediate left and right.

Moving on to the left edge.

Those pixels are on the map but also at the very edge of the frame so it makes sense why they aren't getting interpolated. However, on the right edge:

There should be enough there to interpolate these pixels. Finally, on the bottom of the frame:

No problem here because there is enough space between the focus pixels and the edge of the frame.

The reason we only do horizontal is for dual ISO (which I would be interested to see some examples of to see if it works now without artifacts, it should).

So let's look at a dual ISO example. First without the 80000331_1872x1060.fpm:

I pushed the ACR settings to exaggerate the HDR look and cranked the Vibrance all the way so the focus pixels pop.

And this is what it looks like with the 80000331_1872x1060.fpm file inside of the MLVFS.workflow/Contents:

I'd say it is working!

Ok, there are still a few focus pixels visible on the left and top edge if you really look for them but dmilligan will fix that on his next update. 

[dfort]

Another test. This time shooting every possible combination of image size and aspect ratio in crop mode hack mode to stress test the focus pixel fixer on MLVFS.

With the EOSM the maximum possible image size is 1792x1026. The camera can record only a few seconds at that resolution but there are ways to shoot the maximum width and the maximum height by adjusting the aspect ratio. Here are the extremes.

1792x358

640x1026

I'm glad to report that the focus pixel map file in MLVFS is working on all combinations of image sizes and aspect ratios in crop hack mode. Moving on to other video modes and camera models.

[dmilligan]

I'll make an update.

https://bitbucket.org/dmilligan/mlvfs/commits/a387b44405bca3aa9528c976553af08a94e961e4

[AWPStar]

 @dfort

perhaps you should consider changing your pixel maps so 0,0 is at the upper left instead of in the center.

I'm all for it! But in this case i need(somehow to get Full sensor pixel maps for all cameras. And handle crop and high fps modes.

[dfort]

I'll make an update.

Damn you're good--and quick too!

The update worked perfectly on my end--no more stray pixels around the edges. Looks like we can check off CROP_MODE_HACK for the EOSM. This should also work for the 650D|REBEL T4i|Kiss X6i by renaming the EOS pixel map file (80000331_1872x1060.fpm) to the number that identifies the 650D (80000301_1872x1060.fpm).

5x crop mode is going to be an even greater challenge than the CROP_MODE_HACK. To be perfectly honest with you I didn't even know about 5x crop mode until I started using a 5D3, I didn't know other cameras didn't have the crop mode hack and didn't know what digital dolly was all about. There seems to be so much to ML that I don't know about. In any case, when 5x crop is centered the results should appear identical to the crop mode hack but the big difference between these modes is the ability to pan around the frame in 5x crop mode. The issue is that the raw buffer also moves around the frame.

I started out with the image centered and the size set to 1280 wide. Here's what it looks like with the full sensor pixel map overlaid with the raw buffer and the cropped video overlaid on top of that:

Taking a close look at the raw buffer notice that it almost covers all of the focus pixels on the EOSM except for a few on the far right side:
[EDIT: These images are taken from the CROP_MODE_HACK. The 5x crop mode raw buffer covers these pixels but there are some focus pixels to the bottom of 5x crop mode raw buffer that aren't covered. See Reply #115 for more details.]

That should give about the same results as the CROP_MODE_HACK set at 1280x720 but after recording a few seconds it stops because of skipped frames while the CROP_MODE_HACK can record until the card fills up. So why use 5x crop mode over CROP_MODE_HACK? Because you can pan around the image just like when you're checking focus using the 5x and 10x magnifier. When I first set this up I ended up running off the edge of my makeshift chart as noted in Reply #95 of this topic.

So what to do about this? I've got some ideas which I'll save for my next post.

@dfortI'm all for it! But in this case i need(somehow to get Full sensor pixel maps for all cameras. And handle crop and high fps modes.

Everything I'm doing is free for everyone to use. So far I'm not entirely sure the full sensor map is 100% accurate. That's because I'm getting slightly different sizes reported from a CR2 and a Full-res Silent Picture DNG. It is only 1-pixel different but of course that makes all the difference in the world in this case. The working focus pixel map file along with a test MLV file is posted on my bitbucket download area: https://bitbucket.org/daniel_fort/magic-lantern/downloads

[AWPStar]

Funny thing, all footages that i have with focus pixels have CameraModel=0.
Is it about old ml builds?

[DeafEyeJedi]

Stay Calm ... Foam Party!!!

I'm even more surprised by the 1 pixel differences ... Great work as always, D & D!

One day in the future there'll be those who've never heard of Focus Pixels ... That's how big of a deal this is.

This is HiSToRy to be made and you guys are part of this small step for all mankind!

[dfort]

Funny thing, all footages that i have with focus pixels have CameraModel=0.
Is it about old ml builds?

Maybe old builds, maybe it is raw instead of mlv? There was a recent commit that is making all of this work so I would suggest getting new samples. I can help you out with that. Give me a few days and I'll start posting some sample frames from various cameras and some more focus pixel maps.

One day in the future there'll be those who've never heard of Focus Pixels ...

That's like so much of the jobs that I have worked on--if you notice it that probably means that I made a mistake. (I worked in editing and post production.)

[AWPStar]

@dfort
I'm making little experiment.
i've made full frame pattern  And i want try to scale it to another crop modes.

(edit)
it's not working.

[dmilligan]

EOSM was missing a stub for a while and didn't record camera model correctly.

[dfort]

@dfort
I'm making little experiment.
i've made full frame pattern  And i want try to scale it to another crop modes.

(edit)
it's not working.

Here is a link to my master pixel map in PNG format. Note that I use simple Portable Bit Map files when working with it. It is still a work in progress so don't be surprised if I come up with a revised version of it. The main goal is to come up with a list with the coordinates of all of the focus pixels. It would be nice to have one full frame pattern that works with all cameras, image sizes and video modes but it doesn't work that way. Looking at your map it appears that you're trying to hit ever possible location where a focus pixel might show up by repeating the pattern over the entire sensor. It doesn't work like that. You are hitting lots of areas that don't need treatment and possibly degrading the image. In addition, you can't scale it. You might be able to line skip the pattern to come up with a different pattern but I haven't tried that yet.

It seems that the 'right' way to do this is to map each of the raw buffers which varies depending on the camera and video mode. The test files I posted include the 80000331_1872x1060.fpm file. It should also work with the 650D by simply renaming the file to 80000301_1872x1060.fpm. So far that's the only pixel map that has been tested and appears to be working. If you've been following along you'll know this is for the EOSM in crop hack mode video. I believe there will only be 3 focus pixel maps for each camera: crop hack mode, 5x crop mode and full frame mode. The software should be able to handle all possible image sizes from those three maps.

[dfort]

@dmilligan

Is there a preference to using a space or a tab as a delimiter in the focus pixel map files ".fpm" ? The reason that I ask is because they both seem to work but maybe there's an advantage to one over the other?


Today I posted CROP_MODE_HACK and 5x crop mode focus pixel map files for 650D and EOSM cameras. You can find them in my bitbucket download area: https://bitbucket.org/daniel_fort/magic-lantern/downloads

Inside of "2016-01-24_FocusPixelMaps_650D_EOSM_CropModes" you'll find:

Code: [Select]

80000301_1872x1060.fpm
80000301_2592x1108.fpm
80000331_1872x1060.fpm
80000331_2592x1108.fpm
80000331 = EOSM (tested)
80000301 = 650D (untested)
1872x1060 = CROP_MODE_HACK (finished)
2592x1108 = 5x crop mode (WIP)

The reason I'm marking 5x crop mode as a work in progress is because it might not be quite ready yet. The focus pixel map works fine as long as the image is centered but may not work when panning around the screen.

This is the mode that I didn't even know existed before I started this topic. The biggest difference between 5x crop mode and CROP_MODE_HACK is that you can pan around in 5x crop mode. It is a little tricky with the EOSM because this camera is the only one in the group with the focus pixel issue that doesn't have a magnifier button.

First of all press the INFO. button until you get to a Canon menu that looks like this:

Note that the camera is in Movie mode and in this case RAW Video (MLV) ON, 1280x432 Aspect Ratio 16:9. Now if you're math abilities is greater than mine you'll immediately note that 128x432 isn't 16:9. That's because in full frame video mode there is some line skipping. I haven't started on full frame video mode yet so let's just file that for future reference.

Touch the magnifying glass icon and a couple of boxes will appear on the screen.

The box in the middle shows the 5x crop area and the number "x1" over the solid white box on the lower right shows that you are viewing the full frame. Press the magnifying glass icon again and:

At this point you can press the INFO. button and find your way back the the ML overlays but the Canon and ML menus will be clashing with each other and the ML crop overlay will be missing so it is best to line things up and perhaps make marks on your screen when using crop mode. In any case, if you press the movie record button the video will be recorded in 5x video crop mode. All the screen overlays will disappear except for a small movie camera icon with a MB/s indicator when recording, I believe this applies to all MLV and RAW video modes.

You can press the magnifier once more and the screen will show "x10" magnification but the video will still be recorded in 5x mode.

The big difference between 5x crop mode and CROP_MODE_HACK is that you can pan around the entire sensor in 5x crop mode. Here I moved a bit to the left and up using the triangles at the edges of the screen.

Now before you get all excited and plan on doing shots that smoothly pan around the entire sensor be warned that there are issues with 5x crop mode. One problem I encountered is when recording 1280x720 5x crop mode it stops recording after a few seconds due to skipped frames while in CROP_MODE_HACK it will record continuously (provided your card is fast enough of course). In addition, I've been shooting test after test in 5x crop mode and it seems that even though you can pan to various positions even while recording, the results don't always match what you see on the screen.

Here is what the raw buffer looks like when laid over the entire sensor.

The reason the full sensor image looks green is because I've been experimenting with the RawTherapee trick @otherman used to show the Bayer pattern. This gives one more confidence check that the layers are lined up properly but the focus pixels are harder to find.

Switching over to the master focus pixel map it is obvious that a few focus pixels fall outside of the centered 5x raw buffer area.

(Erratum: I showed a 5x crop mode raw buffer in Reply #108 but it was actually a CROP_MODE_HACK raw buffer. The author apologizes for this error and hopes nobody was unduly harmed. )

The raw buffer doesn't cover the entire sensor yet in 5x crop mode you can pan around the entire sensor--how? The raw buffer also moves around the sensor. Here is a second raw buffer covering the upper left area of the sensor

Note that only the out of bounds area of the raw buffers overlap. So how many possible 5x raw buffer positions are there? How to handle the focus pixels all of these raw buffers? That's why the 5x crop mode focus pixel map file I posted is still a work in progress.

When I started on this project I thought that creating one master focus pixel map would cover all of the various video modes. Now I'm realizing that probably isn't possible unless there is a way to find the location of the raw buffer in relationship to the entire sensor.

An alternative is to find all the possible positions of the 5x crop mode raw buffer over the entire sensor and keep adding points to the focus pixel map file until all the possible focus pixels are covered. This isn't ideal because it will be interpolating pixels that aren't focus pixels but it will probably work.

Feedback anyone?

[a1ex]

For the 5x crop mode, I'm afraid the metadata regarding raw frame position relative to full image is not saved. The position of the raw buffer changes as you pan around (notice there is some slowdown when you move the focus box in certain locations - that's when the raw buffer changes its position).

An approximate (!) position can be found with focus_box_get_raw_crop_offset. A more exact position can be probably obtained from CMOS registers, the math is probably not straightforward, and the trick will be camera-specific (the meaning of those registers is not the same on all cameras).

So I'd say we have two options here:
- just give up and use chroma smoothing in this mode
- use a heuristic (pan the bad pixel map a few pixels around until you find a good match) - difficult to implement and error-prone, but with a bit of robustness (clever math), one can make it work in most practical situations.

[poromaa]

I should probably keep my mouth shut (so you can just disregard this in case I have missed some info), but can't you just reset crop position before filming in 5x crop mode. (the lens is more sharp in the middle anyway).
Alt. only support this feature at middle fixed crop mode...?

[dfort]

Well another option is to use CROP_MODE_HACK instead of 5x crop mode. That's probably what I would do because at the same 1280x720 image size recording is only limited by card size while with 5x crop mode it stops after a few seconds. Probably due to the much larger raw image buffer used by the 5x crop mode?

In any case, since I got myself so deep into this I thought I'd check to see what is going on by shooting Silent Simple frames while panning around the sensor. It turned out that there are "only" 12 possible positions for the raw buffer. Three across by four high. This screenshot of some raw buffers in Photoshop should make it clear.

Just for the sake of experimenting I'm thinking about making a second focus pixel map for 5x crop mode that includes all the possible focus pixel positions. Doing an interpolation on a pixel that doesn't require it isn't ideal but it might work better for dual iso than chroma smoothing. As far as anything requiring some clever math---

The height is completely off:
3477 + 52 = 3549 height -- Wrong!

3477 + 52 = 3529

[EDIT: @poromaa I just thought up of a use for panning around in 5x crop mode video mode--it can give you some of the capabilities of a PC (perspective control) lens. For example shifting up instead of tilting the camera up can keep vertical lines straight when shooting architecture.]

[a1ex]

Just for the sake of experimenting I'm thinking about making a second focus pixel map for 5x crop mode that includes all the possible focus pixel positions.

Indeed, this should work really well.

[dmilligan]

Agreed. And I just had an idea to help the "minimize the damage" for pixels that would be falsely accused of being focus pixels: don't change the value if it's already between the value of the pixel to the left and to the right. In fact, using this simple heuristic might identify given a set of focus pixels, which set is the correct one: set with the fewest false positives wins. Since there are only 12 possibilities, it should be easy to find a clear winner and quickly.

Can you send some samples from a couple different crop positions and corresponding maps.

[DeafEyeJedi]

This is just beautiful...

[dfort]

Can you send some samples from a couple different crop positions and corresponding maps.

Certainly. The centered 5x crop is the only one that I have finished and tested so here is the focus pixel map with a Simple DNG of my makeshift test chart.

https://bitbucket.org/daniel_fort/magic-lantern/downloads

Filename: 2016-01-25_5x_crop_centered.zip

Once the sun comes up I'll shoot a short MLV file with this setting.

I'm going to work on the focus pixel map that includes all possible 5x crop focus pixel positions today. It is still very early and I'm not quite awake yet but as soon as I have another 5x crop position ready I'll post it. My scripts run really slow and this is very tedious work but by the end of the day I hope to have the final 5x crop focus pixel map finished.

If anyone out there is following along and is thinking of trying this experiment--the trick is to do this on a very steady tripod and lock everything down. Even then an entire test has to be done at once because once you move the camera, let's say to change the card or battery, the alignment is shot and you have to start the test all over. I had to start over again several times, mostly because the results I was getting wasn't what I was expecting. The test that finally worked was to shoot literally hundreds of Simple Silent Pictures in DNG format starting from the top left corner scanning left to right like TV scan lines. I tried it using MLV format for the Simple Silent Pictures but every time it switched to another buffer position the image got corrupted. Also, panning around and shooting MLV video wasn't working as expected. Lots of unexpected things going on here but that's part of the process.

[dfort]

Ok--so I didn't even wait for the sun to come up to start testing. I leaned out my window and shot Dual ISO towards the sunrise with an 8mm fisheye lens. Here's a full frame Dual ISO CR2 just to get your bearings.

This is 5x crop mode centered with the MLVFS 80000331_2592x1108.fpm in place.

And without the focus pixel map file:

Close up of the focus pixels for dramatic effect:

Finally, moving the 5x focus box all the way to the top left corner:

This is so extreme that even without the focus pixel map file it misses all of the focus pixels because on the EOSM they are concentrated in the center of the sensor.

(the lens is more sharp in the middle anyway).
Alt. only support this feature at middle fixed crop mode...?

Yes, this lens isn't that sharp in the corners and for that matter not too sharp in the center when shooting wide open. I'll admit I was also a bit sloppy as I was hanging out the window in the dark. For all you gear heads out there I used a Rokinon 8mm T3.8 Cine UMC Fisheye CS II Lens with Nikon F Mount and Fotasy NIK-EOS M adapter.

I can see some creative possibilities using the corners of a fisheye lens for a sort of "Leaving Las Vegas" effect. There are certainly some other uses like the pseudo PC lens I mentioned earlier so let's support this feature no matter where on the sensor you put the 5x crop box.

I'll post the sample files when my script finishes processing the next focus pixel map file. (Slow running script and even slower Photoshop user.)

[dfort]

Ok--sample files and focus pixel map files from this morning's test can be downloaded from here:

https://bitbucket.org/daniel_fort/magic-lantern/downloads

2016-01-25_5x_crop_sample_files.zip

This has just the center and the top left corner MLV and .fpm files as shown in my previous post. Some interesting observations. The center raw buffer area map shows 8,622 positions and has almost all of the focus pixels that are on the sensor. The upper left raw buffer has only 390 positions and the 1280x720 active area doesn't contain any focus pixels.

I'm also starting to work with the SL1 / 100D. In order to map the entire sensor it seems logical to start with the 5x raw buffers and it looks like this camera has 21 possible raw buffers, three across and seven up. Makes me wonder if I should redo the EOSM test to make sure that it only has 12 possible raw buffers in 5x crop mode.

[EDIT: Just ran another test on the EOSM and this time also came up with 21 possible 5x raw buffers. Glad I caught it before merging all the focus pixels.]