Dealing with Focus Pixels in raw video

[dmilligan]

Don't worry about 'Crop' it's just 'Pan' rounded to the nearest 8x8 block (IDK why this is needed, it seems totally useless to me). What you need to focus on is 'Pan' it is the x,y position of the top left corner of the recorded image, relative to the actual full LV raw buffer for the current video mode. You can get the size of the full raw buffer from the raw_info structure contained within the rawi block.

FRSP is irrelevant. That's a totally different raw buffer from the LV raw one. When I say full raw buffer, I don't mean the full sensor, just the original "LV raw" buffer the recorded image was taken from.

Maybe it'll make more sense if I explain.


(0,0)
  ____________________________________________
  |                                         |^
  |                                         ||
  |     _______________________________     ||
  |     |\                          |^      ||
  |     | CropPos(x,Y)              ||      ||
  |     |                           ||      ||
  |     |                           |Rec    |Raw Buffer
  |     |                           |Height |Height
  |     |                           ||      ||
  |     |                           ||      ||
  |     |                           |V      ||
  |     |__________________________ |_      ||
  |     |<---- Recorded Width  ---->|       ||
  |                                         |V
  |_________________________________________|_
  |<----------Raw Buffer Width------------->|


How raw video works:
1. We find the memory location and dimensions of a buffer (called the LV buffer) that is created and continuously filled with sensor data by the Canon firmware. The dimensions of this buffer vary based on the current video mode (and this is what is limiting your resolution choices). This is the outer square pictured above.
2. User selects some resolution that is no larger than this buffer (this is where your limits come from) and divisible by 32*.
3. If the selected resolution is smaller than the raw buffer, ML selects some 'pan' position that approximately centers the recorded area within the raw buffer (the user can move this position around while recording for a 'digital dolly' effect).
4. Recording started.
5. Each VSYNC (when each frame is ready), ML directs some specialized hardware called "EDMAC" to copy the rectangle of our choosing (the inner rectangle pictured above), to a memory area controlled by ML (and then eventually on to the CF/SD card interface).

How simple silent pictures work:
1. We find the memory location and dimensions of a buffer (called the LV buffer) that is created and continuously filled with sensor data by the Canon firmware. The dimensions of this buffer vary based on the current video mode.
2. We copy the whole thing to a file.


So if you map the focus pixels using a simple silent picture, then for a particular raw video frame (of the same video mode) the position of the focus pixel is simply:
FP_raw(x)=FP_silent(x) - PanPosX
FP_raw(y)=FP_silent(y) - PanPosY

Note that you may get negative values or values larger than W x H, this just means there's a focus pixels outside of the recorded frame, and you just need to skip it.

*the reason it must be divisible by 32 is that the EDMAC needs the image data to be aligned on 4-byte word boundaries. Since the image data is packed 14 bit and is stored in 'row major order' (row by row) in memory, many pixels' values start in the middle of a byte or word and we can't copy parts of bytes and copying in the middle of a word (even if on a byte boundary) is unreliable/inefficient due to the nature of the ARM CPU and EDMAC.

[dfort]

Wow what a fantastic post. Thank you so much!

I never knew that simple silent pictures varied depending on the video setting. I thought that the still picture and movie settings didn't affect one another. Looks like I've got a lot to learn.

So now it makes sense why I saw a difference between the two methods of shooting crop mode video on the EOS-M. Comparing the RAWI block between the two modes:

Code: [Select]

CROP_MODE_HACK 1280x720
Block: VIDF
     Pan: 330x181
Block: RAWI
    Res:  1280x720
      height           1060
      width            1872
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872


5x Zoom crop center 1280x720
Block: VIDF
     Pan: 585x335
Block: RAWI
    Res:  1280x720
      height           1108
      width            2592
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1102
      active_area.x2   2592

So far all of the CROP_MODE_HACK mlv files that I have checked have the same raw buffer size. I'll check all of them just to make sure but that would make things very simple because there is no way that I know of to do a "digital dolly" using the CROP_MODE_HACK so only one mlv frame needs to be checked for cropping. Working it out with the 5x digital zoom will be more work but it should be doable.

So if you map the focus pixels using a simple silent picture, then for a particular raw video frame (of the same video mode) the position of the focus pixel is simply:
FP_raw(x)=FP_silent(x) - PanPosX
FP_raw(y)=FP_silent(y) - PanPosY

I've got a problem with simple silent picture when checking crop mode video because it looks like simple silent picture does some line skipping and crop mode video doesn't skip lines. At least that's how I understand it, got to look into it some more.

[dfort]

More tests, more data to absorb, more lessons learned. I'll start off by addressing my own ignorance:

I've got a problem with simple silent picture when checking crop mode video because it looks like simple silent picture does some line skipping and crop mode video doesn't skip lines. At least that's how I understand it, got to look into it some more.

I was switching the camera to still photo mode to shoot a simple silent picture. Doing that put the camera out of crop mode so I got the vertically squashed/horizontally stretched image which turned out to be the raw buffer for non-cropped raw video. I thought I'd just concentrate on one video mode at a time but it looks like I'm now juggling three modes. The good news is that it looks like there are only three raw buffer sizes to deal with, one for each video mode. I still need to continue testing the various resolutions but so far only these three are showing up.

Code: [Select]

1798x1026 CROP_MODE_HACK
2518x1074 5x crop mode
1734x693 standard non-crop video mode
Those dimensions are actually in the RAWI Block but it isn't apparent right away. For example on a CROP_MODE_HACK mlv there is:

Code: [Select]

      height           1060
      width            1872
       active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872

RAW BUFFER(x) = active_area.x2 - active_area.x1
RAW BUFFER(y) = active_area.y2 - active_area.y1

In the above example:

1798 = 1872 - 74
1026 = 1054 - 28

Which is exactly the dimensions I got from a simple silent picture in CROP_MODE_HACK. This makes sense (sort of) because the maximum possible resolution in this mode is 1792x1026--not that you can actually shoot raw video at this size because of the write speed limit. Now is this really the raw buffer size? I'm not too sure because the pan information in the VIDF would put it way off center. For example, for a 1280x720 image centered inside of a 1798x1026 frame the upper left CropPos(x,y) PanPos(x,y) should be 259,153 but the VIDF Block has it at 585x335 330x181.

[EDIT: Correction, we're talking about pan not crop and I copied the wrong numbers in this post.]

Now that I've got a good image file of the focus pixels I'm thinking about loading some of the test shots as layers in Photoshop and seeing just where the focus pixels line up. Maybe that will help make sense of these numbers.

[dmilligan]

PanPos(X,Y) could actually be the number of bytes from the top left, not the number of pixels. If this is the case, then just multiply by 8/14 and see if you get numbers that make sense/line up.

[dfort]

You mean like this?

PanPos(X)

Code: [Select]

echo 330*8/14 | bc
188
PanPos(Y)

Code: [Select]

echo 181*8/14 | bc
103
Closer but still far off center.

Using Photoshop I lined up the a simple silent still (a.k.a. raw buffer) with a crop hack movie frame using 16:9 and 1280x720 and came up with a PanPos(X,Y) of 262,152. That's with the images and the focus pixel pattern lined up so I'm pretty sure this is correct. Note that the Photoshop pixel ruler starts at 0 as does the graphic in Reply #50 so I'm pretty sure this is the correct PanPos(X,Y).

Of course this assumes that the image size in Photoshop is the same as what mlv_dump is using and I'm not sure that's a correct assumption. I found some interesting information by running mlv_dump and exiftool on the raw buffer and the mlv video images that I lined up in Photoshop.

Simple Silent Picture (a.k.a. raw buffer)

Code: [Select]

Settings:
1280x720
16:9
Crop Hack

Photoshop = 1798 x 1026

mlv_dump

      height           1060
      width            1872
      pitch            3276
      frame_size       0x0034FCB0
      bits_per_pixel   14
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872
 
   Crop: 0x0
     Pan: 0x0

exiftool

Rows Per Strip                  : 1060
Default Crop Origin             : 0 0
Default Crop Size               : 1798 1026
Active Area                     : 28 74 1054 1872

Crop movie

Code: [Select]

Settings:
1280x720
16:9

Photoshop = 1280 x 720

mlv_dump

      height           1060
      width            1872
      pitch            3276
      frame_size       0x0034FCB0
      bits_per_pixel   14
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872

    Crop: 336x184
     Pan: 330x181

exiftool

Rows Per Strip                  : 720
Default Crop Origin             : 0 0
Default Crop Size               : 1280 720
Active Area                     : 0 0 720 1280

So maybe the raw buffer is actually:

(active_area.x2 + active_area.x1) x (active_area.y2 + active_area.y1)
(1872 + 74) x (1054 + 28)
1946 x 1082

That would put a 1280 x 720 image centered at:
(1946 - 1200) / 2 x (1082 - 720) / 2
(666) / 2 x (362) / 2
333 x 181

That's a lot closer to mlv_dump:

Code: [Select]


     Pan: 330x181


[a1ex]

From source code:

Code: [Select]

    uint16_t    cropPosX;    /* specifies from which sensor row/col the video frame was copied (8x2 blocks) */
    uint16_t    cropPosY;    /* (can be used to process dead/hot pixels) */

Why 8x2?

- when copying raw data, you can only do so starting from a multiple of 8 pixels (14 bytes). Why? see the definition of struct raw_pixblock.
- to keep things simple, all the code in ML assumes a Bayer order of RG;GB, so you should always skip an even number of lines

Therefore, for focus pixel correction, you are interested in these two offsets.

Code: [Select]

    uint16_t    panPosX;    /* specifies the panning offset which is cropPos, but with higher resolution (1x1 blocks) */
    uint16_t    panPosY;    /* (it's the frame area from sensor the user wants to see) */

These were meant to be used for digital dolly at pixel-level resolution; however, I don't remember this feature being actually implemented. You do not need them - in theory (there is a bug, see below).

Now, cropPosX/Y are actually skip_x/y rounded to multiples of 8:

Code: [Select]

    hdr->cropPosX = (skip_x + 7) & ~7;     /* same as (skip_x + 7) / 8 * 8, using integer math */
    hdr->cropPosY = (skip_y + 7) & ~7;
    hdr->panPosX = skip_x;
    hdr->panPosY = skip_y;

How are those skip offsets used when copying the raw buffer?

Code: [Select]

    edmac_copy_rectangle_cbr_start(ptr, raw_info.buffer, raw_info.pitch, (skip_x+7)/8*14, skip_y/2*2, res_x*14/8, 0, 0, res_x*14/8, res_y, &mlv_rec_dma_cbr_r, &mlv_rec_dma_cbr_w, NULL);

So, clearly they are pixels, not bytes, and they are the top-left corner of the image.

Notice cropPosY does not respect the specification. It should be: hdr->cropPosY = skip_y & ~1. That's a bug.

To work around for existing MLV files, assume cropPosY = panPosY & ~1. That should do the trick.

So maybe the raw buffer is actually:
(active_area.x2 + active_area.x1) x (active_area.y2 + active_area.y1)

Sorry, this is nonsense. Raw buffer size is raw_info.width x height. Active area is the part that contains useful image data. What's outside active area is optical black area. Pitch is width in bytes.

To get the entire raw buffer from a DNG, including OB area: dcraw -4 -E
To get the active area only: dcraw -4 -D

Hope this helps.

[dfort]

Thanks for this.

So with 1280x720 crop mode dcraw -4 -E on the simple silent picture resulted in a 1872 x 1060 image (full raw buffer) and dcraw -4 -D resulted in a 1798 x 1026 image (active area).

Photoshop, mlv_dump and exiftool are all in agreement about this and I'll stop my nonsense of trying to change these dimensions. I'll continue focus pixel hunting tomorrow with the new build.

[DeafEyeJedi]

Sounds like an excellent plan and thanks @a1ex for pointing us into the right direction.

[dfort]

Today I took the silent picture showing the full raw buffer, not just the action area, using dcraw -4 -E as instructed by a1ex and put that into Photoshop then added another layer with a frame of 1280x720 crop hack mode video. I lined up a makeshift test chart, verifying that the focus pixels also line up and presto:

Note that the full 1872 x 1060 raw buffer is showing the inactive area on the top and left of the image, so much for my assumption that the active area is centered on the full buffer.

Taking a close look at where the top left corner of the video frame lines up with the image of the full raw buffer:

336x180

This matches up perfectly with the Crop information reported by mlv_dump.

Code: [Select]

    Crop: 336x180
     Pan: 330x181

Note that this is after yesterday's commit by a1ex. The numbers mlv_dump was reporting before the fix were:

Code: [Select]

    Crop: 336x184
     Pan: 330x181

Of course in order to fix the focus pixels it needs to be exact so it looks like everything is working properly now.

More work to do but finally moving forward. Thanks @a1ex and @dmilligan for the expert advice.

I do have a thought to throw out there. I'm not much of a coder and what I'm doing is writing bash scripts in order prototype something that would work with dcraw in order to eliminate the focus pixels but wouldn't it be best to zap the focus pixels in mlv_dump or MLVFS or whatever program is used to create DNG frames from the mlv original? That way the dot-free DNG frames can be used in DaVinci Resolve or Premiere or whatever the user's choice transcoding/editing/grading program. I believe that the old PinkDotRemover tool 650D actually modified the original raw video files. Maybe that's something to consider?

[a1ex]

I think it makes sense, and probably should be enabled by default for the cameras that need it.

The video mode can be probably identified by (uncropped) raw buffer size - hopefully there aren't video modes with identical sizes, but different maps.

Offtopic: one may also create an overlay in ML that displays only these focus pixels, as a small bitmap. There may be useful info inside, and could probably be used for focus confirmation, or as a manual focus hint (to tell you in which direction you should turn the focus ring, for example).

[DeafEyeJedi]

Holy Moly ... That's some serious work you got going on, Dan!   

Offtopic: one may also create an overlay in ML that displays only these focus pixels, as a small bitmap. There may be useful info inside, and could probably be used for focus confirmation, or as a manual focus hint (to tell you in which direction you should turn the focus ring, for example).

+1

[Walter Schulz]

650D at your service. Direction advice needed.

[dfort]

650D at your service. Direction advice needed.

Fantastic Walter! For starters make sure you're on the latest Nightly with the mlv_rec fix in it. Then shoot a short 1280x720 mlv using crop-mode from the ML menu and run on that mlv file:

Code: [Select]

mlv_dump -m -v [filename.mlv]

Report back with the image dimensions from "Block: RAWI" and "Block: VIDF" from one of the frames. For example, I'm getting this on a 1280x720 crop mode video.

Code: [Select]

Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 212.350000 ms
    Res:  1280x720
    raw_info:
      api_version      0x00000001
      height           1060
      width            1872
      pitch            3276
      frame_size       0x0034FCB0
      bits_per_pixel   14
      black_level      2047
      white_level      15000
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1

Block: VIDF
  Offset: 0x06c02280
    Size: 1616864
    Time: 3365.025000 ms
   Frame: #0070
    Crop: 336x180
     Pan: 330x181
   Space: 4032

The important stuff in "Block: RAWI" is Res: and the active area height width information and in "Block: VIDF" I'm looking for the Crop: and Pan:
[EDIT: I originally pointed out the wrong fields--oops!]

That would be great for starters. Later I might ask you for some other raw modes and resolutions. By the way, can you get into crop mode using either the ML menu or zoom in with the magnifying glass button on the 650D? That was a new trick for me, didn't know the EOSM could do that. The two crop mode variations have different raw buffer sizes.

The video mode can be probably identified by (uncropped) raw buffer size - hopefully there aren't video modes with identical sizes, but different maps.

That's one of the tests I want to run--shoot every resolution in every raw mode just to make sure.

Offtopic: one may also create an overlay in ML that displays only these focus pixels, as a small bitmap. There may be useful info inside, and could probably be used for focus confirmation, or as a manual focus hint (to tell you in which direction you should turn the focus ring, for example).

Those focus pixels should good for something. I never get a shot that shows all of the focus pixels in one frame, sometimes they don't even appear at all. Maybe there's some sort of a hidden message in there? Anyway, for now I'll be happy if I can get rid of them reliably--there are reports that sometimes they come back in areas where the highlights are clipped. There's another test to run.

[Walter Schulz]

Report back with the image dimensions from "Block: RAWI" and "Block: VIDF" from one of the frames.

Canon Setting 1920x1080@30. ML started with default settings -> MLV_REC.mo enabled -> RAW Video (MLV) resolution set to 1280x720.


First attempt: Crop mode by Zoom-In button

Code: [Select]

Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 164.857000 ms
    Res:  1280x720
    raw_info:
      api_version      0x00000001
      height           1108
      width            2592
      pitch            4536
      frame_size       0x004CB060
      bits_per_pixel   14
      black_level      2047
      white_level      15000
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1102
      active_area.x2   2592
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: VIDF
  Offset: 0x03476260
    Size: 1616896
    Time: 1646.340000 ms
   Frame: #0034
    Crop: 592x334
     Pan: 585x335
   Space: 4064
Second set: Zoom in using ML menu "Movie crop mode ON"

Code: [Select]

Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 192.627000 ms
    Res:  1280x720
    raw_info:
      api_version      0x00000001
      height           1060
      width            1872
      pitch            3276
      frame_size       0x0034FCB0
      bits_per_pixel   14
      black_level      2047
      white_level      15000
      active_area.y1   28
      active_area.x1   74
      active_area.y2   1054
      active_area.x2   1872
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: VIDF
  Offset: 0x0140f260
    Size: 1616896
    Time: 1037.776000 ms
   Frame: #0011
    Crop: 336x180
     Pan: 330x181
   Space: 4064


[dfort]

Thanks Walter. Looks like the 650D and the EOSM match. That was expected. Could you run exiftool on one of your CR2 files and see what the full sensor size is? The EOSM is 5280x3528.

[Walter Schulz]

Sensor Width                    : 5280
Sensor Height                   : 3528
Sensor Left Border              : 84
Sensor Top Border               : 64
Sensor Right Border             : 5267
Sensor Bottom Border            : 3519

Original Image Width            : 5184
Original Image Height           : 3456

Image Size                      : 5184x3456

[dfort]

Perfect! Here's the EOSM:

Code: [Select]

Sensor Width                    : 5280
Sensor Height                   : 3528
Sensor Left Border              : 84
Sensor Top Border               : 64
Sensor Right Border             : 5267
Sensor Bottom Border            : 3519

Original Image Width            : 5184
Original Image Height           : 3456

Image Size                      : 5184x3456

Thanks Walter. I'll get back to testing on Monday and will let you know if there's anything else to check out but for now we can assume that both the 650D and EOSM will be the same. Now there's the 100D and 700D to check out too. Any other cameras with the focus pixel issue?

[Walter Schulz]

Any other cameras with the focus pixel issue?

70D, I suppose.
EDIT: Oops ... Dual-Pixel AF, I see.

[otherman]

Made some test exporting not-demosaicing picture taken with "simple" Silent Picture whit Eos M (using rawtherapee, and photoshop to view single color channel), and it seems that focus pixels involved are only the red ones, maybe some blue ones, but not green. I'm tryng to figure out if the influence on the blue ones is some sort of "ghost" due to my test condition.

image here: https://drive.google.com/open?id=0B2JqnkAmnNAZTEhjNkZrdHlYdTg

[a1ex]

it seems that focus pixels involved are only the red ones, maybe some blue ones, but not green.

That's right - it is the reason chroma smoothing from mlv_dump fixes those pixels (it does median filtering on r-g and b-g, and keeps green channel unchanged).

[dfort]

Made some test exporting not-demosaicing picture taken with "simple" Silent Picture whit Eos M (using rawtherapee, and photoshop to view single color channel), and it seems that focus pixels involved are only the red ones, maybe some blue ones, but not green. I'm tryng to figure out if the influence on the blue ones is some sort of "ghost" due to my test condition.

Interesting discovery. Exactly how did you do this?

Looks like a focus pixel at 1076,372 and 1080,378. I haven't made a pixel map for non-crop on the EOSM yet so I can't verify it.

That's right - it is the reason chroma smoothing from mlv_dump fixes those pixels (it does median filtering on r-g and b-g, and keeps green channel unchanged).

I've had cases where chroma smoothing didn't get rid of all of the focus pixels. It also doesn't seem to work well when removing focus pixels from dual_iso video. Are there any issues using chroma smoothing over the entire image besides longer processing time?

Something else that came up is that in order to get this working automatically the camera model needs to be read from the metadata. mlv_dump is having a problem with the 700D. Can anyone verify this?

mlv_dump -m -v [700D_mlv_file]

Code: [Select]

Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 0.004000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000

Hopefully this is just an old file and is fine on the latest nightly build.

Another issue is the dng files created by mlv_dump is showing the wrong information for "Unique Camera Model" on the EOSM and 100D:

exiftool [file.dng]

Code: [Select]

Camera Model Name               : Canon EOS REBEL SL1
Unique Camera Model             : Canon EOS 650D

Camera Model Name               : Canon EOS M
Unique Camera Model             : Canon EOS 650D

Note that this is only an issue when the dng's are created with mlv_dump. MLVFS dng's show the correct Unique Camera Model.

[otherman]

Chroma smoothing over the entire image means some loss in contrast and "real" resolution, imho, and wrong results in dual iso.
For my test, I take a SP in "simple" mode of my room wall; in rawtherapee I've selected "none" for demosaicing filter, than exported as png; in Photoshop, simply selected single channels and made considerations by eyes.

[Oswald]

I have 100d and i can help with developing this. What I can do?

[dfort]

I have 100d and i can help with developing this. What I can do?

Great! Do you know how to use mlv_dump and exiftool? What country are you in--or more precisely what country did your camera come from?

For starters I'd like to find out if your camera is identified as "Canon EOS REBEL SL1" or Canon "EOS 100D" and if you can run the same tests that Walter ran that would be a good start. Of course the big deal is mapping all of the focus pixels so if you shoot a blank wall or paper and bracket your exposure you should find a few frames that clearly show the focus pixels. Set your camera to movie mode and shoot "simple silent pictures" at various video settings. The 100D is a fairly new port so there's no features listing--does it have the crop mode hack?

[nikfreak]

yes it features crop mode hack