Dealing with Focus Pixels in raw video

[a1ex]

There are two flavors of 10-bit:

- uncompressed, obtained by telling the image processing hardware to simply trimming the lowest bits;
  Canon's image input/output modules can be configured for 10,12,14 or 16-bit uncompressed Bayer data.

- compressed - lossless, but we can't / don't know how to configure the encoder to output 10-bit (but 10-bit input works);
  workaround: restricting the useful levels in a 14-bit container does the trick;
  this is done by darkening the image using digital gain, operating on integer math.

[theBilalFakhouri]

I used both MLVProducer 3203v (a quicker way to see focus pixels, remove it and whatever)
and orginal MLV Dump & on steriods version. Playing with maps and comparing the results.
as a1ex said
yeah I am using MLVProducer too to see bit-depth by Numbers(black-white level maybe) very nice and fast for Couples MLV files with some darkening image issue(at 12  11-8bit lossless it hadn't solved yet)

This another MLV Samples at 14  12  11-8bit lossless to compare
https://drive.google.com/open?id=1rhTS1i4pbqi6lLceIT6Tp3B4Ci52k_WM
and it's looks same problem in mv720 with crop_rec
thank you hard workers!

[dfort]

Made a new 80000326_1808x1190.fpm file that should cover up those pixels that are showing up on 12-bit lossless files. Problem is, I'm not seeing an improvement when using MLVFS. Try it out on your system and let me know if it works for you.

The SourceTree app is showing this in the diff:



I'll also prepare a new mv720 map file and take a look at the other video modes. If it works I'll also need to update the script and pass along this information to the MLV app developers.

[theBilalFakhouri]

It's the same problem no improvement yet; only the middle was cleared.
I am using mlv_dump_on_steroids same as MLVFS method.
Any reason why?

[bouncyball]

Because none of RAW data from 700D had focus pixels all over the image until now.

@dfort is that mv720 mode?

[dfort]

@dfort is that mv720 mode?

That one is mv1080 mode, haven't gotten around to the other modes yet. I'm having problems getting it working with MLVFS.

It is easy enough to extend the pattern so that it covers the entire sensor. It shouldn't hurt the other video modes so we probably don't need to complicate things by checking to see if the MLV file was recorded in 12bit and applying a new map file.

[bouncyball]

@dfort:
Got that contrasty pattern MLV. It has raw "height 1189" and for MLVFS '.fpm' have to be named "80000326_1808x1189.fpm". Is there other reason why MLVFS does not work?

@Danne:
Btw this contrasty MLV from @theBilalFakhouri is the very good example why MLVFS interpolation method is better then RAW2DNG. Try yourself switching '--fpi' method and you'll see a big difference

[Danne]

Still looks like shit over here 

[dfort]

@dfort:
Got that contrasty pattern MLV. It has raw "height 1189" and for MLVFS '.fpm' have to be named "80000326_1808x1189.fpm". Is there other reason why MLVFS does not work?

Thanks, keep forgetting about that because your app doesn't need it. Still having some problems but I'll have to work on it later.

[bouncyball]

Still looks like shit over here 

I did not say it cleans all up (because this is unusual case). I said see the difference where pixels are cleaned (believe me it's HUGE, lot of uncleaned focus dots/artifacts in dark areas of this "cage" pattern with RAW2DNG)

[bouncyball]

Thanks, keep forgetting about that because your app doesn't need it. Still having some problems but I'll have to work on it later.

Good luck

[DeafEyeJedi]

I dig your work @theBilalFakhouri and best of luck to @dfort!

[dfort]

No solution yet but I'm getting closer. It looks like the focus pixel pattern we've been using on the 700D works fine for the middle section but extending that same pattern so that it covers the entire sensor didn't work. This might be a case similar to what we've seen on the crop_rec pattern for the EOSM.

In the process of trying to figure this out I found other issues so instead of working this out by myself I'm going to document the process I went through.

Got that contrasty pattern MLV. It has raw "height 1189" and for MLVFS '.fpm' have to be named "80000326_1808x1189.fpm". Is there other reason why MLVFS does not work?

According to the current mlv_dump from the crop_rec_4k branch the height is 1188:

Code: [Select]

    Res:  1736x976
    raw_info:
      api_version      0x00000001
      height           1188
      width            1808

However, mlv_dump on steroids shows something different:

Code: [Select]

mlv_dump_on_steroids

    Res:  1736x976
    raw_info:
      api_version      0x00000001
      height           1189
      width            1808

Under the guidance of @a1ex and @dmilligan I measured all the full raw buffer sizes a while back and mv1080 for the 700D measured 1808x1190. I assumed that the full buffer size never changes -- but apparently it does and now we've got three possible sizes? Luckily all of them seem to line up from the upper left (0x0) so we don't need to change the map files, just the names.

Since I'm testing with MLVFS which map file is it using? In order to find out let's run MLVFS from the command line. This is how it is done on the Mac:

In your home directory, create a folder called MLVFS_TEST
Place the mlvfs executable in this folder.
Place your fpm files in this folder

In your home directory create another folder called MLV_TEST
Place your MLV files in this folder.

In your home directory create another folder called MOUNT_TEST

Type EXACTLY this in the console

Code: [Select]

cd ~/MLVFS_TEST
./mlvfs ~/MOUNT_TEST --mlv-dir=~/MLV_TEST -f -o local -o volname="TEST" -o allow_other


It needed some action like opening one of the DNG files in Adobe Camera Raw and this showed up in the terminal:

Code: [Select]

Loading focus pixel map '80000326_1808x1189.fpm'...
Todo -- Shoot some simple silent DNG's using the various modes and measure the size of the full raw buffers. The DNG's need to processed through dcraw like this in order to include the out of bounds zones:

Code: [Select]

dcraw -4 -E *.dng
-or- [EDIT] This won't get you the full raw buffer size.

I specified:

Code: [Select]

dcraw -D -E -T INPUT.DNGSeems to work and we get a tiff instead of a pgm file. Output came out all black with -4 so skipped that.

Last time I tried measuring the full raw buffers I was getting different images sizes in Photoshop and exiftool so I'm still not sure what to believe. [EDIT] On my latest tests they are showing the same size.

Question -- shouldn't the full raw buffer height alway be an even number? Otherwise wouldn't that mess with the Bayer pattern? Maybe the raw buffer itself hasn't changed, just the way it was either rounded up or down to report it as an even number of pixels?

Ok--let's get back to that test shot @theBilalFakhouri sent. I lined up the DNG with the new map file I created and saw this:



What we are seeing a close up of the area between the middle where the focus pixel map (the black pixels) lined up and the upper area where it didn't. Seems like the focus pixels are in a different position and possibly a different pattern. [EDIT] My bad. I didn't line up the map file properly when I extended it to cover the top and bottom areas.

How do I know that I lined up the focus pixel map properly? The offsets can be found using mlv_dump  with the -v option:

Code: [Select]

Block: VIDF
  Offset: 0x0018c400
  Number: 14
    Size: 1620480
    Time: 175.773000 ms
   Frame: #0001
    Crop: 72x118
     Pan: 72x118
   Space: 32

The upper left corner of the DNG lines up with 72x118 on the full raw buffer image file. In this case the Crop and Pan values are the same but if they are different use the Crop. (or is it the Pan? ugh--that's why we should save our notes.) If anyone wants to help figure out new focus pixel map files, all the tools you should need are in my ML Focus Pixels Bitbucket repository.

[Danne]

Heavy. Tip. You can run tests via Switch too and you'll see the focus pixel maps loading in terminal. Simply add the new maps into mlvfs folder inside Contents folder inside Switch.
Then run (ml) from main menu.

[dfort]

Another tip, fpm.sh has a -v (verbose) mode:

Code: [Select]

./fpm.sh -v Contrast\ Edge.MLV

Camera:               700D
Camera Model:         80000326
Resolution:           1736x976
Video Mode:           mv1080
Full Raw Buffer Size: 1808x1188
Pan:                  72x118
Crop:                 72x118
Output File:          700D_mv1080_1736x976.txt
File Format:          dcraw

Writing file
Finished

I've got mlv_dump from the crop_rec_4k branch installed as my system default. mlv_dump on steroids already has the focus pixels algorithm. 

[Danne]

And I'm using fpmutil(bouncyball) as well

[theBilalFakhouri]

MLVFS interpolation method is better then RAW2DNG. Try yourself switching '--fpi' method and you'll see a big difference

RAW2DNG interpolation method it's look like only soften the pixels dots didn't remove it In many different shots it wasn't good enough as MLFVS method.

Todo -- Shoot some simple silent DNG's using the various modes and measure the size of the full raw buffers.

I didn't have all this knowledge yet to do this things. So maybe I can shoot silent DNGs and upload it?

[dfort]

So maybe I can shoot silent DNGs and upload it?

The problem with shooting silent DNG's is that I don't think you can save a 12-bit lossless compressed simple silent still. That's what you are using, right?

Seems all the testing was done on 14bit/10bit and not enough attention to 12bit lossless. That should be the sweet spot for the 700D. It is what the Black Magic Pocket Cinema Camera is using, right?

In any case, I've got a 700D so I should be able to get this figured out--eventually. There's a lot of dots in thar!

[theBilalFakhouri]

So right, Good luck 

[bouncyball]

According to the current mlv_dump from the crop_rec_4k branch the height is 1188:
...

Code: [Select]

Loading focus pixel map '80000326_1808x1189.fpm'......
Question -- shouldn't the full raw buffer height alway be an even number? Otherwise wouldn't that mess with the Bayer pattern? Maybe the raw buffer itself hasn't changed, just the way it was either rounded up or down to report it as an even number of pixels?

It's bad because getting height of the raw buffer is so straightforward. This value physically stored in the RAWI header's 'raw_info' part. If mlv_dump from 4k branch reports 1188 then it just rounds it to the lower even number before printing log. I have to look at the code.

[bouncyball]

Found this piece of code in latest mlv_dump from 4k branch:

Code: [Select]

/* well, it appears to happen that MLVs with odd sizes were written, restrict that */
block_hdr.raw_info.height &= ~1;
block_hdr.yRes &= ~1;

Doing this is just wrong (misleading information) because the change only applies to the MLV exported from mlv_dump and to the log printed to the screen, original MLV indeed has original odd number. The correct way to "restrict" it, is implementing this restriction to mlv_rec/mlv_lite modules.

[dfort]

Thanks @bouncyball -- things are starting to clear up. About a year ago @Danne noticed that the full raw buffer heights have changed so he did this adjustment to the focus pixel script. Since mv1080 went from 1190 to 1189 or 1188 we're allowing for a huge range of possibilities in the script:

Code: [Select]

case $raw_buffer in
-  1808x1190)
+  1808x11**)
Add a comment to this line
     video_mode=mv1080
I've been shooting simple silent DNG's on the 700D and keep getting 1808x1189 as measured in exiftool and in Photoshop after processing with dcraw. (By the way, just found out that saving to a Portable Bit Map file with the -4 and -E options shows the full raw buffer size but saving to a tiff does not.)

@theBilalFakhouri -- Almost there! I cleaned up the top. Of course I've got to do the same for the bottom part.

[a1ex]

The problem was introduced when trying to autodetect resolution from registers, rather than hardcoding it for each video mode. The method did not give exact results; they were sometimes off by one. Previously, we took the resolution from EDMAC configurations, but these were harder to autodetect with EDMAC_RAW_SLURP - because in this case we had to set up the EDMAC at the right resolution, rather than taking it for granted.

Back then, having exact resolution didn't seem important; at least nobody noticed it when reviewing this PR.

Later, shortly after crop_rec_4k was announced for 5D3, this adjustment was required for 8...12-bit lossless modes - otherwise, only every other frame was correct. You'll see the issue right away if you comment out the height adjustment. However, if you reduce the autodetected value by 2 pixels or a bit more, all frames will still be correct. So, it's critical to have the EDMAC resolution not higher than what the sensor is set up to deliver (but apparently it can be a little lower).

Now, why did I not adjust all the other models in the same way?

Resolution autodetection is inconsistent - sometimes this method gives the EDMAC resolution + 1, other times - 1 and other times the exact value. Here's a script that scans the ROMs for possible resolutions (pattern matching):

scan_raw_res.py and results: raw_res.txt.

Interpretation:
- some models give exact values (with exceptions - some hidden video modes are configured differently):
  6D, 100D, EOSM2
- most models have EDMAC resolutions configured at the one from capture registers + 1 (vertically)
  1100D, 1200D, 1300D, 600D, 60D, 650D, 700D, EOSM
- one model has EDMAC resolutions configured at the one from capture registers - 1 (vertically)
  5D3
- not analyzed: old DIGIC 4 models with LVState.

So... rather than hardcoding resolutions in each video mode (as we did some time ago), looks like we still have to hardcode a resolution adjustment (besides column_factor, which probably shows how many columns are read out in parallel).

Notice the EDMAC (used for transferring 14-bit Bayer data into RAM) is configured sometimes at even resolutions and sometimes at odd resolutions - in Canon code.

[dfort]

Oh yeah, I remember that pull request. Probably wasn't looking that closely at the focus pixels at the time but I did do a pull request for the EOSM on top of that pull request.

Couldn't resist running that script on the cameras affected by the focus pixel issue--assuming the EOSM2 also has the issue:

Code: [Select]

for f in */ROM1.BIN; do echo; echo $f; python scan_raw_res.py $f | sort | uniq; done

100D.101/ROM1.BIN
C0F06800: 1808 x 1190 (timer A = 520)
C0F06800: 1808 x 1190 (timer A = 640)
C0F06800: 1808 x 298  (timer A = 568)
C0F06800: 1808 x 727  (timer A = 528)
C0F06800: 1808 x 727  (timer A = 640)
C0F06800: 1872 x 1060 (timer A = 576)
C0F06800: 1872 x 1060 (timer A = 600)
C0F06800: 2592 x 1108 (timer A = 732)
C0F06804: 5280 x 3505
C0F06804: 5280 x 3529
EDMAC: 1808 x 1190
EDMAC: 1808 x 220
EDMAC: 1808 x 299
EDMAC: 1808 x 727
EDMAC: 2592 x 1108

650D.104/ROM1.BIN
C0F06800: 1808 x 1189 (timer A = 528)
C0F06800: 1808 x 1189 (timer A = 640)
C0F06800: 1808 x 726  (timer A = 528)
C0F06800: 1808 x 726  (timer A = 640)
C0F06800: 1872 x 1059 (timer A = 546)
C0F06800: 1872 x 1059 (timer A = 640)
C0F06800: 2592 x 1107 (timer A = 716)
C0F06804: 5280 x 3505
C0F06804: 5280 x 3529
EDMAC: 1808 x 1190
EDMAC: 1808 x 220
EDMAC: 1808 x 727
EDMAC: 2592 x 1108

700D.115/ROM1.BIN
C0F06800: 1808 x 1189 (timer A = 528)
C0F06800: 1808 x 1189 (timer A = 640)
C0F06800: 1808 x 726  (timer A = 528)
C0F06800: 1808 x 726  (timer A = 640)
C0F06800: 1872 x 1059 (timer A = 546)
C0F06800: 1872 x 1059 (timer A = 640)
C0F06800: 2592 x 1107 (timer A = 716)
C0F06804: 5280 x 3505
C0F06804: 5280 x 3529
EDMAC: 1808 x 1190
EDMAC: 1808 x 220
EDMAC: 1808 x 727
EDMAC: 2592 x 1108

EOSM.202/ROM1.BIN
C0F06800: 1808 x 1189 (timer A = 528)
C0F06800: 1808 x 1189 (timer A = 640)
C0F06800: 1808 x 726  (timer A = 528)
C0F06800: 1808 x 726  (timer A = 640)
C0F06800: 1872 x 1059 (timer A = 546)
C0F06800: 1872 x 1059 (timer A = 640)
C0F06800: 2592 x 1107 (timer A = 716)
C0F06804: 5280 x 3505
C0F06804: 5280 x 3529
EDMAC: 1808 x 1190
EDMAC: 1808 x 220
EDMAC: 1808 x 727
EDMAC: 2592 x 1108

EOSM2.103/ROM1.BIN
C0F06800: 1808 x 1190 (timer A = 520)
C0F06800: 1808 x 1190 (timer A = 640)
C0F06800: 1808 x 298  (timer A = 568)
C0F06800: 1808 x 416  (timer A = 522)
C0F06800: 1808 x 727  (timer A = 528)
C0F06800: 1808 x 727  (timer A = 640)
C0F06800: 1872 x 1060 (timer A = 576)
C0F06800: 1872 x 1060 (timer A = 600)
C0F06800: 2592 x 1108 (timer A = 732)
C0F06804: 5280 x 3505
C0F06804: 5280 x 3529
EDMAC: 1808 x 1190
EDMAC: 1808 x 220
EDMAC: 1808 x 417
EDMAC: 1808 x 727
EDMAC: 2592 x 1108

I looked at firmware updates on the EOSM AND EOSM2 and it didn't change the results, at least not on those updates. In any case, all of the simple silent stills I've been shooting on the 700D in mv1080 mode have been coming in at 1808x1189, same with MLV files as reported by mlv_dump on steroids and MLVFS. The "official" crop_rec_4k mlv_dump seems to be the only app that I tested that reports the height differently for the reason that bouncyball pointed out.

@theBilalFakhouri -- Here is a brand new focus pixel map file that will work on the 700D when recording 12-bit lossless compression in mv1080 mode. Test it out and report back. This is what your test file looks in MLVFS using the new map file--no chroma smoothing this time.



To do -- figure out what to do with all of the other focus pixel map files. Maybe they should all be covering the entire sensor just in case? It doesn't seem to hurt the image. At least it looks better than having colored dots all over the place! [EDIT] We're pretty much doing that already on the 100D.

[Danne]

Amazing work on those 700D pixels dfort.