Pixel binning patterns in LiveView

[a1ex]

Original discussion: http://www.magiclantern.fm/forum/index.php?topic=10111.msg118232#msg118232

I wanted to split the topic, but that would make the original discussion harder to follow, so I'm just copying the relevant parts here.

Finally finished stuffing around, and here is a good bunch of results.  Enjoy!
https://www.dropbox.com/sh/ktc2kyabmozjofq/AAC7kSBnzX3E5cwoLn9nrsbYa

From the above data, I'll try to guess the pixel binning factors from LiveView (and I'll ask SpcCb to double-check what follows):

My quick test, at ISO 6400:

Code Select Expand


         gain       read noise     ratio (compared to 5x)
720p:    1.43       14.79          14.74
1080p:   0.88       14.75          9.07
5x:      0.097      23.64          1


Numbers from Audionut:

Code Select Expand


         gain       read noise     ratio (compared to 5x)

ISO 100:
720p:    73.48      6.93           11.9        (note: it's very hard to tell how much is read noise
1080p:   53.78      6.54           8.7          and how much is Poisson noise from a nearly straight line)
5x:       6.15      5.98           1
photo:    5.11      6.77           0.83

ISO 200:
720p:    44.87      7.22           14.4
1080p:   27.50      6.76           8.84
5x:       3.11      6.26           1
photo:    2.58      7.08           0.83

ISO 400:
720p:    22.50      7.34           14.6
1080p:   13.94      6.90           9.05
5x:       1.54      6.70           1
photo:    1.27      7.61           0.82


ISO 800:
720p:    11.40      7.77           14.6
1080p:    7.07      7.32           9.06
5x:       0.78      7.32           1
photo:    0.66      8.60           0.85

ISO 1600:
720p:     5.80      8.78           14.7
1080p:    3.54      8.34           8.98
5x:       0.394     9.94           1
photo:    0.324    11.10           0.82

ISO 3200:
720p:     2.91      10.82          14.9
1080p:    1.81      10.45          9.23
5x:       0.196     14.75          1
photo:    0.166     16.28          0.85

ISO 6400:
720p:     1.41      14.81          14.7
1080p:    0.87      14.67          9.06
5x:       0.096     23.90          1
photo:    0.082     30.09          0.85

ISO 12800:
720p:     0.71      29.69          14.2
1080p:    0.44      29.44          8.8
5x:       0.050     58.40          1


Raw buffer sizes (active area):
- photo mode: 5796x3870
- 1080p: 1932x1290
- 1932x672 stretched (covers roughly 16:9 in LiveView)

Ratio between photo mode and 5x zoom: 0.83. If the 5x zoom captures a little more highlight detail, it's OK. The difference may be also because LiveView uses electronic shutter, while photo mode uses mechanical shutter. So, I'll use the 5x zoom as reference for the other LiveView modes.

From the above data, I now have very strong reasons to believe that 5D3 does a 3x3 binning in 1080p, and a 5x3 binning in 720p (5 lines, 3 columns).

(if you shoot 720p on 5D3, the desqueezing factor - to correct the aspect ratio of your footage-  is therefore exactly 5/3 = 1.67x)

A possible 3x3 binning (and easy to implement in hardware) would be to average each sensel and its 8 neighbours of the same color (considering the two greens as separate colors, as in the well-known four-color demosaicing algorithms). This binning scheme can be easily extended to 720p (5x3), but might cause some interesting artifacts on resolution charts.

 

A more complex 3x3 binning (very unlikely to be implemented in hardware, since it requires complex logic and knowledge about each pixel's color) could be:


(I'm showing it just for completeness, but I think the first pattern is the much more likely to be used).

If anybody could shoot some resolution charts in LiveView (silent pictures in 5x, 1080p and 720p, without touching the camera - I need more than pixel-perfect alignment), I can verify if these patterns are indeed the correct ones or not. If you don't use a remote release, you can take this test with the "Silent zoom bracket" option from the latest raw_diag to avoid camera movement.

Side note: the registers that control the downsizing factors are:
- Horizontal: CMOS[2], which also controls the horizontal offset; you can select full-res (1:1) or downsized by 3
- Vertical: ADTG 0x800C (2 for 1080p, 4 for 720p and 0 for zoom, so it should be the downsizing factor minus 1; other values are valid too)

Other cameras: I don't have much data, but from what I have, the binning factor seems to be 3. For example, the data from 50D (dsManning) looks like this:

Code Select Expand


         gain       read noise     ratio (compared to photo)

ISO 100:
1080p:    7.67      5.34           3.4
photo:    2.26      6.15           1

ISO 200:
1080p:    4.20      5.48           3.85
photo:    1.09      6.52           1

ISO 400:
1080p:    2.04      5.89           3.4
photo:    0.60      7.97           1

ISO 800:
1080p:    1.04      7.30           3.4
photo:    0.31     10.94           1

ISO 1600:
1080p:    0.53     10.32           3.5
photo:    0.15     16.12           1

ISO 3200:
1080p:    0.53     10.45          nonsense :)
photo:    0.08     38.06          1


and from 500D (Greg):

Code Select Expand


         gain       read noise     ratio (compared to photo)
ISO 100:
photo LV: 7.38      6.34           3.3
photo:    2.23      6.82           1


From the resolution charts (the first one I could find was this), most cameras (except 5D3) show artifacts as if they were skipping lines, but not skipping columns.

Therefore, I believe the binning pattern looks like this:



but I'm waiting for your raw_diag tests to confirm (or reject) this theory.

Edit: confirmed on EOS M and 5D Mark II. From visual inspection, this method appears to be used on most other Canons.



An interesting conclusion is that 5D3 does not throw away any pixel in LiveView. Then you may wonder, why binning a full-res CR2 by 3x3 in post is cleaner? Simple: binning in software will average out all noise sources, while binning in the analog domain (like the 5D3 does) will only average out the noise that got introduced before binning (here, the shot noise and maybe a small part of other types of noise), but cannot average out the noise that gets introduced after binning (here, the read noise, which is quite high on Canon sensors).

Therefore, at high ISO (where the shot noise is dominant), the per-pixel SNR on 5D3 1080p is improved by up to^*) log2(sqrt(9)) = 1.58 EV, compared to per-pixel SNR in crop mode. On the other cameras (3x1 binning), per-pixel SNR is improved by up to log2(sqrt(3)) = 0.79 EV.

So, the noise improvement from the better binning method is up to 0.8 EV at 1080p (ranging from 0 in deep shadows to 0.8 in highlights). That's right - throwing away 2/3 of your pixels will worsen the SNR by only 0.8 stops (maybe not even that).

^*) If the binning would be done in software, you would simply drop the "up to" - the quoted numbers would be the real improvement throughout the entire picture

[a1ex]

I PMd you a link with a Dropbox .zip 100 megs of shots

Some quick results:

5D3, 1080p vs 5x zoom:

Source images: 1x.DNG and 5x.DNG; convert to pgm with "dcraw -4 -E *.DNG".



A few binning methods, applied on the 5x crop image:


Difference between downsized 5x and unmodified 1x:


5D3, 720p vs 5x zoom:

Source images: 1x.DNG and 5x.DNG; convert to pgm with "dcraw -4 -E *.DNG".



A few binning methods, applied on the 5x crop image:


Difference between downsized 5x and unmodified 1x:


Debayer method: simply applied white balance and rendered as grayscale; since the image was monochrome, no real debayer was needed.

Binning methods:
1. 3x3 binning of pixels with the same color (the two greens are considered different colors)
2. 1x3 binning of pixels with the same color (1 line, 3 columns, which results in line skipping, as I believe to be done on most Canon DSLRs)
3. 5x3 binning of pixels with the same color (5 lines, 3 columns)
4. 3x3 binning of adjacent pixels (can only be done after debayering)
5. octave's imresize (can only be done after debayering)

Octave scripts, if you want to review the experiment or try it on your own data: 1080p and 720p.

I'd say the theory is confirmed - 3x3 binning in 1080p, 5x3 binning in 720p, averaging pixels of the same color, no line skipping (all pixels are used).

 

Wanted: test images for some camera(s) other than 5D3, to confirm the binning method with line skipping.

[a1ex]

Still looking for test data for cameras other than 5D3.

If anybody could shoot some resolution charts in LiveView (silent pictures in 5x, 1080p and 720p, without touching the camera - I need more than pixel-perfect alignment), I can verify if these patterns are indeed the correct ones or not. If you don't use a remote release, you can take this test with the "Silent zoom bracket" option from the latest raw_diag to avoid camera movement.

[KelvinK]

Is it possible to download somewhere and print resolution chart?

[KelvinK]

Does it make sence to do it with 6D?

[Walter Schulz]

http://www.graphics.cornell.edu/~westin/misc/res-chart.html

[KelvinK]

http://www.graphics.cornell.edu/~westin/misc/res-chart.html

thanks

Alex, will it be enough to print it on A4 or A3 is a must?

[a1ex]

As long as you get moire on the camera, any size is fine.

[dfort]

EOSM and SL1 ('A' firmware a.k.a. 100D)

https://www.dropbox.com/sh/cj4ztz7bplv0g4m/AADRfBvCElinNgDeCM2gNeFCa?dl=0

The EOSM was no problem but this is the first time using the 100D. I couldn't get the touchscreen working with ML menus so I couldn't get raw_diag to do its thing. Also the 1080p silent picture doesn't look right but I did this three times and it always came out the same. Hope this is what you're asking for.

[SpcCb]

I just took a look over this, tonight I'll see it closer. Sounds very interesting!
Specially about the question 'hardware or software binning?'.

[a1ex]

Heh, the message was taken from an older discussion, where you already answered. But revisiting the topic shouldn't hurt, right?

[SpcCb]

It never hurts!
I remember the original discussion, I thought you are re-lunching it.

[dfort]

Added files for 700D

https://www.dropbox.com/sh/cj4ztz7bplv0g4m/AADRfBvCElinNgDeCM2gNeFCa?dl=0

Whatever you find out about Pixel Binning and line skipping also helps with the Focus Pixel Fixer project. 

[dfort]

If you don't use a remote release, you can take this test with the "Silent zoom bracket" option from the latest raw_diag to avoid camera movement.

I'd like to shoot some more tests but please refresh my mind. Is raw_diag a module, branch or what? Also, the link to that resolution chart is dead. Is there a recommended chart someone can link to?

[Walter Schulz]

http://www.bealecorner.org/red/test-patterns/

[reddeercity]

I'd like to shoot some more tests but please refresh my mind. Is raw_diag a module, branch or what?

It a module from the iso research branch

[a1ex]

It's here as well: http://builds.magiclantern.fm/modules.html

[dfort]

Thanks Walter, David and Alex!

I take it the ISO 12233 Chart is preferred one to use? Anything that creates a moiré pattern, right?

[dfort]

Did some more tests. This time I tried to shoot the pixel binning test using the same chart and settings on the EOSM, 100D and 700D.

https://www.dropbox.com/sh/prilw5vvrl9d0e0/AABbX8xoiU5o_AEZW2vJMXvya?dl=0

Some interesting observations.

The 100D always recorded the second (5x) frame from the raw_diag "Silent Zoom Bracket" blank. I tried the build that is on the 100D forum topic along with the posted raw_diag module, building from the @nikfreak bitbucket repository and several combinations including merging the latest unified but it always came up blank. I would have looked into it some more but I need to return the camera which belongs to @DeafEyeJedi. Shot the test on a solid tripod the best I could. I also included those blank frames in case there's a clue in there as to what is going on.

[a1ex]

How did you manage to get the test files, then? Manually pressing half-shutter, or remote release cable?

Did the blank picture also appear with manual trigger? If the third picture was OK, you can edit the "script" to take 3 pictures as a workaround (although I'd be interested to find out why it happens). Were there any messages on the console?

There is a memory allocation fix for the silent picture module (where it was missing the first frame on 5D3, iirc) on the raw_fixes branch. Maybe it's related.

BTW, I have a feeling ISO might have been too high on these samples (they are quite noisy).

[dfort]

How did you manage to get the test files, then?

Manually -- yeah, that's a lousy test.

Did the blank picture also appear with manual trigger? If the third picture was OK, you can edit the "script" to take 3 pictures as a workaround (although I'd be interested to find out why it happens). Were there any messages on the console?

Yes, manual trigger will also give a blank picture on the second trigger. Interesting is that the second picture should be the 5x exposure but it is the same file size as the first, only blank so the "set_lv_zoom(5);" didn't seem to take though LiveView went into 5x before it fired the second shot. I edited the script to take multiple shots in 5x mode and all the subsequent shots were fine. I also played with the delay before the second shot but it didn't seem to make a difference. Putting the camera in 5x mode and shooting a single shot worked fine. Weird. The error message that flashes on the screen is, "Buffer error..." I had to run it several times to be sure because sometimes it wouldn't display the error or maybe I'm too slow.

BTW, I have a feeling ISO might have been too high on these samples (they are quite noisy).

ISO was set at 200. The noise you see is probably from my printer. I used smooth paper but the print doesn't look very sharp.

Give me a few minutes to set up and I'll upload another 100D test.

[dfort]

100D test reshoot is up. I trashed the invalid tests.

A few notes on the error message.

The full frame is saved normally though there is a message that says "Buffer: 2 frames (90%)" LV zooms into 5x and next message says "Saving 1736 x 699" which doesn't make sense because that isn't the size of the 5x zoom buffer. Next message is "Preparing..." with "Buffer error" below that followed by "Preparing..." with "Capturing frame 1". The next couple of 5x capture frames I added show the normal "Preparing..." followed by "Saving 2520 x 1080..." message.

Hope this helps.

[EDIT] BTW doing a shutter half press to shoot a silent picture would also sometimes trigger raw_diag to save a chart. Thought I'd post this one in case anyone finds it interesting.

[ilia3101]

(I hope this thread is right place)

Pixel binning patterns are interesting and annoying... I've been thinking about it for a while.

Cameras such as 5D2, 7D, 70D, 100D, they all have the horrible 3x1 binning, and 5D3 has the good 3x3 binning.

So... In my opinion biggest issue with RAW video for me and almost everyone is the massive moire that comes with it,
it ruins the rendering of detail, and we use all kinds of methods to try and avoid it, such as debayering algorithms like AMaZE...
I don't know if this is right place to start this idea, but maybe an algorithm specially optimised for demosaicing 3x1 binned images could be created.
AMaZE is optimised almost perfectly for square pixels, or that have equal vertical and horizontal coverage, but with the 3x1 binning it still looks crap.

I have been experimenting with binning raw images and I've come up with binning_test (github link).
It's inefficient and not too flexible yet, but mainly, it creates 2 binned images from an MLV file... one 3x3 and one 3x1, binning patterns are customisable.
It's most accurate with crop mode/FRSP. I think its a good starting point.

I really care about this because in my opinion the moire is the biggest problem with Magic Lantern RAW video on most cameras,
and I think it would be worth all the time it will take to make this algorithm, as it could potentially bring these crippled cameras up to 5D3 levels..

Here's the difference between how 3x1 and 3x3 bayer images look:



It shows that 3x3 is significantly smoother(look at roof too).

...another idea is:train a neural network to accurately transform a 3x1 image to look as if it was 3x3 binned.
Neural networks are good at finding subtle connections between values, so maybe it would recreate the 3x3 binning relatively good even with the missing information...
Except I don't know how to do neural networks ... something worth looking in to (this might be a better idea than first one)

I don't know really what the aim of this post is, but who thinks its a good idea? Does anyone have any suggestions or input? Can anyone help?

[a1ex]

Some random notes:

Input image: https://www.magiclantern.fm/forum/index.php?topic=15088.msg191844#msg191844

Code Select Expand


pkg load image
system("dcraw 2-normal.DNG");
b = imread("2-normal.ppm");
x = b(700, 500:600, 2); t = 1:length(x);
xf = imfilter(x, [1 0 1 0 1] / 3);
xnf = xf; for i = 1:10000, xnf = imfilter(xnf, [1 0 1 0 1] / 3); end
cla; stem(t, x, 'b'); hold on; plot(t, xf, 'o-r'); plot(t, xnf, '.-g'); axis tight




Food for thought: can we recover blue from red?

(same problem as Ilia's, just 1-D and monochrome)

[theBilalFakhouri]

@a1ex

I didn't know how you got 3x1 Binning what the registers to override?

I found a register maybe will fix horizontal stretching in LiveView when using 3x1 which is: C0F11A8C you can stretch the LiveView horizontally to the right and the left. I couldn't test it because adtg_gui doesn't work when using crop_rec or iso_regs in the same time, How can we use them both?

I can get with 3x1 mode 3840x720 (stretch it to get 3840x2160) @ 23.976 FPS but with horrible aliasing and no details at all. I am thinking if
we can do 3x2 Binning (Less aliasing in 3840 and more details) or also 2x2 Binning (Less crop factor) to be used for Monochrome, can we get this?