[SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording

[a1ex]

Okay, I'll fix it if you do the pixel peeping part.

5D3 bin: autoexec.bin

Code: [Select]

    for (int type = 0; type < 100; type++)
    {
        /* from lv_af_raw -> lv_set_raw_type(arg0 ? 4 : 7) -> MEM(0x2D168) = a bunch of values, default 34 */
        MEM(0x2D168) = type;
       
        /* this enables a LiveView debug flag that gives us 14-bit RAW data. Cool! */
        raw_lv_request();
        msleep(50);
       
        /* after filling one frame, disable the flag so we can dump the data without tearing */
        raw_lv_release();
        msleep(50);

        /* update raw geometry, autodetect black/white levels etc */
        raw_update_params();

        /* save it to card */
        char* fn = silent_pic_get_name();
        bmp_printf(FONT_MED, 0, 60, "Saving raw type %d, %d x %d...", type, raw_info.jpeg.width, raw_info.jpeg.height);
        save_dng(fn);
    }

Take a picture to reset the file counter.

Then, hit "don't click me" to try these raw flags (they are related to bad pixel handling).

Then, write down for each picture type whether it has bad pixels, banding, vertical stripes and whatever else you can find.

[bass1012dash]

Thank you.
one more detail, only getting noise (at least noticing noise) at 1250 ISO+

[1%]

Code: [Select]


RAM:0001DABC                 ADR     R0, aRawTypeRshd0_0 ; "RAW TYPE ::: RSHD       ==>  0\n"
RAM:0001DAC0                 BL      sub_FF0C1F28
RAM:0001DAC4                 ADR     R0, aShade1_0   ; "             SHADE      ==>  1\n"
RAM:0001DAC8                 BL      sub_FF0C1F28
RAM:0001DACC                 ADR     R0, aHivshd2_0  ; "             HIVSHD     ==>  2\n"
RAM:0001DAD0                 BL      sub_FF0C1F28
RAM:0001DAD4                 ADR     R0, aOrbit3_0   ; "             ORBIT      ==>  3\n"
RAM:0001DAD8                 BL      sub_FF0C1F28
RAM:0001DADC                 ADR     R0, aDefcorre4_0 ; "             DEFCORRE   ==>  4\n"
RAM:0001DAE0                 BL      sub_FF0C1F28
RAM:0001DAE4                 ADR     R0, aCcd5_0     ; "             CCD        ==>  5\n"
RAM:0001DAE8                 BL      sub_FF0C1F28
RAM:0001DAEC                 ADR     R0, aDefmark6_0 ; "             DEFMARK    ==>  6\n"


I think this is what your bits mean?


6D address is 0x7E020

[a1ex]

Almost. There's a lookup table between those numbers and the value written to digic.

[1%]

Rawtype is the first bit tho?

[a1ex]

Not sure. After 64, the images repeat, so there should be 6 bits.

[sergiocamara93]

Okay, I'll fix it if you do the pixel peeping part.

I don't know if you need more tests or if 1% have taken care of that. I'll be happy to help with the pixel peeping 
As I side note, to keep all the test data in this post, it doesn't happen either in 19th May Build.

[a1ex]

Still need the tests. I don't see any significant difference on my 5d3, there's one bright pixel that's been there since the beginning, and I don't see any signs of vertical stripes or banding...

[sergiocamara93]

I put a DNG comparison in another post http://www.magiclantern.fm/forum/index.php?topic=5413.msg39600#msg39600 . If you zoom in the big "N" or in other letters the pixels are massive.

I'll run the tests and post them ASAP

[1%]

So I ran the tests... I get hard lock at type 23 and when I counted backwards type 52.... I see no difference in these, they are all 4.2 MB


BTW, 1320x562 or whatever that was is fixed now.

[a1ex]

If anyone is able to notice a difference between older and newer builds, run a hg bisect to find the offending changeset.

[1%]

Also some of the noise bits like noisyboy showed me only ended up in the tiffs when I used rawanizer.. didn't see them in actual dngs and when I exported video.

* One of those raw functions is to set white point?

There are some more fun regs:

0xC0F1A014
0xC0F083C0

0xC0F37014 - what we're messing with I think


This is what was complained about:

http://i41.tinypic.com/149n40.jpg

[sergiocamara93]

Test done!

Summary:
Several images show image with color pixels but they are not the issue described.

I've found two RAW types which match exactly the problem (they are in caps): 19 y 83. They are also 1/3 more exposed that the ones marked as "NORMAL". They are equal images for what I can see (the two have same exposure and same hot pixels)

I've found 6 types which shows a normal image with no problems, they seem to be equal among them: 17, 35, 37, 39, 81, 99
*The ones marked as "Color points" are color points covering the whole image in which appears to be the sensor pattern.

I've saved the recorded frames in case you need them for reference. I've double-checked the NORMAL and ISSUE type numbers.

Code: [Select]

1 Horizontal color lines
2 Color points
3 Image tearing mostly Green/magenta
4 Color points but with some form from the objects photographed
5 Normal image with a few color pixels
6 Color points but with all the form from the objects photographed – Green/magenta
7 Normal image with a few color pixels
8 Normal image with a few color pixels – less exposure
9 Normal image with a few color pixels
10 Color points but with all the form from the objects photographed – Green/magenta
11 Normal image with a few color pixels
12 Color points but with all the form from the objects photographed – Green/magenta
13 Normal image with a few color pixels
14 Color points but with all the form from the objects photographed – Green/magenta
15 Normal image with a few color pixels
16 Color points but with all the form from the objects photographed – Green/magenta
17 NORMAL IMAGE, nothing I can see
18 Color points but with all the form from the objects photographed – Green/magenta
19 IMAGE SHOWING THE ISSUE DESCRIBED – Hot pixels along contrasted lines
20 Color points but with all the form from the objects photographed – Green/magenta
21 Horizontal color lines
22 Color points
23 Horizontal color lines
24 Normal image with a few color pixels
25 Color points but with all the form from the objects photographed – Green/magenta
26 Normal image with a few color pixels
27 Color points but with all the form from the objects photographed – Green/magenta
28 Normal image with a few color pixels
29 Normal image with a few color pixels – more exposure
30 Color points but with all the form from the objects photographed – Green/magenta
31 Normal image with a few color pixels
32 Color points but with all the form from the objects photographed – Green/magenta
33 Rubbish + horizontal lines
34 Color points
35 NORMAL IMAGE, nothing I can see
36 Color points but with all the form from the objects photographed – Green/magenta
37 NORMAL IMAGE, nothing I can see
38 Color points but with all the form from the objects photographed – Green/magenta
39 NORMAL IMAGE, nothing I can see
40 Image tearing mostly Green/magenta
41 Weird horizontal lines-points
42 Color points
43 Image tearing mostly Green/magenta
44 Color points but with some form from the objects photographed
45 Image tearing mostly Green/magenta
46 Color points but with some form from the objects photographed
47 Image tearing mostly Green/magenta
48 Color points but with some form from the objects photographed
49 Rubbish + Color points
50 Color points
51 Image tearing mostly Green/magenta
52 Color points but with some form from the objects photographed
53 Horizontal color lines
54 Color points
55 Horizontal lines
56 Image tearing mostly Green/magenta
57 Image tearing mostly, some magental and vertical stripes one info+one black pattern
58 Color points but with some form from the objects photographed
59 Image tearing mostly Green/magenta
60 Color points
61 Image tearing mostly Green/magenta
62 Color points but with some form from the objects photographed
63 Image tearing mostly Green/magenta
64 Color points but with some form from the objects photographed
65 Horizontal color lines
66 Color points
67 Image tearing mostly Green/magenta
68 Color points but with some form from the objects photographed
69 Normal image with a few color pixels
70 Color points but with all the form from the objects photographed – Green/magenta
71 Normal image with a few color pixels
72 Normal image with a few color pixels – less exposure
73 Normal image with a few color pixels
74 Color points but with all the form from the objects photographed – Green/magenta
75 Normal image with a few color pixels
76 Color points but with all the form from the objects photographed – Green/magenta
77 Normal image with a few color pixels
78 Color points but with all the form from the objects photographed – Green/magenta
79 Normal image with a few color pixels
80 Color points but with all the form from the objects photographed – Green/magenta
81 NORMAL IMAGE, nothing I can see
82 Color points but with all the form from the objects photographed – Green/magenta
83 IMAGE SHOWING THE ISSUE DESCRIBED – Hot pixels along contrasted lines
84 Color points but with all the form from the objects photographed – Green/magenta
85 Horizontal color lines
86 Color points
87 Horizontal color lines
88 Normal image with a few color pixels
89 Color points but with all the form from the objects photographed – Green/magenta
90 Normal image with a few color pixels
91 Color points but with all the form from the objects photographed – Green/magenta
92 Normal image with a few color pixels
93 Normal image with a few color pixels – more exposure
94 Color points but with all the form from the objects photographed – Green/magenta
95 Normal image with a few color pixels
96 Color points but with all the form from the objects photographed – Green/magenta
97 Rubbish + horizontal lines
98 Color points
99 NORMAL IMAGE, nothing I can see
100 Color points but with all the form from the objects photographed – Green/magenta

EDIT: I used your autoexec.bin with the other files from 22nd build and the RAW module loaded.

[1%]

So I'm not crazy and the color points show up in the image with certain types? I saw sprinkles in raw studio when moving among the different types then they disappear when it processes.

[sergiocamara93]

A couple crops (without any other adjustments)
This is type 17 (NORMAL):
https://dl.dropboxusercontent.com/u/21789470/ML%20PIXELS/Type%2017%20CROP.jpg
This is type 19 (ISSUE):
https://dl.dropboxusercontent.com/u/21789470/ML%20PIXELS/Type%2019%20CROP.jpg

I'm amazed with the exposure change, it remembers me to the 1/3 reported difference between the H.264 and the HDMI output in the new 1.2.1 firmware.

[1%]

Totally see the difference... no more sprinkles.

Guess this has to go in raw.c? So that we can account for the address on different cameras.

Shit, mine all have them. Have to try 26-51

26, 27 freezes... 28-46 doesn't freeze and its all noise free. Haven't tried 48-52 yet but I guess I don't have to.

[nanomad]

It turns out that lv_af_raw enables something that looks very similar to a vertical edge detection algorithm instead of defect correction (at least on the 650D) 

https://dl.dropboxusercontent.com/u/3904468/DNG/AF-SERVO.DNG

You can even enable something that looks like a full-blown contrast-detection algorithm

https://dl.dropboxusercontent.com/u/15964163/MAGIC_4.DNG

[1%]

Completely defect free at 28 tho.

[sergiocamara93]

It turns out that lv_af_raw enables something that looks very similar to a vertical edge detection algorithm instead of defect correction (at least on the 650D) 

https://dl.dropboxusercontent.com/u/3904468/DNG/AF-SERVO.DNG

I think that's exactly the issue we were describing!

[fascina]

Thanks for the awesome RAW hack! I really hope this hot pixels isue can be solved, because there are so many of those hot pixels in my 5d MIII footage that they make it unusable.

As far as I can assess, the hot pixels problems started from 19 May, exactly since the banding problem was solved.

I really wish you guys luck with solving the pixel problem so that the 5d Mark III can film perfect professional footage without picture isues.

Thanks for your work.

[fascina]

I've attached a 400x magnified picture crop of a developed RAW DNG (5D Mark III, developed with Adobe Camera RAW).

Here one can see that it is actually not a "hot pixel" problem: Because those faulty pixels are only in those areas where there is a transition from dark to bright (see the radiator in the lower half of the photo).
There are no pixel problems on the wall, because on the wall there are no transitions from darker to brighter areas.

You can download the original DNG here: http://www.webpage-design.org/000003.dng

[RenatoPhoto]

Reporting 5D3

May 15 good
May 16 good
May 17 good
May 18 good
May 19 good
May 20 BAAAAAD!  mercurial  3ab36d8a50de+
May 21 Bad
May 22 Ban
May 23 Bad
May 26 (now) Bad

I am running the autoexec.bin now

[RenatoPhoto]

Here is my description of what I saw.

Best images in: 17, 35, 37, 39, 81, 83, 99

Code: [Select]

1 Horizontal  lines , colors off
2 All Black..Phtoshop cannot open this file
3 Some Vertical lines, image tearing, colors off
4 Vertical, horizonta, tearting, colors off
5 Good but sprinkles of many colors
6 Vertical lines , colors off, image is vissible
7 Good but sprinkles of many colors
8 Good but sprinkles of many colors
9 Good but sprinkles of many colors
10 Vertical lines , colors off, image is vissible
11 Good but sprinkles of many colors
12 Vertical lines , colors off, image is vissible
13 Good but sprinkles of many colors
14 Vertical lines , colors off, image is vissible
15 Good but sprinkles of many colors
16 Vertical lines , colors off, image is vissible
17 PERFECT IMAGE
18 Vertical lines , colors off, image is vissible
19 Good buT SOME sprinkles of many colors
20 Vertical lines , colors off, image is vissible
21 hORIZONTAL  Lines
22 Sort of banding near top, criss croos no image
23 Horizontal  lines , colors off
24 Good but sprinkles of many colors
25 Vertical lines , colors off, image is vissible
26 Good but sprinkles of many colors
27 Vertical lines , colors off, image is vissible
28 Good but sprinkles of many colors
29 Good but sprinkles of many colors
30 Vertical lines , colors off, image is vissible
31 Good but sprinkles of many colors
32 Vertical lines , colors off, image is vissible
33 Horizontal lines, off colors, image barely apearing on top
34 Balck ??
35 Excellent
36 Vertical lines , colors off, image is vissible
37 Excellent
38 Vertical lines , colors off, image is vissible
39 Excellent
40 Horizontal lines, off colors,tearing on bottom, image near top, small band on bottom
41 Horizontal lines, off colors, image barely apearing on top
42 Sort of banding near top, then criss croos no image , small band on top
43 Horizontal lines, off colors,tearing on bottom, image near top, small band on bottom
44 Sort of banding near top, then criss croos no image , small band on top
45 Same as above less less the criss cross pattern
46 Sort of banding near top, then criss croos no image , small band on top
47 Same as above less less the criss cross pattern
48 Black?
49 Horizontal lines, off colors, image barely apearing on top
50 Sort of banding near top, then criss croos no image , small band on top
51 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image
52 Same as above with noise (criss cross=)
53 Horizontal Lines, colors off, no image
54 Criss cross, no image, some banding above
55 Horizontal Lines, colors off, no image
56 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image
57 Same as above but less contrast?
58 Same as above with criss-cross
59 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image
60 Same as above with criss-cross
61 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image
62 Same as above with criss-cross
63 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image, more noise
64 Black?
65 Horizontal Lines, colors off, no image
66 Black?
67 Horizontal lines, off colors, image barely apearing on top, tering on center, bootom image
68 Same as above with criss-cross
69 Good but sprinkles of many colors
70 Horizontal lines on image, color off
71 Good but sprinkles of many colors
72 Good but sprinkles of many colors
73 Good but sprinkles of many colors
74 Horizontal lines on image, color off
75 Good but sprinkles of many colors
76 Horizontal lines on image, color off
77 Good but sprinkles of many colors
78 Horizontal lines on image, color off
79 Good but sprinkles of many colors
80 Horizontal lines on image, color off
81 Excellent
82 Horizontal lines on image, color off
83 Excellent
84 Horizontal lines on image, color off
85 Horizontal Lines, colors off, no image
86 Sort of banding near top, criss croos no image
87 Horizontal lines on image, color off
88 Good but sprinkles of many colors
89 Horizontal lines on image, color off
90 Good but sprinkles of many colors
91 Horizontal lines on image, color off
92 Good but sprinkles of many colors
93 Good but sprinkles of many colors
94 Horizontal lines on image, color off
95 Good but sprinkles of many colors
96 Horizontal lines on image, color off
97 Horizontal Lines, colors off, image above only
98 Black?
99 Excellent
100 Horizontal lines on image, color off


[RenatoPhoto]

One more thing I think is important to note:

I spent about an hour shooting this and that, and I could not get the dam specs, or whatever they are.  Finally I shot some video of a very busy area of books and magazines just as shown in one of these post and then I could see them all over!  Worst problems with may 20th but getting better with the latest build but still noticeable.

[1%]

I thought I had some perfect ones and then they had sprinkles later on.

I think lv_af_raw is for something else. I dunno if its hurting or helping, the noise pattern just looks slightly different with it off.