[600D RAW] Dots, dots and more dots.

[vickersdc]

I've been plagued by dots on raw recording since moving away from Tragic Lantern [New System]... I'm now using NewMem as the base, and updated with the nightly build from Dec 22nd. Whether I just use NewMem, or the nightly build, I get dots; for a long while I thought they were black dots which you can clearly see in this video, especially where there is a very light background.

And then I took a much closer look at them, and there is a sort of pattern to these invaders, as the image below shows:



You can see that the dots are not black at all, but take on the surrounding hue; there's always a central 2x2 pixel formation which is surrounded by a sort of brighter halo.

I've seen and read loads of posts about pink dots, green dots and all sorts of dots, but haven't seen anything about this particular pattern... anyone know why it occurs, and how to get rid of it? It's the only blot on the otherwise excellent raw recording. Unfortunately it's a bit of a showstopper for me - I love using the raw workflow and the capabilities it brings, but an image full of clearly visible dots is of no use (to me).

I've tried the MLV recording too, using chroma smoothing options, but that made it worse 

[vickersdc]

This was something of a last ditch attempt to find a quick way of getting rid of / reducing the dots in raw recording. I decided to shoot a few seconds of my wife knitting - lit by a compact fluorescent light (camera right), a halogen light (above camera right) and tungsten light (directly above camera)... so a total mish-mash of lights!

Anyway, the clips were shot at 1152x432 (2.66:1) which gets automatically upres'ed to 1280x720 in Davinci Resolve; I wondered if scaling it down to 1024x383 to retain the 2.66:1 aspect ratio would get rid of the dots.

It didn't

But the raw handled different light sources really well

Unfortunately, it's quite difficult to see the detail that is still in the MP4 file that I uploaded - the Vimeo compression seems to have made it rather darker than it really is (Oh well, not to self - must take that into account next time).

[xaled]

Had the same problems with the black dots on my 600d with TL versions. I just gave up on RAW back then as the tweaked TL H.264 version is fine for my use cases.

It would be great though if there is an answer to the issue.

[Luiz Roberto dos Santos]

This was something of a last ditch attempt to find a quick way of getting rid of / reducing the dots in raw recording. I decided to shoot a few seconds of my wife knitting - lit by a compact fluorescent light (camera right), a halogen light (above camera right) and tungsten light (directly above camera)... so a total mish-mash of lights!

Anyway, the clips were shot at 1152x432 (2.66:1) which gets automatically upres'ed to 1280x720 in Davinci Resolve; I wondered if scaling it down to 1024x383 to retain the 2.66:1 aspect ratio would get rid of the dots.

It didn't

But the raw handled different light sources really well

Unfortunately, it's quite difficult to see the detail that is still in the MP4 file that I uploaded - the Vimeo compression seems to have made it rather darker than it really is (Oh well, not to self - must take that into account next time).

I'm really curious to know the real reason you do not use TL and not ML NB ... @a1ex has said several times that packets of TL 2.0 for 600D are outdated (since June) and that the version is unstable (overheating, and memory blocks are not as vast as the current ML NB). Why use? Even in H.264, bitrate is the same.

I believe that only software we will be able to remove the dead pixels. Unfortunately I have no programming skills, but I'm studying to do so, since the matter related to the topic seems to be dead. No one is posting anything, not even discussing ... well, this is not the correct way to resolve things in an open source project.

[vickersdc]

I'm really curious to know the real reason you do not use TL and not ML NB ... @a1ex has said several times that packets of TL 2.0 for 600D are outdated (since June) and that the version is unstable (overheating, and memory blocks are not as vast as the current ML NB). Why use? Even in H.264, bitrate is the same.

I believe that only software we will be able to remove the dead pixels. Unfortunately I have no programming skills, but I'm studying to do so, since the matter related to the topic seems to be dead. No one is posting anything, not even discussing ... well, this is not the correct way to resolve things in an open source project.

I use both TL and the nightly builds... both give me the dots, which I do not believe are dead pixels at all. If I fire up TL [New System] from back in June, I can shoot raw with none of these pixels occurring. I've now stopped using raw, as much as I love the quality, having so many of these dots all over the image is no good 

I'm currently back to using nightly builds and H.264.

[RenatoPhoto]

Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0

Also Adobe Camera RAW will fix most of these problems. 

[Luiz Roberto dos Santos]

So, TL dont give "dots"? Or less?

[Luiz Roberto dos Santos]

Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0

Also Adobe Camera RAW will fix most of these problems.

I test a1ex application, AHD like chroma smooth 2x2. It's help, but dont remove. But, ACR really solve it:



But, the workflow with ACR/LR is too slow for using on real life... at least for my weak hardware.
The black dots seems to be more "easier" to remove it than the chroma dots, indeed. Perhaps the problem could be solved, in future, we could create another application, merging with CS 2x2...
But, anyway, we would have several dng's and would have to re-use some other raw enginers to process, it's to slow workflow.
There is no way to mux DNG to RAW again?

[vickersdc]

I tried the MLV video recording module and used mlv_dump with the no-cs, cs2x2, cs3x3 and cs5x5 chroma smoothing options and none of those worked.

Interesting results with Lightroom 5.3 - I don't have LR on my iMac (just RAWMagic -> Resolve 10 -> FCP X)... if it does a good job of removing the dots across the entire frame I may well pay the £70 / $100 for it as it would be worth it; but is the workflow with it?

Do you have to batch process the DNG's after raw2dng, deal with them one by one, or...?

DV.

[vickersdc]

Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0

Also Adobe Camera RAW will fix most of these problems.

Thank you!!!

I've actually had RAWTherapee on my Mac for ages and never used it! It took me a while to find the relevant 'Hot/Dead Pixel' tickbox, but when I did...  Dots gone! Brilliant!

Now I need to work out how to batch process and save them - ideally back into DNG, but I think the only options are JPEG, TIFF or PNG. Anyway, I need to do some searching about using RAWTherapee, but I feel I'm a step closer to getting rids of those dots.

Many thanks for that link.
DV.

[Luiz Roberto dos Santos]

I tried the MLV video recording module and used mlv_dump with the no-cs, cs2x2, cs3x3 and cs5x5 chroma smoothing options and none of those worked.

Interesting results with Lightroom 5.3 - I don't have LR on my iMac (just RAWMagic -> Resolve 10 -> FCP X)... if it does a good job of removing the dots across the entire frame I may well pay the £70 / $100 for it as it would be worth it; but is the workflow with it?

Do you have to batch process the DNG's after raw2dng, deal with them one by one, or...?

DV.

You can batch process, just only synchronize your changes... it's simple, but for a work "in real life" is a very slow process, especially if it will be lossless (you would have to export 16bit TIF, which are very heavy and slow to process).
RAW only have reason and will be popularized if workflow is fast enough, not everyone uses i7 to process their recordings
I think the process. RAW> RAW2GPCF> 444 10bit quite usable (for win, for mac RAWMagic works fine, I think), but if you have to go through some application, 'unpack' .raw, no longer usable ... if it were possible to fix this problem with an application and make multiplexing for .RAW again, would be the ideial, indeed.

RAWTherappe is even slower. You can make batch: modify one DNG, click with right button on icon, copy settings, place on others, and put it on qeue... but the AMaZE is very slow too, and using the highlight recover is impossible in my oppinion.

Any developer can explain how these artifacts occur in the image? It is the line skipping? On 3x crop this occur too, so... I really don't understand it.

[escho]

These bad pixel of 600D are not very easy to handle. I know only two opensource tools, which can remove them:

1. rawtherapee, as you said before
2. cr2hdr from the dualiso-modul of ML (but it only works for dualiso-stuff)

I think, if the bad-pix-code from cr2hdr can somehow  be ported to raw2dng and to mlvdump, the problem has gone.

Edgar

[vickersdc]

Any developer can explain how these artifacts occur in the image? It is the line skipping? On 3x crop this occur too, so... I really don't understand it.

I agree... it's odd that the 'prehistoric' TL [New System] didn't suffer from this, but anything subsequent does seem to...

[vickersdc]

RAWTherappe is even slower. You can make batch: modify one DNG, click with right button on icon, copy settings, place on others, and put it on qeue... but the AMaZE is very slow too, and using the highlight recover is impossible in my opinion.

RAWTherapee is not too bad - using the command line to 'batch' process an entire directory full of DNGs doesn't take so long, although it does mean that you no longer have raw files to deal with, but 16-bit TIFF instead (and they take up more space!).

But I think RAWTherapee is do-able as part of the workflow... just.

[a1ex]

I've described the proper solution here: http://www.magiclantern.fm/forum/index.php?topic=9625

It's very very easy, that's why I'm not implementing it

[escho]

I've described the proper solution here: http://www.magiclantern.fm/forum/index.php?topic=9625

It's very very easy, that's why I'm not implementing it

I saw this thread some days ago. And I tried to understand the code, but I didn´t suceed. Yes, this is very easy for you, Alex. But for me with no coding skills in C, it´s a little bit harder. I will try it once again in the next days, hoping my little knowledge in bash-programming can help me. Let´s see...

Edgar

[a1ex]

The first step is to compile the half-working code: it should remove most of the bad pixels, except for adjacent ones.

Once that works, simply extend the algorithm to handle groups of pixels instead of isolated ones.

[tupp]

The dots have a distinctive pattern, and, hence, each is larger than a single pixel.  Each dot appears to affect 24 pixels.

The dots have a set location.  They seem to gradually fade/sharpen depending on whether or not the adjacent area of the image is in focus (or, perhaps, on whether or not the camera thinks that  the adjacent area is in focus).

[a1ex]

Look at the dots before debayering, not after

[Luiz Roberto dos Santos]

So, this is the source of cr2hdr to solve the bad pixels... it's just it a1ex? I can place it after the chroma smooth operations and compile?

Code: [Select]

static void find_and_fix_bad_pixels(int dark_noise, int bright_noise, int* raw2ev, int* ev2raw)
{
    int w = raw_info.width;
    int h = raw_info.height;

    int black = raw_info.black_level;
    //~ int white = raw_info.white_level;

    printf("Looking for hot/cold pixels...\n");

    /* hot pixel map */
    unsigned short* hotpixel = malloc(w * h * sizeof(unsigned short));
    memset(hotpixel, 0, w * h * sizeof(unsigned short));

    int hot_pixels = 0;
    int cold_pixels = 0;
    int x,y;
    for (y = 6; y < h-6; y ++)
    {
        for (x = 6; x < w-6; x ++)
        {
            int p = raw_get_pixel(x, y);

            int is_hot = 0;
            int is_cold = 0;

            /* really dark pixels (way below the black level) are probably noise */
            is_cold = (p < black - dark_noise*8);

            /* we don't have no hot pixels on the bright exposure */
            /* but we may have cold pixels */
            if (!BRIGHT_ROW || is_cold)
            {
                /* let's look at the neighbours: is this pixel clearly brigher? (isolated) */
                int neighbours[100];
                int k = 0;
                int i,j;
                int fc0 = FC(x, y);
                int b0 = is_bright[y%4];
                for (i = -4; i <= 4; i++)
                {
                    for (j = -4; j <= 4; j++)
                    {
                        if (i == 0 && j == 0)
                            continue;

                        /* only look at pixels of the same brightness */
                        if (is_bright[(y+i)%4] != b0)
                            continue;

                        /* only look at pixels of the same color */
                        if (FC(x+j, y+i) != fc0)
                            continue;

                        int p = raw_get_pixel(x+j, y+i);
                        neighbours[k++] = -p;
                    }
                }

                if (k <= 4) /* not enough data to draw a conclusion */
                    continue;

                int max = -kth_smallest_int(neighbours, k, 1);
                is_hot = (raw2ev[p] - raw2ev[max] > EV_RESOLUTION) && (max > black + 8*dark_noise);

                if (fix_bad_pixels == 2)    /* aggressive */
                {
                    int second_max = -kth_smallest_int(neighbours, k, 2);
                    is_hot = ((raw2ev[p] - raw2ev[max] > EV_RESOLUTION/4) && (max > black + 8*dark_noise))
                          || (raw2ev[p] - raw2ev[second_max] > EV_RESOLUTION/2);
                }

                if (is_hot)
                {
                    hot_pixels++;
                    hotpixel[x + y*w] = -kth_smallest_int(neighbours, k, 2);
                }

                if (is_cold)
                {
                    cold_pixels++;
                    hotpixel[x + y*w] = -median_int_wirth(neighbours, k);
                }
            }
        }
    }

    /* apply the correction */
    for (y = 0; y < h; y ++)
        for (x = 0; x < w; x ++)
            if (hotpixel[x + y*w])
                raw_set_pixel16(x, y, debug_bad_pixels ? black : hotpixel[x + y*w]);

    if (hot_pixels)
        printf("Hot pixels      : %d\n", hot_pixels);

    if (cold_pixels)
        printf("Cold pixels     : %d\n", cold_pixels);

    free(hotpixel);
}


edit: I analyse the non-demosaicing data, and it's just a dark pixel (ohhh, really?), the chroma artefacts is the effect of interpolation algorithm... but seems really simple to resolve it, but I am not a programmer. I will try.

[a1ex]

Yeah, remove the hot pixel code, add some glue code, and should work.

To try my incomplete code, compile raw2dng from this changeset: https://bitbucket.org/hudson/magic-lantern/commits/5bb7af2d168bda9ce3f6f4587a5ceb07703ea47a

=> you get some glue code in fix_bad_pixels, where you can start to experiment. That code will solve the problem for isolated bad pixels, so it's worth trying it.

Note that raw2dng uses very fast code via PA...PH / SET_PA...SET_PH macros. With raw_get_pixel / raw_set_pixel it's much easier to code, but it can be a little slower. So, bonus points if you just fix my incomplete code.

[tupp]

Look at the dots before debayering, not after

Thanks!  24 RGB pixel groups seems like a huge spread for debayering.

How does the debayering create the fading/sharpening effect in the dots?

[escho]

To try my incomplete code, compile raw2dng from this changeset: https://bitbucket.org/hudson/magic-lantern/commits/5bb7af2d168bda9ce3f6f4587a5ceb07703ea47a

=> you get some glue code in fix_bad_pixels, where you can start to experiment. That code will solve the problem for isolated bad pixels, so it's worth trying it.

Note that raw2dng uses very fast code via PA...PH / SET_PA...SET_PH macros. With raw_get_pixel / raw_set_pixel it's much easier to code, but it can be a little slower. So, bonus points if you just fix my incomplete code.

I just copied this "incomplete" code into the newest version of raw2dng.c and compiled it.

I compared it on an older rec of the moon. The result was amazing. All the bad pixels are removed, the big ones too. Darktable and ufraw didn´d succed in doing so, only rawtherapee showed the same amazing result.
Why don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.

Edgar

[a1ex]

Why don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.

In theory, the current solution (tagging the bad pixel in DNG files) is more elegant. In practice, only ACR reads these tags.

Why don't we use the code? because I thought this problem is too easy and I chose to wait for others to fix it. I just kept bugging people until you actually tried it

Or, as dmilligan said these days: it's no fun if somebody else does all the work for you.

[1%]

Heh, from some videos, clearly stuck red/purple pixels stay even in ACR.