Author Topic: [600D RAW] Dots, dots and more dots. (Read 75371 times)

escho · « **Reply #25 on:** January 12, 2014, 09:10:40 PM »

I tried to bring the code from cr2hdr into raw2dng, but I run into problems, I cannot solve with my knowledge. You say, it´s very simple, but for me, it is impossible at the moment. So others have to work on this, if they want the badpixel-removal. I for my part have it now on my HD amd can use it.
If you want, I create a pull-request

Edgar

Luiz Roberto dos Santos · « **Reply #26 on:** January 12, 2014, 09:54:51 PM »

Hi!
I try compile, but the "int32" cannot be processed by the gcc. Can someone help me?
I start to learn "C" language, and this is really, really simple, but I can't compile the raw2dng (all "includes", lib, are here) .
I try it on CodeBlock, on Win8, but I am now with Xubuntu Saucy, and I dont try it here...

escho · « **Reply #27 on:** January 12, 2014, 10:11:02 PM »

How did you compile? With a

Code: [Select]

make clean
make

in module raw_rec?

Edgar

edit:
and maybe, you could use 32bit gcc

Luiz Roberto dos Santos · « **Reply #28 on:** January 13, 2014, 01:43:07 AM »

I try just on Win8 with codeblocks, not on linux yet... I will try this night

RenatoPhoto · « **Reply #29 on:** January 15, 2014, 06:42:05 PM »

Quote from: escho on January 12, 2014, 08:42:36 PM

I just copied this "incomplete" code into the newest version of raw2dng.c and compiled it.

I compared it on an older rec of the moon. The result was amazing. All the bad pixels are removed, the big ones too. Darktable and ufraw didn´d succed in doing so, only rawtherapee showed the same amazing result.
Why don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.

Wow! That is very interesting for the people who have this problem with 5D3... I thought this was resolved by Alex for 5D3 a long time ago but maybe it is not incorporated yet! It should bring some people back to RAW and make RAW recording more popular and friendly.

a1ex · « **Reply #30 on:** January 15, 2014, 06:45:18 PM »

5D3 still has this problem?! have a link?

RenatoPhoto · « **Reply #31 on:** January 15, 2014, 07:19:02 PM »

I did a summary of the old issues here so people could report the banding. I remember that you (Alex) developed an app for the 5D3 in which it went throug a series of bad and good images in which you seleceted the best images and some of those images had to deal with colored dots.

Here is the summary: http://www.magiclantern.fm/forum/index.php?topic=9564.msg91691#msg91691

The post that I recovered where these discussions happened are in the link above and also listed here:
[SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording ...
Lost
Jun 2
Recovered Here: https://drive.google.com/file/d/0B9z8Y0rg-pu8c2RIeUpUNHdIMHc/edit?usp=sharing

[SOLVED] Vertical banding in raw Canon 5D mark III - Magic Lantern
LOST
May 23 2013
Recovered Here: https://drive.google.com/file/d/0B9z8Y0rg-pu8c3J6dzVZQmdUdnM/edit?usp=sharing

5D MK3 Raw sensor ISO noise samples
http://www.magiclantern.fm/forum/index.php?topic=5609.0
May 23

a1ex · « **Reply #32 on:** January 15, 2014, 07:29:18 PM »

I'm asking about the current state. Are there still any bad pixel problems on 5D3?

I could not find any in your summary, and your other links are several months old.

escho · « **Reply #33 on:** January 19, 2014, 10:20:56 PM »

I just tried to write the "find_and_fix_bad_pixels"-function of cr2hdr into raw2dng. After a big fight, I finally succeded in compiling this stuff. Badpixels are removed now, but not all. Maybe, I moved to many things to the trash. I think, I must play a little bit more...

Edgar

escho · « **Reply #34 on:** January 20, 2014, 08:32:58 PM »

I gave the "find_and_fix_bad_pixels"-function of cr2hdr another try and now, all cold pixels are removed. Nice! But I cannot say, that I understand all of this stuff, I did. Maybe, I have some questions, later.

Edgar

escho · « **Reply #35 on:** January 20, 2014, 08:54:23 PM »

This is my "dirty" change of raw2dng: https://bitbucket.org/escho/escho-magiclantern/commits/10d46e39c9dc79300dece92f873100750ca4fd5d?at=unified

but, what can I do with

Code: [Select]

dark_noise
is_bright

I really don´t understand, where do they get their values from.

Edgar

a1ex · « **Reply #36 on:** January 20, 2014, 08:57:41 PM »

dark_noise is the standard deviation from noise at low ISO (remember that you got the code from dual ISO). You don't know it from raw video files, but typical DR at ISO 100 is around 11 stops => consider it 8.

is_bright is to identify bright and dark lines in dual ISO. Some of the bad pixel problems are only present in one of the two exposures, so checking the other one will only take CPU time and maybe give false positives.

escho · « **Reply #37 on:** January 20, 2014, 09:06:25 PM »

That means, I can say dark_noise = 8 ? Or am I wrong here?
And the is_bright-stuff can go to trash ?

Edgar

a1ex · « **Reply #38 on:** January 20, 2014, 09:12:52 PM »

Yep.

How does the speed compare with plain raw2dng?

If the bad pixels are consistent (present in all frames at the same location), you can estimate the bad pixel locations from the first frame only (since they are under black level it doesn't matter what image is in the first frame), transform the hotpixel map into a small list (so you check 50 pixels instead of 2 million), and for subsequent frames, only compute the correction from the pixels on the list. That should run without any visible slowdown.

escho · « **Reply #39 on:** January 20, 2014, 09:24:30 PM »

I will do the speedtest later, maybe tomorrow, because I´m away now for some hours.

The math for transforming the map is a thing, I don´t know how to do. No clue about this.

Edgar

escho · « **Reply #40 on:** January 20, 2014, 11:07:16 PM »

Back again with some short speed tests, decoding a 4GB file, 6188 frames with a resolution of 832x480

plain raw2dng with no coldpix-removal: 55s
coldpix-removal with the imcomplete code of alex: 56s
coldpix-removal with the stuff from dual_iso: 64s

All tests started like this:
./raw2dng *.RAW > /dev/null

Edgar

escho · « **Reply #41 on:** January 21, 2014, 12:21:43 AM »

Updated my repo: https://bitbucket.org/escho/escho-magiclantern/commits/c5736ebb03648322af134c29fd34d4ecec006b4f

Edgar

a1ex · « **Reply #42 on:** January 21, 2014, 01:08:52 AM »

That's faster than I've expected. HDD or SSD?

escho · « **Reply #43 on:** January 21, 2014, 05:51:14 PM »

Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
ST3000DM001-1CH166 HDD SATA
RAM 8GB

Edgar

escho · « **Reply #44 on:** January 22, 2014, 10:30:38 PM »

Quote from: a1ex on January 20, 2014, 09:12:52 PM

Yep.

How does the speed compare with plain raw2dng?

If the bad pixels are consistent (present in all frames at the same location), you can estimate the bad pixel locations from the first frame only (since they are under black level it doesn't matter what image is in the first frame), transform the hotpixel map into a small list (so you check 50 pixels instead of 2 million), and for subsequent frames, only compute the correction from the pixels on the list. That should run without any visible slowdown.

succeeded, not quick, but dirty.
https://bitbucket.org/escho/escho-magiclantern/commits/0246e4467562f4a501de0feb72613c5f8695fafc

I have to test this with bigger files and with other sizes of the reduced badpixel map arrays. For the moment, I set the size of this array to 100 for 100 badpixel, which can be removed.

Speedtest:
same file as before
without badpix-remove: 49s
with badpix-remove: 49s

Edgar

a1ex · « **Reply #45 on:** January 22, 2014, 10:59:58 PM »

Getting close

Some comments:

Functional:
- Only the pixel location remain constant, not the corrected value. You need to recompute the median for each frame.

Non-functional:
- in C, arrays are indexed from 0 to N-1 (and there's no bounds checking). With your code, you can run into a off-by-one error.
- here, 100 is a "magic number" used in more than one place; replace with a constant macro
- style: you can group X and Y into a "struct xy" or something similar:

Code: [Select]

struct xy { int x; int y; };
struct xy bad_pixel_list[1000];
do something with bad_pixel_list[i].x and bad_pixel_list[i].y

escho · « **Reply #46 on:** January 22, 2014, 11:48:39 PM »

Quote from: a1ex on January 22, 2014, 10:59:58 PM

- in C, arrays are indexed from 0 to N-1 (and there's no bounds checking). With your code, you can run into a off-by-one error.

Reducing to

Code: [Select]

if (hotpixel[x + y*w] && z <= 99) should do the trick in this case?

Quote from: a1ex on January 22, 2014, 10:59:58 PM

- here, 100 is a "magic number" used in more than one place; replace with a constant macro

Something like this?

Code: [Select]

#define REDUCED_SIZE 99
For the other stuff I must sleep a night and work a day, before continuing

Edgar

a1ex · « **Reply #47 on:** January 23, 2014, 08:01:04 AM »

Now you've got 2 magic numbers

escho · « **Reply #48 on:** January 25, 2014, 04:05:46 PM »

Code: [Select]

void repair_badpixel() /*Funktion zum Reparieren der gefundenen badpixel. Wird für jedes Frame ausgeführt*/
{
    int badpix_list;
    
    for (badpix_list = 0; badpix_list <= cold_pixels; badpix_list++) /*Schaue die gefundenen badpixel der Reihe nach an*/
    {	int x = reduced_x[badpix_list];
	int y = reduced_y[badpix_list];
      
	int neighbours[100];
	int i,j;
	int k = 0;
	int fc0 = FC(x, y);
	
	for (i = -4; i <= 4; i++) /*Schaue die Nachbarn des gefundenen badpixel an, und zwar 4 in jede Richtung*/
	{
	    for (j = -4; j <= 4; j++) /* dito */
	    {
		if (i == 0 && j == 0) /* das badpixel selber wird nicht beachtet*/
		    continue;
                        
		if (FC(x+j, y+i) != fc0) /*und es werden nur Nachbarn der gleiche Farbe berpcksichtigt, die das badpixel haben sollte*/
		    continue;
                        
		int p = raw_get_pixel(x+j, y+i); /*Hole den Wert des jeweiligen Nachbarn*/
		neighbours[k++] = -p; /*und speichere ihn in einer Liste*/
	    }
	}
	if (k <= 4)
	    continue;
	
	reduced_pixmap[badpix_list] = -median_int_wirth(neighbours, k);
	raw_set_pixel(x, y, reduced_pixmap[badpix_list]);
    }
}

Why if(k <= 4)? On the first look I see no big differece, If I have this in the code or commented it out.

Edgar

a1ex · « **Reply #49 on:** January 25, 2014, 04:10:42 PM »

Right; it has no effect and can be removed from both programs.

News:

Author Topic: [600D RAW] Dots, dots and more dots. (Read 75371 times)