I've been plagued by dots on raw recording since moving away from Tragic Lantern [New System]... I'm now using NewMem as the base, and updated with the nightly build from Dec 22nd. Whether I just use NewMem, or the nightly build, I get dots; for a long while I thought they were black dots which you can clearly see in this video, especially where there is a very light background.
And then I took a much closer look at them, and there is a sort of pattern to these invaders, as the image below shows:
(https://lh5.googleusercontent.com/-ZDU5GfLsyeQ/Ur166ZqafkI/AAAAAAAACXI/ifdYXMf4S9Y/s144/Screen%2520Shot%25202013-12-27%2520at%252012.13.59.png)
You can see that the dots are not black at all, but take on the surrounding hue; there's always a central 2x2 pixel formation which is surrounded by a sort of brighter halo.
I've seen and read loads of posts about pink dots, green dots and all sorts of dots, but haven't seen anything about this particular pattern... anyone know why it occurs, and how to get rid of it? It's the only blot on the otherwise excellent raw recording. Unfortunately it's a bit of a showstopper for me - I love using the raw workflow and the capabilities it brings, but an image full of clearly visible dots is of no use (to me).
I've tried the MLV recording too, using chroma smoothing options, but that made it worse :'(
This was something of a last ditch attempt to find a quick way of getting rid of / reducing the dots in raw recording. I decided to shoot a few seconds of my wife knitting - lit by a compact fluorescent light (camera right), a halogen light (above camera right) and tungsten light (directly above camera)... so a total mish-mash of lights!
Anyway, the clips were shot at 1152x432 (2.66:1) which gets automatically upres'ed to 1280x720 in Davinci Resolve; I wondered if scaling it down to 1024x383 to retain the 2.66:1 aspect ratio would get rid of the dots.
It didn't :(
But the raw handled different light sources really well :)
Unfortunately, it's quite difficult to see the detail that is still in the MP4 file that I uploaded - the Vimeo compression seems to have made it rather darker than it really is :( (Oh well, not to self - must take that into account next time).
Had the same problems with the black dots on my 600d with TL versions. I just gave up on RAW back then as the tweaked TL H.264 version is fine for my use cases.
It would be great though if there is an answer to the issue.
Quote from: vickersdc on December 27, 2013, 11:07:14 PM
This was something of a last ditch attempt to find a quick way of getting rid of / reducing the dots in raw recording. I decided to shoot a few seconds of my wife knitting - lit by a compact fluorescent light (camera right), a halogen light (above camera right) and tungsten light (directly above camera)... so a total mish-mash of lights!
Anyway, the clips were shot at 1152x432 (2.66:1) which gets automatically upres'ed to 1280x720 in Davinci Resolve; I wondered if scaling it down to 1024x383 to retain the 2.66:1 aspect ratio would get rid of the dots.
It didn't :(
But the raw handled different light sources really well :)
Unfortunately, it's quite difficult to see the detail that is still in the MP4 file that I uploaded - the Vimeo compression seems to have made it rather darker than it really is :( (Oh well, not to self - must take that into account next time).
I'm really curious to know the real reason you do not use TL and not ML NB ... @a1ex has said several times that packets of TL 2.0 for 600D are outdated (since June) and that the version is unstable (overheating, and memory blocks are not as vast as the current ML NB). Why use? Even in H.264, bitrate is the same.
I believe that only software we will be able to remove the dead pixels. Unfortunately I have no programming skills, but I'm studying to do so, since the matter related to the topic seems to be dead. No one is posting anything, not even discussing ... well, this is not the correct way to resolve things in an
open source project.
Quote from: Luiz Roberto dos Santos on December 28, 2013, 02:53:06 PM
I'm really curious to know the real reason you do not use TL and not ML NB ... @a1ex has said several times that packets of TL 2.0 for 600D are outdated (since June) and that the version is unstable (overheating, and memory blocks are not as vast as the current ML NB). Why use? Even in H.264, bitrate is the same.
I believe that only software we will be able to remove the dead pixels. Unfortunately I have no programming skills, but I'm studying to do so, since the matter related to the topic seems to be dead. No one is posting anything, not even discussing ... well, this is not the correct way to resolve things in an open source project.
I use both TL and the nightly builds... both give me the dots, which I do not believe are dead pixels at all. If I fire up TL [New System] from back in June, I can shoot raw with none of these pixels occurring. I've now stopped using raw, as much as I love the quality, having so many of these dots all over the image is no good :(
I'm currently back to using nightly builds and H.264.
Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0
Also Adobe Camera RAW will fix most of these problems.
So, TL dont give "dots"? Or less?
Quote from: RenatoPhoto on December 28, 2013, 06:28:58 PM
Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0
Also Adobe Camera RAW will fix most of these problems.
I test a1ex application, AHD like chroma smooth 2x2. It's help, but dont remove. But, ACR really solve it:
(http://img62.imageshack.us/img62/6615/w93.gif)
But, the workflow with ACR/LR is too slow for using on real life... at least for my weak hardware.
The black dots seems to be more "easier" to remove it than the chroma dots, indeed. Perhaps the problem could be solved, in future, we could create another application, merging with CS 2x2...
But, anyway, we would have several dng's and would have to re-use some other raw enginers to process, it's to slow workflow.
There is no way to
mux DNG to RAW again?
I tried the MLV video recording module and used mlv_dump with the no-cs, cs2x2, cs3x3 and cs5x5 chroma smoothing options and none of those worked.
Interesting results with Lightroom 5.3 - I don't have LR on my iMac (just RAWMagic -> Resolve 10 -> FCP X)... if it does a good job of removing the dots across the entire frame I may well pay the £70 / $100 for it as it would be worth it; but is the workflow with it?
Do you have to batch process the DNG's after raw2dng, deal with them one by one, or...?
DV.
Quote from: RenatoPhoto on December 28, 2013, 06:28:58 PM
Lost of information here: http://www.magiclantern.fm/forum/index.php?topic=6658.0
Also Adobe Camera RAW will fix most of these problems.
Thank you!!!
I've actually had RAWTherapee on my Mac for ages and never used it! It took me a while to find the relevant 'Hot/Dead Pixel' tickbox, but when I did... ;D Dots gone! Brilliant!
Now I need to work out how to batch process and save them - ideally back into DNG, but I think the only options are JPEG, TIFF or PNG. Anyway, I need to do some searching about using RAWTherapee, but I feel I'm a step closer to getting rids of those dots.
Many thanks for that link.
DV.
Quote from: vickersdc on December 28, 2013, 11:45:27 PM
I tried the MLV video recording module and used mlv_dump with the no-cs, cs2x2, cs3x3 and cs5x5 chroma smoothing options and none of those worked.
Interesting results with Lightroom 5.3 - I don't have LR on my iMac (just RAWMagic -> Resolve 10 -> FCP X)... if it does a good job of removing the dots across the entire frame I may well pay the £70 / $100 for it as it would be worth it; but is the workflow with it?
Do you have to batch process the DNG's after raw2dng, deal with them one by one, or...?
DV.
You can batch process, just only synchronize your changes... it's simple, but for a work "in real life" is a very slow process, especially if it will be lossless (you would have to export 16bit TIF, which are very heavy and slow to process).
RAW only have reason and will be popularized if workflow is fast enough, not everyone uses i7 to process their recordings :P
I think the process.
RAW> RAW2GPCF> 444 10bit quite usable (for win, for mac RAWMagic works fine, I think), but if you have to go through some application, 'unpack' .raw, no longer usable ... if it were possible to fix this problem with an application and make multiplexing for .RAW again, would be the ideial, indeed.
RAWTherappe is even slower. You can make batch: modify one DNG, click with right button on icon, copy settings, place on others, and put it on qeue... but the AMaZE is very slow too, and using the highlight recover is impossible in my oppinion.
Any developer can explain how these artifacts occur in the image? It is the line skipping? On 3x crop this occur too, so... I really don't understand it.
These bad pixel of 600D are not very easy to handle. I know only two opensource tools, which can remove them:
1. rawtherapee, as you said before
2. cr2hdr from the dualiso-modul of ML (but it only works for dualiso-stuff)
I think, if the bad-pix-code from cr2hdr can somehow be ported to raw2dng and to mlvdump, the problem has gone.
Edgar
Quote from: Luiz Roberto dos Santos on December 29, 2013, 12:15:17 AM
Any developer can explain how these artifacts occur in the image? It is the line skipping? On 3x crop this occur too, so... I really don't understand it.
I agree... it's odd that the 'prehistoric' TL [New System] didn't suffer from this, but anything subsequent does seem to...
Quote from: Luiz Roberto dos Santos on December 29, 2013, 12:15:17 AM
RAWTherappe is even slower. You can make batch: modify one DNG, click with right button on icon, copy settings, place on others, and put it on qeue... but the AMaZE is very slow too, and using the highlight recover is impossible in my opinion.
RAWTherapee is not too bad - using the command line to 'batch' process an entire directory full of DNGs doesn't take so long, although it does mean that you no longer have raw files to deal with, but 16-bit TIFF instead (and they take up more space!).
But I think RAWTherapee is do-able as part of the workflow... just. :)
I've described the proper solution here: http://www.magiclantern.fm/forum/index.php?topic=9625
It's very very easy, that's why I'm not implementing it ;)
Quote from: a1ex on December 29, 2013, 10:49:59 PM
I've described the proper solution here: http://www.magiclantern.fm/forum/index.php?topic=9625
It's very very easy, that's why I'm not implementing it ;)
I saw this thread some days ago. And I tried to understand the code, but I didn´t suceed. Yes, this is very easy for you, Alex. But for me with no coding skills in C, it´s a little bit harder. I will try it once again in the next days, hoping my little knowledge in bash-programming can help me. Let´s see...
Edgar
The first step is to compile the half-working code: it should remove most of the bad pixels, except for adjacent ones.
Once that works, simply extend the algorithm to handle groups of pixels instead of isolated ones.
The dots have a distinctive pattern, and, hence, each is larger than a single pixel. Each dot appears to affect 24 pixels.
The dots have a set location. They seem to gradually fade/sharpen depending on whether or not the adjacent area of the image is in focus (or, perhaps, on whether or not the camera thinks that the adjacent area is in focus).
Look at the dots before debayering, not after ;)
So, this is the source of cr2hdr to solve the bad pixels... it's just it a1ex? I can place it after the chroma smooth operations and compile?
static void find_and_fix_bad_pixels(int dark_noise, int bright_noise, int* raw2ev, int* ev2raw)
{
int w = raw_info.width;
int h = raw_info.height;
int black = raw_info.black_level;
//~ int white = raw_info.white_level;
printf("Looking for hot/cold pixels...\n");
/* hot pixel map */
unsigned short* hotpixel = malloc(w * h * sizeof(unsigned short));
memset(hotpixel, 0, w * h * sizeof(unsigned short));
int hot_pixels = 0;
int cold_pixels = 0;
int x,y;
for (y = 6; y < h-6; y ++)
{
for (x = 6; x < w-6; x ++)
{
int p = raw_get_pixel(x, y);
int is_hot = 0;
int is_cold = 0;
/* really dark pixels (way below the black level) are probably noise */
is_cold = (p < black - dark_noise*8);
/* we don't have no hot pixels on the bright exposure */
/* but we may have cold pixels */
if (!BRIGHT_ROW || is_cold)
{
/* let's look at the neighbours: is this pixel clearly brigher? (isolated) */
int neighbours[100];
int k = 0;
int i,j;
int fc0 = FC(x, y);
int b0 = is_bright[y%4];
for (i = -4; i <= 4; i++)
{
for (j = -4; j <= 4; j++)
{
if (i == 0 && j == 0)
continue;
/* only look at pixels of the same brightness */
if (is_bright[(y+i)%4] != b0)
continue;
/* only look at pixels of the same color */
if (FC(x+j, y+i) != fc0)
continue;
int p = raw_get_pixel(x+j, y+i);
neighbours[k++] = -p;
}
}
if (k <= 4) /* not enough data to draw a conclusion */
continue;
int max = -kth_smallest_int(neighbours, k, 1);
is_hot = (raw2ev[p] - raw2ev[max] > EV_RESOLUTION) && (max > black + 8*dark_noise);
if (fix_bad_pixels == 2) /* aggressive */
{
int second_max = -kth_smallest_int(neighbours, k, 2);
is_hot = ((raw2ev[p] - raw2ev[max] > EV_RESOLUTION/4) && (max > black + 8*dark_noise))
|| (raw2ev[p] - raw2ev[second_max] > EV_RESOLUTION/2);
}
if (is_hot)
{
hot_pixels++;
hotpixel[x + y*w] = -kth_smallest_int(neighbours, k, 2);
}
if (is_cold)
{
cold_pixels++;
hotpixel[x + y*w] = -median_int_wirth(neighbours, k);
}
}
}
}
/* apply the correction */
for (y = 0; y < h; y ++)
for (x = 0; x < w; x ++)
if (hotpixel[x + y*w])
raw_set_pixel16(x, y, debug_bad_pixels ? black : hotpixel[x + y*w]);
if (hot_pixels)
printf("Hot pixels : %d\n", hot_pixels);
if (cold_pixels)
printf("Cold pixels : %d\n", cold_pixels);
free(hotpixel);
}
edit: I analyse the non-demosaicing data, and it's just a dark pixel (ohhh, really?), the chroma artefacts is the effect of interpolation algorithm... but seems really simple to resolve it, but I am not a programmer. I will try.
Yeah, remove the hot pixel code, add some glue code, and should work.
To try my incomplete code, compile raw2dng from this changeset: https://bitbucket.org/hudson/magic-lantern/commits/5bb7af2d168bda9ce3f6f4587a5ceb07703ea47a
=> you get some glue code in fix_bad_pixels, where you can start to experiment. That code will solve the problem for isolated bad pixels, so it's worth trying it.
Note that raw2dng uses very fast code via PA...PH / SET_PA...SET_PH macros. With raw_get_pixel / raw_set_pixel it's much easier to code, but it can be a little slower. So, bonus points if you just fix my incomplete code.
Quote from: a1ex on December 30, 2013, 07:02:24 AM
Look at the dots before debayering, not after ;)
Thanks! 24 RGB pixel groups seems like a huge spread for debayering.
How does the debayering create the fading/sharpening effect in the dots?
Quote from: a1ex on December 30, 2013, 03:46:19 PM
To try my incomplete code, compile raw2dng from this changeset: https://bitbucket.org/hudson/magic-lantern/commits/5bb7af2d168bda9ce3f6f4587a5ceb07703ea47a
=> you get some glue code in fix_bad_pixels, where you can start to experiment. That code will solve the problem for isolated bad pixels, so it's worth trying it.
Note that raw2dng uses very fast code via PA...PH / SET_PA...SET_PH macros. With raw_get_pixel / raw_set_pixel it's much easier to code, but it can be a little slower. So, bonus points if you just fix my incomplete code.
I just copied this "incomplete" code into the newest version of raw2dng.c and compiled it.
I compared it on an older rec of the moon. The result was amazing. All the bad pixels are removed, the big ones too. Darktable and ufraw didn´d succed in doing so, only rawtherapee showed the same amazing result.
Why don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.
Edgar
QuoteWhy don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.
In theory, the current solution (tagging the bad pixel in DNG files) is more elegant. In practice, only ACR reads these tags.
Why don't we use the code? because I thought this problem is too easy and I chose to wait for others to fix it. I just kept bugging people until you actually tried it :D
Or, as dmilligan said these days: it's no fun if somebody else does all the work for you.
Heh, from some videos, clearly stuck red/purple pixels stay even in ACR.
I tried to bring the code from cr2hdr into raw2dng, but I run into problems, I cannot solve with my knowledge. You say, it´s very simple, but for me, it is impossible at the moment. So others have to work on this, if they want the badpixel-removal. I for my part have it now on my HD amd can use it.
If you want, I create a pull-request 8) :) ;D
Edgar
Hi!
I try compile, but the "int32" cannot be processed by the gcc. Can someone help me?
I start to learn "C" language, and this is really, really simple, but I can't compile the raw2dng (all "includes", lib, are here) .
I try it on CodeBlock, on Win8, but I am now with Xubuntu Saucy, and I dont try it here...
How did you compile? With a
make clean
make
in module raw_rec?
Edgar
edit:
and maybe, you could use 32bit gcc
I try just on Win8 with codeblocks, not on linux yet... I will try this night ;)
Quote from: escho on January 12, 2014, 08:42:36 PM
I just copied this "incomplete" code into the newest version of raw2dng.c and compiled it.
I compared it on an older rec of the moon. The result was amazing. All the bad pixels are removed, the big ones too. Darktable and ufraw didn´d succed in doing so, only rawtherapee showed the same amazing result.
Why don´t we use this code in raw2dng? Since it is incomplete, it can be completed later on. But for now the result is fine.
Wow! That is very interesting for the people who have this problem with 5D3... I thought this was resolved by Alex for 5D3 a long time ago but maybe it is not incorporated yet! It should bring some people back to RAW and make RAW recording more popular and friendly.
5D3 still has this problem?! have a link?
I did a summary of the old issues here so people could report the banding. I remember that you (Alex) developed an app for the 5D3 in which it went throug a series of bad and good images in which you seleceted the best images and some of those images had to deal with colored dots.
Here is the summary: http://www.magiclantern.fm/forum/index.php?topic=9564.msg91691#msg91691
The post that I recovered where these discussions happened are in the link above and also listed here:
[SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording ...
Lost
Jun 2
Recovered Here: https://drive.google.com/file/d/0B9z8Y0rg-pu8c2RIeUpUNHdIMHc/edit?usp=sharing
[SOLVED] Vertical banding in raw Canon 5D mark III - Magic Lantern
LOST
May 23 2013
Recovered Here: https://drive.google.com/file/d/0B9z8Y0rg-pu8c3J6dzVZQmdUdnM/edit?usp=sharing
5D MK3 Raw sensor ISO noise samples
http://www.magiclantern.fm/forum/index.php?topic=5609.0
May 23
I'm asking about the current state. Are there still any bad pixel problems on 5D3?
I could not find any in your summary, and your other links are several months old.
I just tried to write the "find_and_fix_bad_pixels"-function of cr2hdr into raw2dng. After a big fight, I finally succeded in compiling this stuff. Badpixels are removed now, but not all. Maybe, I moved to many things to the trash. I think, I must play a little bit more...
Edgar
I gave the "find_and_fix_bad_pixels"-function of cr2hdr another try and now, all cold pixels are removed. Nice! But I cannot say, that I understand all of this stuff, I did. Maybe, I have some questions, later.
Edgar
This is my "dirty" change of raw2dng: https://bitbucket.org/escho/escho-magiclantern/commits/10d46e39c9dc79300dece92f873100750ca4fd5d?at=unified
but, what can I do with
dark_noise
is_bright
I really don´t understand, where do they get their values from.
Edgar
dark_noise is the standard deviation from noise at low ISO (remember that you got the code from dual ISO). You don't know it from raw video files, but typical DR at ISO 100 is around 11 stops => consider it 8.
is_bright is to identify bright and dark lines in dual ISO. Some of the bad pixel problems are only present in one of the two exposures, so checking the other one will only take CPU time and maybe give false positives.
That means, I can say dark_noise = 8 ? Or am I wrong here?
And the is_bright-stuff can go to trash ?
Edgar
Yep.
How does the speed compare with plain raw2dng?
If the bad pixels are consistent (present in all frames at the same location), you can estimate the bad pixel locations from the first frame only (since they are under black level it doesn't matter what image is in the first frame), transform the hotpixel map into a small list (so you check 50 pixels instead of 2 million), and for subsequent frames, only compute the correction from the pixels on the list. That should run without any visible slowdown.
I will do the speedtest later, maybe tomorrow, because I´m away now for some hours.
The math for transforming the map is a thing, I don´t know how to do. No clue about this.
Edgar
Back again with some short speed tests, decoding a 4GB file, 6188 frames with a resolution of 832x480
plain raw2dng with no coldpix-removal: 55s
coldpix-removal with the imcomplete code of alex: 56s
coldpix-removal with the stuff from dual_iso: 64s
All tests started like this:
./raw2dng *.RAW > /dev/null
Edgar
Updated my repo: https://bitbucket.org/escho/escho-magiclantern/commits/c5736ebb03648322af134c29fd34d4ecec006b4f
Edgar
That's faster than I've expected. HDD or SSD?
Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
ST3000DM001-1CH166 HDD SATA
RAM 8GB
Edgar
Quote from: a1ex on January 20, 2014, 09:12:52 PM
Yep.
How does the speed compare with plain raw2dng?
If the bad pixels are consistent (present in all frames at the same location), you can estimate the bad pixel locations from the first frame only (since they are under black level it doesn't matter what image is in the first frame), transform the hotpixel map into a small list (so you check 50 pixels instead of 2 million), and for subsequent frames, only compute the correction from the pixels on the list. That should run without any visible slowdown.
succeeded, not quick, but dirty.
https://bitbucket.org/escho/escho-magiclantern/commits/0246e4467562f4a501de0feb72613c5f8695fafc
I have to test this with bigger files and with other sizes of the reduced badpixel map arrays. For the moment, I set the size of this array to 100 for 100 badpixel, which can be removed.
Speedtest:
same file as before
without badpix-remove: 49s
with badpix-remove: 49s
Edgar
Getting close :)
Some comments:
Functional:
- Only the pixel location remain constant, not the corrected value. You need to recompute the median for each frame.
Non-functional:
- in C, arrays are indexed from 0 to N-1 (and there's no bounds checking). With your code, you can run into a off-by-one error.
- here, 100 is a "magic number" used in more than one place; replace with a constant macro
- style: you can group X and Y into a "struct xy" or something similar:
struct xy { int x; int y; };
struct xy bad_pixel_list[1000];
do something with bad_pixel_list[i].x and bad_pixel_list[i].y
Quote from: a1ex on January 22, 2014, 10:59:58 PM
- in C, arrays are indexed from 0 to N-1 (and there's no bounds checking). With your code, you can run into a off-by-one error.
Reducing to
if (hotpixel[x + y*w] && z <= 99) should do the trick in this case?
Quote from: a1ex on January 22, 2014, 10:59:58 PM
- here, 100 is a "magic number" used in more than one place; replace with a constant macro
Something like this?
#define REDUCED_SIZE 99 For the other stuff I must sleep a night and work a day, before continuing ;)
Edgar
Now you've got 2 magic numbers :D
void repair_badpixel() /*Funktion zum Reparieren der gefundenen badpixel. Wird für jedes Frame ausgeführt*/
{
int badpix_list;
for (badpix_list = 0; badpix_list <= cold_pixels; badpix_list++) /*Schaue die gefundenen badpixel der Reihe nach an*/
{ int x = reduced_x[badpix_list];
int y = reduced_y[badpix_list];
int neighbours[100];
int i,j;
int k = 0;
int fc0 = FC(x, y);
for (i = -4; i <= 4; i++) /*Schaue die Nachbarn des gefundenen badpixel an, und zwar 4 in jede Richtung*/
{
for (j = -4; j <= 4; j++) /* dito */
{
if (i == 0 && j == 0) /* das badpixel selber wird nicht beachtet*/
continue;
if (FC(x+j, y+i) != fc0) /*und es werden nur Nachbarn der gleiche Farbe berpcksichtigt, die das badpixel haben sollte*/
continue;
int p = raw_get_pixel(x+j, y+i); /*Hole den Wert des jeweiligen Nachbarn*/
neighbours[k++] = -p; /*und speichere ihn in einer Liste*/
}
}
if (k <= 4)
continue;
reduced_pixmap[badpix_list] = -median_int_wirth(neighbours, k);
raw_set_pixel(x, y, reduced_pixmap[badpix_list]);
}
}
Why if(k <= 4)? On the first look I see no big differece, If I have this in the code or commented it out.
Edgar
Right; it has no effect and can be removed from both programs.
My new version: https://bitbucket.org/escho/escho-magiclantern/commits/73cdb580bb144c461abfb481da0b57f51822bd00
The struct isn´t implemented yet, must first look, how to work with such structs. And the german comments are only for my understanding.
Here some testpics, with and without coldpix-removing
(http://farm8.staticflickr.com/7356/12135315004_247e91d4ed.jpg) (http://www.flickr.com/photos/112024673@N05/12135315004/)
without_badpix (http://www.flickr.com/photos/112024673@N05/12135315004/) von seescho (http://www.flickr.com/people/112024673@N05/) auf Flickr
(http://farm3.staticflickr.com/2826/12135576776_ceb9ef7923.jpg) (http://www.flickr.com/photos/112024673@N05/12135576776/)
with_badpix (http://www.flickr.com/photos/112024673@N05/12135576776/) von seescho (http://www.flickr.com/people/112024673@N05/) auf Flickr
Edgar
Nice!
Structure is more of a style thing, don't worry too much about this.
About the bad pixel list, you don't know in advance how many bad pixels you have. So, you need to check if you still have space in the array before inserting a new item (otherwise you get a buffer overflow - of course, only when you will have more than 1000 pixels). You can use the COUNT() macro for this.
Also, it's clearer to start the counter from 0, not from -1 (so it always represents the number of items you have in the array, not N-1).
Quote from: a1ex on January 25, 2014, 05:22:51 PM
You can use the COUNT() macro for this.
Where do I find this makro for to have a look into it?
Edgar
It's used in many places in ML source.
Next version: https://bitbucket.org/escho/escho-magiclantern/commits/d8ee79b52f15834d1e94d2aca14a9f2dcf1c7d5c
working with struct (nice thing)
malloc isn´t needed anymore
And now, I´m looking, how to avoid this buffer overflow
Edgar
And the next one: https://bitbucket.org/escho/escho-magiclantern/commits/6411b87ba71a09c073fa6e756e8bfc3ff9e3a58a
fix buffer overflow
What so you mean about this solution?
A speed test gives the same results for with and without badpixel-removing. Doing this speedtest, I see some thing. The more frames are decoded, the more slow the decoding works (with the original raw2dng ). Here some times:
Start decoding: 0s
1000 frames: 3s
2000 frames: 6s
3000 frames: 10s
4000 frames: 19s
5000 frames: 26s
6000 frames: 37s
Why?
Edgar
Good job escho. I apologize to the community, to have walked away, but did not really have the ability to solve this problem... I need to learn more about code yet. :P
But, if I can help in something, such as testing, I'm here.
This seems to be moving along well! How can I help?
escho has an exe file in the downloads section; test it and report the results ;)
For to be exact, it´s a linux "exe-file". I cannot provide an exe file for windows.
And yes, it would be nice, if you could do some tests and report, what´s going on.
Edgar
Sorry, didn't notice.
Here's a binary compiled directly from escho's source. Run some tests, and if all is fine I'll update the main raw2dng with this one.
raw2dng_escho.exe (http://acoutts.com/a1ex/raw2dng_escho.exe)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Quote from: a1ex on January 27, 2014, 06:30:56 PM
Sorry, didn't notice.
Here's a binary compiled directly from escho's source. Run some tests, and if all is fine I'll update the main raw2dng with this one.
raw2dng_escho.exe (http://acoutts.com/a1ex/raw2dng_escho.exe)
So, I try it with footage from 600D, and I cant find any difference. I use your compilation vs oficial from g3gg0 topic, and process it with ufraw (ahd). Any idea?
I think I've compiled the wrong branch, can you download it again?
I will do a complete test on Friday. How well does it do with FPN? I know there's a fix for 5D3, but does the same fix work for the 600D?
There are no fixes about FPN yet. For this, I might have to record some optical black data. I want to study it on silent pictures first, because they include full optical black data; so if you have a burst of silent pictures that show this problem, upload them here: www.magiclantern.fm/forum/index.php?topic=9564
Quote from: a1ex on January 28, 2014, 06:54:05 AM
I think I've compiled the wrong branch, can you download it again?
Work very good now.
Any idea for calculate the line and column of the bad pix? There's some badpix than escho modifications not interpolate... I can indicate it on some file, but I think better indicate the line/column.
In gimp, put the mouse cursor on the bad pixel and read the coords from status bar. I believe MS Paint does it too.
0k.
On here:
X: 47,26
Y: 3,26
And [maybe] here:
X: 2,70
Y: 0,37
There's bad pix not removed.
I not test very much, but it's very effective! But, I think this is slow yet... on my garbage HD, this run approx. 89% more slow...
My crystal ball is brand new and shiny, there are no bad pixels on it at these coords ;)
Are you sure a1ex? I tested at least 4 footage with the 600D and the first coordinate badpix appear... ???
edit: see the first coord. on this archives https://app.box.com/s/t1voim3ew086ko0aze81
Can you upload a short raw-video, containing these bad pixels, please?
Edgar
Here (https://app.box.com/s/lyrjbuqsry4hk8031sxn)
Thanks for the file.
I downloaded and decoded it with the badpix-fix. All the coldpixels are removed:
(http://farm8.staticflickr.com/7378/12195287213_d98048a629.jpg) (http://www.flickr.com/photos/112024673@N05/12195287213/)
000036 (http://www.flickr.com/photos/112024673@N05/12195287213/) von seescho (http://www.flickr.com/people/112024673@N05/) auf Flickr
Opening with gimp shows no badpixels too
Edgar
any chance of a mac version?
I have no mac, so I cannot help
Edgar
For to discuss: https://bitbucket.org/escho/escho-magiclantern/commits/dad031cdc77d1a00e7c4984b48dbbedc371ad282
Edgar
Congrats escho, great.
Maybe an implementation of these code in MLV2DNG? Just one un-official mod, for now?
Thanks
mlv2dng is to big for my understanding at the moment, but I´m working on it (to understand it) :)
Edgar
Just shot a promo vid in ML raw on 5d3. I used batchelor 2.3 and swaped the raw2dngs.
Anyway i shot a lot of low FPS+high iso clips in this so there should be some pretty bad offenders that i would like to use this for so ill give it a whirl
i processed one though file though raw2dng_esco and the same file through regular raw2dng and in resolve both had equal amount of bad pix. i didnt notice a difference in the 2 clips. It was high iso probably 12800. Im sure i did something wrong. The pixels i usually see are white and a lot of the times they are in similar spots. strange thing is there are not as noticeable in acr. Im sure i did something wrong. I downloaded after the link was fixed. also are these white pix considered hot pix? and will this remedy hot pix?
Please upload a short clip somewhere.
Edgar
would you like the raw file?
I might be able to compile on Mac, someone would have to guide me through it though. I've only ever done small projects with just a few source and class files.
Quote from: a1ex on January 28, 2014, 07:40:53 AM
There are no fixes about FPN yet. For this, I might have to record some optical black data. I want to study it on silent pictures first, because they include full optical black data; so if you have a burst of silent pictures that show this problem, upload them here: www.magiclantern.fm/forum/index.php?topic=9564
I'll go ahead and shoot some tomorrow and upload them. It'd be awesome to get FPN correction because it's very apparent in the footage.
In regards to the current raw2dng_escho, it worked very well. Nice and speedy. The only exception was with clips that seemed to have cold pixels everywhere, other than that it works as expected.
Can you upload such a clip?
Quote from: escho on February 01, 2014, 10:43:18 AM
Yes
Edgar
Ok I used the raw2dng_escho from A1ex post. I sent you a pm with a test clip
I just found a high-iso-file, I recorded yesterday night, where the frames have cold pixels in different positions, too. The only difference to my other videos is, that I recorded it with mlv_rec and converted it to legacy raw with mlv_dump. Not sure, what happened.
Edgar
I remember some discussions saying that first frames in mlv_rec might be corrupted, but didn't check it myself. Worth looking into it.
I changed raw2dng to let it print the locations of the cold pixels. And I compared different frames (not only the first one). I see three things:
1. A part of the cold pixels have the same position in every frame
2. Other cold pixels have have the same position in the frames, but not in every frame.
3. The rest has completely different positions in the frames
I will try some other clips from yesterday later
Edgar
In this case, you may want to extend the algorithm by scanning for example the first 10 frames for bad pixels. If they all stay in the same location, you don't need to continue scanning for the rest of the frames (just use the cached locations). If there are changes, just scan all the frames, a little slower, but should remove them all.
This optimization is a little more complex though; I can look into it hopefully next week. Probably only matters for those who use SSDs.
Yes, scanning all frames for cold pixels removes all of them. But I´m testing yet. An interim result:
ISO 6400 shows this behaviour with the different locations.
ISO 100 - ISO 1600 (tested not all ISOs in this range): The cold pixels have the same positon in all frames.
Edgar
Yes, comfirmed for my camera.
ISO 6400 different locations
ISO 1600 same locations.
I didn´t try ISO 3200 yesterday.
Edgar
Quote from: escho on February 01, 2014, 07:51:00 PM
Yes, comfirmed for my camera.
ISO 6400 different locations
ISO 1600 same locations.
I didn´t try ISO 3200 yesterday.
Edgar
Maybe because they are not real ISO's, but pushed digitally?
Quote from: a1ex on February 01, 2014, 09:40:30 AM
Can you upload such a clip?
The one's with a lot of dead pixels or the FPN clips?
A nice test result for my 600D:
no crop, cold pixels per frame
ISO 100: 34
ISO 200: 34
ISO 400: 34
ISO 800: 42
ISO 1600: 57
ISO 3200: 797
ISO 6400: 797 - 799
5x-crop cold pixels per frame
ISO 100: 16
ISO 200: 16
ISO 400: 16
ISO 800: 19
ISO 1600: 27
ISO 3200: 109
ISO 6400: 111 - 128
Edgar
What about commandline-options?
An option for scanning all frames for cold pixel (default scanning only the first frame)
And chroma smooth could be enabled by such an option, too.
Just an idea...
Edgar
This modification is really necessary? I mean, people have come to conclusion that ISO's greater than 1600 are pushed digitally by the camera, and a1ex has said he prefers to push this information in post-production than letting the camera do.
What your thoughts on this? I think we could save time about it and turn in the study of ADTG, which seems the most promising. It is just a warning in the official announcement... :P
Quote from: hjfilmspeed on January 30, 2014, 07:14:51 PM
Just shot a promo vid in ML raw on 5d3. I used batchelor 2.3 and swaped the raw2dngs.
Anyway i shot a lot of low FPS+high iso clips in this so there should be some pretty bad offenders that i would like to use this for so ill give it a whirl
i processed one though file though raw2dng_esco and the same file through regular raw2dng and in resolve both had equal amount of bad pix. i didnt notice a difference in the 2 clips. It was high iso probably 12800. Im sure i did something wrong. The pixels i usually see are white and a lot of the times they are in similar spots. strange thing is there are not as noticeable in acr. Im sure i did something wrong. I downloaded after the link was fixed. also are these white pix considered hot pix? and will this remedy hot pix?
Hi
I looked at your raw-file and decoded it. There are no cold pixels in the frames. The dark level is 2044. That means, all dots with a value of -2044 will get recognized as cold pixels and removed. The darkest pixel in your frames is -145, that´s somewhere around dark noise, I guess. At the right side of the frames, I see a red dot, This pixel has a value of a little bit over 1000.
So the cold pixel fix works correct, if it is doing nothing. This fix is not designed to correct hot pixels.
I looked at the white dots, too. The all have values inbetween 1000 and 1500.
Edgar
Quote from: Luiz Roberto dos Santos on February 02, 2014, 01:42:03 AM
This modification is really necessary? I mean, people have come to conclusion that ISO's greater than 1600 are pushed digitally by the camera, and a1ex has said he prefers to push this information in post-production than letting the camera do.
What your thoughts on this? I think we could save time about it and turn in the study of ADTG, which seems the most promising. It is just a warning in the official announcement... :P
I do not force someone to code this into raw2dng, so all you guys can work in ADTG, if you want :P ;) I will try to work this out for myself as a next step in my C-learning curve. And I for myself would like to have the possibility of the commandline-options, because this would prevent me from working with different raw2dng-builds, as I´m doing now. :)
Edgar
@escho i had a feeling they were hot pix and that its different then cold. Thank you for taking the time to download and look. I think this is something worth investing time into since it will save you tons of post work
fix cold pixels and mlv_dump
I played a bit with mlv_dump-code in order to bring the cold pixel fix into mlv_dump. It seem not to be as complicated as I thougt before. I think, I´ll give it a try, if g3gg0 don´t want to port it himself.
Edgar
now that - cold pixels and mlv_dump would be just great!
I think, I´ve got it:
edgar@linux-uoww:~/mlv_test/mit/1> ./mlv_dump --dng --fixcp *.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'M07-2217.MLV'
- Convert to DNG frames
- Output into 'M07-2217_frame_'
File M07-2217.MLV opened
File M07-2217.M00 not existing.
Processing...
Vertical stripes correction:
1.000 1.000 1.002 1.001 1.001 1.001 1.002 1.003
Cold pixels : 456
Reached end of chunk 1/1 after 291 blocks
Processed 124 video frames
Done
Must do some tests
Edgar
What do you think about this?
link removed (experimental code)
Edgar
I ported the cold-pixel-fix from raw2dng to mlv_dump. It is in ML-source-code now. If you are interested in, give it a try. The commandline option for mlv_dump is:
--fixcp
Edgar
Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.
Quote from: 1% on February 15, 2014, 05:51:02 PM
Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.
Do you mean this one: https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/8c602fcd3dd8ed6a1fb77417dd2ef4a41a4892f7 ?
For giving it a try, I need a short explanation in easy words, what this stuff :) is meant for and how I can test it with my camera. Is this commit the only change, which has to be done? Or are there other things to take into account?
Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.
Edgar
Quote from: escho on February 15, 2014, 10:55:58 PM
Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.
These things are far from easy. You can follow this thread for details: www.magiclantern.fm/forum/index.php?topic=10443
but EDMAC is not yet fully understood, so we don't know much more either (at this stage is mostly trial and error). But we are making small steps in understanding it (and this is one of them).
A general description of EDMAC is here: http://magiclantern.wikia.com/wiki/Register_Map#EDMAC
Quote from: escho on February 07, 2014, 10:24:09 PM
What do you think about this?
https://bitbucket.org/escho/escho-magiclantern/branch/cold-pix-fix
Edgar
It's a dead link
Quote from: escho on February 15, 2014, 04:31:48 PM
I ported the cold-pixel-fix from raw2dng to mlv_dump. Should be in the next nightly. If you are interested in, give it a try. The commandline option for mlv_dump is:
--fixcp
Edgar
Do I need to download a newer version of mlv_dump.exe to use this? Or is it in camera where this fix takes place?
This is the main commit, the other one picks a better edmac channel:
Try a few more :)
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/3c3f7f903e2540135d58c119432647ed5d392128?at=unified
Quote from: duey101 on February 16, 2014, 04:09:34 AM
It's a dead link
Yes, I removed the link. This was experimental sourcecode from my repo
Edgar
Quote from: duey101 on February 16, 2014, 06:32:23 AM
Do I need to download a newer version of mlv_dump.exe to use this? Or is it in camera where this fix takes place?
Oh, I didn´t realize, that the nightlys only provide the modules and not tools like raw2dng or mlv_dump. Sorry for this misinformation. You must compile this by yourself or wait, till mlv_rec, g3gg0 provides on his site, is updated.
Edgar
slurp stuff...
Will read a bit to understand, what´s going on
Edgar
slurp stuff...
I read the code and looked at the links, you gave me. I have to admit, that this all is beyond my understanding at the moment (maybe in a year or two it´s better? :) ). So I cannot help.
@ 1%
Maybe you can create a little pull request with this stuff to ML, if you think, it is ready for use?
Edgar
raw2dng
I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.
The cold pixels in the first frame are detected and fixed, no problem. But the second frame gives this error. Code from raw2dng.c:
int framenumber;
for (framenumber = 0; framenumber < lv_rec_footer.frameCount; framenumber++)
{
printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);
fflush(stdout);
printf("\rdebug_frame : %d \n", (framenumber));
int r = fread(raw, 1, lv_rec_footer.frameSize, fi);
CHECK(r == lv_rec_footer.frameSize, "fread");
raw_info.buffer = raw;
/* uncomment if the raw file is recovered from a DNG with dd */
//~ reverse_bytes_order(raw, lv_rec_footer.frameSize);
char fn[100];
snprintf(fn, sizeof(fn), "%s%06d.dng", prefix, framenumber);
fix_vertical_stripes();
find_and_fix_cold_pixels(fix_cold_pixels, framenumber);
#ifdef CHROMA_SMOOTH
chroma_smooth();
#endif
dng_set_framerate(lv_rec_footer.sourceFpsx1000);
save_dng(fn, &raw_info);
}
Working on the second frame, "printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);" is correctly displayed.
Than comes this fflush, which don´t understand.
And my debug-print, which follows to the fflush, isn´t displayed anymore: "frintf("\rdebug_frame : %d \n", (framenumber));"
No clue, what´s going on here.
To avoid this error, I can set the max_cold_pixels to maybe 4000. Than all is fine and the 3500 cold pixels, I forced in an record, are correctly removed. But that is no solution. I want to know, why this error comes in, if there are to much cold pixels found.
Edgar
escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)
escho,
I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:
Cold pixels : 0
I looked quickly through the source code and thought I saw something about not targeting pixels that are in over-exposed regions. The dark pixels I have are ONLY in over-exposed regions
If you have time maybe you can investigate? If not I will try, but I am just starting to learn C so it may be a long while!
EDIT: Also, here's is a more thorough description of my problem:
https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png
http://www.magiclantern.fm/forum/index.php?topic=7473.msg103033#msg103033
Quote from: sarotaz on February 24, 2014, 09:25:50 PM
escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)
mlv_dump calls the cold-pixel-fix-function from raw2dng. the commandline-option, I integrated in mlv_dump, is --fixcp. You may have to compile mlv_dump by yourself to get this option.
Edgar
Quote from: escho on February 17, 2014, 01:14:11 AM
raw2dng
I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.
fixed
Edgar
Quote from: davegravy on February 25, 2014, 04:53:23 AM
escho,
I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:
Please, can you upload somewhere a very short RAW or MLV file with your problem?
The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours
Edgar
Quote from: escho on February 25, 2014, 08:48:59 PM
Please, can you upload somewhere a very short RAW or MLV file with your problem?
The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours
Edgar
Hi Edgar,
Here's a short RAW:
https://dl.dropboxusercontent.com/u/30586759/M25-2141.RAW
In this example there is only 1 pixel that is bad and it happens on the 12th frame (I have some footage that has many more of the artifacts if you need it for testing but the RAW files are huge). Look for the bad pixel towards the bottom left of the flame. It is there if you convert to DNG with raw2dng cs2x2 but not if you use rawanizer.
EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!
Quote from: davegravy on February 26, 2014, 04:04:18 AM
EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!
I give it a short try. Yes, the badpix comes in with croma smoothing (cs3x3 and cs5x5 modes too). the cold pixel fix can do nothing here, because cold pixels (and hot pixels, I,m experimenting with at the moment) are fixed before chroma smoothing.
Edgar