[600D RAW] Dots, dots and more dots.

[hjfilmspeed]

@escho i had a feeling they were hot pix and that its different then cold. Thank you for taking the time to download and look. I think this is something worth investing time into since it will save you tons of post work

[escho]

fix cold pixels and mlv_dump

I  played  a bit with mlv_dump-code in order to bring the cold pixel fix into mlv_dump. It seem not to be as complicated as I thougt before. I think, I´ll give it a try, if g3gg0 don´t want to port it himself.

Edgar

[xaled]

now that - cold pixels and mlv_dump would be just great!

[escho]

I think, I´ve got it:

Code: [Select]

edgar@linux-uoww:~/mlv_test/mit/1> ./mlv_dump --dng --fixcp *.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-2217.MLV'
   - Convert to DNG frames
   - Output into 'M07-2217_frame_'
File M07-2217.MLV opened
File M07-2217.M00 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  1.002  1.001  1.001  1.001  1.002  1.003
Cold pixels : 456                             
Reached end of chunk 1/1 after 291 blocks
Processed 124 video frames
Done
Must do some tests

Edgar

[escho]

What do you think about this?

link removed (experimental code)

Edgar

[escho]

I ported the cold-pixel-fix from raw2dng to mlv_dump. It is in ML-source-code now. If you are interested in, give it a try. The commandline option for mlv_dump is:

Code: [Select]

--fixcp

Edgar

[1%]

Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

[escho]

Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

Do you mean this one: https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/8c602fcd3dd8ed6a1fb77417dd2ef4a41a4892f7 ?

For giving it a try, I need a short explanation in easy words, what this stuff is meant for and how I can test it with my camera. Is this commit the only change, which has to be done? Or are there other things to take into account?

Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

Edgar

[a1ex]

Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

These things are far from easy. You can follow this thread for details: www.magiclantern.fm/forum/index.php?topic=10443
but EDMAC is not yet fully understood, so we don't know much more either (at this stage is mostly trial and error). But we are making small steps in understanding it (and this is one of them).

A general description of EDMAC is here: http://magiclantern.wikia.com/wiki/Register_Map#EDMAC

[duey101]

What do you think about this?
https://bitbucket.org/escho/escho-magiclantern/branch/cold-pix-fix
Edgar

It's a dead link

[duey101]

I ported the cold-pixel-fix from raw2dng to mlv_dump. Should be in the next nightly. If you are interested in, give it a try. The commandline option for mlv_dump is:

Code: [Select]

--fixcp

Edgar

Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place? 

[1%]

This is the main commit, the other one picks a better edmac channel:
Try a few more

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/3c3f7f903e2540135d58c119432647ed5d392128?at=unified

[escho]

It's a dead link

Yes, I removed the link. This was experimental sourcecode from my repo

Edgar

[escho]

Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place?

Oh, I didn´t realize, that the nightlys only provide the modules and not tools like raw2dng or mlv_dump. Sorry for this misinformation. You must compile this by yourself or wait, till mlv_rec, g3gg0 provides on his site, is updated.

Edgar

[escho]

slurp stuff...

Will read a bit to understand, what´s going on

Edgar

[escho]

slurp stuff...

I read the code and looked at the links, you gave me. I have to admit, that this all is beyond my understanding at the moment (maybe in a year or two it´s better? ). So I cannot help.

@ 1%
Maybe you can create a little pull request with this stuff to ML, if you think, it is ready for use?

Edgar

[escho]

raw2dng

I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

The cold pixels in the first frame are detected and fixed, no problem. But the second frame gives this error. Code from raw2dng.c:

Code: [Select]

int framenumber;
    for (framenumber = 0; framenumber < lv_rec_footer.frameCount; framenumber++)
    {
        printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);
       
        fflush(stdout);
       
        printf("\rdebug_frame : %d                             \n", (framenumber));
        int r = fread(raw, 1, lv_rec_footer.frameSize, fi);
        CHECK(r == lv_rec_footer.frameSize, "fread");
        raw_info.buffer = raw;
       
        /* uncomment if the raw file is recovered from a DNG with dd */
        //~ reverse_bytes_order(raw, lv_rec_footer.frameSize);
       
        char fn[100];
        snprintf(fn, sizeof(fn), "%s%06d.dng", prefix, framenumber);

        fix_vertical_stripes();
         
        find_and_fix_cold_pixels(fix_cold_pixels, framenumber);

        #ifdef CHROMA_SMOOTH
        chroma_smooth();
        #endif

        dng_set_framerate(lv_rec_footer.sourceFpsx1000);
        save_dng(fn, &raw_info);
    }   
Working on the second frame, "printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);" is correctly displayed.
Than comes this fflush, which  don´t understand.
And my debug-print, which follows to the fflush, isn´t displayed anymore: "frintf("\rdebug_frame : %d                             \n", (framenumber));"

No clue, what´s going on here.

To avoid this error, I can set the max_cold_pixels to maybe 4000. Than all is fine and the 3500 cold pixels, I forced in an record, are correctly removed. But that is no solution. I want to know, why this error comes in, if there are to much cold pixels found.

Edgar

[sarotaz]

escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

[davegravy]

escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:

Code: [Select]

Cold pixels : 0
I looked quickly through the source code and thought I saw something about not targeting pixels that are in over-exposed regions. The dark pixels I have are ONLY in over-exposed regions
If you have time maybe you can investigate? If not I will try, but I am just starting to learn C so it may be a long while!

EDIT: Also, here's is a more thorough description of my problem:

https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png

http://www.magiclantern.fm/forum/index.php?topic=7473.msg103033#msg103033

[escho]

escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

mlv_dump calls the cold-pixel-fix-function from raw2dng. the commandline-option, I integrated in mlv_dump,  is --fixcp. You may have to compile mlv_dump by yourself to get this option.

Edgar

[escho]

raw2dng
I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

fixed

Edgar

[escho]

escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:

Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar

[davegravy]

Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar

Hi Edgar,

Here's a short RAW:

https://dl.dropboxusercontent.com/u/30586759/M25-2141.RAW

In this example there is only 1 pixel that is bad and it happens on the 12th frame (I have some footage that has many more of the artifacts if you need it for testing but the RAW files are huge). Look for the bad pixel towards the bottom left of the flame. It is there if you convert to DNG with raw2dng cs2x2 but not if you use rawanizer.

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

[escho]

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

I give it a short try. Yes, the badpix comes in with croma smoothing (cs3x3 and cs5x5 modes too). the cold pixel fix can do nothing here, because cold pixels (and hot pixels, I,m experimenting with at the moment) are fixed before chroma smoothing.

Edgar