[600D RAW] Dots, dots and more dots.

Started by vickersdc, December 27, 2013, 10:12:58 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

hjfilmspeed

@escho i had a feeling they were hot pix and that its different then cold. Thank you for taking the time to download and look. I think this is something worth investing time into since it will save you tons of post work

escho

fix cold pixels and mlv_dump

I  played  a bit with mlv_dump-code in order to bring the cold pixel fix into mlv_dump. It seem not to be as complicated as I thougt before. I think, I´ll give it a try, if g3gg0 don´t want to port it himself.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

xaled

now that - cold pixels and mlv_dump would be just great!

escho

I think, I´ve got it:

edgar@linux-uoww:~/mlv_test/mit/1> ./mlv_dump --dng --fixcp *.MLV

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-2217.MLV'
   - Convert to DNG frames
   - Output into 'M07-2217_frame_'
File M07-2217.MLV opened
File M07-2217.M00 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  1.002  1.001  1.001  1.001  1.002  1.003
Cold pixels : 456                             
Reached end of chunk 1/1 after 291 blocks
Processed 124 video frames
Done


Must do some tests

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

What do you think about this?

link removed (experimental code)

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

I ported the cold-pixel-fix from raw2dng to mlv_dump. It is in ML-source-code now. If you are interested in, give it a try. The commandline option for mlv_dump is:

--fixcp


Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

1%

Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

escho

Quote from: 1% on February 15, 2014, 05:51:02 PM
Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

Do you mean this one: https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/8c602fcd3dd8ed6a1fb77417dd2ef4a41a4892f7 ?

For giving it a try, I need a short explanation in easy words, what this stuff :) is meant for and how I can test it with my camera. Is this commit the only change, which has to be done? Or are there other things to take into account?

Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

a1ex

Quote from: escho on February 15, 2014, 10:55:58 PM
Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

These things are far from easy. You can follow this thread for details: www.magiclantern.fm/forum/index.php?topic=10443
but EDMAC is not yet fully understood, so we don't know much more either (at this stage is mostly trial and error). But we are making small steps in understanding it (and this is one of them).

A general description of EDMAC is here: http://magiclantern.wikia.com/wiki/Register_Map#EDMAC

duey101

Gear: Canon 5D Mark III, 60D, EF 70-200L, EF 85mm, EF 50mm, EF 10-22mm.  Platform: Dual Boot Win 7 Ultimate x64/Mavericks Hackintosh, PPro CC, LR5.

duey101

Quote from: escho on February 15, 2014, 04:31:48 PM
I ported the cold-pixel-fix from raw2dng to mlv_dump. Should be in the next nightly. If you are interested in, give it a try. The commandline option for mlv_dump is:

--fixcp


Edgar

Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place? 
Gear: Canon 5D Mark III, 60D, EF 70-200L, EF 85mm, EF 50mm, EF 10-22mm.  Platform: Dual Boot Win 7 Ultimate x64/Mavericks Hackintosh, PPro CC, LR5.

1%


escho

Quote from: duey101 on February 16, 2014, 04:09:34 AM
It's a dead link

Yes, I removed the link. This was experimental sourcecode from my repo

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

Quote from: duey101 on February 16, 2014, 06:32:23 AM
Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place?

Oh, I didn´t realize, that the nightlys only provide the modules and not tools like raw2dng or mlv_dump. Sorry for this misinformation. You must compile this by yourself or wait, till mlv_rec, g3gg0 provides on his site, is updated.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

slurp stuff...

Will read a bit to understand, what´s going on

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

slurp stuff...

I read the code and looked at the links, you gave me. I have to admit, that this all is beyond my understanding at the moment (maybe in a year or two it´s better? :) ). So I cannot help.

@ 1%
Maybe you can create a little pull request with this stuff to ML, if you think, it is ready for use?

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

raw2dng

I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

The cold pixels in the first frame are detected and fixed, no problem. But the second frame gives this error. Code from raw2dng.c:

int framenumber;
    for (framenumber = 0; framenumber < lv_rec_footer.frameCount; framenumber++)
    {
        printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);
       
        fflush(stdout);
       
        printf("\rdebug_frame : %d                             \n", (framenumber));
        int r = fread(raw, 1, lv_rec_footer.frameSize, fi);
        CHECK(r == lv_rec_footer.frameSize, "fread");
        raw_info.buffer = raw;
       
        /* uncomment if the raw file is recovered from a DNG with dd */
        //~ reverse_bytes_order(raw, lv_rec_footer.frameSize);
       
        char fn[100];
        snprintf(fn, sizeof(fn), "%s%06d.dng", prefix, framenumber);

        fix_vertical_stripes();
         
        find_and_fix_cold_pixels(fix_cold_pixels, framenumber);

        #ifdef CHROMA_SMOOTH
        chroma_smooth();
        #endif

        dng_set_framerate(lv_rec_footer.sourceFpsx1000);
        save_dng(fn, &raw_info);
    }

   
Working on the second frame, "printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);" is correctly displayed.
Than comes this fflush, which  don´t understand.
And my debug-print, which follows to the fflush, isn´t displayed anymore: "frintf("\rdebug_frame : %d                             \n", (framenumber));"

No clue, what´s going on here.

To avoid this error, I can set the max_cold_pixels to maybe 4000. Than all is fine and the 3500 cold pixels, I forced in an record, are correctly removed. But that is no solution. I want to know, why this error comes in, if there are to much cold pixels found.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

sarotaz

escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

davegravy

escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:

Cold pixels : 0

I looked quickly through the source code and thought I saw something about not targeting pixels that are in over-exposed regions. The dark pixels I have are ONLY in over-exposed regions
If you have time maybe you can investigate? If not I will try, but I am just starting to learn C so it may be a long while!

EDIT: Also, here's is a more thorough description of my problem:

https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png

http://www.magiclantern.fm/forum/index.php?topic=7473.msg103033#msg103033


escho

Quote from: sarotaz on February 24, 2014, 09:25:50 PM
escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

mlv_dump calls the cold-pixel-fix-function from raw2dng. the commandline-option, I integrated in mlv_dump,  is --fixcp. You may have to compile mlv_dump by yourself to get this option.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

Quote from: escho on February 17, 2014, 01:14:11 AM
raw2dng
I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

fixed

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

Quote from: davegravy on February 25, 2014, 04:53:23 AM
escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:


Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

davegravy

Quote from: escho on February 25, 2014, 08:48:59 PM
Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar

Hi Edgar,

Here's a short RAW:

https://dl.dropboxusercontent.com/u/30586759/M25-2141.RAW

In this example there is only 1 pixel that is bad and it happens on the 12th frame (I have some footage that has many more of the artifacts if you need it for testing but the RAW files are huge). Look for the bad pixel towards the bottom left of the flame. It is there if you convert to DNG with raw2dng cs2x2 but not if you use rawanizer.

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

escho

Quote from: davegravy on February 26, 2014, 04:04:18 AM

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

I give it a short try. Yes, the badpix comes in with croma smoothing (cs3x3 and cs5x5 modes too). the cold pixel fix can do nothing here, because cold pixels (and hot pixels, I,m experimenting with at the moment) are fixed before chroma smoothing.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed