Author Topic: [600D RAW] Dots, dots and more dots.  (Read 69616 times)

hjfilmspeed

  • Senior
  • ****
  • Posts: 479
  • 5D III and IV
Re: [600D RAW] Dots, dots and more dots.
« Reply #100 on: February 02, 2014, 02:58:31 PM »
@escho i had a feeling they were hot pix and that its different then cold. Thank you for taking the time to download and look. I think this is something worth investing time into since it will save you tons of post work

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #101 on: February 07, 2014, 03:28:30 PM »
fix cold pixels and mlv_dump

I  played  a bit with mlv_dump-code in order to bring the cold pixel fix into mlv_dump. It seem not to be as complicated as I thougt before. I think, I´ll give it a try, if g3gg0 don´t want to port it himself.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

xaled

  • New to the forum
  • *
  • Posts: 42
Re: [600D RAW] Dots, dots and more dots.
« Reply #102 on: February 07, 2014, 04:02:25 PM »
now that - cold pixels and mlv_dump would be just great!

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #103 on: February 07, 2014, 09:40:43 PM »
I think, I´ve got it:

Code: [Select]
edgar@linux-uoww:~/mlv_test/mit/1> ./mlv_dump --dng --fixcp *.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M07-2217.MLV'
   - Convert to DNG frames
   - Output into 'M07-2217_frame_'
File M07-2217.MLV opened
File M07-2217.M00 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  1.002  1.001  1.001  1.001  1.002  1.003
Cold pixels : 456                             
Reached end of chunk 1/1 after 291 blocks
Processed 124 video frames
Done

Must do some tests

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #104 on: February 07, 2014, 10:24:09 PM »
What do you think about this?

link removed (experimental code)

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #105 on: February 15, 2014, 04:31:48 PM »
I ported the cold-pixel-fix from raw2dng to mlv_dump. It is in ML-source-code now. If you are interested in, give it a try. The commandline option for mlv_dump is:

Code: [Select]
--fixcp

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: [600D RAW] Dots, dots and more dots.
« Reply #106 on: February 15, 2014, 05:51:02 PM »
Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #107 on: February 15, 2014, 10:55:58 PM »
Give the slurp stuff a go, see how well it works. Might be worth putting it in the ML tree.

Do you mean this one: https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/8c602fcd3dd8ed6a1fb77417dd2ef4a41a4892f7 ?

For giving it a try, I need a short explanation in easy words, what this stuff :) is meant for and how I can test it with my camera. Is this commit the only change, which has to be done? Or are there other things to take into account?

Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [600D RAW] Dots, dots and more dots.
« Reply #108 on: February 15, 2014, 11:09:29 PM »
Sorry for asking such may be easy things, but I would like to understand, what I do, before I really do it.

These things are far from easy. You can follow this thread for details: www.magiclantern.fm/forum/index.php?topic=10443
but EDMAC is not yet fully understood, so we don't know much more either (at this stage is mostly trial and error). But we are making small steps in understanding it (and this is one of them).

A general description of EDMAC is here: http://magiclantern.wikia.com/wiki/Register_Map#EDMAC

duey101

  • Freshman
  • **
  • Posts: 79
  • Surf first, then everything else.
Re: [600D RAW] Dots, dots and more dots.
« Reply #109 on: February 16, 2014, 04:09:34 AM »
Gear: Canon 5D Mark III, 60D, EF 70-200L, EF 85mm, EF 50mm, EF 10-22mm.  Platform: Dual Boot Win 7 Ultimate x64/Mavericks Hackintosh, PPro CC, LR5.

duey101

  • Freshman
  • **
  • Posts: 79
  • Surf first, then everything else.
Re: [600D RAW] Dots, dots and more dots.
« Reply #110 on: February 16, 2014, 06:32:23 AM »
I ported the cold-pixel-fix from raw2dng to mlv_dump. Should be in the next nightly. If you are interested in, give it a try. The commandline option for mlv_dump is:

Code: [Select]
--fixcp

Edgar

Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place? 
Gear: Canon 5D Mark III, 60D, EF 70-200L, EF 85mm, EF 50mm, EF 10-22mm.  Platform: Dual Boot Win 7 Ultimate x64/Mavericks Hackintosh, PPro CC, LR5.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: [600D RAW] Dots, dots and more dots.
« Reply #111 on: February 16, 2014, 07:55:21 AM »
This is the main commit, the other one picks a better edmac channel:
Try a few more :)

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/3c3f7f903e2540135d58c119432647ed5d392128?at=unified

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #112 on: February 16, 2014, 10:33:16 AM »
It's a dead link

Yes, I removed the link. This was experimental sourcecode from my repo

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #113 on: February 16, 2014, 10:38:16 AM »
Do I need to download a newer version of mlv_dump.exe to use this?  Or is it in camera where this fix takes place?

Oh, I didn´t realize, that the nightlys only provide the modules and not tools like raw2dng or mlv_dump. Sorry for this misinformation. You must compile this by yourself or wait, till mlv_rec, g3gg0 provides on his site, is updated.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #114 on: February 16, 2014, 10:40:39 AM »
slurp stuff...

Will read a bit to understand, what´s going on

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #115 on: February 16, 2014, 11:52:50 AM »
slurp stuff...

I read the code and looked at the links, you gave me. I have to admit, that this all is beyond my understanding at the moment (maybe in a year or two it´s better? :) ). So I cannot help.

@ 1%
Maybe you can create a little pull request with this stuff to ML, if you think, it is ready for use?

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #116 on: February 17, 2014, 01:14:11 AM »
raw2dng

I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

The cold pixels in the first frame are detected and fixed, no problem. But the second frame gives this error. Code from raw2dng.c:
Code: [Select]
int framenumber;
    for (framenumber = 0; framenumber < lv_rec_footer.frameCount; framenumber++)
    {
        printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);
       
        fflush(stdout);
       
        printf("\rdebug_frame : %d                             \n", (framenumber));
        int r = fread(raw, 1, lv_rec_footer.frameSize, fi);
        CHECK(r == lv_rec_footer.frameSize, "fread");
        raw_info.buffer = raw;
       
        /* uncomment if the raw file is recovered from a DNG with dd */
        //~ reverse_bytes_order(raw, lv_rec_footer.frameSize);
       
        char fn[100];
        snprintf(fn, sizeof(fn), "%s%06d.dng", prefix, framenumber);

        fix_vertical_stripes();
         
        find_and_fix_cold_pixels(fix_cold_pixels, framenumber);

        #ifdef CHROMA_SMOOTH
        chroma_smooth();
        #endif

        dng_set_framerate(lv_rec_footer.sourceFpsx1000);
        save_dng(fn, &raw_info);
    }
   
Working on the second frame, "printf("\rProcessing frame %d of %d...", framenumber+1, lv_rec_footer.frameCount);" is correctly displayed.
Than comes this fflush, which  don´t understand.
And my debug-print, which follows to the fflush, isn´t displayed anymore: "frintf("\rdebug_frame : %d                             \n", (framenumber));"

No clue, what´s going on here.

To avoid this error, I can set the max_cold_pixels to maybe 4000. Than all is fine and the 3500 cold pixels, I forced in an record, are correctly removed. But that is no solution. I want to know, why this error comes in, if there are to much cold pixels found.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

sarotaz

  • Freshman
  • **
  • Posts: 82
Re: [600D RAW] Dots, dots and more dots.
« Reply #117 on: February 24, 2014, 09:25:50 PM »
escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

davegravy

  • New to the forum
  • *
  • Posts: 20
Re: [600D RAW] Dots, dots and more dots.
« Reply #118 on: February 25, 2014, 04:53:23 AM »
escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:

Code: [Select]
Cold pixels : 0
I looked quickly through the source code and thought I saw something about not targeting pixels that are in over-exposed regions. The dark pixels I have are ONLY in over-exposed regions
If you have time maybe you can investigate? If not I will try, but I am just starting to learn C so it may be a long while!

EDIT: Also, here's is a more thorough description of my problem:

https://www.dropbox.com/s/ijgu5z62cia2zzi/Screenshot%202014-02-21%2022.47.33.png

http://www.magiclantern.fm/forum/index.php?topic=7473.msg103033#msg103033


escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #119 on: February 25, 2014, 08:22:10 PM »
escho your code resolves definitely the 60D bad pixel problem with raw shooting. (without ACR workflow - i use davinci Resolve).
But I would use new MLV rec module and chroma smoothing function included in mlv_dump doesn't fix the pixel.
If you (or someone) could include your code in mlv_dump, make converting process more easy. (without generating raw and reconvert in dng)

mlv_dump calls the cold-pixel-fix-function from raw2dng. the commandline-option, I integrated in mlv_dump,  is --fixcp. You may have to compile mlv_dump by yourself to get this option.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #120 on: February 25, 2014, 08:36:14 PM »
raw2dng
I run into an error (Speicherzugriffsfehler), if there are more cold pixels found, than definied in max_cold_pixels.

fixed

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #121 on: February 25, 2014, 08:48:59 PM »
escho,

I'm getting dark dots from my 650D (not 600D). I compiled and ran the latest raw2dng (w/ cs2x2) but the dark dots do not get treated and gives this output:


Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed

davegravy

  • New to the forum
  • *
  • Posts: 20
Re: [600D RAW] Dots, dots and more dots.
« Reply #122 on: February 26, 2014, 04:04:18 AM »
Please, can you upload somewhere a very short RAW or MLV file with your problem?

The cold pixel fix fixes only true cold pixels (sensor defects). It make no differece, whether the cold pixels are located in dark or bright areas, they are fixed with the median of their neighbours

Edgar

Hi Edgar,

Here's a short RAW:

https://dl.dropboxusercontent.com/u/30586759/M25-2141.RAW

In this example there is only 1 pixel that is bad and it happens on the 12th frame (I have some footage that has many more of the artifacts if you need it for testing but the RAW files are huge). Look for the bad pixel towards the bottom left of the flame. It is there if you convert to DNG with raw2dng cs2x2 but not if you use rawanizer.

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

escho

  • Contributor
  • Hero Member
  • *****
  • Posts: 563
Re: [600D RAW] Dots, dots and more dots.
« Reply #123 on: February 27, 2014, 10:56:17 PM »

EDIT: I just tried with plain raw2dng (no cs2x2) and it's NOT there, so it seems to be related to cs2x2!

I give it a short try. Yes, the badpix comes in with croma smoothing (cs3x3 and cs5x5 modes too). the cold pixel fix can do nothing here, because cold pixels (and hot pixels, I,m experimenting with at the moment) are fixed before chroma smoothing.

Edgar
https://sternenkarten.com/
600D, 6D, openSUSE Tumbleweed