Author Topic: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording  (Read 104476 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #150 on: June 02, 2013, 08:56:49 PM »
Can you upload some samples? (the first DNG from the sequence is enough, along with the conversion log).

fascina

  • New to the forum
  • *
  • Posts: 27
  • 5D Mark III
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #151 on: June 02, 2013, 09:26:32 PM »
OK, shot with 5D Mark III, converted with latest RAW2DNG (30 May).

First, for clarification: The full frame RAW of my 5D Mark III usually shows 8-pixel-banding (unless corrected by new RAW2DNG), but the 5x magnification RAW of my 5D Mark III never shoes 8-pixel-banding.


Sample 1: When converting this full frame RAW shot, the RAW2DNG conversion log does not mention any stripe correction, therefore the stripes remain to be seen:

http://www.webpage-design.org/sample000001.dng

Maybe the algorythm became confused by the dark area in the lower part of the shot.


Sample 2: When converting this 5x magnification RAW shot, the RAW2DNG conversion log does perform stripe correction (1   1   1   1    1.006    1.008    1.0010    1.0012), although 5x magnification RAW never has banding. Thus it creates banding although this banding would not be there with old RAW2DNG conversion:

http://www.webpage-design.org/sample000002.dng


Unfortunately the RAW files belonging to this DNGs are way too big to upload.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #152 on: June 03, 2013, 11:22:01 AM »
Good catch, gradient in source image was the problem here.

Got a few solutions that seem a bit better at first sight:

1) raw2dng_noweight.exe - it addresses the gradient issue and uses the old weighting method (constant weights for all pixels that are -9 EV under white level or brighter, the others are discarded). It is able to undo most of the banding from sample 2 (not quite exact, 1.000  1.000  0.999  1.000  0.991  0.992  0.987  0.987). According to your log, they should be 1 1 1 1 0.994 0.992 0.990 0.988.

2) raw2dng_weight.exe This addresses the gradient issue and uses variable weight for pixels brighter than -12 EV under white level; the weight is equal to EV value relative to black level, so a bright pixel can be ~10x more important than a dark one. Undo coefficients: 1.000  1.000  1.000  1.001  0.994  0.994  0.991  0.990.

3) raw2dng_weight_hilight.exe Same as 2, but ignores very bright areas when estimating the coefficients. It only makes a difference when the image contains overexposed areas.

Can you run some tests and see which one seems closest to reality?

Note: these are debug versions, and they will always do some correction, no matter what.

fascina

  • New to the forum
  • *
  • Posts: 27
  • 5D Mark III
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #153 on: June 03, 2013, 09:46:18 PM »
Great Job, A1ex, banding in my two sample RAW footage is gone with the new RAW2DNGs.  :-)

I also tested them with some other footage and they seem to work without any banding problems (only time will tell if there are some subjects left which the algorythm does not recognize).

Visually, all 3 solutions are the same as perfect. My eyes cannot see any visible differences in those 3 solutions, they are all good.  :)

( By the way, when I drag and drop several RAW files to RAW2DNG (in Windows), it only convertes one of them. )

mannfilm

  • Freshman
  • **
  • Posts: 88
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #154 on: June 04, 2013, 12:32:26 AM »
5Dm3 banding issues. An attempt for clarity for dumb guys like me.
Banding was caused by some of the earlier versions of raw2dng. (Not the camera itself, yeah!)
Raw2DNG v11 fixes the problem.
Where can we find v11 for the mac?
..........
You guys are incredible, you are re-defining the entire film landscape keep up the good work, and many thnaks from all of us.


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #155 on: June 04, 2013, 12:41:17 AM »
It is caused by the camera itself; raw2dng tries to correct it.

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #156 on: June 04, 2013, 05:44:26 AM »
Hi Guys! It´s ready for mac??? thanks!

aaphotog

  • Senior
  • ****
  • Posts: 356
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #157 on: June 04, 2013, 07:01:13 AM »
Recorded a very short video, I notice that it's underexposed a lot(on purpose) but I didn't bring the shadwos up any.
The lines ALSO show in the bright areas of the video as well.
I used the latest raw2dng(for mac)
I also used the ML from June 1st(I believe)
Any idea as to why these black bars are across the entire image???

https://www.dropbox.com/s/o9zasksokmkgubj/M01-2200.mov

aaphotog

  • Senior
  • ****
  • Posts: 356
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #158 on: June 04, 2013, 07:04:49 AM »
Seems as though dropbox compressed the video
here's a still taken from the video
If anyone knows of a website where I can upload the video(for free) with the best quality, let me know


frenchps49

  • Freshman
  • **
  • Posts: 54
  • Canon 5D Mk III
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #159 on: June 04, 2013, 07:25:24 AM »
"Raw2DNG v11 fixes the problem.
Where can we find v11 for the mac?"

That's right... any clue ?

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #160 on: June 04, 2013, 07:27:36 AM »
There is no 0.11 for mac yet.

mannfilm

  • Freshman
  • **
  • Posts: 88
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #161 on: June 06, 2013, 04:26:52 PM »
Sorry to be a bother, but was wondering if there will be a 0.11 for the Mac. In the meantime, is there a no-banding version available for the PC?

aaphotog

  • Senior
  • ****
  • Posts: 356
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #162 on: June 06, 2013, 04:46:54 PM »
Sorry to be a bother, but was wondering if there will be a 0.11 for the Mac. In the meantime, is there a no-banding version available for the PC?
It was updated for Mac yesterday with the banding fix. Check the raw2dng thread

gerk.raisen

  • Member
  • ***
  • Posts: 161
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #163 on: June 07, 2013, 10:03:07 AM »
Hello A1ex,

at the end what of the three possible solutions (weight, noweight, noweight_hilight ) had you choosen?
It's already implemented in the last commit?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #164 on: June 07, 2013, 10:05:31 AM »
Right now it's the last one, but Thomas Worth found a test case where there's room for improvement.

Edit: this seems to be a lot better: https://bitbucket.org/hudson/magic-lantern/commits/d4b1e587cfab

(at least it covers all the DNGs that I've tried it on, with extremely strong curves)

marekk

  • Senior
  • ****
  • Posts: 331
  • www.oddajmisie.com
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #165 on: June 10, 2013, 06:54:50 PM »
Based on original chdk's dng.c file I figured out how to fix bad/hot pixels in DNG file.

Pls look at original chdk dng.c
http://subversion.assembla.com/svn/chdk/trunk/modules/dng.c
They implemented bad pixel removal using a special map of bad pixels created on cameras manually (they shot two times and choose a file with higher sum of bad pixels and create file called badpixel.bin). They set DNG version to 1.1 because 1.3 or 1.4 isn't working with this method. However there is an option to use special badpixel opcode in idf1 structure, automatically using camera_sensor.cfa_pattern. If DNG version is set to 1.3 ACR automatically fixes bad and hot pixels. I tested it and it works. Pls try it:

Modified chdk-dng.c
https://www.dropbox.com/s/pyby320bap6rhwg/chdk-dng.c

DNG with bad/hot pixels:
https://www.dropbox.com/s/9dh1tp4d3vpxo43/000000_bad_pixels.dng

DNG processed with modified chdk-dng.c (without raw2dng fix_bad_pixels)
https://www.dropbox.com/s/000pzcw9l05pw4o/000000.dng

DNG 1.3 specification (http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf)
"There are two ways to deal with defective pixels in DNG. The first is to map out (interpolate over) the defective pixels before storing the raw data in DNG. The second is to include a bad pixel fixing opcode in the OpcodeList1 tag."

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564

bouncyball

  • Contributor
  • Hero Member
  • *****
  • Posts: 849
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #167 on: May 13, 2016, 08:46:10 PM »
Does it mean we'll be able to select PREFERRED_RAW_TYPE in the menu? Cool.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: [SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording
« Reply #168 on: May 13, 2016, 08:48:22 PM »
The menu option is only meant for debugging; I don't want to expose it in the public builds.