[SOLVED!] Dead pixels - Hot pixels - Banding in RAW recording

[a1ex]

Can you upload some samples? (the first DNG from the sequence is enough, along with the conversion log).

[fascina]

OK, shot with 5D Mark III, converted with latest RAW2DNG (30 May).

First, for clarification: The full frame RAW of my 5D Mark III usually shows 8-pixel-banding (unless corrected by new RAW2DNG), but the 5x magnification RAW of my 5D Mark III never shoes 8-pixel-banding.


Sample 1: When converting this full frame RAW shot, the RAW2DNG conversion log does not mention any stripe correction, therefore the stripes remain to be seen:

http://www.webpage-design.org/sample000001.dng

Maybe the algorythm became confused by the dark area in the lower part of the shot.


Sample 2: When converting this 5x magnification RAW shot, the RAW2DNG conversion log does perform stripe correction (1   1   1   1    1.006    1.008    1.0010    1.0012), although 5x magnification RAW never has banding. Thus it creates banding although this banding would not be there with old RAW2DNG conversion:

http://www.webpage-design.org/sample000002.dng


Unfortunately the RAW files belonging to this DNGs are way too big to upload.

[a1ex]

Good catch, gradient in source image was the problem here.

Got a few solutions that seem a bit better at first sight:

1) raw2dng_noweight.exe - it addresses the gradient issue and uses the old weighting method (constant weights for all pixels that are -9 EV under white level or brighter, the others are discarded). It is able to undo most of the banding from sample 2 (not quite exact, 1.000  1.000  0.999  1.000  0.991  0.992  0.987  0.987). According to your log, they should be 1 1 1 1 0.994 0.992 0.990 0.988.

2) raw2dng_weight.exe This addresses the gradient issue and uses variable weight for pixels brighter than -12 EV under white level; the weight is equal to EV value relative to black level, so a bright pixel can be ~10x more important than a dark one. Undo coefficients: 1.000  1.000  1.000  1.001  0.994  0.994  0.991  0.990.

3) raw2dng_weight_hilight.exe Same as 2, but ignores very bright areas when estimating the coefficients. It only makes a difference when the image contains overexposed areas.

Can you run some tests and see which one seems closest to reality?

Note: these are debug versions, and they will always do some correction, no matter what.

[fascina]

Great Job, A1ex, banding in my two sample RAW footage is gone with the new RAW2DNGs.  :-)

I also tested them with some other footage and they seem to work without any banding problems (only time will tell if there are some subjects left which the algorythm does not recognize).

Visually, all 3 solutions are the same as perfect. My eyes cannot see any visible differences in those 3 solutions, they are all good. 

( By the way, when I drag and drop several RAW files to RAW2DNG (in Windows), it only convertes one of them. )

[mannfilm]

5Dm3 banding issues. An attempt for clarity for dumb guys like me.
Banding was caused by some of the earlier versions of raw2dng. (Not the camera itself, yeah!)
Raw2DNG v11 fixes the problem.
Where can we find v11 for the mac?
..........
You guys are incredible, you are re-defining the entire film landscape keep up the good work, and many thnaks from all of us.

[a1ex]

It is caused by the camera itself; raw2dng tries to correct it.

[budafilms]

Hi Guys! It´s ready for mac??? thanks!

[aaphotog]

Recorded a very short video, I notice that it's underexposed a lot(on purpose) but I didn't bring the shadwos up any.
The lines ALSO show in the bright areas of the video as well.
I used the latest raw2dng(for mac)
I also used the ML from June 1st(I believe)
Any idea as to why these black bars are across the entire image???

https://www.dropbox.com/s/o9zasksokmkgubj/M01-2200.mov

[aaphotog]

Seems as though dropbox compressed the video
here's a still taken from the video
If anyone knows of a website where I can upload the video(for free) with the best quality, let me know

[frenchps49]

"Raw2DNG v11 fixes the problem.
Where can we find v11 for the mac?"

That's right... any clue ?

[squig]

There is no 0.11 for mac yet.

[mannfilm]

Sorry to be a bother, but was wondering if there will be a 0.11 for the Mac. In the meantime, is there a no-banding version available for the PC?

[aaphotog]

Sorry to be a bother, but was wondering if there will be a 0.11 for the Mac. In the meantime, is there a no-banding version available for the PC?

It was updated for Mac yesterday with the banding fix. Check the raw2dng thread

[gerk.raisen]

Hello A1ex,

at the end what of the three possible solutions (weight, noweight, noweight_hilight ) had you choosen?
It's already implemented in the last commit?

[a1ex]

Right now it's the last one, but Thomas Worth found a test case where there's room for improvement.

Edit: this seems to be a lot better: https://bitbucket.org/hudson/magic-lantern/commits/d4b1e587cfab

(at least it covers all the DNGs that I've tried it on, with extremely strong curves)

[marekk]

Based on original chdk's dng.c file I figured out how to fix bad/hot pixels in DNG file.

Pls look at original chdk dng.c
http://subversion.assembla.com/svn/chdk/trunk/modules/dng.c
They implemented bad pixel removal using a special map of bad pixels created on cameras manually (they shot two times and choose a file with higher sum of bad pixels and create file called badpixel.bin). They set DNG version to 1.1 because 1.3 or 1.4 isn't working with this method. However there is an option to use special badpixel opcode in idf1 structure, automatically using camera_sensor.cfa_pattern. If DNG version is set to 1.3 ACR automatically fixes bad and hot pixels. I tested it and it works. Pls try it:

Modified chdk-dng.c
https://www.dropbox.com/s/pyby320bap6rhwg/chdk-dng.c

DNG with bad/hot pixels:
https://www.dropbox.com/s/9dh1tp4d3vpxo43/000000_bad_pixels.dng

DNG processed with modified chdk-dng.c (without raw2dng fix_bad_pixels)
https://www.dropbox.com/s/000pzcw9l05pw4o/000000.dng

DNG 1.3 specification (http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf)
"There are two ways to deal with defective pixels in DNG. The first is to map out (interpolate over) the defective pixels before storing the raw data in DNG. The second is to include a bad pixel fixing opcode in the OpcodeList1 tag."

[a1ex]

I'm trying to find a better solution for this issue:

https://bitbucket.org/hudson/magic-lantern/pull-requests/732/5d3-attempt-to-fix-vertical-stripes-so-it/diff

[bouncyball]

Does it mean we'll be able to select PREFERRED_RAW_TYPE in the menu? Cool.

[a1ex]

The menu option is only meant for debugging; I don't want to expose it in the public builds.