PinkDotRemover tool 650D

[maxotics]

I have finally had success mapping out the static focus pixels, sending a bad pixel list to dcraw, and getting good TIFFs out of it.  In the next day or so I hope to have it integrated into my Windows scripts for processing RAW into either focus-dot free TIFFs or a video clip with 422 color.

@Rewind, this stuff is hard!!!   I couldn't get Windows to read the pixel values of a bayer pattern output by dcraw.  I had to use the libTiff.net library.  PDR has never completely eliminated all the dots for me, often doesn't work ("file is corrupted") and is difficult to understand and configure (dot start/end/offsets).  I tried Raw Therapee, but it crashes when you try to use too many DNGs.  I think Mixer2 may be back at some point.  Hopefully what I'm learning will help PDR too.

[Rewind]

@Rewind, this stuff is hard!!!

To map pixels is not too hard really, but what is hard is to achieve good results with interpolation algorithm.
Try to experiment with images like this (with colorful contrasty edges):


And pay an attention to the edges behavior. Don't know much about dcraw and it's hot pixel removal methods, but neither of raw converters out there do this job properly.

This picture below shows how PDR is better than ACR at this point:

so it would be interesting if you make similar test with dcraw

[maxotics]

To map pixels is not too hard really,

If that were true, wouldn't PDR work perfectly  

The problems you're showing, I now believe, are not from weak de-bayering algorithms, but focus-pixels that were not removed properly. 

Here is a picture of the focus pixels in 1280x720.  In addition to the bright red pixels, there are pixels that are blue, green and some pixels aren't bright, but are actually turned off. 



And here in BW



My current algorithm to find focus pixels locates about 4,112 pixels.  I believe there are more (because I still see some).  Eventually, I think I'll get them all.

dcraw has a few different de-bayering algos. 

What you're also noticing, I think, in your images, is that some focus pixels don't fire unless they're on a hard edge. 

What I don't know is if this is really true, that they are activated on hard edges, or focus pixels always change for focus, but the changes is not noticed when nearby pixels are also properly exposed, so to speak--that is, they blend in.

If they don't fire when not focused, then we probably want a way to leave them alone.

If they always fire, but is not always noticed, then we should just remove them all.

I suspect that removing them all will not noticeably degrade the image and that PDR, in it's current state, make that process more complicated than it needs to be.  Of course PDR is in a great position to do all this stuff.  Java just isn't my strong suit.

[Rewind]

If that were true, wouldn't PDR work perfectly

PDR actually do works perfectly for 650D (at least in terms of removing all the dots).

The problems you're showing, I now believe, are not from weak de-bayering algorithms, but focus-pixels that were not removed properly.

No. I told you several times, the positions are not a problem. We know exact positions of all these pixels with absolute accuracy. Period. Edges are the problem. You can use any simplified method to interpolate in uniform areas, but you forced to use some form of adaptive interpolation if you're concerned in quality on edges.

PDR, in it's current state, make that process more complicated than it needs to be.

No. The math behind this method seems too complicated only for those who didn't study the actual paper describing it, but it works better than any raw processor built-in filtration I've seen so far.

[maxotics]

PDR actually do works perfectly for 650D (at least in terms of removing all the dots).

It doesn't work for my EOS-M.  Others have complained. 

I've posted this video showing the dots, I keep point them out you keep saying it's not a problem.  I used your PDR on these clips. 

No. I told you several times, the positions are not a problem. We know exact positions of all these pixels with absolute accuracy. Period. Edges are the problem. You can use any simplified method to interpolate in uniform areas, but you forced to use some form of adaptive interpolation if you're concerned in quality on edges.

All I know is it isn't working for me.  It isn't for others.  Knowing it works perfectly for your camera doesn't help me.  Sorry!

No. The math behind this method seems too complicated only for those who didn't study the actual paper describing it, but it works better than any raw processor built-in filtration I've seen so far.

The math of static pixels is easy.  The math describing pixels as start/end/offsets, from center image is MORE COMPLEX, period.  I don't care which method is used.  You seem to think I do.  All I know is that PDR is not working for me (and others).  You're implying that I'm not smart enough to understand the calculations, ignoring the fact that it is not working for me on your calculations (not that I blame you, you don't have the camera, and I thank you for your time/effort).   I talk static positioning because if I'm going to solve a problem I believe in lowering complexity, if I can.

I agree, it doesn't matter what calculations you use when it comes to RAW processor built-in filtration.  I've used both Raw Therapee and ufRAW and know as well as you do!

I don't think you're taking my research and thoughts on this matter seriously.  You're a bit defensive about PDR (which again, is not working for everyone).  I'd love it if you either made PDR work for everyone or worked with me a bit on the approach I'm taking.  Or, I'm happy to work with you on PDR. 

Soon I will have an alternative to PDR.  I would think that a good thing.  I wish you thought so too.

What we all need is a way for some software to scan the DNGs, find the pattern, and write that out, either as PDR formulas or static locations.  That what we should probably talk about?  Right?

[animal413]

 

[gary2013]

anything new happening with PDR? I am very frustrated with it not working most of the time. I still get corrupted file error messages at the end, but not all the time. I cannot tell when or why this happens. I have the crop and non crop PDR and sometimes they work but not a lot of times. Again, I don't see why or when it does this. I have tried many different ways shooting raw and obviously none of it can be used with pink dots still there. I am using Sept 30 version of ML

Gary
all my older raw files also do not work with pdr. so something changed in the newer ml ver.

[maxotics]

Gary I work on it every night.  It is VERY frustrating to me too.  In a sense I'm glad you're having problems too because I wonder if I'm just crazy.  I don't know why others don't have these problems.  I suspect that people don't shoot enough in real-world situations. 

This is what I believe so far.  There are more than 4,000 pixels used for focus.  They are clumped together in groups.  There are few green sensels used, mostly red and blue.  Sometimes the pixels fire when the scene is dark, sometimes when over-exposed.  PDR gets rid of the basic focus pixels, but does not remove the pixels that fire when over-exposed (at least for me).  Removing all these pixels will probably degrade the image more than I'd like.

Fortunately, these pixels are in definitive patterns.  Each pixel in a horizontal line is separated by 24 pixels.  I'm working on an Algo that figures out when the pixels fire, for each individual frame, and then will interpolate like PDR (hopefully port over their routine, which, as Rewind points out, works very well).

The ultimate solution should be to incorporate these alogrithms in raw2dng.  If I end up finishing it I'll make it something like hafraw2dng (Hybrid Autofocus). 

In short, PDR works great for an edge to edge properly exposed frame.  Other shots fall apart.

For a whole host of programming headaches, this is not going quickly.  I am working on it.  And on the moire issue too, which has some similarities I'm seeing. 

Every other day I think of giving up on ML with this camera.  Glad to know I'm not alone

[gary2013]

max, don't give up, please. I modified my post while you replied. All my old raw files will not pdr at all. you know a hell of a lot more than i do on this. is there some way where a pdr app can just scan the first image and find all the crap and then remove it all for every frame in the seq? something keeps changing cus pdr works then later it won't work at all even on the older raw files that used to work.

Gary
if we can't get some sort of pdr to work, then we don't have raw video and then the camera is not much better than any other consumer cam. We really need pdr, raw video and audio to work to make this little david beat goliath just like Trammel did in the beginning with his 5dm2 and the giants.

[maxotics]

If you've shot them all at the same resolution, send me a DNG from a frame with the most pink dots and I can look into it.  It has to be a DNG because I need to see the original sensor readings before de-bayering.  You can post link here or email me at max@maxotics.com  Sorry I don't have a fix right this second.  I'm sitting on a lot of useless footage too!

[a1ex]

If you have a difficult shot where the interpolation struggles, and a list with bad pixels (just a list of x,y points), send it to me; I'd like to try the median filter that I use in cr2hdr to fix the hot pixels.

[Rewind]

Guys with EOS-M, just one note. If you will send examples to A1ex, please make something with pronounced vertical and horisontal edges and with high contrast detail. Because these are the things in which different interpolation algorithms.. well... differ.
I mean, if you shoot your cat, then median filter is obviously ok, but if you try something like this, it's more interesting (look closely to the edges)

[a1ex]

Yep, this shot in DNG with a list of pixels would be perfect.

[maxotics]

Hi A1ex, I've uploaded a bunch of files for you at maxotics.com/photos  Enter "focusdotshotpixels", then you'll see a link for download

Rewind, if you're interested, you can see the focus pixels in the blown out areas AFTER PDRing in "1280x434_PDRd_HotPixels_BlownOutAreas"  I wish you'd go shoot more so you can confirm that this won't be a problem for you (others) down the road.  The edge pixels you're seeing, is part of the problem PDR struggles against.  There are too many focus pixels for it to deal with without removing more than one would want. 

A1ex, you can see hot pixels in the "HotPixelsInRoad" example.  (There is a lot of moire with this camera in non-crop mode.)

In the zip, you'll find frames that I shoot in a light table to get focus dots.  I then use Photivo (as Rewind helped me with) to isolate the dots through elimination of green channel and other contrast/image type adjustments.  I want light pixels on black.  I wrote a .NET program that will scan the image and create a dcRAW format hot pixel list.  I did one for both red and blue and red only.

In the file "1280x434_PDRd_HotPixels_BlownOutAreas_SeeDeadPixels.jpg" you will see that these hot pixels are actually black pixels from the sensor.  My hot pixel locator would have to be adjusted to find those.  I can do that later.  (This while I'm porting this over to C# and trying to integrate/use g3gg0's code, and C stuff from the libraries---my head is spinning!).

I've been thinking about the median filters a bit too.  I think there is an even better application for what you're looking into--reducing moire.

Look at "1280x434_MoirePattern.dng"  Seems like a lot of interference patterns that could be predicted/isolated?

It seems to me, much of the moire is because the chroma information for what should be one scan line is split up into 2, so to speak.  I understand moire isn't talked about that way.  But I believe a median filter on, say a group of 3-7 pixels, with a lower/higher line, slightly offset to right or left, will eliminate the moire, though will blur the image somewhat (which is what a VAF filter would do anyway).   Such moire reduction would need a rules-based algo matched to some sort of median filter.  Anyway, I may be crazy, but I believe one/we could do through programming what the VAF fitler does physically (better join skipped lines from the sensor).

I understand what you may really want is a RAW frame with some hot pixels mapped to dcRAW.  I could shoot something with my 50D (which doesn't have the Hybrid AF dot issue).  But I hope you'll segue into our EOS-M nightmare

As you can see, I have some equipment set up for testing.  If you need any types of images (you too Rewind), let me know!

[Rewind]

Rewind, if you're interested, you can see the focus pixels in the blown out areas AFTER PDRing

I'm not interested since these areas are blown out. There is no problem at all to delete them.

[maxotics]

Rewind, easy for who?  I'm trying to get the camera to work for film-makers, not academics.  I don't expect them to go through their videos, frame by frame, and eliminate hot pixels.  I respect if you're not interested, but I think it sad you won't join us in this quest.  Why are you fighting me on this?

I mean, I think you have done great work with this and have said it a million times.  But anything I say seems to bother you.  I'm sorry. 

[Rewind]

Rewind, easy for who?  I'm trying to get the camera to work for film-makers, not academics.  I don't expect them to go through their videos, frame by frame, and eliminate hot pixels.  I respect if you're not interested, but I think it sad you won't join us in this quest.  Why are you fighting me on this?

Oh, man. Excuse me. I mean, raw processors doing that nicely for us.
All the cameras with hybrid focusing system have this issue of overexposed areas, but ACR for example eliminates them easy.

Since i'm using ACR in my workflow, i never see these pixels in overexposed areas. But I understand, not everyone wants adobe, and some of the raw processors has no dead pixel filters may be

[Rewind]

By the way, just tried with Photivo and RawTherapee. Just turn Hot/Dead pixel filter on, and overexposed areas are pure as driven snow ) This is what i meant by 'easy'

There is no any point to apply some fancy interpolation algorithm to overexposed areas, since they are... well, overexposed = plain white. This is what hot\dead pixel filters are done for.

For the 'real' focusing pixels PDR does the job. So I just can't see the problem anymore.

[maxotics]

The dots in the window ARE focus pixels.  They are not hot/dead pixels.  PDR is not removing them because they are dormant when the sensor pixel is properly exposed.  Our definitions of what works is different.  PDR, to me, half works.

I have Adobe.  I can go out and buy a 5D3 and be done with all this sh_t   But I love the small size of the EOS-M and think this can be a killer camera. 

PDR is better than nothing, but it is not a complete solution.  As Gary and others have pointed out, it needs to do more than apply a calculated map of pixels that need interpolating.  It needs some smart analysis up front.  It either needs to figure out those maps and apply them, or do what I think is a better, harder, but more fool-proof solution--find all patterns and eliminate them.

You're one of the few people around who can make this a reality.  But you keep insisting PDR is doing all it can.  Why would you WANT another program to remove hot pixels in blown out areas?  They're focus pixels.  They should be ALL taken care of.

Again, we have a disconnect   Are they, or are they not, focus pixels?

[Rewind]

The pixels in overbrighten areas ARE NOT usual focusing pixels. You should NOT interpolate them all over the image because you will loose detail! They appear ONLY in overexposed areas. This is why you'd better just filter them hell off — this is darn easy.

PDR is not ideal — that's fact! But in terms of interpolation algorithm only. PDR removes ALL of the focusing pixels. But neither mine, nor Mixer2's algorithm is not ideal. This is what we should work on.

[maxotics]

Rewind, Almost everyone I know who argues like you do does well in law school!   You'll at least a double threat.  Programmer and Trial Lawyer!

Okay, they're not the "usual" focus pixels.  We won't put them in focus-pixel prison   I'm going to continue on with my strategy of building a C#/C workbench to deal with these issues. 

If you can improve PDR, I'm sure that would be much appreciated.  But I think the future is incorporating what you know in existing C routines in the general ML sphere.

[Rewind]

But I think the future is incorporating what you know in existing C routines in the general ML sphere.

Sounds reasonable.

[gary2013]

stay calm guys. we all want the same thing. I am just confused as to why all my older test raw files that used to PDR ok are not doing anything now. By that I mean I use the original untouched raw file again from the start with both of the pdr apps I have for crop and non crop. i think it was version 8.

I will try and make some new short raw test files and clean dng files to post for anyone t see and try out. also a PDR dng showing all the pink dots.

Gary

[maxotics]

Thanks Gary. We're arguing, not fighting.  That would be good if you posted some DNGs gary.   We need to get to the bottom of differences between ML builds, if there are any.  I'm trying to organize my collection of stuff so we can better work through these issues in the future.  Rewind, I don't have your camera, so any files you want to share would be great.

[a1ex]

@maxotics: your pink dot list (badpixels_red_and_blue.txt) is incomplete. Here's why.