EOS-M RAW dots come back in blown out areas

Started by maxotics, September 09, 2013, 04:11:32 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

maxotics

After a new build by 1% the Pink Dot Remover stopped working.   Rewind was kind enough to figure out the new dot-matrix locations and fix it.  In this video you can see how the dots come back in over-exposed areas.  I don't believe this is a fault in Rewind's fix, but a deeper problem with EOS-M video, which I discuss a bit here: http://www.magiclantern.fm/forum/index.php?topic=3648.msg73969#msg73969

Rewind pointed out that there is an error in my logic, about dot clusters, in that TIFFs have interpolated Bayer patterns.  Nevertheless, as you can see from this video, it's an open issue about the number and significance of these dots in the 1280x720 area of the sensor used in crop mode.

Today it looked like some users may have come close to fixing the "shutter-bug" problem with the 18-55mm IS lens.  Until that is solved, I don't see getting a 10-20mm IS lens from Canada.  It may have the same problems.

I was very enthusiast about this camera a couple of weeks ago but am now wary.  There seems to be only one dev working on it and he seems to be split between many projects.   I would not buy the EOS-M over any other Canon camera at this point.  The builds are still alpha and are not documented.  I went days where I didn't shoot video because I wasn't sure I'd ever be able to remove the pink dots. 

If it wasn't for the fact (I believe) of the 50D having to crop 5x and the EOS-M only 3x, I'd probably have changed my experimental focus to the 50D. 

I'm also wondering if there can be a better fix to moire problems with the EOS-M (and other cameras) in normal mode.  Maybe the devs could alternate line skipping (HA!  Bring back interlacing).  Maybe someone could produce down-sampling software, specifically for these Canons, that minimizes it.  So many questions.  So few answers.


deleted.account

Haven't read the thread and this may have been mentioned but if you're investigsting removing the dots before demosaicing or as part of a demosaicing algo why not contact the dcraw dev David Coffin or libraw developer at www.libraw.org or the DCB or Amaze dev it's all open source code and see if they can help?

You're more likely to get a solution there then via commercial software and DCB or Amaze in dcraw or dcraw variation as in libraw is no bad thing. Maybe Elle at Nine Below Zero Photography with her dcraw variation might be an another dev route.

a1ex

Median filter from dual ISO tools didn't help?

Rewind

Since this crap happens in overexposed areas, it is possible to use any simplified interpolation allgorithm.
By the way, ACR does it for you. This problem shows up only in dcraw.

zuzukasuma

just encountered with af dots on dual iso files. when I check cr2s they are clean but after cr2hdr.exe processing they are all over the photo. I can not uplad the photo but the screen shot if its necessary. PDR seems not to work with 18mp DNGs, I'd like to have a solution for them, or I'll stick to old version.
in a complicated relationship with eos m.

maxotics

@y3llow  dcraw has a way to interpolate around dead/hot pixels.  You just need to attach a pixel-location file during conversion in the form of column, row location.  I've tried to create one of these files but it didn't work, probably because I didn't calculate the positions properly.  My problem is an effective way of identifying these pixels.  I don't know how to extract a RAW image from the RAW video file, or know it if is much different than the DNGs.  In any case, using the DNGs to TIFFs and writing a small program to give me coordinates of the dots has not worked for me (though I may have used a bad frame, need to try again at some point).  Anyway, I've tried many types of shooting methods to come up with these focus pixels, but there is always noise, etc.  I think I can get there, but hope someone here already has the best procedure.

I believe that those who have calculated these positions for PDR, are doing it manually, visually (at 2,000% magnification) identifying grids of dots and entering their start/end/offsets into PDR.  Right now, in crop mode, it seems they have identified 5 patterns.

I've asked the question in the PDR thread about strategies of analyzing a RAW file to get these locations.  I think the original devs of PDR are busy with other stuff.  Only Rewind is around and he's new to it.  Anyway, no answer yet.

I would think, since it seems this issue is creeping into other aspects of ML, that we would want to get a good map of any Canon camera that sets aside pixels for focus, or whatever.  I would think they are always the same.  Once we have the bad pixel map it could be incorporated into current workflows because most use dcraw.

@a1ex  I'd LOVE to get to dual ISO.  Are there any instructions?  EDIT: just skimmed through thread.  Oh brother!  I have enough problems with single video frames ;)




deleted.account

Quote from: Rewind on September 09, 2013, 08:29:16 AM
Since this crap happens in overexposed areas, it is possible to use any simplified interpolation allgorithm.
By the way, ACR does it for you. This problem shows up only in dcraw.

The crap being focus dots in pink or crap being the focus dots whatever color?

When you say 'overexposed' what do you mean? Clipped channel before white balance or blown pixels after white balance?

Does changing or setting the sensor saturation level for the camera make any difference or highlight recovery mode?

Rewind

Quote from: y3llow on September 10, 2013, 06:43:28 AM
The crap being focus dots in pink or crap being the focus dots whatever color?

When you say 'overexposed' what do you mean? Clipped channel before white balance or blown pixels after white balance?

Does changing or setting the sensor saturation level for the camera make any difference or highlight recovery mode?

Focusing pixels exists in red and blue channels. Green channel wisely stays untouched. So none of them actually pink. it is interpolation that makes them looks pink.

Overexposed in this case means clipped.

This 'crap' happens when you shooting like the sun directly with dual ISO. And Im not sure these artifacts have the same nature as focusing pixels.
I mean, af dots is the problem of LV raw stream, right? There are no af dots in usual cr2's. When the mirror pops up, these af dots miraculously became an usual pixels/ (correct me if i'm wrong). So why would they appear at all in dual iso shots?

deleted.account

Quote from: Rewind on September 10, 2013, 09:59:52 AM
Focusing pixels exists in red and blue channels. Green channel wisely stays untouched. So none of them actually pink. it is interpolation that makes them looks pink.

Interpolation? ok, so pink is not from dcraw's RGB multiplier scaling for white balance based on the clipped channel? Like if dcraw -H 1 is used for blown highlight handling and a channel is clipped then highlights discolor? Do pink dots appear with -H 0 and sensor saturation level having been set correctly for the particular camera, dcraw's -S CLI option.?

QuoteOverexposed in this case means clipped.
So full well? Clipped before demosacing and WB multipliers?

QuoteThis 'crap' happens when you shooting like the sun directly with dual ISO. And Im not sure these artifacts have the same nature as focusing pixels.

Is this different crap to the focus dots you mean here, not been funny, I just haven't read up on the pink dot thing.

zuzukasuma

Quote from: y3llow on September 10, 2013, 02:30:02 PM
Is this different crap to the focus dots you mean here, not been funny, I just haven't read up on the pink dot thing.

if you're using after effects for DNG import you can always try to use mask tool* for dots, its a painful thing to do but gets the job done.

*you know, for dust, birds etc.
in a complicated relationship with eos m.

maxotics

I spent all night on this.  As Rewind says, the pink dots are active red and blue sensor pixels.  Rewind, I believe the DNGs created by raw2dng discard the Bayer pattern, so I can't look at each individual sensor pixel (R,G,B,).  Are you able to extract this information from the RAW file?

I've done a lot of experiments on how to get the best "shot" of the focus point.  I focus on a light table at camera-good exposure, and through some simple gamma adjustments, get this image.  Where the image is out of focus the dots are faint or don't appear!



I'm now trying to generate a "dark frame" from it that will work with dcraw but having problems.

I used this file to create "bad pixel" locations and feed that through dcraw.  BTW, I calculate that this focus pixels take up 4.4% of the 1280x720 center crop of the sensor.

Unfortunately, I believe dcraw only does interpolation about bad pixels from a true RAW file, not from DNG.  Anyway, I can't get it to work.

The good news is that ufRAW has a very good procedure for finding these hot pixels and interpolating around them.  Can be used in batch mode.  Still working on it; will write up procedures later.

It seems the devs in ML think the "pink dot" problem is solved.  I believe, for any modern Camera with this focus design, this problem is going to rise up again and demand to be taken seriously by the devs ;)

PDR treas the dots as squares.  I believe they are diagonals, as the patent application graphic suggests.  They start from the center.  It is why dots to the far left right don't have the same density.  So much to figure out with this stuff.

Rewind

Quote...I believe the DNGs created by raw2dng discard the Bayer pattern, so I can't look at each individual sensor pixel (R,G,B,)...
Jesus Christ, man ))
dng is raw. Nothing discarded in it.
In order to see each individual pixel of bayer pattern you should use some soft, which allows you to do that. Photivo for example:


QuotePDR treas the dots as squares.  I believe they are diagonals
Well, you definetely need some sleep, fellow ))

maxotics

Jesus Christ Man, you better be right )) :)  I've read that some RAW converters do throw out the exact Bayer pattern. I checked with Imatest.  I'll go open one in Photiva and see! 

maxotics

Cool beans Rewind!  So how can we go into the DNG's Bayer pattern data and interpolate around those specific red and blue pixel locations?   You can draw squares around the dots, but I don't think that's the logic of how they're constructed.  I need more than sleep!  I need your young brain ;)

Here's a blow up of the Bayer patterns (thanks for tip Rewind!)  Guess what, I think you rushed to judgment on those green pixels ;)  Hard to see in this shot, but I believe they are used too EDIT: Dang, optical illusion!  You're right again :)


Rewind

In Photivo you can use the channel mixer to view only channels you want.
For example, to disable green channel, which is not suffering from af dots, you should use it like that:


But the main question is: what exactly are you trying to do?

Pink Dot Remover tool already uses a very clever-smart-ass-shiny adaptive interpolation algorithm from this paper: http://www.tanbakuchi.com/Publications/Papers/2003AdaptivePixelDefCorPub.pdf

You think you can do better interpolation or what?

maxotics

Quote from: Rewind on September 10, 2013, 03:42:49 PM
You think you can do better interpolation or what?

Now, now, don't suggest I'm THAT dumb ;)  I was using my EOS-M and everything going well when the pink dots came back.  If it wasn't for you where would I be?  The footage and camera would be useless.  So I want another method to fix the pink dots if the devs do something that causes them to return.  I don't want a better plan.  I want an alternate, back-up, plan.

That said, I'm not convinced PDR is perfect (which is why someone I know starting looking for improvements!).  After all, as you see, the pink dots came back in the clipped areas of those videos I took. 

It seems to me PDR makes it more complicated than it needs to be.  We should know the EXACT location of each focus pixel.  So why not mark it bad and interpolate around it?  As long as you know the exact x/y position of the frame, relative to the sensor, an algorithm should be able to do this.  Right?

With you telling me what to do, I believe we can create this map :) 



Rewind

Well, the thing is, exact positions already calculated. Otherwise, how PDR could even work? ))
You can mark them all via PDR by utilizing Set Dead Pixel option and left the interpolation for your raw processor.
By the way, they suck at interpolation: http://www.magiclantern.fm/forum/index.php?topic=6658.msg70661#msg70661
PDR with my modified adaptive algorithm does the better job.

I gave you the actual map, I described the maths of how the positions obtained.
You have everything you need.
How else can I help you? I'd be glad really, but I just don't undestand what do you want )

maxotics

Quote from: Rewind on September 10, 2013, 04:01:43 PM
Well, the thing is, exact positions already calculated. Otherwise, how PDR could even work? ))

The dots that appeared in clipped areas.  Also, I do see their artifacts, as you point out in your article.  Don't know if cause is in-exact locations or bad interpolation.  I think we both agree, there are issues?

Quote from: Rewind on September 10, 2013, 04:01:43 PM
I gave you the actual map, I described the maths of how the positions obtained.
You have everything you need.
How else can I help you? I'd be glad really, but I just don't undestand what do you want )

You gave me a way of describing start/end/offset positions for 5 pixel grids for PDR.  THANKS!  However, does that account for all the pixels?  What I want is an x/y coordinate list of all these pixels.  I'm not sure PDR's "dead pixel" works perfectly.  Does it? 

You understand PDR, so have more confidence in it and see less reason for me to worry.  What I want, again, is a way to calculate these positions without PDR and use other software to interpolate around them.  Do you think this unreasonable?  PDR relies on Java and a formulaic way of looking for bad pixels. 

Or maybe another way I should put it, is how do we create a dark-frame that would work in other kinds of interpolation software?  Like Photiva even?

Rewind

Now I see your point.
A couple of hints:

If you want that your alternative plan works well for every resolutions and aspect ratios, you will unavoidably end up with similar to PDR method to calculate positions. I mean the offsets from center. That seems as an obvious and most reasonable technique for me.

To get clean exact map for af dots, what I usually do:
— Shoot something evenly lit (something flat and underexposed, or even with a cap on works fine for 650d)
— extract 50 to 100 frames with raw2dng
— open one of dngs in photivo and make adjustments: 1:1 view, turn off the green channel, use bayer pattern instead of any interpolation, output as 16 bit png, then export settings as a job file
— make the batch job to export all those frames as pngs
— open all pngs in photoshop as layers (document depth must be at least 16 bit )
— select all of the layers and convert them into the smart object
— go to Layer - Smart Objects - Stack Mode, and set something like mean or median. This gives you a better signal to noise ratio, thus ensures none of the random noise pixels will be identified as hot/dead pixel
— use curves to bring up the af dots and suppress the remaining noise.

From there it is not so hard to write down the positions in any coords system you like.

PDR does use exact positions, otherwise it just won't work.

If Set Dead Pixels option is enabled, it simply puts zero to every af dot. Clean zero interprets as dead pixel by every raw procwssor afaik. You can compare the interpolation algorithm by yourself, but my research gives me verdict that PDR does the best job so far. Just for example, PDR vs ACR: look closely to the edges treatment:


You can not just create a dark frame, because af dots intensities changes dynamically. The only way is to interpolate.

maxotics

Do you know why the pink dots appeared in the clipped areas?  That's what got me worried again.  Thanks! 

Rewind

Quote from: maxotics on September 10, 2013, 05:55:49 PM
Do you know why the pink dots appeared in the clipped areas?  That's what got me worried again.  Thanks!

Not yet. Interesting problem, but I can't reproduce it on my 650D.

maxotics

When it happens you'll be my best friend again ;)

I have my nephew helping me out on this for practice.  He wants to get a job as a developer here.   Long story.  Right now, I'm having him go through Mixer2's PDR code.

What project/work do you think best to do? 

Sorry to keep sounding dense!  Why can't we send an array to come command-line program that would go through the RAW file and interpolate around all the known focus pixels.   I realize this is what PDR does, but seems we could have a raw2depink.exe or something like that?

Again, thoughts on what dev might make sense here?  I'm happy to let this whole thing drop if you think there's a bigger issue with shooting RAW on these cameras.

Rewind

QuoteWhy can't we send an array to come command-line program that would go through the RAW file and interpolate around all the known focus pixels.   I realize this is what PDR does, but seems we could have a raw2depink.exe or something like that?

So what's the point of raw2depink.exe if PDR does already exactly the same and operates on RAW files nicely? Just another funky name?

The real problem is... the nature of af dots. Can we find a solution to 'switch them off' ?
Obviously, sensor pixels can operate in two different states, because when the mirror pops up, they become an usual pixels. Something to do with LV raw stream? This is the grave we should dig in ))

maxotics

Quote from: Rewind on September 10, 2013, 06:40:06 PM
Obviously, sensor pixels can operate in two different states, because when the mirror pops up, they become an ususal pixels. Something to do with LV raw stream? This is the grave we should dig in ))

I agree, this has been asked for in the thread.  BTW, my EOS-M is mirrorless.  Yes, when ML writes the RAW file it should just interpolate or deaden those pixels.   I have no idea what's involved in ML code.  Have you forked any of the ML builds? 

I wasn't looking for a new funky name, though thank you!  Just a way of getting away from Java and using a "static" data approach to DNG fixes.  It's easier for me to batch Windows programs instead of Java.  And I don't trust Oracle for a second ;)