PinkDotRemover tool 650D

[gary2013]

Has there been anything new on this issue of PDR? Max, any new advancements on your project?

Gary

[Rewind]

Has there been anything new on this issue of PDR?

What issue?

[gary2013]

What issue?

so that's a no.

[maxotics]

Gary, I've been using Rewind's "crop mode" version for any videos I do in 1280x720 and the latest original for everything else and it is working okay as long as I don't shoot any blown out areas. 

I have separate folders "PDR", "PDR Rewind Crop Version" with each version.  I run the RAW through RAWanizer to see what aspect ratio the clips are, and then PDR accordingly.  Though I do everything crop mode now.  The other isn't worth the moire risk.

I have a method for generating focus pixels and creating a bad pixel list for dcRAW type convesion, but when PDR works, as Rewind says, that's the best method (it interpolates better than dcRAW).  However, i still hope to get that working with the same interpolation method as PDR and incorporate into raw2dng or mlv2dng, etc.

What we really need, for now, is a PDR that has updated drop down selections. I looked at the code but could see it wouldn't be a quick fix for me.  I imagine it's a little complicated or Rewind would have done the same.  But Rewind, if you can incorporate your crop mode with the earlier stuff that would be optimal.

We need something like

650 Normal
650 Crop
EOS-M Normal
EOS-M Crop

Even if both are the same, would allow adjustments to each if something changes in builds.

[gary2013]

I am still trying to understand why I do not have any pink dots when I do not use the PDR and I only use Alex's latest   
raw2dng_cs3x3_ahdlike_noise.exe along with the Neat Video filter for noise reduction in Premiere Pro editing.

Gary

[a1ex]

Here's another raw2dng that does a little less chroma smoothing (it only considers top, bottom, left and right neighbours). It still removes all the pink dots.

raw2dng_cs2x2_ahdlike_noise.exe (fewer artifacts)

(actually this kernel is more like 2x2+1)

Previous versions:
raw2dng_cs3x3_ahdlike_noise.exe
raw2dng_cs5x5_ahdlike_noise.exe (for really noisy shots)

Original raw2dng (without chroma smoothing):
raw2dng.exe



@gary: this raw2dng applies a median filter to red-green and blue-green channels, and since the pink dots are only on red and blue channels, it removes them all (without requiring PDR). The disadvantage is that it doesn't know where these pink dots are, so the filter is applied to all the red and blue pixels.

Sometimes this filter does good (it may reduce noise or hide some color aliasing), but in other situations it blurs fine details or introduces artifacts. Experiment with it and see what works best.

To remove pink dots with minimal alteration of the original raw data, you still need to use PDR (when it works), with the original raw2dng (without chroma smoothing).

[maxotics]

A1ex, I think you asked about this before you started doing this, but now that you've got the smoothing part done it may be a good time to revisit.

I created a set of files for you if you want to smooth only the focus pixels.  There are three 24 frame RAW files at these resolutions

1728x584
1600x642
1280x720 crop mode

Zip is 200meg

https://app.box.com/s/6q3nmjih6g54mfeucgo1

I also included bad pixel lists (dcraw format).

I shot clouds out of a dirty window.  Should be good enough for testing I hope.  The 1280 bad pixel list  may be too aggressive. 

All the pixel lists can probably be improved.

If you're up to this and it works, I can generate bad pixel lists for all resolutions.  At some point I hope to upload my bad pixel mapper to bitbucket too.

Having some version of raw2dng taking care of this would be so freaking amazing as everyone with these cameras would tell you!

[britom]

what does AHD mean? i couldnt find anywhere

[Rewind]

what does AHD mean? i couldnt find anywhere

Adaptive Homogeneity-Directed interpolation.

[britom]

@Rewind thanks a lot for the quick answer

[delorossa]

Hi Guyz i downloaded the Tool

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip

but when i click or double click on PinkDotRemover (Executable Jar File ) it does not open?! I intalled of course Java have both installed 32bit and 64bit.

Regards


HEY ADMIN PLEASE DELETE MY ACCOUNT! because my comment must be approved bevor it is Online,I take my help elsewhere.

[dsagilles]

Here's another raw2dng that does a little less chroma smoothing (it only considers top, bottom, left and right neighbours). It still removes all the pink dots.

raw2dng_cs2x2_ahdlike_noise.exe (fewer artifacts)

(actually this kernel is more like 2x2+1)

I've just tested it and it seems to work pretty well, remove a lot of dots/artifacts where PDR doesn't !
I've updated my RAW post processing tutorial and allowed myself to link your raw2dng version, I quoted you of course (http://www.gna-productions.com/tutos/how-to-process-magic-lantern-raw-videos , part 3), I hope you don't see problems.

Thanks, and again (not again but it was in my mind since the beginning) : good job !

[a1ex]

If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)

[gary2013]

I got a pop up from Java saying there is a newer version and after installing it, I can't run PDR. I have tried uninstalling java and then reinstalling the older version of Java, but PDR will still not work. Any clues or fixes?

Gary
Edit- I did a fresh install of windows 7 and java. that fixed whatever was wrong.

[dsagilles]

If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)

The problem is I directly extract frames with a chroma smoothing, so I can't really see the difference.
But if I see any issue, I will report them here, no problem !

[maxotics]

If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)

I couldn't see any differences in my experiments, the lost of detail was too great. The reason chroma smoothing won't work, IMO, is the chroma distortions are line based, not pixel based; that is, you can't sample around a pixel (it will just pick up the distortion from the pixels to the left or right), you must sample above and below a line and probably greatly reduce the chroma of the offending pixels.

However, I do beleive chroma smoothing CAN work on aliasing.  You would need an algo to figure out a line of pixels, say 8 pixels wide, and then an offset to the left or right (what creates a moire effect), and smooth using that information.  You would smooth 2-3 lines.

I have a light table and moire charts and can create lots of good aliasing/ moire images for anyone to work with.  It's on my list of things to do, but would do today if someone took an interest.

Another thing I'm curious about is if any algos exist out there.  I would think the camera manufacturers do something similar in their in-camera software.

Each line distortion seems to me a predictable interference pattern along a sensor line,  RGRGRGRGRG or BGBGBGBGBG.

[roterrion]

I did a comparison between raw2dng_cs3x3_ahdlike vs raw2dng_cs2x2_ahdlike_noise to process dual iso dngs.



I see big difference on highlights. Hope that helps.

[a1ex]

Can you upload the original DNG? I think the proper way is to get rid of the dots somewhere after exposure matching, but I don't have any sample to try (none of my cameras have pink dots).

[roterrion]

I was using my camera upside down. sorry. but here's the link:

raw2dng_cs3x3_ahdlike.DNG https://docs.google.com/file/d/0BwGjP3nTWukGRWxXeEhseTRUaDg/edit?usp=sharing
raw2dng_cs2x2_ahdlike_noise.DNG https://docs.google.com/file/d/0BwGjP3nTWukGRHlCYTQzaVJ4X0E/edit?usp=sharing

[a1ex]

I mean the unprocessed one (with the plain raw2dng, without any modifications).

[roterrion]

original raw2dng https://drive.google.com/file/d/0BwGjP3nTWukGc19uTy1tOTFfS1E/edit?usp=sharing

[a1ex]

The 2x2 raw2dng does a pretty good job (I have a hard time trying to beat it). Nice find.

But if you can run PinkDotRemover, I think that one will give noticeably better results. Chroma smoothing artifacts are still visible with dual ISO.

[roterrion]

right PDR does better, I just found an artifact on a frame from raw2dng 2x2. but I just love the workflow of raw2dng 3x3/2x2. drag n drop and it extracts clear frames.

[gary2013]

I did a comparison between raw2dng_cs3x3_ahdlike vs raw2dng_cs2x2_ahdlike_noise to process dual iso dngs.



I see big difference on highlights. Hope that helps.

which one do you like better? there is also a 3x3 noise version you can add to compare.

Gary

[Letni]

"raw2dng_cs2x2_ahdlike_noise"   looks better because it does not leave streaks on the dress.
But there are still artifacts available.