PinkDotRemover tool 650D

[foorgol]

There you go mate, I only skipped one line at the very top, which was the command in order to include it all in one screenshot

Thanks for the data!

Just by quickly looking at the log messages, I realized that I might have made a very stupid beginner's mistake: I blindly assumed that all picture file names always have a path component, i. e. "/this/is/your/picture.dng" and not just "picture.dng".

I'll fix it tonight, should be a no-brainer.

In the meantime you can try to call the program like

java -jar PinkDotRemover.jar .\5.dng  or java -jar PinkDotRemover.jar ./5.dng   (depending on your OS)

or simply have pictures and tool in different directories. If my assumption above is correct (haven't looked at the code yet), this should fix your issues for the time being.

[donjames150]

Thanks for the data!

Just by quickly looking at the log messages, I realized that I might have made a very stupid beginner's mistake: I blindly assumed that all picture file names always have a path component, i. e. "/this/is/your/picture.dng" and not just "picture.dng".

I'll fix it tonight, should be a no-brainer.

In the meantime you can try to call the program like

java -jar PinkDotRemover.jar .\5.dng  or java -jar PinkDotRemover.jar ./5.dng   (depending on your OS)

or simply have pictures and tool in different directories. If my assumption above is correct (haven't looked at the code yet), this should fix your issues for the time being.

what worked for me was to create a folder, then a subfolder and from the folder to use this syntax:
java -jar pinkdotremover.jar subfoldername
then all .dngs in the subfolder would get converted

For setting the system path variable, select the start icon, then computer, system properties, advanced system settings,
environmental variables. In the system variables window, scroll down to path and edit it. At the end of the variable, add
this

;C:\Program Files (x86)\Java\jre7\bin

then keep selecting ok until you back out of that section
of course, that's for Win7, 64bit. If you have a different OS, then append the variable with wherever your java.exe is located.

[1%]

This should be built into chdk-dng.c... I just dunno how to mark the pixels exactly.

650D + eos M are different raw sizes tho.

[donjames150]

This should be built into chdk-dng.c... I just dunno how to mark the pixels exactly.

650D + eos M are different raw sizes tho.

I should mention that I output at 1280 x 720 and all the pink pixels seem to get eliminated.
On my canon settings I chose the 1920 x 1080 (24fps) before going into the ML and choosing
the 1280 x 720. That was the only way I could get that size at a data rate that the card can handle.

[1%]

I only get 1 size in LV and one size in zoom except when recording... the default LV is like 50D, stuck at low res. So only need 2 sizes.. actually 1 size.. no dots in zoom mode.

[donjames150]

I only get 1 size in LV and one size in zoom except when recording... the default LV is like 50D, stuck at low res. So only need 2 sizes.. actually 1 size.. no dots in zoom mode.

Sorry 1%, I am definitely in over my head at this point    I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.

[donjames150]

Sorry 1%, I am definitely in over my head at this point    I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.

Note the monitor with "Hello World" on it at 2:26

[foorgol]

Alright, here's a new build. What has changed:

  * Fixed the crash when using file names without path (see previous postings)

  * Added the option to simply mark the pink dots as "Dead Pixel" which then leaves the actual color correction to your RAW tool

  * Due to popular request: added a simple GUI

Here's a screenshot:



Should be pretty self-explanatory. Throw in whatever files you find (e. g. by selecting whole directories). The tool should be pretty tolerant against unrecognized or unprocessable files. Try your luck.

If you want to, you can still use the tool on the command line or in scripts, just like before. The GUI is only started when you provide no or only invalid command line arguments.

I assume that now with the GUI a simple double-click on the JAR-file is sufficient if Java is properly installed on your normal (Windows-) working computer. However, I couldn't try it because I only have a (Linux) development computer here, which behaves a bit different.

Download the binary here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.zip

I hope the ZIP has everthing you need to run the GUI on a normal machine. Let me know if you face troubles.


Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__004.src.zip

For those of you interesting in the source code:
The code is pretty crammed with debug messages now which makes it hard to read. I'll remove all the messages once this version proves to be more or less stable. And the GUI stuff lacks comments... you'll have to figure out yourself what's happening there...

Enjoy!

[foorgol]

I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.

Sorry, I can't help you with LV or similar. I have absolutely no clue what happens inside ML and how data is converted back and forth inside ML before being dumped to the card. I'm not an ML developer.

But regarding your idea to integrate the dot removal into ML:
From an architectural point of view, I would not do that. Writing the RAW data in higher resolutions is already time critical enough today (okay, mostly limited by writing speed, AFAIK). So I would push as much data processing as possible out of ML and up to your normal workstations with practically unlimited processing power, time and disk space. You have to touch all the image files anyway for post-processing, so adding the dot removal is not so much of a pain here.

Dot removal inside ML would only make sense if the built-in camera processors are extremely optimized for these kind of tasks (what's not unlikely) and/or if you have additional information inside the camera at runtime which would support/optimize the removal process but which can't be exported to the outside world.

From all I know today, I would rather strive for an integration of the removal process into raw2dng....

[foorgol]

Oh, I just found out that taking silent DNGs in movie mode gives you nice 1726x1150 images with correct aspect ratio and better exposure control. Guess which pink dot locations I'll tackle next...

[Aborgh]

Oh, I just found out that taking silent DNGs in movie mode gives you nice 1726x1150 images with correct aspect ratio and better exposure control. Guess which pink dot locations I'll tackle next...

Thank you for the tool foorgol

I guess 2.35:1?

[demetrisag]

Finally it works! thanks for this a lot Foorgol! Though I noticed, that dots are being removed very very well at the center of the image whereas at the two sides many dots are still here. I guess you have not tackled those yet?

Thanks again!

[midnite]

Rawtherapiee program did a good job, but some pink dots are still visible. They are annoying as hell.


after using Rawtherapiee

http://oi43.tinypic.com/2mi1grp.jpg

[donjames150]

Rawtherapiee program did a good job, but some pink dots are still visible. They are annoying as hell.


after using Rawtherapiee

http://oi43.tinypic.com/2mi1grp.jpg

I got this using the pinkdot removal tool

http://kozenrufu.com/_000116.jpg

First I set my canon movie record size to 1920 x 1080 (24fps)
Then I set ML to 1280 x 720, Aspect 16:9, Framing Center

It had all the usual pink dots before I ran the removal tool

Here's the video before and after.  Watch at 720p to clearly see all the pink dots

[pileman]

Could you add support to 1344x572, and also 1472x626 resolutions? Thanks.

[dimadan]

And for 1280x544. I have only 45mb/s sd card, and 1280x720 is stopped after 8 seconds of recording, but on 1280x544 - no stops!

Very thanks!

[mixer2]

the 1808x727 silentdng data can be used for all unzoomed video resolutions. the smaller resolutions are just crops. therefore the app just has to calculate the correct offset for other resolutions. as soon as that feature is supported all resolutions should work. so there is no need to ask for any specific resolution.

[Danne]

Hi! THis looks like a real nice program. Is it possible to have it in rawtodng tool? Ocassionally I get some annoying dead pixels showing and I don,t go through adobe acr but straight to proRes frpm dng with the rawtodng tool.
thanks
//D

[foorgol]

the 1808x727 silentdng data can be used for all unzoomed video resolutions. the smaller resolutions are just crops. therefore the app just has to calculate the correct offset for other resolutions.

This is fantastic news! This would save me/us a lot of work, because it's quite tedious to record the pixel locations for new or other resolutions. If instead I would only need to determine a x,y-offset, adding a new resolution would only be a matter of seconds!

However, there is one thing that doesn't support your assumptions:
The grid-like dots in the picture's center have a distance of 10 pix between two grid rows in 1280x720 while it's 12 pix between two rows in 1808x727. If 1280x720 would just be a crop of 1808x727, the row offset should be identical.

Any ideas why the row distance changes? Or is 1280x720 a "scaled" resolution?

[mixer2]

there are two different raw streams. normal and while LV is zoomed in. so all other cases should be crops of one of those 2 streams. using the same data for other resolutions is working on eos-m. have to check the 650d images to see what's the problem, but ml doesn't scale anything, just cropping.

[mixer2]

Hi! THis looks like a real nice program. Is it possible to have it in rawtodng tool? Ocassionally I get some annoying dead pixels showing and I don,t go through adobe acr but straight to proRes frpm dng with the rawtodng tool.
thanks
//D

you can use a script to generate dng with rawtodng first and then automatically process it with the dotremover. will that work for you? which os do you use?
i think it will take some time, until there is a single program solution, since the dot problem is specific for some camera models (650d, eos-m and i think 700d+100d too).

[Danne]

Sounds nice. I,m on a mac 10.6.8 I think. I have a 5d mark 3. The problems occur in high iso and dark areas. Still it,s only when outside acr when I want to export to ProRes

[mixer2]

hurm, on 5d mark 3 the type of dots, which this tool tries to remove, doesn't exist.
sure that it's not a kind of noise that you get? are the dots on the same position all the time?

[dariSSight#1]

what & where is the pink spot removal tool?

[Danne]

hurm, on 5d mark 3 the type of dots, which this tool tries to remove, doesn't exist.
sure that it's not a kind of noise that you get? are the dots on the same position all the time?

Yeah, they,re hot/dead pixels. Maybe not so good with this particular tool?