PinkDotRemover tool 650D

[donjames150]

what & where is the pink spot removal tool?

Read from page 1 (only 2 pages) . Basically it's for the 650D which gets clusters of pink pixels when shooting raw video. This tool removes them, but only
for a 1280 x 720 crop selection in raw.

[mixer2]

Yeah, they,re hot/dead pixels. Maybe not so good with this particular tool?

no, sorry, that tool isn't useful for that case. the dead pixel detection of the raw editor or your choice should solve that problem.

[Danne]

Aight, thanks

[demetrisag]

no, sorry, that tool isn't useful for that case. the dead pixel detection of the raw editor or your choice should solve that problem.

Both do the best job so far. But foorgol from what I know is still wokring on it, so I believe if foorgol finish it it will be the best thing on its own

[foorgol]

Okay guys, short status update:

  * I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions

  * Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version

  * Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well

Give me two or three days and by then I should have a new release out for testing....

[papkee]

Okay guys, short status update:

  * I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions

  * Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version

  * Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well

Give me two or three days and by then I should have a new release out for testing....

Foorgol, this tool is simply awesome. I think you need an official name for it now that it's up and working well.

[donjames150]

Okay guys, short status update:

  * I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions

  * Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version

  * Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well

Give me two or three days and by then I should have a new release out for testing....

thank foorgol, I've been wanting to play around with other sizes than the 1280 x 720

[dimadan]

Thanks for useful tool!
Can you add option for overwrite old files with dots?

Thanks again!

[donjames150]

Alright, here's a new build. What has changed:

  * Fixed the crash when using file names without path (see previous postings)

  * Added the option to simply mark the pink dots as "Dead Pixel" which then leaves the actual color correction to your RAW tool

  * Due to popular request: added a simple GUI

Here's a screenshot:



Should be pretty self-explanatory. Throw in whatever files you find (e. g. by selecting whole directories). The tool should be pretty tolerant against unrecognized or unprocessable files. Try your luck.

If you want to, you can still use the tool on the command line or in scripts, just like before. The GUI is only started when you provide no or only invalid command line arguments.

I assume that now with the GUI a simple double-click on the JAR-file is sufficient if Java is properly installed on your normal (Windows-) working computer. However, I couldn't try it because I only have a (Linux) development computer here, which behaves a bit different.

Download the binary here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.zip

I hope the ZIP has everthing you need to run the GUI on a normal machine. Let me know if you face troubles.


Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__004.src.zip

For those of you interesting in the source code:
The code is pretty crammed with debug messages now which makes it hard to read. I'll remove all the messages once this version proves to be more or less stable. And the GUI stuff lacks comments... you'll have to figure out yourself what's happening there...

Enjoy!

yes, on Windows machines, it opens with a double-click of the PinkDotRemover.jar file. Thanks!

[foorgol]

Well then, here it is.

As announced, the program is now able to handle multiple camera types and you can easily add new resolutions for dot removal. This build ships with support for the following 650D image types:

  * Silent DNGs taken in photo mode

  * Silent DNGs taken in video mode

  * RAW video frames with 1152, 1280, 1344, 1472, 1600 and 1728 pixels width and an aspect ratio of 16:9, 1.85:1, 2:1, 2.2:1, 2.35:1, 2.39:1, 2.5:1, 2.67:1 or 3:1

Beware: I have only verified the silent DNGs, 16:9 and 2.35:1 with real footage (except for 1152 width). The pink dot locations for all other modes are only calculated and not verified. Due to rounding, they sometime might be off by one pixel and then the whole dot removal doesn't work anymore.

If you've used one of these non-verified modes and it proved to work, please drop me a line. Then we can tick that one off as "verified".

Here's the binary:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__006.zip

In the GUI, you'll notice a new drop-down box for the selection of the camera model. The default is 650D. If you're using the tool on the command line, you can only process 650D images -- there are no means to choose the cam type on the command line.

The database is not yet filled with real EOS M data yet, but that should come soon.

Since I coded all the new functionality quite in a rush, the code quality suffered a bit. I did not much testing and even less commenting. The "Happy Path" through the program should work quite alright, though. Let me know if you encounter problems.

The source code is now hosted on GitHub:
https://github.com/Foorgol/PinkDotRemover
https://github.com/Foorgol/SimpleTIFFlib
https://github.com/Foorgol/LoggingLib

Enjoy!

[foorgol]

Can you add option for overwrite old files with dots?

Nope. I will not overwrite your files, not even as an option which you have to activate first. I feel bad enough by blindly creating / overwriting the _xxxxxxx.dng files without confirmation.

The risk is too high that you loose precious data, e. g. due to a tool malfunction.

If you want to delete the old files, please do so in your file manager or on the command line. It's just a few clicks or a few keystrokes and the risk is significantly lower....

[donjames150]

Foorgol, I'm getting an error when I double-click the .jar file

[spider]

After processing the files with the dotremover the dots are weak visible.
When I push the picture 1,5 Av the dots are again very good visible.

I shoot 1280*720 16:9 with an EOS650D
Is it normal?

[donjames150]

After processing the files with the dotremover the dots are weak visible.
When I push the picture 1,5 Av the dots are again very good visible.

I shoot 1280*720 16:9 with an EOS650D
Is it normal?

I shot this the other day at 1280 x 720, before and after .dng files

[spider]

after: https://www.dropbox.com/s/3jk2yuiip3wauml/_000000.jpg

before: https://www.dropbox.com/s/qjj9sz8vmjrcdgw/000000.jpg


Two in LR4 cropped 1280*720 24p footage pictures pushed 2Av from Canon 650D

You see some dots are completely removed, some dots are still there and some dots are a bit removed.


Here the two frames without crop
https://www.dropbox.com/s/nmc2r3yradmm964/_000000f.jpg
https://www.dropbox.com/s/21d248ei77w1sa1/000000f.jpg

[papkee]

Hey, I'm having a bit of an issue.

I shoot at 2.35:1 with 1472 width. (this is the max I can get with the t4i) I ran the program using "650D" and "Interpolate" but almost all dots are still visible. And when I attempt to use "dead pixel" it doesn't convert, simply saying "one or more files could not be converted"

Would like some help here.

[demetrisag]

Foorgol, I'm getting an error when I double-click the .jar file

same here

[donjames150]

same here

Since Foorgol was clearly born to program and uses a Linux OS, I suggest he do what I did in 1989
when I took a class on Unix. I put both Linux and Dos on two different partitions and had a dual-boot setup.
Then you could test on Windows machines which is what most people have (except the rich ones who have
macs  )

[foorgol]

Okay, one by one....

--------------------------------------------------
@donjames150 and @demetrisag:

Could you please run the program in a terminal window (instead of just double-clicking the jar) and send me the output? I have a faint idea what the reason for error could be. This idea would be confirmed if the program would run normal if you start it from the command line...
--------------------------------------------------

--------------------------------------------------
@spider:
There are two kinds of dots. The "block" in the center and the "strip" all across the image. It's characteristic for the strip-dots to only show up in dark areas or when you lighten the image.

Your issue has to with the way the position of the strip-dots are located. Currently it's a trade-off between removing fewer of these pixels and losing image details by accidentally removing too many non-pink dots . Mixer2 found an interesting approach to solve this which he describes here:
http://www.magiclantern.fm/forum/index.php?topic=3648.msg54162#msg54162

I'm working with Mixer2 on an improved "dot location determination" for these dots which will come with the next release.

Until then, you can try to edit the file 650D_Video_1280x720.txt which you find in the dotData directory. The 4th line ends with "2068". You can try to gently reduce this value, e. g. to 2050 or 2040. This should then remove more of these dots.
--------------------------------------------------

--------------------------------------------------
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
--------------------------------------------------

[papkee]

--------------------------------------------------
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
--------------------------------------------------

I'm not quite sure where that setting is. Is it an ML thing or a canon thing?

[donjames150]

Okay, one by one....

--------------------------------------------------
@donjames150 and @demetrisag:

Could you please run the program in a terminal window (instead of just double-clicking the jar) and send me the output? I have a faint idea what the reason for error could be. This idea would be confirmed if the program would run normal if you start it from the command line...----------------------------------------

looks like this

[donjames150]

I'm not quite sure where that setting is. Is it an ML thing or a canon thing?

framing is in the submenu under raw video in ML

[foorgol]

looks like this

Okay, my assumption was that the error has to do with the determination of the JAR's directory. I thought it would fail completely. But instead, it's just a back-and-forth conversion issue, where a space " " finally ends up as %20 like in an URL. So my guess was close

Should be easy to fix. For the time being, just avoid spaces in the directory name.

You always catch my sloppy ways to handle paths....

[demetrisag]

on the other hand it foorgol it worked with me on CMD

[spider]

I'm working with Mixer2 on an improved "dot location determination" for these dots which will come with the next release.

Thank you very much, this sounds great.

Until then, you can try to edit the file 650D_Video_1280x720.txt which you find in the dotData directory. The 4th line ends with "2068". You can try to gently reduce this value, e. g. to 2050 or 2040. This should then remove more of these dots.

I tried it, but the improvement is not so big an I cant set smaller values than 1058 if I reduce the value more the program sometimes freeze.