PinkDotRemover tool 650D

[talosectos]

Well if I use raw2dng and then PDR I can't edit it in Davinci Resolve. Is there a way how to get rid of pink mess and also convert it to CinemaDNG?

Yes, just be patient. RawMagic is producing CinemaDNG. Mixer2 and foorgol are working to adapt the tool for it.

Short update on the dot remover:

  * Changed algorithms to support EOS M better

  * Fine-tuning EOS-M dot location database

  * First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed


Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!

[khurra]

thank you so much for this tool!!

here is a test video i did as well as my process for getting the final image: https://vimeo.com/69773294

process:
shot RAW footage on my t4i w/ Magic Lantern.

settings 1280x720 at 23.976 - 23.978 (by default) on a 32GB SanDisk EXTREME PRO 95MB/s SD/HC class 10.

used raw2dng on my iMac to convert RAW to dng
used the pinkdotremover tool from the ML forum to remove pink dots
opened pinkdotremoved DNG's in PS
graded
exported images from PS as TIFF
opened TIFF sequence in Compressor
Exported Apple Pro Res file making sure frame rate was 23.976.

NOTES: Let's use a DNG folder with 5 frames in it as an example. When using pinkdotremover the DNG's come back as single files (if you have 5 files it will look like 5 files) but when you select the 5 files it will show that there are 10 files selected. When you open the pinkdotremover processed DNG's in PS it will have a sequence of 5 images with the pink dots removed followed by another 5 image sequence that still has the pink dots. Before exporting to TIFF delete these files from the sequence, or before exporting in Compressor make sure the TIFF files that still have the pink dots are deleted.

[350D]

Guys, i'm not sure, but did you made any research about dot removal via simple color overlay?
I have tried to create couple layers with different overlay modes here:

[foorgol]

Yes, just be patient. RawMagic is producing CinemaDNG. Mixer2 and foorgol are working to adapt the tool for it.

... and basically the current status is unchanged. THEORETICALLY the dots should be removed, but PRACTICALLY a few of them survive. I guess I'll release the new version anyway, because I need your feedback and a few more sample pictures to hunt down the bug...

[foorgol]

When using pinkdotremover the DNG's come back as single files (if you have 5 files it will look like 5 files) but when you select the 5 files it will show that there are 10 files selected.

The name of the image files created by the dot remover that with an underscore "_". Is it possible that an underscore indicates a hidden file on your system or in your file manager? Or does you file manager show "aaa.dng" and "_aaa.dng" only as one line, because it assumes that the underscore file is something like a backup file?

[foorgol]

Okay, just got the GO! from Mixer2 not to wait for him and release a new version
(He's working hard to setup and test a lot of resolutions for the EOSM, so no blaming him here)

Well then, here it is:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__007.zip

What's new since version 006 (partially already available in the previously posted debug version):

  * Fix for the wrong path handling that lead to a crash for some of you

  * Initial support for EOSM

  * Support for all 650D resolutions (see comment below)

  * Faster image conversion due to optimized algorithms

  * Faster start-up

  * Initial support for the 16-bit DNGs generated by RawMagic

  * When selecting a bunch of files or a directory, only DNG files will be added to the conversion list


Comment on the "all resolutions for 650D" feature:
We've found a way how to extrapolate a single set of dot locations to arbitrary image resolutions. And it works. Mostly.
During testing, we found one or two resolutions which do not fit into our extrapolation scheme.

What this means for you:
The tool will convert the files without complaining. You'll see no error message. But the resulting file might still contain all or at least some dots. In this case, please send me a message with the image resolution and I'll set it up as one the few "special cases" that need individual dot location definitions.

Tested and verified resolutions for the 650D are:
            1280x720
            1344x572
            1344x756
            1472x626
            1472x828
            1600x680
            1600x900
            1728x736
            1728x972
            1808x1190 (Silent DNGs Movie Mode)
            1808x727 (Silent DNGs Picture Mode)

Source Code:
https://github.com/Foorgol/PinkDotRemover

Enjoy!

[foorgol]

I will soon make low light timelapse video just to see how the dot removal tool handles lower light conditions. Shouldn't be much difference?

Right. The tool now removes the dots all over the image, no matter what the lighting conditions are.

Let me give you a more detailed explanation:

In previous builds, I removed only the "light dots" in the center based on a fixed "grid definition" while the "border dots" were removed based on empirical/statistical data read from sample DNGs. Which was wrong.

Reason: Both "center dots" and "border dots" are expressions of the same regular grid of dots, just with different visibility.

Since the debug version (and officially since the 007 version) "center dots" and "border dots" are now handled alike: based on a fixed grid-like definition of dot locations. All over the image, from left to right. Always.

Theoretically there is one disadvantage: I also apply interpolation to border locations where you wouldn't see the dots in your particular image, because e. g. it's a very bright image. So I'm applying interpolation to locations although there's nothing wrong them. This could lead to a small loss of detail.

But in practice I couldn't notice this disadvantage. And even if you could notice it: it would be surely outweighed by the disadvantage of having pink dots accidentally popping up because the removal wasn't thorough enough.

[spider]

Thank you for the new version.
I have a question is it possible to output the files into a new folder?
So it would be easier to delete the old files.

[talosectos]

... and basically the current status is unchanged. THEORETICALLY the dots should be removed, but PRACTICALLY a few of them survive. I guess I'll release the new version anyway, because I need your feedback and a few more sample pictures to hunt down the bug...

Thanks for this new version.

I made a few tests and it seems to work perfect with rawmagic.

I'll make more tries this evening after work.

[khurra]

The name of the image files created by the dot remover that with an underscore "_". Is it possible that an underscore indicates a hidden file on your system or in your file manager? Or does you file manager show "aaa.dng" and "_aaa.dng" only as one line, because it assumes that the underscore file is something like a backup file?

I realized that I was missing the underscore. All the files without the underscore are the originals. It doesn't completely overwrite the old files. THANK YOU so much for the tool. It is ABSOLUTELY amazing.

I wish I knew C better so I could help!!

[saulbass]

Thank you for developing the PinkDotRemover - it works perfectly.

Now I can use my 650D to make beautiful 1280 x 720 raw movies.

[mixer2]

@foorgol: still don't think that the method causes loss in detail at all. not all af dots are visible in all conditions, but they are still there with an incorrect value. the difference of the af-pixel value and the value that the pixel should have is just to small to notice it. however even in that case the interpolated value should be more exact than the wrong af-pixel value.

@spider: some new features that enhance the usability will come soon. we discuss features like optional overwriting the dngs, extra output folders and removing dots directly in the raw instead of the dngs. you can add feature requests on github: https://github.com/Foorgol/PinkDotRemover/issues
that way you can be sure, that the feature idea doesn't get lost between the other posts in the forum.

[Fuma]

@foorgol thx for all the good development. I think now the EOS 650D needs an option for Crop Mode too.

[foorgol]

@foorgol thx for all the good development. I think now the EOS 650D needs an option for Crop Mode too.

I'm not sure that I understand what you mean... are you referring to additional resolutions that need to be supported by the PinkDotRemover? Or do you mean ML-support for additional shooting modes/resolutions?

[Fuma]

Pls forget my post.
I have seen, that i can use EOSM_Crop even for my Videos that i have taken in Movie Crop Mode with the 650D.
My fault

[mixer2]

hurm... that is interessting. are the dot positions in crop mode for 650d and eos m identical? they should be, since they share the same sensor, but we handled them seperatly, since the downscaling without crop mode of 650d and eos m are completely different (totally useless for eos m) and that caused different dot positions on those models. we should rename the eos m crop mode to crop mode then.
the 007 version is btw still buggy with crop mode. all resolutions that are not divisible by 4 shouldn't work. there will be an update to 008 soon.

[demetrisag]

Looking forward to the new update for better support for crop mode too!

btw 1280*720 crop mode on 650D doesnt work with pinkdotremover with eosm_crop. is this resolution not divisible by 4?

edit, it works after i downloaded the latest raw2dng.exe

[foorgol]

So, let's break the silence here! Time for a new debug release!

The last days I gave the code a major re-write in order to support PinkDotRemoval directly in the RAW files. Nevertheless, removal in DNG-files is still possible, of course.

I did some testing, but it's not really what I would call thorough testing. Therefore this is a debug release only. Please give it a shot and tell me if it works for you. Download the binaries here:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__debug.zip

BEWARE:
Other than DNG files, RAW files will be OVERWRITTEN IN PLACE. Make sure to create a backup of your RAW file before you start the conversion.

Implementing it this way was easier for me. So much to my paradigm of not overwriting user data...

Enjoy!

P.S.: The in-RAW-removal together with your feedback and a few improvements that Mixer2 has in the pipe will constitute the next "official" release 008, I think...

[spider]

If I select the RAW files they are not added to the conversion list. I use the latest JAVA7

  * When selecting a bunch of files or a directory, only DNG files will be added to the conversion list

Maybe it depends on this?

[COMMANDES]

I have the same problem

[foorgol]

Aaargh... my fault... I packed the wrong JAR file. Actually the bug you mentioned was the last one I fixed. But then I forgot to built a new "distribution JAR"... I only tested it with the "development environment JAR".... well, whatever...

Now I've uploaded the correct version. The link posted above is still valid.

Sorry for the confusion!

[demetrisag]

Seems to work great for me here mate! It's even easier now with direct raw proccesing. and it seems faster too? or maybe just my idea!

[papkee]

Great news! The in-RAW idea really makes a lot more sense. No more triple-sized folders full of 3 different versions of the video!

[spider]

RAW processing works great I think and it is so awesome fast. 

[spider]

@foorgol

May you have a look at this thread http://www.magiclantern.fm/forum/index.php?topic=5645.new#new

I think you can give fatpig better information than me. Because I asked if it is possible to integrate the pinkdotremover into BATCHelor.