I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...
I) The easy way: use RawTherapee (http://rawtherapee.com)
For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.
You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.
RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.
II) The DIY way: start coding
I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...
Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:
Original DNG:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.jpg)
Modified DNG with dots removed:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.jpg)
(both JPGs developed with RawTherapee and identical settings)
If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip (https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip)
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at your own risk and cross your fingers... :)
As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.
I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.
Here are a few sample DNGs to compare the original with the modified image:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...
Quote from: foorgol on June 15, 2013, 08:51:57 AM
I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...
I) The easy way: use RawTherapee (http://rawtherapee.com)
For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.
You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.
RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.
II) The DIY way: start coding
I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...
Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:
Original DNG:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.jpg)
Modified DNG with dots removed:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.jpg)
(both JPGs developed with RawTherapee and identical settings)
If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip (https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip)
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at you own risk and cross your fingers... :)
As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.
I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.
Here are a few sample DNGs to compare the original with the modified image:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...
There's already a superior way than either of these for now. In Lightroom or ACR, turn Noise Removal in both Luminance and Color all the way up and then adjust the detail sliders until just around the point where the dots disappear enough. Finish developing the rest of the settings (exposure, vibrance, brushes, graduated filters, WB, etc) then just select all the photos in LR or Bridge and sync your develope settings. Voila.
Quote from: foorgol on June 15, 2013, 08:51:57 AM
I saw the pink / blue dots in the DNGs as well and I think I have a solution. Or even two (!) solutions...
I) The easy way: use RawTherapee (http://rawtherapee.com)
For those of you who don't know: RawTherapee is a really powerful open source tool for developing RAWs. Runs under Windows, Linux and Mac.
You can directly open a DNG-file and if you then go to the "Raw"-Tab, unfold the "Preprocessing"-section and check "Hot/Dead pixel filter", the disturbing dots will magically disappear. Okay, most of them. Before you do this, you should select a processing profile from the drop down list under the histogram, because selecting a profile resets your "Hot/Dead pixel filter" setting (and all other settings as well) with the profile's default.
RawTherapee can do batch processing, so you can convert a whole image sequence for a movie with just a few clicks.
II) The DIY way: start coding
I hacked a small command line tool that removes the pink pixels directly in the DNG file. It's written in Java, so it should run under Windows and Mac, although I developed it under Linux... but who knows...
Currently the program can only remove pixels in DNGs with 1280x720 resolution, because it uses pre-stored pixel locations. I "calibrated" it to my 650D, but I guess/hope the pixel positions are identical on all 650Ds. Here's an example:
Original DNG:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.jpg)
Modified DNG with dots removed:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.jpg)
(both JPGs developed with RawTherapee and identical settings)
If you want to play with the tool, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip (https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip)
Just run it like "java -jar PinkDotRemover.jar YourFile.dng". If you're lucky, the program creates a _YourFile.dng with removed dots. Files will be overwritten without confirmation. Currently the tool is just a quick hack without focus on security or stability... use it at you own risk and cross your fingers... :)
As you can see above, there are still a very few dots left which the tool doesn't remove yet. But that's only a matter of fine tuning.
I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.
Here are a few sample DNGs to compare the original with the modified image:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_before.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M3_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_before.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M4_after.dng (DNG for the JPG above)
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/M5_after.dng
Let me know what you think... if you think it's useful, I could extend the tool to batch-convert all DNGs in a directory... or to handle resolutions other than 1280x720...
You should do a batch tool :)
Quote from: a1ex on June 15, 2013, 01:18:02 PM
Can you find the dot locations? I believe their position is fixed (maybe they change with resolution), but if you know where they are, you can just set them as 0 (bad pixel mark) in raw2dng.
It seems you have two kinds of distorted pixels: first the ones that show up in the center in light areas and second a lot more all over the image in dark areas. For the light ones, I manually picked the start and end coordinates of each "dot grid block" and stored them in my program to iterate over them. For the dark ones, I recorded a completely dark movie frame, pre-processed it a little bit (basically I applied a threshold to suppress noise) and got this one:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/DarkDots.jpg)
I extracted the coordinates of each non-black pixel with a little script and stored them in a text file. During the dot removal procedure I read this file and fix the dots at the stored locations. The coordinate file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/pixCoord_threshold2068.txt
If you think it's useful I could easily create a combined file with the "light" and the "dark" coordinates. Could you integrate that with raw2dng?
And, as I said: so far I only have the data for 1280x720. Simply scaling them mathematically to other resolutions is unlikely to work well, because you would get rounding errors for non-integer scaling factors. And if you're just one pixel off with the scaled coordinate, the dot removal doesn't work anymore...
Quote from: 1ricca on June 15, 2013, 01:06:43 PM
There's already a superior way than either of these for now. In Lightroom or ACR, [...]
You shouldn't forget that not everyone runs Adobe software.
Quote from: foorgol on June 15, 2013, 05:00:54 PM
It seems you have two kinds of distorted pixels: first the ones that show up in the center in light areas and second a lot more all over the image in dark areas. For the light ones, I manually picked the start and end coordinates of each "dot grid block" and stored them in my program to iterate over them. For the dark ones, I recorded a completely dark movie frame, pre-processed it a little bit (basically I applied a threshold to suppress noise) and got this one:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/DarkDots.jpg)
I extracted the coordinates of each non-black pixel with a little script and stored them in a text file. During the dot removal procedure I read this file and fix the dots at the stored locations. The coordinate file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/pixCoord_threshold2068.txt
If you think it's useful I could easily create a combined file with the "light" and the "dark" coordinates. Could you integrate that with raw2dng?
And, as I said: so far I only have the data for 1280x720. Simply scaling them mathematically to other resolutions is unlikely to work well, because you would get rounding errors for non-integer scaling factors. And if you're just one pixel off with the scaled coordinate, the dot removal doesn't work anymore...
What you are doing is incredible!!!!Thanks so much
Quote from: foorgol on June 15, 2013, 08:51:57 AM
I'll clean up the code tomorrow and upload the sources as well. If you're (rightfully!) hesitant to execute a stranger's JAR-files on your machine, you can compile the JAR yourself by then.
Ok, here's a new version! You can now pass a directory name as a command line argument and all files ending in "dng" or "DNG" in this directory will be batch-processed. Or you pass multiple file names as arguments. Or mix it. As you like.
The JAR-file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.zip
Just unzip it into a separate folder and execute the JAR file with
java -jar PinkDotRemover.jar <TheFilesAndDirectoriesYouWantToProcess.dng>The source code for the main application:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.src.zip
And the source code for a primitive TIFF-lib to access the DNG-files:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__001.src.zip
You need the lib in order to compile the main application. The source ZIPs contain complete NetBeans-Project trees. If you use a different IDE, just pick the .java-files from the ZIPs. There are no other libs or dependencies for this application.
Please let me know if something doesn't work for you. I've never deployed a java application outside of my little development pond, so maybe I missed something... :)
And: it's still 1280 x 720 only
Quote from: foorgol on June 16, 2013, 05:50:19 PM
Ok, here's a new version! You can now pass a directory name as a command line argument and all files ending in "dng" or "DNG" in this directory will be batch-processed. Or you pass multiple file names as arguments. Or mix it. As you like.
The JAR-file is here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.zip
Just unzip it into a separate folder and execute the JAR file with
java -jar PinkDotRemover.jar <TheFilesAndDirectoriesYouWantToProcess.dng>
The source code for the main application:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__001.src.zip
And the source code for a primitive TIFF-lib to access the DNG-files:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__001.src.zip
You need the lib in order to compile the main application. The source ZIPs contain complete NetBeans-Project trees. If you use a different IDE, just pick the .java-files from the ZIPs. There are no other libs or dependencies for this application.
Please let me know if something doesn't work for you. I've never deployed a java application outside of my little development pond, so maybe I missed something... :)
And: it's still 1280 x 720 only
Sweet! Thanks but I am not sure if it works for me.
I do this:
1. open terminal (using mac)
2. type "cd <and the way to my folder with the jar>
3. type in "java -jar PinkDotRemover.jar 000000.dng"
4. I get this text "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/nodomain/volkerk/PinkDotRemover/PinkDotRemoverMain : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)"
Quote from: Aborgh on June 16, 2013, 06:07:00 PM
I get this text "Exception in thread "main" java.lang.UnsupportedClassVersionError: org/nodomain/volkerk/PinkDotRemover/PinkDotRemoverMain : Unsupported major.minor version 51.0
Hmmm... could it be you're running a not-so-much-up-to-date version of java? :)
The program requires Java 1.7; in Java's weird logic, that the "51.0" in the error message you posted.
Try to run "
java -version" to check the java version you're using.
I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....
Quote from: foorgol on June 16, 2013, 07:59:57 PM
Hmmm... could it be you're running a not-so-much-up-to-date version of java? :)
The program requires Java 1.7; in Java's weird logic, that the "51.0" in the error message you posted.
Try to run "java -version" to check the java version you're using.
I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....
I have java 7 update 21 , so the latest but in terminal it shows 1.6 :S
Tried it in windows and it worked...
Quote from: foorgol on June 16, 2013, 07:59:57 PM
I'll create a new version that runs with the older Java 1.6 later, maybe today. During more testing, found a few ugly pixel distortions in certain regions on certain images, so that I need to create an improved version anyway....
Well then, try this one:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.zip
* Works with Java 1.5 or later
* Improved dot positions -- I had a few gaps here and there
* Better interpolation of defective pixels along edges -- applying a gradient-based algorithm now instead of plain, static averaging
Enjoy!
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__002.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.src.zip
Quote from: foorgol on June 16, 2013, 09:38:56 PM
Well then, try this one:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.zip
* Works with Java 1.5 or later
* Improved dot positions -- I had a few gaps here and there
* Better interpolation of defective pixels along edges -- applying a gradient-based algorithm now instead of plain, static averaging
Enjoy!
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__002.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__002.src.zip
The result is great but some dots still remains in the shadows but its good enough
Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SilentDNG_before.jpg)
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SilentDNG_after.jpg)
Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip
By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...
Pretty sweet @foorgol. I would love to try it but, yeah... No, I can't. I don't know how to play with java. I feel like dead weight. The only thing I can do is to wave my hands in the air and send you guys positive energy ???. Would it be possible to run this using raw2dng?
By the way @Sandsnor. You mentioned that your camera turned off because the battery died, and then you camera died as well.
It happens That my camera turned off automatically. It wouldn't turn on until I used other battery with the memory still inside.
What is ironic, is that the camera was telling me that the battery was full before it turned off, but it wasn't. What I wanted to say is that the code is doing funny things, but i'm sure your camera is alright.
Quote from: foorgol on June 18, 2013, 03:34:31 AM
Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SilentDNG_before.jpg)
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SilentDNG_after.jpg)
Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip
By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...
Foorgol I have the latest java installed on my system. So when I go on CMD and CD to the pindotremover folder when I Write the code you mentioned above it says "bad command, unreconisable"
Quote from: foorgol on June 18, 2013, 03:34:31 AM
Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:
\\img
\\img
Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip
By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...
Hi foorgol, i get this error
(http://imageshack.us/a/img259/5673/ub8.png)
what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"
Quote from: foorgol on June 18, 2013, 03:34:31 AM
Ok guys, time to bother you with a new version of the pink dot remover! This time I added the dot locations for the silent DNGs. So now the program can process both, 1280x720 and 1734x965. Here's a "before" and "after" example of a silent DNG with identical RAW development settings and corrected aspect ratio:
\img
\img
Here's the binary, just unzip it and run java -jar PinkDotRemover.jar <YourFileNameOrDirectory> in the shell:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.zip
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__003.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__003.src.zip
By the way: I was wrong in one of my previous posts; the programm definitely requires Java 1.7... sorry...
thanks so much for your work on this, it works very well, much better than Rawtherapee, although that works relatively well, but still leaves some dots.
If you do use Rawtherapee, use version 3, version 4 kept crashing on me with Win7
It took me awhile to find out I needed to set my environmental variable path so that Windows could locate the java command, but now it works well.
Watch at 720 HD, dots are most noticeable
For Sandsnor,
If you don't have a 2nd battery, I would go to Best Buy or somewhere and get one, that would rule out
that possibility. Radio Shack has them for $22. Lenmar LP-E8
Quote from: COMMANDES on June 18, 2013, 03:59:24 PM
Hi foorgol, i get this error
(http://imageshack.us/a/img259/5673/ub8.png)
what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"
exact same problem here
Quote from: COMMANDES on June 18, 2013, 03:59:24 PM
Hi foorgol, i get this error
(http://imageshack.us/a/img259/5673/ub8.png)
what am I doing wrong? I have java 7 upd 21
For shooting mode used "silent picture"
Quote from: demetrisag on June 18, 2013, 09:19:41 PM
exact same problem here
Hmmm... weird... this is most likely an error around reading / writing files. As I am a lazy guy, I don't check for these errors, because I simply assume that I have always the rights and space to read and write :)
On COMMANDES' screenshot I can also see that you're using special characters in your path name. Maybe that's part of the problem.
So could you please try again and...
* use path and filenames without special characters. Choose something simple like "C:\temp". No Cyrillic characters or similar, please. Just plain ASCII.
* make sure that you have permission to read and write in this directory
If this does not solve the problem, please let me know. I would then send you a special debug version with excessive logging to trace down the issue.
Quote from: Edgar Matos on June 18, 2013, 05:00:20 AM
I would love to try it but, yeah... No, I can't. I don't know how to play with java. I feel like dead weight.
*sigh*
That sounds to me as if you're not yet fully in love with all the amenities of the command line. :)
Well, I also got PMs asking for a GUI. Give me a few days and I'll try to fit the program with a simple GUI, e. g. a file selection dialog. If Java is correctly installed on your machine, a double-click on the jar-file should be sufficient to fire up the GUI.
Next they'll be asking for iOS themed GUI... :P
</Applerant>
Quote from: foorgol on June 19, 2013, 03:52:08 AM
Hmmm... weird... this is most likely an error around reading / writing files. As I am a lazy guy, I don't check for these errors, because I simply assume that I have always the rights and space to read and write :)
On COMMANDES' screenshot I can also see that you're using special characters in your path name. Maybe that's part of the problem.
So could you please try again and...
* use path and filenames without special characters. Choose something simple like "C:\temp". No Cyrillic characters or similar, please. Just plain ASCII.
* make sure that you have permission to read and write in this directory
If this does not solve the problem, please let me know. I would then send you a special debug version with excessive logging to trace down the issue.
So ... Again, this error ... Sad ...
(http://imageshack.us/a/img834/7078/fwka.png)
Quote from: COMMANDES on June 19, 2013, 06:03:03 PM
So ... Again, this error ... Sad ...
(http://imageshack.us/a/img834/7078/fwka.png)
For all of you who got that error: could you please download this version
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
and send me the output via PM?
Please maximize the terminal window, because the debug texts can become quite long...
Thanks a lot!
Quote from: foorgol on June 20, 2013, 03:51:25 AM
For all of you who got that error: could you please download this version
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
and send me the output via PM?
Please maximize the terminal window, because the debug texts can become quite long...
Thanks a lot!
There you go mate, I only skipped one line at the very top, which was the command in order to include it all in one screenshot
(http://i.imgur.com/OqqoHM4.jpg)
Thanks a lot for doing that!
And satriani, you are an angel!
I have moved this topic out of the 650D alpha topic. http://www.magiclantern.fm/forum/index.php?topic=6320.250 (http://www.magiclantern.fm/forum/index.php?topic=6320.250)
This should provide a better overview for this nice tool.
Kind regards
Blade
Quote from: demetrisag on June 20, 2013, 10:00:46 AM
There you go mate, I only skipped one line at the very top, which was the command in order to include it all in one screenshot
Thanks for the data!
Just by quickly looking at the log messages, I realized that I might have made a very stupid beginner's mistake: I blindly assumed that all picture file names always have a path component, i. e. "/this/is/your/picture.dng" and not just "picture.dng".
I'll fix it tonight, should be a no-brainer.
In the meantime you can try to call the program like
java -jar PinkDotRemover.jar .\5.dng or
java -jar PinkDotRemover.jar ./5.dng (depending on your OS)
or simply have pictures and tool in different directories. If my assumption above is correct (haven't looked at the code yet), this should fix your issues for the time being.
Quote from: foorgol on June 20, 2013, 02:08:46 PM
Thanks for the data!
Just by quickly looking at the log messages, I realized that I might have made a very stupid beginner's mistake: I blindly assumed that all picture file names always have a path component, i. e. "/this/is/your/picture.dng" and not just "picture.dng".
I'll fix it tonight, should be a no-brainer.
In the meantime you can try to call the program like
java -jar PinkDotRemover.jar .\5.dng or java -jar PinkDotRemover.jar ./5.dng (depending on your OS)
or simply have pictures and tool in different directories. If my assumption above is correct (haven't looked at the code yet), this should fix your issues for the time being.
what worked for me was to create a folder, then a subfolder and from the folder to use this syntax:
java -jar pinkdotremover.jar subfoldername
then all .dngs in the subfolder would get converted
For setting the system path variable, select the start icon, then computer, system properties, advanced system settings,
environmental variables. In the system variables window, scroll down to path and edit it. At the end of the variable, add
this
;C:\Program Files (x86)\Java\jre7\bin
then keep selecting ok until you back out of that section
of course, that's for Win7, 64bit. If you have a different OS, then append the variable with wherever your java.exe is located.
This should be built into chdk-dng.c... I just dunno how to mark the pixels exactly.
650D + eos M are different raw sizes tho.
Quote from: 1% on June 20, 2013, 02:47:13 PM
This should be built into chdk-dng.c... I just dunno how to mark the pixels exactly.
650D + eos M are different raw sizes tho.
I should mention that I output at 1280 x 720 and all the pink pixels seem to get eliminated.
On my canon settings I chose the 1920 x 1080 (24fps) before going into the ML and choosing
the 1280 x 720. That was the only way I could get that size at a data rate that the card can handle.
I only get 1 size in LV and one size in zoom except when recording... the default LV is like 50D, stuck at low res. So only need 2 sizes.. actually 1 size.. no dots in zoom mode.
Quote from: 1% on June 20, 2013, 03:04:38 PM
I only get 1 size in LV and one size in zoom except when recording... the default LV is like 50D, stuck at low res. So only need 2 sizes.. actually 1 size.. no dots in zoom mode.
Sorry 1%, I am definitely in over my head at this point :( I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.
Quote from: donjames150 on June 20, 2013, 03:13:50 PM
Sorry 1%, I am definitely in over my head at this point :( I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.
Note the monitor with "Hello World" on it at 2:26
Alright, here's a new build. What has changed:
* Fixed the crash when using file names without path (see previous postings)
* Added the option to simply mark the pink dots as "Dead Pixel" which then leaves the actual color correction to your RAW tool
* Due to popular request: added a simple GUI :)
Here's a screenshot:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/gui.jpg)
Should be pretty self-explanatory. Throw in whatever files you find (e. g. by selecting whole directories). The tool should be pretty tolerant against unrecognized or unprocessable files. Try your luck. :)
If you want to, you can still use the tool on the command line or in scripts, just like before. The GUI is only started when you provide no or only invalid command line arguments.
I assume that now with the GUI a simple double-click on the JAR-file is sufficient if Java is properly installed on your normal (Windows-) working computer. However, I couldn't try it because I only have a (Linux) development computer here, which behaves a bit different.
Download the binary here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.zip
I hope the ZIP has everthing you need to run the GUI on a normal machine. Let me know if you face troubles.
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__004.src.zip
For those of you interesting in the source code:
The code is pretty crammed with debug messages now which makes it hard to read. I'll remove all the messages once this version proves to be more or less stable. And the GUI stuff lacks comments... you'll have to figure out yourself what's happening there... :P
Enjoy!
Quote from: donjames150 on June 20, 2013, 03:13:50 PM
I think we need Foorgol to check in about now. It does seem that if his program can
locate and zap those dots, that it could be incorporated into ML, which would be really great.
Sorry, I can't help you with LV or similar. I have absolutely no clue what happens inside ML and how data is converted back and forth inside ML before being dumped to the card. I'm not an ML developer.
But regarding your idea to integrate the dot removal into ML:
From an architectural point of view, I would not do that. Writing the RAW data in higher resolutions is already time critical enough today (okay, mostly limited by writing speed, AFAIK). So I would push as much data processing as possible out of ML and up to your normal workstations with practically unlimited processing power, time and disk space. You have to touch all the image files anyway for post-processing, so adding the dot removal is not so much of a pain here.
Dot removal inside ML would only make sense if the built-in camera processors are extremely optimized for these kind of tasks (what's not unlikely) and/or if you have additional information inside the camera at runtime which would support/optimize the removal process but which can't be exported to the outside world.
From all I know today, I would rather strive for an integration of the removal process into raw2dng....
Oh, I just found out that taking silent DNGs in movie mode gives you nice 1726x1150 images with correct aspect ratio and better exposure control. Guess which pink dot locations I'll tackle next... :)
Quote from: foorgol on June 21, 2013, 04:23:40 AM
Oh, I just found out that taking silent DNGs in movie mode gives you nice 1726x1150 images with correct aspect ratio and better exposure control. Guess which pink dot locations I'll tackle next... :)
Thank you for the tool foorgol :D
I guess 2.35:1? :D
Finally it works! thanks for this a lot Foorgol! Though I noticed, that dots are being removed very very well at the center of the image whereas at the two sides many dots are still here. I guess you have not tackled those yet?
Thanks again!
Rawtherapiee program did a good job, but some pink dots are still visible. They are annoying as hell.
after using Rawtherapiee
http://oi43.tinypic.com/2mi1grp.jpg
Quote from: midnite on June 21, 2013, 10:46:09 PM
Rawtherapiee program did a good job, but some pink dots are still visible. They are annoying as hell.
after using Rawtherapiee
http://oi43.tinypic.com/2mi1grp.jpg
I got this using the pinkdot removal tool
http://kozenrufu.com/_000116.jpg
First I set my canon movie record size to 1920 x 1080 (24fps)
Then I set ML to 1280 x 720, Aspect 16:9, Framing Center
It had all the usual pink dots before I ran the removal tool
Here's the video before and after. Watch at 720p to clearly see all the pink dots
Could you add support to 1344x572, and also 1472x626 resolutions? Thanks.
And for 1280x544. I have only 45mb/s sd card, and 1280x720 is stopped after 8 seconds of recording, but on 1280x544 - no stops!
Very thanks!
the 1808x727 silentdng data can be used for all unzoomed video resolutions. the smaller resolutions are just crops. therefore the app just has to calculate the correct offset for other resolutions. as soon as that feature is supported all resolutions should work. so there is no need to ask for any specific resolution.
Hi! THis looks like a real nice program. Is it possible to have it in rawtodng tool? Ocassionally I get some annoying dead pixels showing and I don,t go through adobe acr but straight to proRes frpm dng with the rawtodng tool.
thanks
//D
Quote from: mixer2 on June 23, 2013, 01:58:32 PM
the 1808x727 silentdng data can be used for all unzoomed video resolutions. the smaller resolutions are just crops. therefore the app just has to calculate the correct offset for other resolutions.
This is fantastic news! This would save me/us a lot of work, because it's quite tedious to record the pixel locations for new or other resolutions. If instead I would only need to determine a x,y-offset, adding a new resolution would only be a matter of seconds!
However, there is one thing that doesn't support your assumptions:
The grid-like dots in the picture's center have a distance of 10 pix between two grid rows in 1280x720 while it's 12 pix between two rows in 1808x727. If 1280x720 would just be a crop of 1808x727, the row offset should be identical.
Any ideas why the row distance changes? Or is 1280x720 a "scaled" resolution?
there are two different raw streams. normal and while LV is zoomed in. so all other cases should be crops of one of those 2 streams. using the same data for other resolutions is working on eos-m. have to check the 650d images to see what's the problem, but ml doesn't scale anything, just cropping.
Quote from: Danne on June 23, 2013, 02:24:23 PM
Hi! THis looks like a real nice program. Is it possible to have it in rawtodng tool? Ocassionally I get some annoying dead pixels showing and I don,t go through adobe acr but straight to proRes frpm dng with the rawtodng tool.
thanks
//D
you can use a script to generate dng with rawtodng first and then automatically process it with the dotremover. will that work for you? which os do you use?
i think it will take some time, until there is a single program solution, since the dot problem is specific for some camera models (650d, eos-m and i think 700d+100d too).
Sounds nice. I,m on a mac 10.6.8 I think. I have a 5d mark 3. The problems occur in high iso and dark areas. Still it,s only when outside acr when I want to export to ProRes
hurm, on 5d mark 3 the type of dots, which this tool tries to remove, doesn't exist.
sure that it's not a kind of noise that you get? are the dots on the same position all the time?
what & where is the pink spot removal tool?
Quote from: mixer2 on June 24, 2013, 04:09:43 PM
hurm, on 5d mark 3 the type of dots, which this tool tries to remove, doesn't exist.
sure that it's not a kind of noise that you get? are the dots on the same position all the time?
Yeah, they,re hot/dead pixels. Maybe not so good with this particular tool?
Quote from: dariSSight#1 on June 24, 2013, 04:17:45 PM
what & where is the pink spot removal tool?
Read from page 1 (only 2 pages) . Basically it's for the 650D which gets clusters of pink pixels when shooting raw video. This tool removes them, but only
for a 1280 x 720 crop selection in raw.
Quote from: Danne on June 24, 2013, 04:47:56 PM
Yeah, they,re hot/dead pixels. Maybe not so good with this particular tool?
no, sorry, that tool isn't useful for that case. the dead pixel detection of the raw editor or your choice should solve that problem.
Aight, thanks
Quote from: mixer2 on June 24, 2013, 06:46:23 PM
no, sorry, that tool isn't useful for that case. the dead pixel detection of the raw editor or your choice should solve that problem.
Both do the best job so far. But foorgol from what I know is still wokring on it, so I believe if foorgol finish it it will be the best thing on its own
Okay guys, short status update:
* I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions
* Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version
* Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well
Give me two or three days and by then I should have a new release out for testing....
Quote from: foorgol on June 25, 2013, 12:56:27 AM
Okay guys, short status update:
* I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions
* Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version
* Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well
Give me two or three days and by then I should have a new release out for testing....
Foorgol, this tool is simply awesome. I think you need an official name for it now that it's up and working well. ;)
Quote from: foorgol on June 25, 2013, 12:56:27 AM
Okay guys, short status update:
* I'm currently modifying the tool to support dot locations stored in external files (not hardcoded in the tool anymore) and more cam types and resolutions
* Mixer2 already patched the code and made it usable for the EOS M as well... I'll integrate that into the new version
* Mixer2 also found out that you can handle many resolutions by just providing an offset to a few basic resolutions (e.g., the silent DNG). I'll provide support for that as well
Give me two or three days and by then I should have a new release out for testing....
thank foorgol, I've been wanting to play around with other sizes than the 1280 x 720
Thanks for useful tool!
Can you add option for overwrite old files with dots?
Thanks again!
Quote from: foorgol on June 21, 2013, 03:43:46 AM
Alright, here's a new build. What has changed:
* Fixed the crash when using file names without path (see previous postings)
* Added the option to simply mark the pink dots as "Dead Pixel" which then leaves the actual color correction to your RAW tool
* Due to popular request: added a simple GUI :)
Here's a screenshot:
(https://dl.dropboxusercontent.com/u/22843507/MagicLantern/gui.jpg)
Should be pretty self-explanatory. Throw in whatever files you find (e. g. by selecting whole directories). The tool should be pretty tolerant against unrecognized or unprocessable files. Try your luck. :)
If you want to, you can still use the tool on the command line or in scripts, just like before. The GUI is only started when you provide no or only invalid command line arguments.
I assume that now with the GUI a simple double-click on the JAR-file is sufficient if Java is properly installed on your normal (Windows-) working computer. However, I couldn't try it because I only have a (Linux) development computer here, which behaves a bit different.
Download the binary here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.zip
I hope the ZIP has everthing you need to run the GUI on a normal machine. Let me know if you face troubles.
Source code:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__004.src.zip
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/SimpleTIFFlib__004.src.zip
For those of you interesting in the source code:
The code is pretty crammed with debug messages now which makes it hard to read. I'll remove all the messages once this version proves to be more or less stable. And the GUI stuff lacks comments... you'll have to figure out yourself what's happening there... :P
Enjoy!
yes, on Windows machines, it opens with a double-click of the PinkDotRemover.jar file. Thanks!
Well then, here it is.
As announced, the program is now able to handle multiple camera types and you can easily add new resolutions for dot removal. This build ships with support for the following 650D image types:
* Silent DNGs taken in photo mode
* Silent DNGs taken in video mode
* RAW video frames with 1152, 1280, 1344, 1472, 1600 and 1728 pixels width and an aspect ratio of 16:9, 1.85:1, 2:1, 2.2:1, 2.35:1, 2.39:1, 2.5:1, 2.67:1 or 3:1
Beware: I have only verified the silent DNGs, 16:9 and 2.35:1 with real footage (except for 1152 width). The pink dot locations for all other modes are only calculated and not verified. Due to rounding, they sometime might be off by one pixel and then the whole dot removal doesn't work anymore.
If you've used one of these non-verified modes and it proved to work, please drop me a line. Then we can tick that one off as "verified".
Here's the binary:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__006.zip
In the GUI, you'll notice a new drop-down box for the selection of the camera model. The default is 650D. If you're using the tool on the command line, you can only process 650D images -- there are no means to choose the cam type on the command line.
The database is not yet filled with real EOS M data yet, but that should come soon.
Since I coded all the new functionality quite in a rush, the code quality suffered a bit. I did not much testing and even less commenting. The "Happy Path" through the program should work quite alright, though. Let me know if you encounter problems.
The source code is now hosted on GitHub:
https://github.com/Foorgol/PinkDotRemover
https://github.com/Foorgol/SimpleTIFFlib
https://github.com/Foorgol/LoggingLib
Enjoy!
Quote from: dimadan on June 25, 2013, 10:38:13 AM
Can you add option for overwrite old files with dots?
Nope. I will not overwrite your files, not even as an option which you have to activate first. I feel bad enough by blindly creating / overwriting the _xxxxxxx.dng files without confirmation.
The risk is too high that you loose precious data, e. g. due to a tool malfunction.
If you want to delete the old files, please do so in your file manager or on the command line. It's just a few clicks or a few keystrokes and the risk is significantly lower....
Foorgol, I'm getting an error when I double-click the .jar file
(http://kozenrufu.com/Capture.JPG)
After processing the files with the dotremover the dots are weak visible.
When I push the picture 1,5 Av the dots are again very good visible.
I shoot 1280*720 16:9 with an EOS650D
Is it normal?
Quote from: spider on June 26, 2013, 04:01:37 PM
After processing the files with the dotremover the dots are weak visible.
When I push the picture 1,5 Av the dots are again very good visible.
I shoot 1280*720 16:9 with an EOS650D
Is it normal?
I shot this the other day at 1280 x 720, before and after .dng files
(http://kozenrufu.com/Capture1.JPG)
(http://kozenrufu.com/Capture2.jpg)
after: https://www.dropbox.com/s/3jk2yuiip3wauml/_000000.jpg
before: https://www.dropbox.com/s/qjj9sz8vmjrcdgw/000000.jpg
Two in LR4 cropped 1280*720 24p footage pictures pushed 2Av from Canon 650D
You see some dots are completely removed, some dots are still there and some dots are a bit removed.
Here the two frames without crop
https://www.dropbox.com/s/nmc2r3yradmm964/_000000f.jpg
https://www.dropbox.com/s/21d248ei77w1sa1/000000f.jpg
Hey, I'm having a bit of an issue.
I shoot at 2.35:1 with 1472 width. (this is the max I can get with the t4i) I ran the program using "650D" and "Interpolate" but almost all dots are still visible. And when I attempt to use "dead pixel" it doesn't convert, simply saying "one or more files could not be converted"
Would like some help here.
Quote from: donjames150 on June 26, 2013, 03:56:46 PM
Foorgol, I'm getting an error when I double-click the .jar file
(http://kozenrufu.com/Capture.JPG)
same here
Quote from: demetrisag on June 26, 2013, 09:38:38 PM
same here
Since Foorgol was clearly born to program and uses a Linux OS, I suggest he do what I did in 1989
when I took a class on Unix. I put both Linux and Dos on two different partitions and had a dual-boot setup.
Then you could test on Windows machines which is what most people have (except the rich ones who have
macs :o)
Okay, one by one....
--------------------------------------------------
@donjames150 and @demetrisag:
Could you please run the program in a terminal window (instead of just double-clicking the jar) and send me the output? I have a faint idea what the reason for error could be. This idea would be confirmed if the program would run normal if you start it from the command line...
--------------------------------------------------
--------------------------------------------------
@spider:
There are two kinds of dots. The "block" in the center and the "strip" all across the image. It's characteristic for the strip-dots to only show up in dark areas or when you lighten the image.
Your issue has to with the way the position of the strip-dots are located. Currently it's a trade-off between removing fewer of these pixels and losing image details by accidentally removing too many non-pink dots . Mixer2 found an interesting approach to solve this which he describes here:
http://www.magiclantern.fm/forum/index.php?topic=3648.msg54162#msg54162
I'm working with Mixer2 on an improved "dot location determination" for these dots which will come with the next release.
Until then, you can try to edit the file 650D_Video_1280x720.txt which you find in the dotData directory. The 4th line ends with "2068". You can try to gently reduce this value, e. g. to 2050 or 2040. This should then remove more of these dots.
--------------------------------------------------
--------------------------------------------------
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
--------------------------------------------------
Quote from: foorgol on June 27, 2013, 01:17:39 AM
--------------------------------------------------
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
--------------------------------------------------
I'm not quite sure where that setting is. Is it an ML thing or a canon thing?
Quote from: foorgol on June 27, 2013, 01:17:39 AM
Okay, one by one....
--------------------------------------------------
@donjames150 and @demetrisag:
Could you please run the program in a terminal window (instead of just double-clicking the jar) and send me the output? I have a faint idea what the reason for error could be. This idea would be confirmed if the program would run normal if you start it from the command line...----------------------------------------
looks like this
(http://kozenrufu.com/Capture3.JPG)
(http://kozenrufu.com/Capture4.JPG)
Quote from: papkee on June 27, 2013, 01:21:05 AM
I'm not quite sure where that setting is. Is it an ML thing or a canon thing?
framing is in the submenu under raw video in ML
Quote from: donjames150 on June 27, 2013, 01:33:15 AM
looks like this
Okay, my assumption was that the error has to do with the determination of the JAR's directory. I thought it would fail completely. But instead, it's just a back-and-forth conversion issue, where a space " " finally ends up as %20 like in an URL. So my guess was close :)
Should be easy to fix. For the time being, just avoid spaces in the directory name.
You always catch my sloppy ways to handle paths.... :)
on the other hand it foorgol it worked with me on CMD
Quote from: foorgol on June 27, 2013, 01:17:39 AM
I'm working with Mixer2 on an improved "dot location determination" for these dots which will come with the next release.
Thank you very much, this sounds great.
QuoteUntil then, you can try to edit the file 650D_Video_1280x720.txt which you find in the dotData directory. The 4th line ends with "2068". You can try to gently reduce this value, e. g. to 2050 or 2040. This should then remove more of these dots.
I tried it, but the improvement is not so big an I cant set smaller values than 1058 if I reduce the value more the program sometimes freeze.
Quote from: donjames150 on June 27, 2013, 01:52:32 AM
framing is in the submenu under raw video in ML
It's not there on mine. RAW Video only displays Resolution, Aspect Ratio, Preview, Digital Dolly, Frame Skipping and Playback
Framing was removed because you dont got any speed improvement from it.
Quote from: spider on June 27, 2013, 04:32:03 PM
Framing was removed because you dont got any speed improvement from it.
Quote from: foorgol on June 27, 2013, 01:17:39 AM
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
There seems to be some conflicting messages here.
No conflict, in the past there was a setting called framing.
Now it is gone https://bitbucket.org/hudson/magic-lantern/commits/145e69e340eb9a868f29eda8f0acae0a4fb6cbc3
Only the dolly mode which was a part of framing is still there.
Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
@papkee:
Does this version solve your issue with pixels not being properly removed @ 1472x626?
@donjames150:
Does this version work with spaces in the directory name for you?
@demetrisag:
If you're lucky this version could solve your database-not-found issue as well... give it a shot, please...
Quote from: foorgol on June 28, 2013, 06:35:02 AM
Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
@donjames150:
Does this version work with spaces in the directory name for you?
So far it looks like it's working fine, I'll be testing it today at different crop sizes. Again, thanks
for all your hard work on this.
This is completely raw after pink dot removal, no other corrections.
1728 x 972 at 12 FPS
ISO 200 F2.8 1/30
(http://kozenrufu.com/raw.jpg)
Hi there,
Thank you for the great job :)
I've got a problem with the tool (I'm certainly doing something wrong :-[).
Here's the link for an original frame:
https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng (https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng)
The resolution of the raw video was set at 1472x626 (2.35:1), shot with the ML build from 2013-06-26.
When I use the tool (on W7), I get this: https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng (https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng)
What am I doing wrong?
Quote from: foorgol on June 28, 2013, 06:35:02 AM
Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
@papkee:
Does this version solve your issue with pixels not being properly removed @ 1472x626?
@donjames150:
Does this version work with spaces in the directory name for you?
@demetrisag:
If you're lucky this version could solve your database-not-found issue as well... give it a shot, please...
Extremely well mate! you are a legend!
Quote from: talosectos on June 28, 2013, 08:21:19 PM
Hi there,
Thank you for the great job :)
I've got a problem with the tool (I'm certainly doing something wrong :-[).
Here's the link for an original frame:
https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng (https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng)
The resolution of the raw video was set at 1472x626 (2.35:1), shot with the ML build from 2013-06-26.
When I use the tool (on W7), I get this: https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng (https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng)
What am I doing wrong?
I just took a short clip at that resolution and it removed the dots for me.
Did you maybe select EOSM by mistake?Edit: But then when I tried it on YOUR file, it did what it did for you, made it 10 times worse.
but if I open your original in Rawtherapee and apply the dead pixels profile, it acts normally (for that program)
(http://kozenrufu.com/fx.jpg)
Then I installed the same build as yours (which is wow, btw) and again took a short clip and the tool removed the dots.
This is my setup in RAW: Digital dolly OFF, Frame skipping OFF. That's all I got for ideas for now :-\
Quote from: talosectos on June 28, 2013, 08:21:19 PM
What am I doing wrong?
This looks very, very strange. And honestly I have no idea right now how this happened. But I'm also not sober enough right now to look deeper into it... :)
The only thing I noticed at a first glance is that your original image contains weird meta data. Looking at the tags model, manufacturer, software, bits per pixel and DNG version, I would expect "Canikon", "Canon", "Magic Lantern", 14 and "1.3.0.0", whereas your file reports nothing, nothing, "Rarevision RAWMagic 1.0", 16 and "1.1.0.0".
What's most striking is the difference between the "normal" 14 bit color depth and the 16 bit in your file.
So, frankly speaking, your source file doesn't seem to come straight from a 650D. Is that true?
My program is somehow tailored to the 650D-files. Maybe this is part of the problem.
Quote from: demetrisag on June 28, 2013, 08:43:57 PM
Extremely well mate! you are a legend!
Thanks a lot, but keep it low, my friend :)
This just an interesting image processing job, done in a well defined Java environment with virtually unlimited resources like memory, time and processing power. This is nothing compared to the embedded, reverse-engineering development that the real ML developers do...
Anyway, I'm glad that some of guys find that little tool useful... makes it even more fun to code :)
Quote from: donjames150 on June 28, 2013, 09:02:25 PM
I just took a short clip at that resolution and it removed the dots for me. Did you maybe select EOSM by mistake?
Edit: But then when I tried it on YOUR file, it did what it did for you, made it 10 times worse.
but if I open your original in Rawtherapee and apply the dead pixels profile, it acts normally (for that program)
Ok thanks for your answer :)
Quote from: foorgol on June 29, 2013, 05:38:28 AM
The only thing I noticed at a first glance is that your original image contains weird meta data. Looking at the tags model, manufacturer, software, bits per pixel and DNG version, I would expect "Canikon", "Canon", "Magic Lantern", 14 and "1.3.0.0", whereas your file reports nothing, nothing, "Rarevision RAWMagic 1.0", 16 and "1.1.0.0".
What's most striking is the difference between the "normal" 14 bit color depth and the 16 bit in your file.
So, frankly speaking, your source file doesn't seem to come straight from a 650D. Is that true?
My program is somehow tailored to the 650D-files. Maybe this is part of the problem.
Ok, I think that the problem comes from the tool I used to convert the raw (RawMagic (http://www.magiclantern.fm/forum/index.php?topic=6218.0)), this part was done on my Mac.
I'm going to try with another tool and let you know the result.
Edit:I've tried with raw2dng, and this time it works perfect.
I should have check that before, sorry :-\
Thanks foorgol!
Second editThe matter is that RAWMagic convert into CinemaDNG files (that can be used directly into Resolve) whereas raw2dng convert into photo-dng.
Some differences between the two formats that cause problems with the PinkDotRemove tool and CinemaDNG files.
Quote from: foorgol on June 29, 2013, 05:43:39 AM
Thanks a lot, but keep it low, my friend :)
This just an interesting image processing job, done in a well defined Java environment with virtually unlimited resources like memory, time and processing power. This is nothing compared to the embedded, reverse-engineering development that the real ML developers do...
Anyway, I'm glad that some of guys find that little tool useful... makes it even more fun to code :)
My friend, maybe technically on that matter you are not as deep as the ML guys, which of course you are right, but practicely you are a legend, you gave us a way to use raw even if thats maybe temporary, cause you never know! the other legends on the ML side might discover something! (they always do, dont they?) What am saying is that you give hope to this project just like the rest of the guys! and also indirectly helping the developers by taking our minds of the pink dots so leaving them developers do what they do best! :)
I tried compare interpolation, without PDR and dead pixel option.
(http://i43.tinypic.com/rtj9qx.jpg)
It's 1280x720. Maybe I have done something wrong, dont't know.
Quote from: multi.flexi on July 01, 2013, 06:03:15 PM
I tried compare interpolation, without PDR and dead pixel option.
From the pattern of the removed pixels I would guess that you used an old version of the tool. A new release is on its way and for the time being just use the debug version mentioned above:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
This should fix our issues.
Oh, and make sure you're using raw2dng to get the DNGs.
Quote from: demetrisag on June 29, 2013, 03:04:16 PM
And also indirectly helping the developers by taking our minds of the pink dots so leaving them developers do what they do best! :)
Hehe... you're right... I remember the discussions in the 650D-thread being quite emotional... about whether the developers should spend time on improving RAW or not... I guess there's some relief now on that front :)
Short update on the dot remover:
* Changed algorithms to support EOS M better
* Fine-tuning EOS-M dot location database
* First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed
Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!
Quote from: foorgol on July 02, 2013, 05:26:54 AM
Short update on the dot remover:
* Changed algorithms to support EOS M better
* Fine-tuning EOS-M dot location database
* First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed
Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!
thanks to both of you
Alright folks, I have a new video out showcasing RAW on the 650D, cleaned up with the remover.
Quote from: papkee on July 02, 2013, 07:46:14 PM
Alright folks, I have a new video out showcasing RAW on the 650D, cleaned up with the remover.
Nice video
Quote from: foorgol on July 02, 2013, 05:26:54 AM
Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!
I am so excited... thank you foorgol and Mixer2
Quick preview of what I could max out with 650D raw + pink dot removal, not perfect. Has some frame skipping.
Actors: dog.
Also note the first scene is with the latest pink dot removal therefore it has less noise or dots compared to the earlier build which is used in scene 2.
I will soon make low light timelapse video just to see how the dot removal tool handles lower light conditions. Shouldn't be much difference?
these two were taken one after the other with the same camera settings. Kit lens 18-55 at about 33mm iso100 and f22.
The H264 was 1920 x 1080 (24fps) and the raw was 1728 x 972 (10fps). Then output at 1280 x 720. I adjusted the colors
on the raw to match the h264 as best as I could and applied very minor stabilization to get rid of the jerkiness that I always
see in my raw videos (from the process I guess). You can clearly see better into the shadows with the raw.
Quote from: foorgol on July 02, 2013, 02:34:20 AM
From the pattern of the removed pixels I would guess that you used an old version of the tool. A new release is on its way and for the time being just use the debug version mentioned above:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip
This should fix our issues.
Oh, and make sure you're using raw2dng to get the DNGs.
Well if I use raw2dng and then PDR I can't edit it in Davinci Resolve. Is there a way how to get rid of pink mess and also convert it to CinemaDNG?
Quote from: multi.flexi on July 05, 2013, 10:05:16 AM
Well if I use raw2dng and then PDR I can't edit it in Davinci Resolve. Is there a way how to get rid of pink mess and also convert it to CinemaDNG?
Yes, just be patient. RawMagic is producing CinemaDNG. Mixer2 and foorgol are working to adapt the tool for it.
Quote from: foorgol on July 02, 2013, 05:26:54 AM
Short update on the dot remover:
* Changed algorithms to support EOS M better
* Fine-tuning EOS-M dot location database
* First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed
Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!
thank you so much for this tool!!
here is a test video i did as well as my process for getting the final image: https://vimeo.com/69773294
process:
shot RAW footage on my t4i w/ Magic Lantern.
settings 1280x720 at 23.976 - 23.978 (by default) on a 32GB SanDisk EXTREME PRO 95MB/s SD/HC class 10.
used raw2dng on my iMac to convert RAW to dng
used the pinkdotremover tool from the ML forum to remove pink dots
opened pinkdotremoved DNG's in PS
graded
exported images from PS as TIFF
opened TIFF sequence in Compressor
Exported Apple Pro Res file making sure frame rate was 23.976.
NOTES: Let's use a DNG folder with 5 frames in it as an example. When using pinkdotremover the DNG's come back as single files (if you have 5 files it will look like 5 files) but when you select the 5 files it will show that there are 10 files selected. When you open the pinkdotremover processed DNG's in PS it will have a sequence of 5 images with the pink dots removed followed by another 5 image sequence that still has the pink dots. Before exporting to TIFF delete these files from the sequence, or before exporting in Compressor make sure the TIFF files that still have the pink dots are deleted.
Guys, i'm not sure, but did you made any research about dot removal via simple color overlay?
I have tried to create couple layers with different overlay modes here:
(http://f.cl.ly/items/2V250n0u2t2f3Z3U2L3d/canon_eos_m_dots_overlay.gif)
Quote from: talosectos on July 05, 2013, 07:35:35 PM
Yes, just be patient. RawMagic is producing CinemaDNG. Mixer2 and foorgol are working to adapt the tool for it.
... and basically the current status is unchanged. THEORETICALLY the dots should be removed, but PRACTICALLY a few of them survive. I guess I'll release the new version anyway, because I need your feedback and a few more sample pictures to hunt down the bug...
Quote from: khurra on July 06, 2013, 01:00:55 AM
When using pinkdotremover the DNG's come back as single files (if you have 5 files it will look like 5 files) but when you select the 5 files it will show that there are 10 files selected.
The name of the image files created by the dot remover that with an underscore "_". Is it possible that an underscore indicates a hidden file on your system or in your file manager? Or does you file manager show "aaa.dng" and "_aaa.dng" only as one line, because it assumes that the underscore file is something like a backup file?
Okay, just got the GO! from Mixer2 not to wait for him and release a new version :)
(He's working hard to setup and test a lot of resolutions for the EOSM, so no blaming him here)
Well then, here it is:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__007.zip
What's new since version 006 (partially already available in the previously posted debug version):
* Fix for the wrong path handling that lead to a crash for some of you
* Initial support for EOSM
* Support for all 650D resolutions (see comment below)
* Faster image conversion due to optimized algorithms
* Faster start-up
* Initial support for the 16-bit DNGs generated by RawMagic
* When selecting a bunch of files or a directory, only DNG files will be added to the conversion list
Comment on the "all resolutions for 650D" feature:
We've found a way how to extrapolate a single set of dot locations to arbitrary image resolutions. And it works. Mostly. :)
During testing, we found one or two resolutions which do not fit into our extrapolation scheme.
What this means for you:
The tool will convert the files without complaining. You'll see no error message. But the resulting file might still contain all or at least some dots. In this case, please send me a message with the image resolution and I'll set it up as one the few "special cases" that need individual dot location definitions.
Tested and verified resolutions for the 650D are:
1280x720
1344x572
1344x756
1472x626
1472x828
1600x680
1600x900
1728x736
1728x972
1808x1190 (Silent DNGs Movie Mode)
1808x727 (Silent DNGs Picture Mode)
Source Code:
https://github.com/Foorgol/PinkDotRemover
Enjoy!
Quote from: pileman on July 04, 2013, 10:36:25 PM
I will soon make low light timelapse video just to see how the dot removal tool handles lower light conditions. Shouldn't be much difference?
Right. The tool now removes the dots all over the image, no matter what the lighting conditions are.
Let me give you a more detailed explanation:
In previous builds, I removed only the "light dots" in the center based on a fixed "grid definition" while the "border dots" were removed based on empirical/statistical data read from sample DNGs. Which was wrong.
Reason: Both "center dots" and "border dots" are expressions of the same regular grid of dots, just with different visibility.
Since the debug version (and officially since the 007 version) "center dots" and "border dots" are now handled alike: based on a fixed grid-like definition of dot locations. All over the image, from left to right. Always.
Theoretically there is one disadvantage: I also apply interpolation to border locations where you wouldn't see the dots in your particular image, because e. g. it's a very bright image. So I'm applying interpolation to locations although there's nothing wrong them. This could lead to a small loss of detail.
But
in practice I couldn't notice this disadvantage. And even if you could notice it: it would be surely outweighed by the disadvantage of having pink dots accidentally popping up because the removal wasn't thorough enough.
Thank you for the new version.
I have a question is it possible to output the files into a new folder?
So it would be easier to delete the old files.
Quote from: foorgol on July 07, 2013, 04:48:39 PM
... and basically the current status is unchanged. THEORETICALLY the dots should be removed, but PRACTICALLY a few of them survive. I guess I'll release the new version anyway, because I need your feedback and a few more sample pictures to hunt down the bug...
Thanks for this new version.
I made a few tests and it seems to work perfect with rawmagic.
I'll make more tries this evening after work.
Quote from: foorgol on July 07, 2013, 04:51:51 PM
The name of the image files created by the dot remover that with an underscore "_". Is it possible that an underscore indicates a hidden file on your system or in your file manager? Or does you file manager show "aaa.dng" and "_aaa.dng" only as one line, because it assumes that the underscore file is something like a backup file?
I realized that I was missing the underscore. All the files without the underscore are the originals. It doesn't completely overwrite the old files. THANK YOU so much for the tool. It is ABSOLUTELY amazing.
I wish I knew C better so I could help!!
Thank you for developing the PinkDotRemover - it works perfectly.
Now I can use my 650D to make beautiful 1280 x 720 raw movies.
@foorgol: still don't think that the method causes loss in detail at all. not all af dots are visible in all conditions, but they are still there with an incorrect value. the difference of the af-pixel value and the value that the pixel should have is just to small to notice it. however even in that case the interpolated value should be more exact than the wrong af-pixel value.
@spider: some new features that enhance the usability will come soon. we discuss features like optional overwriting the dngs, extra output folders and removing dots directly in the raw instead of the dngs. you can add feature requests on github: https://github.com/Foorgol/PinkDotRemover/issues
that way you can be sure, that the feature idea doesn't get lost between the other posts in the forum.
@foorgol thx for all the good development. I think now the EOS 650D needs an option for Crop Mode too.
Quote from: Fuma on July 15, 2013, 01:17:39 PM
@foorgol thx for all the good development. I think now the EOS 650D needs an option for Crop Mode too.
I'm not sure that I understand what you mean... are you referring to additional resolutions that need to be supported by the PinkDotRemover? Or do you mean ML-support for additional shooting modes/resolutions?
Pls forget my post.
I have seen, that i can use EOSM_Crop even for my Videos that i have taken in Movie Crop Mode with the 650D.
My fault :)
hurm... that is interessting. are the dot positions in crop mode for 650d and eos m identical? they should be, since they share the same sensor, but we handled them seperatly, since the downscaling without crop mode of 650d and eos m are completely different (totally useless for eos m) and that caused different dot positions on those models. we should rename the eos m crop mode to crop mode then.
the 007 version is btw still buggy with crop mode. all resolutions that are not divisible by 4 shouldn't work. there will be an update to 008 soon.
Looking forward to the new update for better support for crop mode too!
btw 1280*720 crop mode on 650D doesnt work with pinkdotremover with eosm_crop. is this resolution not divisible by 4?
edit, it works after i downloaded the latest raw2dng.exe
So, let's break the silence here! Time for a new debug release! :)
The last days I gave the code a major re-write in order to support PinkDotRemoval directly in the RAW files. Nevertheless, removal in DNG-files is still possible, of course.
I did some testing, but it's not really what I would call thorough testing. Therefore this is a debug release only. Please give it a shot and tell me if it works for you. Download the binaries here:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__debug.zip
BEWARE:
Other than DNG files, RAW files will be OVERWRITTEN IN PLACE. Make sure to create a backup of your RAW file before you start the conversion.
Implementing it this way was easier for me. So much to my paradigm of not overwriting user data... :)
Enjoy!
P.S.: The in-RAW-removal together with your feedback and a few improvements that Mixer2 has in the pipe will constitute the next "official" release 008, I think...
If I select the RAW files they are not added to the conversion list. I use the latest JAVA7
Quote from: foorgol on July 07, 2013, 05:13:47 PM
* When selecting a bunch of files or a directory, only DNG files will be added to the conversion list
Maybe it depends on this?
I have the same problem
Aaargh... my fault... I packed the wrong JAR file. Actually the bug you mentioned was the last one I fixed. But then I forgot to built a new "distribution JAR"... I only tested it with the "development environment JAR".... well, whatever... :)
Now I've uploaded the correct version. The link posted above is still valid.
Sorry for the confusion!
Seems to work great for me here mate! It's even easier now with direct raw proccesing. and it seems faster too? or maybe just my idea!
Great news! The in-RAW idea really makes a lot more sense. No more triple-sized folders full of 3 different versions of the video!
RAW processing works great I think and it is so awesome fast. 8)
@foorgol
May you have a look at this thread http://www.magiclantern.fm/forum/index.php?topic=5645.new#new
I think you can give fatpig better information than me. Because I asked if it is possible to integrate the pinkdotremover into BATCHelor.
Quote from: spider on July 31, 2013, 12:24:50 PM
it is so awesome fast. 8)
Yes, I had the same impression, although I didn't make any specific performance tests. Since the code for the dot removal itself is identical for both DNGs and RAWs, my assumption is that the opening/closing of many DNGs files consumes time. Even the overall amount of data read and written from/to disk is identical between processing a RAW and its DNGs. So it must be the handling of maaany files....
Quote from: spider on July 31, 2013, 06:02:59 PM
May you have a look at this thread http://www.magiclantern.fm/forum/index.php?topic=5645.new#new
I think you can give fatpig better information than me. Because I asked if it is possible to integrate the pinkdotremover into BATCHelor.
Done:
http://www.magiclantern.fm/forum/index.php?topic=5645.msg63402#msg63402
Thank you.
And I think the same it is faster because there are not so many files operations. The speed improvement should be the biggest on 5800 rpm drives and maybe only a little improvement on SSD.
Little update:
Exchanged a few messages about the BATCHelor integration with fatpig the last days. Looks as if he could just call the PinkDotRemover from the command line. For this purpose I added a few lines of code to enable camera type selection for the command line mode of the PDR. Up to now, the cam type was immutably set to 650D when using the command line.
In the meantime, mixer2 provided a few goodies for the GUI: he improved the file list, added the ability to delete individual files from the list and introduced support for Drag'n'Drop!
So be prepared for a new release coming out soon! :)
Hola , se agradecería si incluyes la resolución 1728x526 en el próximo pinkdotremove , en oras resoluciones funciona perfecto. Muchas gracias y sigan así. Saludos .
Hello, it would be appreciated if you include the resolution 1728x526 in the next pinkdotremove in resolutions pray it works perfect. Thank you very much and keep it up. Greetings.
Quote from: foorgol on August 08, 2013, 03:21:39 AM
So be prepared for a new release coming out soon! :)
Thank you very much.
drag and drop is a dream coming true!
cool, thanks for your effort :)
and btw. wtfpl is a cool license choice, haha.
Would be cool if you could add the signature for the 700D :)
http://puu.sh/40nze.png
http://puu.sh/40nwP.png
Looks like I figured the problem out, no need for this anymore ;)
https://bitbucket.org/hudson/magic-lantern/pull-request/184/700d-650d-remove-pink-dots-from-raw/diff
This is good news! I have to be patient for satriani's new build tomorrow ... very patient.
Ciao, Walter
Quote from: TheUnkn0wn on August 17, 2013, 07:58:54 PM
Looks like I figured the problem out, no need for this anymore ;)
https://bitbucket.org/hudson/magic-lantern/pull-request/184/700d-650d-remove-pink-dots-from-raw/diff
is this for real?
Quote from: TheUnkn0wn on August 17, 2013, 07:58:54 PM
Looks like I figured the problem out, no need for this anymore ;)
https://bitbucket.org/hudson/magic-lantern/pull-request/184/700d-650d-remove-pink-dots-from-raw/diff
:o :D ;D
How does it work? Are these pixels interpolate in the camera?
Quote from: spider on August 17, 2013, 09:55:29 PM
:o :D ;D
How does it work? Are these pixels interpolate in the camera?
Nope, nothing is interpolated. Which means your not losing any details anymore. I've set the RAW_TYPE to a stream that doesn't contain those anomalies.
Raw 99 is full of green dots on the 650D :/
The same for me nanomad. How it looks with 700D is the picture clean?
Quote from: nanomad on August 17, 2013, 11:07:35 PM
Raw 99 is full of green dots on the 650D :/
For me the 700D has a perfect picture with 99. When I was pixel peeping the RAW_TYPE's I found that 78 and 99 were the best.
Try 78? Looks like the 650D and 700D aren't "exactly" the same after all. If that doesn't work then run this
debug.c - based off work from a1ex
#include "raw.h"
static void run_test()
{
char fn[100];
for (int type = 0; type < 100; type++) {
MEM(0x350B4) = type;
raw_lv_request();
msleep(50);
raw_lv_release();
msleep(50);
raw_update_params();
snprintf(fn, sizeof(fn), "%s/%04d.DNG", get_dcim_dir(), type);
bmp_printf(FONT_MED, 0, 60, "Saving raw type %d, %d x %d...", type, raw_info.jpeg.width, raw_info.jpeg.height);
save_dng(fn);
}
}
You will have to pixel peep all of the pictures, I used Lightroom to do this.
78 looks fine ;)
And here third and the last for today build (https://www.dropbox.com/sh/demgitqk7a6ixsv/KeBhooSBHp/ML_N_17.08.2013_03.zip) :D
https://bitbucket.org/hudson/magic-lantern/pull-request/186/700d-650d-found-better-raw_type/diff
Can you upload a DNG with 78 while focusing on text or hard H / V edges?
Sorry guys, it was a bit dark in my room. When the light is on, then this looks not good :/
(https://photos-2.dropbox.com/t/0/AADeLF-vLS1QJ2DWFbfTFqRZk3Wb8li43xbj3mPDGsiuxA/12/14345341/png/2048x1536/3/1376780400/0/2/Bildschirmfoto%20vom%202013-08-17%2023%3A55%3A53.png/dlA9Blc5YEOufuX2VUkqd4mTcY2tByggdkpInC_Y-c4)
(https://photos-2.dropbox.com/t/0/AABCME_k5Dh9Je-d1HHC0sq58Ac4052WxDY3YH1zqz3gcg/12/14345341/png/2048x1536/3/1376784000/0/2/Bildschirmfoto%20vom%202013-08-17%2023%3A56%3A26.png/I13a07ijFLRg7AODxyzfjDkTaCFQodcs6wLi9xi2chg)
(https://photos-4.dropbox.com/t/0/AAB9gt1x4K0SkzF3NVBO3hr8dEIIG2NQHIjLfn_Atc3sDw/12/14345341/png/2048x1536/3/1376784000/0/2/Bildschirmfoto%20vom%202013-08-17%2023%3A57%3A15.png/2_AUqdp80SJV3gWyl2ZwptEQ976Fgv4Ny_de4khW26I)
Looks like someone with a 650D will have to run this
http://www.magiclantern.fm/forum/index.php?topic=6658.msg67745#msg67745
P.S : After making the changes in debug.c, to run it go to the Debug menu in ML and click on Don't click on this!
BTW, the key difference between the 650D and the 700D/100D is the LV cont. AF algorithm.
650D: v1
EOSM with FW upgrade: v1.1 (simply a v1 tweaked for for more speed probably)
700D/100D: v1.5 (LV feed @ 60 FPS according to canon, possibly other tweaks)
70D: v2 (dual pixel sensor)
Who wants to do some pixel peeping for the 650D?
https://www.dropbox.com/s/8gkifgx2lwwq2rx/650D.104-RAW_TYPES.zip
Download that, replace the files on your 650D SD card. Enter Movie Mode, go to the debug menu and click on "Don't click on this!"
It will start taking 100 frames, might take a while. Make sure you keep your camera on something static so you can pixel peep properly.
Once it has done you should have 0000-0099 DNG's on your SD card. Go through them all (I recommend light room) and find the BEST one out of them all, report your findings REPORT THE FILE NAME of the best one you've found.
Error: Cannot be opened as zip ...
Quote from: TheUnkn0wn on August 18, 2013, 02:10:34 AM
Who wants to do some pixel peeping for the 650D?
https://www.dropbox.com/s/8gkifgx2lwwq2rx/650D.104-RAW_TYPES.zip
Download that, replace the files on your 650D SD card. Enter Movie Mode, go to the debug menu and click on "Don't click on this!"
It will start taking 100 frames, might take a while. Make sure you keep your camera on something static so you can pixel peep properly.
Once it has done you should have 0000-0099 DNG's on your SD card. Go through them all (I recommend light room) and find the BEST one out of them all, report your findings REPORT THE FILE NAME of the best one you've found.
I tried.
What I did: with lens cap, I clicked on "Don't click me!". So I get 100 unexposed DNGs. I opened it in Adobe Came RAW, and add +3 EV in EXPOSURE, so I can see the dots. I looked one by one, and
for me they all look not good. So I export a 867x579 pixels (50%) version of each in JPG, and uploaded the zip file (with 100 JPGs). Here is the link (https://www.dropbox.com/s/wroyh5xwaokks36/650D_104_RAW-types_jpg-small.rar) (file size: 23 MB).
Hi , solo se ven puntos rosados , en las 99 fotos todas mas o menos igual. https://www.dropbox.com/s/t0e3xnanwmbx44x/0099.TIF
Please don't take pictures of a cap, use a well exposed book cover with hard edges and high contrast then also check for small pink green dots around the letters. If you think you found a good candidate post the DNG
Yes, user error and a stupid one, I have to confess!
Quote from: nanomad on August 18, 2013, 08:53:13 AM
Please don't take pictures of a cap, use a well exposed book cover with hard edges and high contrast then also check for small pink green dots around the letters. If you think you found a good candidate post the DNG
Oh. Sorry, I didn't know. Thank you for explanation.
Anyway, I repeated the test, and all 100 options seems not good to me. Ugly spots in all DNG files.
Ooops! My fault, no wonder they all look the same, I still had the 700D RAW_TYPE address in there...
https://www.dropbox.com/s/dougzp7ebgpvb6w/650D.104-RAW_TYPE2.zip
Try this one, it also goes to 255 instead of 99
Ok, me and nanomad have kind of figured out the problem. With raw_type 14 you don't get any image anomalies using Adobe software. Using other software causes these anomalies. It's probably down to how the DNG is created?
It's also occurred to me, but I did not understand why this is so :D
Anyway, it's not ok. Not everyone can or want to use Adobe. I would like to know why that is so and what is it.
How does the anomalies look like?
ACR filters Hot-pixel automatically, maybe these anomalies are only one pixel big and ACR filters them?
Is it on the 700D the same?
I tried satrianis builds of 17th.
You see the green dots in the Faststone Imageviewer and shortly in the LR preview.
After rendering a high quality preview LR filters them out, but the filter quality is crappy.
Indeed. I'm not willing to make people rely on a particular image processor, let alone an Adobe one. I'm going back to pink dots on all the cameras that had them
Try RAW_TYPE 2 it looks clean and it works with both dcraw and adobe software.
EDIT : nvm, I've found dead pixels in it...
Thats looks exactly like a RAW_TYPE 121 or something like this, which I tested yesterday. But the streaming seems to be unstable and give sometimes a broken pics.
Look at this:
this is a good pic
(http://s1.directupload.net/images/130820/yew9xo2v.png)
and this is a broken picture, which was taken two seconds later
(http://s7.directupload.net/images/130820/bb6ukny5.png)
But I think we are already closer to the solution :)
I already have a solution, I've re-created raw2dng using the Adobe DNG SDK. Since Adobe created the DNG format its best practice to use what they give you. dcraw seems to ignore some meta-data and misinterprets the raw data causing that green mesh. My raw2dng will include a host of extra features including compression.
You make me so hopefull ;D
Quote from: TheUnkn0wn on August 20, 2013, 04:23:58 PM
I already have a solution, I've re-created raw2dng using the Adobe DNG SDK. Since Adobe created the DNG format its best practice to use what they give you. dcraw seems to ignore some meta-data and misinterprets the raw data causing that green mesh. My raw2dng will include a host of extra features including compression.
now this could be cool!
Say I noticed on another instructional video that one can use QuickTime to recompile the video after post processing. Anyone here do that? Seems really simple and more options than Virtual Dub.
dave
(http://puu.sh/46SsU.png)
Yes, this will properly process the DNG's so you don't get that green pattern you see when using RAW_TYPE 14.
Which means there's no need to remove any pink dots since there wont be any and no quality will be lost.
Quote from: TheUnkn0wn on August 21, 2013, 03:15:21 AM
(http://puu.sh/46SsU.png)
Yes, this will properly process the DNG's so you don't get that green pattern you see when using RAW_TYPE 14.
Which means there's no need to remove any pink dots since there wont be any and no quality will be lost.
Ok I will admit my lack of knowledge, but can you explain what this. Sorry to be a pest. Just trying to learm
Dave
Update. Do you mean qt or something else?
On the 700D and 650D you get pink dots don't you? That's because of the default raw stream (or raw type) from the camera. If you change it to one that doesn't contain any anomalies; for some reason draw (used by software such as RAWTherapee) can't process the dng's correctly causing a green tinge/mesh over the image. To fix this I've created a new RAW2DNG tool using Qt so its multi-platform. Here are some of the features.
- Multi-Threading (Makes use of all your CPU cores)
- Full DNG support (Using Adobes own DNG SDK, this fixes a lot of things like thumbnails and image anomalies)
- Compression (Can use Lossless JPEG for DNG's)
- Save directly to TIF
Quote from: TheUnkn0wn on August 21, 2013, 03:49:37 AM
On the 700D and 650D you get pink dots don't you? That's because of the default raw stream (or raw type) from the camera. If you change it to one that doesn't contain any anomalies; for some reason draw (used by software such as RAWTherapee) can't process the dng's correctly causing a green tinge/mesh over the image. To fix this I've created a new RAW2DNG tool using Qt so its multi-platform. Here are some of the features.
- Multi-Threading (Makes use of all your CPU cores)
- Full DNG support (Using Adobes own DNG SDK, this fixes a lot of things like thumbnails and image anomalies)
- Compression (Can use Lossless JPEG for DNG's)
- Save directly to TIF
Well this is cool. How do can we try this? Cost?
Dave
As I explained before the green pattern stands out a lot even in the Luma only image before debayering. It's not a converter fault.
Quote from: satriani on August 20, 2013, 01:40:23 AM
Thats looks exactly like a RAW_TYPE 121 or something like this, which I tested yesterday. But the streaming seems to be unstable and give sometimes a broken pics.
Look at this:
this is a good pic
(http://s1.directupload.net/images/130820/yew9xo2v.png)
Sadly it is not, look near the sharp contrasty edges (the microSD writing) and you'll see contrast detection "control" points that are impossible to remove.
Quote from: nanomad on August 21, 2013, 09:27:34 AM
As I explained before the green pattern stands out a lot even in the Luma only image before debayering. It's not a converter fault.
It's not a converter fault, its dcraws fault :P
http://forums.adobe.com/thread/478513
Guys, may I ask you please to share a build with RAW_TYPE 14. I compiled it by myself, but I cant see any difference: the pink dots pattern is the same. May be i'm doing something wrong.
Quote from: Rewind on August 23, 2013, 12:19:50 AM
Guys, may I ask you please to share a build with RAW_TYPE 14. I compiled it by myself, but I cant see any difference: the pink dots pattern is the same. May be i'm doing something wrong.
https://bitbucket.org/theunkn0wn/magic-lantern
Well, I don't think these raw_type manipulations are exactly the Solution. Pink dots around high contrast edges are still there.
I mean, this method doesn't excludes the focusing pixels from raw stream, it just adds correct hot/dead pixel info, so we can chose whatever soft we want to deal with them. Right?
Also, TheUnknOwn said, that "With raw_type 14 you don't get any image anomalies using Adobe software". Anyone can confirm that? In my experience, when i change the string in raw.c to #define PREFERRED_RAW_TYPE 14, compile and process with ACR, i see the same pink-green artifacts around high contrast edges:
(http://ipic.su/img/img7/fs/kiss_22kb.1377242118.jpg)
Quote from: Rewind on August 23, 2013, 08:01:00 AM
Well, I don't think these raw_type manipulations are exactly the Solution. Pink dots around high contrast edges are still there.
I mean, this method doesn't excludes the focusing pixels from raw stream, it just adds correct hot/dead pixel info, so we can chose whatever soft we want to deal with them. Right?
Also, TheUnknOwn said, that "With raw_type 14 you don't get any image anomalies using Adobe software". Anyone can confirm that? In my experience, when i change the string in raw.c to #define PREFERRED_RAW_TYPE 14, compile and process with ACR, i see the same pink-green artifacts around high contrast edges:
(http://ipic.su/img/img7/fs/kiss_22kb.1377242118.jpg)
Use RAW_TYPE 78 or 99
I missed the focus points in 14
Quote
Use RAW_TYPE 78 or 99
I missed the focus points in 14
These raw_types acts exactly the same (at least for 650D).
Pink dot remover tool gives better results for now.
First crop below shot with older build, cleaned up with pink dot remover tool and processed with ACR.
Second crop was made with RAW_TYPE 78 and then processed with ACR.
As you can see, in the case of using raw_types ACR has to interpolate not only the focusing pixels, but also that green pattern all over the frame, causing the black spot appears like a whole mess of green-magenta stuff. First picture is much cleaner. (I use the mosaic engineering vaf filter, so there's no place to aliasing and moire at these examples, and first one in video looks acceptable and usable, but second is not).
(http://ipic.su/img/img7/fs/kiss_29kb.1377264266.jpg)
(http://ipic.su/img/img7/fs/kiss_22kb.1377264293.jpg)
Foorgol, I have a question:
Using PDR tool in interpolate mode gives an overall good results (it eliminates the consistent pink dots around high contrast edges, which set dead pixel method can't do), but it produces another problem: pronounced somewhat random colorful artifacts in detailed contrasty areas, such this:
(http://ipic.su/img/img7/fs/kiss_35kb.1377331825.jpg)
Let's say I can live with some quality loss, if so, is there any simple method to interpolate those things hell of a lot? I mean, turn off that judgement algorithm, you've been talking earlier ( http://www.magiclantern.fm/forum/index.php?topic=6658.msg55325#msg55325 ), and just interpolate all of them?
Should I change something in dotData files for example? Or, if there's no way to do this from dotData files, wouldn't it be a good idea to integrate some option in your program, like slider gives a user a possibility to determine how sensitive this decision algorithm is?
EDIT: The positions of these artifacts are not random of course, they belong the same grid. It seems like just something is wrong with the interpolation method. Watching carefully to particular dot, shows that when surrounding is smooth, it interpolates fine, but when the surrounding is some kind of an edge, then fancy colors appears: sometimes green, sometimes magenta or yellow etc. for the same dot.
May be this is something to do with the "finding the minimum gradient" in your PinkDotRemover.java source.
I'd like to dig in and experiment with that, but I need some help: can you guys please point me to the resources on how to build from java?
@Rewind:
current pdr removes all dots for the defined pattern. there isn't any algorithm that checks, if it should fix the dot, or something like that.
there will be a better algorithm for interpolation in the next version which should lead to much less artifacts. have to talk to Foorgol when we'll finish the new version, that you can try it.
Quotethere isn't any algorithm that checks, if it should fix the dot, or something like that.
Got it. Figured that out already, studying the source :)
Quotethere will be a better algorithm for interpolation in the next version which should lead to much less artifacts
That would be great, we'll be looking forward to that. Guys you rock!
By the way, mixer2, which IDE you'd recommend for java development?
if you're interessted in the sourcecode you can check the version with the new interpolation algorithm... very unclean so far, but it's already working with much better results if you want to try it -.-
https://github.com/mixer2/PinkDotRemover/tree/ImprovedInterpolation
it's basically the algorithm from this paper:
http://www.tanbakuchi.com/Publications/Papers/2003AdaptivePixelDefCorPub.pdf
about the IDE... i really don't do any java beside pdr. the only reason i installed a java IDE was because Foorgol wrote the pdr in java and i wanted it to be compatible to the eos m. did some java at the university before... but not much at all. so i don't know much about java IDEs. that said i would always use the IDE the other developers of the project also use. imho it's just a good idea if the whole development team has the same environment. on the pdr i work with netbeans, since Foorgol developed it with netbeans. neatbeans is ok... it's fast and it does work. but, for example, on another project i currently work on, that is written in c++, i use eclipse, because all others working on that project do use eclipse. it would be possible to develop both in eclise or both in netbeans... but just use what the others do. for small new java or c++ projects i would use netbeans, because i have it already installed and i prefer it over eclipse.
if you ask what i think would be the best IDE for java development i would suggest giving IntelliJ IDEA a try. i have never used it and don't have it installed, that's why i said i would use netbeans, which i have already installed, for small projects. since i never used it i'm not sure that it is the best IDE for java development, but i do work every day with PhpStorm, which is from the same company, and it's the best IDE i've ever used.
mixer2, thank you. So much useful info.
Ok, just confirmed it's not a de-bayering error. The green pixles are burnt into the image
Aloha, brotherhood )
Bring you some good stuff.
Well, I've studied Foorgol's and Mixer2's source, installed the NetBeans and dig in ) I've spend a couple of nights to do my own research on Bayer pattern, defect correction techniques and that kind of stuff.
I've tried a couple of interpolation methods, they seemed to work fine in some situations, but fail in others, so finally i settled on some form of adaptive interpolation, same as the developers of PDR did.
While the original Tanbakuchi's implementation of an adaptive pixel defect correction seems not working for me (doesn't hold the edges correctly and has too many artifacts in high-frequency areas), this slightly modified method alraedy gives a quite acceptable results: It holds the critical vertical and horizontal edges nicely, has a lot less artifacts and a reduced overall false colorization.
The gif below shows the comparsion between current interpolation method in PDR and my modified adaptive interpolation (200% crop, 1 sec change)
(http://moredo.ru/raw/PDR_interp_comparsion_1.gif)
I'm not a programmer, so I don't know your rules here guys. May I or may not share my build? Can we share the knowledge with the developers of PDR or am I just sticking my nose where I don't belong?
Anyways, in conjunction with VAF filter, i've already turned my 650D into the hell of a raw machine )) Now PDR gives me acceptable interpolation quality, modified raw_rec module allows me to shoot stable 1520x636 (1:2.4 cinemascope) for more than a minute, so we are going to shoot our sci-fi short entirely in raw workflow this autumn. That's so damn great!
@rewind:
nice work! it's open source so you're free to share any builds of it.
it would be great if you could make a pullrequest on github, that it could be included into all future pdr versions.
Wheeew... a little more than two weeks of vacation and business trips and the good, old PDR-world is not what it used to be anymore :-)
I just flipped through the last pages of the thread and understood that the PDR is not (yet) obsolete. Right? So I keep my promise to release a new version 008 which brings you, compared to the previous official version 007:
* Better support for "ugly" image resolutions which are not divisible by 4
* Improved pink dot locations for crop mode and EOSM
* In-RAW dot removal for videos files (no need to process hundreds of DNGs anymore)
* Better file list in the GUI, including support for Drag&Drop and deletion of selected files with the Del-key
* Persistent storage of the selected camera type for the GUI
* Possibility to provide the cam type as first command line parameter for batch operations
Some of the features were already available in the last debug release. And credit were credit is due: all the GUI and dot location data improvements were contributed my Mixer2. I concentrated only on the in-RAW-removal and on merging Mixer2's pull requests... :)
Download the binaries here: https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__008.zip
I'm looking forward to include an improved removal algorithm in the next release. Again, Mixer2 already contributed a very promising implementation and obviously Rewind is also working on some neat stuff. Perhaps we'll just add a drop-down menu to the GUI so that you can choose the algorithm. That would be the easiest (intermediate?) solution.
Happy dot removing and thanks again to all testers and contributors,
foorgol
Quote from: Rewind on August 27, 2013, 11:54:46 AM
I'm not a programmer, so I don't know your rules here guys. May I or may not share my build? Can we share the knowledge with the developers of PDR or am I just sticking my nose where I don't belong?
I can only second Mixer2's statement: please share your code so that we can merge it with the official releases. If you're familiar with GitHub, issue a pull request; that would be the easiest and preferred way of sharing your code. As an alternative, you can send me your sources and I'll merge them manually.
@Mixer2:
What do you think? Replace my old and busted removal algorithm with yours AND Rewind's and introduce a drop-down box (or radio buttons) to select the preferred method?
Well, seems to me, i've made a pull request :)
https://github.com/mixer2/PinkDotRemover/pull/1
But GitHub is still over my head for now, so i'd like to duplicate my code here. Just for clarity sake (not sure, if i did a request right)
protected void interpolPixel(Generic_CFA_PixBuf srcBuf, Generic_CFA_PixBuf dstBuf, int[][] dotList)
{
int w = (int) srcBuf.imgWidth();
int h = (int) srcBuf.imgHeight();
for (int[] dot : dotList)
{
int x = dot[0];
int y = dot[1];
if ((x < 3) || (x > (w - 4)) || (y < 3) || (y > (h - 4))) continue;
//Modified adaptive pixel defect correction algorithm from
//http://www.tanbakuchi.com/Publications/Papers/2003AdaptivePixelDefCorPub.pdf
//Takes to account only two vectors (horizontal and vertical)
//Not entirely shure why, but this gives better results in all methods i've tried (both high-frequency false colors and edges treatment)
//Maybe simply because of squared nature of raster images themselves
// 1. Retrieve vectors from 7x7 kernel
// d[0] — vertical vector
// d[1] — horizontal vector
// index reference:
// paper -3 -2 -1 0 +1 +2 +3
// actual 0 1 2 3 4 5
int[][] d = {
{srcBuf.CFA_getPixel(x,y-3), srcBuf.CFA_getPixel(x,y-2), srcBuf.CFA_getPixel(x,y-1), srcBuf.CFA_getPixel(x,y+1), srcBuf.CFA_getPixel(x,y+2), srcBuf.CFA_getPixel(x, y+3)},
{srcBuf.CFA_getPixel(x-3,y), srcBuf.CFA_getPixel(x-2,y), srcBuf.CFA_getPixel(x-1,y), srcBuf.CFA_getPixel(x+1,y), srcBuf.CFA_getPixel(x+2,y), srcBuf.CFA_getPixel(x+3, y)}
};
// 2,3 — We don't need these stepse because of diagonal af dots arrangement
// 4. Normalizing vectors
// vertical norm.
d[0][2] = d[0][1]+((d[0][2]-d[0][0])/2);
d[0][3] = d[0][4]+((d[0][3]-d[0][5])/2);
// horizontal norm.
d[1][2] = d[1][1]+((d[1][2]-d[1][0])/2);
d[1][3] = d[1][4]+((d[1][3]-d[1][5])/2);
// 5. Deltas and Weights
int dVert = Math.abs(d[0][2]-d[0][3]);
int dHoriz = Math.abs(d[1][2]-d[1][3]);
int Delta = dVert + dHoriz;
float wVert = 1-((float) dVert / Delta);
float wHoriz = 1-((float) dHoriz / Delta);
// 6. Calculating new pixel value
float newVal = wVert*((d[0][2]+d[0][3])/2) + wHoriz*((d[1][2]+d[1][3])/2);
dstBuf.CFA_setPixel(x, y, (int) newVal);
}
}
For those, who wants to try this method, here is the link (http://moredo.ru/raw/PDR%20Rewind%20interp.7z) for the build. It made of latest foorgol's and mixer2's source, so also includes drag&drop support etc.
@rewind and @foorgol:
your pull request is correct. i just didn't merge it so far, since i'm not completely sure if it will perform better in every case. removing the diagonals may lead to problems with diagonal lines.
i'll test both algorithms in multiple situations and do some further investigations if we can improve the algorithm without removing the diagonals. if there is a way to get best of both worlds we may just use a single algorithm. if each algorithm performs better in some situations we may let the user choose.
The pinkDotRemover (v 0.8 and 0.7) doesn't remove pink dots on RAW/DNG files shot with the latest ML for 650d (31.08.2013).
I've try 2 RAW videos in the same condition except ML version :
- Nothing is removed with PDR on the latest version (31.08.2013).
- Pink Dots are removed fine with an older build (08.08.2013).
The pink dots pattern seems to be the same with the 2 versions.
I guess something had been changed recently concerning the RAW module (plus I can't install the old raw module on the latest build : it says it's in "V2" and a "V3" is expected).
@mixer2, @foorgol Sorry to clutter the thread but I was to make ABSOLUTELY sure your work is greatly appreciated! As you can see on vimeo.com/maxotics, I'm using PDR every day. A few days ago I decided I had to make a big decision, focus on the 50D with more resolution and no pink dots, or go with the smaller, less resolution EOS-M with pink dot workflow. I chose the latter because PDR works great (or I can't see any problems). My wish list would be a PC exe that automatically detected between crop and non crop mode and did the conversion. But again, am perfectly happy and wanted to thank you both, and your new musketeer @rewind, again and again and again!
Anyone else who has problems with PDR with the actual builds(31.08 and 03.09) of ML when Video Crop mode is activated?
hurm, i don't have an 650d, so i can't find the compatibility problem with pdr and the current 650d release.
if you send me a dng that doesn't work i can investigate and fix it.
https://www.dropbox.com/s/do6ol4p8veadf2d/000000.dng
it's crop mode, isn't it? i have no idea why the dot positions moved. maybe they changed something how they calculate the crop position.
try this temporary dot location file, till we know why the dot positions have changed.
http://dl.dropboxusercontent.com/s/by314z0kb8pfvsb/CropMode.txt
thank you for the quick help,
the ML Version before 31.08. had worked pretty well in crop mode.
Mixer2,
I am having a problem with PDR not removing all the dots. There is a faint leftover after processing files. Am I doing something wrong or is there something with the latest ver of ML causing the dots to not completely be removed? here is one dng that was processed by PDR-
https://www.transferbigfiles.com/05231aab-4072-4d49-95dc-8c0f979d4a95/zBh_ykiyjkZigXt8I_776g2
Gary
Mixer2,
I got your reply in the M forum.
Sorry for the double posting.
Gary
Hi guys, i've been desperately trying to get this RAW footage sorted for my 650d, spent all night installing it on to my 650d, filmed some RAW footage, which I really liked, I'm going out of my mind, trying to fix the pink dots issue, I'm sorry if I'm being extremely stupid, but it is possible for someone to give me a short run through on how I use RawTherepee and/or install/use the PinkDotRemover, I am having an absolute nightmare, as I just can't seem to work it out, after numerous attempts at it,
Any help would be greatly appreciated, a simple step by step instruction, may be beneficial to many fellow videographers who wish to take advantage of these excellent tools that have been developed.
Thank you in advance,
Kash
Quotespent all night installing it
Holy cr2! ))
Well, I assume you're running windows:
1. Go to java.com and install the runtimes if you don't have them already;
2. Download PinkDotRemover (http://moredo.ru/raw/PDR%20Rewind%20interp.7z) (this is my modified bild which has better interpolation for now, less artifacts), extract the contents wherever you want;
3. Go inside extracted folder and just double-click the PinkDotRemover.jar — if java installed correctly, double-click from windows explorer will execute the program;
4. Jast drag and drop your *.RAW files onto the PDR's window and hit "Convert Files"
That's it! Wait for a few seconds, then process your raw files the way you used to (extract the dngs directly by raw2dng, or use Rawanizer, Batchelor etc.)
Rewind, Mixer2
Neither of your versions of PDR seem to be removing pink dots form my EOS-M in crop mode anymore. I think Gary might also be stuck. Any thoughts? I have a new build from 1%, which is about a week old. Mixer2, your version will process the files (though pink dots stay). Rewind, your version doesn't process; says they are corrupted. I'm using a Sigma 10-20mm EF on the EOS-M. Was working a couple of weeks ago just fine.
Quote from: maxotics on September 06, 2013, 08:30:42 PM
Rewind, Mixer2
Neither of your versions of PDR seem to be removing pink dots form my EOS-M in crop mode anymore. I think Gary might also be stuck. Any thoughts? I have a new build from 1%, which is about a week old. Mixer2, your version will process the files (though pink dots stay). Rewind, your version doesn't process; says they are corrupted. I'm using a Sigma 10-20mm EF on the EOS-M. Was working a couple of weeks ago just fine.
Have you tried this dot data file?
http://www.magiclantern.fm/forum/index.php?topic=3648.msg73461#msg73461
I copied that to you build, but didn't work. Copies it to Rewind2's, ran, but pink dots still there. BTW, 1% has gotten higher resolutions in crop mode so imagine that's going to create more need for PDR.
http://www.magiclantern.fm/forum/index.php?topic=3648.msg73840#msg73840 (http://www.magiclantern.fm/forum/index.php?topic=3648.msg73840#msg73840)
In the meantime, what is the best alternative method to remove pink dots? Is there one? I'm on a PC. Thanks!
Well, I have no EOS-M, and I'm not interested in crop mode recordings, so I can't help you atm.
But I would suggest you to take two DNGs from old build and from the new one, compare the positions of af dots and make a corresponding changes to dot data file.
Or just wait for foorgol or mixer2, when they hopefuly figure the problem out (seems like something changed with centering of the crop box)
Thanks for your help Rewind! Much appreciated!
I've figured out how to open the DNGs and get pixel locations of the pattern, but have no idea how they match the config files. Are those x/y coordinates? Are they the center lines? I do notice that a DNG form a video that works does not match the one that doesn't currently work. I'm shooting EOS-M, 1280x720 crop mode. Used to work. Thanks!
I'm not sure if any of the devs are following PDR anymore, so I wrote this in EOS-M, which is the camera I use, and in which I have come up against these limitations.
http://www.magiclantern.fm/forum/index.php?topic=3648.msg73969#msg73969
Temporary solution for EOS-M crop mode:
http://www.magiclantern.fm/forum/index.php?topic=3648.msg74097#msg74097
Devs? Rewind?
If I want to open DNGs created from video RAW and find the pixels that the camera uses for auto-focus, the pixels that create "pink dots", which attributes of each pixel best identify them? Or, is there are definitive attribute that can be used? My plan at moment is to use some low-grade statistical analysis to distinguish them from neighboring pixels that also seem to be hot. That plan doesn't excite me ;) Thanks!
When PDR was down I began looking into a hot-pixel mapping solution that would output a bad pixel file for dcraw which could be used to remove pink dots. Don't know if you know, but PDR wasn't working after 1% latest (greatest!) release. Thanks to REWIND again for saving the day!
Rewind, if we can get such a process it would at least give you a clean bad-pixel file from which to work if we need you to put on your superman cape again :)
Quote from: foorgol on June 21, 2013, 03:55:34 AM
Sorry, I can't help you with LV or similar. I have absolutely no clue what happens inside ML and how data is converted back and forth inside ML before being dumped to the card. I'm not an ML developer.
But regarding your idea to integrate the dot removal into ML:
From an architectural point of view, I would not do that. Writing the RAW data in higher resolutions is already time critical enough today (okay, mostly limited by writing speed, AFAIK). So I would push as much data processing as possible out of ML and up to your normal workstations with practically unlimited processing power, time and disk space. You have to touch all the image files anyway for post-processing, so adding the dot removal is not so much of a pain here.
Dot removal inside ML would only make sense if the built-in camera processors are extremely optimized for these kind of tasks (what's not unlikely) and/or if you have additional information inside the camera at runtime which would support/optimize the removal process but which can't be exported to the outside world.
From all I know today, I would rather strive for an integration of the removal process into raw2dng....
A mi me dá permanentemente el error mostrado en el siguiente link http://www.flickr.com/photos/98427703@N08/9715433322/ (http://www.flickr.com/photos/98427703@N08/9715433322/)
Alguien puede ayudarme como solucionarlo?
Quote from: coco_net on September 10, 2013, 05:13:45 AM
A mi me dá permanentemente el error mostrado en el siguiente link http://www.flickr.com/photos/98427703@N08/9715433322/ (http://www.flickr.com/photos/98427703@N08/9715433322/)
Alguien puede ayudarme como solucionarlo?
Use PDR with your RAW files, not dng.
By the way, this is an english board.
Quote from: maxotics on September 08, 2013, 05:49:17 PM
...If I want to open DNGs created from video RAW and find the pixels that the camera uses for auto-focus, the pixels that create "pink dots", which attributes of each pixel best identify them? Or, is there are definitive attribute that can be used?...
These pixels changes their values dynamically, so there are no any distinctive attributes, except may be they LOOKS wrong ))
Quote from: Rewind on September 10, 2013, 09:42:48 AM
Use PDR with your RAW files, not dng.
By the way, this is an english board.
El mismo error me lo daba con los archivos RAW. Pero encontré otra version de pinkdotremover que si lo hace pero quedaban los puntos rosas. Insistí hasta que le encontré la vuelta haciendole una primera pasada con la opcion "EOS-M" y despues otra pasada con la opcion "650D" y logré hacer desaparecer los puntos rosas. Esto anda bien con la configuración de captura de 960 x 540. Con la opción 864 x 486 quedan algunos puntos indeseables.
Ya se Rewind que es un foro en inglés, pero me imagino que por ello no me irán a discriminar, no es así?
PDR alternative Windows BATCH Script using ufRAW
When PDR broke a couple of weeks ago I became paranoid about using my EOS-M. Been looking at alternatives. Here is one.
First, I mapped out all the focus pixels and sent a "bad pixel" map to dcraw, but it didn't interpolate around them. Apparently, dcraw only does that to original RAW files.
I then experimented with ufRAW http://ufraw.sourceforge.net/ and noticed that it has a function to find hot pixels and interpolate around them. It seems to do a good job on the pink dots.
Then, reading through Hudson's source code I noticed a quick BASH script that makes an avi file out of jpgs. That gave me the idea to modify it to process pink-dot files under Windows.
Here is the code I have so far. You could easily modify this to have ufRAW create TIFFs.
REM Inspired by https://bitbucket.org/hudson/magic-lantern/src/
REM 26f79287de2bf06a6ecaa320962b0fd25c0d68fe/modules/
REM Assumes you've extracted the DNGs from RAW using raw2dng
REM This is the simplest process, you can change ufRAW arguments
REM to create TIFFs instead, and with any other processing you want.
REM You could also change ffmpeg to save
REM ffmpeg higher quality use -vcodec ffv1 ?
REM ******** SET PATHS ***************
set "ufraw_exe=C:\Files2013_VidPhotoSoft\ufRAW\ufraw-batch"
set "ffmpeg_exe=C:\Files2013_VidPhotoSoft\RAWCinema\ffmpeg_latest\ffmpeg"
set "framerate=24"
REM **********************************
REM Delete target files first
del *.jpg
del video.avi
%ufraw_exe% *.dng --out-type=jpg --compression=100 --hotpixel-sensitivity=80
Goto ufRAW
Goto FFMPEG
:ufRAW
@echo off & setlocal
set "ext=jpg" & set "Count=100000"
for /f "tokens=*" %%a in (
'dir/b/on/a-d *.%ext%^|findstr/ie "[0-9]\.%ext%"2^>nul') do (
set/a "Count +=1"&&call :ReNum %%Count:~-6%% %%~xa "%%~na")
:FFMPEG
%ffmpeg_exe% -i 1%%5d.jpg -vcodec mjpeg -q:v 0 -r %framerate% video.avi
REM del *.jpg
Goto:End
:ReNum
if not exist %1%2 ren "%~3%2" %1%2
Goto:End
Anyone ever get the feeling that this project is abandoned?
Quote from: demetrisag on September 16, 2013, 06:22:26 AM
Anyone ever get the feeling that this project is abandoned?
Neither fire nor wind, birth nor death can erase our good deeds
Buddah
Quote from: Rewind on September 16, 2013, 09:02:22 AM
Neither fire nor wind, birth nor death can erase our good deeds
Buddah
True that! but how about them deeds keeps evolving! and getting greater and greater!
Anybody stopping you from doing that?
Welcome to open source world ;)
Quote from: a1ex on September 16, 2013, 11:19:04 AM
Anybody stopping you from doing that?
Welcome to open source world ;)
Man you have no idea how greatful I am from that!
But on the other hand I'm not one of those guys who understand nich about programming. So am on the side that waits and use whatever its given as much as I can! :)
PDR is not working correctly now with the latest builds on the M.
Gary
Quote from: maxotics on September 06, 2013, 08:30:42 PM
Rewind, Mixer2
Neither of your versions of PDR seem to be removing pink dots form my EOS-M in crop mode anymore. I think Gary might also be stuck. Any thoughts? I have a new build from 1%, which is about a week old. Mixer2, your version will process the files (though pink dots stay). Rewind, your version doesn't process; says they are corrupted. I'm using a Sigma 10-20mm EF on the EOS-M. Was working a couple of weeks ago just fine.
same here. i get messages from this version saying files are corrupted. I then try the older version and none of the pink dots get removed. I get different results from crop mode and non crop mode.
Gary
Hey folks,
I'm new to shooting RAW video, but I downloaded Raw Therapee on my Mac and have been running into a myriad of issues:
The program doesn't seem to eliminate the pink dots in my DNGs.
The program is extremely laggy/crashes often.
My Macbook Pro Retina is brand new/top of the line, so I can't account for the slowness of the program. What's the solution?
Quote from: benlen on September 25, 2013, 05:45:17 AM
Hey folks,
I'm new to shooting RAW video, but I downloaded Raw Therapee on my Mac and have been running into a myriad of issues:
The program doesn't seem to eliminate the pink dots in my DNGs.
The program is extremely laggy/crashes often.
My Macbook Pro Retina is brand new/top of the line, so I can't account for the slowness of the program. What's the solution?
That's exactly why you should use PDR tool ))
Jokes aside, Raw Therapee is the great program. Not the fastest, and has nothing to do with removing focusing pixels, but gives you almost full control over your dngs (advanced debayering algorithms, true color management etc). I'm using it a lot, and kind of satisfied. May be you should read some manuals first.
I have finally had success mapping out the static focus pixels, sending a bad pixel list to dcraw, and getting good TIFFs out of it. In the next day or so I hope to have it integrated into my Windows scripts for processing RAW into either focus-dot free TIFFs or a video clip with 422 color.
@Rewind, this stuff is hard!!! :) I couldn't get Windows to read the pixel values of a bayer pattern output by dcraw. I had to use the libTiff.net library. PDR has never completely eliminated all the dots for me, often doesn't work ("file is corrupted") and is difficult to understand and configure (dot start/end/offsets). I tried Raw Therapee, but it crashes when you try to use too many DNGs. I think Mixer2 may be back at some point. Hopefully what I'm learning will help PDR too.
Quote from: maxotics on September 25, 2013, 03:32:01 PM
@Rewind, this stuff is hard!!! :)
To map pixels is not too hard really, but what is hard is to achieve good results with interpolation algorithm.
Try to experiment with images like this (with colorful contrasty edges):
(http://ipic.su/img/img7/fs/kiss_40kb.1380116978.jpg)
And pay an attention to the edges behavior. Don't know much about dcraw and it's hot pixel removal methods, but neither of raw converters out there do this job properly.
This picture below shows how PDR is better than ACR at this point:
(http://moredo.ru/raw/PDRvsACR.gif)
so it would be interesting if you make similar test with dcraw
Quote from: Rewind on September 25, 2013, 04:03:27 PM
To map pixels is not too hard really,
If that were true, wouldn't PDR work perfectly :)
The problems you're showing, I now believe, are not from weak de-bayering algorithms, but focus-pixels that were not removed properly.
Here is a picture of the focus pixels in 1280x720. In addition to the bright red pixels, there are pixels that are blue, green and some pixels aren't bright, but are actually turned off.
(http://maxotics.com/images/pixelmap1280x720color.jpg)
And here in BW
(http://maxotics.com/images/pixelmap1280x720bw.jpg)
My current algorithm to find focus pixels locates about 4,112 pixels. I believe there are more (because I still see some). Eventually, I think I'll get them all.
dcraw has a few different de-bayering algos.
What you're also noticing, I think, in your images, is that some focus pixels don't fire unless they're on a hard edge.
What I don't know is if this is really true, that they are activated on hard edges, or focus pixels always change for focus, but the changes is not noticed when nearby pixels are also properly exposed, so to speak--that is, they blend in.
If they don't fire when not focused, then we probably want a way to leave them alone.
If they always fire, but is not always noticed, then we should just remove them all.
I suspect that removing them all will not noticeably degrade the image and that PDR, in it's current state, make that process more complicated than it needs to be. Of course PDR is in a great position to do all this stuff. Java just isn't my strong suit.
QuoteIf that were true, wouldn't PDR work perfectly
PDR actually do works perfectly for 650D (at least in terms of removing all the dots).
QuoteThe problems you're showing, I now believe, are not from weak de-bayering algorithms, but focus-pixels that were not removed properly.
No. I told you several times, the positions are not a problem. We know exact positions of all these pixels with absolute accuracy. Period. Edges are the problem. You can use any simplified method to interpolate in uniform areas, but you forced to use some form of adaptive interpolation if you're concerned in quality on edges.
QuotePDR, in it's current state, make that process more complicated than it needs to be.
No. The math behind this method seems too complicated only for those who didn't study the actual paper describing it, but it works better than any raw processor built-in filtration I've seen so far.
Quote from: Rewind on September 25, 2013, 05:55:15 PM
PDR actually do works perfectly for 650D (at least in terms of removing all the dots).
It doesn't work for my EOS-M. Others have complained.
I've posted this video showing the dots, I keep point them out you keep saying it's not a problem. I used your PDR on these clips.
Quote from: Rewind on September 25, 2013, 05:55:15 PM
No. I told you several times, the positions are not a problem. We know exact positions of all these pixels with absolute accuracy. Period. Edges are the problem. You can use any simplified method to interpolate in uniform areas, but you forced to use some form of adaptive interpolation if you're concerned in quality on edges.
All I know is it isn't working for me. It isn't for others. Knowing it works perfectly for your camera doesn't help me. Sorry!
Quote from: Rewind on September 25, 2013, 05:55:15 PM
No. The math behind this method seems too complicated only for those who didn't study the actual paper describing it, but it works better than any raw processor built-in filtration I've seen so far.
The math of static pixels is easy. The math describing pixels as start/end/offsets, from center image is MORE COMPLEX, period. I don't care which method is used. You seem to think I do. All I know is that PDR is not working for me (and others). You're implying that I'm not smart enough to understand the calculations, ignoring the fact that it is not working for me on your calculations (not that I blame you, you don't have the camera, and I thank you for your time/effort). I talk static positioning because if I'm going to solve a problem I believe in lowering complexity, if I can.
I agree, it doesn't matter what calculations you use when it comes to RAW processor built-in filtration. I've used both Raw Therapee and ufRAW and know as well as you do!
I don't think you're taking my research and thoughts on this matter seriously. You're a bit defensive about PDR (which again, is not working for everyone). I'd love it if you either made PDR work for everyone or worked with me a bit on the approach I'm taking. Or, I'm happy to work with you on PDR.
Soon I will have an alternative to PDR. I would think that a good thing. I wish you thought so too.
What we all need is a way for some software to scan the DNGs, find the pattern, and write that out, either as PDR formulas or static locations. That what we should probably talk about? Right?
:D
anything new happening with PDR? I am very frustrated with it not working most of the time. I still get corrupted file error messages at the end, but not all the time. I cannot tell when or why this happens. I have the crop and non crop PDR and sometimes they work but not a lot of times. Again, I don't see why or when it does this. I have tried many different ways shooting raw and obviously none of it can be used with pink dots still there. I am using Sept 30 version of ML
Gary
all my older raw files also do not work with pdr. so something changed in the newer ml ver.
Gary I work on it every night. It is VERY frustrating to me too. In a sense I'm glad you're having problems too because I wonder if I'm just crazy. I don't know why others don't have these problems. I suspect that people don't shoot enough in real-world situations.
This is what I believe so far. There are more than 4,000 pixels used for focus. They are clumped together in groups. There are few green sensels used, mostly red and blue. Sometimes the pixels fire when the scene is dark, sometimes when over-exposed. PDR gets rid of the basic focus pixels, but does not remove the pixels that fire when over-exposed (at least for me). Removing all these pixels will probably degrade the image more than I'd like.
Fortunately, these pixels are in definitive patterns. Each pixel in a horizontal line is separated by 24 pixels. I'm working on an Algo that figures out when the pixels fire, for each individual frame, and then will interpolate like PDR (hopefully port over their routine, which, as Rewind points out, works very well).
The ultimate solution should be to incorporate these alogrithms in raw2dng. If I end up finishing it I'll make it something like hafraw2dng (Hybrid Autofocus).
In short, PDR works great for an edge to edge properly exposed frame. Other shots fall apart.
For a whole host of programming headaches, this is not going quickly. I am working on it. And on the moire issue too, which has some similarities I'm seeing.
Every other day I think of giving up on ML with this camera. Glad to know I'm not alone ;)
max, don't give up, please. I modified my post while you replied. All my old raw files will not pdr at all. you know a hell of a lot more than i do on this. is there some way where a pdr app can just scan the first image and find all the crap and then remove it all for every frame in the seq? something keeps changing cus pdr works then later it won't work at all even on the older raw files that used to work.
Gary
if we can't get some sort of pdr to work, then we don't have raw video and then the camera is not much better than any other consumer cam. We really need pdr, raw video and audio to work to make this little david beat goliath just like Trammel did in the beginning with his 5dm2 and the giants.
If you've shot them all at the same resolution, send me a DNG from a frame with the most pink dots and I can look into it. It has to be a DNG because I need to see the original sensor readings before de-bayering. You can post link here or email me at
[email protected] Sorry I don't have a fix right this second. I'm sitting on a lot of useless footage too!
If you have a difficult shot where the interpolation struggles, and a list with bad pixels (just a list of x,y points), send it to me; I'd like to try the median filter that I use in cr2hdr to fix the hot pixels.
Guys with EOS-M, just one note. If you will send examples to A1ex, please make something with pronounced vertical and horisontal edges and with high contrast detail. Because these are the things in which different interpolation algorithms.. well... differ.
I mean, if you shoot your cat, then median filter is obviously ok, but if you try something like this, it's more interesting (look closely to the edges)
(http://moredo.ru/raw/PDRvsACR.gif)
Yep, this shot in DNG with a list of pixels would be perfect.
Hi A1ex, I've uploaded a bunch of files for you at maxotics.com/photos Enter "focusdotshotpixels", then you'll see a link for download
Rewind, if you're interested, you can see the focus pixels in the blown out areas AFTER PDRing in "1280x434_PDRd_HotPixels_BlownOutAreas" I wish you'd go shoot more so you can confirm that this won't be a problem for you (others) down the road. The edge pixels you're seeing, is part of the problem PDR struggles against. There are too many focus pixels for it to deal with without removing more than one would want.
A1ex, you can see hot pixels in the "HotPixelsInRoad" example. (There is a lot of moire with this camera in non-crop mode.)
In the zip, you'll find frames that I shoot in a light table to get focus dots. I then use Photivo (as Rewind helped me with) to isolate the dots through elimination of green channel and other contrast/image type adjustments. I want light pixels on black. I wrote a .NET program that will scan the image and create a dcRAW format hot pixel list. I did one for both red and blue and red only.
In the file "1280x434_PDRd_HotPixels_BlownOutAreas_SeeDeadPixels.jpg" you will see that these hot pixels are actually black pixels from the sensor. My hot pixel locator would have to be adjusted to find those. I can do that later. (This while I'm porting this over to C# and trying to integrate/use g3gg0's code, and C stuff from the libraries---my head is spinning!).
I've been thinking about the median filters a bit too. I think there is an even better application for what you're looking into--reducing moire.
Look at "1280x434_MoirePattern.dng" Seems like a lot of interference patterns that could be predicted/isolated?
It seems to me, much of the moire is because the chroma information for what should be one scan line is split up into 2, so to speak. I understand moire isn't talked about that way. But I believe a median filter on, say a group of 3-7 pixels, with a lower/higher line, slightly offset to right or left, will eliminate the moire, though will blur the image somewhat (which is what a VAF filter would do anyway). Such moire reduction would need a rules-based algo matched to some sort of median filter. Anyway, I may be crazy, but I believe one/we could do through programming what the VAF fitler does physically (better join skipped lines from the sensor).
I understand what you may really want is a RAW frame with some hot pixels mapped to dcRAW. I could shoot something with my 50D (which doesn't have the Hybrid AF dot issue). But I hope you'll segue into our EOS-M nightmare ;)
As you can see, I have some equipment set up for testing. If you need any types of images (you too Rewind), let me know!
QuoteRewind, if you're interested, you can see the focus pixels in the blown out areas AFTER PDRing
I'm not interested since these areas are blown out. There is no problem at all to delete them.
Rewind, easy for who? I'm trying to get the camera to work for film-makers, not academics. I don't expect them to go through their videos, frame by frame, and eliminate hot pixels. I respect if you're not interested, but I think it sad you won't join us in this quest. Why are you fighting me on this?
I mean, I think you have done great work with this and have said it a million times. But anything I say seems to bother you. I'm sorry.
Quote from: maxotics on October 02, 2013, 05:53:18 PM
Rewind, easy for who? I'm trying to get the camera to work for film-makers, not academics. I don't expect them to go through their videos, frame by frame, and eliminate hot pixels. I respect if you're not interested, but I think it sad you won't join us in this quest. Why are you fighting me on this?
Oh, man. Excuse me. I mean, raw processors doing that nicely for us.
All the cameras with hybrid focusing system have this issue of overexposed areas, but ACR for example eliminates them easy.
Since i'm using ACR in my workflow, i never see these pixels in overexposed areas. But I understand, not everyone wants adobe, and some of the raw processors has no dead pixel filters may be
By the way, just tried with Photivo and RawTherapee. Just turn Hot/Dead pixel filter on, and overexposed areas are pure as driven snow ) This is what i meant by 'easy'
There is no any point to apply some fancy interpolation algorithm to overexposed areas, since they are... well, overexposed = plain white. This is what hot\dead pixel filters are done for.
For the 'real' focusing pixels PDR does the job. So I just can't see the problem anymore.
The dots in the window ARE focus pixels. They are not hot/dead pixels. PDR is not removing them because they are dormant when the sensor pixel is properly exposed. Our definitions of what works is different. PDR, to me, half works.
I have Adobe. I can go out and buy a 5D3 and be done with all this sh_t ;) But I love the small size of the EOS-M and think this can be a killer camera.
PDR is better than nothing, but it is not a complete solution. As Gary and others have pointed out, it needs to do more than apply a calculated map of pixels that need interpolating. It needs some smart analysis up front. It either needs to figure out those maps and apply them, or do what I think is a better, harder, but more fool-proof solution--find all patterns and eliminate them.
You're one of the few people around who can make this a reality. But you keep insisting PDR is doing all it can. Why would you WANT another program to remove hot pixels in blown out areas? They're focus pixels. They should be ALL taken care of.
Again, we have a disconnect ;) Are they, or are they not, focus pixels?
The pixels in overbrighten areas ARE NOT usual focusing pixels. You should NOT interpolate them all over the image because you will loose detail! They appear ONLY in overexposed areas. This is why you'd better just filter them hell off — this is darn easy.
PDR is not ideal — that's fact! But in terms of interpolation algorithm only. PDR removes ALL of the focusing pixels. But neither mine, nor Mixer2's algorithm is not ideal. This is what we should work on.
Rewind, Almost everyone I know who argues like you do does well in law school! :) You'll at least a double threat. Programmer and Trial Lawyer!
Okay, they're not the "usual" focus pixels. We won't put them in focus-pixel prison :) I'm going to continue on with my strategy of building a C#/C workbench to deal with these issues.
If you can improve PDR, I'm sure that would be much appreciated. But I think the future is incorporating what you know in existing C routines in the general ML sphere.
Quote from: maxotics on October 02, 2013, 06:37:02 PM
But I think the future is incorporating what you know in existing C routines in the general ML sphere.
Sounds reasonable.
stay calm guys. we all want the same thing. I am just confused as to why all my older test raw files that used to PDR ok are not doing anything now. By that I mean I use the original untouched raw file again from the start with both of the pdr apps I have for crop and non crop. i think it was version 8.
I will try and make some new short raw test files and clean dng files to post for anyone t see and try out. also a PDR dng showing all the pink dots.
Gary
Thanks Gary. We're arguing, not fighting. That would be good if you posted some DNGs gary. We need to get to the bottom of differences between ML builds, if there are any. I'm trying to organize my collection of stuff so we can better work through these issues in the future. Rewind, I don't have your camera, so any files you want to share would be great.
@maxotics: your pink dot list (badpixels_red_and_blue.txt) is incomplete. Here's why. (http://a1ex.magiclantern.fm/bleeding-edge/1280x434_FocusDotsExampleLightTable.DNG)
here one dng filke that has all the pink dotsa after running pdr.
https://www.transferbigfiles.com/fac70a0d-9e74-4ce0-a056-96f38f3b3a2f/XsYoIriLIBw7_x9kHlMCdA2
gary
here one dng file with no pdr applied.
https://www.transferbigfiles.com/95034153-aa39-4f7c-8b7c-967af22b2b91/fG0Wfu1hgu8mLQGv0slcDA2
gary
They look the same. Both have pink dots.
here is one dng where pdr just worked. i just shot this at 1280 non crop 24 fps. i applied the raw file to pdr ver 8
https://www.transferbigfiles.com/d02e118a-045a-4ffe-9fbb-5388a5c9ee0f/hPUFDyJ_cEnxKWMVcwqgBQ2
the other files i did at 1728 30 fps, if i remember correctly.
gary
Quote from: maxotics on October 02, 2013, 10:33:15 PM
They look the same. Both have pink dots.
correct. pdr did nothing to fix it. i also tried both versions of pdr with the same results
Interesting that a simple 3x3 chroma smoothing like the one from cr2hdr seems to fix the pink dots without any prior info about them.
http://a1ex.magiclantern.fm/bleeding-edge/1280x434_FocusDotsExampleLightTable-cs3x3.DNG
here is one more dng file where pdr did not work. this is from the raw file i shot earlier today. the raw file is fresh and i ran it thru pdr successfully. i then ran raw2dng and here is one frame form that sequence that shows pdr did not work.
https://www.transferbigfiles.com/b23d6469-3344-4956-8ab0-2034652e14ce/IDQnDV3dn3_zwJT2oaXFRg2
gary
the only thing different in mine were the earlier one being shot at 1728 non crop and the second one where pdr worked being 1280 non crop
Quote from: a1ex on October 02, 2013, 10:44:39 PM
Interesting that a simple 3x3 chroma smoothing like the one from cr2hdr seems to fix the pink dots without any prior info about them.
A1ex, can you apply the chroma smoothing to this dng (http://moredo.ru/raw/000000.dng), so I can compare results with PDR?
Here you go: http://a1ex.magiclantern.fm/bleeding-edge/rewind.dng
Can you generate the jpeg for pixel-peeping?
Edit: here are some experimental versions of raw2dng that perform chroma smoothing:
http://acoutts.com/a1ex/raw2dng_cs3x3.exe
http://acoutts.com/a1ex/raw2dng_cs5x5.exe
They may help with color aliasing too, so try them both and see which gives the best results.
Quote from: a1ex on October 02, 2013, 11:08:56 PM
Can you generate the jpeg for pixel-peeping?
Two jpegs:
by Chroma smoothing (http://moredo.ru/raw/compare/Croma-Smoothing-1.jpg)
by PDR (http://moredo.ru/raw/compare/PDR-1.jpg)
Color fringing is more pronounced in first picture. This is quite expected result i think.
(http://moredo.ru/raw/compare/compare.gif)
Looks good! However, I would think that it would forfeit 30% of your color information--going down that slippery H.264 slope. Will do more tests... Thanks! (BTW, besides g3ggo, has anyone ported any of these routines/libraries to C#?)
Updated the chroma smoothing algorithm to reduce fringing.
http://acoutts.com/a1ex/raw2dng_cs3x3_v2.exe
http://acoutts.com/a1ex/raw2dng_cs5x5_v2.exe
http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_v2.dng
http://a1ex.magiclantern.fm/bleeding-edge/rewind_5x5_v2.dng
This update will also improve dual ISO shots in aliased areas. Thanks for the relevant DNG!
Pixel peepers: which one do you like best and why?
original dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_original.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_original.png)
3x3 (first version) dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3.png)
3x3_v2 dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_v2.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_v2.png)
3x3_median4 dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_median4.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_median4.png)
3x3_ahdlike dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_ahdlike.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_ahdlike.png)
Quote from: a1ex on October 03, 2013, 12:07:38 PM
Pixel peepers: which one do you like best and why?
ahdlike has less color artifacts
For me the last one looks best, because there is nearly no color aliasing.
Good, here's the raw2dng that does it:
raw2dng_cs3x3_ahdlike.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike.exe)
by the way, a1ex, since green channel stays untouched (focusing pixels are red and blue only), is it possible to calculate chroma on R and B channels only? This way we will preserve more detail.
I mean, somehow make the AHD-like calculations on R&B channels only
I'm not touching the green channel.
Quote from: a1ex on October 03, 2013, 12:22:56 PM
Good, here's the raw2dng that does it:
raw2dng_cs3x3_ahdlike.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike.exe)
Thank you, it looks very good.
Small update: noticed that "ahdlike" affected noise structure in Gary's shot (which is extremely underexposed), so I've added a threshold below which it no longer tries to choose the interpolation direction (it averages both).
raw2dng_cs3x3_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe)
raw2dng_cs5x5_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs5x5_ahdlike_noise.exe) (for really noisy shots)
For regular shots, the noise threshold should not make any difference (if it does, post some test shots that show the difference). For noisy shots, the noise should be whiter (less correlated, easier to clean).
Unfortunately, this method produces pretty obvious artifacting in small detail, as well as expectable overall color detail loss.
(http://moredo.ru/raw/compare/compare2.gif)
Then, extract a list of pink dots and only interpolate these points. The list I've received was incomplete.
Quote from: a1ex on October 03, 2013, 01:45:58 PM
Then, extract a list of pink dots and only interpolate these points. The list I've received was incomplete.
This is exactly what we are doing in PDR, but to try this AHD method, I need it's math description. Would you mind to share one?
Will commit it soon.
What puzzles me right now is that all these new smoothing methods seem worse when used with cr2hdr (at least on the resolution charts and some tricky video frames). So I'll commit after I'll find out what's going on.
Commited:
- initial version: https://bitbucket.org/hudson/magic-lantern/commits/4c1b0c2e65eb
- v2: https://bitbucket.org/hudson/magic-lantern/commits/2d0671859c1e
- ahd-like: https://bitbucket.org/hudson/magic-lantern/commits/b24b9f189203
- with noise threshold: https://bitbucket.org/hudson/magic-lantern/commits/530908efe192
It's disabled by default; define CHROMA_SMOOTH to enable it.
The algorithm is inspired from dcraw median_filter.
Some details:
/**
* for each red pixel, compute the median value of red minus interpolated green at the same location
* the median value is then considered the "true" difference between red and green
* same for blue vs green
*
*
* each red pixel has 4 green neighbours, so we may interpolate as follows:
* - mean or median(t,b,l,r)
* - choose between mean(t,b) and mean(l,r) (idea from AHD)
*
* same for blue; note that a RG/GB cell has 6 green pixels that we need to analyze
* 2 only for red, 2 only for blue, and 2 shared
* g
* gRg
* gBg
* g
*
* choosing the interpolation direction seems to give cleaner results
* the direction is choosen over the entire filtered area (so we do two passes, one for each direction,
* and at the end choose the one for which total interpolation error is smaller)
*
* error = sum(abs(t-b)) or sum(abs(l-r))
*
* interpolation in EV space (rather than linear) seems to have less color artifacts in high-contrast areas
*
* we can use this filter for 3x3 RG/GB cells or 5x5
*/
Hi Alex. I'm thrilled you're working on incorporating other debayering algos and filters into RAW conversion. For the non focus-pixel cameras, your work is going to pay instant dividends. 1% has been asking about noise processing, and what you're doing will improve on that. When I said above that you're losing 30% color information I didn't mean to also imply that that was a bad thing. Even with less color information your work should bring footage closer to H.264 in out-of-the-box usability, with plenty of extra dynamic range. A great option to have, which I've wanted too.
I just want to point out again, because I shoot every day and I'm not sure you or anyone will see this who doesn't. There are too many focus pixels then can be dealt with effectively with a general algo in certain shots. You alluded to this with not having a complete bad pixel list. One of the problems is that the main red focus pixels are easy to remove (with PDR for example), but there are blue pixels and other red pixels, a pixel or two away from the main ones. And there are dark pixels. So you might interpolate around a main red pixel fine, but the blue pixel fired and another red went black or something, and then you generate another hot pixel. You can expand your interpolation around more pixels, but the more you do, the more you degrade the whole image.
Again, not a big issue with the other cameras.
I guess what I'm saying is you want to have realistic expectations about what you can accomplish, at least with the focus pixel footage.
Continuing on your current effort...
One of the problems I have is there is no easy workflow where I can send a bad pixel list to dcRAW, for example, and get a DNG back. If we had something like raw2dng with the bad pixels interpolated around, like PDR (using a bad pixel list), it would be easier to see what's falling apart with what you're doing and what's a focus pixel problem. I think this would be helpful to us all (I'm working on this alone, but am not up to speed with C yet).
raw2dng_removebadpixels.exe (A version that either interpolates /deadens pixels identified in a pad pixel list. Similar to PDR)
dng2dng_alexfilters (does your filtering around the stuff)
With those two workflows we can compare footage with focus dots removed, AND exactly how much your stuff cleans up afterwards. Right now, hard to know where the problems come from.
Thanks!
Quote from: a1ex on October 03, 2013, 03:22:00 PM
The algorithm is inspired from dcraw median_filter.
On that subject, you may find this interesting...
http://ninedegreesbelow.com/photography/floating-point-dcraw.html
Integrating a badpixel list is pretty simple. Instead of sweeping the entire image in chroma_smooth, use an array of x,y coords from the file.
These pixels seem to follow a regular pattern, so just include all of them, even if they are not always noticeable.
maxotics, if there is any combination of resolution / aspect ratio, in which PDR doesn't work for your EOS-M, send me an example along with a short series of underexposed flat frames to stack, so i can build the new map and tune up the PDR for you.
Upd. Just a 50-80 frames of flat field would be fine. No need a reference frames at all. Just like you did last time recently.
Rewind, thanks! Let me take a little time to put together a good reference set for us all. I know Gary would love a new map too!!!
Alex, here are a couple of shots that show what you're up against. Rewind might be right, a little tuning and the problem can be eliminated. But you can see how there are a lot of pixels used for focus type work, both red and blue. It's difficult for me to calculate the bad pixel list because the software I built to identify them still needs work. It isn't as easy as it looks because the pixels are not uniform in brightness. These images are AFTER a lot of Photivo adjustments.
I would just download these
http://maxotics.com/images/1600x542_20130929.jpg
http://maxotics.com/images/1600x542_20130929.bmp
Here is 1280x720,
(http://maxotics.com/images/EOSM1280x720FocusPoints.jpg)
What might work well is a combination of basic hot pixels PDRd away using Rewinds new version and then your stuff for clean-up. (Wish the dev of RAWanizer was around or it was open source).
Quote from: a1ex on October 03, 2013, 01:32:49 PM
Small update: noticed that "ahdlike" affected noise structure in Gary's shot (which is extremely underexposed), so I've added a threshold below which it no longer tries to choose the interpolation direction (it averages both).
raw2dng_cs3x3_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe)
raw2dng_cs5x5_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe) (for really noisy shots)
For regular shots, the noise threshold should not make any difference (if it does, post some test shots that show the difference). For noisy shots, the noise should be whiter (less correlated, easier to clean).
I have noticed that a lot of my raw files show up in ACR as underexposed, yet when I make the movie, I am exposed ok. I am trying to push this camera to it's limit for shooting documentaries with low light, available light. Also having the camera being so small does not draw attention to when I shoot, looking like I am tourist.
Obviously i run into a limit of raising ISO and having more digital noise then what I like. I am still seeking that limit of acceptance. I can use a light if you want for now on these tests. But, why are the underexposed when the LCD shows a decent exposure when shooting?
Gary
Expose to the right (try ETTR) and forget the myth that raising ISO increases noise (it actually decreases noise, until around ISO 1600; after that you get no more improvement in noise, you only lose highlight detail).
Quote from: a1ex on October 03, 2013, 12:07:38 PM
Pixel peepers: which one do you like best and why?
original dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_original.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_original.png)
3x3 (first version) dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3.png)
3x3_v2 dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_v2.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_v2.png)
3x3_median4 dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_median4.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_median4.png)
3x3_ahdlike dng (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_ahdlike.dng) png (http://a1ex.magiclantern.fm/bleeding-edge/rewind_3x3_ahdlike.png)
i don't see the green fringing on the last pic in the list
gary
just noticed the other replies on this. I am trying to catch up reading the replies I have not seen
I'm using this build on some non-dotted but noisy footage, seems to be helping a bit.
Sometimes it helps, sometimes not.
Old Style
http://www.filedropper.com/m28-2303-000239
3x3 exe
http://megafileupload.com/en/file/457243/M28-2303-000239-dng.html
I just tried shooting some raw 1290 non crop mode and i then dropped the raw file in to the raw2dng_cs3x3_ahdlike_noise.
is this correct? no PDR? I then put the dng sequence into AE CC with ACR adjustment and then used the Neat Video filter. Looks pretty good except i see one vertical row of pink dots along the left edge of the frame.
gary
never mind. i must have had a bad corrupted file. newer tests look good on crop and non crop so far.
Quote from: gary2013 on October 04, 2013, 02:24:56 AM
ii just tried shooting some raw 1290 non crop mode and i then dropped the raw file in to the raw2dng_cs3x3_ahdlike_noise.
is this correct? no PDR?
Did you missed the whole latest discussion?
http://www.magiclantern.fm/forum/index.php?topic=6658.msg80241#msg80241
Yes, you can use this method of chroma aliasing, but you losing detail, and if you concerned in quality, you'd better read the latest posts )
Quote from: a1ex on October 03, 2013, 01:32:49 PM
raw2dng_cs3x3_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe)
raw2dng_cs5x5_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe) (for really noisy shots)
I don't know if this matters but both links are pointing to the same file: raw2dng_cs3x3_ahdlike_noise.exe
Rewind, I've been shooting only in 1280x720 these past few days and your latest version of PDR is working great. I'm going to give up on non-crop mode RAW on the EOS-M for now.
My guess is that a combination of your PDR and Alex's latest raw2dng options will cover all the shooting situations. I don't mind if I lose some quality on shots where there is a lot of dots in blown out areas. As long as I know I HAVE a solution at hand.
At some point we need to write an updated man page with links to the latest software and a shooter's guide for anyone with all these questions.
Quote from: maxotics on October 04, 2013, 06:52:49 PM
Rewind, I've been shooting only in 1280x720 these past few days and your latest version of PDR is working great. I'm going to give up on non-crop mode RAW on the EOS-M for now.
My guess is that a combination of your PDR and Alex's latest raw2dng options will cover all the shooting situations. I don't mind if I lose some quality on shots where there is a lot of dots in blown out areas. As long as I know I HAVE a solution at hand.
At some point we need to write an updated man page with links to the latest software and a shooter's guide for anyone with all these questions.
i also saw that with crop mode 1290x720 i was not getting the aliasing on lines like i was getting with non crop. I tried a few different ways just using alex's raw2dng_cs3x3_ahdlike.exe and i did not get any pink dots. and that was no pdr applied. i just use adobe PS CC with ACR to ge the tif seq made and then Neat Video filter in PPro CC.
Gary
Quote from: Rewind on October 04, 2013, 09:41:24 AM
Did you missed the whole latest discussion?
http://www.magiclantern.fm/forum/index.php?topic=6658.msg80241#msg80241
Yes, you can use this method of chroma aliasing, but you losing detail, and if you concerned in quality, you'd better read the latest posts )
no, i have read everything. i am only going by what i see as a result on a large 42" hd display using HDMI. It looks preety good from what I see what I shot. I will try some more, re read what was said before and see if there is anything I am missing.I am also using ETTR as Alex suggested so my exposure now do look better in ACR.
gary
Quote from: maxotics on October 04, 2013, 06:52:49 PM
Rewind, I've been shooting only in 1280x720 these past few days and your latest version of PDR is working great. I'm going to give up on non-crop mode RAW on the EOS-M for now.
My guess is that a combination of your PDR and Alex's latest raw2dng options will cover all the shooting situations. I don't mind if I lose some quality on shots where there is a lot of dots in blown out areas. As long as I know I HAVE a solution at hand.
At some point we need to write an updated man page with links to the latest software and a shooter's guide for anyone with all these questions.
where is this latest version of PDR you mention?
Gary
Quote from: gary2013 on October 04, 2013, 08:41:52 PM
where is this latest version of PDR you mention?
try this one http://moredo.ru/raw/PDR Rewind for EOS-M crop.zip (http://moredo.ru/raw/PDR%20Rewind%20for%20EOS-M%20crop.zip)
One note: Although this build is dedicated to crop mode specially, You should NOT use the 'crop mode' option with this build. Just select EOS-M and hit 'convert'
Quote from: Rewind on October 04, 2013, 08:52:13 PM
try this one http://moredo.ru/raw/PDR Rewind for EOS-M crop.zip (http://moredo.ru/raw/PDR%20Rewind%20for%20EOS-M%20crop.zip)
One note: Although this build is dedicated to crop mode specially, You should NOT use the 'crop mode' option with this build. Just select EOS-M and hit 'convert'
thank you. are you saying do not shoot raw with crop mode on? Then why is your pdr called crop?
Quote from: gary2013 on October 04, 2013, 09:00:12 PM
thank you. are you saying do not shoot raw with crop mode on? Then why is your pdr called crop?
))) I'm talking about these settings
(http://ipic.su/img/img7/fs/kiss_11kb.1380913077.jpg)
Quote from: Rewind on October 04, 2013, 09:02:46 PM
))) I'm talking about these settings
(http://ipic.su/img/img7/fs/kiss_11kb.1380913077.jpg)
ok. i never knew crop mode was there . i always used eosm
Quote from: Rewind on October 04, 2013, 08:52:13 PM
try this one http://moredo.ru/raw/PDR Rewind for EOS-M crop.zip (http://moredo.ru/raw/PDR%20Rewind%20for%20EOS-M%20crop.zip)
One note: Although this build is dedicated to crop mode specially, You should NOT use the 'crop mode' option with this build. Just select EOS-M and hit 'convert'
this version worked good for me on a clip at 1280x720 crop mode on. I used the original raw2dng to convert the files after pdr. i ran pdr using the raw file, not all the dng files.
gary
I am seeing some pink dots in the raw files I shot at 1536x520 "non crop mode" using this version of PDR.
https://www.transferbigfiles.com/5706a02e-9af5-4ec7-b334-688d3e5cbe10/V9AvG-Ha2ICC4HP2G3bqyg2
I tried some more raw video testing today and PDR is not working. I am using different resolutions and ratios.It seems to me PDR is only working for 1280 x7 20. There are many variables people can use for resolutions and aspect and I think PDR cannot handle or figure out the differences. Leads me back to thinking of how we might need some app that can scan one image of each new sequence of DNGs and determine where the pink dots are and then do its magic to remove them.
Gary
@Rewind: MLV has metadata about crop offset (the exact position of each frame within the full raw buffer), exactly for this purpose. To get the full (not cropped) raw buffer, take a silent picture.
Has there been anything new on this issue of PDR? Max, any new advancements on your project?
Gary
Quote from: gary2013 on October 12, 2013, 04:31:22 AM
Has there been anything new on this issue of PDR?
What issue?
Gary, I've been using Rewind's "crop mode" version for any videos I do in 1280x720 and the latest original for everything else and it is working okay as long as I don't shoot any blown out areas.
I have separate folders "PDR", "PDR Rewind Crop Version" with each version. I run the RAW through RAWanizer to see what aspect ratio the clips are, and then PDR accordingly. Though I do everything crop mode now. The other isn't worth the moire risk.
I have a method for generating focus pixels and creating a bad pixel list for dcRAW type convesion, but when PDR works, as Rewind says, that's the best method (it interpolates better than dcRAW). However, i still hope to get that working with the same interpolation method as PDR and incorporate into raw2dng or mlv2dng, etc.
What we really need, for now, is a PDR that has updated drop down selections. I looked at the code but could see it wouldn't be a quick fix for me. I imagine it's a little complicated or Rewind would have done the same. But Rewind, if you can incorporate your crop mode with the earlier stuff that would be optimal.
We need something like
650 Normal
650 Crop
EOS-M Normal
EOS-M Crop
Even if both are the same, would allow adjustments to each if something changes in builds.
I am still trying to understand why I do not have any pink dots when I do not use the PDR and I only use Alex's latest
raw2dng_cs3x3_ahdlike_noise.exe along with the Neat Video filter for noise reduction in Premiere Pro editing.
Gary
Here's another raw2dng that does a little less chroma smoothing (it only considers top, bottom, left and right neighbours). It still removes all the pink dots.
raw2dng_cs2x2_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs2x2_ahdlike_noise.exe) (fewer artifacts)
(actually this kernel is more like 2x2+1)
Previous versions:
raw2dng_cs3x3_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs3x3_ahdlike_noise.exe)
raw2dng_cs5x5_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs5x5_ahdlike_noise.exe) (for really noisy shots)
Original raw2dng (without chroma smoothing):
raw2dng.exe (http://acoutts.com/a1ex/raw2dng.exe)
(http://acoutts.com/a1ex/pink.gif)
@gary: this raw2dng applies a median filter to red-green and blue-green channels, and since the pink dots are only on red and blue channels, it removes them all (without requiring PDR). The disadvantage is that it doesn't know where these pink dots are, so the filter is applied to all the red and blue pixels.
Sometimes this filter does good (it may reduce noise or hide some color aliasing), but in other situations it blurs fine details or introduces artifacts. Experiment with it and see what works best.
To remove pink dots with minimal alteration of the original raw data, you still need to use PDR (when it works), with the original raw2dng (without chroma smoothing).
A1ex, I think you asked about this before you started doing this, but now that you've got the smoothing part done it may be a good time to revisit.
I created a set of files for you if you want to smooth only the focus pixels. There are three 24 frame RAW files at these resolutions
1728x584
1600x642
1280x720 crop mode
Zip is 200meg
https://app.box.com/s/6q3nmjih6g54mfeucgo1
I also included bad pixel lists (dcraw format).
I shot clouds out of a dirty window. Should be good enough for testing I hope. The 1280 bad pixel list may be too aggressive.
All the pixel lists can probably be improved.
If you're up to this and it works, I can generate bad pixel lists for all resolutions. At some point I hope to upload my bad pixel mapper to bitbucket too.
Having some version of raw2dng taking care of this would be so freaking amazing as everyone with these cameras would tell you! :)
what does AHD mean? i couldnt find anywhere
Quote from: britom on October 15, 2013, 07:46:22 PM
what does AHD mean? i couldnt find anywhere
Adaptive Homogeneity-Directed interpolation.
@Rewind thanks a lot for the quick answer
Hi Guyz i downloaded the Tool
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover.zip
but when i click or double click on PinkDotRemover (Executable Jar File ) it does not open?! I intalled of course Java have both installed 32bit and 64bit.
Regards
HEY ADMIN PLEASE DELETE MY ACCOUNT! because my comment must be approved bevor it is Online,I take my help elsewhere.
Quote from: a1ex on October 15, 2013, 01:07:42 PM
Here's another raw2dng that does a little less chroma smoothing (it only considers top, bottom, left and right neighbours). It still removes all the pink dots.
raw2dng_cs2x2_ahdlike_noise.exe (http://acoutts.com/a1ex/raw2dng_cs2x2_ahdlike_noise.exe) (fewer artifacts)
(actually this kernel is more like 2x2+1)
I've just tested it and it seems to work pretty well, remove a lot of dots/artifacts where PDR doesn't !
I've updated my RAW post processing tutorial and allowed myself to link your raw2dng version, I quoted you of course (http://www.gna-productions.com/tutos/how-to-process-magic-lantern-raw-videos , part 3), I hope you don't see problems.
Thanks, and again (not again but it was in my mind since the beginning) : good job !
If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)
I got a pop up from Java saying there is a newer version and after installing it, I can't run PDR. I have tried uninstalling java and then reinstalling the older version of Java, but PDR will still not work. Any clues or fixes?
Gary
Edit- I did a fresh install of windows 7 and java. that fixed whatever was wrong.
Quote from: a1ex on October 19, 2013, 04:59:56 PM
If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)
The problem is I directly extract frames with a chroma smoothing, so I can't really see the difference.
But if I see any issue, I will report them here, no problem !
Quote from: a1ex on October 19, 2013, 04:59:56 PM
If you have some shots with aliasing (caused by line skipping), can you check whether chroma smoothing has any effect on them? (be it good or bad)
I couldn't see any differences in my experiments, the lost of detail was too great. The reason chroma smoothing won't work, IMO, is the chroma distortions are line based, not pixel based; that is, you can't sample around a pixel (it will just pick up the distortion from the pixels to the left or right), you must sample above and below a line and probably greatly reduce the chroma of the offending pixels.
However, I do beleive chroma smoothing CAN work on aliasing. You would need an algo to figure out a line of pixels, say 8 pixels wide, and then an offset to the left or right (what creates a moire effect), and smooth using that information. You would smooth 2-3 lines.
I have a light table and moire charts and can create lots of good aliasing/ moire images for anyone to work with. It's on my list of things to do, but would do today if someone took an interest.
Another thing I'm curious about is if any algos exist out there. I would think the camera manufacturers do something similar in their in-camera software.
Each line distortion seems to me a predictable interference pattern along a sensor line, RGRGRGRGRG or BGBGBGBGBG.
I did a comparison between raw2dng_cs3x3_ahdlike vs raw2dng_cs2x2_ahdlike_noise to process dual iso dngs.
(http://farm8.staticflickr.com/7332/10540564223_6971a61f87_o.jpg)
I see big difference on highlights. Hope that helps.
Can you upload the original DNG? I think the proper way is to get rid of the dots somewhere after exposure matching, but I don't have any sample to try (none of my cameras have pink dots).
I was using my camera upside down. sorry. but here's the link:
raw2dng_cs3x3_ahdlike.DNG https://docs.google.com/file/d/0BwGjP3nTWukGRWxXeEhseTRUaDg/edit?usp=sharing (https://docs.google.com/file/d/0BwGjP3nTWukGRWxXeEhseTRUaDg/edit?usp=sharing)
raw2dng_cs2x2_ahdlike_noise.DNG https://docs.google.com/file/d/0BwGjP3nTWukGRHlCYTQzaVJ4X0E/edit?usp=sharing (https://docs.google.com/file/d/0BwGjP3nTWukGRHlCYTQzaVJ4X0E/edit?usp=sharing)
I mean the unprocessed one (with the plain raw2dng, without any modifications).
original raw2dng https://drive.google.com/file/d/0BwGjP3nTWukGc19uTy1tOTFfS1E/edit?usp=sharing (https://drive.google.com/file/d/0BwGjP3nTWukGc19uTy1tOTFfS1E/edit?usp=sharing)
The 2x2 raw2dng does a pretty good job (I have a hard time trying to beat it). Nice find.
But if you can run PinkDotRemover, I think that one will give noticeably better results. Chroma smoothing artifacts are still visible with dual ISO.
right PDR does better, I just found an artifact on a frame from raw2dng 2x2. but I just love the workflow of raw2dng 3x3/2x2. drag n drop and it extracts clear frames.
Quote from: roterrion on October 28, 2013, 07:12:46 PM
I did a comparison between raw2dng_cs3x3_ahdlike vs raw2dng_cs2x2_ahdlike_noise to process dual iso dngs.
(http://farm8.staticflickr.com/7332/10540564223_6971a61f87_o.jpg)
I see big difference on highlights. Hope that helps.
which one do you like better? there is also a 3x3 noise version you can add to compare.
Gary
"raw2dng_cs2x2_ahdlike_noise" looks better because it does not leave streaks on the dress.
But there are still artifacts available.
I just uploaded a new RAW Video test I made, where - unfortunately - we can see a lot of aliasing & artifacts.
I used the raw2dng_cs2x2_ahdlike_noise to extract the frames (and tried the original raw2dng to compare but the quality was quite the same - except there was the focus pink dot pattern of course).
I thought it looked nice. BTW, A1ex was the one who added chroma smoothing algos to the raw2dng. However, he doesn't have a camera with the focus pixel (pink dot) problem, so is handicapped by how much he can address that specific problem. There is the Java PDR software, but it doesn't always work (long story). I've been working on algos for the focus pixel problem and hope that if I'm onto something A1ex can make it sing.
My thinking is we
1. Detect general pattern in frame if we can
2. Run the following idea on all pixels most likely to be focus pixels based on above. The routine always works pretty well on all pixels. By targeting known pixels and lowering the threshold I think the problem will be nailed.
After the focus pixel problem is taken care of, film-makers can use A1ex's smoothing routines for low-light shots mostly, my guess.
The main point, dsagilles, is some people are putting a lot of effort into making your camera kick butt ;)
// Interpolate METHOD 2
if (cboFocusDotRemovalMethod.SelectedIndex == 1)
{
// ***
// REDS ONLY!
// ***
for (int x = 2; x < Footer.xRes-2; x += 2)
{
for (int y = 2; y < Footer.yRes-2; y +=2 )
{
// Let's nail our suspect onto a cross of red pixels
CP = (double)PixelData[y, x]; // Center pixel
LP = (double)PixelData[y, x-2]; // Left
RP = (double)PixelData[y, x+2]; // Right
TP = (double)PixelData[y+2, x]; // Top
BP = (double)PixelData[y-2, x]; // Bottom
AvgLR = (LP + RP) / 2; // average horizontal
AvgTB = (TP + BP) / 2; // average vertical
// If all surrounding pixels are lighter or darker, we could do simple interpolatation
// with any neighboring red pixel.
// BUT IF THERE IS an "edge/line" to left, right, top, or bottom, the pixel might end up
// incorrectly smoothed and show up as an artifact to that edge
// Therefore, we want to detect any edges, and if found
// try to smooth into pixel row ALONGSIDE edge
// For example, if a focus dot is next to a flag pole, on right, against the sky, nd we use the
// pole's red pixel to interpolate, we'd end up with a red pixel next to the pole
// that has the pole's chroma. It will look like a dot stuck next to pole
// We'd want to interpolate with the sky pixels, top, bottom.
CPInterpolated = 0;
// If differs by more then 10% left right
if (Math.Abs((CP / AvgLR) - 1) > 0.10)
{
// Remember, we interpolate using the perpindicular line (top/bottom), assuminge this might be a line
CPInterpolated = (ushort)AvgTB;
}
// If differs by more then 10% left right
if (Math.Abs((CP / AvgTB) - 1) > 0.10)
{
// Perpindicular cross line
CPInterpolated = (ushort)AvgLR;
}
//We found a distortion so blend in...
if (CPInterpolated > 0)
{ PixelData[y, x] = CPInterpolated; }
}
}
// ***
// Blue ONLY!
// ***
for (int x = 3; x < Footer.xRes - 3; x += 2)
{
for (int y = 3; y < Footer.yRes - 3; y += 2)
{
// Let's nail our suspect onto a cross of red pixels
CP = (double)PixelData[y, x]; // Center pixel
LP = (double)PixelData[y, x - 2]; // Left
RP = (double)PixelData[y, x + 2]; // Right
TP = (double)PixelData[y + 2, x]; // Top
BP = (double)PixelData[y - 2, x]; // Bottom
AvgLR = (LP + RP) / 2; // average horizontal
AvgTB = (TP + BP) / 2; // average vertical
CPInterpolated = 0;
// If differs by more then 10% left right
if (Math.Abs((CP / AvgLR) - 1) > 0.09)
{
CPInterpolated = (ushort)AvgTB;
}
// If differs by more then 10% left right
if (Math.Abs((CP / AvgTB) - 1) > 0.09)
{
// Perpindicular cross line
CPInterpolated = (ushort)AvgLR;
}
//We found a distortion so blend in...
if (CPInterpolated > 0)
{ PixelData[y, x] = CPInterpolated; }
}
}
} // false?
Can we get PDR to allow us to drag and drop all the new raw files from the card to PDR. Then PDR creates a folder with the current date as the name for all these new raw files and places the files there. Then it makes subfolders for all the new raw files with each named from the raw file name. Then it processes all the new copies of raw files with PDR and then it runs raw2dng (or whatever raw2dng file we choose to use inside PDR). All one drag and drop and then it's finished with every thing we have to now do manually. Make it so we have to setup the options first, like crop, non crop for EOS M and the location for the first folder (or use the location from the last time we used PDR) and which raw2dng we want to use. Then we do the drag n drop of all new raw files from the card.
Max, maybe you know this already and are working on it. I just wanted to make another request for an easier workflow. You also probably want one more step at the end for converting DNG to Cineform file. or DNxHD or ProRes. I would be happy to just get the DNG sequences for now. I am aware of Batchelor and Rawinizer, but they lack a few things like making copies of original raw files for bakup, allow new choice of raw2dng.exe , etc.
Gary
Hi Gary, I'm working on it. We can do all the things you mentioned, and more, once I get past the problem of saving the interpolated pixels back to the RAW file, like PDR does it. g3gg0's C# code doesn't have a write routine, PDR's is in Java and Alex's is in C and meant to write to dng. None of this code is what I would call trivial. Because these raw values are in 14 bits, you have to chop, slice and move around all the bits in 8/16 bit chunks. What a pain! But pressing on!
cool. thanx :)
i also had a new problem with some raw i shot the other day. I have a bunch of random very bright green pixels that I can't get rid of with pdr or neat video filter in PPro?? Never seen the green things before like this. I think i also shot it in non crop instead of crop like I usually use. Maybe the non crop had something to do with it??
why can't i pin the PinkDotRemover.jar to the taskbar or start menu for easier opening of the program? I always have to dig thru explorer folders to find and double click on PinkDotRemover.jar
Quote from: gary2013 on November 03, 2013, 07:50:44 PM
i also had a new problem with some raw i shot the other day. I have a bunch of random very bright green pixels that I can't get rid of with pdr
I get the green dots in similar situations with my 50d. Usually sunlight through trees. It is from poor debayering (like the one included in the RAW to Cineform converter).
What is your workflow? PDR will not deal with it, it isn't a focus pixel problem I believe.
Quote from: maxotics on November 03, 2013, 08:04:10 PM
I get the green dots in similar situations with my 50d. Usually sunlight through trees. It is from poor debayering (like the one included in the RAW to Cineform converter).
What is your workflow? PDR will not deal with it, it isn't a focus pixel problem I believe.
Sorry I answered. :)
Sorry I asked ;) Can you post an image with the green dot problem? Or email it to me at max at maxotics com and I'll post it for you.
Max,
i need to shoot a new one
Max,
here is ar DNG frame I just shot.
https://www.transferbigfiles.com/e39af3f6-40cc-4a63-ba21-6930ab9a308c/VQ29xRxyNXRuIWZW3GYbWg2
Max, here is a screen grab from PPro CC showing the green dots from the DNG I last sent here.
https://www.transferbigfiles.com/ecb1ee61-113d-40cb-aeae-55b6425c4a9a/otIaHrJgcit4Slrwc4qxQQ2
Gary
Any news on the Pink dots effort? Too much silence in here :)
Quote from: demetrisag on November 21, 2013, 12:54:09 AM
Any news on the Pink dots effort? Too much silence in here :)
Too few developers :-[
Shame, wish i knew a thing our two about these staff, all i know is how to be patient and wait.
Quote from: demetrisag on November 21, 2013, 08:16:25 AM
Shame, wish i knew a thing our two about these staff, all i know is how to be patient and wait.
But what are you waiting for exactly? PDR kinda works, right?
Quote from: Rewind on November 21, 2013, 10:01:47 AM
But what are you waiting for exactly? PDR kinda works, right?
Well, like you said it
Kinda, under perfectly controllable case scenarios and specific resolutions should be ok.
Real world scenarios not so much :)
Quote from: demetrisag on November 21, 2013, 12:33:53 PM
Well, like you said it
Kinda, under perfectly controllable case scenarios and specific resolutions should be ok.
Real world scenarios not so much :)
Which is why I built my own version in C#. It needs to be tweaked a little bit for sensitivity to exposure and I missed a couple of rows of dots. The Java version of PDR is 90% there. Rewind always shoots within that 90% :) Anyway, there is now a Java and C# open-source version.
I was looking at a lot of my old footage, that was properly exposed, and used PDR. Looks great to me.
Quote from: demetrisag on November 21, 2013, 12:33:53 PM
Well, like you said it
Kinda, under perfectly controllable case scenarios and specific resolutions should be ok.
Real world scenarios not so much :)
Then may be you should be more specific, because I use my 650D for commercial work now quite fine, PDR does it's dedicated job nicely, that's why I don't bother updating or fixing it.
If you tell me scenarios, when it doesn't work, then there will be a point to deal with it.
Quote from: Rewind on November 21, 2013, 02:47:55 PM
Then may be you should be more specific, because I use my 650D for commercial work now quite fine, PDR does it's dedicated job nicely, that's why I don't bother updating or fixing it.
If you tell me scenarios, when it doesn't work, then there will be a point to deal with it.
Rewind, I said it a gazillion times! If you shoot anything with clipped/overexposed parts of the frame the dots re-appear. You may be using a workflow that interpolates around them, so I'm not saying you're not wrong, for your use. (BTW Do you shoot outdoors?) It usually works perfectly for me too. And I know you're trying to help! And have! What would be really helpful is a new version with the latest good settings. Right now, I have to switch back and forth. I did look into it. Seems to me they punted a bit in the code, so not that easy. Anyway, once that's done, I can probably help out with fine-tuning any of the dot matrixes.
Also, now that I've worked with this stuff in serious depth there are some lines of dots that do not fit a formula.
Here's a video where you can see the problems. The dots appear in over-exposed sections of the frame.
PDR tool is for removing focusing pixels. These green dots in overexposed areas has different nature. They do not belong common pattern, so you should remove them with bad pixel removal functions of any raw processor in order to remove them only in overexposed areas, and not all over the image.
It is not so hard to drag a single slider in your raw app ;)
download this, it is a free (old) plugin, but it can inspire you...
http://www.apefos.com/dualiso/svpr.rar (http://www.apefos.com/dualiso/svpr.rar)
They aren't green pixels. However, there may be pixels that PDR would have removed. Rewind, I can't remember if that clip was processed before or after you fixed the patterns. But I did see them again in footage I shot a few days ago, but am not going to put up on Vimeo. In any case, I want a tool that takes care of all "fixed-pattern" problem pixels. It's why I wrote FocusPixelFixer. Like PDR, it can work 100%. Just too much work for me to do at moment.
Apefox, thanks for posting the plugin. However, without source code, doesn't help me.
I just bought a BMPCC. I still believe in the EOS-M, and other Canon cameras, but I haven't been able to get anyone involved in ML to get more professional. If you use ML professionally, Remind, I commend you! I don't need this stuff professionally. This is a hobby for me. Even as that, I find it too aggravating. It's hard enough trying to fix the focus pixel issues with these camera AND wrestle with bugs. This isn't anyone's fault. It's a collective failure.
Again, I've gotten pretty good at Photivo, a lot because of Rewinds help! So if someone gets the Java PDR into a single more usable code base going forward, i can handle the EOS-M camera settings, which I have.
But Rewind, I'm not asking you! God knows you've done enough. We need someone with dev skills beside you and I! Anyone?
Hey user of an 650D,
I just got a test crop video shot with 12.5 fps @ 1920x1074.
First I unpacked the RAW file with raw2dng_cs3x3_ahdlike_noise.
But the result was not satisfactory, because I got weird green dot's in the white lights.
(http://s7.directupload.net/images/131123/temp/25mrxteb.jpg) (http://s7.directupload.net/file/d/3450/25mrxteb_jpg.htm)
Then I tried it with PDR and crop mode, whereupon the green dot's disappeared, but the normal pink Focus pixel not disappeared.
(http://s7.directupload.net/images/131123/temp/498j4xfy.jpg) (http://s7.directupload.net/file/d/3450/498j4xfy_jpg.htm)
Now is not very important, because you can not turn really nice video with 12.5 fps, but it has me then puzzled.
Can you upload a DNG obtained with the vanilla (unmodified) raw2dng?
Here are the dng's started with the pure "raw2dng"
http://www.file-upload.net/download-8322907/raw2dng.dng.html
http://www.file-upload.net/download-8322914/raw2dng_cs2x2_ahdlike_noise.dng.html
http://www.file-upload.net/download-8322922/raw2dng_cs3x3_ahdlike.dng.html
http://www.file-upload.net/download-8322927/raw2dng_cs3x3_ahdlike_noise.dng.html
http://www.file-upload.net/download-8322930/raw2dng_cs3x3_v2.dng.html
Here's a quick fix:
raw2dng_cs2x2_whitefix.exe (http://acoutts.com/a1ex/raw2dng_cs2x2_whitefix.exe)
I've only tried it on the above DNG. It might have trouble with focus pixels in overexposed areas, because this is what I changed (try it and post some examples). If it looks alright, I'll commit this when I'll get back home (1-2 weeks). If not, it will require more investigation.
Thanks
Will test it later and reply the results :).
Nice done a1ex.
http://www.file-upload.net/download-8325224/raw2dng_cs2x2_whitefix.dng.html
It looks as if ISO 200 produces the most green pixel.
http://www.file-upload.net/download-8333915/raw2dng_cs2x2_ahdlike_noise-iso100.dng.html
http://www.file-upload.net/download-8333919/raw2dng_cs2x2_ahdlike_noise-iso200.dng.html
http://www.file-upload.net/download-8333918/raw2dng_cs2x2_ahdlike_noise-iso400.dng.html
http://www.file-upload.net/download-8333933/raw2dng_cs2x2_ahdlike_noise-iso800.dng.html
And if I use raw2dng_cs2x2_whitefix the green pixels around the flashlight change the color to white.
http://www.file-upload.net/download-8333931/raw2dng_cs2x2_whitefix-iso200.dng.html
So for me, working your whitefix great.
Thank you for that :)
Hi guys,
I´m realy happy to found that topic but first - I´m a german guy, sorry about my english.
I read from the beginning of topic and i never found my problem with dng files from a video-raw.
If I convert my raw (700D) video (720p) with raw2dng.exe, I get DNG´s with Pink and green dots! >:(
I have tested all version from that topic, but nothing removes the green dots. It may be true that I don´t understand all the discussion. I´m in the dark - what the hell I´m doing wrong??
here 2 DNG´s, first convert with raw2dng
Bild "orig8kset.jpg" anzeigen. (http://abload.de/image.php?img=orig8kset.jpg)
second converted by raw2dng_cs3x3
Bild "cs3x3qms1l.jpg" anzeigen. (http://abload.de/image.php?img=cs3x3qms1l.jpg)
Thanks to all guys, who work hard for those tools. ;D
I hope someone can give me tipps, to remove the green dots and all is perfect with my camera.
Hello, just out of curiosity, I was wanting to know if there is a present updated PDR Tool version or technique that will remove the red/green dots on the raw produced by the newer builds for the 650d? I think I read of an "earlier" build that the newest PDR (posted in this thread) worked on (including 1472x626) but it's older than what the nightly builds have for download. Could someone possibly be so kind as to post a link or email me a good build version that works well with the newer PDR tool listed in this thread? I shoot at 1280x720 and 1472x626. Currently, neither version 7 or 8 of PDR clean up my 1472x626 raw. I am currently using a build from 2014. By the way, thanks for all the hard work and commintment! Thanks, Anthony
Observation with various versions of raw2dng:
primary version: keeps pink dots: https://www.dropbox.com/s/ys03ilotm5kdfya/raw2dng.DNG
2X2 keeps white dots: https://www.dropbox.com/s/c59ha7lhne6row9/2X2.DNG
3X3 removes all dots: https://www.dropbox.com/s/s4487l5zbyx6e99/3X3.dng
5X5 removes all dots: https://www.dropbox.com/s/63paco5h0j4ol2v/5X5.dng
I can barely see a few dots in the first one and 5x5 one...
Hello guys,
I had not registered but I have been following the 650D threads for a while.
Yesterday, suddently the PinkDotRemover v7 stopped working. All that I get is this: https://www.dropbox.com/s/abxj1grbpbql583/Captura%20de%20pantalla%202014-02-10%2010.06.00.png (One or more files could not be converted).
I am on a Mac so I have to use this .jar, I cannot run .exe files. If there is another way to remove the pink dots, please tell me.
Edit: In the path there was an accent in one letter... Solved.
Thank you and keep on the good work!
Is there a build that I could use to fix the pink dots issue on my 700D ? I've tried the v7 but couldn't find a way to make it work with my 700D raw video (maybe the resolution which is not 720p or 1080p ?)
A1ex's programs work a lot better than PDR for most cases at the moment.
Do you have a link of it ? I'm pretty new around here :D
raw2dng_cs2x2_ahdlike.exe (http://acoutts.com/a1ex/raw2dng_cs2x2_ahdlike_noise.exe) is the latest version.
Page 13 has the other versions as well.
Quote from: papkee on February 13, 2014, 01:41:08 AM
raw2dng_cs2x2_ahdlike.exe (http://acoutts.com/a1ex/raw2dng_cs2x2_ahdlike_noise.exe) is the latest version.
Page 13 has the other versions as well.
I wish this worked for MLV files since we are moving in that direction instead of raw. I only see a way of either trying to still shoot just raw (not mlv) or a two step with converting mlv to raw and mores steps is not something anyone wants to do. Unless I am missing something here and someone can explain it to me.
MLV has --cs2x2 (didn't try it).
I understand mlv results in a drop in max resolution one can continuously record at on the 650d, or a reduced number of frames one can record for a given resolution. (I'm not sure why, my best guess is that more data - metadata - is being saved with EACH frame).
In order to maintain the max performance but allow an algorithm that targets bad pixels instead of blanket chroma smoothing which Rewind points out is non-ideal, could the raw recording module be modified to spit out a separate .fpd (frame position data) file which can be passed to the PDR tool as a reference point to properly determine bad pixel coordinates?
There's probably a good reason why this hasn't been proposed, but I'm just too new to know!
Quote from: a1ex on February 13, 2014, 08:30:08 AM
MLV has --cs2x2 (didn't try it).
For my planets and me --cs5x5 works best
Edgar
Can you update PinkDotRemover to remove dots from (2.20:1) 1344x610 .RAW file (recorded with mlv_rec)?
Or is there more programs made for remove pink dots?
DNG: http://www.upload.ee/image/3899487/000000.dng (http://www.upload.ee/image/3899487/000000.dng)
https://www.dropbox.com/s/4hc75c879w51j3i/000000.dng
Can someone tell me how to fix all the aliasing? I think all the pink dots have been removed but the picture looks really bad. Is there any way to fix that?
0) challenge the Nyquist theorem
1) try a VAF filter (not sure if available for your camera)
2) shoot very slightly defocused
3) try super-resolution
4) hide it by desaturating
5) ???
Quote from: HrRaiko on February 20, 2014, 03:18:41 PM
Can you update PinkDotRemover to remove dots from (2.20:1) 1344x610 .RAW file (recorded with mlv_rec)?
Or is there more programs made for remove pink dots?
DNG: http://www.upload.ee/image/3899487/000000.dng (http://www.upload.ee/image/3899487/000000.dng)
Have you tried raw2dng cs2x2?
Quote from: davegravy on February 21, 2014, 05:34:40 PM
Have you tried raw2dng cs2x2?
Jep it work but i liked that javascript :) But i got it work now.
Is there anywhere a updated download list of that 2x2, 3x3 etc raw2dng files?
Quote from: anakin316 on February 20, 2014, 10:44:12 PM
https://www.dropbox.com/s/4hc75c879w51j3i/000000.dng
Can someone tell me how to fix all the aliasing? I think all the pink dots have been removed but the picture looks really bad. Is there any way to fix that?
use resolve. if you dont know how to use ACR properly you will see lot of aliasing thing
Since MLV is the standard raw format, how do u remove the pink dots? What is new flow process? Is there a switch in mlv2dng? please help :D
or Please have builtin pink dot remove in the next ML build for 650? :D
if u want pink dot removal builtin MLRawViewer VOTE there. Need YOUR vote here!
https://bitbucket.org/baldand/mlrawviewer/issue/26/hot-dead-other-use-pixel-removal
So, I think this is a known issue with Pink dot remover but I figured I would re-iterate. I have successfully removed all the "dots" in all my video recorded in 1280x720. However, I am having trouble with 2.35:1 ratio or 1472x626. Pinkdotremover doesn't seem to be able to remove 1472x626. I am using v8. I've tried v7 and Rewind's crop version unsuccessfully.
Here are a couple of dng files. One is 1280x720 and the other is 1472x626.
https://www.dropbox.com/sh/nldkpyq8wdhhej8/Mv2ObP4SHg
Workflow was PDR v7 (converted the raw files directly) -> RAWMagic
I'm surprised you got PDR working. This works for me:
raw2dng_cs2x2.exe (http://acoutts.com/a1ex/raw2dng_cs2x2.exe)
It should remove the pink dots (actually focus pixels which can appear pink or green) on any image size or crop factor raw movie file.
I'm struggling to get it working on my Mac.
PDR works fine for resolutions multiples of 4.
For example, for cinemascope, i'm using 1536 x 640 on my 650D, which is exact (2.4:1)
That's good info. I didn't realize it would work with multiples of 4. I'll try that first, because my workflow is on a Mac. I'll give the exe file a shot too. If you've figured out a way to get the exe to work on a Mac, please let us know. Thanks for the replies!
Here's a list of max resolutions (anamorphic aspect ratios) given hardware/software constraints.
Assume: Max transfer speed 37mbps
width and length resolution need to be multiples of 4 for PDR to work.
2.39:1 1472x616 36.3 mbps
2.67:1 1536x576 35.4 mbps
2.50:1 1472x588 34.6 mbps
2.35:1 1344x572 30.7 mbps
2.35:1 1504x640 38.5 mbps seems to work continuously, but only tried it for about 1 min.
My nightly build of ML doesn't have a 2.4:1 aspect ratio. Closest is 2.39:1
i'm using modified raw_rec build. Resolution choices are in the begining of the file. Just edit them, add the resolutions whatever you want and compile.
Default res. choices are not particularly good for SD cameras.
Quote from: kaguilar on April 10, 2014, 11:11:25 AM
That's good info. I didn't realize it would work with multiples of 4. I'll try that first, because my workflow is on a Mac. I'll give the exe file a shot too. If you've figured out a way to get the exe to work on a Mac, please let us know. Thanks for the replies!
Hi kaguilar,
I just wrote up my experience on how I was able to get raw2dng compiled on my Mac.
Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng] (http://www.magiclantern.fm/forum/index.php?topic=7645.msg111168#msg111168)
Mine is a fairly new 64-bit PowerBook running 10.9.2. If your Mac is similar, PM me and I'll email you my compiled raw2dng with CHROMA_SMOOTH switched on.
why noone is developing pink dot remover for mlv like pink dot remover for raw. its a pain in the ass converting all mlv files to raw, removed pink dots and convert to cdng.
The --cs2x2 option from mlv_dump is not working?
I didn't try it (no mlv samples here).
Quote from: a1ex on April 14, 2014, 11:32:37 AM
The --cs2x2 option from mlv_dump is not working?
I didn't try it (no mlv samples here).
i dont want to convert to dng.
A small request to people using camera with Pink Dots...
MlRawViewer 1.1.6 introduced a new GPU-based bayer pre-processing option that tries to identify any "bad pixels". This is mainly for dead or hot pixels, but I have also been testing with a little material I have from PinkDot cameras, and I believe it can also detect and suppress many of those with a minimum of impact to the rest of the scene.
The feature is still in development, and would need some more tuning to be totally effective in the pink dot case. In particular I probably need to change the work flow so you can "train it" on a certain resolution, and then re-use that training data in other shots.
If anyone is working with raw on those cameras, and would be interested to help me develop this further, please try out the current code and let me know how it goes. I would also appreciate being sent more such MLVs to test it against.
Thanks!
Quote from: baldand on May 29, 2014, 05:49:16 PM
A small request to people using camera with Pink Dots...
MlRawViewer 1.1.6 introduced a new GPU-based bayer pre-processing option that tries to identify any "bad pixels". This is mainly for dead or hot pixels, but I have also been testing with a little material I have from PinkDot cameras, and I believe it can also detect and suppress many of those with a minimum of impact to the rest of the scene.
The feature is still in development, and would need some more tuning to be totally effective in the pink dot case. In particular I probably need to change the work flow so you can "train it" on a certain resolution, and then re-use that training data in other shots.
If anyone is working with raw on those cameras, and would be interested to help me develop this further, please try out the current code and let me know how it goes. I would also appreciate being sent more such MLVs to test it against.
Thanks!
If your code works on RAW, not just MLV, then I'd be happy to help.
I haven't moved on from RAW to MLV yet since, last I checked, you can't achieve the same resolution (continuous record) with MLV.
In any case, I can provide you some .RAW files if they're of any use in your testing. Let me know.
Quote from: davegravy on May 30, 2014, 05:26:03 AM
If your code works on RAW, not just MLV, then I'd be happy to help.
I haven't moved on from RAW to MLV yet since, last I checked, you can't achieve the same resolution (continuous record) with MLV.
In any case, I can provide you some .RAW files if they're of any use in your testing. Let me know.
Yes, it supports .RAW files also. They would also be useful for me.
Now, if you could find a way to remove the pink dots without removing dual iso data that would be amazing.
Quote from: baldand on May 30, 2014, 05:35:43 AM
Yes, it supports .RAW files also. They would also be useful for me.
I've PMed you some RAW files
I nearly trade my 650D for 50D, until a1ex introduce raw2dng with chroma smoothing. Its a great program. it makes the workflow more simple. :)
but there's so many version, so which one is the best for raw rec v1 with resolution 1280x720 ? :o
raw2dng cs2x2
raw2dng cs2x2 ahdlike noise
raw2dng cs2x2 whitefix
I use "raw2dng cs2x2 whitefix" without PDR and you can see the result (had to stabilize the video, because of this distortion). Without pink and green noise.
Sorry for my English, I use Google Translate.
Hi.. I can see pink lines flashing then and there in .mlv footage shot with 5d mark III, July build of magic lantern. Please help me fix this problem for the already shot videos. The lines show up in many footage..
http://s1163.photobucket.com/user/MrCunctator/media/linesmlvprob.jpg.html (http://s1163.photobucket.com/user/MrCunctator/media/linesmlvprob.jpg.html)
Can I use the raw2dng.exe for mac? Please help! I cannot remove the Pink dots on my RAW. :(
Quote from: shin~ on October 11, 2014, 07:35:28 PM
Can I use the raw2dng.exe for mac? Please help! I cannot remove the Pink dots on my RAW. :(
There isn't a version for Mac, however if you want to try and use @Baldand's remarkable app which includes similar PDR effects.
Here's the link: https://bitbucket.org/baldand/mlrawviewer/downloads/MlRawViewer_1_3_2_mac.dmg
Enjoy...
Or you can use MVLFS, it's wonderfull
@a1ex @escho
short question: i found in the ml-source two variants of chroma smoothing (2x2). a short one (in dual_iso and mlvfs (https://bitbucket.org/dmilligan/mlvfs/src/74a998f9c63820a7919377f4fdf98b7a3cb229ab/mlvfs/chroma_smooth.c?at=master)) and a long one in lv_rec/raw2dng (https://bitbucket.org/dmilligan/mlvfs/src/74a998f9c63820a7919377f4fdf98b7a3cb229ab/mlvfs/chroma_smooth.c?at=master). are both equal in result? i just coded the longer one into raw2cdng (c#) but stucked/hot pixels (red green and blue ones) just change into white, dont disappear. my fault while recoding?
regards chmee
The one from raw2dng is fine-tuned for clearing the focus pixels from recent small cameras (like EOS-M).
The one from dual iso is an older version (I should update it).
They will not filter the hot pixels on the green channel (only those on red and blue channels).
ok, now on a a/b-comparison i see, its working, but other turn into "more visible bugs". dont know if i do a second time cs2x2, maybe a 3x3 is better or another algorithm-idea will do that much faster.. kind of hotpixel-finder.. used example-mlv (http://www.magiclantern.fm/forum/index.php?topic=11899.msg130484#msg130484)
(http://dl.phreekz.de/raw2cdng/raw2cdng1.6.2_chromasmooth.jpg)
Where can I find "raw2dng cs2x2 ahdlike noise" or "raw2dng cs2x2 whitefix"?
I downloaded raw2dng cs2x2 and raw2dng and neither seem to open anything.
Using Windows 7. I tried PDR but to no avail. Thanks.
raw2dng is a command line tool.
Here's information on how to work with it:
http://www.magiclantern.fm/forum/?topic=5404.0
And here are the versions you're looking for:
http://acoutts.com/a1ex/raw2dng_cs2x2_ahdlike_noise.exe
http://acoutts.com/a1ex/raw2dng_cs2x2_whitefix.exe
These are Windows versions. I'm on a Mac and can't verify that they are working. If you really want to experiment download the source code and try compiling it yourself. I did and was able to get something that works for my camera (EOS-M) and operating system.
Sorry for the delay in responding. It has been a while since I checked the forum and even longer since playing around with RAW.
Some test with RAW 1.0 from Canon 700d
WITHOUT using PDR
1. Raw2CDNG 1.7.4
2. Raw2DNG cs2x2_whitefix
3. Raw2DNG cs5x5
But better watch in full prores quality:
https://mega.co.nz/#!I9FgFYAb!Omq1JVYCowyPZXiWWq8UuxCyMaE5lWFAY3Bxks53dAo
WITH using PDR
1. Raw2CDNG 1.7.4
2. Raw2DNG cs2x2_whitefix
3. Raw2DNG cs5x5
But better watch in full prores quality:
https://mega.co.nz/#!08sRnSQQ!LcqtgwmlD89h8z6tIl_EuEl0GnyJOg--Bxe4RYgjdHk
As you can see:
1. Сolor artifacts and pixels with broken colors
2. Very good result, but some color artifacts
3. Less color artifacts, but in some details Raw2DNG cs2x2_whitefix better.
There is no need o use PDR!
I think the best solution will be
Raw2DNG cs5x5_whitefix, if A1ex will make it :)
Everything is green basically in your test :)
Try to shoot something colorful. PDR is way better solution if you interested in color fidelity. Because, as it says, cs algorithm, well, smoothes (just blurs) all the color information. But PDR is adaptively interpolates just the "pink dots" themselves.
http://www.magiclantern.fm/forum/index.php?topic=5780.msg80822#msg80822
Hi
I am shooting 768 x 462 on my 650D and need to use the PINK DOT REMOVAL tool.
I have run it straight out the box and the RAW file when I look at it in MLRawViewer is still showing the dots.
Does anyone know what I have to do to get it to work ok?
Cheers
Mat
Hello everyone,
I buy a 700D and try to remove the pink/blue dots from raw video using PDR 0.8 but without any result.
At moment the best is CS2x2 that remove all or near all dots.
I mainly use ML for astrophotography and video on planets, moon and sun (a-alpha).
This last application could be very interesting using ML raw 14 bit video (on the planet not more than 640x640 - the planet are very small also with 5 meters of focal length) but this damned pink/blue dots are very annoying.
I try also to know better the ML option on 700D.
I experiment with "Movie crop mode" option on MOVIE tab.
I found (probably you know before that) that it took movie exactly 1720x979 expanded digitally to 1980x1088.
This is just a mere 10% and if you increase the CBR factor to reduce the compression you get a good 8 bit MOV video ready to use and without pink/blue dots.
Why in this 1.1:1 pixels crop mode we don't get pink/blue dots? The little (10%) interpolation cancel them?
The 70D has the same Pink Dot problem?
Hey
I have an Canon 650D (T4i) with a sigma 17-50 2.8. When i use pink dot remover the dots are still there after the programme have finnished. I run version 8. Any tips for help?
Thanks
Im also have dots on my dngs (canon 700d)
But the question is other I see videos from 700d and other canons And there is no pink dots on it Then problem in my camera ??? ot what ? why this happens ? maybe reinstall ML ?
@dirty_dollar
Tip - always read the first post of a topic.
There is nothing wrong with your camera or ML. PinkDotRemover tool was made for the 650D and while it also works with other camera models the 700D has a different focus pixel pattern. There are several ways to deal with these dots. Search the forum for dots or focus pixels.
Hi all - I'm on a Mac OS 10.11.2 and tried PDR_007 without luck to rid 650D footage of the pink focus dots.
Right now, I'm using MLV Mystic (Mac) to rid the focus dots by setting Chroma Smooth (2x2). Then taking the resulting DNG folder into Resolve 12 (Free) and exporting ProRes. A little cumbersome but it works for me.
I tried MlRawViewer and found that if I set Stripes off and hit 0 (zero), that it would cause the dots to disappear on playback. Unfortunately, upon export, the dots somehow persist. So close.
Hope this helps someone.
@ikoniq
I can't get PDR working for me either.
I started a topic to better understand how to deal with these focus pixels:
http://magiclantern.fm/forum/index.php?topic=16054.0
What works best for me is using MLV and put in the numbers I mapped off of my EOSM--it should also work for 650D. I only have 1280x720 cropped video mapped but plan to eventually map all resolutions.
MLP is here:
http://magiclantern.fm/forum/index.php?topic=13512.0
In the dcraw_FFmpeg_settings folder you'll find a dcraw_fix_pixels folder with a dead_pixel_list.txt file in there. Read the file, it will instruct you to add your own map of pixels. I've got my map saved here: https://bitbucket.org/daniel_fort/magic-lantern/downloads/focus_pixels_EOSM_1280X720_crop_mode.txt
Simply copy and paste my numbers after these lines:
-P ../../PRORES/A_lut_hold/dead_pixel_list.txt
-P ../A_lut_hold/dead_pixel_list.txt
-P A_lut_hold/dead_pixel_list.txt
Then add your edited dead_pixel_list.txt into the A_lut_hold--that's it. MLP also works with MLVFS so you don't even need to make intermediate DNG's.
Another way is to use MLVProducer but it is a Windows program so you'll have to run it under Wine.
The problem with these processes is that they won't remove the dots from the DNG files and that's what is needed if you want to go back to the raw files for the final color grading. I'd like mlp_dump to clean up the dots and create clean DNG's or in an ideal world, do it inside the camera so we never have to deal with them in post.
Updated MLP with your pixel coordinates as a starting template dfort. Hopefully more of these lists can be created and added.
@dfort and Danne - thank you for the tip. Will give MLP a shot, much appreciated.
Hi all, i´m trying to test MLP and dead pixel list, Danne has helped me alot :D Tell me if a can help someone by testing!
Im on the latest MLP and also ML. 700d.
Cool, here are the more relevant topics for this maybe. Try asking there.
http://www.magiclantern.fm/forum/index.php?topic=16054.msg162539#msg162539
MLP
http://www.magiclantern.fm/forum/index.php?topic=13512.625
I use a mackintosh (Mac OS X on ordinary computer hardware) and i have downloaded pink dot remover and MLrawViewer. I get MLrawViewer to work but can't get pink dot remover to run and i don't understand how to get it run. When i double click on pinkdotremover.jar it just makes a new folder with the same files. I have read somewhere that you should do something in terminal. Can someone please explain for me what to write in terminal? I have downloaded and installed java. My footage are on my third hard drive called WD and pink dot remover are on the main SSD where the OS is.
Thanks
SOLVED
Installed a development kit for Java and it worked when i use Terminal to start it.
I have recently bought a mac. When i used windows i just used raw to DNG file drag and drop and it removed the pink dots and converted the pictures to DNG. No when I'm on mac i would like to fins a workflow that works. I haven't goten anything to work. I can convert files from raw to DNG thats not a problem. The problem i have is getting rid of the pink dots. I have tried pink dot remover but can't get it to start working. When i start the program it just makes a new mac with the same program files and not a single picture. Can someone help me please!
@filip1727 use mlvfs or mlp
@andy kh but i can't use those? or can i? I just record raw into the raw format i, don't record MLVs
Raw format works well with MLP.
If you have a focus pixel camera, you really need to use MLV format. The raw format lacks the necessary metadata to properly, easily, and automatically remove focus pixels.
There are many other reasons you should use the MLV format as well.
Guys I'm lost. :(
I'm a Mac user with very low confidence using Terminal or anything too code-related. I'm really enjoying shooting RAW on my new EOS M and processing it with MLVRawViewer, both of which are very smooth and I'm liking the results I'm getting. Hopwever I can't for the life of me figure out how to get rid of the pink dots. I have to be able to do it on Mac as I don't own or really know how to drive a PC. Ideally I'd like to continue my MLV to MLVRawViewer workflow but if that aint possible I'm prepared to take another route.
In case it has any bearing, I'm shooting RAW in crop mode with a 4:3 ratio (to mimic Super-8) and would like to continue this.
Many thanks if anyone can help me get rid of the pink dots on the Mac.
Thanks as always to Alex etc for so much work over the years. I don't post here but have followed with delight. :)