Messages

I played around with dual ISO today and I also noticed the "ISOless...." error message. Whenever this message appeared, dual ISO didn't work, even if it was activated. Read: the cam took a normal picture.

More or less by coincidence I found out that the message did NOT appear and dual ISO worked like a charm if I started the cam in "P" mode. I tried it like 10 times or so and it was reproducible. On the other hand, if I booted the cam in A, T or M, the error kept popping up.

I used yesterday's build (Sep. 6).

Maybe booting in "P" works for you as well... Good Luck!

I'm not a programmer, so I don't know your rules here guys. May I or may not share my build? Can we share the knowledge with the developers of PDR or am I just sticking my nose where I don't belong?

I can only second Mixer2's statement: please share your code so that we can merge it with the official releases. If you're familiar with GitHub, issue a pull request; that would be the easiest and preferred way of sharing your code. As an alternative, you can send me your sources and I'll merge them manually.

@Mixer2:
What do you think? Replace my old and busted removal algorithm with yours AND Rewind's and introduce a drop-down box (or radio buttons) to select the preferred method?

Wheeew... a little more than two weeks of vacation and business trips and the good, old PDR-world is not what it used to be anymore :-)

I just flipped through the last pages of the thread and understood that the PDR is not (yet) obsolete. Right? So I keep my promise to release a new version 008 which brings you, compared to the previous official version 007:

  * Better support for "ugly" image resolutions which are not divisible by 4

  * Improved pink dot locations for crop mode and EOSM

  * In-RAW dot removal for videos files (no need to process hundreds of DNGs anymore)

  * Better file list in the GUI, including support for Drag&Drop and deletion of selected files with the Del-key

  * Persistent storage of the selected camera type for the GUI

  * Possibility to provide the cam type as first command line parameter for batch operations


Some of the features were already available in the last debug release. And credit were credit is due: all the GUI and dot location data improvements were contributed my Mixer2. I concentrated only on the in-RAW-removal and on merging Mixer2's pull requests...

Download the binaries here: https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__008.zip


I'm looking forward to include an improved removal algorithm in the next release. Again, Mixer2 already contributed a very promising implementation and obviously Rewind is also working on some neat stuff. Perhaps we'll just add a drop-down menu to the GUI so that you can choose the algorithm. That would be the easiest (intermediate?) solution.


Happy dot removing and thanks again to all testers and contributors,
foorgol

Little update:

Exchanged a few messages about the BATCHelor integration with fatpig the last days. Looks as if he could just call the PinkDotRemover from the command line. For this purpose I added a few lines of code to enable camera type selection for the command line mode of the PDR. Up to now, the cam type was immutably set to 650D when using the command line.

In the meantime, mixer2 provided a few goodies for the GUI: he improved the file list, added the ability to delete individual files from the list and introduced support for Drag'n'Drop!

So be prepared for a new release coming out soon!

May you have a look at this thread http://www.magiclantern.fm/forum/index.php?topic=5645.new#new

I think you can give fatpig better information than me. Because I asked if it is possible to integrate the pinkdotremover into BATCHelor.

Done:

http://www.magiclantern.fm/forum/index.php?topic=5645.msg63402#msg63402

As to the PinkDotRemover:

@spider:
I can maybe include something like it,
but I am not aware of the functionality of the tool.

I read something about it being different for each camera..
which cameras need this tool?
How do you normally work with it?

As spider explained, the tool is needed by the 650D, EOSM and (perhaps) the 700D to remove annoying artifacts (AF dots) from raw video recordings and silent pictures.

You can use the tool either with its GUI or from the command line. Only drawback when using the command line: you can't select the camera type; it's fixed to the 650D. But this restriction could be improved with reasonable effort.

The program itself is written in Java. If the BATCHelor is also written in Java, I could easily create a lib / JAR for inclusion in the BATCHelor. Otherwise a command line call from within the BATCHelor is most likely the easiest approach...

@fatpig:
PM me if you need more info!

Hope that helps....

it is so awesome fast. 

Yes, I had the same impression, although I didn't make any specific performance tests. Since the code for the dot removal itself is identical for both DNGs and RAWs, my assumption is that the opening/closing of many DNGs files consumes time. Even the overall amount of data read and written from/to disk is identical between processing a RAW and its DNGs. So it must be the handling of maaany files....

Aaargh... my fault... I packed the wrong JAR file. Actually the bug you mentioned was the last one I fixed. But then I forgot to built a new "distribution JAR"... I only tested it with the "development environment JAR".... well, whatever...

Now I've uploaded the correct version. The link posted above is still valid.

Sorry for the confusion!

[directly in the RAW files]

So, let's break the silence here! Time for a new debug release!

The last days I gave the code a major re-write in order to support PinkDotRemoval directly in the RAW files. Nevertheless, removal in DNG-files is still possible, of course.

I did some testing, but it's not really what I would call thorough testing. Therefore this is a debug release only. Please give it a shot and tell me if it works for you. Download the binaries here:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__debug.zip

BEWARE:
Other than DNG files, RAW files will be OVERWRITTEN IN PLACE. Make sure to create a backup of your RAW file before you start the conversion.

Implementing it this way was easier for me. So much to my paradigm of not overwriting user data...

Enjoy!

P.S.: The in-RAW-removal together with your feedback and a few improvements that Mixer2 has in the pipe will constitute the next "official" release 008, I think...

@foorgol thx for all the good development. I think now the EOS 650D needs an option for Crop Mode too.

I'm not sure that I understand what you mean... are you referring to additional resolutions that need to be supported by the PinkDotRemover? Or do you mean ML-support for additional shooting modes/resolutions?

Turns out its a module- i downloaded a build for the 60D, then copied the ettr.mo file to the modules folder on the sd card. It now comes up in the expo menu, and seems to be auto exposing, and creating the .uwr files.

Wait, let me get that straight:
You're using a module built for the 60D and it works on the 650D? Without doing anything harmful or at least make ML on the 650D crash or hang?

I will soon make low light timelapse video just to see how the dot removal tool handles lower light conditions. Shouldn't be much difference?

Right. The tool now removes the dots all over the image, no matter what the lighting conditions are.

Let me give you a more detailed explanation:

In previous builds, I removed only the "light dots" in the center based on a fixed "grid definition" while the "border dots" were removed based on empirical/statistical data read from sample DNGs. Which was wrong.

Reason: Both "center dots" and "border dots" are expressions of the same regular grid of dots, just with different visibility.

Since the debug version (and officially since the 007 version) "center dots" and "border dots" are now handled alike: based on a fixed grid-like definition of dot locations. All over the image, from left to right. Always.

Theoretically there is one disadvantage: I also apply interpolation to border locations where you wouldn't see the dots in your particular image, because e. g. it's a very bright image. So I'm applying interpolation to locations although there's nothing wrong them. This could lead to a small loss of detail.

But in practice I couldn't notice this disadvantage. And even if you could notice it: it would be surely outweighed by the disadvantage of having pink dots accidentally popping up because the removal wasn't thorough enough.

Okay, just got the GO! from Mixer2 not to wait for him and release a new version
(He's working hard to setup and test a lot of resolutions for the EOSM, so no blaming him here)

Well then, here it is:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover__007.zip

What's new since version 006 (partially already available in the previously posted debug version):

  * Fix for the wrong path handling that lead to a crash for some of you

  * Initial support for EOSM

  * Support for all 650D resolutions (see comment below)

  * Faster image conversion due to optimized algorithms

  * Faster start-up

  * Initial support for the 16-bit DNGs generated by RawMagic

  * When selecting a bunch of files or a directory, only DNG files will be added to the conversion list


Comment on the "all resolutions for 650D" feature:
We've found a way how to extrapolate a single set of dot locations to arbitrary image resolutions. And it works. Mostly.
During testing, we found one or two resolutions which do not fit into our extrapolation scheme.

What this means for you:
The tool will convert the files without complaining. You'll see no error message. But the resulting file might still contain all or at least some dots. In this case, please send me a message with the image resolution and I'll set it up as one the few "special cases" that need individual dot location definitions.

Tested and verified resolutions for the 650D are:
            1280x720
            1344x572
            1344x756
            1472x626
            1472x828
            1600x680
            1600x900
            1728x736
            1728x972
            1808x1190 (Silent DNGs Movie Mode)
            1808x727 (Silent DNGs Picture Mode)

Source Code:
https://github.com/Foorgol/PinkDotRemover

Enjoy!

When using pinkdotremover the DNG's come back as single files (if you have 5 files it will look like 5 files) but when you select the 5 files it will show that there are 10 files selected.

The name of the image files created by the dot remover that with an underscore "_". Is it possible that an underscore indicates a hidden file on your system or in your file manager? Or does you file manager show "aaa.dng" and "_aaa.dng" only as one line, because it assumes that the underscore file is something like a backup file?

Yes, just be patient. RawMagic is producing CinemaDNG. Mixer2 and foorgol are working to adapt the tool for it.

... and basically the current status is unchanged. THEORETICALLY the dots should be removed, but PRACTICALLY a few of them survive. I guess I'll release the new version anyway, because I need your feedback and a few more sample pictures to hunt down the bug...

Sorry for being slightly OT, but can anyone point me to a spec or description of the RAW files created by ML? Or do I have to reverse-engineer it from the raw2dng source code and its headers?

Thanks!

Im thinking of shooting video in a Club, we are holding Go Go Girl Auditions, should I film in raw or should I just film with the cannons native software?

Besides all photographic considerations, I would start with a pragmatic question: are you willing / able to shovel really huge amounts of data? One RAW frame 1472x626 is about 1.7 MB. If you shoot 24 or 25 fps, you end up with 2.5 GIGABYTE PER MINUTE. If you shoot e. g. 30 minutes footage (what I would consider not much given your setting), you have 75 GB sitting on your CF cards and (later) on your disk.

And at least during pink-dot-removal, this doubles for a short while: 150 GB.

And after you develop the RAWs into MJPG, you might have approx. 1/5 of the size, so about 15 GB...

And other figure: for me, rawtherapee and all four cores at 2.9 GHz ("Turbo Mode" ) need approx. 2 seconds / frame for applying some color correction, tone mapping and noise removal on a 1280x720 image. So my machine would need 50 minutes to convert one minute of 25 fps RAW material to JPG. Depending on your computer, your pace my vary, of course.

All I want to say is: RAW is nice but not really handy...

Short update on the dot remover:

  * Changed algorithms to support EOS M better

  * Fine-tuning EOS-M dot location database

  * First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed


Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!

And also indirectly helping the developers by taking our minds of the pink dots so leaving them developers do what they do best!

Hehe... you're right... I remember the discussions in the 650D-thread being quite emotional... about whether the developers should spend time on improving RAW or not... I guess there's some relief now on that front

I tried compare interpolation, without PDR and dead pixel option.

From the pattern of the removed pixels I would guess that you used an old version of the tool. A new release is on its way and for the time being just use the debug version mentioned above:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

This should fix our issues.

Oh, and make sure you're using raw2dng to get the DNGs.

Extremely well mate! you are a legend!

Thanks a lot, but keep it low, my friend

This just an interesting image processing job, done in a well defined Java environment with virtually unlimited resources like memory, time and processing power. This is nothing compared to the embedded, reverse-engineering development that the real ML developers do...

Anyway, I'm glad that some of guys find that little tool useful... makes it even more fun to code

What am I doing wrong?

This looks very, very strange. And honestly I have no idea right now how this happened. But I'm also not sober enough right now to look deeper into it...

The only thing I noticed at a first glance is that your original image contains weird meta data. Looking at the tags model, manufacturer, software, bits per pixel and DNG version, I would expect "Canikon", "Canon", "Magic Lantern", 14 and "1.3.0.0", whereas your file reports nothing, nothing, "Rarevision RAWMagic 1.0", 16 and "1.1.0.0".

What's most striking is the difference between the "normal" 14 bit color depth and the 16 bit in your file.

So, frankly speaking, your source file doesn't seem to come straight from a 650D. Is that true?

My program is somehow tailored to the 650D-files. Maybe this is part of the problem.

Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

@papkee:
Does this version solve your issue with pixels not being properly removed @ 1472x626?

@donjames150:
Does this version work with spaces in the directory name for you?

@demetrisag:
If you're lucky this version could solve your database-not-found issue as well... give it a shot, please...

looks like this

Okay, my assumption was that the error has to do with the determination of the JAR's directory. I thought it would fail completely. But instead, it's just a back-and-forth conversion issue, where a space " " finally ends up as %20 like in an URL. So my guess was close

Should be easy to fix. For the time being, just avoid spaces in the directory name.

You always catch my sloppy ways to handle paths....

Okay, one by one....

--------------------------------------------------
@donjames150 and @demetrisag:

Could you please run the program in a terminal window (instead of just double-clicking the jar) and send me the output? I have a faint idea what the reason for error could be. This idea would be confirmed if the program would run normal if you start it from the command line...
--------------------------------------------------

--------------------------------------------------
@spider:
There are two kinds of dots. The "block" in the center and the "strip" all across the image. It's characteristic for the strip-dots to only show up in dark areas or when you lighten the image.

Your issue has to with the way the position of the strip-dots are located. Currently it's a trade-off between removing fewer of these pixels and losing image details by accidentally removing too many non-pink dots . Mixer2 found an interesting approach to solve this which he describes here:
http://www.magiclantern.fm/forum/index.php?topic=3648.msg54162#msg54162

I'm working with Mixer2 on an improved "dot location determination" for these dots which will come with the next release.

Until then, you can try to edit the file 650D_Video_1280x720.txt which you find in the dotData directory. The 4th line ends with "2068". You can try to gently reduce this value, e. g. to 2050 or 2040. This should then remove more of these dots.
--------------------------------------------------

--------------------------------------------------
@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.
--------------------------------------------------