PinkDotRemover tool 650D

[papkee]

framing is in the submenu under raw video in ML

It's not there on mine. RAW Video only displays Resolution, Aspect Ratio, Preview, Digital Dolly, Frame Skipping and Playback

[spider]

Framing was removed because you dont got any speed improvement from it.

[papkee]

Framing was removed because you dont got any speed improvement from it.

@papkee:
If "almost all" of the points have not been removed... hmmm... what "Framing" setting did you use when shooting the video? The framing must set to "Centered", otherwise the assumed dot locations are wrong.

There seems to be some conflicting messages here.

[spider]

No conflict, in the past there was a setting called framing.
Now it is gone https://bitbucket.org/hudson/magic-lantern/commits/145e69e340eb9a868f29eda8f0acae0a4fb6cbc3

Only the dolly mode which was a part of framing is still there.

[foorgol]

Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

@papkee:
Does this version solve your issue with pixels not being properly removed @ 1472x626?

@donjames150:
Does this version work with spaces in the directory name for you?

@demetrisag:
If you're lucky this version could solve your database-not-found issue as well... give it a shot, please...

[donjames150]

Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

@donjames150:
Does this version work with spaces in the directory name for you?

So far it looks like it's working fine, I'll be testing it today at different crop sizes.  Again, thanks
for all your hard work on this.

[donjames150]

This is completely raw after pink dot removal, no other corrections.
1728 x 972 at 12 FPS
ISO 200 F2.8 1/30

[talosectos]

Hi there,

Thank you for the great job 

I've got a problem with the tool (I'm certainly doing something wrong  ).

Here's the link for an original frame:
https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng
The resolution of the raw video was set at 1472x626 (2.35:1), shot with the ML build from 2013-06-26.

When I use the tool (on W7), I get this: https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng

What am I doing wrong?

[demetrisag]

Debug version:
https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

@papkee:
Does this version solve your issue with pixels not being properly removed @ 1472x626?

@donjames150:
Does this version work with spaces in the directory name for you?

@demetrisag:
If you're lucky this version could solve your database-not-found issue as well... give it a shot, please...

Extremely well mate! you are a legend!

[donjames150]

Hi there,

Thank you for the great job 

I've got a problem with the tool (I'm certainly doing something wrong  ).

Here's the link for an original frame:
https://www.dropbox.com/s/y3jb96kyp6sc2zs/M26-1615_00000.dng
The resolution of the raw video was set at 1472x626 (2.35:1), shot with the ML build from 2013-06-26.

When I use the tool (on W7), I get this: https://www.dropbox.com/s/nna78o9qw1r4xry/_M26-1615_00000.dng

What am I doing wrong?

I just took a short clip at that resolution and it removed the dots for me. Did you maybe select EOSM by mistake?

Edit: But then when I tried it on YOUR file, it did what it did for you, made it 10 times worse.

but if I open your original in Rawtherapee and apply the dead pixels profile, it acts normally (for that program)



Then I installed the same build as yours (which is wow, btw) and again took a short clip and the tool removed the dots.
This is my setup in RAW: Digital dolly OFF, Frame skipping OFF. That's all I got for ideas for now

[foorgol]

What am I doing wrong?

This looks very, very strange. And honestly I have no idea right now how this happened. But I'm also not sober enough right now to look deeper into it...

The only thing I noticed at a first glance is that your original image contains weird meta data. Looking at the tags model, manufacturer, software, bits per pixel and DNG version, I would expect "Canikon", "Canon", "Magic Lantern", 14 and "1.3.0.0", whereas your file reports nothing, nothing, "Rarevision RAWMagic 1.0", 16 and "1.1.0.0".

What's most striking is the difference between the "normal" 14 bit color depth and the 16 bit in your file.

So, frankly speaking, your source file doesn't seem to come straight from a 650D. Is that true?

My program is somehow tailored to the 650D-files. Maybe this is part of the problem.

[foorgol]

Extremely well mate! you are a legend!

Thanks a lot, but keep it low, my friend

This just an interesting image processing job, done in a well defined Java environment with virtually unlimited resources like memory, time and processing power. This is nothing compared to the embedded, reverse-engineering development that the real ML developers do...

Anyway, I'm glad that some of guys find that little tool useful... makes it even more fun to code

[talosectos]

I just took a short clip at that resolution and it removed the dots for me. Did you maybe select EOSM by mistake?

Edit: But then when I tried it on YOUR file, it did what it did for you, made it 10 times worse.

but if I open your original in Rawtherapee and apply the dead pixels profile, it acts normally (for that program)

Ok thanks for your answer

The only thing I noticed at a first glance is that your original image contains weird meta data. Looking at the tags model, manufacturer, software, bits per pixel and DNG version, I would expect "Canikon", "Canon", "Magic Lantern", 14 and "1.3.0.0", whereas your file reports nothing, nothing, "Rarevision RAWMagic 1.0", 16 and "1.1.0.0".

What's most striking is the difference between the "normal" 14 bit color depth and the 16 bit in your file.

So, frankly speaking, your source file doesn't seem to come straight from a 650D. Is that true?

My program is somehow tailored to the 650D-files. Maybe this is part of the problem.

Ok, I think that the problem comes from the tool I used to convert the raw (RawMagic), this part was done on my Mac.

I'm going to try with another tool and let you know the result.


Edit:
I've tried with raw2dng, and this time it works perfect.
I should have check that before, sorry 

Thanks foorgol!

Second edit
The matter is that RAWMagic convert into CinemaDNG files (that can be used directly into Resolve) whereas raw2dng convert into photo-dng.
Some differences between the two formats that cause problems with the PinkDotRemove tool and CinemaDNG files.

[demetrisag]

Thanks a lot, but keep it low, my friend

This just an interesting image processing job, done in a well defined Java environment with virtually unlimited resources like memory, time and processing power. This is nothing compared to the embedded, reverse-engineering development that the real ML developers do...

Anyway, I'm glad that some of guys find that little tool useful... makes it even more fun to code

My friend, maybe technically on that matter you are not as deep as the ML guys, which of course you are right, but practicely you are a legend, you gave us a way to use raw even if thats maybe temporary, cause you never know! the other legends on the ML side might discover something! (they always do, dont they?) What am saying is that you give hope to this project just like the rest of the guys! and also indirectly helping the developers by taking our minds of the pink dots so leaving them developers do what they do best!

[multi.flexi]

I tried compare interpolation, without PDR and dead pixel option.


It's 1280x720. Maybe I have done something wrong, dont't know.

[foorgol]

I tried compare interpolation, without PDR and dead pixel option.

From the pattern of the removed pixels I would guess that you used an old version of the tool. A new release is on its way and for the time being just use the debug version mentioned above:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

This should fix our issues.

Oh, and make sure you're using raw2dng to get the DNGs.

[foorgol]

And also indirectly helping the developers by taking our minds of the pink dots so leaving them developers do what they do best!

Hehe... you're right... I remember the discussions in the 650D-thread being quite emotional... about whether the developers should spend time on improving RAW or not... I guess there's some relief now on that front

[foorgol]

Short update on the dot remover:

  * Changed algorithms to support EOS M better

  * Fine-tuning EOS-M dot location database

  * First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed


Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!

[donjames150]

Short update on the dot remover:

  * Changed algorithms to support EOS M better

  * Fine-tuning EOS-M dot location database

  * First promising tests on DNGs created with RawMagic; used talosectos' file for testing and have almost all dots removed


Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!

thanks to both of you

[papkee]

Alright folks, I have a new video out showcasing RAW on the 650D, cleaned up with the remover.

[donjames150]

Alright folks, I have a new video out showcasing RAW on the 650D, cleaned up with the remover.

Nice video

[spider]

Mixer2 and me are currently optimizing and testing, but we should have a new version in a few days!

I am so excited... thank you foorgol and Mixer2

[pileman]

Quick preview of what I could max out with 650D raw + pink dot removal, not perfect. Has some frame skipping.

Actors: dog.

Also note the first scene is with the latest pink dot removal therefore it has less noise or dots compared to the earlier build which is used in scene 2.

I will soon make low light timelapse video just to see how the dot removal tool handles lower light conditions. Shouldn't be much difference?

[donjames150]

these two were taken one after the other with the same camera settings. Kit lens 18-55 at about  33mm iso100 and f22.
The H264 was 1920 x 1080 (24fps) and the raw was 1728 x 972 (10fps). Then output at 1280 x 720. I adjusted the colors
on the raw to match the h264 as best as I could and applied very minor stabilization to get rid of the jerkiness that I always
see in my raw videos (from the process I guess). You can clearly see better into the shadows with the raw.

[multi.flexi]

From the pattern of the removed pixels I would guess that you used an old version of the tool. A new release is on its way and for the time being just use the debug version mentioned above:

https://dl.dropboxusercontent.com/u/22843507/MagicLantern/PinkDotRemover_debug.zip

This should fix our issues.

Oh, and make sure you're using raw2dng to get the DNGs.

Well if I use raw2dng and then PDR I can't edit it in Davinci Resolve. Is there a way how to get rid of pink mess and also convert it to CinemaDNG?