[600D RAW] Dots, dots and more dots.

[escho]

I have no mac, so I cannot help

Edgar

[escho]

For to discuss: https://bitbucket.org/escho/escho-magiclantern/commits/dad031cdc77d1a00e7c4984b48dbbedc371ad282

Edgar

[Luiz Roberto dos Santos]

Congrats escho, great.
Maybe an implementation of these code in MLV2DNG? Just one un-official mod, for now?

[escho]

Thanks

mlv2dng is to big for my understanding at the moment, but I´m working on it (to understand it) 

Edgar

[hjfilmspeed]

Just shot a promo vid in ML raw on 5d3. I used batchelor 2.3 and swaped the raw2dngs.

Anyway i shot a lot of low FPS+high iso clips in this so there should be some pretty bad offenders that i would like to use this for so ill give it a whirl

 i processed one  though file though raw2dng_esco and the same file through regular raw2dng and in resolve  both had equal amount of bad pix. i didnt notice a difference in the 2 clips. It was high iso probably 12800. Im sure i did something wrong. The pixels i usually see are white and a lot of the times they are in similar spots. strange thing is there are not as noticeable in acr. Im sure i did something wrong. I downloaded after the link was fixed. also are these white pix considered hot pix? and will this remedy hot pix?

[escho]

Please upload a short clip somewhere.

Edgar

[hjfilmspeed]

would you like the raw file?

[iaremrsir]

I might be able to compile on Mac, someone would have to guide me through it though. I've only ever done small projects with just a few source and class files.

[iaremrsir]

There are no fixes about FPN yet. For this, I might have to record some optical black data. I want to study it on silent pictures first, because they include full optical black data; so if you have a burst of silent pictures that show this problem, upload them here: www.magiclantern.fm/forum/index.php?topic=9564

I'll go ahead and shoot some tomorrow and upload them. It'd be awesome to get FPN correction because it's very apparent in the footage.

In regards to the current raw2dng_escho, it worked very well. Nice and speedy. The only exception was with clips that seemed to have cold pixels everywhere, other than that it works as expected.

[a1ex]

Can you upload such a clip?

[escho]

would you like the raw file?

Yes

Edgar

[hjfilmspeed]

Yes

Edgar

Ok I used the raw2dng_escho from A1ex post. I sent you a pm with a test clip

[escho]

I just found a high-iso-file, I recorded yesterday night, where the frames have cold pixels in different positions, too.  The only difference to my other videos is, that I recorded it with mlv_rec and converted it to legacy raw with mlv_dump. Not sure, what happened.

Edgar

[a1ex]

I remember some discussions saying that first frames in mlv_rec might be corrupted, but didn't check it myself. Worth looking into it.

[escho]

I changed raw2dng to let it print the locations of the cold pixels. And I compared different frames (not only the first one). I see three things:

1. A part of the cold pixels have the same position in every frame
2. Other cold pixels have have the same position in the frames, but not in every frame.
3. The rest has completely different positions in the frames

I will try some other clips from yesterday later

Edgar

[a1ex]

In this case, you may want to extend the algorithm by scanning for example the first 10 frames for bad pixels. If they all stay in the same location, you don't need to continue scanning for the rest of the frames (just use the cached locations). If there are changes, just scan all the frames, a little slower, but should remove them all.

This optimization is a little more complex though; I can look into it hopefully next week. Probably only matters for those who use SSDs.

[escho]

Yes, scanning all frames for cold pixels removes all of them. But I´m testing yet. An interim result:

ISO 6400 shows this behaviour with the different locations.
ISO 100 - ISO 1600 (tested not all ISOs in this range): The cold pixels have the same positon in all frames.

Edgar

[escho]

Yes, comfirmed for my camera.
ISO 6400 different locations
ISO 1600 same locations.

I didn´t try ISO 3200 yesterday.

Edgar

[Luiz Roberto dos Santos]

Yes, comfirmed for my camera.
ISO 6400 different locations
ISO 1600 same locations.

I didn´t try ISO 3200 yesterday.

Edgar

Maybe because they are not real ISO's, but pushed digitally?

[iaremrsir]

Can you upload such a clip?

The one's with a lot of dead pixels or the FPN clips?

[escho]

A nice test result for my 600D:

no crop, cold pixels per frame

ISO 100:  34
ISO 200:  34
ISO 400:  34
ISO 800:  42
ISO 1600: 57
ISO 3200: 797
ISO 6400: 797 - 799

5x-crop cold pixels per frame

ISO 100:  16
ISO 200:  16
ISO 400:  16
ISO 800:  19
ISO 1600: 27
ISO 3200: 109
ISO 6400: 111 - 128

Edgar

[escho]

What about  commandline-options?

An option for scanning all frames for cold pixel (default scanning only the first frame)
And chroma smooth could be enabled by such an option, too.

Just an idea...

Edgar

[Luiz Roberto dos Santos]

This modification is really necessary? I mean, people have come to conclusion that ISO's greater than 1600 are pushed digitally by the camera, and a1ex has said he prefers to push this information in post-production than letting the camera do.
What your thoughts on this? I think we could save time about it and turn in the study of ADTG, which seems the most promising. It is just a warning in the official announcement... 

[escho]

Just shot a promo vid in ML raw on 5d3. I used batchelor 2.3 and swaped the raw2dngs.

Anyway i shot a lot of low FPS+high iso clips in this so there should be some pretty bad offenders that i would like to use this for so ill give it a whirl

 i processed one  though file though raw2dng_esco and the same file through regular raw2dng and in resolve  both had equal amount of bad pix. i didnt notice a difference in the 2 clips. It was high iso probably 12800. Im sure i did something wrong. The pixels i usually see are white and a lot of the times they are in similar spots. strange thing is there are not as noticeable in acr. Im sure i did something wrong. I downloaded after the link was fixed. also are these white pix considered hot pix? and will this remedy hot pix?

Hi

I looked at your raw-file and decoded it. There are no cold pixels in the frames. The dark level is 2044. That means, all dots with a value of -2044 will get recognized as cold pixels and removed. The darkest pixel in your frames is -145, that´s somewhere  around dark noise, I guess. At the right side of the frames, I see a red dot, This pixel has a value of a little bit over 1000.
So the cold pixel fix works correct, if it is doing nothing. This fix is not designed to correct hot pixels.

I looked at the white dots, too. The all have values inbetween 1000 and 1500.

Edgar

[escho]

This modification is really necessary? I mean, people have come to conclusion that ISO's greater than 1600 are pushed digitally by the camera, and a1ex has said he prefers to push this information in post-production than letting the camera do.
What your thoughts on this? I think we could save time about it and turn in the study of ADTG, which seems the most promising. It is just a warning in the official announcement... 

I do not force someone to code this into raw2dng, so all you guys can work in ADTG, if you want  I will try to work this out for myself as a next step in my C-learning curve. And I for myself would like to have the possibility of the commandline-options, because this would prevent me from working with different raw2dng-builds, as I´m doing now. 

Edgar