Vertical stripes revisited (5D Mark III)

[MarS]

Thank you very much a1ex for flat processing. I was just searching for such a feature! It will be useful for high-resolution planetary astronomical imaging. 

First shots on Moon with MLV files and crop 3x (without dark and flat yet) make me really enthusiast...
http://www.magiclantern.fm/forum/index.php?topic=17825.0

Thank you GutterPump.  I'm working on a Mac application to produce avi files for astronomical tools like Registax, AviStack 2 and AutoStakkert 2. You're binary will be useful.

I would have a question about processing dark frame. Cause there is no temperature stabilization with Canon cameras, can we compensate the temperature difference between main video and dark?

[a1ex]

TBH, I didn't really look into the thermal variations of dark frame, so I'm afraid I cannot answer.

For raw video, on 5D3 at least (but not on 5D2), you can set the exposure time to 0, capturing bias frames every now and then (for example, one bias frame every 10 regular frames or whatever). It's not implemented in menus; if there is interest in exploring this feature, I can look into it.

[MarS]

Thank you for your feedback a1ex. 

So I've just made some dark tests with my 5D Mk III. As far as I can see, thermal noise has no real impact with short exposures. Here is a video showing dark with growing temperature (22°c to 52°c according to ML display)...

-dark signal was stretched for visualization-
It's a good news for calibration. I don't think we really need to take care about bias. Vertical stripes seem to be constant so dark and flat should do the job. I will test on astronomical images as soon as possible.

I've just finished my own Mac GUI to convert easily MLV to lossless AVI (needed for astronomical softwares). -It can also export mov files-
Dark and flat calibration are already integrated...

I'll post a download link quickly.
Wish it helps. 

[Danne]

Great app. Is it open source? I,m interested to see how you created your gui and how your workflow is going from MLV to mov? Are you using FFmpeg? I had my own puzzling with darkframe subtraction workflow a while ago.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg164871#msg164871

[cmccullum]

Maybe this is stupid, and a little late as the conversation has moved on,, but could high ISO noise reduction be a factor here?
I know it's recommended to turn it off, but I don't know why. Even if it absolutely is not a factor, could someone take a moment to explain why we turn it off?

[Lars Steenhoff]

ooh that gui looks very nice!

good work, and if it can export to prores with a lut in the future? 

[a1ex]

but could high ISO noise reduction be a factor here?

If you are talking about vertical stripes, then you may want to know the effect is more obvious at lower ISOs. At higher ISOs, the pixel noise becomes greater than the column amplification differences, so the noise will effectively hide this artifact.

I know it's recommended to turn it off, but I don't know why

I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

You may want to mention the source of this recommendation, and maybe ask the author what difference it makes.

[MarS]

Thank you guys! I've just created a new topic for MLVToMovie here...
http://www.magiclantern.fm/forum/index.php?topic=17893

Danne, yes I'm using MLV_Dump, DCRaw and FFmpeg. I don't think to make it open source cause it's a signed application and I'm using personal proprietary code for multi-tasking but I can send you code source for study. Send me a private message with you mail if you want.

Lars Steenhoff, I'm not a video master but if your needs can be done with FFmpeg I can look at it. Feel free to switch on the MLVToMovie topic to talk about It.

[Kharak]

I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

as you write it, I remember long time ago the talk was to deactivate it because it would leave more free memory.. But I too have no confirmation of this, just something that got stuck in my head reading through all the hundreds of posts here back in 2013 when the whole ML Raw ship set sails.

[cmccullum]

If you are talking about vertical stripes, then you may want to know the effect is more obvious at lower ISOs. At higher ISOs, the pixel noise becomes greater than the column amplification differences, so the noise will effectively hide this artifact.

I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

You may want to mention the source of this recommendation, and maybe ask the author what difference it makes.

Thanks that answers everything. I actually wasn't aware that the stripes were more of an issue at lower ISOs (I've only noticed them when I crank the ISO, but I haven't done any kind of real testing)

I don't remember where I read about disabling it, but I think it was in the crop 2.5k 50fps thread. Could very likely just be misinformation on my part though. I'll look into it

[a1ex]

Here we are talking about vertical stripes in highlights. Updated the first post with some background info.

[Danne]

Reading up upon flatfield frames and it seems a combination of darkframes and flatfield frames can be obtained here which in theory could be some optimal correction combo for hot pixels, banding, color cast, dust etc. One question. Are the flatfield frames depending on being matched in iso, fps, aspect ratio etc the same as for creating applying darkframe averaging? Idea is to test it out and also implementing it to a script so want to minimize user error first.

[a1ex]

I didn't do any tests regarding this, but I would expect most gain defects to be the same at all ISOs.

[Danne]

Some initial very loosely testing. Think the result is usable though.
Filmed a short sequence with canon eos m shutter 50fps iso 800 f2.0 MLV.
Then filmed a short flat frame by simply filming a white surface at f22. Not sure if f22 was the best setting. Might be even better wide open? And of course even better doing it right(http://wilson.bronger.org/lens_calibration_tutorial/#id3) More tests to be done.
Vinjetting is affected as stated.
Focus pixels are affected in some way.
Purple fringing?
All in all very promising feature.

original


flat frame


original


flat frame

[Danne]

Did another test. Created a flatfield mlv by filming at f2.0 straight at a photo bouncer (eos M). Experimenting. Check what it did to the focus pixels! Not all of them are gone but damn close. Developed in dcraw. Applied log C gamma for a flat revealing look.

original


flatfield


original


flatfield




*Update
Ran a lot of different tests around the clip just posted. Some feedback here.

Without lens - Vignetting seems gone but focus pixels more pronounced than withouth a lens on
With lens attached - Aperture wide open seems to take care of vignetting and most of the focus pixels. With aperture set to 22 focus pixels will be more pronounced.
Changing aperture while creating the flatframe - Will not show any difference to the dng files created. Changing exposure doesn,t seem to change anything.

Ok so a lot talk about focus pixels but maybe this could be a solution to how to eliminate those in post or even in mlv_dump?
Here is a flat frame created from an eos m
https://drive.google.com/file/d/0B4tCJMlOYfirLS1wTTh2dDk5aHc/view?usp=sharing

Oops, sorry about the off topic eos m testing. Realize it,s about 5D mark III.

[hyalinejim]

Thanks A1ex for the updated mlv_dump which totally sorts out the vertical banding I was seeing in 5d3 crop mode using Apr 03 build. However, I have some DNG sequences that I'd like to correct, but I don't have the MLVs anymore. Is there anything I can do to the DNGs now in order to fix banding? An example is here:

DNG file extracted with MLVFS

https://www.dropbox.com/s/xc2xgi9vo2365xl/M02-0916_000000.dng?dl=0

This is what it looks like with aggressive colour correction (and I do actually want to push it this far for the short doc I'm working on) - screenshot is cropped from the full 1920x1080 and at 100%

[MarS]

I've just tested flat calibration this morning on the crescent moon. As far as I can see, the dark signal is not removed automatically from the flat field when applying -s and -t options together. It's not really a mistake but the good mathematical process should be:

Calibrated raw = (raw-dark)/(flat-dark)*(average value of flat-dark)

As a result, take care when using flat calibration. We have to remove dark from flat with mlv_dump before using flat...

mlv_dump -s dark.mlv -o flat-dark.mlv flat.mlv

Otherwise noise will be reintroduced by the flat.

-I hope my explanation is clear enough.-

[MarS]

A grab from "The Handbook of Astronomical Image Processing" by Richard Berry & James Burnell to illustrate my last message...

[a1ex]

Agree; however, this will require some extra code paths (to handle both cases).

Do you mind showing the difference it makes? If the flats are exposed to the right, I think the difference should be minimal (except for hot pixels, which are probably best measured, rather than just dark-subtracted).

[MarS]

Everything works fine to me with mlv_dump. I don't think this job has to be handle by mlv_dump. We just need to know we have to be aware for the flat dark subtraction. I'll update my GUI.

I've processed a movie from this morning (very poor seeing but it doesn't matter for tests). Here is the first frame with different calibration...
http://astronomy.eliotis.com/tests/m29-0657-6400Iso-calibration-test.zip
- (raw-dark)/(flat-dark): tmp000000_raw-dark-div-by-flat-dark.tiff
- (raw-dark)/flat: tmp000000_raw-dark-div-by-flat.tiff
- raw-dark: tmp000000_raw-dark
- raw/flat: tmp000000_raw-div-by-flat
- raw (No calibration): tmp000000_raw

And here are the movies to see more easily random and constant noise:
- (raw-dark)/(flat-dark):

- (raw-dark)/flat:

- raw-dark:

- raw/flat:

- raw (No calibration):

Do not take care of corners. My focal Reducer/Corrector reduces the field. As a result, flat can only improve noise in the borders.

In my opinion, I do not recommend using flat without flat dark. But as you say a1ex, It can be less significant with high lights.

Danne, if you want to test. Flat dark could be the ultimate trick to remove focus pixels perfectly (or not).

[Danne]

Flat dark? How is that created? Shooting at a complete dark area? Isn,t that a darkframe?
Checked your movies. Darkframe works best? Vignetting issue is too pronounced in the raw/flat example therefore revealing pixel circle?
Not really sure what to think about flatframe. It will even out the pixel gain on the overall picture but if too pronounced other artefacts starts to appear?

[MarS]

Danne, yes it's a simple dark. flat dark is a dark shooted with the same parameters (Speed and Iso) as your flat. It's the same concept as the dark witch must use the same parameters as the raw images. It's really easy to do. After recording your flat as usually, you put the lens cover and launch a second record.

Vignetting issue with my shoot is only due to the fact that my focal reducer on my telescope limits the field so there is no more light to calibrate in the borders. You do not encounter such an effect with standard lens.

If you look to this video (raw-dark)/(flat-dark):

The field of view is quite perfect in the image circle. There is only random noise (no more vignetting and no more vertical or horizontal stripped lines). It's not the case with the other videos.

[Danne]

After recording your flat as usually, you put the lens cover and launch a second record.

Yes, for creating a darkframe?

The field of view is quite perfect in the image circle.

Yes, noticed now. Looks very clean.

If you look to this video (raw-dark)/(flat-dark):

Raw-dark, flat-dark. What,s the difference?
I understand it as flatframe created with light diffusor. Darkframe created with lens cap on. Flat-dark??

Two formats right not a third one ?

If I recall I tried applying a darkframe to the focus pixel footage but didn,t work.

[MarS]

Yes it's a second dark. Mathematically, each dark must respect the raw image it will be subtracted from:
- the dark "classic" must respect ISO and Exposure of the raw mlv you want to process.
- the flat dark must respect ISO and Exposure of the flat mlv.

I wish this diagram helps...


When you shot your flat mlv with a light diffusor, you get the same repeatable noise that you get in the raw mlv. If you subtract the dark from your raw mlv you get no more repeatable noise. Now, if you apply a simple flat you inject the repeatable noise it also contains. As a result, the best way is to subtract a "flat dark" to remove repeatable noise from flat before the flat division.

I hope my explanation (and my English) is good enough. Lol 

[DeafEyeJedi]

The image calibration diagram definitely gets me going... [emoji7] Thanks for sharing @MarS!