Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: a1ex on August 24, 2016, 11:10:46 AM

Title: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 24, 2016, 11:10:46 AM
Topic split from http://www.magiclantern.fm/forum/index.php?topic=17021.0

Old discussion about vertical stripes: http://www.magiclantern.fm/forum/index.php?topic=5614.50

With crop_rec, these stripes also appear in H.264 (http://"http://www.magiclantern.fm/forum/index.php?topic=17021.msg166354#msg166354"). They can be fixed in post (http://www.magiclantern.fm/forum/index.php?topic=17021.msg166432#msg166432), but it would be best if we could avoid them in the first place.

Vertical stripe fix for H.264:

stripe-fix-h264.html (http://a1ex.magiclantern.fm/bleeding-edge/stripefix/stripe-fix-h264.html)

Note: the stripes we are talking about in this thread are visible in highlights (e.g. sky at low ISO).

Is the vertical banding present in the highlights or shadows?

If the banding is in the highlights, darkframe subtraction won't help. Banding in the highlights is a sign of a multiplicative defect (the column gains are off slightly, which is fixed by multiplying pixels values by some correction factor).

If the banding is in the shadows, the "vertical stripe fix" won't help. Banding in the shadows is a sign of an additive defect (there is some linear offset, which is fixed by simply subtracting some correction value from the pixel values).

Original message:
[...]
I'd say let's review the vertical stripe fix on 1.1.3, and if it's everything alright, I'll include it in the main builds (and fix the 1.2.3 crop build) soon.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 24, 2016, 11:21:42 AM
I just did a test,

You can see the video there :


pass : ML

Of course images are overexposed and underexposed for a best view of the stripes, it's not a final grade.

Hope it will help.

My conclusion is, the vertical stripes fix is working, the strips are softer. But in fact, it is invaded by the evil pixels. Is there a more effective solution than PixelPatcher that leaves no trace after correction ?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 24, 2016, 11:35:27 AM
I'm unable to see the video; can you upload some small 1:1 crops?

Was the test done in crop mode or not? I actually expect some problems in crop mode, similar to the H.264 issue from above. What software did you use for raw conversion? I thought most of them already handle bad pixels very well.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 24, 2016, 11:56:39 AM
crop3x.2016Apr02.5D3113
Extract MLV with raw2cdng v 1.7.5 | Vertical banding correction : On
1920x1080 | 200 ISO

(http://image.noelshack.com/fichiers/2016/34/1472032008-v1-0002-m24-1019-23augapr-mp4-snapshot-00-01-2016-08-24-11-43-01.png)


crop3x.2016Aug23.5D3113
Extract MLV with raw2cdng v 1.7.5 | Vertical banding correction : On
1920x1080 | 200 ISO

(http://image.noelshack.com/fichiers/2016/34/1472032007-v1-0002-m24-1019-23augapr-mp4-snapshot-00-04-2016-08-24-11-43-15.png)

crop3x.2016Aug23.5D3113
Extract MLV with raw2cdng v 1.7.5 | Vertical banding correction : On
PixelPatcher OFX : On
1920x1080 | 200 ISO
(red arrow are showing PixelPatcher traces issues)
(http://image.noelshack.com/fichiers/2016/34/1472032425-v1-0002-m24-1019-23augapr-mp4-snapshot-00-09-2016-08-24-11-51-45.png)


The test was did without crop mode, but i can do the same with it.

What software did you use for raw conversion? I thought most of them already handle bad pixels very well.

What softwares do you have in mind for correct bad pixels ? I always use raw2cdng 1.7.5 but i dont saw any options for correct it.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 24, 2016, 12:21:25 PM
Since the vertical stripe fix is in camera, you must not also correct it in post-processing. I thought this was obvious.

For the bad pixels, can you try MLVFS or mlv_dump?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 24, 2016, 01:00:29 PM
Since the vertical stripe fix is in camera, you must not also correct it in post-processing. I thought this was obvious.
For the bad pixels, can you try MLVFS or mlv_dump?

Yes i know, but i did the test with vertical stripes correction ON and OFF with raw2cdng 1.7.5 and MLV file from 23aug build, there is really no difference.


Vertical banding correction : On
(http://image.noelshack.com/fichiers/2016/34/1472036248-cropbuild-23aug-stripescorrection-on.png)


Vertical banding correction : Off
(http://image.noelshack.com/fichiers/2016/34/1472036254-cropbuild-23aug-stripescorrection-off.png)

MLVFS make my computer crashing, i'm not on my computer's work, it's my laptop for this time, i will try later. But i can already say bad pixels are gone very well with MlRawViewer v1.4.3
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Kharak on August 24, 2016, 01:30:24 PM
Vertical Stripe Fix, in camera??

Wow, how is this done @Alex?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 24, 2016, 01:47:50 PM
Canon code does it, see the pull request description.

This was probably a bad decision taken back in 2013 (http://www.magiclantern.fm/forum/index.php?topic=5614.msg41070#msg41070) (see the comments in raw.c).
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 24, 2016, 02:29:25 PM
Do you think it will be possible to design a new version of raw2cdng with the option to correct the bad pixels ?
Unless this is already the case but I did not see the option provided for this purpose ..
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 24, 2016, 02:37:35 PM
It's very easy, but you need to ask chmee. I thought it's already done on most converters because some cameras (60D for sure) have such bad pixels in their raw output. Not to be confused with focus pixels.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 24, 2016, 08:14:05 PM
Hi! Recorded a small clip mlv and 3x crop mode to check in the new dots. Exciting stuff. Mlv_dump --fixcp(fix cold pixels) does indeed a good job. Processing comes with a cost. It takes longer and I believe it has to be optimized. Check especially the shadowy parts in the pushed exposure photo. This dng was also opened up in acr which usually takes care of cold pixels.
Hard to see the improvement in the overall image since a1ex vertical banding code always given me stellar results but could this other raw stream deliver even better images you say?

DNG orginals(13mb)
https://drive.google.com/file/d/0B4tCJMlOYfirV2FvRjFQYWRTOGM/view?usp=sharing

(https://s14.postimg.cc/7h6t3mfdd/Screen_Shot_2016_08_24_at_20_04_36.png)

(https://s14.postimg.cc/3m3f11w7l/Screen_Shot_2016_08_24_at_20_07_31.png)

(https://s14.postimg.cc/5rxpvjzo1/Screen_Shot_2016_08_24_at_20_05_01.png)

(https://s14.postimg.cc/ikltvhb9t/Screen_Shot_2016_08_24_at_20_07_13.png)

(https://s9.postimg.cc/kdiwomvi7/Screen_Shot_2016_08_24_at_19_46_14.png)



Original size
(https://s9.postimg.cc/qams5gcn3/Screen_Shot_2016_08_24_at_19_41_19.png)

(https://s9.postimg.cc/i7olu4s1r/Screen_Shot_2016_08_24_at_19_46_29.png)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 24, 2016, 08:40:12 PM
hm, another issue came up with mlv_dump when I tried processing through scripting in MLP. I usually process a mlv file with four streams in parallell t o speed up things. When selecting fixcp mlv_dump only seems to work if it does the sequence starting from zero. When running something like mlv_dump -f 2-30 --dng --fixcp M24-1930.MLV dng is spitted out but without fixcp applied. This breaks parallel processing.

Original script looks like this
Code: [Select]

if ls "$jo"A_lut_hold/"MLV_mlv_dump_settings.txt"
then
mlv=$(cat "$jo"A_lut_hold/"MLV_mlv_dump_settings.txt" | head -1)
fi

~/Library/Services/MLP.workflow/Contents/mlv_dump -f 0-1 --dng $mlv -o "$output_dng"${BASE}_1_"$date_01"_ $FILE | awk '/Processed/ { print $2; }' > /tmp/magic_l_prores/framecount
    frct=$(cat /tmp/magic_l_prores/framecount) ;
    rm /tmp/magic_l_prores/framecount
~/Library/Services/MLP.workflow/Contents/mlv_dump -f 2-$(($frct/4)) --dng $mlv -o "$output_dng"${BASE}_1_"$date_01"_ $FILE &
~/Library/Services/MLP.workflow/Contents/mlv_dump -f $(($frct/4+1))-$(($frct/4+1+$frct/4)) --dng $mlv -o "$output_dng"${BASE}_1_"$date_01"_ $FILE &
~/Library/Services/MLP.workflow/Contents/mlv_dump -f $(($frct/4+1+$frct/4+1))-$(($frct/4+1+$frct/4+1+$frct/4)) --dng $mlv -o "$output_dng"${BASE}_1_"$date_01"_ $FILE &
~/Library/Services/MLP.workflow/Contents/mlv_dump -f $(($frct/4+1+$frct/4+1+$frct/4+1))-$frct --dng $mlv -o "$output_dng"${BASE}_1_"$date_01"_ $FILE
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: axelcine on August 24, 2016, 10:37:48 PM
Hooray! Chapeau!
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 25, 2016, 12:06:40 AM
When selecting fixcp mlv_dump only seems to work if it does the sequence starting from zero.

Just pushed a change that hopefully solves it (didn't actually try). It's in the bad_pixels branch.

Also did a few other related changes to mlv_dump:
- cold pixel fix is now enabled by default
- vertical stripe fix is now disabled by default
- option to scan every single frame for cold pixels (slow, but I remember some reports with moving cold pixels, for example with very bright specular highlights)

Processing comes with a cost. It takes longer [...]

Unless you have LOTS of hot pixels, processing time should not be noticeable. This step only processes a tiny percentage of the entire image (unlike the stripe fix, which processes every single pixel).

Benchmarks on a 1GB full-res MLV I had around (with only a few cold pixels), after a few test runs to cache the file:
Code: [Select]
time mlv_dump 59600001.MLV --dng
real 0m6.457s
user 0m1.260s
sys 0m2.912s

time mlv_dump 59600001.MLV --dng --no-fixcp
real 0m6.393s
user 0m1.044s
sys 0m2.984s

time mlv_dump 59600001.MLV --dng --fixcp2
real 0m14.115s
user 0m6.652s
sys 0m3.032s

time mlv_dump 59600001.MLV --dng --fix-stripes
real 0m29.101s
user 0m12.200s
sys 0m2.756s
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 25, 2016, 02:34:06 AM
Just compiled mlv_dump from the bad_pixel branch. Tried shortly and you fixed parallell processing and fixcp is on by default. Great!
Here is the mlv_dump binary for anyone who wants to test out.
*updated 29/8 2016
mlv_dump (https://drive.google.com/file/d/0B4tCJMlOYfirbXNEbHVYcFNGX3c/view?usp=sharing)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 25, 2016, 07:11:00 AM
Just reran my test dng sequence and the fixcp seems to clean up the photos more or less perfectly to this sequence. Pushed the exposure and no color cracking color blotches to be seen hiding. (Opened up in acr which probably cleaned up the dng a little bit more by default).
(https://s10.postimg.org/xvx3tak3d/Screen_Shot_2016_08_25_at_07_00_27.png)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 26, 2016, 11:07:55 AM
I posted a MLP version which will work with files coming from the crop_rec.mo module.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg171272#msg171272

Tested with time command and cold pixel fix is way faster than vertical stripes correction. Probably around twice the speed.Maybe even faster. My mistake before was I had both vertical stripes and cold pixel fix turned on so of course slower.

Just tested briefly with h.264 material and it had some pixels showing on higher isos. Found a filter from ffmpeg which probably can get rid of those in post.
https://ffmpeg.org/ffmpeg-filters.html#delogo

dmilligan talked about metadata to find out what stream is doing what , mv720, mv1080. Could the modes be specified something like
crop_mode_1
crop_mode_2
crop_mode_3

or

crop3xmode_1
crop3xmode_2
crop3xmode_3

Maybe even shorter

crop3xm_1
crop3xm_2
crop3xm_3

Thanks a lot for this great module.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 26, 2016, 12:46:43 PM
I just discover an issue between the 2 april and 23 aug version.

I can't properly recovery  hightlight with the 23aug version in Resolve, highlight stay pink.

Exemple :


02 april build
(http://image.noelshack.com/fichiers/2016/34/1472208543-02augbuild.jpg)

23aug build
(http://image.noelshack.com/fichiers/2016/34/1472208543-23augbuild.jpg)

With the same setting (Balance color, tint etc..)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 26, 2016, 12:55:51 PM
Put up a dng file? Could you check white levels with exiftool?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 26, 2016, 01:24:43 PM
Yes i always work with DNG files in Resolve.

Here DNG from  02 apr and 23 aug builds

https://www.dropbox.com/s/lc7w0caliwc55s1/M24-1019_23augapr00000.dng?dl=0

https://www.dropbox.com/s/2xwo4w3jtepm9hl/M24-1024_2apr00000.dng?dl=0

MLV :

https://www.dropbox.com/s/h4nbsqkujmil6ue/M24-1019_23augapr.MLV?dl=0

https://www.dropbox.com/s/e6igo1ivfo97b4g/M24-1024_2apr.MLV?dl=0

With level is the same for both : 65535
Extraction was done with raw2cdng 1.7.5
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 26, 2016, 01:51:02 PM
In the first DNG from above, the white level is at 60375 (this DNG was promoted to 16-bit data). However, I cannot reproduce this behavior - white level on my camera, with the latest build, is at 16383 (for 14-bit data). ISO 100, 1/33, 3x crop, 1080p25.

Just in case - can you upload DNGs created with mlv_dump? The above DNGs were scaled by the converter.

I don't expect this issue because I didn't change the register configuration. The problem might be random and appearing on both builds - can you rule out this case?

To recover existing footage, reduce the white level in the DNGs with exiftool (for the above DNG, 60374 should do the trick).
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 26, 2016, 01:57:01 PM
I can't use mlv_dump, i don't work on MAC, and MLVFS don't work well with my PC (this is why i use raw2cdng)

But i have upload MLV, if someone has the courage to download 700 mb ..

I will made others test with highligt and see and see how it goes.


EDIT :
I don't expect this issue because I didn't change the register configuration. The problem might be random and appearing on both builds - can you rule out this case?

After have done some new tests i think it was a false alert too, especially if you have not changed the portion of code that might generate this error, but in case I prefer to report if others people encounter this problem.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 26, 2016, 03:33:54 PM
I can't use mlv_dump

You can. (http://www.magiclantern.fm/forum/index.php?topic=7122.0)

After have done some new tests i think it was a false alert too

Well, you've got a DNG with the white level incorrect. Can you find out what settings trigger the issue?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 26, 2016, 04:01:14 PM
You can. (http://www.magiclantern.fm/forum/index.php?topic=7122.0)

Thanks I did not know there was a windows version !

DNG from mlv_dump here :

https://www.dropbox.com/s/twruf4cc9jbsr9p/M24-1019_23aug_frame_000000.dng?dl=0

Well, you've got a DNG with the white level incorrect. Can you find out what settings trigger the issue?

It will be a long and laborious work for find it

But i will try because i want see this new build working well :)


Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Quentin on August 26, 2016, 06:22:00 PM
Tried with my 5D 1.1.3, R/G/B pixels here and there.
Need to do some more experiments
Thanks for the efforts
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 12:38:41 AM
I'm back with news tests. Today i shot some footage ,as usually, in ETTR / 200 iso / 1/48 shutter speed / WB = sunlight.

I still get weird hightligt recovery in Resolve, with this pink color. I never get this before the 23aug crop build.

With the 02apr crop build, when i get pink color in highlight, i'm able to erase it with changing balance color/ tint options, but not with the 23aug build, it's like "frozen" on this pink color.

I can't really know what option/configuration is the cause of this issue

Here a DNG sample convert with mlv_dump (i removed bad pixels with it too) :


https://www.dropbox.com/s/vwssn3aup1mvojf/M27-1808_000000.dng?dl=0

EDIT : in waiting to find a solution ,  it is possible to use the qualifer option for fix it in Resolve.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 08:46:22 AM
I,m not sure what the problem is with this dng. I just tested it in resolve with highlight recovery on and also pulled back highlight abnormally much and no pinks. White level set to 15000 as mlv_dump usually does.

(https://s16.postimg.cc/vh6xy4llx/Screen_Shot_2016_08_28_at_08_39_31.png)

(https://s11.postimg.cc/xcivg1nub/Screen_Shot_2016_08_28_at_08_43_04.png)


*update

Ok, so selected BMD and pink highlights are appearing. How can you rule out that it isn,t resolve doing a bad job here?
(https://s21.postimg.cc/93b2jlv3b/Screen_Shot_2016_08_28_at_08_51_34.png)

For the last dng file I doubt it has anything to do with the builds. Test working with it in acr and you,ll see both dng files works great.  For the dual iso dng file with white level around 65535 isn,t this caused by raw2cdng? Could you film the same sequence with two different builds?
*For some reason raw2cdng won,t let me overwrite white level in the dng example. 

Could you upoad the dual iso MLV file example? You could shorten the actual MLV file with mlv_dump like this.
mlv_dump -o new.MLV -f 5 [drag/your/MLV/file/here] hit enter

See to it that you create a different output name (where I put in the name new.MLV) than the name of the original MLV file or mlv_dump will overwrite your original file.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 28, 2016, 12:10:20 PM
Could you film the same sequence with two different builds?

+1. Once you get a quick and repeatable way to reproduce the issue, we can narrow it down.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 12:25:16 PM

Ok, so selected BMD and pink highlights are appearing. How can you rule out that it isn,t resolve doing a bad job here?
(https://s21.postimg.cc/93b2jlv3b/Screen_Shot_2016_08_28_at_08_51_34.png)

The pink color in highlight is worse when I use cinelog-C or if i put REC709 in color space (yes usually we don't need to use rec709 for raw, but it's just more revelant for this issue.)


For the last dng file I doubt it has anything to do with the builds. Test working with it in acr and you,ll see both dng files works great.  For the dual iso dng file with white level around 65535 isn,t this caused by raw2cdng? Could you film the same sequence with two different builds?
*For some reason raw2cdng won,t let me overwrite white level in the dng example. 

I will do a second test with 2 differents builds yes, but what about your tests ? You don't met the same issue than me on Resolve with footage in highlight ?

Quote
white level around 65535 isn,t this caused by raw2cdng?

I did the test with MLV_dump it's the same problem.

Could you upoad the dual iso MLV file example? You could shorten the actual MLV file with mlv_dump like this.
mlv_dump -o new.MLV -f 5 [drag/your/MLV/file/here] hit enter

See to it that you create a different output name (where I put in the name new.MLV) than the name of the original MLV file or mlv_dump will overwrite your original file.

Here the short MLV :

https://www.dropbox.com/s/jr6viin3g2d6vk0/new.MLV?dl=0

 but i don't understand what you mean about "dual iso MLV" ? I don't shot any video in dual iso with the new build. (I never use dual iso)

ps: Sorry if my English is approximative, I try to do my best to answer you.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 12:38:18 PM
I meant for you to upload the MLV that is creating dng files with 65536 in white level. This high numbers are present in dual iso not for regular MLV files. ACR can work your files cause it works from unique camera model tag and applies default white level for 5D mark III. I,m almost sure resolve isn,t doing the same so could be the issue here. Still, it seems it has something to do with ml builds so narrowing that down will more likely solve the issue. Meanwhile upload the MLV.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 01:01:17 PM
Ok i understand !

Here the short MLV from this post. (http://www.magiclantern.fm/forum/index.php?topic=17021.msg171279#msg171279)

https://www.dropbox.com/s/twrai93wv5zdler/M24-1019_23aug_light.MLV?dl=0
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Andy600 on August 28, 2016, 01:12:34 PM
@Danne - ACR gets white level from metadata. ACR also has a better HL recovery algorithm than Resolve. The white level tag is probably set too high for Resolve (for this particular shot) but ACR handles it fine.

@GutterPump - The shot is slightly over exposed. Try to shoot with zebras.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 01:32:28 PM
The files processes fine 15000 white level through mlv_dump and highlights are handled well with resolve. No problems with the builds but raw2cdng? I also downloaded your dng files which you converted with mlv_dump and they are 15000 as well. Only the dng files from here are giving the ultrahigh 65535 white levels.
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171284#msg171284

@Andy600
Is there any upper limit to white level in acr or resolve? When I change from 15000 to 16383 the change is clearly visible in resolve but not in acr. Now how can 65535 even work in either NLE without some sort of default limit to metadata?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Andy600 on August 28, 2016, 01:58:55 PM
I think the white level for ML DNGs is somewhat arbitrary. The theoretical maximum is relative to the bit depth of the image and the ADC so 16384 is the maximum RGB channel saturation (white level) for a 14bit file i.e. the upper level that 'could' (but probably wouldn't) have an effect. The absolute maximum for DNG is 16bit or 0-65535 because DNG is essentially a TIFF. The difference is less noticeable in ACR because of it's different application of HL recovery -  Resolve and ACR either do this processing in different colorspaces or there is a procedural difference in 'when' the HL recovery algorithm is used.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 02:21:56 PM
Can raw2cdng output 14-bit? Maybe try that?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Andy600 on August 28, 2016, 02:28:06 PM
No, it's 12bit, 16bit or 16bit maximized. I only ever use 16bit. If you use 2x2 chroma smoothing it has to be 16bit maximized AFAIK. Of course, 16bit is padded, it's not full 16bits.

I don't believe the converter is at fault with regards to GutterPump's issue. It's exposure.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 02:43:18 PM
In those tests. I intentionally overexposed scenes. I always use zebra for check HL. But even if i get overexposed footage with previous build i never met this issue before.

I dont think it's generated by hazard or bad exposure.

I will now made news tests and made a new comparison, always with intentional  overexposed footage.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: vstrglv on August 28, 2016, 03:01:42 PM
I have tested raw2cdng1.6.5:
magiclantern-crop3x.2016Apr02.5D3113.zip - Crop 1:1 and No crop modes - There are NO Hot pixels.
magiclantern-crop3x.2016Aug23.5D3113.zip - Crop 1:1 and No crop modes - There are  Hot pixels.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 03:06:28 PM
I have tested raw2cdng1.6.5:
magiclantern-crop3x.2016Apr02.5D3113.zip - Crop 1:1 and No crop modes - There are NO Hot pixels.
magiclantern-crop3x.2016Aug23.5D3113.zip - Crop 1:1 and No crop modes - There are  Hot pixels.

You have to use MLVViewer or Mlv_dump (with the commande line "--fixcp")/MLVFS for remove them.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: vstrglv on August 28, 2016, 03:17:43 PM
Yes, i have used Mlv_dump. But it means that there is something wrong in Aug build, compare to Apr one.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 03:25:04 PM
Hey guys, I think I have discover what's wrong with my tests !

Before, for build 02apr and 23 aug, i always use the ISO tweak :

ML Digital ISO : -0.3 EV

When it's enabled for previous builds like 02apr, it's work well on Resolve, dont see any bad highlight.

But when I enabled it on the new 23aug it's make wrong pink highlight recovery in Resolve.
Then I disabled it, and i didnt see any pink highlight issue.

I will share shorts MLV to my differents tests

EDIT:

In waiting to upload MLV files, i have seen some bad pixels in my footage, even if fixed by MLV_dump.

e.g: It's visible in top of my previous footage, but in my next footage it will be more revelant.

(http://image.noelshack.com/fichiers/2016/34/1472392494-cp.jpg)

Is there a solution for a more agressive cold pixels fix with mlv_dump ?

I just tired with MLV_viewer 1.4.3, the CP fix is best, i dont see any bad pixels.

Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 04:35:05 PM
Great bugfind gutterpump.
Havn,t seen your pixel fixing footage but did you try the fixcp2 setting in the specially created mlv_dump ?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 04:41:11 PM
You will see in this zip :

https://www.dropbox.com/s/9efqoxex5o5xm1e/23augBuild_HL_issue.zip?dl=0

These hot pixels, especially in the border of footage made with 23aug build, i don't tried the fix cp2 setting, what command should i try ?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 04:54:36 PM
You need to compile a version for windows from this branch.
https://bitbucket.org/hudson/magic-lantern/branch/bad_pixels
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 05:07:08 PM
I dont know how to compile i dont have enough dev knowledges.. If anyone would like to try.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 05:33:01 PM
I couldn,t see any pixels from these files? Did you select fixcp in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171393#msg171393
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 05:49:02 PM
I couldn,t see any pixels from these files? Did you select fixcp in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171393#msg171393

Yes i select --fixcp in mlv_dump, you can see bad pixels in Resolve in the right/top border.
Or else if i dont use this command, entire footage is contamined by cold pixels.

e.g. : (http://image.noelshack.com/fichiers/2016/34/1472399116-cp2.png)

from "M28-1516_23augbuild_isoteak_On_light" or "M28-1516_23augbuild_isoteak_Off_light" DNG folders
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 28, 2016, 06:07:38 PM
Dead/stuck pixel? I have two, one blue and a red one which shines on higher isos in shadows. Mlv_dump wouldn,t bite on them. However I use dcraw to map those single ones out by specifying coordinates. Maybe mlv_dump could work with such pixelmaps if implemented.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 28, 2016, 06:31:08 PM
I m not sure about dead pixels. Because when I convert the same footage with mlv viewer all image is clean. IMO the current Mlv_dump version can't analyse and fix all bad pixels.

I did the test with a white wall i get the same between mlv_dump and mlv viewer and with the boths builds.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 28, 2016, 08:54:53 PM
ML Digital ISO : -0.3 EV

Reproduced; the issue is present without crop_rec as well. Didn't check the main builds, but it may be there as well.

Who's going to run a "hg bisect" for me? The issue is obvious on the histogram, no need to take any test pictures.

edit: narrowed down, it's from the vertical stripe fix. There are raw streams effected by digital ISO, and streams that are not. So far, I've assumed the LiveView raw stream is not affected by digital ISO, and the photo raw stream is always affected by it. Looks like I was wrong.

On 5D3, these raw types are affected by digital ISO: 2, 4, 14, 18.
These are not: 7, 8, 16, 23, 28, 34, 39, 50, 55, 58.
These have bad pixels: 4, 7, 8, 14, 23, 28, 39, 55.
These do not: 2, 16, 18, 34, 50, 58.

Please double-check my list (I may have missed some numbers) and check the vertical stripe status of those raw types. Ideally, we should pick a raw type that has neither bad pixels, nor hot pixels, and nor affected by digital ISO. If that's not possible, we'll pick the closest approximation (my current best guess is 8 ).

Test build (you can select the LV raw type from the Debug menu): zip (http://a1ex.magiclantern.fm/bleeding-edge/raw/raw-type-test/magiclantern-raw-type-test.2016Aug28.5D3113.zip)

Yes i select --fixcp in mlv_dump, you can see bad pixels in Resolve in the right/top border.

Solved that as well. Windows binary here (https://builds.magiclantern.fm/jenkins/view/Other%20tasks/job/mlv_dump/); for Mac I'll leave it up to Danne.

Will split the topic later.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 29, 2016, 12:06:14 AM
Well thanks you Alex for your efficiency

Are we nearing the goal ? i will made ​​news tests tommorow if I have time. I hope we will see this crop build + vertical stripes fix in official version soon.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Kharak on August 29, 2016, 05:01:25 AM
So ML digital ISO affects Raw? As you I didnt think it affected the raw ouput. Can the ML digital ISO increase the DR as in the infamous ISO 66 ?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 29, 2016, 07:31:48 AM
Updated mac mlv_dump binary here with the latest cold pixel code. Great stuff.
mlv_dump (https://drive.google.com/file/d/0B4tCJMlOYfirbXNEbHVYcFNGX3c/view?usp=sharing)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 29, 2016, 08:04:42 AM
Can the ML digital ISO increase the DR as in the infamous ISO 66 ?

Just did a quick test and no, it doesn't increase DR at all, it simply scales down the data.

But, on these builds you can load the modules from the ISO research thread (http://www.magiclantern.fm/forum/index.php?topic=10111) (binaries from first post) and try out the ISO tweaks.

Also, since data range on 5D3 appears to be up to 16383, you can get a small boost of DR (0.15 stops in highlights) from existing footage by setting white level (with exiftool) to 16382. I'll commit this one now.

Other cameras may need white level tweaks as well; you may check with raw_diag (binary available in the ISO thread).

Are we nearing the goal ?

Of course, every small progress like this brings us closer to the goal.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: DeafEyeJedi on August 29, 2016, 08:25:21 AM
Thanks @GutterPump for all of your reports. Nice work! Sorry guys I've been out on vacation in Chicago (USA) and have been using the latest build from this thread and messed with a bunch of settings on the plane flight (3 hours and 15 mins) with the crop_rec.mo as much as I could think of in short clips. Results should rather be interesting and will post my findings as soon as I can.

Updated mac mlv_dump binary here with the latest cold pixel code. Great stuff.
mlv_dump (https://drive.google.com/file/d/0B4tCJMlOYfirbXNEbHVYcFNGX3c/view?usp=sharing)

Thanks @Danne for this stuff!  8)


Also, since data range on 5D3 appears to be up to 16383, you can get a small boost of DR (0.15 stops in highlights) from existing footage by setting white level (with exiftool) to 16382. I'll commit this one now.

Absolutely @a1ex this is definitely worth the extra .15 stops of DR especially in HL!
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 29, 2016, 03:00:38 PM
Thanks @GutterPump for all of your reports. Nice work!

You're welcome ! I was so frustrated with this issue, need to find the problem fast !



edit: narrowed down, it's from the vertical stripe fix. There are raw streams effected by digital ISO, and streams that are not. So far, I've assumed the LiveView raw stream is not affected by digital ISO, and the photo raw stream is always affected by it. Looks like I was wrong.

On 5D3, these raw types are affected by digital ISO: 2, 4, 14, 18.
These are not: 7, 8, 16, 23, 28, 34, 39, 50, 55, 58.
These have bad pixels: 4, 7, 8, 14, 23, 28, 39, 55.
These do not: 2, 16, 18, 34, 50, 58.

Please double-check my list (I may have missed some numbers) and check the vertical stripe status of those raw types. Ideally, we should pick a raw type that has neither bad pixels, nor hot pixels, and nor affected by digital ISO. If that's not possible, we'll pick the closest approximation (my current best guess is 8 ).

Test build (you can select the LV raw type from the Debug menu): zip (http://a1ex.magiclantern.fm/bleeding-edge/raw/raw-type-test/magiclantern-raw-type-test.2016Aug28.5D3113.zip)



I tried this "special" version of ML, with 8 raw type, i don't discover any problem with and without ML digital ISO tweak at -0.3 EV

(Will do others and others tests but seem work well and better)

And the new version of MLV_dump erase correctly all the bad pixels ! And of course the verical stripes fix seem to work well too. (don't tried in crop mod)

EDIT :

I did a "stress test" in Resolve for view vertical stripes, the result is : there are always "but less" vertical stripes.
With and without ML Digital ISO
.MLV converted with the new mlv_dump
Was done in normal mode (not crop mod)
Always with the 8 raw type

you can watch here :

https://www.dropbox.com/s/f9tdfze9sdiz78v/MLTEST-28aug-vertical-stripes.mov?dl=0

Video is in DNXHD 444 codecs (dont view directly in browser, download is better)

(http://image.noelshack.com/fichiers/2016/35/1472478027-mltest-28aug-vertical-stripes-mov-snapshot-00-00-2016-08-29-15-39-34.png)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 29, 2016, 03:58:29 PM
Got it, so Canon's correction is probably not perfect either, but the defect is probably visible only on extreme grades.

I assume mlv_dump --fix-stripes (or the old version, which does this by default) would fix the remaining stripes, correct?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 29, 2016, 04:20:49 PM
Got it, so Canon's correction is probably not perfect either, but the defect is probably visible only on extreme grades.

I assume mlv_dump --fix-stripes (or the old version, which does this by default) would fix the remaining stripes, correct?

Yes : (http://image.noelshack.com/fichiers/2016/35/1472480325-timeline-1-mov-snapshot-00-00-2016-08-29-16-17-19.png)

https://www.dropbox.com/s/czyypkn52z0k0vq/Timeline%201.mov_snapshot_00.00_%5B2016.08.29_16.17.19%5D.bmp?dl=0

I think it should better to set mlv_dump with --fix-stripes as defaut.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 29, 2016, 05:02:48 PM
I have some high iso material which pronounce dead pixels. Is there anything that keeps those from being detected from the cold pixel algorithm? Is there some numbers to be tweaked in code which could detect them? I tried by changing the number here but no luck.
Code: [Select]
int cold_thr = MAX(0, raw_info.black_level - 200);If I select chroma smoothing it can detect and erase these two pixels but it will introduce some other strange dots to the picture. Conclusion is that cold pixel fix is giving a much cleaner picture than chroma smooth.
What is left to do is maybe to identify the coordinates manually and the somehow let mlv_dump know these coordinates and have them added externally?

Original
https://drive.google.com/file/d/0B4tCJMlOYfird3VXa2VTdU1XZ3c/view?usp=sharing

Example crops.
(https://s12.postimg.cc/bzk7y4o3h/Screen_Shot_2016_08_29_at_16_54_10.png)

(https://s12.postimg.cc/v5xf1b4l9/Screen_Shot_2016_08_29_at_16_54_35.png)

Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 29, 2016, 06:21:46 PM
These are hot pixels. They can be corrected in the same way as cold pixels, but their detection heuristic is a bit different (basically, much brighter than neighbors). There is some code in cr2hdr that finds them.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 29, 2016, 08:53:54 PM
I understood a few more things about how these vertical stripes are corrected in Canon code.

The ADTG gain registers (888x) are used for column groups; 5D3 has 8 of them, so the pattern repeats every 8 column. Each video mode has a set of calibration values, such as 0x455, 0x459, 0x456, 0x456 (an example). These values are stored in ROM, so I guess they were calibrated at the factory, and assumed to stay the same. The assumption probably doesn't work on some cameras. My 5D3, for example, has no such patterns, even with really extreme grading curves.

The values are copied to RAM and can be overridden there. So, in theory, I could copy the algorithm from raw2dng/mlv_dump and run it in the camera to fine-tune these values.

Looking closer at the DNGs from GutterPump, the vertical stripes are also present in the version with bad pixels; maybe just a little less obvious. So, choosing a different raw type might not be the best solution.

I've fine-tuned the stripe correction algorithm a bit in mlv_dump. Didn't test it much, just 3 crops: original, old corrected, new corrected:

(http://a1ex.magiclantern.fm/bleeding-edge/raw/stripe-fix/M28-1505_02aprbuild8isoteak_On_light_frame_000005-nofix.jpg)(http://a1ex.magiclantern.fm/bleeding-edge/raw/stripe-fix/M28-1505_02aprbuild8isoteak_On_light_frame_000005-stripefix-old.jpg)(http://a1ex.magiclantern.fm/bleeding-edge/raw/stripe-fix/M28-1505_02aprbuild8isoteak_On_light_frame_000005-stripefix-new.jpg)



Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on August 29, 2016, 09:22:13 PM
That looks really clean. The difference is clearly visible.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: DeafEyeJedi on August 29, 2016, 09:42:58 PM
O0o0h Totally.  Nice work @a1ex!
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Kharak on August 29, 2016, 10:24:45 PM
Yes that is a clean image, Alexa style :)

edit:

Question: Does the 1:1 Crop mode compared to "vanilla" crop have less rolling shutter? It looks like it to me, but things sometimes look wrong to me too.. So I am not sure.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 30, 2016, 12:38:20 AM
Wow nice find .. Can't wait to try it  !

Finally, the last thing that was a little defect in the RAW video with the 5D Mark III is now completely fixed.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 30, 2016, 09:24:10 PM
@GutterPump: can you take a flat field image? That is, an image without a lens, but with a piece of paper instead, like this (http://wilson.bronger.org/lens_calibration_tutorial/#id3). It should be exposed to the right, but not clipped.

Additionally, can you take a bracketed image at the following settings?
- Subject: a blank out-of-focus wall, or anything else that looks flat without details (not critical, but the scene must be perfectly static)
- Movie mode, ISO 100...400, 1/100, image exposed to the right
- Advanced Bracket: 2 frames, 1 EV spacing, 0++, 1/100 ... 1/50 (that means, Tungsten lights in PAL land are fine)
- Silent Picture: simple (any file format is fine)
- Intervalometer: enabled (to avoid any camera motion), stop after 10 frames (it won't work from the first try).

I just noticed a bug with bracketing + simple silent pictures: in LiveView I need to wait some more after the exposure is changed. That's why I've set the intervalometer to stop after 10 frames - to work around this bug. Please upload only a successful pair of images.

The ratio of these two images (ideally 2) should reveal the stripe pattern.

Related: is there anyone who can consistently reproduce the vertical stripe problem in H.264 with crop mode enabled? I might have a fix for that as well, but I have no way to test - my camera shows absolutely no sign of vertical stripes...
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 30, 2016, 09:56:32 PM
@GutterPump: can you take a flat field image? That is, an image without a lens, but with a piece of paper instead, like this (http://wilson.bronger.org/lens_calibration_tutorial/#id3). It should be exposed to the right, but not clipped.



I need to know if i must do this RAW video or standard RAW photo, anyway I'll do both


Additionally, can you take a bracketed image at the following settings?
- Subject: a blank out-of-focus wall, or anything else that looks flat without details (not critical, but the scene must be perfectly static)
- Movie mode, ISO 100...400, 1/100, image exposed to the right
- Advanced Bracket: 2 frames, 1 EV spacing, 0++, 1/100 ... 1/50 (that means, Tungsten lights in PAL land are fine)
- Silent Picture: simple (any file format is fine)
- Intervalometer: enabled (to avoid any camera motion), stop after 10 frames (it won't work from the first try).

I just noticed a bug with bracketing + simple silent pictures: in LiveView I need to wait some more after the exposure is changed. That's why I've set the intervalometer to stop after 10 frames - to work around this bug. Please upload only a successful pair of images.

The ratio of these two images (ideally 2) should reveal the stripe pattern.

Of course. I will do this tommorow at my work (more light in the piece if i want pass this experience), in waiting, i'm checkind steps by steps what i should do exactly
Is there no matter about the build ? i will use the 23aug  crop mod build

but I have no way to test - my camera shows absolutely no sign of vertical stripes...

You're very lucky .. That want mean, calibration made by Canon isn't so well for all camera ? There is problem problem about the reliability of their plant testing.. Or maybe it's a question of release's date.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 31, 2016, 11:57:15 AM
That want mean, calibration made by Canon isn't so well for all camera?

Or maybe the cameras heavily used for video will lose the calibration over time (not sure, just a hypothesis).

I need to know if i must do this RAW video or standard RAW photo, anyway I'll do both

Video, same settings as your previous sample with the clouds. Any build should be fine.



Some more details on the calibration data, from 5D3 (other cameras may use different values):

- Photo mode and 5x/10x zoom all use the following properties:
ISO 100: 0x1010022, ISO 125: 0x1010023, ISO 200: 0x1010025 ... ISO 12800+: 0x1010037

- 1080p movie mode and regular LiveView use these:
ISO 100: 0x10500d1, ISO 200: 0x10500d2 ... ISO 25600: 0x10500d9

- 720p movie mode use these:
ISO 100: 0x10500dd, ISO 200: 0x10500de ... ISO 25600: 0x10500e5

The contents of these appear to be identical between my 1.1.3 and 1.2.3 ROM dumps (which is why I've assumed the camera is calibrated once at factory and that's it).
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 31, 2016, 12:16:23 PM
First test done with the flat image :

https://www.dropbox.com/s/7d6kgl0cjmqexqz/flat_23aug_160iso.MLV?dl=0

tell me if I didn't made mistakes

I did a bracketing too with the same technic and silent pictures (bad pixels no fix)

https://www.dropbox.com/s/jvpz0qw9ap46pba/flat_DNG_bracketing.zip?dl=0

                                 

Second test

https://www.dropbox.com/s/cm7kdvv4gtyo96x/bracketed_sample_adv_int.MLV?dl=0

Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on August 31, 2016, 08:37:21 PM
Downloaded the first one, looks fine. This fixes the vertical stripes in M28-1509_23augBuild_isoteak_Off_light.MLV *completely*. The --fix-stripes option still leaves some traces of vertical stripes, if you use a really extreme grading.

I've added an option to use a flat field frame in mlv_dump (similar to dark frames; background info here (http://spiff.rit.edu/classes/phys445/lectures/flats/flats.html), here (http://frigg.physastro.mnsu.edu/spect_flat.html) or here (https://en.wikipedia.org/wiki/Flat-field_correction)). This fixes vertical stripes completely, and can be also used to fix lens vignetting and other similar defects. Even extreme vignetting (for example, if you use an APS-C lens on a full-frame camera) can be corrected to some extent with this technique.

For details regarding how to capture a flat-field (light) frame, look here (http://wilson.bronger.org/lens_calibration_tutorial/#id3) (the section about vignetting correction). You may experiment with flat-field frames taken with or without a lens.

Sample workflow:
Code: [Select]
mlv_dump M31-1234.MLV -a -o flat.mlv           # average all flat frames
mlv_dump M28-1509.MLV --dng -t flat.mlv        # use the averaged flat frame to correct the footage

Or, if you also use a dark frame (not tested):
Code: [Select]
mlv_dump M31-1010.MLV -a -o dark.mlv                   # average all dark frames
mlv_dump M31-1234.MLV -a -o flat.mlv                   # average all flat frames
mlv_dump M28-1509.MLV --dng -s dark.mlv -t flat.mlv    # use the averaged flat/dark frames to correct the footage

I'm curious whether this technique can remove the dust spots as well. @GutterPump, can you take a flat field image with the lens on, and a diffuser in front of it, at these settings?
Code: [Select]
     Name:        'EF24-70mm f/2.8L USM'
     Focal Len:   24 mm
     Focus Dist:  356 mm
     Aperture:    f/18.20
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on August 31, 2016, 11:14:15 PM
Sound very usefull, that mean we're able to use this unic "flat.mlv" for correct all our footage with vertical banding issue  and once created, you just execute in default into mlv_dump and be quiet for the next MLV  extractions?

Obviously those that I made could not be applied to everyone and everyone should create his own if need ?


 
I'm curious whether this technique can remove the dust spots as well. @GutterPump, can you take a flat field image with the lens on, and a diffuser in front of it, at these settings?
Code: [Select]
     Name:        'EF24-70mm f/2.8L USM'
     Focal Len:   24 mm
     Focus Dist:  356 mm
     Aperture:    f/18.20

Yes of course, i will try it tommorow !
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 01, 2016, 10:29:21 AM
If anybody wants to try out the flat field frame function I compiled a mac version of mlv_dump here. Will try when I find more time.
https://drive.google.com/file/d/0B4tCJMlOYfirYno1Zi1nN0Z1SEU/view?usp=sharing
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on September 02, 2016, 03:07:10 PM
well i just tried the test with the diffuser in front, my light is powerless and the image isnt clean. I will get LED light in some days and renew this experience later.

in case, here anyway the .MLV :

https://www.dropbox.com/s/2p98vc3uxv83iij/M02-1449.MLV?dl=0



Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Greg on September 07, 2016, 05:20:47 PM
Or, if you also use a dark frame (not tested):

Cold pixel after subtract the black frame has a value of black frame or black level, so fixcp not working properly (old bug).

Code: [Select]
R = 5000
G = 6000
B = 0
G2 = 6000

after subtracting black frame :
Code: [Select]
R = 5000
G = 6000
B = 1024
G2 = 6000

Fixcp will not identify cold pixel.

mlv_dump.c
Code: [Select]
int32_t value = bitextract(src_line, x, current_depth);
int32_t sub_value = bitextract(sub_line, x, current_depth);

value -= sub_value;
value += lv_rec_footer.raw_info.black_level; /* should we really add it here? or better subtract it from averaged frame? */
value = COERCE(value, 0, (1<<current_depth)-1);

bitinsert(src_line, x, current_depth, value);
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: extremelypoorfilmaker on September 07, 2016, 07:33:48 PM
Just posting a reply to keep up to date with the post. :)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 11, 2016, 01:15:37 AM
Thank you very much a1ex for flat processing. I was just searching for such a feature! It will be useful for high-resolution planetary astronomical imaging.  :)

First shots on Moon with MLV files and crop 3x (without dark and flat yet) make me really enthusiast... :)
http://www.magiclantern.fm/forum/index.php?topic=17825.0

Thank you GutterPump.  I'm working on a Mac application to produce avi files for astronomical tools like Registax, AviStack 2 and AutoStakkert 2. You're binary will be useful.

I would have a question about processing dark frame. Cause there is no temperature stabilization with Canon cameras, can we compensate the temperature difference between main video and dark?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on September 12, 2016, 11:05:48 AM
TBH, I didn't really look into the thermal variations of dark frame, so I'm afraid I cannot answer.

For raw video, on 5D3 at least (but not on 5D2), you can set the exposure time to 0, capturing bias frames every now and then (for example, one bias frame every 10 regular frames or whatever). It's not implemented in menus; if there is interest in exploring this feature, I can look into it.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 15, 2016, 01:15:41 PM
Thank you for your feedback a1ex.  :D

So I've just made some dark tests with my 5D Mk III. As far as I can see, thermal noise has no real impact with short exposures. Here is a video showing dark with growing temperature (22°c to 52°c according to ML display)...
-dark signal was stretched for visualization-
It's a good news for calibration. I don't think we really need to take care about bias. Vertical stripes seem to be constant so dark and flat should do the job. I will test on astronomical images as soon as possible.

I've just finished my own Mac GUI to convert easily MLV to lossless AVI (needed for astronomical softwares). -It can also export mov files-
Dark and flat calibration are already integrated...
(http://astronomy.eliotis.com/magic-lantern/mlvtomovie.jpg)
I'll post a download link quickly.
Wish it helps.  :)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 15, 2016, 01:26:28 PM
Great app. Is it open source? I,m interested to see how you created your gui and how your workflow is going from MLV to mov? Are you using FFmpeg? I had my own puzzling with darkframe subtraction workflow a while ago.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg164871#msg164871
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: cmccullum on September 15, 2016, 03:56:29 PM
Maybe this is stupid, and a little late as the conversation has moved on,, but could high ISO noise reduction be a factor here?
I know it's recommended to turn it off, but I don't know why. Even if it absolutely is not a factor, could someone take a moment to explain why we turn it off?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Lars Steenhoff on September 15, 2016, 04:00:51 PM
ooh that gui looks very nice!

good work, and if it can export to prores with a lut in the future? 
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on September 15, 2016, 07:01:45 PM
but could high ISO noise reduction be a factor here?

If you are talking about vertical stripes, then you may want to know the effect is more obvious at lower ISOs. At higher ISOs, the pixel noise becomes greater than the column amplification differences, so the noise will effectively hide this artifact.

Quote
I know it's recommended to turn it off, but I don't know why

I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

You may want to mention the source of this recommendation, and maybe ask the author what difference it makes.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 15, 2016, 07:16:32 PM
Thank you guys! I've just created a new topic for MLVToMovie here...
http://www.magiclantern.fm/forum/index.php?topic=17893

Danne, yes I'm using MLV_Dump, DCRaw and FFmpeg. I don't think to make it open source cause it's a signed application and I'm using personal proprietary code for multi-tasking but I can send you code source for study. Send me a private message with you mail if you want. :)

Lars Steenhoff, I'm not a video master but if your needs can be done with FFmpeg I can look at it. Feel free to switch on the MLVToMovie topic to talk about It. :)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Kharak on September 15, 2016, 07:20:36 PM
I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

as you write it, I remember long time ago the talk was to deactivate it because it would leave more free memory.. But I too have no confirmation of this, just something that got stuck in my head reading through all the hundreds of posts here back in 2013 when the whole ML Raw ship set sails.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: cmccullum on September 16, 2016, 03:12:51 AM
If you are talking about vertical stripes, then you may want to know the effect is more obvious at lower ISOs. At higher ISOs, the pixel noise becomes greater than the column amplification differences, so the noise will effectively hide this artifact.

I'm not aware of this setting causing any difference in video mode. Maybe it affects the amount of free memory on some models?

You may want to mention the source of this recommendation, and maybe ask the author what difference it makes.

Thanks that answers everything. I actually wasn't aware that the stripes were more of an issue at lower ISOs (I've only noticed them when I crank the ISO, but I haven't done any kind of real testing)

I don't remember where I read about disabling it, but I think it was in the crop 2.5k 50fps thread. Could very likely just be misinformation on my part though. I'll look into it
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on September 16, 2016, 07:29:38 AM
Here we are talking about vertical stripes in highlights. Updated the first post with some background info.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 20, 2016, 09:04:44 AM
Reading up upon flatfield frames and it seems a combination of darkframes and flatfield frames can be obtained here which in theory could be some optimal correction combo for hot pixels, banding, color cast, dust etc. One question. Are the flatfield frames depending on being matched in iso, fps, aspect ratio etc the same as for creating applying darkframe averaging? Idea is to test it out and also implementing it to a script so want to minimize user error first.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on September 20, 2016, 09:26:02 AM
I didn't do any tests regarding this, but I would expect most gain defects to be the same at all ISOs.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 20, 2016, 10:52:13 AM
Some initial very loosely testing. Think the result is usable though.
Filmed a short sequence with canon eos m shutter 50fps iso 800 f2.0 MLV.
Then filmed a short flat frame by simply filming a white surface at f22. Not sure if f22 was the best setting. Might be even better wide open? And of course even better doing it right(http://wilson.bronger.org/lens_calibration_tutorial/#id3) More tests to be done.
Vinjetting is affected as stated.
Focus pixels are affected in some way.
Purple fringing?
All in all very promising feature.

original
(http://s16.postimg.cc/5yd7gnpit/image.png)

flat frame
(http://s16.postimg.cc/69untf5yt/image.png)

original
(http://s16.postimg.cc/equ64canp/image.png)

flat frame
(http://s16.postimg.cc/yk69x1o1h/image.png)


Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 20, 2016, 01:28:55 PM
Did another test. Created a flatfield mlv by filming at f2.0 straight at a photo bouncer (eos M). Experimenting. Check what it did to the focus pixels! Not all of them are gone but damn close. Developed in dcraw. Applied log C gamma for a flat revealing look.

original
(http://s9.postimg.cc/c1wxwm2y7/Screen_Shot_2016_09_20_at_13_23_01.png)

flatfield
(http://s9.postimg.cc/u5zyh90mn/Screen_Shot_2016_09_20_at_13_22_52.png)

original
(http://s9.postimg.cc/dtkuuefzj/Screen_Shot_2016_09_20_at_13_23_27.png)

flatfield
(http://s9.postimg.cc/vv3zs7a0f/Screen_Shot_2016_09_20_at_13_23_47.png)



*Update
Ran a lot of different tests around the clip just posted. Some feedback here.

Without lens - Vignetting seems gone but focus pixels more pronounced than withouth a lens on
With lens attached - Aperture wide open seems to take care of vignetting and most of the focus pixels. With aperture set to 22 focus pixels will be more pronounced.
Changing aperture while creating the flatframe - Will not show any difference to the dng files created. Changing exposure doesn,t seem to change anything.

Ok so a lot talk about focus pixels but maybe this could be a solution to how to eliminate those in post or even in mlv_dump?
Here is a flat frame created from an eos m
https://drive.google.com/file/d/0B4tCJMlOYfirLS1wTTh2dDk5aHc/view?usp=sharing

Oops, sorry about the off topic eos m testing. Realize it,s about 5D mark III.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: hyalinejim on September 23, 2016, 01:05:24 PM
Thanks A1ex for the updated mlv_dump which totally sorts out the vertical banding I was seeing in 5d3 crop mode using Apr 03 build. However, I have some DNG sequences that I'd like to correct, but I don't have the MLVs anymore. Is there anything I can do to the DNGs now in order to fix banding? An example is here:

DNG file extracted with MLVFS

https://www.dropbox.com/s/xc2xgi9vo2365xl/M02-0916_000000.dng?dl=0

This is what it looks like with aggressive colour correction (and I do actually want to push it this far for the short doc I'm working on) - screenshot is cropped from the full 1920x1080 and at 100%

(http://s17.postimg.cc/rsb9goubj/crop_banding_2.jpg)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 29, 2016, 09:43:27 AM
I've just tested flat calibration this morning on the crescent moon. As far as I can see, the dark signal is not removed automatically from the flat field when applying -s and -t options together. It's not really a mistake but the good mathematical process should be:

Calibrated raw = (raw-dark)/(flat-dark)*(average value of flat-dark)

As a result, take care when using flat calibration. We have to remove dark from flat with mlv_dump before using flat...
Quote
mlv_dump -s dark.mlv -o flat-dark.mlv flat.mlv
Otherwise noise will be reintroduced by the flat.

-I hope my explanation is clear enough.-
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 29, 2016, 12:55:08 PM
A grab from "The Handbook of Astronomical Image Processing" by Richard Berry & James Burnell to illustrate my last message...
(http://astronomy.eliotis.com/magic-lantern/image-calibration.jpg)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on September 29, 2016, 01:02:58 PM
Agree; however, this will require some extra code paths (to handle both cases).

Do you mind showing the difference it makes? If the flats are exposed to the right, I think the difference should be minimal (except for hot pixels, which are probably best measured (http://www.photonics.com/Article.aspx?AID=44298), rather than just dark-subtracted).
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 29, 2016, 08:26:36 PM
Everything works fine to me with mlv_dump. I don't think this job has to be handle by mlv_dump. We just need to know we have to be aware for the flat dark subtraction. I'll update my GUI.

I've processed a movie from this morning (very poor seeing but it doesn't matter for tests). Here is the first frame with different calibration...
http://astronomy.eliotis.com/tests/m29-0657-6400Iso-calibration-test.zip
- (raw-dark)/(flat-dark): tmp000000_raw-dark-div-by-flat-dark.tiff
- (raw-dark)/flat: tmp000000_raw-dark-div-by-flat.tiff
- raw-dark: tmp000000_raw-dark
- raw/flat: tmp000000_raw-div-by-flat
- raw (No calibration): tmp000000_raw

And here are the movies to see more easily random and constant noise:
- (raw-dark)/(flat-dark): - (raw-dark)/flat: - raw-dark: - raw/flat: - raw (No calibration):
Do not take care of corners. My focal Reducer/Corrector reduces the field. As a result, flat can only improve noise in the borders.

In my opinion, I do not recommend using flat without flat dark. But as you say a1ex, It can be less significant with high lights.

Danne, if you want to test. Flat dark could be the ultimate trick to remove focus pixels perfectly (or not). :)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 29, 2016, 08:56:21 PM
Flat dark? How is that created? Shooting at a complete dark area? Isn,t that a darkframe?
Checked your movies. Darkframe works best? Vignetting issue is too pronounced in the raw/flat example therefore revealing pixel circle?
Not really sure what to think about flatframe. It will even out the pixel gain on the overall picture but if too pronounced other artefacts starts to appear?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on September 29, 2016, 10:04:44 PM
Danne, yes it's a simple dark. flat dark is a dark shooted with the same parameters (Speed and Iso) as your flat. It's the same concept as the dark witch must use the same parameters as the raw images. It's really easy to do. After recording your flat as usually, you put the lens cover and launch a second record.

Vignetting issue with my shoot is only due to the fact that my focal reducer on my telescope limits the field so there is no more light to calibrate in the borders. You do not encounter such an effect with standard lens.

If you look to this video (raw-dark)/(flat-dark):
The field of view is quite perfect in the image circle. There is only random noise (no more vignetting and no more vertical or horizontal stripped lines). It's not the case with the other videos.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on September 29, 2016, 10:21:26 PM
Quote
After recording your flat as usually, you put the lens cover and launch a second record.
Yes, for creating a darkframe?

Quote
The field of view is quite perfect in the image circle.
Yes, noticed now. Looks very clean.

Quote
If you look to this video (raw-dark)/(flat-dark):
Raw-dark, flat-dark. What,s the difference?
I understand it as flatframe created with light diffusor. Darkframe created with lens cap on. Flat-dark??

Two formats right not a third one :)?

If I recall I tried applying a darkframe to the focus pixel footage but didn,t work.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: MarS on October 03, 2016, 12:08:21 AM
Yes it's a second dark. Mathematically, each dark must respect the raw image it will be subtracted from:
- the dark "classic" must respect ISO and Exposure of the raw mlv you want to process.
- the flat dark must respect ISO and Exposure of the flat mlv.

I wish this diagram helps...
(http://astronomy.eliotis.com/magic-lantern/image-calibration-help.jpg)

When you shot your flat mlv with a light diffusor, you get the same repeatable noise that you get in the raw mlv. If you subtract the dark from your raw mlv you get no more repeatable noise. Now, if you apply a simple flat you inject the repeatable noise it also contains. As a result, the best way is to subtract a "flat dark" to remove repeatable noise from flat before the flat division.

I hope my explanation (and my English) is good enough. Lol  :D
Title: Vertical stripes revisited (5D Mark III)
Post by: DeafEyeJedi on October 03, 2016, 03:57:56 AM
The image calibration diagram definitely gets me going... [emoji7] Thanks for sharing @MarS!
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on November 16, 2016, 08:45:39 PM
I,m trying to get the concept of creating a flatframe together with a flat dark.

MarS put this up a few posts up.
Code: [Select]
mlv_dump -s dark.mlv -o flat-dark.mlv flat.mlv
Do I average the flat.mlv and the dark.mlv before doing the darkframe subtraction from the flatframe above? I mean the whole workflow would look something like this.
1 - mlv_dump my_flat.mlv -a -o flat.mlv
2 - mlv_dump  my_dark.mlv -a -o dark.mlv
last step
3 - mlv_dump -s dark.mlv -o flat-dark.mlv flat.mlv

I would now have the darksubtracted flat_dark.mlv? Or do I skip step 1 and 2 and go straight to 3 meaning I would work with unaveraged flatframe and darkframe footage and instead I average the flat.mlv as a last step in the process?


Title: Vertical stripes revisited (5D Mark III)
Post by: DeafEyeJedi on November 17, 2016, 06:28:16 AM
Basically is it better to go...

•MLV > FlatFrame > DarkFraming Avg as Danne suggested or...

•MLV > DF Avg > FF?
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on November 17, 2016, 06:33:47 AM
That is not what I,m asking.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: DeafEyeJedi on November 17, 2016, 06:48:27 AM
Sorry guess I misread your post and I'll sheesh up until @MarS shines upon!
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: a1ex on November 17, 2016, 09:45:23 AM
Are you asking what is the difference between those two?

Code: [Select]
(f1+f2+f3+...+fn)/n - (d1+d2+d3+...+dn)/n
(f1-d1 + f2-d2 + f3-d3 + ... + fn-dn) / n

The only difference would be in round-off errors (one does two integer divisions, other does one). I doubt you'll be able to see it in practice (maybe only if you are stacking a lot of frames).

In this case, you may also want to add some rounding to the integer division:
Code: [Select]
-value /= average_samples;
+value = (value + average_samples/2) / average_samples;

Or you may want to work with higher bit depths.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on November 17, 2016, 10:34:49 PM
I,m testing various ways to create the optimal flatframe. Experimenting with some clips with focus pixels on. Conclusion so far is that a single flatframe MLV seems to be enough to use together with a darkframe MLV to eliminate around 95% of the focus pixels. Close but no cegar. Yet.
Producing a flatframe+flatdark=flatframe + additional darkframe MLV doesn,t seem to add any improvement in this very test. Probably stronger use in astrophotography.
A bit tedious testing this but I soon have a half automated script workflow in cr2hdr.app which will make it very easy to create both flatframes and darkframes.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on November 19, 2016, 11:17:33 PM
Been testing darkframes and flatframes and now I got some scripted automated solutions for both darkframes and flatframes in cr2hdr.app
http://www.magiclantern.fm/forum/index.php?topic=15108.msg146822#msg146822

Darkframe subtraction really cleans up color noise significantly on higher isos(example 800 iso) and flatframes evens out vignetting among other things. I ended up with a flatframe filmed at f5.6 while cat clip was filmed at f1.4.
A lot more to learn and test with in regard of creating the better flatframe. Here is an interested read about it. Note that I didn,t follow the rule.
https://photographingspace.com/how-to-create-dslr-and-ccd-flat-frames-for-astrophotography/

Original dng files here
https://drive.google.com/file/d/0B4tCJMlOYfirS2RoRV9jUGVVOE0/view?usp=sharing

Original
(https://s13.postimg.cc/vfm3tnrjb/Screen_Shot_2016_11_19_at_23_00_46.png)

Darkframe subtraction added
(https://s13.postimg.cc/6l2lzl6p3/Screen_Shot_2016_11_19_at_23_00_54.png)

Darkframe+flatframe added
(https://s13.postimg.cc/da95fls13/Screen_Shot_2016_11_19_at_23_01_00.png)



Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on December 30, 2016, 01:35:26 AM
I back on this topic after did lot of tests and shot, i noticed something strange.

The automatic stripes correction does not always work to some MLV footage that have highlight.
Sometime yes sometime no, depend the footage and maybe the exposition.

I do not know if it's related but i use the MLV_DUMP version from this link :

http://www.magiclantern.fm/forum/index.php?topic=5601.msg177064#msg177064

MLV footage was at 12bits, standard mode (no crop), on 5D mark III 113.

command line was just
Code: [Select]
--dng

I have to try with standard 14bits MLV and MLV_DUMP from last september for check if i get the same issue than 12bits.

Here two exemple of short .MLV, one with the issue, one without.

https://www.dropbox.com/s/41axdf047m16eg9/12bits-stripecorrectionwork.MLV?dl=0
https://www.dropbox.com/s/09z50zr9r9h2ety/12bits-stripescorrectiondontwork.MLV?dl=0


EDIT :

I did some tests with 14bits MLV files and older MLV_DUMP from 05SEPT.

I get the same issue. Look like the stripes fix dont work when the footage is overexposed, but sometime i need to overexpose when i do "Zone system".
Is there a solution for apply this fix even if the footage is overexposed and not just ETTR ?

Here some exemples with others shorts MLV :

https://www.dropbox.com/s/d3c5uqwmxdwck6s/overexposed_stripesfixdontwork.MLV?dl=0
https://www.dropbox.com/s/bt921vhev7qofss/ETTR_stripesfixwork.MLV?dl=0


Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on January 11, 2017, 03:56:45 PM
How does it not work you mean? Very overexposed but seems fine?

(https://s27.postimg.cc/xzfup6flf/Screen_Shot_2017_01_11_at_15_54_13.png)

(https://s27.postimg.cc/qxhx2zbzn/Screen_Shot_2017_01_11_at_15_54_00.png)
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on January 11, 2017, 04:23:30 PM
Here an exemple with the ETTR_stripesfixwork.MLV (https://www.dropbox.com/s/bt921vhev7qofss/ETTR_stripesfixwork.MLV?dl=0) :

(http://image.noelshack.com/fichiers/2017/02/1484147722-no-stripes.png)
(http://image.noelshack.com/fichiers/2017/02/1484147784-mlv-dump-fixstripes.png)

Another with overexposed_stripesfixdontwork.MLV (https://www.dropbox.com/s/d3c5uqwmxdwck6s/overexposed_stripesfixdontwork.MLV?dl=0)

(http://image.noelshack.com/fichiers/2017/02/1484147722-stripes.png)
(http://image.noelshack.com/fichiers/2017/02/1484147784-mlv-dump-nofixstripes.png)


Do you understand what i mean about this issue ?

During the process for extract DNG from the  overexposed_stripesfixdontwork.MLV with MLV_DUMP , the stripes fix don't work.

Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on January 11, 2017, 04:30:53 PM
Understand now. Seem to work on mac

(https://s30.postimg.cc/w9edrl1w1/Screen_Shot_2017_01_11_at_16_29_06.png)

(https://s30.postimg.cc/3l1foczpt/Screen_Shot_2017_01_11_at_16_28_20.png)

Title: Re: Vertical stripes revisited (5D Mark III)
Post by: Danne on January 11, 2017, 04:37:43 PM
You,re right. Older build of mlv_dump code will work but not the latest on your selected examples. Just tested again.
Title: Re: Vertical stripes revisited (5D Mark III)
Post by: GutterPump on January 11, 2017, 04:41:15 PM
Thanks Danne, i'm feel less alone hehe.

On my exemple, i did the test with MLV_DUMP from 05 septembre 2016.