Vertical stripes revisited (5D Mark III)

[GutterPump]

Well thanks you Alex for your efficiency

Are we nearing the goal ? i will made ​​news tests tommorow if I have time. I hope we will see this crop build + vertical stripes fix in official version soon.

[Kharak]

So ML digital ISO affects Raw? As you I didnt think it affected the raw ouput. Can the ML digital ISO increase the DR as in the infamous ISO 66 ?

[Danne]

Updated mac mlv_dump binary here with the latest cold pixel code. Great stuff.
mlv_dump

[a1ex]

Can the ML digital ISO increase the DR as in the infamous ISO 66 ?

Just did a quick test and no, it doesn't increase DR at all, it simply scales down the data.

But, on these builds you can load the modules from the ISO research thread (binaries from first post) and try out the ISO tweaks.

Also, since data range on 5D3 appears to be up to 16383, you can get a small boost of DR (0.15 stops in highlights) from existing footage by setting white level (with exiftool) to 16382. I'll commit this one now.

Other cameras may need white level tweaks as well; you may check with raw_diag (binary available in the ISO thread).

Are we nearing the goal ?

Of course, every small progress like this brings us closer to the goal.

[DeafEyeJedi]

Thanks @GutterPump for all of your reports. Nice work! Sorry guys I've been out on vacation in Chicago (USA) and have been using the latest build from this thread and messed with a bunch of settings on the plane flight (3 hours and 15 mins) with the crop_rec.mo as much as I could think of in short clips. Results should rather be interesting and will post my findings as soon as I can.

Updated mac mlv_dump binary here with the latest cold pixel code. Great stuff.
mlv_dump

Thanks @Danne for this stuff! 

Also, since data range on 5D3 appears to be up to 16383, you can get a small boost of DR (0.15 stops in highlights) from existing footage by setting white level (with exiftool) to 16382. I'll commit this one now.

Absolutely @a1ex this is definitely worth the extra .15 stops of DR especially in HL!

[GutterPump]

Thanks @GutterPump for all of your reports. Nice work!

You're welcome ! I was so frustrated with this issue, need to find the problem fast !

edit: narrowed down, it's from the vertical stripe fix. There are raw streams effected by digital ISO, and streams that are not. So far, I've assumed the LiveView raw stream is not affected by digital ISO, and the photo raw stream is always affected by it. Looks like I was wrong.

On 5D3, these raw types are affected by digital ISO: 2, 4, 14, 18.
These are not: 7, 8, 16, 23, 28, 34, 39, 50, 55, 58.
These have bad pixels: 4, 7, 8, 14, 23, 28, 39, 55.
These do not: 2, 16, 18, 34, 50, 58.

Please double-check my list (I may have missed some numbers) and check the vertical stripe status of those raw types. Ideally, we should pick a raw type that has neither bad pixels, nor hot pixels, and nor affected by digital ISO. If that's not possible, we'll pick the closest approximation (my current best guess is 8 ).

Test build (you can select the LV raw type from the Debug menu): zip

I tried this "special" version of ML, with 8 raw type, i don't discover any problem with and without ML digital ISO tweak at -0.3 EV

(Will do others and others tests but seem work well and better)

And the new version of MLV_dump erase correctly all the bad pixels ! And of course the verical stripes fix seem to work well too. (don't tried in crop mod)

EDIT :

I did a "stress test" in Resolve for view vertical stripes, the result is : there are always "but less" vertical stripes.
With and without ML Digital ISO
.MLV converted with the new mlv_dump
Was done in normal mode (not crop mod)
Always with the 8 raw type

you can watch here :

https://www.dropbox.com/s/f9tdfze9sdiz78v/MLTEST-28aug-vertical-stripes.mov?dl=0

Video is in DNXHD 444 codecs (dont view directly in browser, download is better)

[a1ex]

Got it, so Canon's correction is probably not perfect either, but the defect is probably visible only on extreme grades.

I assume mlv_dump --fix-stripes (or the old version, which does this by default) would fix the remaining stripes, correct?

[GutterPump]

Got it, so Canon's correction is probably not perfect either, but the defect is probably visible only on extreme grades.

I assume mlv_dump --fix-stripes (or the old version, which does this by default) would fix the remaining stripes, correct?

Yes :

https://www.dropbox.com/s/czyypkn52z0k0vq/Timeline%201.mov_snapshot_00.00_%5B2016.08.29_16.17.19%5D.bmp?dl=0

I think it should better to set mlv_dump with --fix-stripes as defaut.

[Danne]

I have some high iso material which pronounce dead pixels. Is there anything that keeps those from being detected from the cold pixel algorithm? Is there some numbers to be tweaked in code which could detect them? I tried by changing the number here but no luck.

Code Select Expand

int cold_thr = MAX(0, raw_info.black_level - 200);
If I select chroma smoothing it can detect and erase these two pixels but it will introduce some other strange dots to the picture. Conclusion is that cold pixel fix is giving a much cleaner picture than chroma smooth.
What is left to do is maybe to identify the coordinates manually and the somehow let mlv_dump know these coordinates and have them added externally?

Original
https://drive.google.com/file/d/0B4tCJMlOYfird3VXa2VTdU1XZ3c/view?usp=sharing

Example crops.

[a1ex]

These are hot pixels. They can be corrected in the same way as cold pixels, but their detection heuristic is a bit different (basically, much brighter than neighbors). There is some code in cr2hdr that finds them.

[a1ex]

I understood a few more things about how these vertical stripes are corrected in Canon code.

The ADTG gain registers (888x) are used for column groups; 5D3 has 8 of them, so the pattern repeats every 8 column. Each video mode has a set of calibration values, such as 0x455, 0x459, 0x456, 0x456 (an example). These values are stored in ROM, so I guess they were calibrated at the factory, and assumed to stay the same. The assumption probably doesn't work on some cameras. My 5D3, for example, has no such patterns, even with really extreme grading curves.

The values are copied to RAM and can be overridden there. So, in theory, I could copy the algorithm from raw2dng/mlv_dump and run it in the camera to fine-tune these values.

Looking closer at the DNGs from GutterPump, the vertical stripes are also present in the version with bad pixels; maybe just a little less obvious. So, choosing a different raw type might not be the best solution.

I've fine-tuned the stripe correction algorithm a bit in mlv_dump. Didn't test it much, just 3 crops: original, old corrected, new corrected:

[Danne]

That looks really clean. The difference is clearly visible.

[DeafEyeJedi]

O0o0h Totally.  Nice work @a1ex!

[Kharak]

Yes that is a clean image, Alexa style

edit:

Question: Does the 1:1 Crop mode compared to "vanilla" crop have less rolling shutter? It looks like it to me, but things sometimes look wrong to me too.. So I am not sure.

[GutterPump]

Wow nice find .. Can't wait to try it  !

Finally, the last thing that was a little defect in the RAW video with the 5D Mark III is now completely fixed.

[a1ex]

@GutterPump: can you take a flat field image? That is, an image without a lens, but with a piece of paper instead, like this. It should be exposed to the right, but not clipped.

Additionally, can you take a bracketed image at the following settings?
- Subject: a blank out-of-focus wall, or anything else that looks flat without details (not critical, but the scene must be perfectly static)
- Movie mode, ISO 100...400, 1/100, image exposed to the right
- Advanced Bracket: 2 frames, 1 EV spacing, 0++, 1/100 ... 1/50 (that means, Tungsten lights in PAL land are fine)
- Silent Picture: simple (any file format is fine)
- Intervalometer: enabled (to avoid any camera motion), stop after 10 frames (it won't work from the first try).

I just noticed a bug with bracketing + simple silent pictures: in LiveView I need to wait some more after the exposure is changed. That's why I've set the intervalometer to stop after 10 frames - to work around this bug. Please upload only a successful pair of images.

The ratio of these two images (ideally 2) should reveal the stripe pattern.

Related: is there anyone who can consistently reproduce the vertical stripe problem in H.264 with crop mode enabled? I might have a fix for that as well, but I have no way to test - my camera shows absolutely no sign of vertical stripes...

[GutterPump]

@GutterPump: can you take a flat field image? That is, an image without a lens, but with a piece of paper instead, like this. It should be exposed to the right, but not clipped.

I need to know if i must do this RAW video or standard RAW photo, anyway I'll do both

Additionally, can you take a bracketed image at the following settings?
- Subject: a blank out-of-focus wall, or anything else that looks flat without details (not critical, but the scene must be perfectly static)
- Movie mode, ISO 100...400, 1/100, image exposed to the right
- Advanced Bracket: 2 frames, 1 EV spacing, 0++, 1/100 ... 1/50 (that means, Tungsten lights in PAL land are fine)
- Silent Picture: simple (any file format is fine)
- Intervalometer: enabled (to avoid any camera motion), stop after 10 frames (it won't work from the first try).

I just noticed a bug with bracketing + simple silent pictures: in LiveView I need to wait some more after the exposure is changed. That's why I've set the intervalometer to stop after 10 frames - to work around this bug. Please upload only a successful pair of images.

The ratio of these two images (ideally 2) should reveal the stripe pattern.

Of course. I will do this tommorow at my work (more light in the piece if i want pass this experience), in waiting, i'm checkind steps by steps what i should do exactly
Is there no matter about the build ? i will use the 23aug  crop mod build

but I have no way to test - my camera shows absolutely no sign of vertical stripes...

You're very lucky .. That want mean, calibration made by Canon isn't so well for all camera ? There is problem problem about the reliability of their plant testing.. Or maybe it's a question of release's date.

[a1ex]

That want mean, calibration made by Canon isn't so well for all camera?

Or maybe the cameras heavily used for video will lose the calibration over time (not sure, just a hypothesis).

I need to know if i must do this RAW video or standard RAW photo, anyway I'll do both

Video, same settings as your previous sample with the clouds. Any build should be fine.



Some more details on the calibration data, from 5D3 (other cameras may use different values):

- Photo mode and 5x/10x zoom all use the following properties:
ISO 100: 0x1010022, ISO 125: 0x1010023, ISO 200: 0x1010025 ... ISO 12800+: 0x1010037

- 1080p movie mode and regular LiveView use these:
ISO 100: 0x10500d1, ISO 200: 0x10500d2 ... ISO 25600: 0x10500d9

- 720p movie mode use these:
ISO 100: 0x10500dd, ISO 200: 0x10500de ... ISO 25600: 0x10500e5

The contents of these appear to be identical between my 1.1.3 and 1.2.3 ROM dumps (which is why I've assumed the camera is calibrated once at factory and that's it).

[GutterPump]

First test done with the flat image :

https://www.dropbox.com/s/7d6kgl0cjmqexqz/flat_23aug_160iso.MLV?dl=0

tell me if I didn't made mistakes

I did a bracketing too with the same technic and silent pictures (bad pixels no fix)

https://www.dropbox.com/s/jvpz0qw9ap46pba/flat_DNG_bracketing.zip?dl=0

                                 

Second test

https://www.dropbox.com/s/cm7kdvv4gtyo96x/bracketed_sample_adv_int.MLV?dl=0

[a1ex]

Downloaded the first one, looks fine. This fixes the vertical stripes in M28-1509_23augBuild_isoteak_Off_light.MLV *completely*. The --fix-stripes option still leaves some traces of vertical stripes, if you use a really extreme grading.

I've added an option to use a flat field frame in mlv_dump (similar to dark frames; background info here, here or here). This fixes vertical stripes completely, and can be also used to fix lens vignetting and other similar defects. Even extreme vignetting (for example, if you use an APS-C lens on a full-frame camera) can be corrected to some extent with this technique.

For details regarding how to capture a flat-field (light) frame, look here (the section about vignetting correction). You may experiment with flat-field frames taken with or without a lens.

Sample workflow:

Code Select Expand


mlv_dump M31-1234.MLV -a -o flat.mlv           # average all flat frames
mlv_dump M28-1509.MLV --dng -t flat.mlv        # use the averaged flat frame to correct the footage


Or, if you also use a dark frame (not tested):

Code Select Expand


mlv_dump M31-1010.MLV -a -o dark.mlv                   # average all dark frames
mlv_dump M31-1234.MLV -a -o flat.mlv                   # average all flat frames
mlv_dump M28-1509.MLV --dng -s dark.mlv -t flat.mlv    # use the averaged flat/dark frames to correct the footage


I'm curious whether this technique can remove the dust spots as well. @GutterPump, can you take a flat field image with the lens on, and a diffuser in front of it, at these settings?

Code Select Expand


     Name:        'EF24-70mm f/2.8L USM'
     Focal Len:   24 mm
     Focus Dist:  356 mm
     Aperture:    f/18.20


[GutterPump]

Sound very usefull, that mean we're able to use this unic "flat.mlv" for correct all our footage with vertical banding issue  and once created, you just execute in default into mlv_dump and be quiet for the next MLV  extractions?

Obviously those that I made could not be applied to everyone and everyone should create his own if need ?

I'm curious whether this technique can remove the dust spots as well. @GutterPump, can you take a flat field image with the lens on, and a diffuser in front of it, at these settings?

Code Select Expand


     Name:        'EF24-70mm f/2.8L USM'
     Focal Len:   24 mm
     Focus Dist:  356 mm
     Aperture:    f/18.20


Yes of course, i will try it tommorow !

[Danne]

If anybody wants to try out the flat field frame function I compiled a mac version of mlv_dump here. Will try when I find more time.
https://drive.google.com/file/d/0B4tCJMlOYfirYno1Zi1nN0Z1SEU/view?usp=sharing

[GutterPump]

well i just tried the test with the diffuser in front, my light is powerless and the image isnt clean. I will get LED light in some days and renew this experience later.

in case, here anyway the .MLV :

https://www.dropbox.com/s/2p98vc3uxv83iij/M02-1449.MLV?dl=0

[Greg]

Or, if you also use a dark frame (not tested):

Cold pixel after subtract the black frame has a value of black frame or black level, so fixcp not working properly (old bug).

Code Select Expand

R = 5000
G = 6000
B = 0
G2 = 6000

after subtracting black frame :

Code Select Expand

R = 5000
G = 6000
B = 1024
G2 = 6000

Fixcp will not identify cold pixel.

mlv_dump.c

Code Select Expand

int32_t value = bitextract(src_line, x, current_depth);
int32_t sub_value = bitextract(sub_line, x, current_depth);

value -= sub_value;
value += lv_rec_footer.raw_info.black_level; /* should we really add it here? or better subtract it from averaged frame? */
value = COERCE(value, 0, (1<<current_depth)-1);

bitinsert(src_line, x, current_depth, value);

[extremelypoorfilmaker]

Just posting a reply to keep up to date with the post.