Vertical stripes revisited (5D Mark III)

[GutterPump]

I'm back with news tests. Today i shot some footage ,as usually, in ETTR / 200 iso / 1/48 shutter speed / WB = sunlight.

I still get weird hightligt recovery in Resolve, with this pink color. I never get this before the 23aug crop build.

With the 02apr crop build, when i get pink color in highlight, i'm able to erase it with changing balance color/ tint options, but not with the 23aug build, it's like "frozen" on this pink color.

I can't really know what option/configuration is the cause of this issue

Here a DNG sample convert with mlv_dump (i removed bad pixels with it too) :


https://www.dropbox.com/s/vwssn3aup1mvojf/M27-1808_000000.dng?dl=0

EDIT : in waiting to find a solution ,  it is possible to use the qualifer option for fix it in Resolve.

[Danne]

I,m not sure what the problem is with this dng. I just tested it in resolve with highlight recovery on and also pulled back highlight abnormally much and no pinks. White level set to 15000 as mlv_dump usually does.






*update

Ok, so selected BMD and pink highlights are appearing. How can you rule out that it isn,t resolve doing a bad job here?


For the last dng file I doubt it has anything to do with the builds. Test working with it in acr and you,ll see both dng files works great.  For the dual iso dng file with white level around 65535 isn,t this caused by raw2cdng? Could you film the same sequence with two different builds?
*For some reason raw2cdng won,t let me overwrite white level in the dng example. 

Could you upoad the dual iso MLV file example? You could shorten the actual MLV file with mlv_dump like this.
mlv_dump -o new.MLV -f 5 [drag/your/MLV/file/here] hit enter

See to it that you create a different output name (where I put in the name new.MLV) than the name of the original MLV file or mlv_dump will overwrite your original file.

[a1ex]

Could you film the same sequence with two different builds?

+1. Once you get a quick and repeatable way to reproduce the issue, we can narrow it down.

[GutterPump]

Ok, so selected BMD and pink highlights are appearing. How can you rule out that it isn,t resolve doing a bad job here?

The pink color in highlight is worse when I use cinelog-C or if i put REC709 in color space (yes usually we don't need to use rec709 for raw, but it's just more revelant for this issue.)

For the last dng file I doubt it has anything to do with the builds. Test working with it in acr and you,ll see both dng files works great.  For the dual iso dng file with white level around 65535 isn,t this caused by raw2cdng? Could you film the same sequence with two different builds?
*For some reason raw2cdng won,t let me overwrite white level in the dng example. 

I will do a second test with 2 differents builds yes, but what about your tests ? You don't met the same issue than me on Resolve with footage in highlight ?

white level around 65535 isn,t this caused by raw2cdng?

I did the test with MLV_dump it's the same problem.

Could you upoad the dual iso MLV file example? You could shorten the actual MLV file with mlv_dump like this.
mlv_dump -o new.MLV -f 5 [drag/your/MLV/file/here] hit enter

See to it that you create a different output name (where I put in the name new.MLV) than the name of the original MLV file or mlv_dump will overwrite your original file.

Here the short MLV :

https://www.dropbox.com/s/jr6viin3g2d6vk0/new.MLV?dl=0

 but i don't understand what you mean about "dual iso MLV" ? I don't shot any video in dual iso with the new build. (I never use dual iso)

ps: Sorry if my English is approximative, I try to do my best to answer you.

[Danne]

I meant for you to upload the MLV that is creating dng files with 65536 in white level. This high numbers are present in dual iso not for regular MLV files. ACR can work your files cause it works from unique camera model tag and applies default white level for 5D mark III. I,m almost sure resolve isn,t doing the same so could be the issue here. Still, it seems it has something to do with ml builds so narrowing that down will more likely solve the issue. Meanwhile upload the MLV.

[GutterPump]

Ok i understand !

Here the short MLV from this post.

https://www.dropbox.com/s/twrai93wv5zdler/M24-1019_23aug_light.MLV?dl=0

[Andy600]

@Danne - ACR gets white level from metadata. ACR also has a better HL recovery algorithm than Resolve. The white level tag is probably set too high for Resolve (for this particular shot) but ACR handles it fine.

@GutterPump - The shot is slightly over exposed. Try to shoot with zebras.

[Danne]

The files processes fine 15000 white level through mlv_dump and highlights are handled well with resolve. No problems with the builds but raw2cdng? I also downloaded your dng files which you converted with mlv_dump and they are 15000 as well. Only the dng files from here are giving the ultrahigh 65535 white levels.
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171284#msg171284

@Andy600
Is there any upper limit to white level in acr or resolve? When I change from 15000 to 16383 the change is clearly visible in resolve but not in acr. Now how can 65535 even work in either NLE without some sort of default limit to metadata?

[Andy600]

I think the white level for ML DNGs is somewhat arbitrary. The theoretical maximum is relative to the bit depth of the image and the ADC so 16384 is the maximum RGB channel saturation (white level) for a 14bit file i.e. the upper level that 'could' (but probably wouldn't) have an effect. The absolute maximum for DNG is 16bit or 0-65535 because DNG is essentially a TIFF. The difference is less noticeable in ACR because of it's different application of HL recovery -  Resolve and ACR either do this processing in different colorspaces or there is a procedural difference in 'when' the HL recovery algorithm is used.

[Danne]

Can raw2cdng output 14-bit? Maybe try that?

[Andy600]

No, it's 12bit, 16bit or 16bit maximized. I only ever use 16bit. If you use 2x2 chroma smoothing it has to be 16bit maximized AFAIK. Of course, 16bit is padded, it's not full 16bits.

I don't believe the converter is at fault with regards to GutterPump's issue. It's exposure.

[GutterPump]

In those tests. I intentionally overexposed scenes. I always use zebra for check HL. But even if i get overexposed footage with previous build i never met this issue before.

I dont think it's generated by hazard or bad exposure.

I will now made news tests and made a new comparison, always with intentional  overexposed footage.

[vstrglv]

I have tested raw2cdng1.6.5:
magiclantern-crop3x.2016Apr02.5D3113.zip - Crop 1:1 and No crop modes - There are NO Hot pixels.
magiclantern-crop3x.2016Aug23.5D3113.zip - Crop 1:1 and No crop modes - There are  Hot pixels.

[GutterPump]

I have tested raw2cdng1.6.5:
magiclantern-crop3x.2016Apr02.5D3113.zip - Crop 1:1 and No crop modes - There are NO Hot pixels.
magiclantern-crop3x.2016Aug23.5D3113.zip - Crop 1:1 and No crop modes - There are  Hot pixels.

You have to use MLVViewer or Mlv_dump (with the commande line "--fixcp")/MLVFS for remove them.

[vstrglv]

Yes, i have used Mlv_dump. But it means that there is something wrong in Aug build, compare to Apr one.

[GutterPump]

Hey guys, I think I have discover what's wrong with my tests !

Before, for build 02apr and 23 aug, i always use the ISO tweak :

ML Digital ISO : -0.3 EV

When it's enabled for previous builds like 02apr, it's work well on Resolve, dont see any bad highlight.

But when I enabled it on the new 23aug it's make wrong pink highlight recovery in Resolve.
Then I disabled it, and i didnt see any pink highlight issue.

I will share shorts MLV to my differents tests

EDIT:

In waiting to upload MLV files, i have seen some bad pixels in my footage, even if fixed by MLV_dump.

e.g: It's visible in top of my previous footage, but in my next footage it will be more revelant.



Is there a solution for a more agressive cold pixels fix with mlv_dump ?

I just tired with MLV_viewer 1.4.3, the CP fix is best, i dont see any bad pixels.

[Danne]

Great bugfind gutterpump.
Havn,t seen your pixel fixing footage but did you try the fixcp2 setting in the specially created mlv_dump ?

[GutterPump]

You will see in this zip :

https://www.dropbox.com/s/9efqoxex5o5xm1e/23augBuild_HL_issue.zip?dl=0

These hot pixels, especially in the border of footage made with 23aug build, i don't tried the fix cp2 setting, what command should i try ?

[Danne]

You need to compile a version for windows from this branch.
https://bitbucket.org/hudson/magic-lantern/branch/bad_pixels

[GutterPump]

I dont know how to compile i dont have enough dev knowledges.. If anyone would like to try.

[Danne]

I couldn,t see any pixels from these files? Did you select fixcp in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171393#msg171393

[GutterPump]

I couldn,t see any pixels from these files? Did you select fixcp in mlv_dump?
http://www.magiclantern.fm/forum/index.php?topic=17021.msg171393#msg171393

Yes i select --fixcp in mlv_dump, you can see bad pixels in Resolve in the right/top border.
Or else if i dont use this command, entire footage is contamined by cold pixels.

e.g. :

from "M28-1516_23augbuild_isoteak_On_light" or "M28-1516_23augbuild_isoteak_Off_light" DNG folders

[Danne]

Dead/stuck pixel? I have two, one blue and a red one which shines on higher isos in shadows. Mlv_dump wouldn,t bite on them. However I use dcraw to map those single ones out by specifying coordinates. Maybe mlv_dump could work with such pixelmaps if implemented.

[GutterPump]

I m not sure about dead pixels. Because when I convert the same footage with mlv viewer all image is clean. IMO the current Mlv_dump version can't analyse and fix all bad pixels.

I did the test with a white wall i get the same between mlv_dump and mlv viewer and with the boths builds.

[a1ex]

ML Digital ISO : -0.3 EV

Reproduced; the issue is present without crop_rec as well. Didn't check the main builds, but it may be there as well.

Who's going to run a "hg bisect" for me? The issue is obvious on the histogram, no need to take any test pictures.

edit: narrowed down, it's from the vertical stripe fix. There are raw streams effected by digital ISO, and streams that are not. So far, I've assumed the LiveView raw stream is not affected by digital ISO, and the photo raw stream is always affected by it. Looks like I was wrong.

On 5D3, these raw types are affected by digital ISO: 2, 4, 14, 18.
These are not: 7, 8, 16, 23, 28, 34, 39, 50, 55, 58.
These have bad pixels: 4, 7, 8, 14, 23, 28, 39, 55.
These do not: 2, 16, 18, 34, 50, 58.

Please double-check my list (I may have missed some numbers) and check the vertical stripe status of those raw types. Ideally, we should pick a raw type that has neither bad pixels, nor hot pixels, and nor affected by digital ISO. If that's not possible, we'll pick the closest approximation (my current best guess is 8 ).

Test build (you can select the LV raw type from the Debug menu): zip

Yes i select --fixcp in mlv_dump, you can see bad pixels in Resolve in the right/top border.

Solved that as well. Windows binary here; for Mac I'll leave it up to Danne.

Will split the topic later.