5D Mk III - completely glitched out clips

[Aeidxst]

Hi everyone,

I encountered a devastating problem. A lot of my footage became completely glitched out.

It seems like this problem is similar to one of my older problems: https://www.magiclantern.fm/forum/index.php?topic=19108.0

First, here are my settings:

I'm recording 14-bit RAW video at 1920 x 490, 60 fps through fps override (I'm going to stretch it to 1920 x 817 in post to get proper 2.35:1 resolution). There are no other ML features being used, except the mlv_lite and mlv_play. I'm using the latest build (February 4th 2018 on 1.1.3).

I'm using a Canon 5D Mark III, with 50mm 1.2 and 24-105mm 4.0 lenses, using SanDisk Extreme PRO as storage (64GB, 160 MB/s). No external monitors connected.

This is in my ML folder > Settings > MLV_LITE.CFG

# Config file for module mlv_lite (MLV_LITE.MO)

raw.video.enabled = 1
raw.aspect.ratio = 6
raw.write.speed = 7938


Yesterday I recorded some (very important) footage. When on the set, I used the magnify button a lot to make focusing easier. When zoomed in I could see some glitches, but when zoomed out everything was fine, so I just started recording. Everything was also fine while recording.

I just converted all my footage to .dng sequences using MLVFS. A lot of footage became completely glitched out. See pic related. About 30% of all the clips I shot turned into pic related.


What caused this, and what can I do about this? Can I still save my 'corrupted' files/footage?

Here are some of the original MLV files that got ruined.

https://we.tl/1wIC7hqg05

https://we.tl/lNUKVcNDkU

[Aeidxst]

@a1ex @Danne @SpcCb @reddeercity @Audionut

You guys perfectly helped me out last time. Is there something you can do? If I can't manage to recover this footage I will have a huge problem with my customers.

[a1ex]

60 fps through fps override

Why not from Canon menu? What was the FPS set in Canon menu?

Managed to get this and this (first two frames), but each frame seems to require manual adjustment - notice they are not aligned horizontally. Can probably be automated, depending on how important the footage is.

Correction for frame 0 (shifted the raw buffer by 3 6 bytes) also applies to frames 2, 3, 4, 6, 7, 8, 10, 12, 14, 16, 18 ..., but these are all identical (first frame repeated all over). So, at least half of the footage is lost.

Correction for frame 1 (shifted the raw buffer by 6 12 bytes) also applies to frames 5, 9, 11, 13, 15, 17, 19, 21... with isolated exceptions that seem recoverable (they just require a different offset; 3 6 bytes works at least for some of these). So, the other half of the frames seems to be recoverable (there may be exceptions).

mlv_dump diff (on top of crop_rec_4k branch):

Code: [Select]

@@ -3441,7 +3444,7 @@
                                 /* if passing through original uncompressed/lossless raw */
                                 case UNCOMPRESSED_ORIG:
                                 case COMPRESSED_ORIG:
-                                    dng_data.image_buf = (uint16_t *)frame_buffer;
+                                    dng_data.image_buf = (uint16_t *)frame_buffer + 6;   /* adjust offset here; trial and error */
                                     dng_data.image_size = frame_buffer_size;
                                     break;
                             }

After patching mlv_dump source, run with:

Code: [Select]

mlv_dump_patched M13-1537.MLV -p --dng --black=2048

BTW - next time you should consider using a higher ISO.

[Aeidxst]

Why not from Canon menu? What was the FPS set in Canon menu?

Managed to get this and this (first two frames), but each frame seems to require manual adjustment - notice they are not aligned horizontally. Can probably be automated, depending on how important the footage is.

BTW - next time you should considered using a higher ISO.

The settings in my Canon menu were 1280 x 720, 50 fps, ALL-I, and I put fps override to 60.

So, you're saying these corrupted files are fixable? I would kill to get this fixed.

Also, has my ISO anything to do with this? I didn't use higher ISO because I was afraid of noise.

And how can I avoid this problem in the future?

[a1ex]

Edited the post - you can recover half of the frames with some tinkering.

ISO had nothing to do with frame corruption - the low ISO is the reason you've got so much noise (since the exposure time was pretty much maxed out)

(already explained why it happens; read the link above)

Though, before increasing ISO, you should have considered recording at some lower FPS (30 or 24) and using a higher exposure time.

And how can I avoid this problem in the future?

Try using 60 FPS from Canon menu.

[Aeidxst]

Edited the post - you can recover half of the frames with some tinkering.

ISO had nothing to do with frame corruption - the low ISO is the reason you've got so much noise (since the exposure time was pretty much maxed out)

Try using 60 FPS from Canon menu.

Thanks for your reply. However, I have no clue what everything you says means. I am very a-technical and I have no clue how to fix this myself. I can pay you to recover my footage, if you would allow me to. Would you accept this? I really have a big problem if I can't recover this footage.

[Aeidxst]

Edited the post - you can recover half of the frames with some tinkering.

ISO had nothing to do with frame corruption - the low ISO is the reason you've got so much noise (since the exposure time was pretty much maxed out)

(already explained why it happens; read the link above)

Though, before increasing ISO, you should have considered recording at 30 FPS and using a higher exposure time.

Try using 60 FPS from Canon menu.

There is not 60 fps in the Canon menu:



Also, would you accept this?

I can pay you to recover my footage, if you would allow me to. Would you accept this? I really have a big problem if I can't recover this footage.

[allemyr]

I had some problem like this some years ago. And in my case it was a counterfeit Lexar 128gb 1066X I've bought and had trouble with one time after the recording shut down unproperly because of the card. It was during a little test when I was filling around 70gb of the card I went into errors, I bought it on ebay and the seller with sort of 100% rating didn't want to helt me even while stating life time garauntee during sales. Send Sanddisk a email with serial number of the Extreme PRO card and let them check if it's they have it in their registeres.

That was more then 3 years ago and have since then used a Lexa 1066X 256gb card without a single problem, Shure it was way over twice as expensive as the counterfeit card, but it was produce by Lexar and was real.

Sorry for long post, and I hope that your card doesn't have these issues as my had.

Other then that, the only other thing that could be failing is MLVFS app but it was some years ago I read about an old app causing these issues and think that every programmer have solved those problems years ago.

[allemyr]

There is not 60 fps in the Canon menu:



Also, would you accept this?

Ehh ok?

Have you read about PAL and NTSC? NTSC support 60fps, i guess you have your camera setting on PAL?

[Aeidxst]

Ehh ok?

Have you read about PAL and NTSC? NTSC support 60fps, i guess you have your camera setting on PAL?

Yep, had it on PAL. Just changed it to NTSC, hope this problem never occurs again.

-snip-

So, could I pay you to fix it to me? I'm very a-technical and there's no way I'd ever be able to do this myself.

Or anyone who can help me with this? I'll pay.

[allemyr]

Yep, had it on PAL. Just changed it to NTSC, hope this problem never occurs again.
So, could I pay you to fix it to me? I'm very a-technical and there's no way I'd ever be able to do this myself.

Or anyone who can help me with this? I'll pay.

The problem will occur again for shure its not the Pal/NTSC setting I'am quite shure of, Iäve record in 70fps form 60fps and its fine no ruined image. You say something about the the zoom/magnifie button, never heard of it causing issues, but could be.

I use it sometimes this button when using the 3X digitalzoom function when recording but never ran into problem.

Don't use the zoom button that much is my advice if thats causing issues, use focus peaking internally on 5D3 or on a seperate display that has it for checking focus.

But my bet is still on the memorycard, or the magnifi thingy, if so, use it very rarely.

Ruined photos like this if the problem is during recording that cause these issues can't be fixed afterwards. I'am vastly shure of that the main creator/coder of Magic Lantern doesn't have time or interest in this.

Ok, I sound a bit rude above, but you can get exceptionel footage from this camera when treated right.

[a1ex]

Alright, our tip jar wasn't filled since last summer

Please find the recovered files: M13-1537.zip and M13-1538.zip.

Luckily, it was easier than I thought - the offsets are either 6 (12 bytes) for most frames or 3+7 (6+14 bytes) for a few of them; then, a script can look at these two sets of images and choose the one which is not "too red". Only one frame required manual tweaking - 622 from the second clip (there are a few pixels lost on the right side).


Reproduced the issue as well. Settings:
- mlv_lite with Framing preview enabled
- 60 FPS from 50, low light, actual FPS 60.024
- press zoom in and back (this one is important; I tried many times to reproduce by going to PLAY mode and back, no success)
- in this state, real-time preview (rendered by Canon code) looks clean (!)

It's not the CF card (100% sure about that).

[allemyr]

Haha I should shut up for shure

[Aeidxst]

Alright, our tip jar wasn't filled since last summer

Please find the recovered files: M13-1537.zip and M13-1538.zip.

Luckily, it was easier than I thought - the offsets are either 6 (12 bytes) for most frames or 3+7 (6+14 bytes) for a few of them; then, a script can look at these two sets of images and choose the one which is not "too red". Only one frame required manual tweaking - 622 from the second clip (there are a few pixels lost on the right side).


Reproduced the issue as well. Settings:
- mlv_lite with Framing preview enabled
- 60 FPS from 50, low light, actual FPS 60.024
- press zoom in and back (this one is important; I tried many times to reproduce by going to PLAY mode and back, no success)
- in this state, real-time preview (rendered by Canon code) looks clean (!)

It's not the CF card (100% sure about that).

That's awesome of you, thank you. Here's the rest: https://fil.email/2kbKnycE

I'll definitely donate, as you basically saved my life when all those clips will be saved.

Really, I'll owe you big time.

[g3gg0]

t.b.h. bitcoin isnt right now the perfect choice for donations.
(even the transaction fees are higher than a typical donation would be. yeah the crash last weeks got it down to a moderate level, but it will raise again)
but due lack of alternatives, still feel free to donate in BTC

(alternatively for the server cost -> paypal@g3gg0.de, it is financed till december '18 right now)

[ilia3101]

What about dogecoin? It has some of the lowest fees.

I might have donated a bit just now (sorry not much)

[a1ex]

Forgot to update the status: recovered everything except for one clip and a few isolated frames, at half FPS (30 instead of 60).

Hopefully OP received the files (did not hear back from him since then).

Thanks Ilia for the tip