h.264 video glitch 5DM3

[JADURCA]

Hi everyone! Recently filmed an event and h.264 video came with some glitches. From 200 clips approximately 120 have it. Thought it was USB data cable that was connected Crane 3 stabilizer for controlling ISO, shutter and ISO, but when disconnected it for a few hand held shots, glitches still there. Used crop_rec_4k.2018Jul225D3123

Here is a sample: https://drive.google.com/drive/folders/1mI8aBj2Ukl-K9qb16uhytcLrt5x6ThHD?usp=sharing

[Danne]

What glitch? Please provide more info and also present a still image of the so called glitch.

[JADURCA]

Just uploaded two snapshots in same folder.

[Danne]

Ok. I´d say the card is not working properly. It would also help if you would be more informative.

[a1ex]

I'm unable to locate the broken frames in the test clip when playing it with VLC.

Best guess: playback problem on your computer. You could try extracting all frames with ffmpeg, for example, and inspect them one by one:

Code Select Expand


ffmpeg -i 6E2A3403.MOV frame%05d.jpg


The only glitch I could identify, was... monitor flicker.

[JADURCA]

Thanks Danne and a1ex!

Using Windows 10 Home 64-bit
Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz
32 GB RAM
NVIDIA GeForce GTX 1050 Ti

Used other players like Movies & TV, Windows Media Player (old one) and VLC. All of them exhibited same glitch at same frames. QuickTime Player curiously is the only one that plays it without any glicth.

[Danne]

If your uploaded file and screenshots are from the uploaded file it could very well be your mediaplayer. Just export your file as images as suggested by A1ex or from ae or similar and examine those to verify your issue source.

[JADURCA]

Is looks like is a playback problem on my computer. How can that even happen? 

[Walter Schulz]

Things to investigate:
- Anti-virus/internet protection software
- Graphics driver. If you have an onboard video port you may want to disable/remove your graphic card for a test.

[yourboylloyd]

Is looks like is a playback problem on my computer. How can that even happen? 

Is the flicker appearing in your editing software too? I know BORISFX can remove flicker but it can be a bit pricey...

Or maybe your monitor refresh rate? Try it on a different monitor?

[JADURCA]

Is the flicker appearing in your editing software too? I know BORISFX can remove flicker but it can be a bit pricey...

Or maybe your monitor refresh rate? Try it on a different monitor?

Yes. Glitch appears in my editing software too. Actually I am concerned about the actual glitch rather than monitor flickering in the background, that's something I learned to live with. At least in live events where is hard to change the scenario.

Things to investigate:
- Anti-virus/internet protection software
- Graphics driver. If you have an onboard video port you may want to disable/remove your graphic card for a test.

Antivirus is OK! About graphic driver, can be the problem. I'm going to disable video card and then come back with the results. Thanks!

I'm unable to locate the broken frames in the test clip when playing it with VLC.

Best guess: playback problem on your computer. You could try extracting all frames with ffmpeg, for example, and inspect them one by one:

Code Select Expand


ffmpeg -i 6E2A3403.MOV frame%05d.jpg


The only glitch I could identify, was... monitor flicker.

Going to extract! Thanks for the suggestion!

[JADURCA]

Here are the results!


<a target='_blank' href='https://imgbb.com/'>pie cartoon image</a>

[Luther]

The video has corrupted intra frames. Your samples glitch on my computer too (using mpv player).
I was able to transcode and it (apparently) fixed the issues. Try this command:

Code Select Expand

ffmpeg -i 6E2A3403.MOV -c:v libx264 -preset slow -crf 18 -tune film -coder cabac -pix_fmt yuv420p -c:a aac -b:a 320k -movflags +faststart 6E2A3403_compressed.mp4

[JADURCA]

The video has corrupted intra frames. Your samples glitch on my computer too (using mpv player).
I was able to transcode and it (apparently) fixed the issues. Try this command:

Code Select Expand

ffmpeg -i 6E2A3403.MOV -c:v libx264 -preset slow -crf 18 -tune film -coder cabac -pix_fmt yuv420p -c:a aac -b:a 320k -movflags +faststart 6E2A3403_compressed.mp4

Thanks Luther! It looks clean now, there still like a bump in the corrupted frames.

What can I do to avoid that when filming next time? Because almost 40% of all the clips got this random glitch. Fixing that much clips takes time.

Could it be the build or CompactFlash Card?

[Luther]

there still like a bump in the corrupted frames.

Yeah, you won't be able to fix those frames, unless you try to interpolate them. Twixtor might help.

What can I do to avoid that when filming next time?

I can't know exactly what caused the corruption in your case. But you should test different settings. In my old 600D, these corruptions used to happen sometimes when using really high bitrate (CBR 3.0x) and FPS Override together, but now the ML build is really stable.

Could it be the build or CompactFlash Card?

Could be, don't know. Try to reduce CBR settings if your card is not fast enough. You can use the benchmark inside ML to see what writing speed it has. If it is more than ~20MB/s, then you're fine. If it's too low (<10MB/s) then you probably need a new/faster card.

[JADURCA]

Card is a Komputer Bay 128GB 1066X UDMA 7

[Luther]

Card is a Komputer Bay 128GB 1066X UDMA 7

If it's working properly, then you should be fine (I have one of those). But you should test using some software, flash memory like CF cards usually don't last too long and get bugged cells. I don't know if the corruption was caused by your card, though.

[JADURCA]

Maybe there is an tool to check if card is ok.

[Walter Schulz]

H2testw
F3/F3X

[reddeercity]

Check the file in the first post & I'm not getting this problem , check on win7pro (cpu AMD FX8350) with mpv player which uses GPU (GTX580 Nvidia).
Even in vlc player looks ok , I think you have a h264 decoding problem .
I Loaded it in MLVProducer and exported the h264 .mov file into JPeg image sequence and found only 3 frames with a small glitch .
Frame 46 , frame 59 & frame 389 . I zipped up the full jpeg sequence (440 frame) to a 6E2A3403.rar , I marked the bad frames in the properties (comments)

[a1ex]

found only 3 frames with a small glitch .
Frame 46 , frame 59 & frame 389 .

Confirmed - this shows how bad I am at pixel peeping

If you look close at these frames, the top part is good, but the bottom part was taken from the previous frame. Other H.264 decoders might choose to fill the bottom part with the last good row of pixels - which may explain how the defect appears on different computers - even with the same software.

Here, it looks "ok" in both vlc and mpv, at first sight; the glitch is, indeed, noticeable if you know where to look.

How did this defect appear? Most likely the card. I cannot imagine how a bug ML would cause such kind of corruption - extremely unlikely in my opinion. You could cross-check with plain Canon firmware, to see if the defects are still there (very likely IMO).

To fix the bad frames from existing footage, you will have to interpolate them. Possible approaches, besides Twixtor: ffmpeg with minterpolate, pyflow, or one of these, or... see also here.

To avoid such situations, I can imagine some future ML features:
- record H.264 video on both SD and CF at the same time (shouldn't Canon firmware already do this? apparently it does only for photos)
- save some kind of checksum (to detect file corruption), or ECC (to correct some bit errors), assuming the CPU is fast enough to do this in real-time (maybe for low bitrates and still photos only?)
- verify (re-read) each saved file when recording stops, and warn the user before further corruption occurs? (this is going to be slow, but if data integrity is important, it might be worth the trouble)

[JADURCA]

If it helps, don't know if has something to do with this glitch, in SD card there is a crash log notepad. Here is the report:

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:983, MoviePlay:b1f8
lv:1 mode:3

MoviePlay stack: 1766e8 [176870-175870]
0xUNKNOWN  @ de48:176868
0xUNKNOWN  @ 17bbc:176840
0x000178B4 @ ff0e31dc:176828
0xUNKNOWN  @ 178e4:176818
0xUNKNOWN  @ 1796c:1767f8
0xFF1E450C @ ff1e4d58:1767b0
0xFF1E3E28 @ ff1e4874:176778
0xFF1EAB14 @ ff1e3e74:176740
0xFF1EC1A8 @ ff1eab68:176730
0x0000B1D0 @ ff1ec1b0:176728
0x00001900 @ b1f4:176720
0x00069878 @ 69c1c:1766e8

Magic Lantern version : crop_rec_4k.2018Jul22.5D3123
Mercurial changeset   : c1e44b8e0183 (crop_rec_4k_mlv_snd) tip
Built on 2018-07-22 13:10:52 UTC by jenkins@nightly.
Free Memory  : 211K + 3644K

[Danne]

Why not run tests with the card without magic lantern installed? Makes little sense to look for issues around ml If your card is broken which probably is the case.Try and exclude most probable cause first.

[a1ex]

In a previous thread, OP had corrupted CR2 files without ML installed. Was it with the same card?

[Danne]

More than likely. Strange if he keep on using the card still assuming ml is causing corruption. Maybe trolling around...

[JADURCA]

H2testw
F3/F3X

Thanks for the suggestion Walter! I'm running test on CompactFlash right now!

[Walter Schulz]

It looks like you are not using an UDMA-7 compatible cardreader. You know Komputerbay is very open about their cards getting corrupted (permanently damaged) when used in incompatible devices?

[JADURCA]

Using this one:

Atech Flash Pro-37U
Specs
1 x CompactFlash Type II (UDMA 7 [167 MB/s])
Supports UHS-1 and UDMA 7 read/write speeds

It is connected to mother board using a USB3 output.

[Walter Schulz]

Good! But data rates shown in snapshot are way, way worse than expected from an USB 3.0 device and such a card. Did card speed decrease over time in your h2testw run or was it that low all the time?
h2testw is not designed to use highest write/read transfer (compared to dedicated tools like CrystalDiskMark) but numbers that bad are alarming!

[JADURCA]

Yeah, still that low all the time. Going to try CrystalDiskMark

[Walter Schulz]

And do a benchmark run inside your cam ...

[JADURCA]

<a target='_blank' href='https://carinsuranceguru.org/does-each-driver-need-car-insurance-or-just-each-car'>insure a car not in your name</a>

[Danne]

Doesn't matter what ratings you get. If sectors are defective your card will still be shitty.
Just run the card with ml uninstalled for a couple of hours.
If you get only one single corrupted frame or movie file like described you can treat the card as garbage. Never to be trusted again.

[JADURCA]

Okey! Uninstalling ML then going to do all photos and video normally to see results.