5D Mk II - completely glitched out clips with wrong resolution

[Aeidxst]

Hi everyone,

I encountered yet another problem, which is even more devastating than my previous one. I have always used raw_rec on my Mark II and I've never really had any problems. I recently switched to the new RAW (mlv_lite).

First, here are my settings:

I'm recording 14-bit RAW video at 1856 x 1044, 23,976 fps. There are no other ML features being used, except the mlv_lite and mlv_play.

I'm using a Canon 5D Mark II, with 50mm 1.2 and 24-105mm 4.0 lenses, using SanDisk Extreme PRO as storage (64GB, 160 MB/s). No external monitors connected

This is in my ML folder > Settings > raw_rec.cfg.

# Config file for module mlv_lite (MLV_LITE.MO)

raw.video.enabled = 1
raw.write.speed = 7335


Yesterday I recorded some (very important) footage. When on the set, I already wasn't able to view back some of the footage with mlv_play. The weird thing is that it was only some of the footage. It just gave me a black screen with an error along the lines: "Could not load blocks" or something.

I just converted all my footage to .dng sequences using MLVFS. All of the footage I couldn't view on the set suddenly turned into 1920x1078 footage, instead of the 1856x1044 I originally shot at. And even worse, all those clips became completely glitched out. See pic related. All the frames of the clips of which the resolution changed (which I couldn't view on set) became like this. About 20% of all the clips I shot turned into pic related.


What caused this, and what can I do about this? Can I still save my 'corrupted' files/footage?

[a1ex]

Can you upload a sample MLV?

Regarding what caused this, I have no idea. The mlv_lite module was available for testing for about one year before it was included in the nightly builds, and no such problems were reported.

[Aeidxst]

Can you upload a sample MLV?

Sure.

Here's one that isn't broken:
https://we.tl/U8cYSOLQEH

And here is its .dng sequence:
https://we.tl/bj19posJK4



Here's one that's broken. I think I even shot it within the same minute, with no change in settings whatsoever.
https://we.tl/efk870YaM5

And here is its .dng sequence:
https://we.tl/MMBDS8x11k


I included the .IDX and .M00 and .M01 and .MRX files too.

I really hope you or someone can help me with this. I have already tried other converters such as cr2hdr, same result. So there's really something wrong with the files.

[a1ex]

The good news is that your file can be recovered with a simple patch to mlv_dump:

Code: [Select]

--- a/modules/mlv_rec/mlv_dump.c
+++ b/modules/mlv_rec/mlv_dump.c
@@ -3173,6 +3173,9 @@
                 
                 lua_handle_hdr(lua_state, buf.blockType, &block_hdr, sizeof(block_hdr));
 
+                block_hdr.xRes = 1856;
+                block_hdr.yRes = 1044;
+
                 video_xRes = block_hdr.xRes;
                 video_yRes = block_hdr.yRes;
                 if(verbose)

then:

Code: [Select]

mlv_dump M02-1834.MLV --dng --black-fix=1780

Now, the question is how to reproduce your error. Best guess (without trying; just by examining the source code): I believe the bug only happens when the following conditions are met:
- in menu, you have selected a resolution higher than your max resolution (here, 1920 vs max=1856)
- you start recording very quickly after returning from zoom mode

Even so, the probability of triggering the bug will be lower than 100%.

Can you confirm?

If yes, the problem is unique to mlv_lite and it's very easy to fix.

[Aeidxst]

The good news is that your file can be recovered with a simple patch to mlv_dump:

Code: [Select]

--- a/modules/mlv_rec/mlv_dump.c
+++ b/modules/mlv_rec/mlv_dump.c
@@ -3173,6 +3173,9 @@
                 
                 lua_handle_hdr(lua_state, buf.blockType, &block_hdr, sizeof(block_hdr));
 
+                block_hdr.xRes = 1856;
+                block_hdr.yRes = 1044;
+
                 video_xRes = block_hdr.xRes;
                 video_yRes = block_hdr.yRes;
                 if(verbose)

then:

Code: [Select]

mlv_dump M02-1834.MLV --dng --black-fix=1780


Okay, awesome to hear that my files can still be saved. Very happy about this, since 20% of everything I shot became corrupted like this.

However, I have absolutely NO clue how to do what you said. I just read what you typed (the entire code thing) and I don't undestand anything of it. What do I do, and how do I do it? Oh, and please mention on what OS you're working. I'm using a Mac atm, but I can do this on Windows if it's needed.

I believe the bug only happens when the following conditions are met:
- in menu, you have selected a resolution higher than your max resolution (here, 1920 vs max=1856)
- you start recording very quickly after returning from zoom mode

Even so, the probability of triggering the bug will be lower than 100%.

Can you confirm?

No, I haven't selected a resolution higher than my max (which is 1856x1044) because my camera doesn't let me.

Yes, I indeed started recording very quickly after returning from zoom mode. I find zoom mode very handy if I focus manually, especially with shallow depth of field. So I think that's the problem then.

[a1ex]

What do I do, and how do I do it? Oh, and please mention on what OS you're working. I'm using a Mac atm, but I can do this on Windows if it's needed.

The fix I've posted does not depend on your operating system. Any community member with command-line skills can help you with OS-specific steps.

No, I haven't selected a resolution higher than my max (which is 1856x1044) because my camera doesn't let me.

The default selection is 1920. The "camera doesn't let me" part didn't work very well, as you can select 1920 in zoom mode, so the internal resolution parameters weren't updated when you started to record.

[Aeidxst]

The fix I've posted does not depend on your operating system. Any community member with command-line skills can help you with OS-specific steps.

Ehm, okay, but I'm kind of stuck now. What would be the next step to recover the damaged/corrupted files?

[reddeercity]

@Aeidxst , I can help you I'm downloading your MLV right now . I'll be off line for a few hours thou but I will check this thread after that.

[Aeidxst]

@Aeidxst , I can help you I'm downloading your MLV right now . I'll be off line for a few hours thou but I will check this thread after that.

Hey that's awesome, thanks. I have more than only one file that has to be taken care of. Do you need them as well? Let me know when you're back online. I'm at GMT +1.

[Danne]

Here is a patched mlv_dump with the black fix number hardcoded as well according to a1ex proposal. Tested with your corrupt file. There is a howto file inside the zip.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_patch.zip

Just put all the files in the folder next to mlv_dump and the MAIN.command script. DO tests with copies first!

What you do is you double click the MAIN.command script and your MLV files will be processed into folders with dng files. Make sure to give the MAIN.command permission rights. It,s explained in the howto what to do. If any questions write back. Do backups. I tested only briefly. Should work for all files including .M00 etc.

[Aeidxst]

Here is a patched mlv_dump with the black fix number hardcoded as well according to a1ex proposal. Tested with your corrupt file. There is a howto file inside the zip.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_patch.zip

Just put all the files in the folder next to mlv_dump and the MAIN.command script. DO tests with copies first!

What you do is you double click the MAIN.command script and your MLV files will be processed into folders with dng files. Make sure to give the MAIN.command permission rights. It,s explained in the howto what to do. If any questions write back. Do backups. I tested only briefly. Should work for all files including .M00 etc.

I tried it out, and worked like a fucking charm. I literally have no idea how it's possible for someone to figure out how to fix a problem like mine. Truly incredible.

Thanks a lot. My important footage that was corrupted is completely restored. Can I keep using this download for future reference, in case this problem keeps occuring?

[Danne]

Great that it worked.

Can I keep using this download for future reference, in case this problem keeps occuring?

Use it anyway you want. Keep in mind it,s patched for your particular issue.
Thanks a1ex for the swift solution.

[Aeidxst]

Great, big thanks everyone. Wish there was a way to repay you. You have no idea how big of a problem you've solved for me.

[Audionut]

Wish there was a way to repay you.

There is.

My bold for emphasis.

Now, the question is how to reproduce your error. Best guess (without trying; just by examining the source code): I believe the bug only happens when the following conditions are met:
- in menu, you have selected a resolution higher than your max resolution (here, 1920 vs max=1856)
- you start recording very quickly after returning from zoom mode

Even so, the probability of triggering the bug will be lower than 100%.

Can you confirm?

If yes, the problem is unique to mlv_lite and it's very easy to fix.

[reddeercity]

@Aeidxst , Looks like you have solution to problem from a1ex , good.
I did fix the MLV Source File in a Hex Editor on my PC , Repaired_M02-1834.MLV and here is the repaired file process thought MLVProducer ,Repaired MLV ProRes
If Those links give you trouble use this one  https://drive.google.com/drive/folders/0BxNY4d4Aa7KuMlNDYVN3OTFoTlE?usp=sharing
I exported as a ProRes444 file in Adobe RGB Color Space and I add a Kodak2383.cube

I know this maybe a little more info then you like to know , but I'm writing this post so other may benefit from the knowledge.
In the RAWI block type at Offset 00000034 , 52 41 57 49 B4 00 00 00 49 02 00 00 00 00 00 00 80 07 34 04
The green highlighted number represent the Frame Size which is 1920x1076
I changed 80 07 34 04 to 40 07 14 04  with that I get 1856x1044
With that fix the MLV file open up in MLVProducer , MlRawViewer_1_3_4 but the black level is still at "0" which it need to be at "1791"
That's what really makes MLVProducer stand above the rest that you can set the Raw Black & white level on the fly in real time to see the results .
So in MLVProducer there's a black level slider or input the desired number in this case "1791"
The one tool that really help in these type of problems is mlvdiag.1.0.1 Big thanks to @chmee  and of course hex editor hxd-hex-editor 
In my 5D2 I never came across this problem before thou I only use Full MLV+Audio so this maybe a issue with MLV_Lite.
 

[Aeidxst]

There is.

- in menu, you have selected a resolution higher than your max resolution (here, 1920 vs max=1856)
- you start recording very quickly after returning from zoom mode

Even so, the probability of triggering the bug will be lower than 100%.

Can you confirm?

If yes, the problem is unique to mlv_lite and it's very easy to fix.

@reddeercity @Audionut @a1ex @Danne

So, I just tried reproducing the bug with the settings exactly the same as in OP. I shot the tests with a 50mm 1.2 lens. It gave me interesting results. What I did is the following:

- [1] I started recording regularly, without using the zoom function right before it.
The bug happened 0/10 times.

- [2] I started recording right after leaving the zoom function.
The bug happened 6/10 times. Two of those six MLV files don't even give me a .dng sequence when converted. Those two don't give me any frames at all. Four of those six MLV files became glitched as before. Pic related (1) is the 'completely glitched out' bug.

- [3] I started recording while in the zoom function. The camera doesn't let me record in x10 zoom (it says 'RAW detect error') but it does let me record in x5.
Another bug happened. Instead of the screen completely glitching out, all the footage had a pink/purple hue over it. This happened 10/10 times. Pic related (2) is the pink/purple hue bug.

- [4] I started recording right after I had recorded in zoom function. To be clear, I was recording exactly the way I did at [1].
The pink/purple hue bug happened 4/10 times. Pic related (3) is the pink/purple hue bug while not being in zoom function.

An important thing too: The pink/purple hue bug sometimes happens INSTEAD OF the completely glitched out screen bug. I have no idea what causes it to be either the GLITCH bug or the HUE bug.


Few more questions I am having now:

- How can I avoid/fix the pink/purple hue bug? It hasn't happened to any of my important footage yet though. But since I encountered it while trying to reproduce the other bug, I figured that it might occur at some of my 'real' recordings too.
EDIT: I just tested the MAIN.command file (from a1ex and Danne) on my pink hued footage, and it works. So that problem is also solved. Still doesn't explain why same of the recordings/MLVs contain 0 frames.

- Why does my camera/ML switch to 1920x1076 instead of the original 1856x1044 when I switch to zoom mode? Why doesn't it stay at 1856x1044? Even weirder, I cannot even select 1856x1044 when I'm in zoom mode. It's always 1920xXXXX or 1600xXXXX (or a lower first number). Why can I only tweak the second number, and not the first one? If I could set it to 1856x1044 in the zoom mode too, I might be able avoid the glitch in the first place.


ATTACHMENTS

Pic related (1) - the 'completely glitched out' bug which Danne and a1ex helped me with.



Pic related (2) - Hue bug in x5 zoom mode



Pic related (3) - Hue bug in regular (no zoom) mode


For reference, this is what the glitched images should've looked like. This is a frame from a test in which the bug didn't occur.


I really hope I'm contributing something to the development of ML with my time and tests. I'm doing it for myself regardless, since I want these questions/bugs answered as well

[reddeercity]

@Aeidxst , this is in mlv_lite/old Raw_rec right not full mlv+audio ?
can you try the same with full MLV+audio (set audio to 44.1 kHz instead of 48khz or turn off audio for more performance)
please post results . Also if you don't mind could you try the experimental 10-12bit build for the 5D2  , there's still a problem in 1:1
Of which I'm looking in to , I maybe close to a solution.
@ 10bit (not useable , half the frames are corrupted) but 12bit is useable thou there is a pink noise bar at top , about 80 to 100 lines
best resolution to use is 1856x1044 & of course there is still 14bit . Look in the Raw Video Tab to make your selection .
3x Crop works without any problem so far (10bit @ 2144x1072 23.976 fps is continuous) ,
be sure to use full MLV+audio as raw_rec(mlv_lite) has many issue that are far from resolved .
I'm interested to see if there a problem with 1:1,  first try 14bit then 12bit ,
do the same thing that you did to cause the problem that has been reported by you.

Edit:
For some time now 5D2  3xCrop in mlv_lite (old raw_rec) has wrong black level (the frames you posted)
This is not a problem in regular or full mlv+audio , but easy to correct and most camera raw software can view
the frames correctly e.g. ACR Adobe After Effects , Blackmagic Resolve  etc... .

[Aeidxst]

-snip-

Yes, these tests are done in mlv_lite.

So, what you are basically saying is that mlv_lite is far from flawless, and I should use full mlv_rec instead? The problem with that is that I usually shoot clips that are 1-2 minutes long, and full mlv_rec doesn't allow me to record more than max. 20 seconds...

[reddeercity]

Yes, these tests are done in mlv_lite.

Ok

So, what you are basically saying is that mlv_lite is far from flawless, and I should use full mlv_rec instead? 

No not really , remember ml can have issue on different cameras and to make the code portable across
19 cams sometimes can be taxing for developer to keep up with bug fixing and if not enough people test new or improved
features then it get missed or discover later down the line .

The problem with that is that I usually shoot clips that are 1-2 minutes long, and full mlv_rec doesn't allow me to record more than max. 20 seconds...

You can try older builds , you don't necessary need to use the latest bleeding edge build in fact most of the new builds benefit the newer digic 5 & 6 cam e.g. eosm , 70d , 5d3, etc... .
there not much for digic 4 e.g. 5d2 , 50d , etc... .

Go to  http://builds.magiclantern.fm/5D2-212.html and near the bottom of the page is Older Builds try this one magiclantern-Nightly.2016Dec19.5D2212.zip
There's a list of details that the nightly builds has just below each one .

One more question , so when you did record mlv_lite 1856x1044 did you kill overlays ?
I tried on my 5d2 with over lays killed I can get about 6500-8000 frames @23.976 ,
but the problem is you don't know when the recording will stop.
and all frames look good no corruption.
I also tested 1856x1044 14bit full mlv+audio with audio turned off with hacked liverview preview and I was getting about 90 seconds.

I guess you need to decided which work around best fits your needs, but at some point in the near future I do expect these issue to be resolved
just need more 5d2 user to help out with R&D.
Once the 10-12bit get stable then there will not be any need for mlv_lite/raw_rec  just full mlv+audio .

[dariSSight]

I'm not great at coding but I'm willing to do what I can to Help on R&D. I use the latest build but I still use Feb142014 MLV2.0 & Oct232013 ML1.0, how stable are the Dec19,2016 in 1.0 and 2.0? I would love to film in audio but I have enough High Quality Audio Devices that I could easily record external audio with excellence, I need clean RAW Video. All day I will be doing a thorough test of Magic Lantern just so I can know which is good under pressure for some project I'll be working on for the next 3 months. Any Advice? Lastly I've decide post-process will be MLVFS to ACR In AfterEffects for White Balancing, AfterEffects for ProRes 4444XQ and Latest FCPX for editing and Grading. Is this the best post-process for Magic Lantern RAW?

[DeafEyeJedi]

...MLVFS to ACR In AfterEffects for White Balancing, AfterEffects for ProRes 4444XQ and Latest FCPX for editing and Grading. Is this the best post-process for Magic Lantern RAW?

If speed turnarounds are part of your priorities then yes this is an excellent workflow otherwise use @Danne's latest & greatest cr2hdr.app which does wonders. It's basically a built-in DIT within the app and can handle pretty much anything you throw at. I cannot live without it, literally!

[SpcCb]

I have already encountered some similar issues while filming with the process "zoom to make focus >> un-zoom to film" when I start the recording too fast after un-zoom, or when I start recording just after a previous recording stopped (ie. while the LCD screen is refreshing and the LED is blinking). Sometimes I got corrupted frames, but in most of cases audio was 'out of time' (worst than desync and impossible to resync/reconstruct, it's like the audio file is fragmented and placed in random order). I haven't tried all builds but it appends with builds made in 2014-02-16, 2016-10-09, 2017-01-27, with regular RAW and MLV, so maybe it is just because the camera CPU is not fast enough or for a question of buffering (?), &so that we have to wait a few seconds between some kind of actions. Since I know that, I wait a couple of seconds before all recording starts and all is working fine.

[a1ex]

Fix pushed. If you can compile faster than nightly build server, please test and report before it explodes.

[dotunololade]

Here is a patched mlv_dump with the black fix number hardcoded as well according to a1ex proposal. Tested with your corrupt file. There is a howto file inside the zip.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_patch.zip

Just put all the files in the folder next to mlv_dump and the MAIN.command script. DO tests with copies first!

What you do is you double click the MAIN.command script and your MLV files will be processed into folders with dng files. Make sure to give the MAIN.command permission rights. It,s explained in the howto what to do. If any questions write back. Do backups. I tested only briefly. Should work for all files including .M00 etc.

@Aeidxst, Could you please help with this zip file? I couldn't find it, but experienced the same problem just yesterday and need to deliver project first thing on Monday at the office. Thanks.

[dotunololade]

I tried it out, and worked like a fucking charm. I literally have no idea how it's possible for someone to figure out how to fix a problem like mine. Truly incredible.

Thanks a lot. My important footage that was corrupted is completely restored. Can I keep using this download for future reference, in case this problem keeps occuring?

Hi, I have a similar problem and tried to download the zip file but it isn’t available anymore. Do you by chance still have it?