Uncompressed 14-bit RAW video testing - 5D Mark III

[jarabmx]

itsDPmikey: That nightly is no longer available, any chance you could upload it here? Thanks

[Walter Schulz]

https://builds.magiclantern.fm/jenkins/
-> 5D3.113
-> Scroll down -> Left link "More" -> All 389 builds available.

But take a look into his newer posts. He changed his mind from "BEAUTIFULLY!" to not so enthusiastic ...

[jarabmx]

Hmm, so is there any consensus on what is causing this issue and how we can avoid it? For me it is almost a deal breaker with currently almost every third take having corrupt frames.

[Danne]

Turn off global draw.

[jarabmx]

Turning off global draws in 50fps means no way of framing the shot, no focus peak, no information about anything. So that is not an option either..

[Danne]

What happens when you shoot with RAW and global draw on? Or with new format mlv_lite? Also turning down image size a bit to not stress the processor could work?

[spins]

I've been using the Dec2015 build - there seems to be no option for shooting in 14 bit raw?  It's been quite the task to downgrade from 1.33 to 1. 23 and now can't even find the raw function, shouldn't this be more obvious?
Thank you!

[beauchampy]

I've been using the Dec2015 build - there seems to be no option for shooting in 14 bit raw?  It's been quite the task to downgrade from 1.33 to 1. 23 and now can't even find the raw function, shouldn't this be more obvious?
Thank you!

Have you enabled mlv_rec in the modules menu?

[jtvision]

Hmm, so is there any consensus on what is causing this issue and how we can avoid it? For me it is almost a deal breaker with currently almost every third take having corrupt frames.

Change to 1280x720 @50 fps from canon menu. You are getting corrupt frames because you're overwriting from 30fps to 50fps. When shooting raw you will still get 1080p even though your settings are at 720p as per canon menu.

[spins]

there is NO mdules menu in this latest build (I just got it over the weekend).  I've seen it on tutorials, but not on my version???

Posted by: beauchampy
« on: March 13, 2016, 03:55:01 PM » Insert Quote
Quote from: spins on March 13, 2016, 06:02:24 AM
I've been using the Dec2015 build - there seems to be no option for shooting in 14 bit raw?  It's been quite the task to downgrade from 1.33 to 1. 23 and now can't even find the raw function, shouldn't this be more obvious?
Thank you!

Have you enabled mlv_rec in the modules menu?

[Walter Schulz]

Browse the screens and read captions.

[jarabmx]

I am switching to 50fps 720p in Canon menu and it is still happening.

MLV_lite does not have sound - I guess I could go back to RAW in that case..

All I want is

no failed frames
sound
global draw on

at whatever resolution it can keep up.

[kitp]

Yeah, I tried with card spanning - no help.

[beauchampy]

Did we ever get to the bottom of these pink frames? Quite keen to use the latest nightly with the new crop mode, but not if we're still getting corrupt frames..

EDIT: Just been testing the April 2nd 2016 nightly. Recorded 9 clips at 50p and two of them had corrupt frames. This is recording MLV on 1.2.3 with GD and Audio on.

I've gone back to a build from August 7th 2014th which doesn't have a problem with the above settings.

Devs - how can we be of help to isolate the issue?

[a1ex]

Can you upload your config files?

[beauchampy]

Can you upload your config files?

Sure, here's a Google drive folder with my config plus two corrupt clips.
https://drive.google.com/open?id=0B-GY59oq1k7eTHNzdms2bHRnWVE

I just ran the test again, and had 3 corrupted clips from 7 attempts. The two clips above are from this second test run, dumped my config settings right after.

[beauchampy]

I'd be really interested to know if anyone else can re-produce the pink corrupt frame glitches
I'm getting. Just load up my ML settings, record 10 50p RAW clips in a row and see if any of them have corrupt frames.

EDIT: a1ex, one thing I've noticed is that builds that don't have the 40gb limit on raw recording are the ones that seem to glitch for me. Could that change be part of the issue?

[DeafEyeJedi]

@beauchampy -- I'll give it a shot with your ML settings and report back.

[squig]

Been testing the April 2 build trying to nail the pink frame bug  I did a fresh 1.2.3 firmware update after a 64Gb 1066x Toshiba card shat itself (it's been resurrected). I fiddled with just about every Canon & ML setting and couldn't track it down. It doesn't appear to be caused by a conflict with any particular setting, or card.

Apr2: 1920x818 24p (exact FPS) mlv_rec & mlv_snd loaded, global draw on (raw histogram) snd on, 180 degree shutter. Recorded 12x 10sec shots and 2x 2min shots. Toshiba 1066x 64GB. No bad frames.

Apr2: 992x596 60p (exact FPS) 3x3 crop, mlv_rec & crop_rec loaded, GD (raw histo) on and off, 180 degree shutter. Recorded 12+ 5, 10, 30sec shots. Toshiba 1066x 64GB. 1 to 2 bad frames on 3 shots. Recorded 1312x472 with 3x3 crop off: same result.

Apr2: 992x596 60p (exact FPS) 3x3 crop, mlv_rec & crop_rec loaded, GD (raw histo) on, 180 degree shutter. Recorded 12x 10sec shots. Lexar 1066x 64GB. 1 bad frames on 2 shots.

Dec20: 1312x472 60p (exact FPS) mlv_rec loaded, GD (raw histo) on, 180 degree shutter. Recorded 12x 10sec shots. Toshiba 1066x 64GB. 1 bad frame on 2 shots.

Aug7/14 1344x502 60p (exact FPS) mlv_rec loaded, GD (raw histo) on, 180 degree shutter. Recorded 12x 10sec shots. Lexar 1066x 64Gb. No bad frames

[beauchampy]

I can throw a ton of abuse at the 7th Aug 2014 build and not one bad frame. I'm almost convinced it has something to do with the 40gb limit. I'm a dumbass when it comes to programming, but can anyone build the latest build again but with whatever change happened to lift the 40gb limit reversed?

[strikermed]

I previously had an issue with Magic lantern Raw producing pink frames.  I tested all the MLRAW to CDNG converters out there that I could find.  Just about all of them had issues producing pink frames.  I did however find this build: raw2cdng.1.5.0.betra7 to be flawless in shooting an entire 128GB compact flash card on 5d Mark III with firmware 1.2.3 and magic lantern build: Nightly.2015Dec20.5D3123.

This build records audio, and didn't produce any pink frames when paired with the raw2cdng.1.5.0.beta7 converter.  Keep in mind this is the MLV moduel, not the RAW module with Magic Lantern.  I also disabled Global draw when I did these tests.  I shot at 1920x1080 24p on 5d3.  Here is a link to the test shoot I did: https://youtu.be/wyH6z47rmL4

I haven't done much testing since then, but this was shot in February, using the said versions of software and Magic Lantern. 

In conclusion, I've come to the realization that some of the issue with pink frames may not come from the MLV files themselves, but from the converters doing the conversion.  I can tell you that I've tested the same footage, and I've got different results from different software. 

I propose that I take a file that you are getting pink frames with and trying it with this converter to verify if it's Magic Lantern causing the issue, or if it's your converting software.

Food for thought!  Love ML and love what you guys have done with it in the last 5 years.  I can't wait until it's actually partnered with Canon (if they will ever do such a thing) to create an all in one package!

[beauchampy]

In conclusion, I've come to the realization that some of the issue with pink frames may not come from the MLV files themselves, but from the converters doing the conversion.  I can tell you that I've tested the same footage, and I've got different results from different software. 

Interesting, I had thought of this too. However, I get pink frames when I play back the MLV in camera! I also get them in MLRawViewer as well as RawMagic (yes, I know it's not welcome around here). Had a look at raw2cdng, but it seems to be PC only? Want to try and convert one of my corrupt clips? There's two right here:

https://drive.google.com/folderview?id=0B-GY59oq1k7eTHNzdms2bHRnWVE&usp=sharing

EDIT: Also please do the record 10 clips at 50p test and let us know your results.

[a1ex]

I'm almost convinced it has something to do with the 40gb limit.

So, are you saying that:

* magiclantern-Nightly.2015Mar02.5D3113.zip (e8cc6e5) is buggy? (has pink frames?)
* magiclantern-Nightly.2015Feb20.5D3113.zip (36ff475) is fine? (as in, can you throw a lot of abuse at it without getting pink frames?)

If - and only if - the answer to both questions is a solid "yes", can you check this mlv_rec.mo on any of the two builds? (ML core is identical in both).

It should display "Last update: df0bbfe" when you press Q in the module dialog, and it's the only intermediate version between the two builds.

If - and only if - the modified mlv_rec.mo works fine, and you can say it with a solid "yes", can you try it (modified mlv_rec.mo) on a recent 1.1.3 nightly?

[beauchampy]

So, are you saying that:

* magiclantern-Nightly.2015Mar02.5D3113.zip (e8cc6e5) is buggy? (has pink frames?)
* magiclantern-Nightly.2015Feb20.5D3113.zip (36ff475) is fine? (as in, can you throw a lot of abuse at it without getting pink frames?)

If - and only if - the answer to both questions is a solid "yes", can you check this mlv_rec.mo on any of the two builds? (ML core is identical in both).

It should display "Last update: df0bbfe" when you press Q in the module dialog, and it's the only intermediate version between the two builds.

If - and only if - the modified mlv_rec.mo works fine, and you can say it with a solid "yes", can you try it (modified mlv_rec.mo) on a recent 1.1.3 nightly?

Hey Alex,

Just spent a good few hours testing. The results are in!

Test:
Record 16 clips, camera settings as follows... 50p, 1920 16:9, MLV Snd enabled and ON, GD ON, Focus Peaking ON, Histogram ON.
Rack focus during recordings to liven up the focus peaking.
Review clips in MLRawViewer at 23.976 stretched to 16:9.
Clips that have 1 or more bad frames marked as BAD.

BENCH TEST
FW: magiclantern-Nightly.2016Apr01.5D3113
BAD: 9
GOOD: 7
FAIL

Test 1:
FW: magiclantern-Nightly.2015Mar02.5D3113.zip
BAD: 7
GOOD: 9
FAIL

Test 2:
FW: magiclantern-Nightly.2015Feb20.5D3113.zip
BAD: 0
GOOD: 16
PASS

(At this point I'm now 100% convinced about my theory)

Test 3:
FW:  magiclantern-Nightly.2015Feb20.5D3113.zip w/ the intermediate mlv_rec.mo linked in previous post matching df0bbfe
BAD: 0
GOOD: 16
PASS

Test 4:
FW: magiclantern-Nightly.2016Apr01.5D3113 w/ the intermediate mlv_rec.mo linked in previous post matching df0bbfe
BAD: 0
GOOD: 16
PASS

YOU'RE THE MAN A1EX!!! IT WORKS!!! 

This intermediate mlv_rec module produces no bad frames on the latest April 1st nightly, also records past the 40gb mark.. My camera is currently rolling next to me and has hit the 11 min mark at 24p.. Well past 40gb.

Question is, what does this all mean? What's the difference in the 'intermediate' mlv_rec module VS the mlv_rec in the 2015march02 nightly?

And also where does this leave 1.2.3?

Thanks so much for your time and efforts.

[a1ex]

Interesting. The 40GB fix works in the modified mlv_rec, right? (edit: found the answer)

What I did:

- between the two builds linked above, there were two changes:

e8cc6e5 | g3gg0
mlv_rec: added DEBG chunks for embedding logs. useful to troubleshoot after a MLV had been written. one side effect: the second file will have log data from the first file after writers were stopped

df0bbfe | g3gg0
mlv_rec: major fixups for card spanning and preallocated file names

The modified mlv_rec only includes df0bbfe (the 40GB fix), but not e8cc6e5 (debug logs embedded in the MLV stream).

You uploaded two corrupted clips. I couldn't download them (I get to a page that says "Trotzdem herunterladen", and clicking it just reloads the page); can you (or someone else) convert them with mlv_dump? Do you get pink frames?