Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[arch]

Hi all,

I'm testing the latest 4/22 - Built on: 2014-04-22 12:35:00 -0700 for 1.2.3

It still has the console error, but I replaced the mlv_rec.mo with the one that g3gg0 linked. And the console error is gone (still has hacked error at 0 when recording / stopping with extra hacks turned on). It seems the mlv_rec.mo may not have been replaced with this build yet.

Thanks!

[ted ramasola]

Testing April 24 nightly on 5D2.

When recording in MLV raw,
GD ON,
the framing/border disappears and will only reappear when recording stops.

Apr 23 did not have this behavior.

edit:

I tested clearing the cf and placing new ML folders and its OK now. Erasing the old magic.cfg must have done the trick.

[gravitatemediagroup]

the last 2 or so nightly builds for 5d3 have been train wrecks    but the one before those was by far the most stable I had used yet. I'm not sure what was changed but on my end I don't think it was for the best. lol

[g3gg0]

can you please clearly say what doesnt work?

[gravitatemediagroup]

I think I figured it out, dual ISO was switched on. 

[ted ramasola]

@g3gg0,

Is it possible to have the Black fix OFF on the 5D2 by default? Its bit me once, good thing it was a test and the command line setting for mlv dump to either 2048 1024 did not fix it and instead produced greenish shadows.

[g3gg0]

Is it possible to have the Black fix OFF on the 5D2 by default?

a) yes, good point.
b) set black level to 1700: http://www.magiclantern.fm/forum/index.php?topic=11551.msg112388#msg112388 / https://bitbucket.org/hudson/magic-lantern/commits/02c9af698af933dfea31212713d5216cb0626ff8

[a1ex]

5D2 uses both 1024 and 1700.

The black level workaround should not be present in the first place in my opinion (I said it back then and I'm saying it again). It should be fixed in the backend.

[ted ramasola]

1750 was a closer fix than the 1700. Resolve does a faster fix than ACR, adding an ARRI LUT also helps.

Its very tricky with it there with default ON, it could be costly for those used to older builds without it and just decided to update, and since it says "black FIX" they might think its important to leave on.

[g3gg0]

The black level workaround should not be present in the first place in my opinion (I said it back then and I'm saying it again). It should be fixed in the backend.

and again i totally agree and want to add that it is hard to "just fix" it in the backend if one didnt write the code.
well, i can add some waits here and there and just submit it into main branch to test if that did the job.

yet i could not reproduce it on 1.1.3 with >50 videos i recorded automatically

[Audionut]

--black-fix doesn't appear to be working in the latest mlv_dump, with this sample.

I also tried outputting directly to DNG with --black-fix, and it doesn't appear to be working.
edit:  exiftool reports the black level on a processed dng, has a value of 2048, after setting 1800.

Code: [Select]

mlv_dump --black-fix=1800 --dng -o e:\test\test.dng m28-1702.mlv

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'm28-1702.mlv'
   - Setting black level to 1800
   - Convert to DNG frames
   - Output into 'e:\test\test.dng'
File m28-1702.mlv opened
File m28-1702.m00 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  0.987  0.983  1.013  0.999  1.005  1.001
Reached end of chunk 1/1 after 459 blocks
Processed 221 video frames
Done

[chmee]

..it is hard to "just fix" it in the backend if one didnt write the code..

could someone explain why this workaround/fix appeared? it seems it's makin a lot of users nervous..

[g3gg0]

could someone explain why this workaround/fix appeared? it seems it's makin a lot of users nervous..

mlv_rec/raw_rec make use of the camera-specific raw backend in ML core.
both modules use the same way of reading raw data using this backend.

the raw backend determines raw data dimensions and black level etc.
now on 1.2.3 (and probably on 1.1.3 too, but not as often as it seems) the digic internal EDMAC
gets disturbed somehow by redirecting and the optical black area used for calibration contains crap.

this crap unfortunately leads to a wrong black level estimation that only affects mlv/raw metadata, but not image content.

this black level correction is *obviously* a workaround for a problem that is hard to pin down.
but it is sufficient to repair corrupted footage - or - to apply that fix already during recording.

my hope was that people test their current module config and workflow before actually going out on a set.

[a1ex]

For an workaround, I'd say the option in mlv_dump should be enough.

With the current approach, in-camera workaround being on by default, the forced value is not correct for all models (and there are cameras that use more than one value - 5D2 for sure, maybe 50D too, since it's from the same generation). This is what Ted was trying to say.

Also, with the workaround enabled by default, the real issue gets hidden and it's easily overlooked.

[Audionut]

Rather then fix this in camera, if that is very difficult or hurts performance, couldn't you use the black level detection from cr2hdr?
The files must be post processed, so it's fine to fix the issue in post IMO.

[chmee]

Because the blacklevel is static per body (not per recording) you could also give an Option to set it manually?

[a1ex]

Rather then fix this in camera, if that is very difficult or hurts performance, couldn't you use the black level detection from cr2hdr?
The files must be post processed, so it's fine to fix the issue in post IMO.

When the raw stream is pink (corrupted), the OB area is gibberish. And, for speed reasons, OB is not recorded in raw videos, so we can't look at it in post.

Because the blacklevel is static per body  (not per recording)...

It's not.

[jimmyD30]

Is there any way to log an error that flashes across the screen when I view an MLV recording for the the first time in-camera (when it's building the .idx file)? It says something about a corrupted header I believe. The error repeats even if I delete the .idx file and view the mlv again as it recreates the .idx file.

It happens on a few different builds, Feb 16 and Apr 29 for sure on 5DM2.

[g3gg0]

Is there any way to log an error that flashes across the screen when I view an MLV recording for the the first time in-camera (when it's building the .idx file)? It says something about a corrupted header I believe. The error repeats even if I delete the .idx file and view the mlv again as it recreates the .idx file.

It happens on a few different builds, Feb 16 and Apr 29 for sure on 5DM2.

you talk about mlv_play?
tis header error is due to a buggy mlv_rec version you used to record the video.
can you try a recent build of mlv_rec, record a video and play in mlv_play ?

you can fix the footage using mlv_dump on your computer if necessary

[jimmyD30]

@g3gg0
Yes, mlv_play. So far I've used 4 builds Feb16, Apr 24, Apr 27, Apr 29 and it happens with all of them. Also, if the .idx is present (in the same folder as the .mlv file) when using MLV Converter 1.9.2 for WINDOWS conversion stops where the error occurs, if the .idx file is not present, conversion completes, but some frames are not converted (missing), about 5 for a 60 sec video.

The error seems to consistently occur at around 45 secs (give or take) when processing 60 sec videos (I've just been using 1 min long videos for my tests, so I don't know if the error would repeat for longer videos). It's never occurred with any of my 30 sec video shots and always with my 60 sec long shots.

I don't know yet if the error is dependent upon the ML settings yet, but I'll keep playing and see what's what and report back any new findings.

I would like to help get MLV working well, so if there is anything specific you want me to try toward that end, just let me know :-)

[g3gg0]

ah i remember smth.
do you use exFAT and MLV is recording files > 4GiB?

[jimmyD30]

Hey, I'm using a 5DM2, so fat32 and files capped at 4GB with auto rollover. Funny thing, even using raw_rec the error occurs at the same frame (using in camera viewer) as mlv_rec and seems to be build dependent, like the Apr 27 build always errored at frame 848 and a different build always errors at frame 835 for both raw_rec and mlv_rec.

I'm not too worried that the in-camera viewer stops playing at the frame error, but wondering if the lost frames after mlv to dng conversion is going to be a problem?

[g3gg0]

that problem will be fixed in next nightly.
was a 4GiB playback issue of mlv_play.

[ted ramasola]

jimmyD30,

While updating to new nightly builds I noticed what I thought was a bug, the way I "usually" update was just to download the nightly and copy paste only the autoexec.bin and the ml folder over to my cf card containing the previous builds, however I realized that some previous settings caused what I thought was a "bug". I deleted all the files inside the settings folder in ML after copy pasting the new nightlies and the "bug" was gone.

Maybe you should also try deleting the files in your settings folder.

[reddeercity]

Try a different Convertor like Raw2CDNGv1.4.9,v1.5.0 or sharp viewer , I use the Feb16 NB for 5d2 and no problems at all with mlv or raw in these convertors. Really no problem at all with that build
"solid"