Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[xNiNELiVES]

I saw benchmarks of the 5D Mark II reaching write speeds slightly over 90 Megabyte/s, just like the 50D.
And the 5D Mark III even reached 130 Megabyte/s. Its SD-Card controller's hardware limit is 21 Megabyte/s.

Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.

[RenatoPhoto]

New test with build 54a545720 TODAY from g3gg0 repository

CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB

TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.

For CF recording only @  1920x1080 at 29 fps for frame skipping
Tested all CF 1 through 9 with buffer fill method =  4

CF=9 
test 1 = 98.5-100.7   
test 2 = 98-101.5
test 3 = 99.1-100.3
Average Min -Max= 98.5-100.8

CF=8
test 1 = 99.101.6
test 2 = 100.1-99.1
test 3 = 102.9-102.4
Average Min -Max= 100.7-101

CF=7
test 1 = 100.1-101.9
test 2 = 100.9-102.1
test 3 = 101.6-101.3
Average Min -Max= 100.9-101.8

CF=6
test 1 = 101.9-102.7
test 2 = 99.5-101.6
test 3 = 97.7-100
Average Min -Max= 99.7-101.4

CF=5 
test 1 = 102.5-103
test 2 = 102.1-102.9
test 3 = 101.8-100.5
Average Min -Max= 102.1-102.1


CF=4
test 1 = 97.4-98.9
test 2 = 101.8-102
test 3 = 100.6-102.1
Average Min -Max= 99-101


CF=3
test 1 = 97.4-98.9
test 2 = 101.8-102 Cont rec ok for a little while
test 3 = 100.6-102.1
Average Min -Max= 99.9-101

CF=2
test 1 = 97.5-100.8
test 2 = 102.9-103-100 Cont rec ok for a little while
test 3 = 100.5-101.8
Average Min -Max= 100.3-101.9

CF=1
test 1 = 95.3-98.9
test 2 = 100.8-101
test 3 = 100-101.1
Average Min -Max=98.7-100.3


CF=0
test 1 = 95.9-100.5
test 2 = 98.2-100
test 3 = 102.3-102.6
Average Min -Max=98.8-101.03

My conclusions. For CF only seem like buffer  4,5,6,7,8 are the better with 5 maybe better than the others.  I repeated some of the  test and repeatability is not very good.
--------------------------

Next test Spaning On.  Too many options. 

zoom= 3x
Resolutions:  2048x1152  at 29.97

CF=0 B=4
test 1 = 113-116.2
test 2 = 114-116.2
test 3 = 116.1-116.2
Average Min -Max=114.4-116.2

CF=3 B=4
test 1 = 112.3-114
test 2 = 116-117.1 Cont rec ok for a little while
test 3 = 116.1-119.7 Cont rec ok for a little while - touched 120!!
Average Min -Max=114.8-116.9

CF=2 B=4
test 1 = 114.9-115.9
test 2 = 116.2-117.1 Cont rec ok for a little while
test 3 = 117-118.4 Cont rec ok for a little while - touched 123!!
Average Min -Max=116-117.1

CF=1 B=4
test 1 = 117-115
test 2 = 113-117.6
test 3 = 115.2-116.9
Average Min -Max=115.1-116.5

CF=4 B=4
test 1 = 116.7-117.7 Cont rec ok for a little while
test 2 = 116.8-117  Cont rec ok for a little while
test 3 = 118.1-118.3 Cont rec ok for a little while
Average Min -Max=117.2-117.6

Note:  At CF=5 and above the SD card does not initialize fast enough so recording stops quickly.

Best is CF=4 B=4

Congrats g3ggo.  Amazing, blazing, speeding machine!

[CFP]

Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.

The 5D Mark III has the same SD-Controller like the 600D. Why do you think the 5D Mark II should have a different CF-Card than the 50D?
Canon uses the same components for multiple cameras to keep the prices low.

[g3gg0]

@g3gg0 what about a version freeze? i would begin to code a converter next days, but the things are changing almost daily. after this version freeze or a first reference mlv-file (with version-number) i could start..

regards chmee

the file format was fixed before i ever started a line of code.
it didnt change since then, except two fields.

so: the converters even from the first day would have been able to read this format the raw_rec outputs today

[chmee]

ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.

regards chmee

[g3gg0]

ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.

regards chmee

what do you want to talk about mlv?
had the proposal two weeks in this forum and there was no discussion
just a few feature requests.

[chmee]

i just propose to split the spanning-mode optimization from this thread - finally i will code and test when im back at home, and then i can ponder about the (pro's and hopefully not con's) of the fileformat.

regards chmee

[g3gg0]

@RenatoPhoto:

i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes

[RenatoPhoto]

@RenatoPhoto:

i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes

no, thank you!!

Just wondering if it would be a better test to turn off skipping frames.  Running each mode until the recording stops, running it again maybe four or five times and then obtaining the total GB recorded.  I suppose the best setup is the one that can record the most without skipping frames, so this test should be more applicable to the needs.  In most cases, if the recording speed is stable, then the fastest recording will give you the longest file.  The problem is that recording speed is not constant so maybe total GB of multiple recordings may be a more reliable number to find the best settings.

[CoresNZ]

so your raw is in E:\ ?

Yes. I have an E drive with a folder caller rawFiles in it and I have dumped all the recorded files from both cards into it along with the mlv_dump.exe.

[1%]

Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.

50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.

[g3gg0]

@RenatoPhoto:
here is a version with "test mode" that tries recording with iterating buffer count and method.
every test case is ran 4 times.
enable tracing and test mode from menu. spanning option is not changed during tests.

with my setup the frame count varies more than the buffering method option changes.
does it look better on your camera?

[RenatoPhoto]

Where is tracing?
edit: you mean Debug trace?

I am running it  and see:
test numb with variables M , B, W, F, and (speed)

What do they mean..

When does it stop?

My comments so far:
* Each test seems very short and only represents the beginning of recording.   In my recordings (not in this test) I noticed that some buffers and CF numbers begin recording fast and then slow down while other options begin slow and increase the recording speed over time.

Is there a log saved?

Looks good

[gnarr]

Here is my first version of mlv2dng. It's not finished yet and there is still a lot to be done.

What is not working:
Spanned files (could someone please supply me with some spanned files? I don't have a 5D3 to try it).
Preview image. (it shows random noise)
Dual ISO.
It only shows very limited amount of EXIF data, as I have still to do mappings from MLV to exif tags.
probably much more.

I have a lot of cleanup to do on the code before I submit it online. Will probably be online tomorrow or the day after.

Usage is the same as with raw2dng for the moment:
mlv2dng.exe file.mlv [prefix]
=> will create prefix000000.dng, prefix000001.dng and so on.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip

[xNiNELiVES]

50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.

Yeah I'm just saying that if anything the 5D2 should be just as fast, if not faster when it comes to write speed. Am I right?

[Canon eos m]

1)
2)
3) Even if I stop recording, the buffer status never disappeared. The rest of the info disappeared after a good 15 seconds or so.
4) I don't enable it but what happens if someone did? I realize this is only an experimental copy of the new MLV Format, I guess you'll just disable it in the final versions.

Will trace the bug with the lag spike.

No bugs for me.

[g3gg0]

@gnarr:

thanks for supporting!
(it is crashing with no output on my pc)

things important to know (most of that is written already i think)
 - files are numbered in the file itself
 - there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
 - no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
 - all M00, M01, ... also contain a file header (mlv_file_hdr_t)
 -> you see no difference in files from card spanning and files that went over 4GiB
 - the initial blocks like IDNT, INFO etc are written in the first file before recording
 - some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
 -> so dont rely on exact positions in file
 - blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
 - look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
 -> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

will upload a file, takes some time.
it will appear here, when its done.

[g3gg0]

(upload finished)

[gnarr]

@gnarr:

thanks for supporting!
(it is crashing with no output on my pc)

things important to know (most of that is written already i think)
 - files are numbered in the file itself
 - there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
 - no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
 - all M00, M01, ... also contain a file header (mlv_file_hdr_t)
 -> you see no difference in files from card spanning and files that went over 4GiB
 - the initial blocks like IDNT, INFO etc are written in the first file before recording
 - some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
 -> so dont rely on exact positions in file
 - blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
 - look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
 -> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

will upload a file, takes some time.
it will appear here, when its done.

Thanks for the notes and the files I will test this when I get home frome work.

[gnarr]

one question for you g3gg0.

Frame size in the raw info header shows 4797520 for this video, because the camera was configured for 2080*1318, but of course you can only record max 1920*1288, so the real real frame size is 4327680 (as reported by the video frame header). I was wondering if this was a bug or if it is supposed to be like this?

[Stedda]

g3gg0 and chmee you guys rock!

[Danne]

Awesomeness x 10

[gnarr]

new version.

Now handles spanned files and hopefully won't crash

With spanned files, you just point to the first file and the program automatically detects if there are more files.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip

[gnarr]

-> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

I have still to do this. Currently it just writes metadata from the last info block that was read. But the whole metadata thing is largely unfinished yet anyways

[Andy600]

gnarr - It's working here. Exif is showing correctly.