Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 3 Guests are viewing this topic.

g3gg0

okay here is a new update to our raw_rec module for MLV writing.
can some of you with a 5D3 can test it again, please?

test procedure:
* disable all modules except raw_rec
* disable audio recoding (canon menu)
* disable global draw (ML menu)
* disable electronic level sensor (ML menu)
* configure raw_rec module:
*   Frame Skipping: Allow
*   Buffer fill method: 0, 1, 2 - can you test all of them?
*   Memory/Small hacks: On
*   Card spanning: Off, On - can you test all of them?
* Format CF card

* Start recording
* wait 10-20 seconds
* write down total write rate (3rd line)

with my setup i get ~96 MiB/s (77.5 + 18.5) with fill method 2. the others are a bit slower unfortunately.


Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Shield

Will gladly help out on testing once my 5d3 comes back from Canon (the SD card slot died!)   :'(

kgv5

All tests 1920x1080 29,97p (CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)

Couldnt find electronic level off - where is it? (so its on during the tests)

Buffer fill method: 0 (there is no 1,2,3,  - there is 0,1,2)


Best: spanning on --85,5   17,5 get continous OK ----total (after 0:20 - 100,2)
         spaning off ----92 and more but many dropped frames, after 30sec reached ~96MB but about 100 dropped frames

Buffer fill: 1

Best: spanning on --85 17,9 get expected couple thousand frames ------ total (after 0:20 - 101,9)
         spanning off ---94 , many skipped frames afer 20sec

Buffer fill: 2

Best: spanning on ---85,1 17,3 get expected couple thousand frames ------ total (after 0:20 - 103,0)
          spanning off ---95 and after 20sec started to drop frames

EDIT

Additional test (spanning ON) >>> 2,5K   2560x1090   2,35:1
Buffer fill 0 - total 111,5 (average 93,3, 18,)frame skip OFF,  recorded 1:47
Buffer fill 1 - total 112 (94,1 17,7) recorded 3:23 and card full but was continous OK,
Buffer fill 2 - total 112,6 sometimes 113 (94,5 18,2) continous OK

www.pilotmovies.pl   5D Mark III, 6D, 550D

g3gg0

thanks again, kgv5!

is the CF only write rate comparable to alex' version or is it far off?
(komputerbay cards are between 90 and 120MB/s afaik)

if i read correctly, you get 95MiB/s with CF only and 93MiB/s on CF plus 18 MiB/s SD?
if i enable spanning, my CF rate drops by a fre megabytes per second.
yours seems quite stable.

@all:
can some of the others provide results too?

hint:
chose a resolution so that you get dropped frames for sure - so you have max data to write
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Milan Skrob

My test of todays raw_rec module:
Cards: CF Komputerbay 64GB 1000x, SD Sandisk Ultra 30MB/s

Settings:
* disable all modules except raw_rec
* disable audio recoding (canon menu)
* disable global draw (ML menu)
* Frame Skipping: Allow
* Preview: Canon

span - on
1920x1080 - 25fps
buffer fill - 0
90 + 11 = 101fps

span - off
1920x1080 - 25fps
buffer fill - 0
98.5fps

-----------------
span - on
1920x1080 - 25fps
buffer fill - 1
91 + 11 = 102fps

span - off
1920x1080 - 25fps
buffer fill - 1
96.2fps

-----------------

span - on
1920x1080 - 25fps
buffer fill - 2
90 + 11 = 101fps

span - off
1920x1080 - 25fps
buffer fill - 2
97.5fps



Canon eos m

Hi g3gg0

With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):

1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s

2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF  - Touched 88.2 mb/s

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Canon eos m

Quote from: Canon eos m on August 10, 2013, 01:18:09 PM
Hi g3gg0,

With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):

Could not find the 'electronic level off'

1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s

2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF  - Touched 88.2 mb/s

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

I was so excited to see the 5DM3 touch 102 mb/s my heart beat must have been about 1000 beats/ second  ;D

One thing, I did notice the camera was warmer in hands than usual & so was the CF card.

Also, noticed the DCIM folder was empty. The root generated the following files:

M10-1619.MLV/ M10-1619.M00 to  M10-1634.MLV/ M10-1634.M00 & M10-1618.MLV

Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

kgv5

Quote from: Canon eos m on August 10, 2013, 01:20:59 PM
Also, noticed the DCIM folder was empty.

The new module writes to card (SD and CF) root folder.
www.pilotmovies.pl   5D Mark III, 6D, 550D

Canon eos m

Thanks. How do you convert to DNG from here onwards.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Toffifee

raw2dng, son of batch, rawmagic.. take your pick

Toffifee

Test equipment:
5D M3
CF - 32Gb Sandisk extreme 60Mb/s
SD - 8Gb Kingston ultimateX 100x

Preview: none (global draw off and still in magic lantern preview mode, not Canon)
Used all settings from g33g0's description above
My camera did not feel warmer than usual, which is not hot at all.
Edit: 1920 x 1080p 24fps ofc

Buffer fill method 0:

Spanning ON:

#1 - 56 + 14  = 70 mb/s
#2 - 55.8 + 14.1 = 69.9 mb/s
#3 - 56.2 +14 = 70.3 mb/s

Spanning OFF:

#1 - 59.7 mb/s
#2 - 58.7 mb/s
#3 - 58.8 mb/s

Buffer fill method 1:

Spanning ON:

#1 - 56.6 + 14.2 = 70.8 mb/s
#2 - 54.5 + 14.3 = 68.8 mb/s
#3 - 53.2 + 14.2 = 67.4 mb/s

Spanning OFF:

#1 - 61.4 mb/s
#2 - 61.7 mb/s
#3 - 61.9 mb/s

Buffer fill method 2:

Spanning ON:

#1 - 57.6 + 14.2 = 71.8 mb/s
#2 - 57.9 + 14.2 = 72.1 mb/s
#3 - 57.2 + 14.3 = 71.5 mb/s

Spanning OFF:

#1 - 61.9 mb/s
#2 - 61.8 mb/s
#3 - 60.2 mb/s



RenatoPhoto

CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB
1920x1080 at 32 fps for frame skiping

There is a lot of variability so I set up for three consecutive test in the following sequence:

TEST 1.
Format CF  Buff=0  and spanning OFF -- Run test
Format CF  Buff=0  and spanning ON -- Run test
Format CF  Buff=1  and spanning OFF -- Run test
Format CF  Buff=1  and spanning ON -- Run test
Format CF  Buff=2  and spanning OFF -- Run test
Format CF  Buff=2  and spanning ON -- Run test
TEST 2
Same as above
TEST 3
Same as above

TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.

Buff=0  - spanning = OFF
test 1 = 91.9-93.3
test 2 = 90.3-91.6
test 3 = 92.1-93.4
Average Min -Max= 91.4 - 92.7

Buff=0 - spanning = ON
test 1 = 100-100.8
test 2 = 97.1-99.6
test 3 = 98-100.3
Average Min -Max= 98.3-100.2

Buff=1  - spanning = OFF
test 1 =90 - 91.5
test 2 = 90 - 91.8
test 3 = 89.7 - 91.7
Average Min -Max= 89.9-91.7

Buff=1 - spanning = ON
test 1 = 99.7 - 100.9
test 2 = 101 - 101.8
test 3 = 97.2 - 100.3
Average Min -Max= 99.3-101

Buff=2  - spanning = OFF
test 1 =92 - 93
test 2 = 89.2 - 92.3
test 3 = 91.3 - 92.1
Average Min -Max= 90.8-92.5

Buff=2 - spanning = ON
test 1 = 100.6 - 100.7
test 2 = 101.3 - 101.9
test 3 = 101.1 - 101.9
Average Min -Max= 101-101.5

My conclusions. 
With spanning the clear winner is Buff 2
Without spanning  maybe buff 0 or buff 2
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

kgv5

I checked again buffer fill method = 2, this time with RenatoPhoto way:
(CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)

For 1920x1080 32fps

Buff=2 - spanning = ON
test 1 = 100.8 - 101.1
test 2 = 101 - 101.2
test 3 = 100.9 - 101.3

What is interesting for 29.97 fps i get better write speed:

Buff=2 - spanning = ON
test 1 = 102.2 - 103.3
test 2 = 102.7 - 103.5
test 3 = 101.9 - 102.8

EDIT
The highest speed so far i get with 2880x1226 24fps (with frameskip ALLOW) total 114,9  (95,8, 18,5), this was after ~ 1 minute recording, ~400 skipped frames.

Seems like bigger frames with lower frame rate gives better performance.
EDIT2:
Checked 2880x1206 (2,39:1) but 20fps - recorded 45 seconds before frame skipped, max total write speed 116,3 (97,5, 18,6)

EDIT3: ok, ultimate test: 3584x1320 (max resolution) - for 15fps - normally 119,2-119,6, touched 120,3 MB/s  121,7 MB/s :o
https://www.dropbox.com/s/c3118snkq0uq5ny/Spanning%203-5K%2015%20fps.jpg
Recorded about 50 sec.
I am starting to worry about my cards  :D

EDIT 4: Recorded 45 sec of 2240x1260 (16:9) 24fps at 112,5-113 total.
www.pilotmovies.pl   5D Mark III, 6D, 550D

Canon eos m

Quote from: Toffifee on August 10, 2013, 02:46:32 PM
raw2dng, son of batch, rawmagic.. take your pick

Sorry for being dumb. Cannot convert the MLV or M00 files to dng with raw2magic. Could please explain how to do that?
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Canon eos m

Quote from: g3gg0 on July 28, 2013, 04:58:21 PM
Update: (see first post)

The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)

see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.

if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built ones. (5D3)
use it on your own risk.


if you can show the mlv_dump logs of your cameras, it would help me a bit.

Found this. Is there a binary for this code yet?
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

g3gg0

@all:
thanks for all the reports!

@Canon eos m:
not for windows, just a linux build.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

morphers

I can't seem to figure out how to record at 2.5K It says it is not possible?

jc

Quote from: Toffifee on August 09, 2013, 06:05:57 PM
There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.

A)My request was that the format allows separate audio length to video length, not the 5 sec feature it's self
B)My second point was that using a pre-established serialisation format would ease things greatly in the future. Assuming the needed number of float or int to string conversions per second is not beyond what the processor can do ontop of the copying work etc.. Just using json would even work just fine.. :-) J

g3gg0

here is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

xNiNELiVES

Quote from: g3gg0 on August 11, 2013, 12:58:06 AM
here is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?

So is this available with other cameras such as the 5D2?

g3gg0

updated the DL link of mlv_dump.exe and fixed a memory access error that triggers on bit depth change

5D2: not yet, didnt look for the new symbols yet.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

arrinkiiii


Canon eos m

Cannot figure out the command chain:

C:\
C:\ cd desktop
C:\desktop\cd (where the mlv_dump.exe and *.mlv files are saved)
C:\desktop\director with mlv_dump\mlv_dump.exe m10-1619.mlv

Nothing happens - what do I type in the last line?
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

gnarr

Quote from: g3gg0 on August 11, 2013, 02:04:14 AM
5D2: autoexec  symbol

can you try it?

What raw_rec.mo should we use with this? I just tried the one from post #200 in this thread and it freezes the camera. I am currently using the 2013/08/09 build from the 5D2 raw thread.

Here you can see how it freezes:
https://plus.google.com/117039326451584292689/posts/dYB7vPW2FsR