Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[mixmastermike]

Card spanning is working 1920x1038 30fps is shooting continuous
Settings: status when recording NONE/Memory Hack ON/Extra hack ON/Buffer Fill 3/CF only 1
Thnx g3gg0!

PS: status when recording DEBUG gives me a around 1300 frames with above settings

Please can you let me know what card and camera you are using. thnx

[tjaja]

Lexar 32GB 1000x and Sandisk Extreme 45MB/s. I also perform card warmup btw. the resolution is  1920x1038 not 1080p.
Try recording with both hacks and no buffer or debug status. it is faster. Don't know if that's some quantum mechanics voodoo trying to go gangster on my 5d, or that the buffer/debug status is making use of the SD card.

[mixmastermike]

Thanks for the the heads up on the 1038, I have 2 CF cards, a Lexar 128Gb 1000X and a Lexar 1066X 64Gb my SD card is a Sandisk Extreme Pro 95MB/s 64GB, Im testing them today.

When I run a quick benchmark on the CF and SD I get:

Lexar 1000X 128 GB

SD 19.8 MB/s
CF 77.5 MB/s

Lexar 1066X  64 GB

SD 19.8 MB/s
CF 76.4 MB/s

My first question is are these results believable? If so that would lead me to believe 96-98 MB/s is a kind of maximum data write achievable even when CF and SD cards are Combined, have we reached maximum bandwidth?

If anyone has achieved higher with earlier builds, Im aware lots has changed recently, this would really interest me and lead me to explore buffer settings etc a bit more

When I use RAW_REC I can, with small hacks on, record 1920 x 1038 30p continuously  at a rate of 99.6-99.8 MB/s with the camera icon green continuously with the Lexar 64Gb 1066x card

With the benchmarks on Markus's Komputerbay 1200x CF much higher than my 1066x Lexar I would think we're close to 1920 x 1080 30p using RAW_REC. That is unless the 100MB/s is a ceiling so to speak.

[tjaja]

I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.

Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.

[vyskocil]

I have run some tests with the last nightly build for the 7D and wend i preview the files in camera and return to Lv the box have move down. Need to turn off and on the camera for return to the right frame in Lv. Im recording at 2.35:1.

Same here, the box move down after preview. I used 16:9 ratio (7D).

[mixmastermike]

I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.

Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.

Agreed on buffers, something doesn't seem to be functioning correctly, the code isn't maximizing the potential bandwidth for whatever reason, like yourself I am no programmer but Im happy to test.

[g3gg0]

I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.

Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.

the buffer details are not printed as fast as the buffer is really filled.
this could explain that behavior.

[noix222]

Card spanning is working 1920x1038 30fps is shooting continuous
Settings: status when recording NONE/Memory Hack ON/Extra hack ON/Buffer Fill 3/CF only 1
Thnx g3gg0!

Lexar 32GB 1000x and Sandisk Extreme 45MB/s. I also perform card warmup btw. the resolution is  1920x1038 not 1080p.
Try recording with both hacks and no buffer or debug status. it is faster.

I did some testing today on a 5Dmk3 using those settings. I am running on a Komputerbay 64gb 1000x and Sandisk Extreme 16gb 45mb/s but i can't get continuous 1920x1038 30fps using mlv_rec.. even though using raw_rec i get continuous 1920x1038 30fps with buffer on 99.6mb/s ... using mlv debug it shows a constant 17MB out of SD but a variable bit rate from CF, higher in the beginning, from 90mb/s to 60mb/s.... sometimes lower.... recording around 30 to 40 sec.

[arrinkiiii]

Make some more test with the last ML Nightly for the 7D and the bug still there.  Wend you record with mlv_rec at 2.35:1 and then you preview the clips in camera with mlv_play and you return to the Lv the black screen for the 2.35:1 stay in the same position that used in the mlv_play and that is different wend you are in Lv ready for record.

For fix this is need to turn the camera off and on. It's not possible to fix this?

[mixmastermike]

Did a quick test today with 1920 x 1080 30p with card spanning on,  managed 15 seconds of footage thats 5 seconds more than yesterday.

With 1920 x 1038 30p I got 25 seconds. much improved., I will play with buffer fill methods later today.

I repeated this with MLV_SND and got 23 Seconds

MLV_SND, Im not sure if this is working the audio meters are not registering any levels when the module is turned on, It did with yesterday build. seems to be working again, ok so temperamental. Ill try to find out what caused it to be on/off.

One question to anyone out there who has experience with card spanning, do you have to set the Canon menu to record to both cards or does ML override that setting and as such it isn't a concern? I currently have the Canon menu set to record to both cards and files are being written to both cards.

[g3gg0]

you dont have to select anything in canon menu.
just make sure your primary card is CF

[mixmastermike]

Thanks!

[sergiocamara93]

Hi there! I'm testing the nightly from the 15th of February in a 5D Mark III and I've had some issues with audio. I have the default MLV settings (2.35 aspect ratio) + MLV_Sound + Global Draw on while recording. The dngs are all fine, but some of the wavs stop in the middle of the recording and others are simply empty (it happens especially in the 3x crop recordings, but those are understandable). I've search around the forum to see if this was a known issue but I didn't find anything. Am I missing something?

Anyway, thanks for the great job. The new format and the audio integration are unbelievably amazing

PS: I tried the 23rd nightly but I had mayor issues with audio stopping and black frames in the middle of the recording (?), so I decided to try the version from the first post of this thread, which I've found to be much more stable and reliable.

[ted ramasola]

5D mkII Feb 24 tests report.

1. Low FPS freeze bug is FIXED

2. Black border bug when initially turned on is still there. It is off centered and one way to correct is by cycling through magnify button.

3. In 3x crop mode, some borders and framing is not accurate.
   ex. 1728 x972 16:9 portion of top and bottom cut off, 16:9 is more like 2:1, recorded image is not affected.

4. tested continuous recording resolutions, tested at least past 10 minutes.
   
These resolutions are based on lexar and komputerbay 32 and 64 gig 1000X cards , komputer bay 128 gig 1050x will have slightly lower performance.

  1X
  1872 x 850
  1856 x 928
  1792 x 968
  1728 x 972

  3X
  2048 x 872
  1920 x 872
  1856 x 844
  1728 x 972

[eyeland]

Trying to absorb the information in this thread, I am still unsure if less frames when recording mlv as opposed to raw is to be expected or not.
In raw, I can do 48fps in 1920x672 continuous on my Komputerbay 1000x 64. With mlv, I get around 10-15 seconds depending on other settings before the first frame drop. This is without card spanning. Te mlv format is quite the upgrade imo. I don't really need sound, but the meta data surely is useful! Thanks

[kgv5]

-

[mixmastermike]

MLV_REC and MLV_SND working on full HD 24p with 25th build and card spanning on. 30p not quite there yet drops after 10-15 seconds

CF and SD Benchmarks are up 5 MB/s for the CF card, looks like the buffers are smoothing out, SD - 18.9MB/s CF - 84.9MB/s

I'll check the footage and sound later today to see if there are any errors

[noix222]

TESTING build magiclantern-v2.3.NEXT.2014Feb26.5D3113.

5d mark 3 - ALL TESTS DONE in 30fps WITH a CF Komputerbay 64gb 1000x and SD Sandisk Extreme 45mb/s.

MLV_REC settings: status recording Debug _ memory hack ON _ extra hacks ON
                             buffer fill method 4 _ cf-only buffers 3 _ card spanning ON _ reserve card space ON

             30fps    1920x1080    16:9

      1x mode_ recorded 30 sec. _ MAX rate 99.3mb/s _ CF +- 80mb/s _ SD always on 17mb/s
      5x mode_ recorded 50 sec. it always skipped a frame on 50 sec and stopped _ MAX rate 101.5mb/s _ CF +- 87mb/s _ SD always on 17mb/s

             30fps    1920x1038    1:85.1

      1x mode_ recorded 50 sec. it always skipped a frame on 50 sec and stopped _ MAX rate 98mb/s _ CF +- 80mb/s _ SD always on 17mb/s
      5x mode_ recorded continuous _ MAX rate 101mb/s _ CF +- 87mb/s _ SD always on 17mb/s

So besides CF been inconstant the only problem i can report is this 50sec stop in both resolutions.. they're kind at the same time and both stops even though little icon is green showing continuous recording... it doesn't show "its a bug report" it only skip a frame and stop...

RAW_REC settings: memory hack ON _ small hacks ON

              30fps    1920x1080     16:9

      1x mode_ recorded 40 sec. _ MAX rate 99.3mb/s
      5x mode_ recorded 1min. _ MAX rate 101mb/s

              30fps    1920x1038     1:85.1

       1x mode_ recorded continuous. _ MAX rate 99.6mb/s
       5x mode_ recorded continuous. _ MAX rate 101mb/s


So it's clearly that the buffer works a bit faster recording in 5x mode. Running with both modules raw_rec & mlv_rec...
CF buffer needs to be a bit more constant in MLV_REC to be able to record continuous 1920x1080 30fps in both modes 1x and 5x...

I also processed a lot of the images i tested today from both modes mlv and raw... could not find any error or pink frame... it's all good to me !

thanks for the work guys amazing.

-- g3gg0 any advice on buffer config? thanks


                                                                             

[andy kh]

can anyone advice me which consumer camera should i buy for recording mlv raw? i got 550D. it can shoots 1152 X 432 continous which is very small resolution so i m planing to upgrade

[kgv5]

can anyone advice me which consumer camera should i buy for recording mlv raw? i got 550D. it can shoots 1152 X 432 continous which is very small resolution so i m planing to upgrade

It depends on how much you want to spend...

[andy kh]

It depends on how much you want to spend...

i think i should go for 650D since it can record 720P. thanks

[dyfid]

Has there been any discussion about writing Magic Lantern DNG metadata in iXML format into the MLV wav file (sidecar) for something like Blackmagic Resolve's extensive metadata use?

I've so far tested a 3 line batch script which dumps the contents of the MLV's into named folders and then uses the open source CLI utility BWFMetaEdit, found here: http://sourceforge.net/projects/bwfmetaedit/files/binary/bwfmetaedit/1.3.0/ to write the contents of an iXML file into the .wav acting as audio + sidecar.

For %%a in (*.MLV) do mkdir "%%~na"
For %%a in (*.MLV) do mlv_dump.exe -f 10 --dng --cs3x3 -o "./%%~na/%%~na_" %%a
For %%a in (*.MLV) do bwfmetaedit.exe "./%%~na/%%~na_.wav" --in-iXML="my_iXML.xml"

Contents of iXML file.

<?xml version="1.0" encoding="UTF-8"?><BWFXML><IXML_VERSION>1.5</IXML_VERSION><PROJECT>Test</PROJECT><NOTE>Blackmagic Metadata Write Test</NOTE><BLACKMAGIC-KEYWORDS>magic,lantern,test</BLACKMAGIC-KEYWORDS</BWFXML>

There are a lot of camera related metadata options for the the usual, like ISO, WB, Shutter, Camera Model, Firmware Version as well as more project related tags. Which I could work through testing the tag names and writing the data to wav.

So is it something a dev would consider implementing DNG EXIF into wav sidecar? This would also open up anyone wanting to develop a metadata entry GUI in camera to write maybe project and multicam type data via ML?

If not then I suppose there's someone writing a MLV extractor which could do the extract EXIF data with EXIFTool, write to XML file, write to wav and so on.

It's then simple in Resolve to Import Media by   "Add Folders & Sub Folders To Media Pool (Create Bins)" to give each MLV folder a separate bin containing the DNG sequence and wav, then choose the "Auto Sync Audio Based On Timecode", to link the wav to the DNG sequence resulting in wav metadata sidecar contents being displayed in the Resolve Metadata Interface for any DNG or wav selected.

[noix222]

TESTING TODAY BUILD magiclantern-v2.3.NEXT.2014Mar02.5D3113

5d mark 3 - ALL TESTS DONE in 30fps WITH a CF Komputerbay 64gb 1000x and SD Sandisk Extreme 45mb/s.

MLV_REC settings: status recording Debug _ memory hack ON _ extra hacks ON
                             buffer fill method 4 _ cf-only buffers 3 _ card spanning ON _ reserve card space ON

Good news i was able to continuous record 1920x1080 30fps in 5x mode... the weird stuff is that in 1x mode the buffer performance changes a lot.. it goes down to 98mb/s max...when in 5x mode it goes around 102-103mb/s... don't know why but i guess it's close... thanks guys

[mixmastermike]

This is something that has confused me greatly, I don't get why the crop mode will do full HD at 30p and 1x mode will not. I'd love to understand why the buffers operate differently under the two modes.

Developers, can you possibly shed some light on this?

[larry 777]

Using nightly march 2nd /14 , 5d3, Mlv , 1920 x 1080 24 fps,
GD allow , Status when recording Debug ,  files> 4GiB (exfat),
card warm up 128 , memory hack on , buffer fill method 4,
cf buffers 3 , card spanning on , reverse card space  on.
Shooting continuous with sound no problem extract perfect with
MLV Mystic. When trying FPS overide loosing sound and the camera bugs completely.
Impossible to go back to the numbers like MLV 1920x1080 continuous but it writes
in yellow now 82.9 MB/s at 23.976p expect around 0 frames at 79.8 MB/s.
( CF card transcend 1000x 64gigs) . Switched off the camera and turn on again same message
exept this time was abble too shoot again continuous but camera erraticly.Take battery off
everythnig back in place. touching the FPS overide function seems to jam every thing .