Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[g3gg0]

the new .mlv approach supports both concepts.
implemented is none of them yet

currently it will break after the 4G were reached and continue a new file.
the blocks might be corrupted in this case.

i prefer to check before recording, if >4GiB files are supported and start a new file in case.

every file gets is own header with a unique ID so you dont have to pick the files by name, but
the post processing tools can detect the sequence number and which video it belongs to - even if the filenames are changed or corrupted.
(e.g. due to copying it with copy stations)

[jc]

Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?

Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?

Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!
Loving the new format proposal! James

[dadinio13]

mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing

[RenatoPhoto]

mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing

Can someone please provide and exe version for testing in windows?

[dadinio13]

MLV converted to RAW with mlv_dump and RAW to DNG with raw2dng (one frame) :
https://docs.google.com/file/d/0B-tM9Z6JauKNSktDTGgxQktnM2M/edit?usp=sharing

ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?

[ilguercio]

Can you keep the first post updated with downloads?

[g3gg0]

Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?

this implies your request

- audio support

Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?

this format already is extensible and is designed to be extensible

some technical facts:
 - structured format
 - extensible layout
 - as a consequence, we can start with the minimal subset (file header, raw info and then video frames)

Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!

the first data field in the file is a version string.

sure you had a look at it?
but thanks for the toughts.

[g3gg0]

ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?

the converter into the legacy format converts into the legacy format
this means, that you dont have all the nifty features of the new format.
so you will be able to use the new format with all existing tools.

yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.
or maybe someone else is willing to do that? combine mlv_dump with raw2dng to get a mlv2dng.

but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10

some warning:
the new version will not only read MLV, but also read the M00, M01 etc when available and merge that into the output.
but i realized that it has a >4GiB bug when converting to legacy format.
will fix that tonight.

[dadinio13]

but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10

Oh yeah! thank you for that

[John Kesl]

I just don't know why you're arguing with me over this... 

There is a reason things work they way they do.  I'm not trying to annoy the developer and I think what he has been able to get out this camera to do is insane!

I'm not special at all, but I'm willing to offer my experience to the developers to help them shape this into something even more awesome.

Honestly, I could be like every other professional and say "psshhh... shoot with the F55", "Yea, but it'll make your camera explode"...

I even went as far as to test this on a paid, 5 day job... had it failed, I could have been sued.  But it worked perfectly, and I've been singing it's praises ever since!

It's great!  Amazing!  I can't believe this quality is THIS accessible for SO CHEAP!  But... the display covers actors faces, and sometimes the images look like vomit before you CC then in Resolve.  I think those issues need to be solved before spanning the data across 2 cards.

Why is that such a threat to you?

You went on a job with no backup? sheesh.

[g3gg0]

can you stop that, please?

[AnotherDave]

You're  a professional who went on a job with no backup? Even saying that proves you aren't.

I didn't say I didn't have a backup if the camera failed... but you don't know if there are problems with the footage until it has been downloaded, converted, and loaded into resolve.  So... you can't check it until the end of the day.

Again, what is with the attitude?  I'm only asking/hoping for features to solve some issues we had.  These would benefit you too... even if you don't understand why.

[g3gg0]

can you stop that, please?

last warning.

[AnotherDave]

So with mlv_dump, we're essentially just (as the name implies) dumping the additional information V2 is recording?

Is there a way to incorporate it back into the DNGs?  Will that come with mlv2dng, or will mlv2dng also remove the information?

Thanks.

[a1ex]

Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.

[AnotherDave]

Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.

No.  I'm asking because I want to test it.  As a programmer, I'm useless... unless you want me to break a computer.  But, I follow instructions well and like to test things...

Wondering when I can get my hands on some metadata! :-)

[g3gg0]

@alex:
haha. i am sure your sarcasm will be misunderstood.
waiting for a: "why are you doing it then?" a few posts later.

@AnotherDave:
let me cite myself:

yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.

[g3gg0]

with the latest commits i added these features to mlv_dump:

 - compress frames using LZMA
 - decompress frames
 - convert bit depth (from 14 bits to anything less than 14 bits per pixel and back)

todo:
 - apply e.g. a log curve instead of just clipping
 - an intelligent reconstruction when upsamping bit depth


if both features are combined you get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.

this feature is better for archiving your footage.
converting back to e.g. legacy raw doesnt need any parameters - it will decompress and convert transparently without any additional parameter.

the LZMA support is tricky and not out-of-the-box yet.
download the latest 7z SDK (v9.2.2) and compile a static version of it using this makefile:

Code: [Select]

PROG = lib7z.a
CXX = g++
LIB =
RM = rm -f
CFLAGS = -c -O2 -Wall -c -D_7ZIP_ST

OBJS = 7zAlloc.o 7zBuf.o 7zBuf2.o 7zCrc.o 7zCrcOpt.o 7zDec.o 7zFile.o 7zIn.o 7zStream.o Alloc.o Bcj2.o Bra.o Bra86.o BraIA64.o CpuArch.o Delta.o  LzFind.o Lzma2Dec.o Lzma2Enc.o Lzma86Dec.o Lzma86Enc.o LzmaDec.o LzmaEnc.o LzmaLib.o Ppmd7.o Ppmd7Dec.o Ppmd7Enc.o Sha256.o Xz.o XzCrc64.o

all: $(PROG)

$(PROG): $(OBJS)
ar -q $(PROG) $(OBJS)

%.o: ../../%.c
$(CXX) $(CFLAGS) $<

clean:
-$(RM) $(PROG) $(OBJS)

then add the created lib7z.a in the raw_rec makefile

Code: [Select]

- $(call build,GCC,gcc mlv_dump.c -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) $(LFS_FLAGS))
+ $(call build,GCC,gcc mlv_dump.c lzma922/C/Util/7z/lib7z.a -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) -Ilzma922/C/ $(LFS_FLAGS))

then uncomment the #define MLV_USE_LZMA in mlv_dump.c and recompile using 'make mlv_dump'


examples:

Code: [Select]

# show mlv content (verbose)
./mlv_dump -v in.mlv

# will dump the first 123 frames into a new file
./mlv_dump -f 123 -o out.mlv in.mlv

# compress input file
./mlv_dump -c -o out.mlv in.mlv

# compress input file with maximum compression level 9
./mlv_dump -c -l 9 -o out.mlv in.mlv

# decompress input file
./mlv_dump -d -o out.mlv in.mlv

# convert to 10 bit per pixel
./mlv_dump -b 10 -o out.mlv in.mlv

# convert to 8 bit per pixel and compress
./mlv_dump -c -b 14 -o out.mlv in.mlv

# create legacy raw, decompress and convert to 14 bits if needed
./mlv_dump -r -o out.raw in.mlv


[RenatoPhoto]

Ran some test with provided build ccec3eeb -aug 3rd
5D3
CF is Lexar Pro 1000x 32 GB
SD is Sandisk Extreme 45 MB/s 32 GB

Preview:Hacked, Memory hack: ON, Small hack: ON, Global Draw: OFF

1920x1080 at 29.27   CF at 87 to 90 and SD at 18 with 3% idle  Continuous OK
1920x1080 at 31 fps   CF at 88 to 90 and SD at 18 with 1% idle Continuous OK
1920x1080 at 32 fps   CF at 72 to 92 and SD at 18 with 132ms idle Continuous OK (sometimes)
1920x1080 at 33 fps   CF at 87-91.2-90 and SD at 18 with 1% idle to finally Skipped frames

1728x606 at 50p fps CF 78 to 81.1 and SD 16.6 at 40% idle  Continuous OK
1856x650 at 50p fps CF 80 to 82.2  and SD 16.6 at 100ms idle  Continuous OK
1920x672 at 50p fps CF 72 to 83  and SD 16.6 at 100ms idle  Skipped

1728x606 st 59.948  CF 64 to 74  and SD 16.3 at 100ms idle  Skipped

SPANNING OFF

1920x1080 at 29.27   CF at 82 to 91.7 Skipped

1728x606 at 50p fps CF 78 to 86.4 Skipped

This is very, very nice and practical way of reaching some new benchmarks.  Also it makes good use of available space on both cards!!

Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further. 

I am finally working on a Virtualbox after struggling for a long time with Cygwin.

I am not a programmer so do not know how to make the mlv_dump work!

I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o  Error1

Congrats G3ggo Excellent work!!!

Update:  Just read your new stuff...  Amazing stuff!

[g3gg0]

thanks for your tests!
do you achieve same or lower rates compared to the old raw_rec version that alex made?

Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further. 

not necessary anymore with current mlv_dump

I am finally working on a Virtualbox after struggling for a long time with Cygwin.

good choice imho

I am not a programmer so do not know how to make the mlv_dump work!

I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o  Error1

basically you have to:
cd modules/raw_rec
make mlv_dump

then it should be done.

about the compile error - can you send all messages you get when compiling the module?

[RenatoPhoto]

do you achieve same or lower rates compared to the old raw_rec version that alex made?

Will run some test with spannig off to compare.  I can compile the new modules except the ime_base, so I will use the complete compile that you provided instead of todays.

about the compile error - can you send all messages you get when compiling the module?

Code: [Select]

magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=ime_base    -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here
ime_base.c: In function 'ime_base_start':
ime_base.c:46:5: warning: incompatible implicit declaration of built-in function 'strncpy' [enabled by default]
ime_base.c:46:5: warning: pointer targets in passing argument 1 of 'strncpy' differ in signedness [-Wpointer-sign]
ime_base.c:46:5: note: expected 'char *' but argument is of type 'unsigned char *'
ime_base.c: In function 'ime_base_test_any':
ime_base.c:92:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alpha':
ime_base.c:96:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_num':
ime_base.c:100:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alnum':
ime_base.c:104:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_punct':
ime_base.c:108:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_math':
ime_base.c:112:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_file':
ime_base.c:116:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_init':
ime_base.c:179:5: warning: pointer targets in passing argument 1 of 'strcpy' differ in signedness [-Wpointer-sign]
In file included from ime_base.c:9:0:
../../src/dryos.h:547:14: note: expected 'char *' but argument is of type 'unsigned char *'
make: *** [ime_base.o] Error 1


[RenatoPhoto]

do you achieve same or lower rates compared to the old raw_rec version that alex made?

Tested with 1920x1080 at 29.97 fps

TEST 1 Spanning enabled>

CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
.MLV 4194304 KB
.M01  4194304 KB
.M02  4194304 KB
.M03  4194304 KB
.M04  4074381 KB
.M05  29269 KB
Frame skipped and stop

SD is Sandisk Extreme 45 MB/s 32 GB
recorded 1 file
.M00 4194304 KB

Total recorded before stop 21.376884.849 bytes on CF plus 4.294.967.296 bytes on SD
Total is 25.671.892.992

I noticed that % Idle SD card began at 27% and continually went down to 2% when finally stopped.
Continuous recording OK was displayed until the idle of SD card went down to 2%, it then changed to estimated frames.
------------------------

TEST 2 Spanning OFF>

CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
0708.MLV 3179009 KB
0705.MLV  1809481 KB
0706.M00  347137 KB
0706.MLV  4194304 KB
0707.M00  1081573 KB
0707.MLV  41934304 KB
Frame skipped and stop
Total recorded before stop 15.161.159.680 bytes
-------------------------------

TEST 3 ALEX1 RAW_REC

CF is Lexar Pro 1000x 32 GB
Recorded files 8 files
0734.R00 1121697 KB
0734.RAW 4194304 KB
0735.R00 600729 KB
0735.RAW 4194304 KB
0736.R00 1327249 KB
0736.RAW 41934304 KB
0737.R00 342017 KB
0737.RAW 41934304 KB
frame skip and stop
Total recorded before stop 20.652.974.080 bytes
Frame skipped and stop
------------------------

In all test it is evident a slowdown of the write procedure until the frame skips and recording stops.

[g3gg0]

Code: [Select]

magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=ime_base    -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here


fixed, didnt notice due to dependencies for modules being odd.

[jc]

Hi g3gg0

Many thanks for your reply!

>this implies your request : audio support
Nope, not quite .. My request was to keep just the first 5 seconds for sync support, not any more. On many cameras the data rate is too high for audio .. Tho if you mean that if audio is supported in the format then the format should be able to handle just 5 seconds then yes it does ( if however the audio time is locked to the same as length as the video capture time in format v2 two then not :-) )

>this format already is extensible and is designed to be extensible
Great but not what I was trying to imply, I meant an already existing format eg Message Pack
When you design any home baked serialisation format you might think you left room for extension but it lacks "arrays in arrays" or "a kv set in a kv set" or some other essential thing that's needed later on + you can use a pre written parser for the desktop app with no need to write your own .. If it was me designing the format I would use message pack, erlang binary terms or similar binary format

>the first data field in the file is a version string.
Oh soz .. Must haves missed it looking first time :-)

[Toffifee]

There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.