Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Canon eos m

Quote from: g3gg0 on August 18, 2013, 12:46:01 AM
okay, fixed.
(here with 5D3 autoexec/sym)
i played with edmac alignment in this version. please report if you get pink frames or lock ups.

@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)

in camera i wanted to prevent 64 bit integer to string conversion, so i decided to put it as hex string.

Hi g3gg0 - quick question - downloaded the most recent build for raw - but am a bit confused - is raw2dng the same as mlv2dng? With the new autoexe.bin, raw_rec.mo, etc. will the camera retain the ability to shoot mlv or will it revert to the previous raw format?
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

g3gg0

Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.

mlv2dng is released by gnarr in separate posts.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Canon eos m

Quote from: g3gg0 on August 18, 2013, 10:56:37 AM
Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.

mlv2dng is released by gnarr in separate posts.

Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

oddname

I understand its for 5D3 only atm, but when reading trough about it, will it work the same on the Mk2 or are there features that will be Mk3 only in the format?

Canon eos m

Quote from: Canon eos m on August 18, 2013, 11:14:03 AM
Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).

Now 102.5mb/s with skipping after 800 frames.

Also, how do I convert from MLV to RAW. I believe that the mlv2dng will convert directly to dng format?
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Danne

Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?

Canon eos m

Quote from: Danne on August 18, 2013, 12:02:51 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?

Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.
Canon 5D Mark III, Gopro Hero Blacks with 3D Casing, A Few Lenses, Adobe CC 2014, MacBook Pro, Windows 8 PC, Lots of Video Rig!

Started Nuke. Loved it but then the 15 day trial ran out. Back to After Effects and loving it :-)

Danne

I see, probably because I tried with some older files. Thanks

Toffifee

Is there anything I can do to help the progress of this development?  :)

chmee

@g3gg0 and all mlv-devs I'm a little bit confused about the data inside/behind the first VIDF-Block. Why are there more other Blocks inside it? (This VIDF-Blocksize is that high, i cant iterate thru the blocks. What did i missed?


red = blocksize
green = blockposition
blue = framespace

Till the VIDF-Block i iterated thru the blocks via (blockPos+blockSize)=newBlockPos

regards chmee
[size=2]phreekz * blog * twitter[/size]

gnarr

Don't forget that framespace is a 32bit number, so it's 0x00000F7C.
So the actual frame size is blocksize - (framspace + 32) and you start reading the frame at the framespace offset.

g3gg0

these blocks are leftovers from memory that hasnt been cleared.
it is just crap, ignore it.

when doing EDMAC copies of the video frames, the destination address must be aligned to 4k borders. (sometimes...)
thats what the frameSpace is for.
ignore anything what is in this space, just follow the block sizes.

btw: the blocks are called VIDF. VideoFrame
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

chmee

lol. of course vidf..

@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.

@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..
[size=2]phreekz * blog * twitter[/size]

g3gg0

Quote from: chmee on August 19, 2013, 10:46:53 PM
@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.

uhmm. do not hardcode 14 bit, please.
the format supports 1-16 bit depth.
frameSize = (width * height * bpp + 7) / 8
the frame data starts at frameOffset bytes behind the VIDF header end.

see the VIDF block as one frame, header as described in mlv.h, some frame space for alignment and finally the data.
behind the frame data there may also be some empty space.
you have to use the RAWI information to get the real frame data size.
again, look into mlv_dump.c how i handle this.
Quote from: chmee on August 19, 2013, 10:46:53 PM
@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well?
which pictureoffset?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

chmee

(which offset) the same as you described. to optimize edmac copies to the sd/cr-card a1ex (i think) moved/saved blocks. framesize!=dataframe

thx. regards chmee
[size=2]phreekz * blog * twitter[/size]

sletts02

Is there any test footage I can play with? I looked through and found M13-0957.MLV but it doesn't seem to open with mlv_dump.exe
EDIT: Oops, I just found the MLV2DNG - works a treat!


Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.

g3gg0

Quote from: chmee on August 19, 2013, 10:46:53 PM
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..

the picture offset (=frameSpace) is at maximum 4k, where the videoframe is several MiB.

about the NULL data:
when there is some metadata like LENS, RTCI, EXPO etc, i am decreasing the frameSpace and moving the VIDF header to use that gained space for these blocks.
this allows me to put additional data without losing any write performance.

here some example:

Code ("c") Select


assume a block with that size, which is quite good for writing
[empty......................... ....................... ............................ .............]

now it gets filled with a VIDF only, and some frameSpace for edmac and write alignment. size stays the same of course.

[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]

now there is some [LENS] to embed.
I remove all frameSpace to zero and put a NULL block with that size instead and insert the [LENS] block before.

[NULL.................................][VIDF__datadatadatadata........datadatadatadata][NULL......]
[LENS][NULL...........................][VIDF__datadatadatadata........datadatadatadata][NULL......]


Code ("c") Select

now there is some [EXPO] and then [RTCI] to embed, too:

[EXPO][LENS][NULL...............][VIDF__datadatadatadata........datadatadatadata][NULL......]
[RTCI][EXPO][LENS][NULL.........][VIDF__datadatadatadata........datadatadatadata][NULL......]

later, when the frame was written and is going to be reused again, the header in front is written again

[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]



and you see that at the same position where [frameSpace...] is now, previously some other blocks like the LENS etc block have been.
i could clear this area to prevent irritation, but i wanted to save CPU load and thus kept the crap there.

to be honest, i could have implemented frameSpace also using NULL, as it is redundant. but i didnt want to touch that again.
in future releases frameSpace might be set to zero and alignments (both edmac and write) could be realized with NULL blocks only.

the only advantage with frameSpace in VIDF is that i dont have to seek while updating the VIDF header from vsync cbr. but thats not much.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

Quote from: sletts02 on August 20, 2013, 06:14:46 AM
Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.

see some posts earlier, there is a package for 5D3
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

aaphotog

I see most of this thread is actually for those smart enough to do coding for it(not I)
So I have a couple of questions.
with the spanning to both cf and sd cards, I have to grab the recordings off of BOTH cards and use a software that will combine the information from both cards to make 1 video?
If so, is the software to do this and allow us to edit these videos and use them(like we can the preview .raw container) available and working? How about for a non programer like myself(running mac os x)?

sletts02

Quote from: g3gg0 on August 20, 2013, 09:46:52 AM
see some posts earlier, there is a package for 5D3

Found it, cheers!

I was hoping that LRTimelapse would help solve my auto exposure hard stopping problem, but it doesn't seem to have :/ Not sure why it's not working, I'd have thought it would have with you EXIF data in each file.

g3gg0

hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?

i know that there will be a few frames delay with the metadata.
going to fix this soon.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

sletts02

Quote from: g3gg0 on August 20, 2013, 12:51:08 PM
hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?

i know that there will be a few frames delay with the metadata.
going to fix this soon.

Will have a look tomorrow night! Keep up the great work g3gg0. This new format seems to be a great improvement, especially if Alex can implement the sidecar method that he has developed for POST FLICKER.

chmee

@g3gg0 thx for the hints and infos.

thinking about the definition, it would be good to write down some restrictions and/or principles. (lot of that sounds logical, but it's worth to say!)

f.i.
(*) Fileheader (file_hdr) is the first block.
(*) info-blocks must be before the first VIDF-Block (f.i. except RTC)
(*) AUDF-block will be found after every VIDF-block. (or else?)
(*) importance-notation of every block-type (f.i. mandatory, recommended, optional)
(*) every VIDF-Block has its own header, likewise splitted VIDF-Blocks
(*) VIDF-Blocks are splitted bytewise

discussion sides before
(*) filesize is depending on fileformat, at least 4GB
or
(*) files will be splitted at 4GB regardless of fileformat

module raw_rec
(*) notation of coding-datetime/version, from when mlv was supported with all mandatory blocks.
(*) notation, when a blocktype was implemented.

regards chmee
[size=2]phreekz * blog * twitter[/size]

AnotherDave

Where can we download the latest MLV2DNG?  I can't find it anywhere... :-/

gnarr

Quote from: Danne on August 18, 2013, 12:02:51 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?

Quote from: Canon eos m on August 18, 2013, 12:18:26 PM
Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.

Quote from: Danne on August 18, 2013, 12:41:21 PM
I see, probably because I tried with some older files. Thanks

This should now be fixed. Old versions of mlv files should be decoded properly.

Please report any errors you have :) Either with a PM or on this thread.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip

and here is a linux version:
https://dl.dropboxusercontent.com/u/3181048/mlv2dng_linux.zip