Magic Lantern Forum

Developing Magic Lantern => Modules Development => Topic started by: g3gg0 on July 15, 2013, 10:58:23 PM

Title: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on July 15, 2013, 10:58:23 PM
Hey.

after alex spent a lot of time to find out how we can squeeze out the last bit of performance while
writing raw video to SD and CF cards, i used the last days to think about how to structure the
raw videos to make the post processing easier and the format more extensible.

the result is our next Magic Lantern Video format (.mlv) i want you to look at.
use it on your own risk.

for users:
mlv_rec: nightly download page (http://www.magiclantern.fm/downloads.html).
mlv_dump: most recent nightly download page (https://builds.magiclantern.fm/jenkins/view/Other%20tasks/job/mlv_dump/lastSuccessfulBuild/artifact/src/modules/mlv_rec/mlv_dump.exe). (binary for WINDOWS only)

mlv_dump: or here (http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip) (binaries for WINDOWS, LINUX and OSX)

for developers:
mlv file structures in C: here (LGPL) (http://www.magiclantern.fm/modules/modules/mlv_structure.zip/mlv_structure.zip)

preferred: you can export .dng frames from the recorded video using "mlv_dump --dng <in>.mlv -o <prefix>"
legacy mode: post processing is still possible with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.


for details see the description below.
see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.

mlv_dump
 - used for debugging and converting .mlv files
 - can dump .mlv to legacy .raw + .wav files
 - can dump .mlv to .dng  + .wav
 - can compress and decompress frames using LZMA
 - convert bit depth (any depth in range from 1 to 16 bits)

compression:
you can get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.
this feature is for archiving your footage.
converting back to e.g. legacy raw doesnt need any parameters - it will decompress and convert transparently without any additional parameter.

parameters:
Code: [Select]
-o output_file      set the filename to write into
 -v                  verbose output

-- DNG output --
 --dng               output frames into separate .dng files. set prefix with -o
 --no-cs             no chroma smoothing
 --cs2x2             2x2 chroma smoothing
 --cs3x3             3x3 chroma smoothing
 --cs5x5             5x5 chroma smoothing

-- RAW output --
 -r                  output into a legacy raw file for e.g. raw2dng

-- MLV output --
 -b bits             convert image data to given bit depth per channel (1-16)
 -z bits             zero the lowest bits, so we have only specified number of bits containing data (1-16) (improves compression rate)
 -f frames           stop after that number of frames
 -x                  build xref file (indexing)
 -m                  write only metadata, no audio or video frames
 -n                  write no metadata, only audio and video frames
 -a                  average all frames in <inputfile> and output a single-frame MLV from it
 -s mlv_file         subtract the reference frame in given file from every single frame during processing
 -e                  delta-encode frames to improve compression, but lose random access capabilities
 -c                  (re-)compress video and audio frames using LZMA (set bpp to 16 to improve compression rate)
 -d                  decompress compressed video and audio frames using LZMA
 -l level            set compression level from 0=fastest to 9=best compression



examples:
Code: [Select]
# show mlv content (verbose)
./mlv_dump -v in.mlv

# will dump frames 0 through 123 into a new file
# note that ./mlv_dump --dng -f 0 in.mlv (or ./mlv_dump --dng -f 0-0 in.mlv) will now extract just frame 0 instead of all of the frames.
./mlv_dump -f 123 -o out.mlv in.mlv

# prepare an .idx (XREF) file
./mlv_dump -x in.mlv

# compress input file
./mlv_dump -c -o out.mlv in.mlv

# compress input file with maximum compression level 9
./mlv_dump -c -l 9 -o out.mlv in.mlv

# compress input file with maximum compression level 9 and improved delta encoding
./mlv_dump -c -e -l 9 -o out.mlv in.mlv

# compress input file with maximum compression level 9, improved delta encoding, 16 bit alignment which improves compression and 12 bpp
./mlv_dump -c -e -l 9 -z12 -b16 -o out.mlv in.mlv

# decompress input file
./mlv_dump -d -o out.mlv in.mlv

# convert to 10 bit per pixel
./mlv_dump -b 10 -o out.mlv in.mlv

# convert to 8 bit per pixel and compress
./mlv_dump -c -b 14 -o out.mlv in.mlv

# create legacy raw, decompress and convert to 14 bits if needed
./mlv_dump -r -o out.raw in.mlv


Play MLV Files

MLRawViewer

baldand implemented an amazing video player that is using OpenGL and is able to convert your .raw/.mlv into ProRes directly.
even i use it as my playback tool, so consider it as the official player. ;)

see: http://www.magiclantern.fm/forum/index.php?topic=9560.0

MLV_Viewer

see here for a MLV player on windows
(http://upload.g3gg0.de/pub_files/e76d16212357343b54eeac44f2e96002/MLV_Viewer.png)
 (http://www.magiclantern.fm/forum/index.php?topic=8447.0)

in-camera mlv_play:
the module mlv_play.mo is shipped with the pre-built binaries.
it is a plugin for file_man.mo to play .raw and .mlv files in camera.
the discussion thread for this module is there (http://www.magiclantern.fm/forum/index.php?topic=9062.0)

Drastic Preview:
the guys over at drastic.tv are currently implementing the MLV format and already have a working non-open beta version. (i tried it already and i love it :) )
i am sure within the next weeks they will release a new version.
http://www.drastic.tv/index.php?option=com_content&view=category&id=42&Itemid=79





some technical facts:
 - structured format
 - extensible layout
 - as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
 - multi-file support (4 GiB splitting is enforced)
 - spanning suport (write to CF and SD in parallel to gain 20MiB/s)
 - out-of-order data support (frames are written some random order, depending on which memory slot is free)
 - audio support
 - exact clock/frametime support (every frame has the hardware counter value)
 - RTC information (time of day etc)
 - align fields in every frame (can differ from frame to frame)

the benefit for post processing will be:
 - files can be easily grouped by processing SW due to UIDs and file header information (autodetect file count and which files belong to each other)
 - file contains a lot of shooting information like camera model, S/N and lens info
 - lens/focus movement can be tracked (if lens reports)
 - exact* frame timing can be determined from hw counter values (*=its accuracy is the limiting thing)
 - also frame drops are easy to detect
 - hopefully exact audio/video sync, even with frame drops
 - unsupported frames can be easily skipped (no need to handle e.g. RTC or LENS frames if the tool doesnt need them)
 - specified XREF index format to make seeking easier, even with out of order data and spanning writes

why a custom format and not reuse e.g. .mov?
 - other formats are good, but none fits to our needs
 - hard to make frames align to sector or EDMAC sizes
 - they dont support 14 bit raw bayer patterns out of the box
 - even when using a flexible container, nearly all sub blocks would need custom additions
 - this means a lot of effort to make the standard libs for those formats compatible
 - its hard to implement our stuff in a clean way without breaking the whole format

thats the reason why i decided to throw out another format.
it is minimalistic when desired (especially the first implementation will only use a subset of the frames)
and can be extended step by step - while even the most minimalistic parser/post processing tool
can process the latest video files where all stuff is implemented.

if you are a developer (ML or even 3rd party tools) - look over it and make yourself comfortable with that format.
in case there is a bug or something doesnt make sense, please report it.
i would love to get feedback.

here is the link of the spreadsheet that is some kind of reference when designing the format:
https://docs.google.com/spreadsheet/ccc?key=0AgQ2MOkAZTFHdHJraTVTOEpmNEIwTVlKd0dHVi1ULUE#gid=0

implementer's notes
green = fully implemented
blue= implemented, but not 100%
red = not implemented yet, just defined

[MLVI] (once)
 - MLVI block is the first block in every .mlv file
 - the MLVI block has no timestamp, it is assumed to have timestamp value 0 if necessary
 - the MLVI block contains a GUID field which is a random value generated per video shoot
 - using the GUID a tool can detect which partial or spanning files belong together, no matter how they are named
 - it is the only block that has a fixed position, all other blocks may follow in random order
 - fileCount field in the header may get set to the number of total chunks in this recording (the current implementation on camera isn't doing this right)

[RAWI] (once, event triggered)
 - this block is known from the old raw_rec versions
 - whenever the video format is set to RAW, this block has to appear
 - this block exactly specifies how to parse the raw data
 - bit depth may be any value from 1 to 16
 - settings apply to all VIDF blocks that come after RAWI's timestamp (this implies that RAWI must come before VIDF - at least the timestamp must be lower)
 - settings may change during recording, even resolution may change (this is not planned yet, but be aware of this fact)

[VIDF] (periodic)
 - the VIDF block contains encoded video data in any format (H.264, raw, YUV422, ...)
 - the format of the data in VIDF blocks have to be determined using MLVI.videoClass
 - if the video format requires more information, additional format specific "content information" blocks have to be defined (e.g. RAWI)
 - VIDF blocks have a variable sized frameSpace which is meant for optimizing in-memory copy operations for address alignment. it may be set to zero or any other value
 - the data right after the header is of the size specified in frameSpace and considered random, unusable data. just ignore it.
 - the data right after frameSpace is the video data which fills up the rest until blockSize is reached
 - the blockSize of a VIDF is therefore sizeof(mlv_vidf_hdr_t) + frameSpace + video_data which means that a VIDF block is a composition of those three data fields
 - if frames were skipped, either a VIDF block with zero sized payload may get written or it may be completele omitted
 - the format of the data in VIDF frames may change during recording (e.g. resolution, bit depth etc)
 - whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks

[WAVI] (once, event triggered)
 - when the audio format is set to WAV, this block specifies the exact wave audio format

[AUDF] (periodic)
 - see [VIDF] block. same applies to audio

[RTCI] (periodic, event triggered)
 - contains the current time of day information that can be gathered from the camera
 - may appear with any period, maybe every second or more often
 - should get written before any VIDF block appears, else post processing tools cannot reliable extract frame time

[LENS] / [EXPO] / ... (periodic, event triggered)
 - whenever a change in exposure settings or lens status (ISO, aperture, focal length, focus dist, ...) is detected a new block is inserted
 - all video/audio blocks after these blocks should use those parameters

[IDNT] (once)
 - contains camera identification data, like serial number and model identifier
 - the camera serial number is written as HEX STRING, so you have to convert it to a 64 bit INTEGER before displaying it

[INFO] (once, event triggered)
 - right after this header the info string with the length blockLen - sizeof(mlv_info_hdr_t) follows
 - the info string may contain any string entered by the user in format "tag1: value1; tag2: value2"
 - tag can for example be strings like take, shot, customer, day etc and value also any string

[NULL] (random)
 - ignore this block - its just to fill some writing buffers and thus may contain valid or invalid data
 - timestamp is bogus

[ELVL] (periodic)
 - roll and pitch values read from acceleration sensor is provided with this block

[WBAL] (periodic, event triggered)
 - all known information about the current white balance status is provided with this block

[XREF] (once)
 - this is the only block written after recording by processing software, but not the camera
 - it contains a list to all blocks that appear, sorted by time
 - the XREF block is saved to an additional chunk
 - files that only contain a XREF block should get named .idx to clarify their use
 - .idx files must contain the same MLVI header like all chunks, but only have the XREF block in it

[MARK]
 - on keypresses, like halfshutter or any other button, this block gets written for e.g. supplying video cutting positions
 - the data embedded into this block is the keypress ID you can get from module.h

[VERS] (any number, usually at the beginning)
 - a string follows that may get used to identify ML and module versions
 - should follow the format "<module> <textual version info>"
 - possible content: "mlv_play built 2017-07-02 15:10:43 UTC; commit c8dba97 on 2016-12-18 12:45:34 UTC by g3gg0: mlv_play: add variable bit depth support. mlv_play requires experi..."


possible future blocks:

[BIAS]
[DARK]
[FLAT]
 - in-camera black and noise reference pictures can be attached here (dark frame, bias frame, flat frame)
 - to be checked if this is useful and doable




[MLV Format]
 - the Magic Lantern Video format is a block-based file format
 - every information, no matter if audio or video data or metadata is written as data block with the same basic structure
 - this basic structure includes block type information, block size and timestamp (exception to this is the file header, which has no timestamp, but a version string instead)
 - the timestamp field in every block is a) to determine the logical order of data blocks in the file and b) to calculate the wall time distance between any of the blocks in the files
 - the file format allows multiple files (=chunks) which basically are in the same format with file header and blocks
 - chunks are either sequentially written (due to e.g. 4 GiB file size limitation) or parallel (spanning over mutiple media)
 - the first chunk has the extension .mlv, subsequent chunks are numbered .m00, m01, m02, ...
 - there is no restriction what may be in which chunk and what not

[processing]
 - to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
 - when sorting, the sorted data can be written into a XREF block and saved to an additional chunk
 - do not rely on any order at all, no matter in which order they were written into a file
 - the only reliable indicator is the timestamp in all headers
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 16, 2013, 01:37:22 AM
It sounds good!
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: HHL on July 16, 2013, 02:11:12 AM
You guys are seriously an inspiration.  Time to give more donations...geez...you guys ROCK! 
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 16, 2013, 03:16:50 AM
Wow, new big news about ML Raw... Nice job gentlemen.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on July 16, 2013, 10:08:13 AM
Looking good! Magic Lantern getting it's own video format  8)

Have a great vacation g3gg0
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Danne on July 16, 2013, 10:20:19 AM
feels totally right. Happy holidays
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: mageye on July 16, 2013, 04:35:46 PM
There are some great specs in there. I am keen to see that the audio situation is addressed with this format.

IF/WHEN this format is delivered it will indeed be very useful. It's certainly very promising and I will be very happy to adopt such a format that will address many of the problems that the current RAW files confront us with. The fact that it will be tailored to the Canon RAW footage is great.

One thing that I certainly hope is that software developers follow suit and support this format too. I hope that there will be enough people (developers) out there with the vision that this format deserves.

Without proper tools, 'industry' support (Adobe, Black Magic etc.) and independent developers (you know who you are ;) and thanks!) we will still be in a difficult corner. Support here is ESSENTIAL and maybe that needs a little subtle persuasion from fellow Magic Lantern users and supporters ;).

The bottom line is that I really want and hope that this works out because the benefits will be huge. This will be groundbreaking; not just for the Canon DSLR fans, but for the whole independent digital cinema movement.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Joachim Buambeki on July 16, 2013, 04:56:40 PM
Quote
specifies from which sensor row/col the video frame was copied (8x2 blocks)
Does that mean this data can be used later to apply *correct* lens correction in any RAW converter that supports it (ACR, Dxo, P1, etc)?
It would be tremendously useful to have that info stored in the .mlv file to use lens profiles for any sensor crop, even though it might needs to be padded with white/black image data to use regular lens profiles (which won't cost any extra bits if lossless compression is used with a DNG converter, right?).
Looking forward to that. :-)
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: marekk on July 16, 2013, 07:19:15 PM
I think we need a white balance value readed directly from the camera during a raw recording.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on July 16, 2013, 07:36:56 PM
Could full EXIF data be captured to a separate file just before recording a raw file then merged with the raw file after recording ends either in-camera or in an app. This data could also then be parsed as metadata for Resolve etc
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: larrycafe on July 16, 2013, 09:53:25 PM
what about putting the bad pixel information into the video file?

any bad pixel handling can be done in post processing according to the information
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: mvejerslev on July 17, 2013, 12:54:41 AM
Full Exif would be wonderful, particularly I miss the ISO tag for Raw denoising.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on July 17, 2013, 03:35:06 PM
In-camera synced audio, omg yes!
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: bumkicho on July 17, 2013, 09:53:32 PM
Audio part is what I am excited about. It is neat to have a little beep sound, but when you are in a place with music/loud noise/etc., that little beep sound is so easily buried under.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Shizuka on July 18, 2013, 05:01:47 PM
Please express frame rate as a rational number: numerator/denominator instead of framerate/1000. This mitigates increasing precision errors for lower framerates.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on July 19, 2013, 09:56:22 AM
Please add also support for markers while recording (for a lot easier life in post-prod)

(as requested here http://www.magiclantern.fm/forum/index.php?topic=6713.0 (http://www.magiclantern.fm/forum/index.php?topic=6713.0)
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 21, 2013, 09:14:56 AM
I updated the format a bit. Added some of the suggestions and some other stuff.
Thats still just paperwork. That has to be implemented step by step.

I would concentrate on the existing frames, if they are right for what we need before adding new ones.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: mauerfuchs on July 21, 2013, 10:49:51 AM
It would be nice to save RAW files with LOG like the technicolor cinestyle for Canon DSLRs
Here is a nice article:
http://www.cinema5d.com/news/?p=6165
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 21, 2013, 12:26:51 PM
Why would this make any sense?
We are saving raw. Do in post whatever you like to do.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: gnarr on July 21, 2013, 06:49:22 PM
It would be awesome if you could reserve a few bytes for bookkeeping information such as:
time : Auto generate with internal clock
Camera # : set manually
Scene # : set manually
Slate # : set manually
Take# : manually set & edit, plus auto count and reset
Ext Sound record # : manually set & edit, plus auto count and reset

and maybe some other info that could be usefull. As was already spoken about here: http://www.magiclantern.fm/forum/index.php?topic=2497.msg60707#msg60707

An option to change the filename according to this info would also be cool, if possible.
E.g.
Scene 1 - Camera 2 - Take 4 - Slate 1 - 2013-07-21 01:05:13.RAW
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 21, 2013, 09:42:14 PM
well, i think you didnt look into the document i posted ;)
its already in there.

mlv_info_hdr_t   5               
      4   uint8_t   blockType[4]   INFO   user definable info string. take number, location, etc.
      4   uint32_t   blockSize   20   
      8   uint64_t   timestamp      
      4   uint32_t   length      
      x   uint8_t[]   string      can be structured: "tag1: value1; tag2: value2"
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: gnarr on July 21, 2013, 10:07:27 PM
ohh, I definitely didn't see that. You clearly know what you are doing 8)
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: mucher on July 22, 2013, 01:03:10 AM
Awesome. It looks like a container to me. Though, I am curious how the file names are generated, because there might be multiple mlv files generated in the camera simultaneously, and written to SD/CF card/cards simultaneously and randomly, as far as I can understand, so the file might end up overwriting each other -- maybe I worried too much.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Audionut on July 22, 2013, 07:21:33 AM
Awesome. It looks like a container to me.

It is.  MP4, AVI, MKV, MOV, etc, etc, are all containers.

-- maybe I worried too much.

Yes :)
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: tin2tin on July 22, 2013, 10:31:51 AM
A reel name tag could be useful(with a default setting as date(max. 8 characters - for EDL files)). Time code would be the same as time stamp right?

For inspiration, the various metadata MediaInfo can extract, can be found in CSV files here:
http://mediaarea.net/download/binary/libmediainfo0/0.7.64/MediaInfo_DLL_0.7.64_Windows_i386_WithoutInstaller.7z
In MediaInfo_DLL_0.7.64_Windows_i386_WithoutInstaller.7z\Developers\List_Of_Parameters\
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: mauerfuchs on July 23, 2013, 01:05:47 AM
Why would this make any sense?
We are saving raw. Do in post whatever you like to do.

Why does Red do?
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 23, 2013, 02:13:56 AM
like said: as we are saving raw, it doesnt make sense.
do this in post processing please.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 23, 2013, 02:23:12 AM
So the ETA for this is 1-2 months?
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Audionut on July 23, 2013, 03:10:36 AM
if you are an end user, wait until we looked over it. then we are going to implement some example code
to write and read this file format. and 1-2 months later it *may* be in the ML sources.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 23, 2013, 04:47:12 AM
Thats what I trying to confirm on.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: Audionut on July 23, 2013, 05:05:57 AM
Yeah that dodgy g3gg0 is always making OP's with incorrect facts  :o
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 23, 2013, 06:48:10 AM
It just seemed worded funny I don't know. I wanted to make sure. I was wondering if there is an update to this eta, that's all.
Title: Re: [DRAFT] Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 23, 2013, 09:40:30 AM
there is no hard schedule for this. it will be implemented when there is some spare time ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 04:58:21 PM
Update: (see first post)

The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)

see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.

if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built (http://upload.g3gg0.de/pub_files/5795906e2820ed5c73514ed4311dbcd4/raw_rec.zip) ones. (5D3)
use it on your own risk.


if you can show the mlv_dump logs of your cameras, it would help me a bit.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 28, 2013, 05:06:20 PM
Can i just drop the contents of raw_rec.zip in the "modules" folder?
This way i can't initialize any module.
I am on the 6D.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 05:20:49 PM
Can i just drop the contents of raw_rec.zip in the "modules" folder?
This way i can't initialize any module.
I am on the 6D.

whats the error message? about _prop_get_value?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 28, 2013, 05:22:10 PM
Yes, it's that one.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 06:12:53 PM
can you retry downloading and testing?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 28, 2013, 06:15:46 PM
Wait, if it's for 5DIII only it can't work, right?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 06:18:21 PM
could work now on others too. just try.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 28, 2013, 06:26:52 PM
It works, what is that you need? :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on July 28, 2013, 06:37:21 PM
Adding the pre-built modules does not work on the 50D (error linking modules). I think the problem is with the IME modules. I get compiler errors with ime_base.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on July 28, 2013, 06:42:00 PM
600d with zip-file
Works. I get mlv-records. And now I´m searching for mlv_dump :)

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on July 28, 2013, 07:03:51 PM
cannot compile mlv_dump

Code: [Select]
[email protected]:~/magic-lantern/modules/raw_rec> make mlv_dump
[ GCC      ]   mlv_dump
In file included from mlv_dump.c:26:0:
../lv_rec/lv_rec.h:24:17: fatal error: raw.h: Datei oder Verzeichnis nicht gefunden
compilation terminated.
make: *** [mlv_dump] Fehler 1

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 07:37:50 PM
true, didnt fix the makefile (cant do it right now)

just run:
'gcc mlv_dump.c -o mlv_dump -I../../src'
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on July 28, 2013, 08:12:49 PM
compiles, ThankYou
but
Code: [Select]
[email protected]:~/test> ~/raw2dng *.RAW
Error: This ain't a lv_rec RAW file

Edgar

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Habitat on July 28, 2013, 08:27:38 PM
Nice vid g3gg0 - just wondering does this version (current 2.0) have audio yet?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on July 28, 2013, 08:37:50 PM
No stupid questions, no badgering, just THANKS for all you do!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on July 28, 2013, 08:45:48 PM
Just for info and giving feedback, see the top of the mlv-header
Code: [Select]
File Header (MLVI)
    Size        : 0x00000038
    Ver         : v2.0
    GUID        : 819A0505F3E5613E
    FPS         : 25.000000
    Frames Video: 312
    Frames Audio: 0
Block: RAWI
    Size: 0x000000B8
    Time: 20.695000 ms
    Res:  864x486
    raw_info:
      api_version      0x04332200
      height           3905328
      width            14
      pitch            2048
      frame_size       0x00003A98
      bits_per_pixel   0
      black_level      0
      white_level      1734
      active_area.y1   1886
      active_area.x1   0
      active_area.y2   0
      active_area.x2   33620224
      exposure_bias    1, 6461
      cfa_pattern      0x00002710
      calibration_ill  -907
Block: INFO
    Size: 0x00000010
    Time: 32.126000 ms
Block: RTCI
    Size: 0x00000030
    Time: 37.598000 ms
     Date:        28.06.2013
     Time:        18:20:04 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 0
     Daylight s.: 0
Block: EXPO
    Size: 0x00000028
    Time: 38.185000 ms
     ISO Mode:   0
     ISO:        800
     ISO Analog: 96
     ISO DGain:  0 EV
     Shutter:    39980 µs (1/25.01)
Block: LENS
    Size: 0x00000060
    Time: 38.226000 ms
     Name:        '70-300mm'
     Serial:      ''
     Focal Len:   70 mm
     Focus Dist:  545 mm
     Aperture:    f/5.60
     IS Mode:     14
     AF Mode:     0
     Lens ID:     0x00000001
     Flags:       0x00000000
Block: IDNT
    Size: 0x00000058
    Time: 38.441000 ms
     Camera Name:   'Canon EOS 600D'
     Camera Serial: '55C28963C'
     Camera Model:  0x80000286
Block: VIDF
    Size: 0x000B374C
    Time: 131.710000 ms
     Pad: 0x000000BC
   Frame: #0
    Crop: 584x368
     Pan: 584x361
   Space: 188
Block: VIDF
    Size: 0x000B374C
    Time: 171.778000 ms
     Pad: 0x000000BC
   Frame: #1
    Crop: 584x368
     Pan: 584x361
   Space: 188

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 10:33:57 PM
@Habitat:
as the video description already said: no audio

@Stedda:
you are welcome

@escho:
perfect! thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 28, 2013, 10:36:48 PM
with this  (http://upload.g3gg0.de/pub_files/6e8bacd4c62e01ef147623c6636a7634/raw_rec.zip)version, the lens id should be correct again (not just 0x00000001)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on July 29, 2013, 02:24:50 PM
LensID is now correct (tested with exiftool).

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 03:28:46 PM
For ML Raw 2.0 can we please (above all additional features) move the writing/card speed information that is displayed during recording to either the upper or lower corner, but off of the image window, for HDMI?

The #1 complaint we had from EVERYONE on our 5 day shoot with the hack was that the display information was in the way of the action.  Often, it ended up completely covering an actors face for the entire scene!!!

Please move it somewhere else!

It is critical for production.

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 03:36:04 PM
Rather than have version 2.0 record audio, IF it is going to have Timecode, would it be possible for it to OUTPUT TC that a sound recordist could jamsync to? 

This MovieSlate app for the iPhone can do it, so I would image it is possible for a 5Dmk3.
https://itunes.apple.com/us/app/movieslate-clapperboard-shot/id320315888?mt=8 (https://itunes.apple.com/us/app/movieslate-clapperboard-shot/id320315888?mt=8)

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 29, 2013, 03:53:03 PM
@AnotherDave:
TimeCode: 64 bit timestamp counters with microsecond resolution is not enough?
Display: for productional use this debug information will get removed.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: nigel on July 29, 2013, 04:01:54 PM
My jaw literally dropped when I saw spanning support. I have a 5DMk3. How can I help?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 29, 2013, 04:35:12 PM
@nigel:
implement multi-file writing support with writer threads in raw_rec.c :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 29, 2013, 05:21:46 PM
One of the simplest and maybe nicest thing to have would be to have the non recorded area blacked out, so who's recording can actually focus on what is in the frame. Maybe leave it as an option, some people might find it useful to see outside of the boundaries.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 29, 2013, 05:37:14 PM
yes.
possible options: rectangle / black out / semi-transparent
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on July 29, 2013, 05:45:29 PM
Oh, the text input system looks almost seamless, you did a very good job and it works well.
:)
Two things:
1)Why does the recording size need to be a multiple of 512 bytes?
2)If i enable small hacks with your modules i get an errore when i start recording raw, it did not happen before.
Are they useless and thus disabled in your build?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 06:13:48 PM
@AnotherDave:
TimeCode: 64 bit timestamp counters with microsecond resolution is not enough?
Display: for productional use this debug information will get removed.

Will we be able to output the TC through the headphone jack (like the iPhone app does)?  Maybe even jam sync with another camera through the mic input?

That is more important to me than having synced audio recorded in camera.  Secondary source audio is always better, and it is good to not-have-to-be-the-sound-guy-too on a shoot.

Blackmagic Resolve can automatically sync secondary source audio if the TC information matches, so syncing in post using this method is a breeze.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 29, 2013, 08:08:24 PM
@ilguercio:
thanks!
1) with the old format, where the frames had no additional information (=headers) this
would give you the best write performance (in net rate / gross rate).
in this version this is not the best of all options. just ignore it for now.

2) i didnt try that yet. i just wonder what has changed..? it should work as before.

@AnotherDave:
no audio output yet and also not planned t.b.h. as i have no clue about that topic.
plus i dont have any tools for that.
are you talking about LTC ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 08:58:22 PM
@AnotherDave:
no audio output yet and also not planned t.b.h. as i have no clue about that topic.
plus i dont have any tools for that.
are you talking about LTC ?

Yes! :-)

LTC, essentially what I want is a module that can generate a free run (time of day based) TC as an audio signal via the headphone jack, and also read that same type of signal to synchronize the internal clock with a secondary source (jam syncing).  This could allow you to sync up to a Sound Devices 788T or even a second 5D3.

Or even record the LTC output to an unused channel from a multitrack recorder.

This, to me, is far more useful than recording audio in camera. 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 29, 2013, 09:34:09 PM
audio reading is too hard to implement.
would a output-only version only help at all then?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 29, 2013, 09:46:18 PM
Yes! :-)

LTC, essentially what I want is a module that can generate a free run (time of day based) TC as an audio signal via the headphone jack, and also read that same type of signal to synchronize the internal clock with a secondary source (jam syncing).  This could allow you to sync up to a Sound Devices 788T or even a second 5D3.

Or even record the LTC output to an unused channel from a multitrack recorder.

This, to me, is far more useful than recording audio in camera.

I don't understand why we can't just get access to the headphone jack in these cameras and allow it to write to the sd card while the CF card does the video writing.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 09:51:54 PM
audio reading is too hard to implement.
would a output-only version only help at all then?

I'd settle for output only.  :-)

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 09:55:12 PM
I don't understand why we can't just get access to the headphone jack in these cameras and allow it to write to the sd card while the CF card does the video writing.

My thinking is this - the audio quality that the 5D3 records isn't very good.  It's ok and useable, but if you're going through the trouble to shoot in uncompressed 14-bit raw... why wouldn't you want a quality of audio that is going to match with the quality of the video?  That can only be done with a secondary system.

I don't think very many people record their audio into their Red cameras... and to be honest, Red is the only thing that compares to the quality of image I'm getting out of my hacked 5D3.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: aaphotog on July 29, 2013, 09:59:31 PM
My thinking is this - the audio quality that the 5D3 records isn't very good.  It's ok and useable, but if you're going through the trouble to shoot in uncompressed 14-bit raw... why wouldn't you want a quality of audio that is going to match with the quality of the video?  That can only be done with a secondary system.

I don't think very many people record their audio into their Red cameras... and to be honest, Red is the only thing that compares to the quality of image I'm getting out of my hacked 5D3.
Much easier to use that horrible camera audio than a clap board even if you do have the funds and man power to record separate audio.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 10:07:13 PM
Much easier to use that horrible camera audio than a clap board even if you do have the funds and man power to record separate audio.

But then why would you be shooting it in raw?  If you don't have the funds or equipment?  Seems to me, that if a TC syncing module for a secondary source was implemented that this would open this camera up to a wider range of productions.

Also I'm not talking about using a clapper, if your TC on your .mlv file matched the audio on your recorder you can just click 1 button in Davinci Resolve and have all of your audio synced within a second.  That could only be done if the camera could output it's free run (TOD) TC via the audio jack.

Dave
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on July 29, 2013, 10:17:39 PM
Could that beepsynch work for premiere or after effects aswell?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on July 29, 2013, 10:20:34 PM
I meant that I want the headphone jack to be able to record the audio from an external mic. I'm gonna get an H4n anyways so it doesn't really matter.

I mean it would be nice if the camera could send a signal for when the H4N (or other recorder) should stop and start recording...

Thats what I meant sorry for the confusion...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 29, 2013, 11:42:33 PM
@Danne
beep sync is ok, but how do you know which clip it belongs with? 

For the short I filmed on the 5D3 ML, we shot around 1,400 clips over 5 days.  We were fortunate enough to have an awesome sound recordist with a 788t and a smart slate.

In resolve, all I had to do was separate the clips and .wavs by the day it was recorded, select any frame where the TC on the slate was visible, and type it in.  Resolve automatically synced it from there.

If we had embedded TC out via the audiojack, I wouldn't even have had to type in the TC.

@xnine

That is probably a much more complex task, though the Zoom does have lanc... so who knows?  With the zoom, and TC out, you could record the TC output sound from the headphone jack into the 1/8" at the top of the H4n and run it through a program that would read the TC track and embed it into the other tracks with recorded audio.

On a side note, it may be worth it to wait and buy an H6 - 4 xlrs and can be expanded into a 6 track recorder.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: aaphotog on July 30, 2013, 12:01:53 AM
But then why would you be shooting it in raw?  If you don't have the funds or equipment?  Seems to me, that if a TC syncing module for a secondary source was implemented that this would open this camera up to a wider range of productions.

Also I'm not talking about using a clapper, if your TC on your .mlv file matched the audio on your recorder you can just click 1 button in Davinci Resolve and have all of your audio synced within a second.  That could only be done if the camera could output it's free run (TOD) TC via the audio jack.

Dave
It doesn't matter WHY. That is of no concern to YOU! Not to say that I don't have the funds/equipment. The fact is, just because YOU may not find it useful, doesn't mean that no one else will. If I want to record raw because I choose to, and have the option of having a recorded audio track on my camera, is that a bad thing? I find it quite ridiculous when people say that NO ONE needs something, an assumption based on their own needs.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 12:14:47 AM
It doesn't matter WHY. That is of no concern to YOU! Not to say that I don't have the funds/equipment. The fact is, just because YOU may not find it useful, doesn't mean that no one else will. If I want to record raw because I choose to, and have the option of having a recorded audio track on my camera, is that a bad thing? I find it quite ridiculous when people say that NO ONE needs something, an assumption based on their own needs.

I apologize, I meant to say... why would you want a great picture with terrible audio?  TC sync would be much more useful for everyone - don't you think?

Implementing a feature that is already in the camera (recording synced audio) is great!  And I hope it is possible, but if you're going for quality filmmaking why would you want to marry great pictures with bad sound when there could be a better way?

It isn't a pissing competition, I'd just like to add a MAJOR professional feature to the hack I've come to love and talk about how great it is all the time.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: jplxpto on July 30, 2013, 12:51:33 AM

Maybe this can help us.
I think that would be an interesting idea VLC come to support the new ML container.
Maybe the guys from the VLC team would like to join us.

http://www.enjoythearchitecture.com/vlc-architecture

http://wiki.videolan.org/Documentation:Documentation/
http://wiki.videolan.org/Documentation:Hacker%27s_Guide/#VLC.27s_Core_.2F_libVLCcore

http://mailman.videolan.org/pipermail/vlc-devel/2011-August/081064.html

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 30, 2013, 01:13:03 AM
with the latest commits, i added periodic updates of LENS, EXPO and RTCI blocks.
they get embedded into the unused space between two frames, so we have no additional slowdown due to those blocks.
this requires the ML core to be compiled again, as i need the msg_queue_count function i just added. (5D3)

by the way - did some of you run speed tests and comparisons between the old format and the new one?
dont have a reliable card to say anything about speeds, but i think the write rate will be a bit lower.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on July 30, 2013, 02:13:04 AM
Great job g3gg0! Could you maybe post a tiny mlv file here, so we can start to develop converters? :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AdamTheBoy on July 30, 2013, 04:30:53 AM
This is really exciting, full of great ideas and I'm really happy to see magic lantern raw looking more and more convenient and usable professionally.  The custom tagging and take number is a fantastic addition, I think something like this was really needed.  I'm wondering if it would be possible to have an option to name the .mlv files after their tag and take number.  It would be awesome if a file would come out looking like this: "Day 1 take 1.MLV" or "Night test take 5.MLV" or "Scene 3 setup 3 take 5.MLV"

On a set a script supervisor could be keeping track of good or bad takes and then you don't even have to process the takes you know were bad, those files could just be deleted and save precious time.  Really, this new format is great and I can't wait to see more, thanks so much everyone for your work.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 04:50:01 AM
Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards.  Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.

I'm on a mac, what can I use to convert and check the files?

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gogauze on July 30, 2013, 05:26:08 AM
In case you are still wanting some of this g3gg0:

Code: [Select]
File Header (MLVI)
    Size        : 0x00000038
    Ver         : v2.0
    GUID        : E2B601B3F770EFC9
    FPS         : 23.976000
    Frames Video: 256
    Frames Audio: 0
Block: RAWI
    Size: 0x000000B8
    Time: 2.933000 ms
    Res:  1920x1080
    raw_info:
      api_version      0x0B152000
      height           4797520
      width            14
      pitch            2046
      frame_size       0x00003A98
      bits_per_pixel   0
      black_level      0
      white_level      1928
      active_area.y1   2074
      active_area.x1   0
      active_area.y2   0
      active_area.x2   33620224
      exposure_bias    1, 6722
      cfa_pattern      0x00002710
      calibration_ill  -635
Block: INFO
    Size: 0x00000010
    Time: 5.250000 ms
Block: RTCI
    Size: 0x00000030
    Time: 7.161000 ms
     Date:        29.06.2013
     Time:        07:44:32 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 209
     Daylight s.: 0
Block: EXPO
    Size: 0x00000028
    Time: 7.832000 ms
     ISO Mode:   0
     ISO:        1600
     ISO Analog: 104
     ISO DGain:  0 EV
     Shutter:    20845 µs (1/47.97)
Block: LENS
    Size: 0x00000060
    Time: 7.866000 ms
     Name:        'EF35mm f/1.4L USM'
     Serial:      ''
     Focal Len:   35 mm
     Focus Dist:  88 mm
     Aperture:    f/0.00
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000001
     Flags:       0x00000000
Block: IDNT
    Size: 0x00000058
    Time: 7.978000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: '131EF8CF14'
     Camera Model:  0x80000285

However, after tooling around with it and getting it to work, raw2dng nor RAWMagic recognize the file as a .raw   :'(

EDIT: Changing the file suffix from .raw to .RAW caused raw2dng to stop denying that it exists at all, and now just returns the standard "This ain't a lv_rec RAW file."
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: crazyrunner33 on July 30, 2013, 06:19:49 AM
Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards.  Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.

I'm on a mac, what can I use to convert and check the files?

Thanks.

My results are almost identical with the 64 GB KomputerBay card.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: tonybeccar on July 30, 2013, 08:50:32 AM
I'd settle for output only.  :-)

Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha

Cheers!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 30, 2013, 12:03:27 PM
About write speeds:
when you chose the largest frame format, whats the difference in the frame count?
i.e. old 300 frames, mlv 250 frames
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Redrocks on July 30, 2013, 02:00:51 PM
5d2 Komputerbay 64GB:

a.d's 728e571a1276 @1872 x 1250 24fps = 260 / 295
g3gg0 25th build same set up = 230 / 250
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 30, 2013, 03:24:22 PM
Can you try two or three next smaller resolutions too?
Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 03:37:12 PM
Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha

Cheers!

How is that easier than generating a TC sync from the camera itself?  I want this as a feature, specifically to allow this camera to be used on more professional shoots.  Making it complex isn't good for the client...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Redrocks on July 30, 2013, 04:32:45 PM
5d2, Komputerbay 64GB, a.d's 728e571a1276 VS g3gg0 28th build. 1872, 24fps, GD off:

1250 @3:2

294 / 229
262 / 234
273 / 246
260 / 218
299 / 236

1054 @16:9

1110 / 913
1184 / 975
1014 / 821
1341 / 660
1215 / 1025
1569 / 683

1012 @1.85:1
a.d. is near enough continuous at this ratio
/ 2754
/ 3005
/ 3042
/ 2334 card full; reformat.
/ 700
/ 3078
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 06:18:50 PM
Will this format store white balance info?  Picture profiles?

Still not sure how to convert .mlv to .dng too :-/
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 30, 2013, 08:44:35 PM
@Redrocks:
thanks a lot!

@AnotherDave:
why picture profiles and wb?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 09:54:12 PM
@AnotherDave:
why picture profiles and wb?

I recently shot a project for a client with the 5d3 ML Raw Hack.  It was a 5 day shoot for a short film.  We filled 42 cards (6 x KB 64gb) for a total sum of 2.68TB and 1,400 clips.

I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.

If it could store that information, it'd mean that I could just hand them footage instead of leaving them with something they don't understand.

*I wouldn't want to hand a client footage that looked as if it was shot @5500k under tungsten lighting.  I know what raw is, and the colorist will know what raw is... but the client would just see it and think I didn't know what I was doing.

Also, I was shooting at -1 EV to aid in ETTR to improve my S/N ratio.  If the image fine tuning data could also be included, that'd be a big help too!

Dave
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 30, 2013, 09:56:13 PM
I can continue to run more tests if someone can tell me how to convert the .mlv files :-/
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on July 31, 2013, 12:09:01 AM
Really excited for audio support. Go g3gg0!
Own a 5D MIII, not familiar with developing, but I'd love to help if you need testers.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 12:45:47 AM
I can continue to run more tests if someone can tell me how to convert the .mlv files :-/

this:

The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 12:49:43 AM
I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.

wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 01:15:20 AM
5D3 autoexec.bin + modules: http://upload.g3gg0.de/pub_files/50f8e2a45595335252a79b19aba2f422/raw_rec.zip

Redrocks:
can you try the latest version with the sizes you already posted?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Redrocks on July 31, 2013, 02:22:23 AM
Can you recheck the files for me or have I messed up? I put all your new .mo files in the modules folder, left the .bin in the root and deleted the .sym. Then the camera refused to fire up. Tried the build on three cards and ended up having to insert a clean card whilst the camera was switched on to roll back to the latest regular nightly.   
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 31, 2013, 04:12:42 AM
wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
If I had to give a reason... it would just be for consistency. 

If I needed a catch phrase... it would be "because the Red camera can do it" :-)

Dave
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 31, 2013, 04:19:27 AM
Tonight's build is far slower for me...  I'm only getting 81.5MB/s 54ms idle, 4000-7000 frames expected,  takes a lot longer to works it way up to the speed too.

5D3, KB 64 1000x, 24fps, 1080p
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on July 31, 2013, 10:25:13 AM
i can't understood why it tell me "not a chdk dng" ?

Code: [Select]
MacBook-Pro-de-david:~ david$ /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/dng2raw /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW dng
reading /Volumes/FREECOM HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW (size=1382573000)
read 1382573000
not a chdk dng: Undefined error: 0
ifd 0: (28804)
ifd 1: (15936)
ifd 2: (16907)
Segmentation fault: 11

and why i get this when i compile mlv_dump ?

Code: [Select]
MacBook-Pro-de-david:raw_rec david$ gcc mlv_dump.c -o mlv_dump -I../../src
mlv_dump.c:109:65: error: invalid suffix "d" on floating constant
mlv_dump.c:163:83: error: invalid suffix "d" on floating constant
MacBook-Pro-de-david:raw_rec david$
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 01:07:04 PM
i can't understood why it tell me "not a chdk dng" ?

why did you specify dng ? thats a raw file.

mlv_dump.c:109:65: error: invalid suffix "d" on floating constant

hm will check later, maybe a copy paste error and my compiler didnt warn?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 01:10:41 PM
Tonight's build is far slower for me...  I'm only getting 81.5MB/s 54ms idle, 4000-7000 frames expected,  takes a lot longer to works it way up to the speed too.

5D3, KB 64 1000x, 24fps, 1080p
thanks. what is "far slower" in numbers?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 01:11:34 PM
Can you recheck the files for me or have I messed up? I put all your new .mo files in the modules folder, left the .bin in the root and deleted the .sym. Then the camera refused to fire up. Tried the build on three cards and ended up having to insert a clean card whilst the camera was switched on to roll back to the latest regular nightly.

this is for 5D3 :)
you just need the module (i assume you have a recent core build)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Redrocks on July 31, 2013, 02:01:40 PM
tcc: error: undefined symbol 'msg_queue_count' [E] failed to link modules

This is with only the .mo files placed in the module folder of a.d's 728e571a1276. Can't see how I'm doing things any different than usual, but I'll see if there is something I've overlooked.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on July 31, 2013, 02:08:28 PM
no, this symbol was introduced a few days ago.
thats the reason why i said you should use a recent ml core ;)

edit: ah forget it, i just looked up that symbol for 5D3. would have to add 5D2 first.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on July 31, 2013, 03:35:14 PM
thanks. what is "far slower" in numbers?

Latest ML RAW 1 - 94.5MB/s always continuous ok

Latest ML RAW 2 - 83.3MB/s, 573ms idle

I recorded 2 mins of footage, and videotaped the display.
Here is a link to the display screen:
https://www.dropbox.com/sh/wvimd6cdqel7m3b/2Dp3_iXYyR

Hope that helps...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 12:09:01 AM
is this (http://upload.g3gg0.de/pub_files/3c2e37fd6ff824bd910a37351b5474e1/raw_rec.mo) one any better regarding performance?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 01, 2013, 12:56:57 AM
I got pretty much the same performance on my 5D2 yesterday as with the old format. Maxed at 76MBps.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 01:53:46 AM
here (http://upload.g3gg0.de/pub_files/4a8662cf1d07a15450ba8c9a5d99d4be/raw_rec.mo) is some special build for the 5D3 with spanning enabled.
can you please try with your fastest SD and fastest CF cards, if you see any speedup over the version in my post before?

-> the output files will for sure be buggy, its just about writing speed

thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 02:02:06 AM
It did something!

it always says continuous ok now...

and it says the write speed is 94.1MB (~80.4mb)

it doesn't display the name of the file it is recording though...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 01, 2013, 03:22:28 AM
Speed seemed the same on my 6D, as far as i could tell.
Haven't tested it properly yet, though.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 10:11:43 AM
the last build has these changes:
 - files are stored in card root folder
 - two writer threads are started
 - one thread writes to SD, the other to CF card
 - one file is named .MLV, the other .M00
 - frames are saved on either the one or the other card
 - the order of the video frames in the .MLV / .M00 is random (well, tbh it is not random but depends on card speed)

in technical words:
 - i have set up a file spanning mechanism over SD and CF

in non-technical words:
 - increasing data rate by using SD and CF in parallel



the good thing:
 - write rate is obviously faster than before
the bad thing:
 - write rate on CF only is not as fast as alex' optimized version, but i am trying to improve that.


ah, one more (obvious, but worth mentioning) conclusion:
 - this is only possible on 5D3 ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: pravdomil on August 01, 2013, 10:55:44 AM
nice, soon I'm going to port it on 5D2. Maybe if you will reach some milestone?
I want to solve the CF speed issue.
What is the write speed CF+SD? 100+?
Do you want some help with coding?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 10:58:10 AM
i cant get beyond 70MB/s with my crappy card.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 01, 2013, 11:08:57 AM
Will try to find a faster sd card. Awesome work by the way!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 12:43:17 PM
What is the write speed CF+SD? 100+?
Do you want some help with coding?

maximum possible: 120 + 20 as alex said once.
but when you have computers, 1+1 is never 2 but 1.8.
so its less than 140MiB/s and more than 120 MiB/s under ideal circumstances.

we will see if theory matches results :)

help is always appreciated in this project - why ask? ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 01, 2013, 01:35:15 PM
why did you specify dng ? thats a raw file.

hm will check later, maybe a copy paste error and my compiler didnt warn?
sorry I was tired I did not realize that I used dng2raw instead of raw2dng   :-\
anyway, i still can't open the .raw file from the mlv dumper :

Code: [Select]
MacBook-Pro-de-david:~ david$ /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/raw2dng /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW
Error: This ain't a lv_rec RAW file

what version of arm none eabi are you using please?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 01:39:49 PM
Does it have to write to both the SD & CF?  I know it will speed things up, but that also makes things even more complicated.

I love the work that everyone is doing with the hack, and to be honest... I think the quality of the images being captured is higher than most professional cameras.

Because the quality is so high, I'm hoping for a simplification of the process of recording in and syncing to raw with the ML 5D3 with version 2.  I want to be able to shoot with it to get amazing image quality... but I also want to be able to be hired out with my camera to shoot in raw.

The more complicated the process gets, the less likely it is that someone will hire a DP with a hacked 5D3.  The more technical knowledge you need to operate a hacked 5D3, the more it is going to be looked at as a "work around" or something "not professional".

The results of this are VERY professional, and I think the format is in the right place at the right time.

But for it to be successful, it really needs to record 1 clip to the CF card that contains ALL the possible information that is gathered while shooting.  When it is opened on a computer, it needs to look as similar to what was shot in the field as possible.  And, syncing the audio needs to be accomplished in 1 click via timecode.

But all of those things together... and you've got a cheap Red camera, and everyone wants to use it.  Continue to make it hard to work with... people will buy or request a BMCP for shooting.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 01, 2013, 02:40:11 PM
I for one like the idea with filespanning between cf and sd. Better ratios for slowmo is one benefit. Really cool work
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 03:08:00 PM
Does it have to ...I'm hoping for .... I want to... I also want to... it really needs to... audio needs to...

uhm, what?
did you forget that this feature is still EXPERIMENTAL?
we still try to figure out how far can we go and what is possible.

i am happy about any recommendation about usability, but please leave us enough room for experiments.
and as the first post says, it IS an experiment.
what will be in the ML release is another story.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: pravdomil on August 01, 2013, 03:19:48 PM
Does it have to ...I'm hoping for .... I want to... I also want to... it really needs to... audio needs to...

g3gg0 please implement 500fps to raw_rec I really need that feature to capture my breakfast also I need sound recording if something horrible will happend
but seriously - one more reason why dont share unfinished features

help is always appreciated in this project - why ask? ;)

well I have some ideas to raw_rec, but now I have to do some other part of ML, I will message you when I will be ready
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 03:49:20 PM
I just think it is important, when experimenting, to keep the end user in mind.

I say something as simple as "Rec 709" to a producer with 20 year of experience... and they look at me like I have 2 heads.

Not everyone is as technologically inclined as the folks on this board, and I want to be able to use these amazing images to improve my work.  The harder it gets on the backend... the less likely it is that will happen.

Dave
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on August 01, 2013, 04:30:55 PM
I just think it is important, when experimenting, to keep the end user in mind.

They are nice enough to share their work in progress with us and clearly mark it as so.

Keep hounding them for features they said they'd work on for the future or for your own personal use and ruin it for the rest of us.

For something that is free and they work hard on it's never enough for some of you....
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 04:58:29 PM
They are nice enough to share their work in progress with us and clearly mark it as so.

Keep hounding them for features they said they'd work on for the future or for your own personal use and ruin it for the rest of us.

For something that is free and they work hard on it's never enough for some of you....

I'm not hounding, I'm trying to help.  I'm a professional, working filmmaker who loves the quality of the image I am getting through my "hacked" 5D3 with their awesome code.  I'm only asking to have changes implemented that will help with the perception that his custom firmware isn't a "hack" or something unprofessional, and can be used artistically in a production.

If they're not interested in my input, ask me to stop offering my experience and I'll just as easily stop posting.

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on August 01, 2013, 05:20:06 PM
Look at g3gg0 quote from you, seems at least one main DEV has the same impression.

Don't ruin it for us patient ones that are happy to get whatever they can come up with because it's 1000 times better than what we had.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 05:30:56 PM
Look at g3gg0 quote from you, seems at least one main DEV has the same impression.

Don't ruin it for us patient ones that are happy to get whatever they can come up with because it's 1000 times better than what we had.

I'm not a developer.  I'm a DP, Director, Editor... when I say "I was hoping" what I should say is "a professional would expect" but I'm also not a jerk who thinks his ideas are the most important.

Bottom line is... this produces an image that I think is better than a Red.  It allows me to work on lower budget projects that can't afford a Red, but still get amazing images. 

It has its flaws though.  I know it is experimental.  But since we can already, rather comfortably, shoot 1080p on the 5D3 with no real issues.  *I say this based on the fact that I used it for a short where we filmed 1,400 clips and 2.68TB of data with it and had no problems.

As a professional, it seems more important to me to focus on what it is missing.  IE: WB information, external TC sync, Picture Profile information, easy backend processing...

If you'all don't think I should be using this thing and asking for improvement cause I am "annoying" them... fine.  The BMPC 4K is only $600 more than a non-hacked 5D3... so instead of getting what professionals want to use, we can just bow to marketing.

I thought the idea was to make it professional.  Was I wrong?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on August 01, 2013, 06:25:09 PM
I get it you're special. You're a director/professional/editor and because of that you have a right to the features you demand. 

I'm not going to argue with you because you're all over the place.  Just remember you're using a tool that wasn't designed to do what YOU want it to do for your one specific use be happy with what you're getting for FREE or use the proper tool. As a professional you should already know this.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 01, 2013, 06:36:43 PM
I get it you're special. You're a director/professional/editor and because of that you have a right to the features you demand. 

I'm not going to argue with you because you're all over the place.  Just remember you're using a tool that wasn't designed to do what YOU want it to do for your one specific use be happy with what you're getting for FREE or use the proper tool. As a professional you should already know this.

I just don't know why you're arguing with me over this... 

There is a reason things work they way they do.  I'm not trying to annoy the developer and I think what he has been able to get out this camera to do is insane!

I'm not special at all, but I'm willing to offer my experience to the developers to help them shape this into something even more awesome.

Honestly, I could be like every other professional and say "psshhh... shoot with the F55", "Yea, but it'll make your camera explode"...

I even went as far as to test this on a paid, 5 day job... had it failed, I could have been sued.  But it worked perfectly, and I've been singing it's praises ever since!

It's great!  Amazing!  I can't believe this quality is THIS accessible for SO CHEAP!  But... the display covers actors faces, and sometimes the images look like vomit before you CC then in Resolve.  I think those issues need to be solved before spanning the data across 2 cards.

Why is that such a threat to you?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 01, 2013, 11:20:48 PM
can you both stop that please?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 02, 2013, 02:41:56 AM
5D3 users:
can you test this (http://upload.g3gg0.de/pub_files/928d2e3d617d0617e27e2d6a84f7bb6e/raw_rec.mo) version for write speeds with spanning OFF and ON (see menu)
hacks both work on my 5D3, so please try with all hacks enabled.

you should see a clean write speed display for SD and CF (upper is CF, lower one is SD)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Audionut on August 02, 2013, 03:23:56 AM
Do I need to compile from your source tree?

Code: [Select]
Linking..
tcc: error: undefined symbol 'msg_queue_count '
  [E]  failed to link modules
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CoresNZ on August 02, 2013, 03:45:47 AM
Looking very promising thanks g3gg0, excited to see where this ends up!

Any chance of posting a compiled version of the mlv_dump utility?

I Would love to have a go at testing this end to end but I'm not set up to compile.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 02, 2013, 06:03:34 AM
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit

MLV_DUMP

BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CoresNZ on August 02, 2013, 07:54:14 AM
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit

MLV_DUMP

BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.

Cheers for that. yeah jumping the gun a bit, but I'll be prepped and ready for when we can convert them ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 02, 2013, 09:57:16 AM
did all that tonight.

 - recording a card spanning movie
 - concatenating using "cat M01-xxx.M01 >> M01-xxx.MLV"
 - converting to raw using "./mlv_dump M01-xxx.MLV out.raw"
 - converting to dng using "./raw2dng out.raw"
 - and finally making a movie from it

works without any problems here.

make sure you have the HEAD revision (the one from today 02 PM GMT+1)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 02, 2013, 10:43:29 AM
So, for now I have only a 15mb/s sd card so I can,t really try the capabilities here but gave it a go. When cardspanning is on I get about 7mb/s on the sd card and around 72mb/s on the cf card. With only cf I once got 84mb/s. Small test below. For now I get less recordingspeed with cardspanning on. Might change with a faster sd-card.

1920x1080 25fps

With filespanning of
about 592 frames
maxspeed on card 84mb/s

With filespanning on
74mb/s (cf-card)
7mb/s (sd-card)
400 frames

Thanks for your efforts g3gg0



Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 02, 2013, 10:52:30 AM
okay i see. its not that straight forward as thought.
thanks again.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 02, 2013, 11:31:00 AM
I tried it again and now it seems that its the other way around. It does give a little boost with the sd-card. However I think when my cf was warmed and suddenly gave me 84mb peaks that the cf itself gave longer recording times than with spanning on. I really like to try a faster sd, should speed up things considerably. This is a great speedboost. Good work
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 02, 2013, 11:00:43 PM
5D3 users:
can you test this (http://upload.g3gg0.de/pub_files/928d2e3d617d0617e27e2d6a84f7bb6e/raw_rec.mo) version for write speeds with spanning OFF and ON (see menu)
hacks both work on my 5D3, so please try with all hacks enabled.

you should see a clean write speed display for SD and CF (upper is CF, lower one is SD)

5D3,global draw OFF
 CF: komputerbay 1000x 64GB (not the fastest one, benchmarks up to 90-92MB/s), SD: Sandisk Extreme 45MB/s 64GB

Tried couple of times with 1920x1080p 30p
CF: write speeds: 83,5-84,5
SD: 17-18,5
The longest clip I managed to record was over 3000 frames, 1:43. But that was once and I didn't take screenshot. Normally many times i get over 1:10.
https://www.dropbox.com/s/id6jjgb6elrftpc/spanning%201.jpg
https://www.dropbox.com/s/asnel2b6tzb36o5/spanning%202.jpg

With "classic" build from 28th July i believe i was getting about 30 seconds or so 30p. MASSIVE IMPROVEMENT!!!

!!!  EDIT - OMG, 2560x1072 24p- previously 370-400 frames. Now i recorded 1:23, over 1900 frames  :D
It was saying continous OK but stopped after 1:23.
Second time i get 2,5K 2,39:1 continous ok - almost 5 minutes !!! and probably card just filled up.
Write speed: CF up to 93MB/s, SD about 17-17,5.

EDIT 2: 2880x1080 (2,67:1) - about 7 seconds, ~300 frames.

Seems 2,5K 2,39:1 would be fully usable  :D
Well...if someone finds a way to process the files.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: MA Visuals on August 02, 2013, 11:09:12 PM
g3gg0... I'm getting different results then the last couple of tests posted here.  For me it's slower for some reason.  Here are my results on 5D3 using your lastest raw_rec (with spanning support). Once the card was warmed up, I ran it 5 times for each test.

1920x1080 33 fps
Komputerbay 64gb 1000X CF
Sandisk Extreme 16gb

raw_rec V2 (CF card only)
177
173
210
157
173
Avg frames: 178

raw_rec V2 (w/Card Spanning to sd)
235
278
239
276
247
Avg frames: 255

Using most recent V1 raw_rec (commit 707c75f)
381
345
430
426
Avg frames: 383

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 03, 2013, 12:06:45 AM
Damn good tests.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 12:08:00 AM
thanks for testing.

here (http://upload.g3gg0.de/pub_files/c3915747a666422558475cc0d1eaab8f/raw_rec.mo) is some new version, but i think its not much better.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 03, 2013, 12:28:54 AM
@kgv5
Nice results. Could you try 50fps in 1920 mode.? Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 12:50:14 AM
Seems 2,5K 2,39:1 would be fully usable  :D
Well...if someone finds a way to process the files.

thanks for the tests.
as described above, it is already possible to process them.
concatenate them with any tool (even the file order should not matter)
and process them with mlv_dump
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 01:00:43 AM
@kgv5
Nice results. Could you try 50fps in 1920 mode.? Thanks

Sure.
canon menu - 1280x720 50p PAL

RAW (requires stretch 1.61x)
1920x672 16:9 360 frames
1920x646 1.85:1 380 frames
1920x544 2.20:1 continous OK

1856x650 16:9 460-480 frames
1856x624 1.85:1 700 frames
1728x606 16:9 continous

Frame numbers varies sometimes, these are more or less the best scores i get.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 03, 2013, 01:16:08 AM
Big thanks for testing slowmo.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 03:18:56 AM
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo

thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 03, 2013, 05:31:47 AM
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo

thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s

Wasn't the speed with CF cards 90mb/s+? Nevertheless its good progress. Should be plenty for 1080p etc.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 11:57:09 AM
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo

thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s

When i put the new raw_rec.mo to the ML folder (yesterday build) it is getting me an error (cannot load any module)., does it need other autoexec.bin?

Bug maybe: when i swith cam on, start shooting with spanning OFF, than turn spanning ON - it cannot initialize SD card. When I restart the cam and shoot spanning ON from the beginning - its OK.

EDIT
Yesterday build further testing:
1920x1280 3:2 24 and 25p - continous OK.
2048x1152 16:9 24 and 25p - continous OK
2048x1228 5:3 24 and 25p - continous OK >>> (should be great for some stabilizing in post and crop to 1920x1080)
2240x1120 2:1 24 and 25p - continous OK

When switching between resolution i get many err70, some examples:
https://www.dropbox.com/s/ezdpxgfoomya6g1/CRASH02.LOG
https://www.dropbox.com/s/oa44zy391pc2rx7/CRASH01.LOG
https://www.dropbox.com/s/2la6kiiacyaioji/CRASH03.LOG
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 01:18:09 PM
Which symbol is missing?
Enable console messages in module menu
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 01:25:02 PM
I just copy/paste .mo from the link to existing and working yesterday build. This is what i get when switch show console ON

tcc: error: undefined symbol 'edmac_memspy_res_unlock'
tcc: error: undefined symbol 'edmac_memspy_res_lock'
[E] failed to link modules

Every modules gives Err
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CoresNZ on August 03, 2013, 01:56:27 PM
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit

MLV_DUMP

BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.

Link dosn't work for me, no permission.

Anyone able to make MLV_Dump available please?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 03, 2013, 02:42:45 PM
Link dosn't work for me, no permission.

Anyone able to make MLV_Dump available please?

Working now.
but i still can't able to convert the out.RAW to .DNG with raw2dng...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 04:21:47 PM
http://upload.g3gg0.de/pub_files/92ff040055c8551996ef13e1be6c43b0/autoexec.bin
http://upload.g3gg0.de/pub_files/55f9003d60f27804ac0924fb50d14da0/5D3_113.SYM

here you have a recent 5D3 ML coer version.
so these errors should be gone
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 04:22:34 PM
but i still can't able to convert the out.RAW to .DNG with raw2dng...

are you just not able to do, or is there an error message..?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 03, 2013, 04:59:57 PM
raw2dng :
Code: [Select]
Error : This ain't a lv_rec RAW file.
and raw2dng for osx by scraxx tell me :
Code: [Select]
image2pipe demuxer @ 0x102021c00] Could not parse framerate: .
pipe:0: Invalid argument

i think this is a framerate detection problem.
i have the latest ml core version.

should i "make clean" mlv_dump?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: marekk on August 03, 2013, 05:46:51 PM
g3gg0, is there a possibility to read current white balance value from the camera and write it to a raw file ? It could improve postprocessing a lot. It's not a problem to edit it manually when we've got 10 raw files, but with 100.. it takes a lot of time..   
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: nandoide on August 03, 2013, 07:05:52 PM
Can anyone try how many seconds is it possible to record at highest resolutions  3k 3.5k at 24fps and 18-20 fps witnout frame skip?
Thanks a lot
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 07:10:43 PM
Its not long, 3,5k 24p is still 1-2sec, 15 gives couple seconds more.

I checked the newest version:
3,5K 3584x1320
23,976 - 1 sec (35 frames) (write speed CF 81-81, SD 17-17,5)
20 fps       - 1,5 sec (39 frames)
18 fps    4 sec (70 frames)
15fps continous OK (write speed CF 101MB/s, SD 17,7MB/s)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 03, 2013, 07:26:51 PM
raw2dng :
Code: [Select]
Error : This ain't a lv_rec RAW file....
should i "make clean" mlv_dump?

cannot preproduce.
neither in CF-only mode nor in CF+SD file spanning mode.
do a 'gcc mlv_dump.c -o mlv_dump -I../../src' instead of make


Its not long, 3,5k 24p is still 1-2sec, 15 gives couple seconds more.

that is how many MiB/s? (CF+SD)
did you use the latest revision?

g3gg0, is there a possibility to read current white balance value from the camera and write it to a raw file ? It could improve postprocessing a lot. It's not a problem to edit it manually when we've got 10 raw files, but with 100.. it takes a lot of time..   

it's planned as many other things.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 03, 2013, 10:41:50 PM
cannot preproduce.
neither in CF-only mode nor in CF+SD file spanning mode.
do a 'gcc mlv_dump.c -o mlv_dump -I../../src' instead of make

I just tried to recompile with this command and still the same error in raw2dng.
could you upload your mlv_dump for testing please i don't know what's wrong  ???
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 10:46:56 PM
that is how many MiB/s? (CF+SD)
did you use the latest revision?

The newest version:
3,5K 3584x1320
23,976 - 1 sec (35 frames) (write speed CF 81-81,5 SD 17-17,5)
15fps continous OK            (write speed CF 101MB/s, SD 17,7MB/s)


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 03, 2013, 10:55:29 PM
Was this with all hacks enabled? (memory, small hacks and hacked preview - this is what I use for max speed).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 10:57:26 PM
5D3,global draw OFF
 CF: komputerbay 1000x 64GB (not the fastest one, benchmarks up to 90-92MB/s), SD: Sandisk Extreme 45MB/s 64GB

Tried couple of times with 1920x1080p 30p
CF: write speeds: 83,5-84,5
SD: 17-18,5
The longest clip I managed to record was over 3000 frames, 1:43. But that was once and I didn't take screenshot. Normally many times i get over 1:10.


Newest build: seem slightly slower, especially SD card
1920x1080 30p - recording 30-40 sec - write speeds CF similar 83-85, SD -slower 15-15,5 (previously 17-18,5)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 03, 2013, 11:00:57 PM
Was this with all hacks enabled? (memory, small hacks and hacked preview - this is what I use for max speed).

memory hack ON, small hacks ON. Preview tested both canon and Hacked - no significant difference


EDIT: I have done more testing in different resolutions and framerates, SD hardly ever exceeds 15-15,5 so even 3 MB/s slower than the previous build.
Problem with initializing SD card is gone.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 04, 2013, 02:27:49 AM
appreciate your work, g3gg0 - if you'd be "ein berliner" you'd get a beer or two :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 04, 2013, 02:51:01 AM
in 23.976 fps mode, i get 74 MiB/s + 17.8 MiB/s. several tries. sometimes my CF is slower, but i know it is a faulty card.

here (http://upload.g3gg0.de/pub_files/74a871a42c606402bcc7b52889228b5d/raw_rec_mlv.zip) is the complete build for 5D3 that gave me these results.
it is the latest version that also embeds level sensor data (if enabled in ML menu) and white balance info.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 04, 2013, 03:26:51 AM
in 23.976 fps mode, i get 74 MiB/s + 17.8 MiB/s. several tries. sometimes my CF is slower, but i know it is a faulty card.

OK, for 3,5K it is right, exceeds 18.
But try 1920x1080 29,97 or even 1920x1280 29,97. Its 15-15,5. Now replace raw_rec.mo for the one from reply #107,with this i am getting 18-18,5 with all the same settings and same SD card. 3,5K also 18-18,5.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 04, 2013, 08:18:08 AM
Oh wait. Does it say continuous ok?
In this case the cf card is preferred and sd might be slower.
In total the write rate should be the same.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mucher on August 04, 2013, 10:26:32 AM
i am wondering that it might be faster to create two different buffers for the twoslots or even two buffers to write to the same slot. just a mere thought on the write speed
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 04, 2013, 11:41:48 AM
two buffers wont give us any improvement imho.
to be correct, we already have n buffers, where n is up to 70 or so on 5D3.
and these buffers get assigned dynamically to CF or SD.


i know we can do another improvement.
right now i am playing with two buffering methods on both queueing and dequeueing.

some problem that is making it hard to get optimal results, is the non-censecutiveness of memory buffers.
we can allocate e.g. memory for 70 frames. but this memory is fragmented, it it not continuous.
this means we have some groups of 2, 8, 20 or any other number of frames (we call it slots btw) that are continuous.
but the best for writing speed is to write a consecutive block of nearly 32MiB at once.

buffer layout e.g.: [____] [__] [_____________] [________] [__]

queueing:
method a) pick the first available free buffer
this method walks through the list of slots and checks for a free one.
it does not care if the previous or next buffers are directly behind the free one.

method b) use alex' algorithm that tries to place frames so that we get the largest block with frames in order.
this was necessary for the old writer where the frames had to be in order

better method:
search for the largest free space and try to fill it up.
strategy behind: build the largest block with frames in any order so that we get highest transfer speed

buffer layout e.g.: [____] [__] [______x1234x___] [________] [__]
the fifth frame would get enqueued at x, even if there is a lot of other free space


dequeueing:
method a) there are two writer threads (CF and SD) that are listening on a queue.
dispatcher thread scans the buffers for the largest consecutive block and places a write job into the queue.
the next "free" thread receives this job and writes it, no matter if SD or CF.

method b) again two threads, but every thread has its own queue.
the dispatcher places the largest found block in the queue for CF, the faster device, and the smallest block in the queue for SD.
strategy: try to keep the CF thread busy with the largest blocks to achieve the highest write rate. SD doesnt suffer that much
from writing smaller blocks. (remember, ~32MiB writes give the fastest possible transfer rate. on SD the speed difference isnt that high with smaller blocks)

i think mehtod b) gives the best results.

but to get back to your idea. one possible improvement could be this:
make the dispatcher aware that it should only qeueue writes to SD from large blocks, if there is nothing else.
background: we dont want the slow SD to permanently disturb the queueing by building large blocks.

[____] [12] [3________] [________] [__]

in this case the SD algorithm will pick frame 3.
this could lead to such fragmentation:
[____] [12] [_____789_] [________] [__]

but CF would benefit from writing large blocks so SD would probably be just annoying.

so i am thinking of building a sorted list of buffers and their sizes.
SD will always try to clear the smallest buffer areas, where CF tries to clear the larger ones.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 04, 2013, 11:53:41 AM
Oh wait. Does it say continuous ok?
In this case the cf card is preferred and sd might be slower.
In total the write rate should be the same.

Nope, continous OK has never show up. With .mo from post #107 i am consistently getting faster SD write by 2-3MB (tested mostly 1920x1080 canons 29,97). Almost always recorded for more than 1 minute.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mucher on August 04, 2013, 03:36:08 PM
two buffers wont give us any improvement imho.
to be correct, we already have n buffers, where n is up to 70 or so on 5D3.
and these buffers get assigned dynamically to CF or SD.


i know we can do another improvement.
right now i am playing with two buffering methods on both queueing and dequeueing.

some problem that is making it hard to get optimal results, is the non-censecutiveness of memory buffers.
we can allocate e.g. memory for 70 frames. but this memory is fragmented, it it not continuous.
this means we have some groups of 2, 8, 20 or any other number of frames (we call it slots btw) that are continuous.
but the best for writing speed is to write a consecutive block of nearly 32MiB at once.

buffer layout e.g.: [____] [__] [_____________] [________] [__]

queueing:
method a) pick the first available free buffer
this method walks through the list of slots and checks for a free one.
it does not care if the previous or next buffers are directly behind the free one.

method b) use alex' algorithm that tries to place frames so that we get the largest block with frames in order.
this was necessary for the old writer where the frames had to be in order

better method:
search for the largest free space and try to fill it up.
strategy behind: build the largest block with frames in any order so that we get highest transfer speed

buffer layout e.g.: [____] [__] [______x1234x___] [________] [__]
the fifth frame would get enqueued at x, even if there is a lot of other free space


dequeueing:
method a) there are two writer threads (CF and SD) that are listening on a queue.
dispatcher thread scans the buffers for the largest consecutive block and places a write job into the queue.
the next "free" thread receives this job and writes it, no matter if SD or CF.

method b) again two threads, but every thread has its own queue.
the dispatcher places the largest found block in the queue for CF, the faster device, and the smallest block in the queue for SD.
strategy: try to keep the CF thread busy with the largest blocks to achieve the highest write rate. SD doesnt suffer that much
from writing smaller blocks. (remember, ~32MiB writes give the fastest possible transfer rate. on SD the speed difference isnt that high with smaller blocks)

i think mehtod b) gives the best results.

but to get back to your idea. one possible improvement could be this:
make the dispatcher aware that it should only qeueue writes to SD from large blocks, if there is nothing else.
background: we dont want the slow SD to permanently disturb the queueing by building large blocks.

[____] [12] [3________] [________] [__]

in this case the SD algorithm will pick frame 3.
this could lead to such fragmentation:
[____] [12] [_____789_] [________] [__]

but CF would benefit from writing large blocks so SD would probably be just annoying.

so i am thinking of building a sorted list of buffers and their sizes.
SD will always try to clear the smallest buffer areas, where CF tries to clear the larger ones.

There should be an arbitration system between CF write thread and SD write thread, I guess, and, possibly, a powerful one.

A sorting list should be a good idea. My wild thought is that it might be good to keep a list of RAW file's address, and let the CF/SD write threads to sort according the list. The question is how much RAW files they can fetch each time by each of them -- then there is a need of arbitration system.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 04, 2013, 09:47:11 PM
was there any discussion about handling of splitted files? in the first implementation its an unnecessary offset-wrapping to get the pics.  the better approach should be "while setting resolution and fps there should be calculated, how many pics could be saved into one file, subtracting one or two pics and we're safe. starting the next file.."

will just play with the new files next hours :) regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 04, 2013, 09:51:02 PM
This requires prior knowledge about whether the filesystem supports large files or not. Current approach does not require such knowledge (the writing algorithm figures it out on the fly).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 05, 2013, 12:47:42 PM
@a1ex and what about to fixing it to f.e. 2GB/File? Finally, you dont work with these files, but with the files after the conversion. So, simply speaking, does it matter, if bigger or smaller? (every file could get his own header - its logically the same as the first file with another picture-offset count)

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 05, 2013, 01:00:09 PM
the new .mlv approach supports both concepts.
implemented is none of them yet ;)

currently it will break after the 4G were reached and continue a new file.
the blocks might be corrupted in this case.

i prefer to check before recording, if >4GiB files are supported and start a new file in case.

every file gets is own header with a unique ID so you dont have to pick the files by name, but
the post processing tools can detect the sequence number and which video it belongs to - even if the filenames are changed or corrupted.
(e.g. due to copying it with copy stations)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: jc on August 06, 2013, 03:01:07 AM
Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?

Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?

Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!
Loving the new format proposal! James
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 06, 2013, 01:16:53 PM
mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 06, 2013, 02:16:49 PM
mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing
Can someone please provide and exe version for testing in windows?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 06, 2013, 04:35:55 PM
MLV converted to RAW with mlv_dump and RAW to DNG with raw2dng (one frame) :
https://docs.google.com/file/d/0B-tM9Z6JauKNSktDTGgxQktnM2M/edit?usp=sharing

ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 06, 2013, 04:42:46 PM
Can you keep the first post updated with downloads?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 06, 2013, 04:44:54 PM
Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?

this implies your request
- audio support

Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?

this format already is extensible and is designed to be extensible
some technical facts:
 - structured format
 - extensible layout
 - as a consequence, we can start with the minimal subset (file header, raw info and then video frames)

Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!

the first data field in the file is a version string.

sure you had a look at it? ;)
but thanks for the toughts.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 06, 2013, 04:56:15 PM
ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?

the converter into the legacy format converts into the legacy format ;)
this means, that you dont have all the nifty features of the new format.
so you will be able to use the new format with all existing tools.

yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.
or maybe someone else is willing to do that? combine mlv_dump with raw2dng to get a mlv2dng.

but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10

some warning:
the new version will not only read MLV, but also read the M00, M01 etc when available and merge that into the output.
but i realized that it has a >4GiB bug when converting to legacy format.
will fix that tonight.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on August 06, 2013, 05:19:50 PM
but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10

Oh yeah! thank you for that :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: John Kesl on August 07, 2013, 11:31:28 AM
I just don't know why you're arguing with me over this... 

There is a reason things work they way they do.  I'm not trying to annoy the developer and I think what he has been able to get out this camera to do is insane!

I'm not special at all, but I'm willing to offer my experience to the developers to help them shape this into something even more awesome.

Honestly, I could be like every other professional and say "psshhh... shoot with the F55", "Yea, but it'll make your camera explode"...

I even went as far as to test this on a paid, 5 day job... had it failed, I could have been sued.  But it worked perfectly, and I've been singing it's praises ever since!

It's great!  Amazing!  I can't believe this quality is THIS accessible for SO CHEAP!  But... the display covers actors faces, and sometimes the images look like vomit before you CC then in Resolve.  I think those issues need to be solved before spanning the data across 2 cards.

Why is that such a threat to you?

You went on a job with no backup? sheesh.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 07, 2013, 11:39:20 AM
can you stop that, please?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 07, 2013, 02:11:33 PM
You're  a professional who went on a job with no backup? Even saying that proves you aren't.

I didn't say I didn't have a backup if the camera failed... but you don't know if there are problems with the footage until it has been downloaded, converted, and loaded into resolve.  So... you can't check it until the end of the day.

Again, what is with the attitude?  I'm only asking/hoping for features to solve some issues we had.  These would benefit you too... even if you don't understand why.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 07, 2013, 03:00:41 PM
can you stop that, please?

last warning.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 07, 2013, 03:17:58 PM
So with mlv_dump, we're essentially just (as the name implies) dumping the additional information V2 is recording?

Is there a way to incorporate it back into the DNGs?  Will that come with mlv2dng, or will mlv2dng also remove the information?

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 07, 2013, 03:20:21 PM
Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 07, 2013, 03:28:00 PM
Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.

No.  I'm asking because I want to test it.  As a programmer, I'm useless... unless you want me to break a computer.  But, I follow instructions well and like to test things...

Wondering when I can get my hands on some metadata! :-)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 07, 2013, 03:34:15 PM
@alex:
haha. i am sure your sarcasm will be misunderstood.
waiting for a: "why are you doing it then?" a few posts later.

@AnotherDave:
let me cite myself:
yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 08, 2013, 02:51:48 AM
with the latest commits i added these features to mlv_dump:

 - compress frames using LZMA
 - decompress frames
 - convert bit depth (from 14 bits to anything less than 14 bits per pixel and back)

todo:
 - apply e.g. a log curve instead of just clipping
 - an intelligent reconstruction when upsamping bit depth


if both features are combined you get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.

this feature is better for archiving your footage.
converting back to e.g. legacy raw doesnt need any parameters - it will decompress and convert transparently without any additional parameter.

the LZMA support is tricky and not out-of-the-box yet.
download the latest 7z SDK (v9.2.2) and compile a static version of it using this makefile:

Code: [Select]
PROG = lib7z.a
CXX = g++
LIB =
RM = rm -f
CFLAGS = -c -O2 -Wall -c -D_7ZIP_ST

OBJS = 7zAlloc.o 7zBuf.o 7zBuf2.o 7zCrc.o 7zCrcOpt.o 7zDec.o 7zFile.o 7zIn.o 7zStream.o Alloc.o Bcj2.o Bra.o Bra86.o BraIA64.o CpuArch.o Delta.o  LzFind.o Lzma2Dec.o Lzma2Enc.o Lzma86Dec.o Lzma86Enc.o LzmaDec.o LzmaEnc.o LzmaLib.o Ppmd7.o Ppmd7Dec.o Ppmd7Enc.o Sha256.o Xz.o XzCrc64.o

all: $(PROG)

$(PROG): $(OBJS)
ar -q $(PROG) $(OBJS)

%.o: ../../%.c
$(CXX) $(CFLAGS) $<

clean:
-$(RM) $(PROG) $(OBJS)

then add the created lib7z.a in the raw_rec makefile
Code: [Select]
- $(call build,GCC,gcc mlv_dump.c -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) $(LFS_FLAGS))
+ $(call build,GCC,gcc mlv_dump.c lzma922/C/Util/7z/lib7z.a -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) -Ilzma922/C/ $(LFS_FLAGS))

then uncomment the #define MLV_USE_LZMA in mlv_dump.c and recompile using 'make mlv_dump'


examples:
Code: [Select]
# show mlv content (verbose)
./mlv_dump -v in.mlv

# will dump the first 123 frames into a new file
./mlv_dump -f 123 -o out.mlv in.mlv

# compress input file
./mlv_dump -c -o out.mlv in.mlv

# compress input file with maximum compression level 9
./mlv_dump -c -l 9 -o out.mlv in.mlv

# decompress input file
./mlv_dump -d -o out.mlv in.mlv

# convert to 10 bit per pixel
./mlv_dump -b 10 -o out.mlv in.mlv

# convert to 8 bit per pixel and compress
./mlv_dump -c -b 14 -o out.mlv in.mlv

# create legacy raw, decompress and convert to 14 bits if needed
./mlv_dump -r -o out.raw in.mlv

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 08, 2013, 03:14:29 AM
Ran some test with provided build ccec3eeb -aug 3rd
5D3
CF is Lexar Pro 1000x 32 GB
SD is Sandisk Extreme 45 MB/s 32 GB

Preview:Hacked, Memory hack: ON, Small hack: ON, Global Draw: OFF

1920x1080 at 29.27   CF at 87 to 90 and SD at 18 with 3% idle  Continuous OK
1920x1080 at 31 fps   CF at 88 to 90 and SD at 18 with 1% idle Continuous OK
1920x1080 at 32 fps   CF at 72 to 92 and SD at 18 with 132ms idle Continuous OK (sometimes)
1920x1080 at 33 fps   CF at 87-91.2-90 and SD at 18 with 1% idle to finally Skipped frames

1728x606 at 50p fps CF 78 to 81.1 and SD 16.6 at 40% idle  Continuous OK
1856x650 at 50p fps CF 80 to 82.2  and SD 16.6 at 100ms idle  Continuous OK
1920x672 at 50p fps CF 72 to 83  and SD 16.6 at 100ms idle  Skipped

1728x606 st 59.948  CF 64 to 74  and SD 16.3 at 100ms idle  Skipped

SPANNING OFF

1920x1080 at 29.27   CF at 82 to 91.7 Skipped

1728x606 at 50p fps CF 78 to 86.4 Skipped

This is very, very nice and practical way of reaching some new benchmarks.  Also it makes good use of available space on both cards!!

Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further. 

I am finally working on a Virtualbox after struggling for a long time with Cygwin.

I am not a programmer so do not know how to make the mlv_dump work!

I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o  Error1

Congrats G3ggo Excellent work!!!

Update:  Just read your new stuff...  Amazing stuff!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 08, 2013, 09:29:13 AM
thanks for your tests!
do you achieve same or lower rates compared to the old raw_rec version that alex made?

Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further. 

not necessary anymore with current mlv_dump

I am finally working on a Virtualbox after struggling for a long time with Cygwin.

good choice imho

I am not a programmer so do not know how to make the mlv_dump work!

I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o  Error1


basically you have to:
cd modules/raw_rec
make mlv_dump

then it should be done.

about the compile error - can you send all messages you get when compiling the module?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 08, 2013, 02:03:29 PM
do you achieve same or lower rates compared to the old raw_rec version that alex made?

Will run some test with spannig off to compare.  I can compile the new modules except the ime_base, so I will use the complete compile that you provided instead of todays.

about the compile error - can you send all messages you get when compiling the module?

Code: [Select]
[email protected]:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=ime_base    -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here
ime_base.c: In function 'ime_base_start':
ime_base.c:46:5: warning: incompatible implicit declaration of built-in function 'strncpy' [enabled by default]
ime_base.c:46:5: warning: pointer targets in passing argument 1 of 'strncpy' differ in signedness [-Wpointer-sign]
ime_base.c:46:5: note: expected 'char *' but argument is of type 'unsigned char *'
ime_base.c: In function 'ime_base_test_any':
ime_base.c:92:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alpha':
ime_base.c:96:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_num':
ime_base.c:100:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alnum':
ime_base.c:104:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_punct':
ime_base.c:108:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_math':
ime_base.c:112:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_file':
ime_base.c:116:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_init':
ime_base.c:179:5: warning: pointer targets in passing argument 1 of 'strcpy' differ in signedness [-Wpointer-sign]
In file included from ime_base.c:9:0:
../../src/dryos.h:547:14: note: expected 'char *' but argument is of type 'unsigned char *'
make: *** [ime_base.o] Error 1
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 08, 2013, 05:50:22 PM
do you achieve same or lower rates compared to the old raw_rec version that alex made?

Tested with 1920x1080 at 29.97 fps

TEST 1 Spanning enabled>

CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
.MLV 4194304 KB
.M01  4194304 KB
.M02  4194304 KB
.M03  4194304 KB
.M04  4074381 KB
.M05  29269 KB
Frame skipped and stop

SD is Sandisk Extreme 45 MB/s 32 GB
recorded 1 file
.M00 4194304 KB

Total recorded before stop 21.376884.849 bytes on CF plus 4.294.967.296 bytes on SD
Total is 25.671.892.992

I noticed that % Idle SD card began at 27% and continually went down to 2% when finally stopped.
Continuous recording OK was displayed until the idle of SD card went down to 2%, it then changed to estimated frames.
------------------------

TEST 2 Spanning OFF>

CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
0708.MLV 3179009 KB
0705.MLV  1809481 KB
0706.M00  347137 KB
0706.MLV  4194304 KB
0707.M00  1081573 KB
0707.MLV  41934304 KB
Frame skipped and stop
Total recorded before stop 15.161.159.680 bytes
-------------------------------

TEST 3 ALEX1 RAW_REC

CF is Lexar Pro 1000x 32 GB
Recorded files 8 files
0734.R00 1121697 KB
0734.RAW 4194304 KB
0735.R00 600729 KB
0735.RAW 4194304 KB
0736.R00 1327249 KB
0736.RAW 41934304 KB
0737.R00 342017 KB
0737.RAW 41934304 KB
frame skip and stop
Total recorded before stop 20.652.974.080 bytes
Frame skipped and stop
------------------------

In all test it is evident a slowdown of the write procedure until the frame skips and recording stops.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 08, 2013, 11:30:48 PM

Code: [Select]
[email protected]:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=ime_base    -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here

fixed, didnt notice due to dependencies for modules being odd.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: jc on August 09, 2013, 02:56:24 AM
Hi g3gg0

Many thanks for your reply!

>this implies your request : audio support
Nope, not quite .. My request was to keep just the first 5 seconds for sync support, not any more. On many cameras the data rate is too high for audio .. Tho if you mean that if audio is supported in the format then the format should be able to handle just 5 seconds then yes it does ( if however the audio time is locked to the same as length as the video capture time in format v2 two then not :-) )

>this format already is extensible and is designed to be extensible
Great but not what I was trying to imply, I meant an already existing format eg Message Pack
When you design any home baked serialisation format you might think you left room for extension but it lacks "arrays in arrays" or "a kv set in a kv set" or some other essential thing that's needed later on + you can use a pre written parser for the desktop app with no need to write your own .. If it was me designing the format I would use message pack, erlang binary terms or similar binary format

>the first data field in the file is a version string.
Oh soz .. Must haves missed it looking first time :-)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 09, 2013, 06:05:57 PM
There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 10, 2013, 12:51:36 AM
okay here (http://upload.g3gg0.de/pub_files/357f92f2cfa8cad36241a2d3b13d49d3/raw_rec_mlv.zip) is a new update to our raw_rec module for MLV writing.
can some of you with a 5D3 can test it again, please?

test procedure:
 * disable all modules except raw_rec
 * disable audio recoding (canon menu)
 * disable global draw (ML menu)
 * disable electronic level sensor (ML menu)
 * configure raw_rec module:
 *   Frame Skipping: Allow
 *   Buffer fill method: 0, 1, 2 - can you test all of them?
 *   Memory/Small hacks: On
 *   Card spanning: Off, On - can you test all of them?
 * Format CF card

 * Start recording
 * wait 10-20 seconds
 * write down total write rate (3rd line)

with my setup i get ~96 MiB/s (77.5 + 18.5) with fill method 2. the others are a bit slower unfortunately.


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Shield on August 10, 2013, 06:09:06 AM
Will gladly help out on testing once my 5d3 comes back from Canon (the SD card slot died!)   :'(
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 10, 2013, 10:07:56 AM
All tests 1920x1080 29,97p (CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)

Couldnt find electronic level off - where is it? (so its on during the tests)

Buffer fill method: 0 (there is no 1,2,3,  - there is 0,1,2)


Best: spanning on --85,5   17,5 get continous OK ----total (after 0:20 - 100,2)
         spaning off ----92 and more but many dropped frames, after 30sec reached ~96MB but about 100 dropped frames

Buffer fill: 1

Best: spanning on --85 17,9 get expected couple thousand frames ------ total (after 0:20 - 101,9)
         spanning off ---94 , many skipped frames afer 20sec

Buffer fill: 2

Best: spanning on ---85,1 17,3 get expected couple thousand frames ------ total (after 0:20 - 103,0)
          spanning off ---95 and after 20sec started to drop frames

EDIT

Additional test (spanning ON) >>> 2,5K   2560x1090   2,35:1
Buffer fill 0 - total 111,5 (average 93,3, 18,)frame skip OFF,  recorded 1:47
Buffer fill 1 - total 112 (94,1 17,7) recorded 3:23 and card full but was continous OK,
Buffer fill 2 - total 112,6 sometimes 113 (94,5 18,2) continous OK

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 10, 2013, 12:26:56 PM
thanks again, kgv5!

is the CF only write rate comparable to alex' version or is it far off?
(komputerbay cards are between 90 and 120MB/s afaik)

if i read correctly, you get 95MiB/s with CF only and 93MiB/s on CF plus 18 MiB/s SD?
if i enable spanning, my CF rate drops by a fre megabytes per second.
yours seems quite stable.

@all:
can some of the others provide results too?

hint:
chose a resolution so that you get dropped frames for sure - so you have max data to write
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Milan Skrob on August 10, 2013, 12:36:25 PM
My test of todays raw_rec module:
Cards: CF Komputerbay 64GB 1000x, SD Sandisk Ultra 30MB/s

Settings:
* disable all modules except raw_rec
* disable audio recoding (canon menu)
* disable global draw (ML menu)
* Frame Skipping: Allow
* Preview: Canon

span - on
1920x1080 - 25fps
buffer fill - 0
90 + 11 = 101fps

span - off
1920x1080 - 25fps
buffer fill - 0
98.5fps

-----------------
span - on
1920x1080 - 25fps
buffer fill - 1
91 + 11 = 102fps

span - off
1920x1080 - 25fps
buffer fill - 1
96.2fps

-----------------

span - on
1920x1080 - 25fps
buffer fill - 2
90 + 11 = 101fps

span - off
1920x1080 - 25fps
buffer fill - 2
97.5fps


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 10, 2013, 01:18:09 PM
Hi g3gg0

With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):

1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s

2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF  - Touched 88.2 mb/s

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 10, 2013, 01:20:59 PM
Hi g3gg0,

With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):

Could not find the 'electronic level off'

1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s

2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF  - Touched 88.2 mb/s

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

I was so excited to see the 5DM3 touch 102 mb/s my heart beat must have been about 1000 beats/ second  ;D

One thing, I did notice the camera was warmer in hands than usual & so was the CF card.

Also, noticed the DCIM folder was empty. The root generated the following files:

M10-1619.MLV/ M10-1619.M00 to  M10-1634.MLV/ M10-1634.M00 & M10-1618.MLV

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 10, 2013, 02:13:11 PM
Also, noticed the DCIM folder was empty.

The new module writes to card (SD and CF) root folder.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 10, 2013, 02:42:53 PM
Thanks. How do you convert to DNG from here onwards.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 10, 2013, 02:46:32 PM
raw2dng, son of batch, rawmagic.. take your pick
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 10, 2013, 02:52:30 PM
Test equipment:
5D M3
CF - 32Gb Sandisk extreme 60Mb/s
SD - 8Gb Kingston ultimateX 100x

Preview: none (global draw off and still in magic lantern preview mode, not Canon)
Used all settings from g33g0's description above
My camera did not feel warmer than usual, which is not hot at all.
Edit: 1920 x 1080p 24fps ofc

Buffer fill method 0:

Spanning ON:

#1 - 56 + 14  = 70 mb/s
#2 - 55.8 + 14.1 = 69.9 mb/s
#3 - 56.2 +14 = 70.3 mb/s

Spanning OFF:

#1 - 59.7 mb/s
#2 - 58.7 mb/s
#3 - 58.8 mb/s

Buffer fill method 1:

Spanning ON:

#1 - 56.6 + 14.2 = 70.8 mb/s
#2 - 54.5 + 14.3 = 68.8 mb/s
#3 - 53.2 + 14.2 = 67.4 mb/s

Spanning OFF:

#1 - 61.4 mb/s
#2 - 61.7 mb/s
#3 - 61.9 mb/s

Buffer fill method 2:

Spanning ON:

#1 - 57.6 + 14.2 = 71.8 mb/s
#2 - 57.9 + 14.2 = 72.1 mb/s
#3 - 57.2 + 14.3 = 71.5 mb/s

Spanning OFF:

#1 - 61.9 mb/s
#2 - 61.8 mb/s
#3 - 60.2 mb/s


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 10, 2013, 03:44:44 PM
CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB
1920x1080 at 32 fps for frame skiping

There is a lot of variability so I set up for three consecutive test in the following sequence:

TEST 1.
Format CF  Buff=0  and spanning OFF -- Run test
Format CF  Buff=0  and spanning ON -- Run test
Format CF  Buff=1  and spanning OFF -- Run test
Format CF  Buff=1  and spanning ON -- Run test
Format CF  Buff=2  and spanning OFF -- Run test
Format CF  Buff=2  and spanning ON -- Run test
TEST 2
Same as above
TEST 3
Same as above

TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.

Buff=0  - spanning = OFF
test 1 = 91.9-93.3
test 2 = 90.3-91.6
test 3 = 92.1-93.4
Average Min -Max= 91.4 - 92.7

Buff=0 - spanning = ON
test 1 = 100-100.8
test 2 = 97.1-99.6
test 3 = 98-100.3
Average Min -Max= 98.3-100.2

Buff=1  - spanning = OFF
test 1 =90 - 91.5
test 2 = 90 - 91.8
test 3 = 89.7 - 91.7
Average Min -Max= 89.9-91.7

Buff=1 - spanning = ON
test 1 = 99.7 - 100.9
test 2 = 101 - 101.8
test 3 = 97.2 - 100.3
Average Min -Max= 99.3-101

Buff=2  - spanning = OFF
test 1 =92 - 93
test 2 = 89.2 - 92.3
test 3 = 91.3 - 92.1
Average Min -Max= 90.8-92.5

Buff=2 - spanning = ON
test 1 = 100.6 - 100.7
test 2 = 101.3 - 101.9
test 3 = 101.1 - 101.9
Average Min -Max= 101-101.5

My conclusions. 
With spanning the clear winner is Buff 2
Without spanning  maybe buff 0 or buff 2
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 10, 2013, 05:53:20 PM
I checked again buffer fill method = 2, this time with RenatoPhoto way:
(CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)

For 1920x1080 32fps

Buff=2 - spanning = ON
test 1 = 100.8 - 101.1
test 2 = 101 - 101.2
test 3 = 100.9 - 101.3

What is interesting for 29.97 fps i get better write speed:

Buff=2 - spanning = ON
test 1 = 102.2 - 103.3
test 2 = 102.7 - 103.5
test 3 = 101.9 - 102.8

EDIT
The highest speed so far i get with 2880x1226 24fps (with frameskip ALLOW) total 114,9  (95,8, 18,5), this was after ~ 1 minute recording, ~400 skipped frames.

Seems like bigger frames with lower frame rate gives better performance.
EDIT2:
Checked 2880x1206 (2,39:1) but 20fps - recorded 45 seconds before frame skipped, max total write speed 116,3 (97,5, 18,6)

EDIT3: ok, ultimate test: 3584x1320 (max resolution) - for 15fps - normally 119,2-119,6, touched 120,3 MB/s  121,7 MB/s :o
https://www.dropbox.com/s/c3118snkq0uq5ny/Spanning%203-5K%2015%20fps.jpg
Recorded about 50 sec.
I am starting to worry about my cards  :D

EDIT 4: Recorded 45 sec of 2240x1260 (16:9) 24fps at 112,5-113 total.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 10, 2013, 08:04:47 PM
raw2dng, son of batch, rawmagic.. take your pick

Sorry for being dumb. Cannot convert the MLV or M00 files to dng with raw2magic. Could please explain how to do that?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 10, 2013, 08:14:20 PM
Update: (see first post)

The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)

see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.

if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built (http://upload.g3gg0.de/pub_files/5795906e2820ed5c73514ed4311dbcd4/raw_rec.zip) ones. (5D3)
use it on your own risk.


if you can show the mlv_dump logs of your cameras, it would help me a bit.

Found this. Is there a binary for this code yet?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 10, 2013, 09:22:03 PM
@all:
thanks for all the reports!

@Canon eos m:
not for windows, just a linux build.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: morphers on August 10, 2013, 11:02:15 PM
I can't seem to figure out how to record at 2.5K It says it is not possible?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: jc on August 11, 2013, 12:49:36 AM
There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.

A)My request was that the format allows separate audio length to video length, not the 5 sec feature it's self
B)My second point was that using a pre-established serialisation format would ease things greatly in the future. Assuming the needed number of float or int to string conversions per second is not beyond what the processor can do ontop of the copying work etc.. Just using json would even work just fine.. :-) J
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 12:58:06 AM
here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 11, 2013, 01:10:17 AM
here (http://upload.g3gg0.de/pub_files/093cc390f95aa89924d529cbc7734c77/mlv_dump.exe) is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?

So is this available with other cameras such as the 5D2?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 01:40:39 AM
updated the DL link of mlv_dump.exe and fixed a memory access error that triggers on bit depth change

5D2: not yet, didnt look for the new symbols yet.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 02:04:14 AM
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin)  symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)

can you try it?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on August 11, 2013, 03:46:59 AM
Possible to try on the 7D?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 05:00:59 AM
Cannot figure out the command chain:

C:\
C:\ cd desktop
C:\desktop\cd (where the mlv_dump.exe and *.mlv files are saved)
C:\desktop\director with mlv_dump\mlv_dump.exe m10-1619.mlv

Nothing happens - what do I type in the last line?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 11, 2013, 05:05:40 AM
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin)  symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)

can you try it?

What raw_rec.mo should we use with this? I just tried the one from post #200 in this thread and it freezes the camera. I am currently using the 2013/08/09 build from the 5D2 raw thread.

Here you can see how it freezes:
https://plus.google.com/117039326451584292689/posts/dYB7vPW2FsR
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: morphers on August 11, 2013, 05:34:42 AM
What is the command to dump an mlv file to raw file for the windows build?

here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 06:01:23 AM
Hi g3gg0

With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):

1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s

2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF  - Touched 88.2 mb/s

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

Small correction (typo error  - by mistake mentioned all buffers fills as 0; whereas, these were 0, 1, 2 in that order):

3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON  - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING

4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 1; Card Spanning: ON  - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING

5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 2; Card Spanning: ON  - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING

Agree with kgv5 that buff 0 and 2 produce the fastest write speeds.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 11, 2013, 07:49:00 AM
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin)  symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)

can you try it?

Well it seems like a user already did. To repeat what he said: it froze. Sorry I couldn't test, I was at traveling 300 miles by car 700 miles by plane... Will test all new builds posted. Thanks for the hard work.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 09:46:35 AM
5D2:

can you try this module version if it is any better?
http://upload.g3gg0.de/pub_files/c73635e82eca629043f7f5d442e82265/raw_rec.mo
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 11, 2013, 01:16:00 PM
Tried to get the mlv_dump for windows through parallels 8, but I couldn't get it working.
Has somebody been more successful than me?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 02:52:23 PM
mlv_dump.exe works in windows with comand:
mlv_dump.exe -o mlv.raw m10-2205.mlv

Where mlv.raw is the name of the new raw file
m10-2205.mlv is the name of the spanned file, it automatically finds the others i.e. M00

How to make it work:
copy the mlv_dump.exe to the directory where the .mlv file is located.
To excecute this command use the dos terminal, navegate to the same directory and type:
mlv_dump.exe -o mlv.raw m10-2205.mlv with your own file names.

It joins the spanned file.....

PROBLEM:   I cannot make the raw2dng.exe (old) work.  I receive the:
This ain't a lv_rec RAW file
message...  when doing it in windows under dos command
raw2dng.exe mlv.raw

I have also tried in linux but I have a virtual disk size which does not allow me to work with the large files, so I can not use the linux commands until I successfully re-size my virtualbox machine.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 02:57:38 PM
Holy crap!!  New compile 8961e22b getting 15% more speed.

Both CF and SD write faster!!!!!!  Getting 110 to 115 MB/s with hacks on.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 11, 2013, 03:17:09 PM
 :o ok, where is it?
110 to 115 with 1920x1080 32fps??
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 03:33:58 PM
You forgot to add -r to write it as legacy raw.
See the help.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 03:42:22 PM
g3gg0,

Sorry to be so difficult, but could you please post the exact command for a windows machine:

Would it be: mlv_dump.exe -o mlv.raw -r m10-2205.mlv ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 03:45:32 PM
Uhm i am on mobile only.
But i am so confident that i already posted an excessive list of commandline examples...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 03:53:45 PM
Holy crap!!  New compile 8961e22b getting 15% more speed.

Both CF and SD write faster!!!!!!  Getting 110 to 115 MB/s with hacks on.

Got damn, holy balony. Gotta test this one. g3gg0 rocks this joint
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 04:01:16 PM
Would it be: mlv_dump.exe -o mlv.raw -r m10-2205.mlv ?

I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv

and ....  made the mlv.raw file

and ... drag and drop on raw2dng.exe and Yes!!!!!!!

Got dng files!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 04:12:31 PM
@RenatoPhoto, Where do you find this build? do you hit the extra megabytes filming raw normal 1920x1080 or in 3x crop?
Thanks
//D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 04:17:56 PM
In linux type

./mlv_dump -h and get the usage RULES

Usage: ./mlv_dump [-o output_file] [-rscd] [-l compression_level(0-9)] <inputfile>
Parameters:
 -o output_file      set the filename to write into
 -v                  verbose output
 -r                  output into a legacy raw file for e.g. raw2dng
 -b bits             convert image data to given bit depth per channel (1-16)
 -f frames           stop after that number of frames
 -c, -d, -l          NOT AVAILABLE: compression support was not compiled into this release
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 04:22:08 PM
@RenatoPhoto, Where do you find this build? do you hit the extra megabytes filming raw normal 1920x1080 or in 3x crop?
Thanks
//D

I compiled it myself.  Finally able to do this with VM machine see> http://www.magiclantern.fm/forum/index.php?topic=7579.0

Yes get 15% more on cropped video mode!!!!!!!!!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 11, 2013, 04:27:38 PM
5D2:

can you try this module version if it is any better?
http://upload.g3gg0.de/pub_files/c73635e82eca629043f7f5d442e82265/raw_rec.mo

It seems to behave in the same way as the other one for me.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 04:28:19 PM
@RenatoPhoto. Sweet. Did you by any chance tried this on 50fps?
Thanks
//D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 04:36:47 PM
I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv

and ....  made the mlv.raw file

and ... drag and drop on raw2dng.exe and Yes!!!!!!!

Got dng files!

I cracked it too. 1973 DNG files at my disposal from a single shoot. Awesome! Long live g3gg0.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 04:44:15 PM
@RenatoPhoto. Sweet. Did you by any chance tried this on 50fps?
Thanks
//D
Yes 1920x672 get 103 to 106 MB/s  I get 1800 to 2600 frames

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 04:51:16 PM
Yes 1920x672 get 103 to 106 MB/s  I get 1800 to 2600 frames

Fantastic!
I, m not a programmer. Could youupload the compile or do I wait for g3gg0 uploading?
Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 04:58:37 PM
I cracked it too. 1973 DNG files at my disposal from a single shoot. Awesome! Long live g3gg0.

In case anyone is wondering what this is all about.

A few days ago, g3gg0 has coded a new format such that the 5D Mark III can harness the power of the CF + SD at the same time for speeds touching 102-110 mb/s for long record of full HD (2.5K) RAW video that was previously impossible even with a fast card unless one was willing to live with dropped frames or compromise on the file size. The files are saved in a new format : MLV (Magic Lantern Video)

The Windows workflow (cannot say for Apple OS, Linux):

1. Download the mlv_dump.exe file from post #228 #218 of this thread. Unzip and extract.

2. Then bring the files recorded in the mlv format (& its shadow in *.m00 format) to the same directory as the unzipped mlv_dump.exe folder (to learn how to record in the new mlv format please scan through g3gg0's posts in this thread these past few days). Note: To begin, you will first have to download the latest modules and autoexe.bin files for recording in mlv format (post #200). The new build is also in one of g3gg0's posts

3. Each footage will be generated in two formats (*.mlv and *.m00 extensions).

4. Then in C:\ type the following thread: cd [with name of main folder (root folder), example, "cd desktop" (without the inverted commas)]
Then type the name of the folder next in the path: Same "cd XXXX"

5. Repeat this process till you reach the folder where the "mlv_dump.exe" file stored

6. Lastly, type:

mlv_dump.exe -o mlv.raw -r m10-2345.mlv
Note: In your case the "m10-2345" will be replaced with the name of your mlv file.
Note: mlv-dump will pick up the *.m00 on its own. The process will be slow - be patient

8. Then the file that is generated is to be brought over the raw2dng.exe file (also in the same folder) to generate DNG's
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 11, 2013, 05:06:43 PM
Latest compile: https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing  (https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing)

ps. not responsible for anything. 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 05:07:03 PM
Thanks Canon eos m. Will try it out. The version you,re referring to, Is it the upload from the post 200?
THanks
//D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 05:07:10 PM
The damn thing is so awesome! I just pulled out one frame from the 1973 frames that were generated. Ran it through the ACR 8.1 and used the 'Flatz1' preset that I got from www.dlwatson.net. The result just without anything else straight from the camera (with a few changes to exposure, contract, shadows) is so amazing. A result to die for.

The worry now is how to process such huge amount of RAW data!  :'(
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 11, 2013, 05:08:51 PM
Latest compile: https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing  (https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing)

ps. not responsible for anything.

Thanks! Noted, at my ow risk, of course :) Maybe I,ll try g3gg0 version first.
Thanks!
//D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 05:09:39 PM
Thanks Canon eos m. Will try it out. The version you,re referring to, Is it the upload from the post 200?
THanks
//D

No. mlv_dump.exe is @ #218
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 11, 2013, 05:10:37 PM
No. mlv_dump.exe is @ #218

Yes. The MLV build it at post #200
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 11, 2013, 05:15:19 PM
Heh, the compression kinda wins me over.

I added the msg queueu stub  for everything I have.. 50D consistently gets at least 80Mb/s with timers disabled on old raw_rec so its an easy test to see if speed is lost. I don't have a cam with dual slots.

Maybe it can be used to count MZ syncs too on cams like M and 600D where it fires too often.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 11, 2013, 07:32:56 PM
Ok I tried out the 5D2 version.

1) It saves to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at [email protected]
3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed (because it doesn't have a dual card slot, as you know)
5) I can't write any tag, no text display comes up.

Here's the quick 96 frame video I took (I have no idea how to process these):
https://dl.dropboxusercontent.com/u/814675/M10-2217.MLV

I labeled the above clip to be take #1...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 11, 2013, 07:40:08 PM
Quote
5) I can't write any tag, no text display comes up.

I think you need the IME module too.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 11, 2013, 08:02:50 PM
I think you need the IME module too.

Ok, where is the download located?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 11, 2013, 08:07:29 PM
I think you'd have to compile it or use the ones from 5DIII package? I'm not sure what/where g3gg0 released.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 11, 2013, 08:11:38 PM
I think you'd have to compile it or use the ones from 5DIII package? I'm not sure what/where g3gg0 released.

Well I'm an idiot when it comes to compiling, I'll guess I'll wait for gegg0 to send it...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on August 11, 2013, 08:13:08 PM
I added the msg queueu stub  for everything I have.. 5

msg_queue_count for 600D, is this 0xFF1F5C80?

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on August 11, 2013, 08:15:28 PM
It's all on the previous page...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 08:21:32 PM
msg_queue_count for 600D, is this 0xFF1F5C80?

it should be at (msg_queue_post + 0x40). at least on three different it was like that.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 11, 2013, 08:22:01 PM
Yes, thats right. I didn't push to the 600D repo yet.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on August 11, 2013, 08:26:11 PM
Yes, thats right. I didn't push to the 600D repo yet.

No problem, I compile it myself

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on August 11, 2013, 09:00:23 PM
doesn´t run on my 600D. The camera tells me an "file create error"

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 11, 2013, 10:34:58 PM
yeah, currently it is hardcoded for CF card cameras.
will change of course somewhen.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 12, 2013, 07:11:08 AM
Could someone here post a small mlv file from a 5D3? I need it to test code that I'm working on.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 07:25:58 AM
Could someone here post a small mlv file from a 5D3? I need it to test code that I'm working on.

I posted a 5D2 clip a few posts above...

Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 08:46:38 AM
I posted a 5D2 clip a few posts above...

Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?

Yes. Take the one from somempackage i uploaded the last time.
Modules are the same on every camera.

Early versions of ime had an error where the camera will suddenly pop up
The input dialog randomly after shooting.
This happens only when you entered something before.
Fixed in the latest releases.

If you have such a buggy version, just repower the camera after you enterd smth
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 08:47:38 AM
Early versions of ime had an error where the camera will suddenly pop up
The input dialog randomly after shooting.
This happens only when you entered something before.
Fixed in the latest releases.

If you have such a buggy version, just repower the camera after you enterd smth
[/quote]
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 09:07:44 AM
I'm gonna go out on a limb and say you used your phone ;D.

Thanks for the response, I'll get the module.

Again I have these bugs though, I don't know if you saw them earlier:

1) Videos save to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at [email protected]
3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed for the 5D2 version (because it doesn't have a dual card slot, as you know)
5) I can't write any tag, no text display comes up.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 11:38:26 AM
the latest IME modules can be found here (http://upload.g3gg0.de/pub_files/65d4d5cc43e5a2e6fe2069ba8cdd4546/ime_modules.zip).
(see also http://www.magiclantern.fm/forum/index.php?topic=6899.0)

the latest raw_rec module including win32 exe and static linux binaries is here (http://upload.g3gg0.de/pub_files/63bd95fd274e04f63b32164dfaa04ca3/raw_rec_mlv.zip).
no guarantee that the linux binaries will work on your system (it's a 32 bit ELF for 80386)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 11:43:25 AM
1) Videos save to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at [email protected]
3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed for the 5D2 version (because it doesn't have a dual card slot, as you know)

1) intention, its still experimental
2) this was on 5D2, right? can you use these (http://upload.g3gg0.de/pub_files/7678c52475d088e9dd4618d93b1203be/trace.zip) both modules (dont forget to enable trace) and send me your raw_rec.txt after recording with a spike? it should be in the root folder. upload it using http://upload.g3gg0.de and send me the [D/L] link (right-click, copy link)
3) if you stop recording, it should disappear
4) uhm, just dont enable it? ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 12, 2013, 12:34:01 PM
.mlv recording and IME modules are working on the 50D but I can't find a way to exit IME when I've input text.

I did a couple of quick tests on a 600x card at 1920x1080p (Global Draw OFF)

old raw_rec = ~344 frames
new raw_rec = ~288 frames

Buffer fill method doesn't have much effect, maybe 2 - 3 frames variance. Fill method 2 gets most frames.

.mlv files convert ok (Windows). They are written to root directory.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 01:47:56 PM
thanks for your test.
about IME: press Q to directly get to the OK/cancel buttons.

raw_rec:the performance is a bit worse than the legacy raw_rec module, yeah.
alex spent a lot of time to get the last frame out of the buffering mechanism.
the new module needs such optimization too.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 12, 2013, 02:07:57 PM
thanks for your test.
about IME: press Q to directly get to the OK/cancel buttons.

raw_rec:the performance is a bit worse than the legacy raw_rec module, yeah.
alex spent a lot of time to get the last frame out of the buffering mechanism.
the new module needs such optimization too.

Thanks. It's good to see it working anyway even with the performance decrease.

Q doesn't seem to be working on the 50D but I'm using Tragic Lantern so it may have been remapped. I'll have to check.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 12, 2013, 03:22:29 PM
We don't really have Q, only joy center. Its acting kind of like set, have to check how its set up exactly.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 03:26:35 PM
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.

select:
Buffer fill method: 4
CF-only buffers: 3

this gives me close to 100MB/s - sometimes :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 12, 2013, 04:01:25 PM
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.

select:
Buffer fill method: 4
CF-only buffers: 3

this gives me close to 100MB/s - sometimes :)

This one is worse on the 50D. I tried 4/3 and various other settings and could only record 240+/- frames on the same card.

@1% - could we not use the PicStyle button for Q? It's not really needed for video and I think most users tend to set PS in the ML menu anyway.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 12, 2013, 04:23:51 PM
I posted a 5D2 clip a few posts above...

Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?

The file has no data in it (it's only 1 kB). But the same is happening for me now.

I just did a "clean install" on my 5D2, that is, I cleaned my CF card and installed the 5D2_Alpha_One package and updated it to the 2013/08/09 version. Then I installed raw_rec.mo from raw_rec_mlv.zip in reply #277 and autoexec and symbol from #221.

It gives me the same results as before. The camera is super slow for ~5 seconds after I press record and then fills the buffer and "stops", but in the raw rec menu it shows "stopping" forever and everything locks up after a few more seconds.

I tried turning on the debug trace, but I can't find the trace file. @g3gg0, are there any log files I can send you or something I can do to help with fixing this? Or are you still focusing on making the 5D3 version work good before looking into debugging other platforms?

Ps.
I still need a 5D3 file :) Anyone willing to upload one for me? It doesn't need to be large.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CoresNZ on August 12, 2013, 04:39:26 PM
Not having much luck converting these.

This is what I'm getting back from the dos command.

E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV
Operating mode:
 - Input MLV file: 'M13-0212.MLV'
 - Convert to legacy RAW
 - Output into 'mlv.raw'
Failed to open file 'M13-0212.MLV'

Any ideas why this would be failing to open?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 12, 2013, 05:27:54 PM
the latest IME modules can be found here (http://upload.g3gg0.de/pub_files/65d4d5cc43e5a2e6fe2069ba8cdd4546/ime_modules.zip).
(see also http://www.magiclantern.fm/forum/index.php?topic=6899.0)

the latest raw_rec module including win32 exe and static linux binaries is here (http://upload.g3gg0.de/pub_files/63bd95fd274e04f63b32164dfaa04ca3/raw_rec_mlv.zip).
no guarantee that the linux binaries will work on your system (it's a 32 bit ELF for 80386)

g3gg0 getting about 104.7 mb/s on my 5D Mark III (1920X1080p) with this raw_rec.mo version. Going to try the one you posted a while ago.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 05:39:24 PM
E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV

so your raw is in E:\ ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 12, 2013, 05:55:27 PM
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.

select:
Buffer fill method: 4
CF-only buffers: 3

this gives me close to 100MB/s - sometimes :)

Touched 106.1 mb/s with the version of raw_rec.mo

Card Lexar 32GB 1000X + 32GB Scandisk 95mb/s
Camera 2032X1040p
Buffer fill: 2
CF-only buffer: 2
No dropped frames
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 12, 2013, 05:56:53 PM
Not having much luck converting these.

This is what I'm getting back from the dos command.

E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV
Operating mode:
 - Input MLV file: 'M13-0212.MLV'
 - Convert to legacy RAW
 - Output into 'mlv.raw'
Failed to open file 'M13-0212.MLV'

Any ideas why this would be failing to open?

Do you have the M13-0212.M00 file in the same directory 'rawFiles'?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 06:18:55 PM
you have set cf only buffers to 2?
Should be better with 3
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 07:05:05 PM
1) intention, its still experimental
2) this was on 5D2, right? can you use these (http://upload.g3gg0.de/pub_files/7678c52475d088e9dd4618d93b1203be/trace.zip) both modules (dont forget to enable trace) and send me your raw_rec.txt after recording with a spike? it should be in the root folder. upload it using http://upload.g3gg0.de and send me the [D/L] link (right-click, copy link)
3) if you stop recording, it should disappear
4) uhm, just dont enable it? ;)

1)
2)

3) Even if I stop recording, the buffer status never disappeared. The rest of the info disappeared after a good 15 seconds or so.
4) I don't enable it but what happens if someone did? I realize this is only an experimental copy of the new MLV Format, I guess you'll just disable it in the final versions.

Will trace the bug with the lag spike.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 12, 2013, 07:06:08 PM
Newest module from #277
This gave me very good results, great work  :D   : Buffer fill method 3, CF-only buffers 2

1920x1080 16:9  29,97fps >>>   ~109-110 (91+19) so shows continous ok.
2240x1260 16:9  24fps      >>>   ~113 (95+19) continous ok
2560x1090 2,35:1 24fps    >>>   ~115 (96+19) continous OK

I tested buffer=3 CF=2 and buff=3 Cf=3, after couple tries i had an impression, that B3CF2 speed was more stable, B3CF3 had similar speed but more variations. But its hard to say really, maybe just coincidence, when tried again it was stable, even maybe a bit faster than B3CF2.


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 07:24:10 PM
Newest module from #277
This gave me very good results, great work  :D   : Buffer fill method 3, CF-only buffers 2

1920x1080 16:9  29,97fps >>>   ~109-110 (91+19) so shows continous ok.
2240x1260 16:9  24fps      >>>   ~113 (95+19) continous ok
2560x1090 2,35:1 24fps    >>>   ~115 (96+19) continous OK

I tested buffer=3 CF=2 and buff=3 Cf=3, after couple tries i had an impression, that B3CF2 speed was more stable, B3CF3 had similar speed but more variations. But its hard to say really, maybe just coincidence, when tried again it was stable, even maybe a bit faster than B3CF2.

These tests are crazy, 2.5k continuous, Jesus.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 12, 2013, 07:33:55 PM
How about making a benchmark test that checks the different methods and logs it?

In a similar fashion A1ex made when RAW video first came out.
A bunch of people can do the tests and you could merge the data into one to compare the differences.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 12, 2013, 07:43:12 PM
These tests are crazy, 2.5k continuous, Jesus.

Yeah, its hard to believe.

Just for testing: If someone is interested in 2,8k 2880x1216 2,39:1 but >>20fps<<   - managed to record for 45 sec.
with super wide aspect 2880x1080 2,67:1 with >>22fps<< i get similar ~45sec. With the same super wide aspect but 24 fps i am getting ~6-7 sec.

EDIT: I was wondering what is real crop factor for different resolutions like 2,5k and 2,8k. 3x crop is for 1920 res i think, so for 2,5k should be about 2,3x (just like bmcc) and for 2,8k is should be about 2x, am I right?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mucher on August 12, 2013, 07:47:59 PM
I have got the feeling that the SD speed looks a bit low.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 12, 2013, 07:49:45 PM
@g3gg0 what about a version freeze? i would begin to code a converter next days, but the things are changing almost daily. after this version freeze or a first reference mlv-file (with version-number) i could start..

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 12, 2013, 07:55:53 PM
I have got the feeling that the SD speed looks a bit low.

5d3 SD controller max speed is about 20-21 i think. It is no 40-ish like 6D or 650D but slower and current versions are reaching almost the top.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 08:00:04 PM
Whats the 5D2's theoretical hardware limit for its CF slot max speed?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 12, 2013, 08:07:31 PM
According to the chart from

http://www.magiclantern.fm/forum/index.php?topic=6215.0

it is 75 MB/s.

For 5D3 is 100 for CF +20 for SD so devs have managed to squeeze almost the full speed.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 08:09:04 PM
According to the chart from

http://www.magiclantern.fm/forum/index.php?topic=6215.0

it is 75 MB/s.

For 5D3 is 100 for CF +20 for SD so devs have managed to squeeze almost the full speed.

LOL I said that because that what the max speeds we are attaining with Raw Video. It doesn't mean its the actual limit.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 12, 2013, 08:20:24 PM
ok, sorry, my mistake.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 12, 2013, 08:32:16 PM
I tried some slowmo but only managed to squeeze about 75mb on cf with the latest compilation. Anybody tried it out? 50 fps 1920x672
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CFP on August 12, 2013, 08:34:16 PM
Whats the 5D2's theoretical hardware limit for its CF slot max speed?
I saw benchmarks of the 5D Mark II reaching write speeds slightly over 90 Megabyte/s, just like the 50D.
And the 5D Mark III even reached 130 Megabyte/s. Its SD-Card controller's hardware limit is 21 Megabyte/s.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 12, 2013, 09:05:17 PM
I saw benchmarks of the 5D Mark II reaching write speeds slightly over 90 Megabyte/s, just like the 50D.
And the 5D Mark III even reached 130 Megabyte/s. Its SD-Card controller's hardware limit is 21 Megabyte/s.

Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 12, 2013, 09:36:25 PM
New test with build 54a545720 TODAY from g3gg0 repository

CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB

TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.

For CF recording only @  1920x1080 at 29 fps for frame skipping
Tested all CF 1 through 9 with buffer fill method =  4

CF=9 
test 1 = 98.5-100.7   
test 2 = 98-101.5
test 3 = 99.1-100.3
Average Min -Max= 98.5-100.8

CF=8
test 1 = 99.101.6
test 2 = 100.1-99.1
test 3 = 102.9-102.4
Average Min -Max= 100.7-101

CF=7
test 1 = 100.1-101.9
test 2 = 100.9-102.1
test 3 = 101.6-101.3
Average Min -Max= 100.9-101.8

CF=6
test 1 = 101.9-102.7
test 2 = 99.5-101.6
test 3 = 97.7-100
Average Min -Max= 99.7-101.4

CF=5 
test 1 = 102.5-103
test 2 = 102.1-102.9
test 3 = 101.8-100.5
Average Min -Max= 102.1-102.1


CF=4
test 1 = 97.4-98.9
test 2 = 101.8-102
test 3 = 100.6-102.1
Average Min -Max= 99-101


CF=3
test 1 = 97.4-98.9
test 2 = 101.8-102 Cont rec ok for a little while
test 3 = 100.6-102.1
Average Min -Max= 99.9-101

CF=2
test 1 = 97.5-100.8
test 2 = 102.9-103-100 Cont rec ok for a little while
test 3 = 100.5-101.8
Average Min -Max= 100.3-101.9

CF=1
test 1 = 95.3-98.9
test 2 = 100.8-101
test 3 = 100-101.1
Average Min -Max=98.7-100.3


CF=0
test 1 = 95.9-100.5
test 2 = 98.2-100
test 3 = 102.3-102.6
Average Min -Max=98.8-101.03

My conclusions. For CF only seem like buffer  4,5,6,7,8 are the better with 5 maybe better than the others.  I repeated some of the  test and repeatability is not very good.
--------------------------

Next test Spaning On.  Too many options. 

zoom= 3x
Resolutions:  2048x1152  at 29.97

CF=0 B=4
test 1 = 113-116.2
test 2 = 114-116.2
test 3 = 116.1-116.2
Average Min -Max=114.4-116.2

CF=3 B=4
test 1 = 112.3-114
test 2 = 116-117.1 Cont rec ok for a little while
test 3 = 116.1-119.7 Cont rec ok for a little while - touched 120!!
Average Min -Max=114.8-116.9

CF=2 B=4
test 1 = 114.9-115.9
test 2 = 116.2-117.1 Cont rec ok for a little while
test 3 = 117-118.4 Cont rec ok for a little while - touched 123!!
Average Min -Max=116-117.1

CF=1 B=4
test 1 = 117-115
test 2 = 113-117.6
test 3 = 115.2-116.9
Average Min -Max=115.1-116.5

CF=4 B=4
test 1 = 116.7-117.7 Cont rec ok for a little while
test 2 = 116.8-117  Cont rec ok for a little while
test 3 = 118.1-118.3 Cont rec ok for a little while
Average Min -Max=117.2-117.6

Note:  At CF=5 and above the SD card does not initialize fast enough so recording stops quickly.

Best is CF=4 B=4

Congrats g3ggo.  Amazing, blazing, speeding machine!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CFP on August 12, 2013, 09:42:28 PM
Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.
The 5D Mark III has the same SD-Controller like the 600D. Why do you think the 5D Mark II should have a different CF-Card than the 50D?
Canon uses the same components for multiple cameras to keep the prices low.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 09:45:01 PM
@g3gg0 what about a version freeze? i would begin to code a converter next days, but the things are changing almost daily. after this version freeze or a first reference mlv-file (with version-number) i could start..

regards chmee

the file format was fixed before i ever started a line of code.
it didnt change since then, except two fields.

so: the converters even from the first day would have been able to read this format the raw_rec outputs today
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 12, 2013, 10:01:12 PM
ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 10:14:00 PM
ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.

regards chmee

what do you want to talk about mlv?
had the proposal two weeks in this forum and there was no discussion :)
just a few feature requests.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 12, 2013, 10:21:11 PM
i just propose to split the spanning-mode optimization from this thread - finally i will code and test when im back at home, and then i can ponder about the (pro's and hopefully not con's) of the fileformat.

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 12, 2013, 10:26:29 PM
@RenatoPhoto:

i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 12, 2013, 10:38:19 PM
@RenatoPhoto:

i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes

no, thank you!!

Just wondering if it would be a better test to turn off skipping frames.  Running each mode until the recording stops, running it again maybe four or five times and then obtaining the total GB recorded.  I suppose the best setup is the one that can record the most without skipping frames, so this test should be more applicable to the needs.  In most cases, if the recording speed is stable, then the fastest recording will give you the longest file.  The problem is that recording speed is not constant so maybe total GB of multiple recordings may be a more reliable number to find the best settings.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: CoresNZ on August 13, 2013, 12:29:53 AM
so your raw is in E:\ ?


Yes. I have an E drive with a folder caller rawFiles in it and I have dumped all the recorded files from both cards into it along with the mlv_dump.exe.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 13, 2013, 01:26:45 AM
Quote
Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.

50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 13, 2013, 01:49:21 AM
@RenatoPhoto:
here (http://upload.g3gg0.de/pub_files/3b98c60717a64e509a889ab81c4d34f8/raw_rec.mo) is a version with "test mode" that tries recording with iterating buffer count and method.
every test case is ran 4 times.
enable tracing and test mode from menu. spanning option is not changed during tests.

with my setup the frame count varies more than the buffering method option changes. :(
does it look better on your camera?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 13, 2013, 02:33:48 AM
Where is tracing?
edit: you mean Debug trace?

I am running it  and see:
test numb with variables M , B, W, F, and (speed)

What do they mean..

When does it stop?

My comments so far:
* Each test seems very short and only represents the beginning of recording.   In my recordings (not in this test) I noticed that some buffers and CF numbers begin recording fast and then slow down while other options begin slow and increase the recording speed over time.

Is there a log saved?

Looks good
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 04:55:52 AM
Here is my first version of mlv2dng. It's not finished yet and there is still a lot to be done.

What is not working:
Spanned files (could someone please supply me with some spanned files? I don't have a 5D3 to try it).
Preview image. (it shows random noise)
Dual ISO.
It only shows very limited amount of EXIF data, as I have still to do mappings from MLV to exif tags.
probably much more.

I have a lot of cleanup to do on the code before I submit it online. Will probably be online tomorrow or the day after.

Usage is the same as with raw2dng for the moment:
mlv2dng.exe file.mlv [prefix]
=> will create prefix000000.dng, prefix000001.dng and so on.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 13, 2013, 07:26:08 AM
50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.

Yeah I'm just saying that if anything the 5D2 should be just as fast, if not faster when it comes to write speed. Am I right?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 13, 2013, 09:42:50 AM
1)
2)

3) Even if I stop recording, the buffer status never disappeared. The rest of the info disappeared after a good 15 seconds or so.
4) I don't enable it but what happens if someone did? I realize this is only an experimental copy of the new MLV Format, I guess you'll just disable it in the final versions.

Will trace the bug with the lag spike.

No bugs for me.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 13, 2013, 11:22:29 AM
@gnarr:

thanks for supporting!
(it is crashing with no output on my pc)

things important to know (most of that is written already i think)
 - files are numbered in the file itself
 - there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
 - no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
 - all M00, M01, ... also contain a file header (mlv_file_hdr_t)
 -> you see no difference in files from card spanning and files that went over 4GiB
 - the initial blocks like IDNT, INFO etc are written in the first file before recording
 - some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
 -> so dont rely on exact positions in file
 - blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
 - look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
 -> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

will upload a file, takes some time.
it will appear here (https://drive.google.com/folderview?id=0BwQ2MOkAZTFHb3lmaTMzUV9uMWc&usp=sharing), when its done.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 13, 2013, 12:04:48 PM
(upload finished)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 12:09:24 PM
@gnarr:

thanks for supporting!
(it is crashing with no output on my pc)

things important to know (most of that is written already i think)
 - files are numbered in the file itself
 - there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
 - no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
 - all M00, M01, ... also contain a file header (mlv_file_hdr_t)
 -> you see no difference in files from card spanning and files that went over 4GiB
 - the initial blocks like IDNT, INFO etc are written in the first file before recording
 - some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
 -> so dont rely on exact positions in file
 - blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
 - look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
 -> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

will upload a file, takes some time.
it will appear here (https://drive.google.com/folderview?id=0BwQ2MOkAZTFHb3lmaTMzUV9uMWc&usp=sharing), when its done.

Thanks for the notes and the files :) I will test this when I get home frome work.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 01:46:45 PM
one question for you g3gg0.

Frame size in the raw info header shows 4797520 for this video, because the camera was configured for 2080*1318, but of course you can only record max 1920*1288, so the real real frame size is 4327680 (as reported by the video frame header). I was wondering if this was a bug or if it is supposed to be like this?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on August 13, 2013, 02:06:33 PM
g3gg0 and chmee you guys rock!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 13, 2013, 02:08:35 PM
Awesomeness x 10
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 03:01:22 PM
new version.

Now handles spanned files and hopefully won't crash :)

With spanned files, you just point to the first file and the program automatically detects if there are more files.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 03:09:56 PM
-> need to build an internal sorted list of all blocks before metadata can be assigned to a frame

I have still to do this. Currently it just writes metadata from the last info block that was read. But the whole metadata thing is largely unfinished yet anyways :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 13, 2013, 03:13:27 PM
gnarr - It's working here. Exif is showing correctly. :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 13, 2013, 03:47:13 PM
new version.

Now handles spanned files and hopefully won't crash :)

With spanned files, you just point to the first file and the program automatically detects if there are more files.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)

crashing still.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 13, 2013, 03:57:46 PM
@RenatoPhoto:
here (http://upload.g3gg0.de/pub_files/3b98c60717a64e509a889ab81c4d34f8/raw_rec.mo) is a version with "test mode" that tries recording with iterating buffer count and method.
every test case is ran 4 times.
enable tracing and test mode from menu. spanning option is not changed during tests.

with my setup the frame count varies more than the buffering method option changes. :(
does it look better on your camera?

how do I delete the recorded raw video from the CF card in the camera
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 13, 2013, 04:34:59 PM
How do I use mlv2dng.exe with dual iso? The files that after converting with mlv2dng still contain the alternating iso traces. Cr2hdr does not help with removing the iso lines.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 04:55:49 PM
it's not yet implemented for dual iso. You will need to go the mlv_dump -> raw2dng way.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 13, 2013, 04:58:54 PM
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 13, 2013, 05:06:43 PM
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.

Download Visual C++ redist from Microsoft: http://www.microsoft.com/en-us/download/details.aspx?id=30679

Download the 64bit version and if that doesn't work try the 32bit one. I had the same issue. Works now :)

Drag and drop works.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 13, 2013, 05:10:38 PM
it's not yet implemented for dual iso. You will need to go the mlv_dump -> raw2dng way.

Hi gnarr,

Does not work with raw2dng.exe

Actually, once the mlv2dng.exe breaks the file up into individual frames it is not possible to run these through raw2dng.exe. The files should be cured with cr2dng.exe. But since these individual frames are already dng neither of the 2 programs works with these individual frames.

Any idea how this could be cured.

Also, while recording with both CF+SD card is there a way to get both the *.MLV and *.M00 files both on one card? Right now the *.MLV files are saved by the camera on the CF card and *.M00 on the SD card.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 13, 2013, 06:20:38 PM
updated DL link in main post
http://upload.g3gg0.de/pub_files/68769d85bdb2f9753de068e9783f162a/raw_rec_mlv.zip


fixed:
 - mlv_dump: handling of INFO chunks (text and take number) when rewriting MLV file
 - mlv_dump: incorrect check of frame size when converting back to 14bpp
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 13, 2013, 06:36:29 PM
So, i take it this is for the 5DIII only.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 13, 2013, 06:38:53 PM
Its CF camera only right now I think. 50D should run it.


Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 13, 2013, 06:44:41 PM
My CF cards are quite slow anyway, 50D is the camera i want to sell amongst the 3.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on August 13, 2013, 06:44:50 PM
Its CF camera only right now I think. 50D should run it.

Confirmed working on 50D but I'm getting ~15-20% less frames ATM for the same write speeds as original raw_rec. Gnarr's mlv2dng app works too. Looking forward to when metadata is working.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 13, 2013, 06:46:25 PM
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.

I just uploaded a version compiled with GCC, so this should be fixed now.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on August 13, 2013, 06:53:34 PM
Quote
My CF cards are quite slow anyway, 50D is the camera i want to sell amongst the 3.

Vs 60D? I don't think it has anything going for it. Just a 600D with a better build and missing audio.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 13, 2013, 06:59:04 PM
What do you mean?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on August 13, 2013, 07:01:28 PM
He means that if any camera you should sell it should be the 60D not the 50D.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 13, 2013, 08:38:52 PM
Download Visual C++ redist from Microsoft: http://www.microsoft.com/en-us/download/details.aspx?id=30679

Download the 64bit version and if that doesn't work try the 32bit one. I had the same issue. Works now :)

Drag and drop works.

Thank you Andy600, it works now (with 32bit version)  :D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 13, 2013, 11:04:34 PM
Does anybody know if somebody is making a os x app?
or updating RAWMagic for MLVs?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 14, 2013, 09:26:21 AM
I know that mlv2dng is very much work in progress and just early stage but is there any possibility of batch file processing yet?
Did somebody succeed to implement this in any existing raw2dng solution?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: PressureFM on August 14, 2013, 09:39:36 AM
I know that mlv2dng is very much work in progress and just early stage but is there any possibility of batch file processing yet?
Did somebody succeed to implement this in any existing raw2dng solution?

You can actually run multiple copies of raw2dng if that is what you are asking?

I usually open 4 instances and convert that way.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on August 14, 2013, 12:21:26 PM
Yeah, Thats what i do, thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 14, 2013, 08:18:49 PM
Alex vs G3ggo compiles for speed without spanning.  In all of G3GGo builds used Buffer=4  and  CF = 3

Lexar Pro 1000x
Frame skip OFF to see who can record longer without dropping frames.

Test 1 1920x1080 at 29.97

Alex
97-102.8
102.4-103.4
102.9-103.7
Total GB recorded 18.7 GB

G3GGo
95-103
103.5-103.4
103.2 101.9
Total GB recorded 17.2 GB

Alex is slightly faster and more stable
----------------------------------------

Test 2 Cropped 3X 2048x1152 at 29.97

Alex
84-103.2 recorded 13 seconds
103-104.7 recorded 13 seconds
81-102.8 recorded 13 seconds
Total GB recorded 4.65 GB

G3GGo
80-103.7 recorded 8 sec
75-86 recorded 5 sec
76-95.8 recorded 5 sec
Total GB recorded 3.07

Alex is slightly faster and more stable
-----------------------

Test 3 1728x606 60 fps

Alex
82-90.3  recorded 15 seconds
80-91.8  recorded 17 seconds
75-90.6 recorded 115 seconds
Total GB recorded 4.91 GB

G3GGo
73-86  recorded 9 sec
78.6-88.5 recorded 11 sec
78-87 recorded 9 sec
Total GB recorded 3.17

Alex is faster and more stable

Hopefully the G3GGo can get up to Alex speed.  Of course with spanning enabled 5D3 can record significantly faster then Alex module.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 15, 2013, 03:45:39 PM
Could someone post a (or send me) short dual-iso mlv file?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mucher on August 15, 2013, 03:59:51 PM
I think their Q/DQ methods tie up the CPU too much, these tasks are really slow to complete, and on a slow cpu the things get even worse. There might be ways to free up the CPU from these seeking jobs to let it do what CPU is supposed to do -- like compressing files. I still sturbbonly believe that the DIGIC4 is fast enough to do something like /4 an interger. :D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Doyle4 on August 15, 2013, 04:55:13 PM
how do I delete the recorded raw video from the CF card in the camera

Use the File Browser in ML and select and highlight delete :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 15, 2013, 06:04:50 PM
I think their Q/DQ methods tie up the CPU too much, these tasks are really slow to complete, and on a slow cpu the things get even worse. There might be ways to free up the CPU from these seeking jobs to let it do what CPU is supposed to do -- like compressing files. I still sturbbonly believe that the DIGIC4 is fast enough to do something like /4 an interger. :D

The cpu is currently not involved at all.
The video is copied by dma controllers and they cant divide.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 15, 2013, 06:35:55 PM
Could someone post a (or send me) short dual-iso mlv file?

Dual iso support isnt implemented yet.
But here some example files: https://docs.google.com/folder/d/0BwQ2MOkAZTFHb3lmaTMzUV9uMWc/edit

Folder contains a spanning mlv, a compressed one, 10 bit uncompressed and compressed etc.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 16, 2013, 02:12:08 AM
I just updated mlv2dng.

Fixed / working:
Headers are now correctly matched to frames via timestamps.
Almost all available exif and ifd info should be included in dng's.
There shouldn't be any memory leaks.
file spanning.
Little bit of error output in case it crashes.

What doesn't work / todo:
preview image. (although I'm closer to getting it right now.. will be there soon)
converting mlv to mlv for changing file compression or bit level.
converting non 14 bit mlv to dng.
converting LZMA compressed files to dng
converting dual iso mlv to dng.
more detailed error logging.

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 16, 2013, 02:12:53 AM
Dual iso support isnt implemented yet.
But here some example files: https://docs.google.com/folder/d/0BwQ2MOkAZTFHb3lmaTMzUV9uMWc/edit

Folder contains a spanning mlv, a compressed one, 10 bit uncompressed and compressed etc.

Thanks :) this will help a lot with development.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 16, 2013, 02:38:02 AM
What doesn't work / todo:
converting mlv to mlv for changing file compression or bit level.

this is not needed, as it is the mlv_dump main use case :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on August 16, 2013, 08:53:11 AM
Is there a version yet for the 5D2 to test run and check? :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 16, 2013, 11:10:35 AM
5D2: no

5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 16, 2013, 11:57:24 AM
@g3gg0
(*) Format-Description - RAWI-Block - isnt it better to refer to the raw.h (https://bitbucket.org/hudson/magic-lantern/src/6f63166b128a0768dc6ed9089875459d929f4fec/src/raw.h?at=unified)?
(*) would it make sense to put a badpixel-block into? i havent seen it. "we" could write a camera-module to mark bad pixels via 100%/200%/400%-photoview, then saving their positions.. this data we could use in the converter-modules to recalculate the badpixels, taking into account, that they're static and marking has to be done only once..

Blocktype - BPIX
Blocksize - 10+bpixCount*4
bpix-count - uint16
array of (x/y) uint16/uint16

regards chmee 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Nikita_MV on August 16, 2013, 01:02:13 PM
For Time Lapse Interested RAW "video" High-resolution, the speed of the card is not as important (for example frequency of 1 frame per second). I can not understand now have that opportunity. Camera 60D.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 16, 2013, 01:42:14 PM
@g3gg0
(*) Format-Description - RAWI-Block - isnt it better to refer to the raw.h (https://bitbucket.org/hudson/magic-lantern/src/6f63166b128a0768dc6ed9089875459d929f4fec/src/raw.h?at=unified)?

right, but i wanted to copy-paste the raw structure into the mlv header file.
not sure if i should put the 'struct raw_info' into mlv.h too, refer to the raw.h or if
i should put the fields from 'struct raw_info' in the RAWI block in the same format.

(*) would it make sense to put a badpixel-block into? i havent seen it.
thanks for the hint.
at the moment there is a rudimentary method for detecting such pixels.
you can record an unlimited number of frames with lens cap on and allowed frame skipping.
using mlv_dump you can average all frames in the video (using './mlv_dump -a -o ref.mlv lenscap.mlv')
the output mlv is a single video frame with all noise (sensor and readout) and deviations (sensel offsets, dead sesels) averaged together.

using the subtract command (using './mlv_dump -s ref.mlv -o out.mlv in.mlv') , this reference frame can get subtracted from every single video frame.
this will eliminate some kinds of noise and will make your dead pixels dark.

this file we could use to reconstruct the dead pixels too.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 16, 2013, 01:44:02 PM
For Time Lapse Interested RAW "video" High-resolution, the speed of the card is not as important (for example frequency of 1 frame per second). I can not understand now have that opportunity. Camera 60D.

not yet. sorry.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 16, 2013, 01:59:44 PM
@g3gg0
i like the idea of simple subtracting - but every time you change the resolution, you have to think of recording the lenscap-video. is there any examplevideo of your approach? practically i think there are no more than ~10badpixels on whole picture - 15 minutes of marking them and having static data for every kind of video (luckily we have the sensor-offset to do the calculations)..

but in general - there's already this approach to start immediately. and its able to neutralize noise and other sideeffects. like it.

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 16, 2013, 04:57:53 PM
Yeah it might no be the perfect solution, but a good start.
As soon we know how bad pixels can be detected, we can specify the format.
Is a simple x/y coordinate of defective pixels enough?
Or do we have to save more?

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 16, 2013, 05:08:03 PM
5D2: no

5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)

 :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 16, 2013, 06:33:51 PM
i tried to find an example-tag inside a cr2-file - but -awakening! - there s no tag for it - so a logical conclusion is, a bad/hot-pixel-tag is only existent in DNG, because they're found/analysed only in the converter :) look into the  chdk-description (http://chdk.wikia.com/wiki/Badpixel_removal).. so, finally it makes sense to give an int8 to any pixelposition - and we try to use them if we know more about hot/dead/bad-pixel- for now it's ok if we tell

0 = defective (recalculate with neighbours-data - general approach if no info about kind of problem)

if we had any information about kind of badpixel and/or differing repairalgorithm
1 = hotpixel on hi-iso
2 = hotpixel on longexposure
4 = recalculate only red
8 = rec.e only blue
16 = rec. only green
and so on.. for now i dont know for sure.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: MA Visuals on August 16, 2013, 08:17:47 PM
5D2: no

5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)

Here are my test results using your latest compared to Alex's latest.  I ran each test 7 times and then removed the 2 outlier results...

  Buffer fill method: 0
  CF-only buffers:    3
  Memory hack:      On
  Small hacks:        On

1920 x 1080, 29.97 fps
----------------------------
  g3gg0   a1ex   g3gg0 
    CF        CF     w/Span   
   634     1417     3002     frames
   754     1446     3594
   633     1622     2563   
   663     1638     3006   
   781     1635     2603   

1920 x 672, 48.00 fps
---------------------------
  g3gg0   a1ex   g3gg0 
    CF        CF     w/Span   
   391      751       550   
   442      751       642
   392      755       634   
   389      680       682   
   439      745       684

For 29.97 fps and spanning enabled, I got double the frames of Alex's latest. Nice :).  But without spanning, I got double the frames using Alex's.
For 48.00 fps, Alex's latest always performed significantly better.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 16, 2013, 09:25:01 PM
Nice test! I really like the spanning feature and all the other possibilities of course adding the new format. Also very curious what spanning would accomplish in Alex,s raw format :).
Really nice to follow the skillfull developing going on.
//D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 17, 2013, 12:54:43 AM
another bit of performance increase, but not much.
here (http://upload.g3gg0.de/pub_files/8371bd0ed7e9de8c55985de2d838cccf/raw_rec.mo)
always try to use buffering method 4.

btw, couldnt get more than 1000 frames with alex code
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 17, 2013, 03:30:31 AM
another bit of performance increase, but not much.
here (http://upload.g3gg0.de/pub_files/8371bd0ed7e9de8c55985de2d838cccf/raw_rec.mo)
always try to use buffering method 4.

New code gave me slower results for 1920x1080, and cropped digital video 2048x1152 but slightly faster rates for high speed at 1728x606 at 60 fps.

Test 1 1920x1080 at 29.97

Yesterdays test G3GGo
95-103
103.5-103.4
103.2 101.9
Total recorded on three tests  17.2 GB

Today's latest code:
86-102
103-100
101-103
Total recorded on three tests  16 GB

Test 2 Cropped 3X 2048x1152 at 29.97

Yesterdays G3GGo
80-103.7 recorded 8 sec
75-86 recorded 5 sec
76-95.8 recorded 5 sec
Total recorded on three tests 3.07 GB

Todays G3GGo
86-96
101-102
78-96
Total recorded on three tests 2.39 GB

Test 3 1728x606 60 fps

Yesterdays G3GGo
73-86  recorded 9 sec
78.6-88.5 recorded 11 sec
78-87 recorded 9 sec
Total recorded on three tests  3.17 GB

Todays G3GGo
86-101
90-90.8
90-85
Total recorded on three tests 3.26

Slightly better speed on this videos.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 17, 2013, 03:04:12 PM
Why is Alex not involved in this thread btw?

It would make sense to involve him, doesn't it?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Michael Zöller on August 17, 2013, 03:33:33 PM
Premature optimization is the root of all evil... :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 17, 2013, 10:24:11 PM
mlv2dng previews are working  8)

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 17, 2013, 10:45:58 PM
really cool :)

btw, do you also get "failed to get properties" for serial numbers with the latest build?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 18, 2013, 12:46:01 AM
okay, fixed.
(here (http://upload.g3gg0.de/pub_files/db058a6bd0bc2c3e6a3ad1c5a9850c54/raw_rec_mlv.zip) with 5D3 autoexec/sym)
i played with edmac alignment in this version. please report if you get pink frames or lock ups.

@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)

in camera i wanted to prevent 64 bit integer to string conversion, so i decided to put it as hex string.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 18, 2013, 02:43:13 AM
@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)

done :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 18, 2013, 10:28:15 AM
okay, fixed.
(here (http://upload.g3gg0.de/pub_files/db058a6bd0bc2c3e6a3ad1c5a9850c54/raw_rec_mlv.zip) with 5D3 autoexec/sym)
i played with edmac alignment in this version. please report if you get pink frames or lock ups.

@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)

in camera i wanted to prevent 64 bit integer to string conversion, so i decided to put it as hex string.

Hi g3gg0 - quick question - downloaded the most recent build for raw - but am a bit confused - is raw2dng the same as mlv2dng? With the new autoexe.bin, raw_rec.mo, etc. will the camera retain the ability to shoot mlv or will it revert to the previous raw format?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 18, 2013, 10:56:37 AM
Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.

mlv2dng is released by gnarr in separate posts.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 18, 2013, 11:14:03 AM
Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.

mlv2dng is released by gnarr in separate posts.

Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on August 18, 2013, 11:15:23 AM
I understand its for 5D3 only atm, but when reading trough about it, will it work the same on the Mk2 or are there features that will be Mk3 only in the format?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 18, 2013, 11:18:19 AM
Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).

Now 102.5mb/s with skipping after 800 frames.

Also, how do I convert from MLV to RAW. I believe that the mlv2dng will convert directly to dng format?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 18, 2013, 12:02:51 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 18, 2013, 12:18:26 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?

Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Danne on August 18, 2013, 12:41:21 PM
I see, probably because I tried with some older files. Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Toffifee on August 19, 2013, 11:28:55 AM
Is there anything I can do to help the progress of this development?  :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 19, 2013, 09:15:38 PM
@g3gg0 and all mlv-devs I'm a little bit confused about the data inside/behind the first VIDF-Block. Why are there more other Blocks inside it? (This VIDF-Blocksize is that high, i cant iterate thru the blocks. What did i missed?

(http://s1.directupload.net/images/130819/mxxjozd7.jpg)
red = blocksize
green = blockposition
blue = framespace

Till the VIDF-Block i iterated thru the blocks via (blockPos+blockSize)=newBlockPos

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 19, 2013, 10:05:51 PM
Don't forget that framespace is a 32bit number, so it's 0x00000F7C.
So the actual frame size is blocksize - (framspace + 32) and you start reading the frame at the framespace offset.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 19, 2013, 10:25:58 PM
these blocks are leftovers from memory that hasnt been cleared.
it is just crap, ignore it.

when doing EDMAC copies of the video frames, the destination address must be aligned to 4k borders. (sometimes...)
thats what the frameSpace is for.
ignore anything what is in this space, just follow the block sizes.

btw: the blocks are called VIDF. VideoFrame
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 19, 2013, 10:46:53 PM
lol. of course vidf..

@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.

@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 19, 2013, 11:25:16 PM
@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.

uhmm. do not hardcode 14 bit, please.
the format supports 1-16 bit depth.
frameSize = (width * height * bpp + 7) / 8
the frame data starts at frameOffset bytes behind the VIDF header end.

see the VIDF block as one frame, header as described in mlv.h, some frame space for alignment and finally the data.
behind the frame data there may also be some empty space.
you have to use the RAWI information to get the real frame data size.
again, look into mlv_dump.c how i handle this.
@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well?
which pictureoffset?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 19, 2013, 11:34:14 PM
(which offset) the same as you described. to optimize edmac copies to the sd/cr-card a1ex (i think) moved/saved blocks. framesize!=dataframe

thx. regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on August 20, 2013, 06:14:46 AM
Is there any test footage I can play with? I looked through and found M13-0957.MLV but it doesn't seem to open with mlv_dump.exe
EDIT: Oops, I just found the MLV2DNG - works a treat!


Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 20, 2013, 09:41:58 AM
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..

the picture offset (=frameSpace) is at maximum 4k, where the videoframe is several MiB.

about the NULL data:
when there is some metadata like LENS, RTCI, EXPO etc, i am decreasing the frameSpace and moving the VIDF header to use that gained space for these blocks.
this allows me to put additional data without losing any write performance.

here some example:

Code: ("c") [Select]

assume a block with that size, which is quite good for writing
[empty......................... ....................... ............................ .............]

now it gets filled with a VIDF only, and some frameSpace for edmac and write alignment. size stays the same of course.

[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]

now there is some [LENS] to embed.
I remove all frameSpace to zero and put a NULL block with that size instead and insert the [LENS] block before.

[NULL.................................][VIDF__datadatadatadata........datadatadatadata][NULL......]
[LENS][NULL...........................][VIDF__datadatadatadata........datadatadatadata][NULL......]

Code: ("c") [Select]
now there is some [EXPO] and then [RTCI] to embed, too:

[EXPO][LENS][NULL...............][VIDF__datadatadatadata........datadatadatadata][NULL......]
[RTCI][EXPO][LENS][NULL.........][VIDF__datadatadatadata........datadatadatadata][NULL......]

later, when the frame was written and is going to be reused again, the header in front is written again

[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]


and you see that at the same position where [frameSpace...] is now, previously some other blocks like the LENS etc block have been.
i could clear this area to prevent irritation, but i wanted to save CPU load and thus kept the crap there.

to be honest, i could have implemented frameSpace also using NULL, as it is redundant. but i didnt want to touch that again.
in future releases frameSpace might be set to zero and alignments (both edmac and write) could be realized with NULL blocks only.

the only advantage with frameSpace in VIDF is that i dont have to seek while updating the VIDF header from vsync cbr. but thats not much.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 20, 2013, 09:46:52 AM
Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.

see some posts earlier, there is a package for 5D3
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: aaphotog on August 20, 2013, 10:01:13 AM
I see most of this thread is actually for those smart enough to do coding for it(not I)
So I have a couple of questions.
with the spanning to both cf and sd cards, I have to grab the recordings off of BOTH cards and use a software that will combine the information from both cards to make 1 video?
If so, is the software to do this and allow us to edit these videos and use them(like we can the preview .raw container) available and working? How about for a non programer like myself(running mac os x)?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on August 20, 2013, 12:27:55 PM
see some posts earlier, there is a package for 5D3

Found it, cheers!

I was hoping that LRTimelapse would help solve my auto exposure hard stopping problem, but it doesn't seem to have :/ Not sure why it's not working, I'd have thought it would have with you EXIF data in each file.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 20, 2013, 12:51:08 PM
hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?

i know that there will be a few frames delay with the metadata.
going to fix this soon.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on August 20, 2013, 01:12:02 PM
hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?

i know that there will be a few frames delay with the metadata.
going to fix this soon.

Will have a look tomorrow night! Keep up the great work g3gg0. This new format seems to be a great improvement, especially if Alex can implement the sidecar method that he has developed for POST FLICKER.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 20, 2013, 07:05:08 PM
@g3gg0 thx for the hints and infos.

thinking about the definition, it would be good to write down some restrictions and/or principles. (lot of that sounds logical, but it's worth to say!)

f.i.
(*) Fileheader (file_hdr) is the first block.
(*) info-blocks must be before the first VIDF-Block (f.i. except RTC)
(*) AUDF-block will be found after every VIDF-block. (or else?)
(*) importance-notation of every block-type (f.i. mandatory, recommended, optional)
(*) every VIDF-Block has its own header, likewise splitted VIDF-Blocks
(*) VIDF-Blocks are splitted bytewise

discussion sides before
(*) filesize is depending on fileformat, at least 4GB
or
(*) files will be splitted at 4GB regardless of fileformat

module raw_rec
(*) notation of coding-datetime/version, from when mlv was supported with all mandatory blocks.
(*) notation, when a blocktype was implemented.

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 20, 2013, 07:45:49 PM
Where can we download the latest MLV2DNG?  I can't find it anywhere... :-/
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 20, 2013, 07:57:39 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?

Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.

I see, probably because I tried with some older files. Thanks

This should now be fixed. Old versions of mlv files should be decoded properly.

Please report any errors you have :) Either with a PM or on this thread. (http://www.magiclantern.fm/forum/index.php?topic=7802)

https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)

and here is a linux version:
https://dl.dropboxusercontent.com/u/3181048/mlv2dng_linux.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng_linux.zip)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 20, 2013, 10:01:03 PM
Now all we need is one for all of us in Macland... :-/
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gnarr on August 20, 2013, 10:10:07 PM
as soon as I finish downloading an OsX vmware image I will compile and post it here :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 20, 2013, 10:20:53 PM
thinking about the definition, it would be good to write down some restrictions and/or principles. (lot of that sounds logical, but it's worth to say!)


yep, good point. let me paraphrase them.

[MLV Format]
 - the Magic Lantern Video format is a block-based file format
 - every information, no matter if audio or video data or metadata is written as data block with the same basic structure
 - this basic structure includes block type information, block size and timestamp (exception to this is the file header, which has no timestamp, but a version string)
 - the timestamp field in every block is a) to determine the logical order of data blocks in the file and b) to calculate the wall time distance between any of the blocks in the files
 - the file format allows multiple chunks which basically are in the same format with file header and blocks
 - chunks are either sequentially written (due to e.g. 4 GiB file size limitation) or parallel (spanning over mutiple media)
 - the first chunk has the extension .mlv, subsequent chunks are numbered .m00, m01, m02, ...
 - there is no restriction what may be in which chunk and what not

[processing]
 - to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
 - when sorting, the sorted data can be written into a XREF block and saved to an additional chunk
 - files that only contain a XREF block should get named .idx to clarify their use
 - do not rely on any order at all, no matter in which order they were written into a file
 - the only reliable indicator is the timestamp in all headers

[MLVI]
 - MLVI block is the first block in every .mlv file
 - the MLVI block has no timestamp, it is assumed to have timestamp value 0 if necessary
 - the MLVI block contains a GUID field which is a random value generated per video shoot
 - using the GUID a tool can detect which partial or spanning files belong together, no matter how they are named
 - it is the only block that has a fixed position, all other blocks may follow in random order
 - fileCount field in the header may get set to the number of total chunks in this recording

[VIDF] (periodic)
 - the VIDF block contains encoded video data in any format (H.264, raw, YUV422, ...)
 - the format of the data in VIDF blocks have to be determined using MLVI.videoClass
 - if the video format requires more information, additional format specific "content information" blocks have to be defined (e.g. RAWI)
 - VIDF blocks have a variable sized frameSpace which is meant for optimizing in-memory copy operations for address alignment. it may be set to zero or any other value
 - the data right after the header is of the size specified in frameSpace and considered random, unusable data. just ignore it.
 - the data right after frameSpace is the video data which fills up the rest until blockSize is reached
 - the blockSize of a VIDF is therefore sizeof(mlv_vidf_hdr_t) + frameSpace + video_data which means that a VIDF block is a composition of those three data fields
 - if frames were skipped, either a VIDF block with zero sized payload may get written or it may be completele omitted
 - the format of the data in VIDF frames may change during recording (e.g. resolution, bit depth etc)
 - whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks

[AUDF] (periodic)
 - see [VIDF] block. same applies to audio

[RTCI] (periodic, event triggered)
 - contains the current time of day information that can be gathered from the camera
 - may appear with any period, maybe every second or more often
 - should get written before any VIDF block appears, else post processing tools cannot reliable extract frame time

[LENS] / [EXPO] / ... (periodic, event triggered)
 - whenever a change in exposure settings or lens status (ISO, aperture, focal length, focus dist, ...) is detected a new block is inserted
 - all video/audio blocks after these blocks should use those parameters


...to be continued
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on August 20, 2013, 11:56:02 PM
Is an Alpha/beta in sight soon for the mark2?
Cant wait to try this new format out :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 21, 2013, 12:00:08 AM
not yet, sorry.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 21, 2013, 02:08:25 AM
How about making a benchmark test that checks the different methods and logs it?

In a similar fashion A1ex made when RAW video first came out.
A bunch of people can do the tests and you could merge the data into one to compare the differences.

Where can I find A1ex's version of the RAW video module (with the binary) or is there no such thing?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: SDX on August 21, 2013, 02:21:00 AM
I suggest you to start looking here: http://www.magiclantern.fm/forum/index.php?board=49.0
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kitp on August 21, 2013, 11:01:03 AM
Works great, thank you, I was able to unpack both to DNGs and to RAW. But I can't see wav file and it doesn't seems that sound is recording - how to switch it on?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ilguercio on August 21, 2013, 11:07:12 AM
Sound recording does not work(yet).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on August 21, 2013, 10:13:37 PM
[MLV Format]
..
 - there is no restriction what may be in which chunk and what not

[processing]
 - to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
..
 - do not rely on any order at all, no matter in which order they were written into a file
 - the only reliable indicator is the timestamp in all headers

[MLVI]
..
 - it is the only block that has a fixed position, all other blocks may follow in random order

 - whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks

...to be continued
@g3gg0 excellent work. i quoted some points i'm not that sure.. i'm a little bit scared about seeking thru the chunks, finding all important info blocks. it's also possible, that in spanned-mode f.i. idnt or wbal could be found in the second file - thats the freedom of the format-description - am i right?

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 22, 2013, 02:04:33 AM
it's also possible, that in spanned-mode f.i. idnt or wbal could be found in the second file - thats the freedom of the format-description - am i right?

thats the point. it might be at the end of the last file or in the middle of the first. anywhere.

this is the reason why there is a XREF block (defined) which contains file number and offsets to the blocks.
the entries in this block are in order, lowest timestamp first, highest timestamp as last.
it looks like this:

Code: [Select]
[XREF]
 (fileNum) (fileOffset)
 0          0x100
 0          0x120
 0          0x1A0
 1          0x80
 0          0x180
 1          0xA0
...

which means: first block is in file 0 at offset 0x100, second in file 0 at 0x120, next in file 0 at 0x1A0, next is in file 1 at offset 0xA0.
... and so on.
of course tools have to build this index and (optionally, but recommended) save it into another file called .idx

if you have this index, reading in correct temporal order, is just going throug that XREF list.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 22, 2013, 11:01:50 PM
Ran some video test which I converted directly to dng mlv2dng.exe

1920x1080 at 29.97 - Perfect
Cropped 1920x1080 at 29.97 - Some pink frames and pink noise frames
60 fps 1600x560 at 60 fps - Some pink frames and pink noise frames
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 23, 2013, 12:13:10 AM
same pink frames with mlv_dump and raw2dng?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 23, 2013, 12:42:22 AM
same pink frames with mlv_dump and raw2dng?
Yes, exactly the same frame number if processed with mlv2dng or mlv_dump + raw2dng
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dariSSight on August 23, 2013, 05:01:17 PM
same pink frames with mlv_dump and raw2dng?

I was checking out the Pre-Built Module of MLV, I only use the rec-mo file because I work on a Mac and the exe files are irrelevant to me on a Mac, but when I use the module on my Canon 5D Mark II, it freeze so I have no choice but to reboot.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on August 23, 2013, 05:56:53 PM
Ran some video test which I converted directly to dng mlv2dng.exe

1920x1080 at 29.97 - Perfect
Cropped 1920x1080 at 29.97 - Some pink frames and pink noise frames
60 fps 1600x560 at 60 fps - Some pink frames and pink noise frames

Sorry to point this out here but the topic is related. On the EOS M, I find if you increase the fps beyond 30, the camera will throw a few pink frames first before it stabilized. Anything up to 30 of less fps is fine.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on August 23, 2013, 10:32:45 PM
I was checking out the Pre-Built Module of MLV, I only use the rec-mo file because I work on a Mac and the exe files are irrelevant to me on a Mac, but when I use the module on my Canon 5D Mark II, it freeze so I have no choice but to reboot.


It is not made for the Mark II, yet =)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on August 24, 2013, 11:14:04 AM
hello,
this all sounds very interesting..
can anyone please post a link to a 5dmk3 build with the new format.. i am looking forward to test it.
thank you.
sw
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: guentergunter on August 24, 2013, 10:17:09 PM
hello,
this all sounds very interesting..
can anyone please post a link to a 5dmk3 build with the new format.. i am looking forward to test it.
thank you.
sw
What about a look into the very first post in this threat?


By the way: 5D2 support is much appreciated ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on August 25, 2013, 11:09:02 AM
thank you.
i see the mlvdump is available for windows and linux..
is there a mac version?
thx.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 25, 2013, 11:51:17 PM
thank you.
i see the mlvdump is available for windows and linux..
is there a mac version?
thx.

Try this: http://www.magiclantern.fm/forum/index.php?topic=7802.0
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on August 26, 2013, 03:11:02 PM
thx, will check that link.

is it possible that someone can provide a new raw rec mlv module? i can see that there were changes.. but the module is from 13_8..
thx.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on August 26, 2013, 03:30:02 PM
Where is the link to the latest build?  This post is getting confusing... :-/
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 26, 2013, 08:58:54 PM
latest build is in first post.
it contains the smpte module, which you have to disable if you dont use smpte. (audio menu or disable module loading)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Steve Kahn on August 26, 2013, 09:27:23 PM
Sorry, I'm a bit confused as well. 

Does this new format currently support synced audio? 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on August 27, 2013, 10:05:08 AM
Any chance someone can do a Mk2 version soon to play around with? =)
Considering it seems to work quite well on the Mk3.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 27, 2013, 10:53:01 AM
it supports no audio recording yet.
SMPTE (LTC coded audio) is an experimental module that outputs a sound signal that can be read and processed (or not) with special equipment.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Doyle4 on August 28, 2013, 02:00:10 PM
is there a mac conversion software? would love to test this out.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 28, 2013, 02:09:45 PM
is there a mac conversion software? would love to test this out.
Reply 420
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on August 28, 2013, 02:42:19 PM
at g3ggo: 5D3 does not compile

error:
module.o: In function `module_shutdown':
module.c:(.text+0x2328): undefined reference to `module_save_configs'
collect2: error: ld returned 1 exit status
make[1]: *** [magiclantern] Error 1
make[1]: Leaving directory `/home/magiclantern/G3GGo/magic-lantern/platform/5D3.113'
make: *** [5D3] Error 2
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on August 29, 2013, 09:07:38 AM
It's possible to add the delta of exposure value (the value in XMP line crs:Exposure2012=) to 2.0 specs?
So that the converter can recreate other than DNG files also xmp file and you can obtain a perfect deflickered files.

I am often in similar situation of this: http://www.magiclantern.fm/forum/index.php?topic=7739.0 or with some little-than-normal-speed timelapse with sunset for example

but sorry I can't offer 100$  :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 29, 2013, 09:21:46 AM
That value has to be computed from raw data, and it's CPU-intensive, so what's the point of including it in the raw format?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on August 29, 2013, 10:03:08 AM
For sunset (especially for the last part) I use often RAW videos because I need to use values under 1 second not available in intervalometer like 2-5fps
but I really miss a lot the deflicker possibility.
It's very important to achieve a good results because the luminosity change a lot in small time.

Maybe is possible to write the raw exposure data and compute it after in the converter?
So we can lighten the CPU.
After all If I understood correctly the frame exposure data used (aperture, iso and shutter value) are already present in 2.0 format.


Thank you much.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 29, 2013, 10:04:57 AM
That would be a feature in the converter, nothing to do with metadata. Not even exposure values.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on August 29, 2013, 10:15:16 AM
Thank you,

I was think that ML compute the XMP exposure number with the difference between the exposure calculated and the exposure that camera apply.
So what raw data use ML to calculate XMP?
Can you calculate it in post without any additional data other than the standard exif?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 29, 2013, 10:18:42 AM
That's how post deflicker is working. Deflicker data is just the median of the green channel (simplest implementation). If you want all channels, you need to apply WB before median.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on August 29, 2013, 10:44:09 AM
Ok, last thing.
For the median of green channel you mean the measured green channel value and not the green channel value of the taken photo, right?
So for an implementation of post deflicker in the converter you need to have (other than the frame exposed data used (aperture, iso and shutter value)) also the measured green channel value that normally you don't have, neither in exif of CR2's files right?

Thank you.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 29, 2013, 10:47:03 AM
No, you compute it in post from the pixels that you find in the video file. You can't do that in the camera...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gerk.raisen on August 29, 2013, 11:38:38 AM
Perfect :) , This time  I hope this time I understood correctly :) :)
With an implementation of post-deflicker in the converter (without add any values other than frame exposed data used) we can obtain the SAME results of the post-deflicker correction value calculated on-camera in case of intervalometer still photo, right?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 29, 2013, 11:39:53 AM
Yes, or better.

I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 30, 2013, 08:41:29 AM
Added a small proposal for the OFFS block (black correction info), and specified the units for digital ISO field in EXPO block (0.001 EV).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 30, 2013, 09:37:33 AM
ok, cool.
will the EXPO update change the format as it is now?
i.e. would the parsers have to update their code to match the changes?
(it is ok, we just have to bump the version then)

I cannot check the changes as google docs is proxy-filtered.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on August 30, 2013, 09:54:07 AM
I don't think any parser is using the digital ISO other than printing it. I've just updated the value from 1 to 1024 (so it's no longer limited to integer EV).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on August 30, 2013, 12:00:14 PM
ok thanks :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on August 31, 2013, 08:01:11 AM
Yes, or better.

I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).

Exciting stuff a1ex! Is there a thread that I can follow regarding this?

Happy to still put the $100 down once you've got it implemented ;)

Great work!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dmk on September 01, 2013, 07:20:15 AM
I'm a bit confused about the plan to implement LZMA compression (MLV_VIDEO_CLASS_FLAG_LZMA in mlv.h)

Will that be happening on the camera side in order to remove (or reduce) the CF card speed as a bottleneck? OR is it not really going to help CF card speed requirements, and it's just about storage?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 01, 2013, 08:12:58 PM
it's just about storage?

this. its for archiving.

test footage can be compressed to 50% of the original size without any information loss.
do this on your computer and you save a lot of space for archiving.

you can also reduce bit depth to e.g. 10 bits and you get it down to 30%-40% of original size.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dmk on September 01, 2013, 08:41:42 PM
Oh... so it wouldn't help the CF Card speed limit to compress on the fly, i.e. time it takes to compress > time it takes to just write the data straight (or some other issue)?

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 01, 2013, 08:53:25 PM
Oh... so it wouldn't help the CF Card speed limit to compress on the fly, i.e. time it takes to compress > time it takes to just write the data straight (or some other issue)?

sure, just tell me how i can configure the EDMAC to compress data with LZMA..
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dariSSight on September 01, 2013, 09:18:55 PM
sure, just tell me how i can configure the EDMAC to compress data with LZMA..
Thanks for everything Team Magic Lantern has accomplished for the Independent Film Industry Community, it has helped me in very advance ways. Also I was wondering is there a light in the tunnel for 5D Mark II to receive Magic Lantern (RAW) Video format v2.0?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 01, 2013, 11:08:58 PM
i am currently focusing on making MLV usable.
but i dont think that 5D2 is a big problem. it just has to be tried and bugfixed :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dmk on September 02, 2013, 09:29:37 AM
sure, just tell me how i can configure the EDMAC to compress data with LZMA..

<joke>Tried calling Canon Customer Support but no luck</joke>
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: John Kesl on September 02, 2013, 04:58:59 PM
Hi,
I compiled all the latest stuff for the 7D from g3gg0/magic-lantern. I'm not able to get any results from mlv_dump other than a read out of the file structure of the .mlv.
I wonder what it is I'm doing wrong? maybe my usage is incorrect?
Code: [Select]
[email protected]:~/Desktop/MLV_Dump0$ ./mlv_dump /home/studio/Desktop/MLV_Dump0/M02-2157.MLV

Code: [Select]
MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: '/home/studio/Desktop/MLV_Dump0/M02-2157.MLV'
   - Verify file structure
[i] File /home/studio/Desktop/MLV_Dump0/M02-2157.MLV opened
[i] Processing...
[i] Reached end of chunk 0/1 after 1379 blocks
[i] Processed 603 video frames
[i] Done


So, I searched and found.

In linux type

./mlv_dump -h and get the usage RULES

I'll keep my post so people can see and learn from my own mistakes

When I try mlv2dng.exe I have no problems getting the .dng output.

BTW. I am having no issues recording continuous frames at 16x9 @ 1782. on a 1066x toshiba 16gb.
no pink frames. no small hacks enabled. works great. I compiled these modules for the 7D. file_man, ime_std, ime_rot, xml_cfg, raw_rec, pic_view. Raw_rec works well. Am checking the buffering strategies for speed gains. right now with my card I get upwards of 67 MB/s write speed whilst recording. the card benchmarks avg. 90MB/s.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: John Kesl on September 02, 2013, 05:23:13 PM
I just ran mlv_dump successfully for linux. works great. I did a verbose dump for 7D. i put the code below.

Code: [Select]
[email protected]:~/Desktop/MLV_Dump0$ ./mlv_dump -o M01-001.RAW -v -r -l0 /home/studio/Desktop/MLV_Dump0/M02-2157.MLV

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: '/home/studio/Desktop/MLV_Dump0/M02-2157.MLV'
   - Verbose messages
   - Convert to legacy RAW
   - Output into 'M01-001.RAW'
[i] File /home/studio/Desktop/MLV_Dump0/M02-2157.MLV opened
[i] Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 11103347201719758039
    FPS         : 23.976000
    File        : 0 / 0
    Frames Video: 603
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 3.927000 ms
    Res:  1728x972
    raw_info:
      api_version      0x00000001
      height           1182
      width            1992
      pitch            3486
      frame_size       0x003EDF84
      bits_per_pixel   14
      black_level      2053
      white_level      15000
      active_area.y1   26
      active_area.x1   256
      active_area.y2   1182
      active_area.x2   1992
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: INFO
  Offset: 0x000000e8
    Size: 16
    Time: 4.145000 ms
Block: RTCI
  Offset: 0x000000f8
    Size: 44
    Time: 4.332000 ms
     Date:        02.08.2013
     Time:        21:57:35 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000124
    Size: 40
    Time: 5.672000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x0000014c
    Size: 96
    Time: 5.744000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 6.047000 ms
     Camera Name:   'Failed to get properties.'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x00000200
    Size: 44
    Time: 6.413000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x0000022c
    Size: 2943424
    Time: 74.161000 ms
   Frame: #0
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x002cebec
    Size: 1600
Block: NULL
  Offset: 0x002cf22c
    Size: 36
Block: RTCI
  Offset: 0x002cf250
    Size: 44
    Time: 107.820000 ms
     Date:        02.08.2013
     Time:        21:57:35 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x002cf27c
    Size: 40
    Time: 109.232000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x002cf2a4
    Size: 96
    Time: 109.297000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: WBAL
  Offset: 0x002cf304
    Size: 44
    Time: 109.504000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x002cf330
    Size: 2943164
    Time: 116.090000 ms
   Frame: #1
    Crop: 256x120
     Pan: 256x118
   Space: 3804
Block: NULL
  Offset: 0x0059dbec
    Size: 1600
Block: VIDF
  Offset: 0x0059e22c
    Size: 2943424
    Time: 157.921000 ms
   Frame: #2
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x0086cbec
    Size: 1600
Block: VIDF
  Offset: 0x0086d22c
    Size: 2943424
    Time: 200.693000 ms
   Frame: #3
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x00b3bbec
    Size: 1600
Block: VIDF
  Offset: 0x00b3c22c
    Size: 2943424
    Time: 241.556000 ms
   Frame: #4
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x00e0abec
    Size: 1600
Block: VIDF
  Offset: 0x00e0b22c
    Size: 2943424
    Time: 282.823000 ms
   Frame: #5
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x010d9bec
    Size: 1600
Block: VIDF
  Offset: 0x010da22c
    Size: 2943424
    Time: 324.598000 ms
   Frame: #6
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x013a8bec
    Size: 1600
Block: VIDF
  Offset: 0x013a922c
    Size: 2943424
    Time: 366.565000 ms
   Frame: #7
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x01677bec
    Size: 1600
Block: VIDF
  Offset: 0x0167822c
    Size: 2943424
    Time: 575.073000 ms
   Frame: #12
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x01946bec
    Size: 1600
Block: VIDF
  Offset: 0x0194722c
    Size: 2943424
    Time: 616.474000 ms
   Frame: #13
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x01c15bec
    Size: 1600
Block: VIDF
  Offset: 0x01c1622c
    Size: 2943424
    Time: 658.341000 ms
   Frame: #14
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x01ee4bec
    Size: 1600
Block: VIDF
  Offset: 0x01ee522c
    Size: 2943424
    Time: 700.022000 ms
   Frame: #15
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x021b3bec
    Size: 1600
Block: VIDF
  Offset: 0x021b422c
    Size: 2943424
    Time: 742.674000 ms
   Frame: #16
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x02482bec
    Size: 1600
Block: VIDF
  Offset: 0x0248322c
    Size: 2943424
    Time: 783.859000 ms
   Frame: #17
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x02751bec
    Size: 1600
Block: VIDF
  Offset: 0x0275222c
    Size: 2943424
    Time: 825.092000 ms
   Frame: #18
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x02a20bec
    Size: 1600
Block: VIDF
  Offset: 0x02a2122c
    Size: 2943424
    Time: 867.115000 ms
   Frame: #19
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x02cefbec
    Size: 1600
Block: NULL
  Offset: 0x02cf022c
    Size: 36
Block: RTCI
  Offset: 0x02cf0250
    Size: 44
    Time: 902.736000 ms
     Date:        02.08.2013
     Time:        21:57:36 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x02cf027c
    Size: 40
    Time: 904.250000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x02cf02a4
    Size: 96
    Time: 904.330000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: WBAL
  Offset: 0x02cf0304
    Size: 44
    Time: 904.589000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x02cf0330
    Size: 2943164
    Time: 908.885000 ms
   Frame: #20
    Crop: 256x120
     Pan: 256x118
   Space: 3804
Block: NULL
  Offset: 0x02fbebec
    Size: 1600
Block: VIDF
  Offset: 0x02fbf22c
    Size: 2943424
    Time: 950.834000 ms
   Frame: #21
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x0328dbec
    Size: 1600
Block: VIDF
  Offset: 0x0328e22c
    Size: 2943424
    Time: 992.293000 ms
   Frame: #22
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x0355cbec
    Size: 1600
Block: VIDF
  Offset: 0x0355d22c
    Size: 2943424
    Time: 408.019000 ms
   Frame: #8
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x0382bbec
    Size: 1600
Block: VIDF
  Offset: 0x0382c22c
    Size: 2943424
    Time: 1117.426000 ms
   Frame: #25
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x03afabec
    Size: 1600
Block: VIDF
  Offset: 0x03afb22c
    Size: 2943424
    Time: 1158.761000 ms
   Frame: #26
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x03dc9bec
    Size: 1600
Block: VIDF
  Offset: 0x03dca22c
    Size: 2943424
    Time: 1200.825000 ms
   Frame: #27
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04098bec
    Size: 1600
Block: VIDF
  Offset: 0x0409922c
    Size: 2943424
    Time: 1243.190000 ms
   Frame: #28
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04367bec
    Size: 1600
Block: VIDF
  Offset: 0x0436822c
    Size: 2943424
    Time: 1284.208000 ms
   Frame: #29
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04636bec
    Size: 1600
Block: VIDF
  Offset: 0x0463722c
    Size: 2943424
    Time: 1326.281000 ms
   Frame: #30
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04905bec
    Size: 1600
Block: VIDF
  Offset: 0x0490622c
    Size: 2943424
    Time: 1367.636000 ms
   Frame: #31
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04bd4bec
    Size: 1600
Block: VIDF
  Offset: 0x04bd522c
    Size: 2943424
    Time: 449.614000 ms
   Frame: #9
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x04ea3bec
    Size: 1600
Block: VIDF
  Offset: 0x04ea422c
    Size: 2943424
    Time: 491.980000 ms
   Frame: #10
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x05172bec
    Size: 1600
Block: VIDF
  Offset: 0x0517322c
    Size: 2943424
    Time: 533.598000 ms
   Frame: #11
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x05441bec
    Size: 1600
Block: VIDF
  Offset: 0x0544222c
    Size: 2943424
    Time: 1409.272000 ms
   Frame: #32
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x05710bec
    Size: 1600
Block: VIDF
  Offset: 0x0571122c
    Size: 2943424
    Time: 1450.895000 ms
   Frame: #33
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x059dfbec
    Size: 1600
Block: VIDF
  Offset: 0x059e022c
    Size: 2943424
    Time: 1493.368000 ms
   Frame: #34
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x05caebec
    Size: 1600
Block: VIDF
  Offset: 0x05caf22c
    Size: 2943424
    Time: 1617.806000 ms
   Frame: #37
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x05f7dbec
    Size: 1600
Block: NULL
  Offset: 0x05f7e22c
    Size: 36
Block: RTCI
  Offset: 0x05f7e250
    Size: 44
    Time: 1652.599000 ms
     Date:        02.08.2013
     Time:        21:57:36 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x05f7e27c
    Size: 40
    Time: 1654.119000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x05f7e2a4
    Size: 96
    Time: 1654.199000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: WBAL
  Offset: 0x05f7e304
    Size: 44
    Time: 1654.463000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x05f7e330
    Size: 2943164
    Time: 1659.786000 ms
   Frame: #38
    Crop: 256x120
     Pan: 256x118
   Space: 3804
Block: NULL
  Offset: 0x0624cbec
    Size: 1600
Block: VIDF
  Offset: 0x0624d22c
    Size: 2943424
    Time: 1701.822000 ms
   Frame: #39
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x0651bbec
    Size: 1600
~~
~~
~~
Block: NULL
  Offset: 0x66299bec
    Size: 1600
Block: VIDF
  Offset: 0x6629a22c
    Size: 2943424
    Time: 24432.173000 ms
   Frame: #584
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x66568bec
    Size: 1600
Block: VIDF
  Offset: 0x6656922c
    Size: 2943424
    Time: 24474.052000 ms
   Frame: #585
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x66837bec
    Size: 1600
Block: NULL
  Offset: 0x6683822c
    Size: 36
Block: RTCI
  Offset: 0x66838250
    Size: 44
    Time: 24502.724000 ms
     Date:        02.08.2013
     Time:        21:57:59 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x6683827c
    Size: 40
    Time: 24504.143000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x668382a4
    Size: 96
    Time: 24504.226000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: WBAL
  Offset: 0x66838304
    Size: 44
    Time: 24504.488000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x66838330
    Size: 2943164
    Time: 24515.765000 ms
   Frame: #586
    Crop: 256x120
     Pan: 256x118
   Space: 3804
Block: NULL
  Offset: 0x66b06bec
    Size: 1600
Block: VIDF
  Offset: 0x66b0722c
    Size: 2943424
    Time: 24557.608000 ms
   Frame: #587
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x66dd5bec
    Size: 1600
Block: VIDF
  Offset: 0x66dd622c
    Size: 2943424
    Time: 24599.616000 ms
   Frame: #588
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x670a4bec
    Size: 1600
Block: VIDF
  Offset: 0x670a522c
    Size: 2943424
    Time: 24641.389000 ms
   Frame: #589
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x67373bec
    Size: 1600
Block: VIDF
  Offset: 0x6737422c
    Size: 2943424
    Time: 24682.607000 ms
   Frame: #590
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x67642bec
    Size: 1600
Block: VIDF
  Offset: 0x6764322c
    Size: 2943424
    Time: 24724.385000 ms
   Frame: #591
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x67911bec
    Size: 1600
Block: VIDF
  Offset: 0x6791222c
    Size: 2943424
    Time: 24891.196000 ms
   Frame: #595
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x67be0bec
    Size: 1600
Block: VIDF
  Offset: 0x67be122c
    Size: 2943424
    Time: 24932.799000 ms
   Frame: #596
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x67eafbec
    Size: 1600
Block: VIDF
  Offset: 0x67eb022c
    Size: 2943424
    Time: 24974.609000 ms
   Frame: #597
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x6817ebec
    Size: 1600
Block: VIDF
  Offset: 0x6817f22c
    Size: 2943424
    Time: 24765.842000 ms
   Frame: #592
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x6844dbec
    Size: 1600
Block: VIDF
  Offset: 0x6844e22c
    Size: 2943424
    Time: 24807.931000 ms
   Frame: #593
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x6871cbec
    Size: 1600
Block: VIDF
  Offset: 0x6871d22c
    Size: 2943424
    Time: 24849.782000 ms
   Frame: #594
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x689ebbec
    Size: 1600
Block: VIDF
  Offset: 0x689ec22c
    Size: 2943424
    Time: 25016.513000 ms
   Frame: #598
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x68cbabec
    Size: 1600
Block: VIDF
  Offset: 0x68cbb22c
    Size: 2943424
    Time: 25057.653000 ms
   Frame: #599
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x68f89bec
    Size: 1600
Block: VIDF
  Offset: 0x68f8a22c
    Size: 2943424
    Time: 25099.865000 ms
   Frame: #600
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x69258bec
    Size: 1600
Block: VIDF
  Offset: 0x6925922c
    Size: 2943424
    Time: 25141.592000 ms
   Frame: #601
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x69527bec
    Size: 1600
Block: VIDF
  Offset: 0x6952822c
    Size: 2943424
    Time: 25182.815000 ms
   Frame: #602
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x697f6bec
    Size: 1600
Block: VIDF
  Offset: 0x697f722c
    Size: 2943424
    Time: 21304.286000 ms
   Frame: #509
    Crop: 256x120
     Pan: 256x118
   Space: 4064
Block: NULL
  Offset: 0x69ac5bec
    Size: 1600
Block: NULL
  Offset: 0x69ac622c
    Size: 36
Block: RTCI
  Offset: 0x69ac6250
    Size: 44
    Time: 19186.942000 ms
     Date:        02.08.2013
     Time:        21:57:54 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 244
     Daylight s.: 0
Block: EXPO
  Offset: 0x69ac627c
    Size: 40
    Time: 19188.471000 ms
     ISO Mode:   0
     ISO:        400
     ISO Analog: 88
     ISO DGain:  0 EV
     Shutter:    20749 µs (1/48.20)
Block: LENS
  Offset: 0x69ac62a4
    Size: 96
    Time: 19188.621000 ms
     Name:        ''
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/5.60
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x00000000
     Flags:       0x00000000
Block: WBAL
  Offset: 0x69ac6304
    Size: 44
    Time: 19188.972000 ms
     Mode:   9
     Kelvin:   3600
     Gain R:   495
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x69ac6330
    Size: 2943164
    Time: 19219.000000 ms
   Frame: #459
    Crop: 256x120
     Pan: 256x118
   Space: 3804
Block: NULL
  Offset: 0x69d94bec
    Size: 1600
[i] Reached end of chunk 0/1 after 1379 blocks
[i] Processed 603 video frames
[i] Done

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 03, 2013, 12:57:03 AM
I did something to my 50D, now I have these .MLV files and am lost in a sea of forum posts.  I'm on Windows.  Can someone point me to where I can get a program/script, whatever, to get me to RAW so I can RAWanizer it :)  I've read just a thing exists, but have also read many problems.  So don't want to download something old and gnarly.  Thanks!  I just hope I can find my way back to this forum thread! :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: John Kesl on September 03, 2013, 02:03:09 AM
I did something to my 50D, now I have these .MLV files and am lost in a sea of forum posts.  I'm on Windows.  Can someone point me to where I can get a program/script, whatever, to get me to RAW so I can RAWanizer it :)  I've read just a thing exists, but have also read many problems.  So don't want to download something old and gnarly.  Thanks!  I just hope I can find my way back to this forum thread! :)

check out mlv2dng by gnarr.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip
the program is command line executable on windows something like.
c:> folder/where/mlv2dng.exe c:> file/where/Mxx-xxx1.MLV > c:> folder/where/you/want/xxxxxx.DNG
I make my folder before and drag and drop it onto the DOS prompt window. works good.
mlv_dump, is available from somewhere around here it's linux, but it's got more options (as far as I can tell)
search mlv_dump. or compile it yourself from g3gg0's repository @ bitbucket.org.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 03, 2013, 05:08:11 AM
Thanks John.  I sent the dev a private note.  Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW.  Anyway. offered to help.  In the meantime, I put my 50D back to one of Andy's stable builds. 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: John Kesl on September 03, 2013, 05:36:10 AM
Thanks John.  I sent the dev a private note.  Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW.  Anyway. offered to help.  In the meantime, I put my 50D back to one of Andy's stable builds.

Hey man, mlvdng.exe goes all the way to .dng no need for raw inter. Also, i'm pretty sure g3gg0 put up a mlv_dump.exe I may be wrong. but i would check developer g3gg0 threads.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 05:43:30 AM
for 50D I made MLV_REC and raw_rec so the choice is yours. raw_rec is still faster, except if you have a 5DIII.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: LEVISDAVIS on September 03, 2013, 07:02:42 AM
Thanks for the 50D development! Looking to test out MLV... Wonder how the 50D will handle the extra info??? Thanks for the extra effort.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 03, 2013, 07:46:24 AM
Thanks John.  I sent the dev a private note.  Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW.  Anyway. offered to help.  In the meantime, I put my 50D back to one of Andy's stable builds.

[ x ] you didn't read my first post, where it says: post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dude on September 03, 2013, 11:46:04 AM
Did you read about Scratch? they are going to drop a Raw Player, the specs say that also dslr raw material will be readeable.
http://www.awn.com/news/business/assimilate-launches-scratch-play
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: NedB on September 03, 2013, 12:13:03 PM
@dude: the link (in the Assimilate press release) for the Scratch player has been pulled by Assimilate, apparently, but I found the new location:

http://www.assimilateinc.com/scratch-play-build (http://www.assimilateinc.com/scratch-play-build)

Cheers!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dude on September 03, 2013, 12:37:42 PM
Nice, thanks!

edit-
installed it, so far it does not load the raw files
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 03, 2013, 02:21:32 PM
[ x ] you didn't read my first post, where it says: post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.

I sent you a private message and you're now publicly saying I don't take the time to read your work?  That's not very nice.  I've read everything you've wrote.  I can't find a mlv_dump.exe for Windows.   All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it.  Really, g3gg0! :) 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 03, 2013, 03:14:10 PM
I sent you a private message and you're now publicly saying I don't take the time to read your work?  That's not very nice.  I've read everything you've wrote.  I can't find a mlv_dump.exe for Windows.   All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it.  Really, g3gg0! :)

sorry, i read forum and answer there before i check any PMs.


look in the main post and download the pre-built version.
> if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built ones. (tested on 5D3)

as i update the link when i make new versions, please use the link from the main post.
there is raw2dng.exe, mlv_dump.exe, raw_rec.mo and a few other modules in that zip.
just open the first page, check my post and search for "pre-built". this is a link to the latest version.....
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 03, 2013, 03:30:26 PM
I sent you a private message and you're now publicly saying I don't take the time to read your work?  That's not very nice.  I've read everything you've wrote.  I can't find a mlv_dump.exe for Windows.   All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it.  Really, g3gg0! :)

Keep your personal stuff off this board!

If you read this thread you will find this on reply 218:

here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump

did not test it for 4GiB problems. can someone else try?

and a bit lower you will find this:

I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv

and ....  made the mlv.raw file

and ... drag and drop on raw2dng.exe and Yes!!!!!!!

Got dng files!

So try it and see.  If it is not compatible with current build then wait until dev makes it available.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 03, 2013, 03:46:27 PM
@g3gg0!

I think the confusion is that you assume someone is going to load your sources and naturally see the utilities, which was natural when you started your effort. 

I ended up with source that creates .MLV, which I didn't realize (and still don't know exactly how that happened).  You might want to edit your first post to make it clear that if someone ends up with MLV files on their camera, that they can download your zip file and get the exe.   Many people like me probably use RAWanizer straight to Cineform to check what they've done, and save on time/space with the whole DNG/TIFF extraction stuff.

I may have missed things, as RenatoPhoto pointed out, but honestly tried very hard.   Again, thanks for all your efforts! 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 03, 2013, 04:39:19 PM
I hate sounding like a broken record... buuut :P
Anyone compiling some tests for 5D2 now that 50D seems to be getting some compiles too :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on September 03, 2013, 04:53:51 PM
I hate sounding like a broken record... buuut :P
Anyone compiling some tests for 5D2 now that 50D seems to be getting some compiles too :)

Try the 50D one https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/mlv_rec.mo should work in the 5D2 (at least for testing).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 05:08:27 PM
That scratch play looks interesting but ads in the free version? Come on! VLC/MPC doesn't put ads in. Quite insulting to have spyware in your media player.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: NedB on September 03, 2013, 05:13:57 PM
@dude: If you convert a .raw file (using raw2cdng.exe rather than raw2dng.exe) to CinemaDNG, you can point Scratch to the first .dng file and it will open and play the clip. Pretty cool. Colors are a bit off but Scratch is apparently a huge application, even the player portion, and there's probably a way to insert a LUT somewhere so that the clip will look reasonably like what you shot. Just FYI, haven't played with it for very long. Cheers!

P.S. Not sure anymore where I got raw2cdng.exe, it was on the ML forum somewhere, you should be able to Google it.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 03, 2013, 05:43:58 PM
@g3gg0!

I think the confusion is that you assume someone is going to load your sources and naturally see the utilities, which was natural when you started your effort. 

I ended up with source that creates .MLV, which I didn't realize (and still don't know exactly how that happened).  You might want to edit your first post to make it clear that if someone ends up with MLV files on their camera, that they can download your zip file and get the exe.   Many people like me probably use RAWanizer straight to Cineform to check what they've done, and save on time/space with the whole DNG/TIFF extraction stuff.

well, thats the reason why i pointed out in the main post that one has to convert it to legacy .raw before he can use it in his normal workflow.
i also provided the binaries so one can run the tools without the hassle of cross-compiling.

as this is no release for the masses, i define my assumption to be perfectly fine.
it is for developers, for testers, BUT NOT for productional use yet.
as the board "General Development Discussion" implies, this is yet in development and end users will get a nifty UI, workflow, guide and whatever as soon it is in "New Releases" forum.

i understand that this is not the allround-happy-packet ready for everyone to use which you seem to have expected, but it was never meant to be that.
this will follow as soon this video format and the code is going to be merged into main tree.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 03, 2013, 06:42:50 PM
@g3gg0

Sorry,  I made it sound like it was your fault, and that's not what I mean.  I meant to say that you might not realize that your code is now seeing its way onto more and more cameras (like my 50d) and that you may want to edit the first page a bit, for people like me who don't know a thing about MLV but now have it on their camera.  Again, I'm sorry if I implied that you had done something wrong.  I realize it was all just a big confusion.  I AM all round happy packet!   Please believe me.  And thank you again.

I can already see how MLV will be a big improvement with the EOS-M.  Right now, I have to pick which mode to clean up the files, normal or crop mode.  With your work, future software can figure that out.  Again, if there is anything I can do to help you with MLV, let me know.  So again, please accept my apologies and thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 07:03:03 PM
I'm going to go out on a limb and say that MLV will be better for SD cameras in general... they already only go ~40mb so +/- 1MB won't make a difference. On 50D I struggle with 1920x1038 and 1080P where before I had idle time.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 03, 2013, 07:42:06 PM
Just for fun, the 50D module allmost works on the 5d2, can setup nicely etc, but recording just locks it up.
But hey, fun to see the options etc :D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 07:46:44 PM
Sure you're not using hacked modes that lock the LV?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 03, 2013, 08:09:29 PM
Very. Menu looks screwed up too after. Completely locked up, with brief unlocked.
Then it just stops automatically. And menu keeps saying Stopping... on the raw-post.

Biggest MLV so far is around 3meg, so I guess its locked up then.

But I didnt expect it to work 100% since its for 50D :)

Just a test
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 08:24:05 PM
5DII missing/broken reslock? I don't have one to test but if hacked is off it shouldn't be freezing.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 03, 2013, 08:33:29 PM
Tried from a scratch made card, only module loaded = mlv.
When not locked I get a datarate of 2-3MB/s so something isnt working properly for the 5D2 =/

But never expected it to work, might be made with 50D-specifics in mind?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 03, 2013, 09:10:27 PM
The handful of 50D specifics only run on 50D. I'll see what happens on SD cameras.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 04, 2013, 01:13:45 AM
Here's that 50D MLV file stuff I shot the other day. Thanks again g3gg0!  There was some broken frames in the beginning of some of the clips, but it could have been anything.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 04, 2013, 01:18:19 AM
you see, all is there ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ItsMeLenny on September 04, 2013, 08:27:46 AM
Here's that 50D MLV file stuff I shot the other day. Thanks again g3gg0!  There was some broken frames in the beginning of some of the clips, but it could have been anything.
Was this taken using the slightly larger camera good for all those suffering from some sort of mid life crisis?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 04, 2013, 12:56:48 PM
Played some more with the 50D-module and still no go. Tried every setting, different bufferings methods, hacks on/off. All with the same results.
So pretty sure the 50D-module has some code that just doesnt wanna play with the 5d2 =)

But it was really nice to see the menu and play around (not being sarcastic, I am uber-positive)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 04, 2013, 02:54:26 PM
Was this taken using the slightly larger camera good for all those suffering from some sort of mid life crisis?

It's even worse.  Her friend Christine, one to the right in the group shot, is an animation major at MassArt.  Emma said she saved up all her money to buy a T3i this summer.  I thought, 'well, she'll appreciate what I'm doing here.'  I tried to explain ML RAW but ended up as a 52-year-old man humiliated by 21-year-old eye-rolling.  As you can see, however, that doesn't STOP ME from shooting anything ;)  It's a lonely world, ML RAW.  In the one camp you have people who don't care, in the other, those who can't face the fact that in-camera H.264, for anyone serious about film/video, is dead..oops, hope I haven't offended anyone here!

The video version doesn't do justice what what was shot.  I want to say again, that in RAWanizer, adding the args -d4 (higher sharpening) and -q5 (higher quality) makes for some very nice footage, good enough for me.

I went back to one of Andy's builds, so am shooting .RAW again.  However, I would like to add .MLV to my processing scripts, maybe help out a bit, but I don't want to get MLV back by doing what I did before, copying modules into the camera willy-nilly.  Is there a stable build for the 50D that will output MLV?  Or can someone let me know when there is one. 

I still can't make sense of which forks I should follow.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 04, 2013, 03:08:59 PM
You can copy both modules and just run raw_rec and leave mlv_rec disabled. Its working and non freezing on 6D (after hard path removal) and other sd cameras but I dunno if this breaks spanning for 5DIII. Slower on SD cameras too... 39.x MB/s with raw_rec... only 36-37 MB/s on MLV rec.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 04, 2013, 08:11:47 PM
1% any news regarding the odd happenings with the 50D module? :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 04, 2013, 08:20:56 PM
@1%, I'd be happy to load new modules, test logs, whatever to devs.   Is there a tutorial, or page, where it explains how to load these modules, what to look out for.   When do I need to re-run the EOScard utility?  Can I just replace the autoexec.bin file.  Can you tell me who is actively developing for the 50D, or any camera.  Should I, can I, use the same code base for my EOS-M and 50D.  While you're here.  Does God exist?  Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 04, 2013, 09:05:56 PM
autoexec + sym go together... one in root the other in the ML/modules folder.

After that you can load any modules with the same API (maybe we've had 3 changes total).

Not sure whats happening on 5DII, only bug I noticed is sound recording is 2 when the defaults are set so I have to fix that.  I don't exactly have one to test and 50D works ok.


50D/EOSM/6D are the same base, it picks what to compile based on the model. 600D is its own but that may change.


This does MLV on SD cameras and probably CF too.

mlv_rec.mo (47 KB)
https://mega.co.nz/#!Nlp1FahY!fyd5Bwlw_uX9ehF21P_Cxk7lX9RbOrrltHmBlokMwzk
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 04, 2013, 09:23:51 PM
1% Thanks, oaky, so I have Andy's latest stable build.  I looked, and see the sym file.  So if I want to save as .MLV, I assume I need that module in my camera's ML folder?

o. Who is working on that?

o. I take it, from what you're saying, that if I put that in the folder it will become a load option on the camera?  I don't need to to anything more.

o. I see builds for you.  What aspect do your builds specialize in?  Which fork of your should I follow?

o. I would think you would want people to test your latest-greatest creation.  But I don't see an easy to follow place for that.  If it exists, can you please tell me how I should approach trying out your new stuff, say.

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 04, 2013, 09:36:48 PM
TL2 is the 600D repo, everything else is in TL6D.

To follow it look at commits, you can set up a VM and compile for yourself. There is a guide on here. Andy is building 50D off a clone of my repo.

For modules, yea pretty much what you said, you add it to the dir and pick what you want loading then reboot.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on September 05, 2013, 12:26:09 PM
Yes, or better.

I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).

Hey a1ex. Just following up - is there a thread that we can follow the progress of this update? Seems great!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on September 05, 2013, 01:34:04 PM
Hey a1ex. Just following up - is there a thread that we can follow the progress of this update? Seems great!

As they've said before when the file format is complete and ready for the masses the tools will be also. Until then be happy with what they put out for testing.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Vegeta on September 06, 2013, 04:25:05 AM
Hey guys, Quick question.
How does one get the new magic lantern video format (.MLV) working?
Is it some plug in you through into the modules folder or is it cooked right into the nightly release or.. what? Lol.

Also, they say it adds 20mbps to the multi card spanning. Now if you have the sandisk 95mbps sd cards.. will that make it more than 20 or do they have it limited to 20?

Using a mark 3.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 06, 2013, 03:33:26 PM
Hey guys, Quick question.
How does one get the new magic lantern video format (.MLV) working?
Is it some plug in you through into the modules folder or is it cooked right into the nightly release or.. what? Lol.
Not available through nightly release.  Read the first post of this thread!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on September 06, 2013, 04:09:55 PM
autoexec + sym go together... one in root the other in the ML/modules folder.

After that you can load any modules with the same API (maybe we've had 3 changes total).

Not sure whats happening on 5DII, only bug I noticed is sound recording is 2 when the defaults are set so I have to fix that.  I don't exactly have one to test and 50D works ok.


50D/EOSM/6D are the same base, it picks what to compile based on the model. 600D is its own but that may change.


This does MLV on SD cameras and probably CF too.

mlv_rec.mo (47 KB)
https://mega.co.nz/#!Nlp1FahY!fyd5Bwlw_uX9ehF21P_Cxk7lX9RbOrrltHmBlokMwzk

Added the mlv_rec.mo to the 5D Mark 3 modules directory. Get the following error: "tcc: error: undefined symbol 'message_queue_count' [E] failed to link modules"
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 06, 2013, 04:20:50 PM
You need a build with msg_qcount and prop body ID. I think 5DIII has it in main, I added those for all in my repo. BTW, the spanning may be broken since the path isn't hardcoded to a:
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sletts02 on September 07, 2013, 12:05:08 AM
As they've said before when the file format is complete and ready for the masses the tools will be also. Until then be happy with what they put out for testing.
Sorry, my misunderstanding. I thought the deflicker tool was completely separate to format 2.0 and there may have been a separate thread that I had missed :(
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Sthirasukha on September 07, 2013, 11:58:42 AM
@RenatoPhoto
Can you explain me the logic of the Raw file value number while recording?
Sometimes it goes like this in a very logical order:
M20-1240.raw then M20-1241.raw, M20-1242.raw
and sometimes it goes like this :
M20-1250.raw and the shot after can be M21-1263.raw and not M20-1251.raw as expected. Did I miss something?
I use MovieSlate to synchronized the soundtrack and this could be helpful to understand the way it works to organize my shooting.
Thanks again to ML team for your amazing work!!!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on September 07, 2013, 12:00:16 PM
Look at the clock ;)

1263 must be a bug, can you show a screenshot?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 07, 2013, 01:57:50 PM
did you check the camera clock?
12:63 is quite a weird time :)

(or maybe it was 12:59 and you recorded four video in that minute?)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on September 07, 2013, 02:02:52 PM
Correct; though very unlikely in normal usage.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 07, 2013, 02:12:30 PM
hehe ok didnt see you already replied :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Sthirasukha on September 08, 2013, 01:08:29 AM
Thanks a1ex and g3gg0 for your quick answer! I understand now. It was so obvious that I didn't get it...
As I didn't know it was the date, I put "a kind of" example and the only reason it's weird is because I thought I could have that kind of number and I didn't check the exact number... Just a little bit slow to understand simple things! Sorry but let me thank you again and again for your great job!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: PhilK on September 08, 2013, 11:15:47 AM
I reported the weird time filename in the 50D thread, I went from a 59 to a 60 with two filenames.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 08, 2013, 01:45:51 PM
(or maybe it was 12:59 and you recorded four video in that minute?)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dadinio13 on September 09, 2013, 06:27:05 PM
i got a ideas : why not use a module for copying immediatly after recording in spanning mode the .M00 file of the SD card to the CF card ?
that is possible ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 09, 2013, 10:28:53 PM
i got a ideas : why not use a module for copying immediatly after recording in spanning mode the .M00 file of the SD card to the CF card ?
that is possible ?

well, i had the idea of a separate module that checks if the camera is idle for a while (e.g. 10 seconds) and then starts
copying all files from CF card to the (WiFi-)SD card for transferring them to the computer automatically.
as soon the files are being copied over to the computer, they are deleted from the cards.

this of course can be combined with copying MLV files to CF card when the camera is idle.
didnt think about that yet, thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: chmee on September 09, 2013, 11:25:08 PM
the criteria for the right moment or enough space are a playfield for not needed code. my opinion. its just one less step on the main computer (or in the file manager, if you're sure time/space is right for moving files)

regards chmee
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on September 09, 2013, 11:36:17 PM
guys, is file spanning present in the newest 5d3 nightly builds?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 09, 2013, 11:52:15 PM
no, as this module is not in nightly build....
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 10, 2013, 04:38:53 AM
Have you sorted out how to merge format 2.0 to unified?  I see the pull request but seems to be stuck, any further discussions needed?  I was hoping we could get a bit of speed boost from some of Alexs work to the mlv files.  That would be awesome.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: pascal on September 10, 2013, 09:57:24 AM
Looking forward to this feature. Are there already estimates what write speeds will be possible with current card tech and what resolutions at continuous 60 fps ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 10, 2013, 10:17:22 AM
Have you sorted out how to merge format 2.0 to unified?  I see the pull request but seems to be stuck, any further discussions needed?  I was hoping we could get a bit of speed boost from some of Alexs work to the mlv files.  That would be awesome.

the pull request is still pending and i fixed some discussion points already.
we are still not 100% sure about properties, as reading admin props (serial number) is something we never tested thoroughly.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Canon eos m on September 10, 2013, 05:11:53 PM
Hi, please help:

I have downloaded the latest 5DM3 ML build from Alex's thread:

http://www.magiclantern.fm/forum/index.php?topic=7139.0

However, as soon as I load the mlv_rec.mo to the module directory, I get loading error (even with the raw_rec.mo on 'off' mode).

How can I manage to run mlv_rec.mo as an alternative to raw_rec.mo when I want?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on September 10, 2013, 05:35:43 PM

Do you already try to format the card and install everything again?  Some times for some errors it works.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 10, 2013, 07:36:47 PM
Quote
I get loading error (even with the raw_rec.mo on 'off' mode).

What error? Missing symbols?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 10, 2013, 08:06:04 PM
Hi, please help:

I have downloaded the latest 5DM3 ML build from Alex's thread:

http://www.magiclantern.fm/forum/index.php?topic=7139.0

However, as soon as I load the mlv_rec.mo to the module directory, I get loading error (even with the raw_rec.mo on 'off' mode).

How can I manage to run mlv_rec.mo as an alternative to raw_rec.mo when I want?

As far as i know Alex build does not have the mlv function yet since g3ggo work has not been merged.
You have to download or compile yourself the entire ML from this thread.  g3ggo has provided raw_rec.zip for 5d3
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Flocksock on September 11, 2013, 04:50:23 PM
My questtion is...
Is it only a container format, or is it more than that?
The thing is. DSLR like 60D will not be able to record RAW nearly FullHD
and more than ca.10 seconds.
But it would be so great to not record in h264.
Something like "uncompressed png, tiff, photojpg".
Not RAW, but also not h.264.

Question is... would it be possible to reencode in camera in
a new file fomat which is not exactly raw but better than h.264.
Something like 8-bit RAW instead of 14-bit RAW.
Simple to minimize the files size, so that Cameras like D60 could
record almost FullHD in "better than h.264" quality.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 11, 2013, 05:20:26 PM
Is it only a container format, or is it more than that?

MLV is a container format for raw video (primary and obvious use case) and also other formats like MJPEG, H264 etc.
other types than raw just weren't specified yet
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Midphase on September 11, 2013, 06:38:09 PM
Question is... would it be possible to reencode in camera in
a new file fomat which is not exactly raw but better than h.264.
Something like 8-bit RAW instead of 14-bit RAW.
Simple to minimize the files size, so that Cameras like D60 could
record almost FullHD in "better than h.264" quality.

This has been discussed so many times, and answered. Please do a search before posting.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mkrjf on September 13, 2013, 06:55:27 PM
This is all great stuff (see I don't always whine ;).

Since some of the motivation for the raw container is performance, are there any back of the envelope numbers on perf / resolutions for 5dmk3raw? Also while this is wandering into dual iso territory - will the actual bit depth of data be the same?
I searched in the forum - but sometimes you need to know what to search for first ;)

And as someone hinted - will there be raw container support for sync-sound?
Thx
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 13, 2013, 08:13:45 PM
This is all great stuff (see I don't always whine ;).

Since some of the motivation for the raw container is performance, are there any back of the envelope numbers on perf / resolutions for 5dmk3raw? A
Same thread here: http://www.magiclantern.fm/forum/index.php?topic=7122.msg66931#msg66931
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 13, 2013, 09:21:04 PM
Pretty much same conclusions I reached... on SD/CF MLV was 1-2MB/s slower... only 5DIII can use the spanning
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 13, 2013, 09:55:03 PM
And as someone hinted - will there be raw container support for sync-sound?

the container is designed to get perfect sync between audio and video.
there is just no implementation for audio yet.

anyone wants to implement?

about performance:
yeah, the new format sacrifices a few MiB/s, but i think there is still some room for optimization.
but the numbers are not always that important - with the new format a lot of other advantages come up - metadata on nearly per-frame basis :)

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 14, 2013, 04:25:40 PM
the container is designed to get perfect sync between audio and video.
there is just no implementation for audio yet.

anyone wants to implement?


I love RAW but all my videos are muted!  I will not carry additional recording audio because it is too complex for wildlife videos.  I really would love to have this implemented or to bring back the out of synch video-audio option.  There is a lot of people like me that could use it!  Not all video recording is done in a studio!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Muf on September 14, 2013, 04:27:41 PM
I love RAW but all my videos are muted!  I will not carry additional recording audio because it is too complex for wildlife videos.  I really would love to have this implemented or to bring back the out of synch video-audio option.  There is a lot of people like me that could use it!  Not all video recording is done in a studio!
Then you'll have to wait. RAW video is after all still experimental.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on September 14, 2013, 05:35:35 PM
An audio thread pertaining to the current sub-discussion: http://www.magiclantern.fm/forum/index.php?topic=7890.msg70058#msg70058
It would should also work for the H4N...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 14, 2013, 05:49:13 PM
Quote
yeah, the new format sacrifices a few MiB/s, but i think there is still some room for optimization.

There definitely is... write test LV open in play mode gives 91.8MB/s... something is using the CPU/taking edmac time.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Doyle4 on September 15, 2013, 01:47:26 PM
Im sure there was a build for 600d that had the mlv module with it... cant seem too find it anymore  :(
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 15, 2013, 03:31:03 PM
So how is this module coming along? Feels like it is some time since one read news :)
Still waiting eagerly for the 5d2 version :P
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 16, 2013, 04:21:34 PM
@ g3g00:  When I compile the modules in https://bitbucket.org/g3gg0/magic-lantern/commits/all  I get the same error in all modules:

Any help would be appreciated.

[email protected]:~/G3GGo/magic-lantern/modules/autoexpo$ make clean
[ RM ]  autoexpo.o autoexpo.mo autoexpo.sym autoexpo.dep module_strings.h *.o *.d *.bmp *.dep *.sym
[email protected]:~/G3GGo/magic-lantern/modules/autoexpo$ make
[ README   ]   module_strings.h
[ DIFF     ]   module_strings.h
/bin/sh: 1: ../module_hginfo_create.sh: Permission denied
make: *** [module_strings.h] Error 126

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on September 16, 2013, 05:20:26 PM
Make module_hginfo_create.sh executable

or change Makefile.modules like this:

Code: [Select]
diff -r e428f411e2ca modules/Makefile.modules
--- a/modules/Makefile.modules  Sun Sep 15 10:55:26 2013 +0200
+++ b/modules/Makefile.modules  Sun Sep 15 14:50:52 2013 +0200
@@ -26,7 +26,7 @@
 
 module_strings.h: FORCE README.rst
        $(call build,README, $(PYTHON) ../readme2modulestrings.py > [email protected])
-       $(call build,DIFF, ../module_hginfo_create.sh >> [email protected])
+       $(call build,DIFF, sh ../module_hginfo_create.sh >> [email protected])
 
 README.rst:
        $(call build,README,echo -e "$(MODULE_NAME)\n========\n\nPlease write a one-page description for your module here.\n\nYou may also add some metadata:\n\n:Author: <your name here>\n:License: <your license here>\n:Website: <your website here>\n\n" > [email protected])

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 16, 2013, 09:01:50 PM
Make module_hginfo_create.sh executable

can you provide a patch for this?
on windows i cant do that :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on September 16, 2013, 11:33:04 PM
Oh, Windows, I see! I didn´t realize, that there exist a operating system called Windows, too. :)
My solution is for compiling from a Linux-machine.

I´m not sure, Windows can understand bash-scripts (foo.sh) natively.

Using Windows I cannot help, because I only use Linux

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 17, 2013, 01:15:24 AM
MSYS is really nice... like power shell + cmd + bash all together... it let me use wildcards in cr2hdr when normal cmd wouldn't. Would probably execute the .sh scripts.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on September 17, 2013, 04:12:58 AM
Been out of a loop for a bit here... wondering where I can find the newest compiled version.

Things in this forum are getting a bit hard to follow...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kaktus on September 20, 2013, 12:52:16 AM
Cant use mlv_rec.mo on 6D. Message "OldAPI". I dont known why... Using "DualIso-FullPack" at 2013-08-11. What am doing wrong? What do I need to adjust?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 21, 2013, 02:43:29 AM
in the main post in this thread thread is a _very_ simple and hackish MLV viewer to check your footage
it will read uncompressed MLV files and display the frames with just a few frames per second.
this tool was programmed in C# on windows, but it uses nothing windows-specific, so it should run on any OS using mono.

please remind:
 - this tool is just a PROOF OF CONCEPT
 - it is not meant as productional tool
 - i used it to check what is necessary to decode and view MLV files, its just my playground
 - it has bugs!
 - it will most likely not be continued
 - i shared it as a last resort tool in case you need something like that


notes:
 - shows the video in half resolution to simplify debayering
 - ramps exposure up/down if there is under/overexposure (so it may not be accurate enough for some of you)
 - has no white balance algorithm
 - just tested on 5D3, other cameras have different bayer patterns - didnt check them yet
 - it uses the coefficients from the raw info block, so color weighting should be correct
 - the scene is scaled to TV black and white levels (16..235) for a better looking playback

thanks to alex for giving me a lot of hints on how to process raw data efficiently

(http://upload.g3gg0.de/pub_files/e76d16212357343b54eeac44f2e96002/MLV_Viewer.png)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on September 21, 2013, 02:54:16 AM
Sweet  :D THANKS  :D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: britom on September 21, 2013, 04:55:53 AM
NICE!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 21, 2013, 08:21:03 AM
That is awesome, allways wanted to be able to preview files before extracting etc =)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Stedda on September 21, 2013, 01:58:17 PM
Awesome work!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on September 21, 2013, 03:04:47 PM
here (http://upload.g3gg0.de/pub_files/d9fe3b7542769792920dfae6bd82fad5/mlv_view_sharp.exe) is a _very_ simple and hackish MLV viewer to check your footage
it will read uncompressed MLV files and display the frames with just a few frames per second.
this tool was programmed in C# on windows, but it uses nothing windows-specific, so it should run on any OS using mono.

Thank you g3g00, and Alex for the help!

I tried it with mlv files from 5D3 and 7D and works very well at all resolutions, cropped video, and 60 fps.  Also payed spanned files!

In windows 7 64 bits I open the program and the drag the file, after a little delay the player functions properly.

I hope your pull request is merged to unified soon!

Thanks for your great work and support. ;D ;D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 21, 2013, 04:22:58 PM
thanks for the feedback.
the initial delay is when the tool scans all MLV chunks and creates an index.
so it knows the order of the blocks to process.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 21, 2013, 05:18:54 PM
update:

 - fixed some crashing situations e.g. when pressing alt (downside: inefficient code)
 - use color matrix from the raw info instead of hardcoded, so the colors should look better
 - scaled colors to TV range (16..235) for a nicer look
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: hookah on September 21, 2013, 06:28:11 PM
Wow what a great tool !!!! VERY useful..
Just cleaned-up/deleted all my small test MLV videos  :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 21, 2013, 08:20:12 PM
Does one dare to ask if there is a module that works on the 5D2 yet? =)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 21, 2013, 10:35:09 PM
i dont know what is wrong on 5D2. there were just reports of "hanging" module.
can you explain the problem in detail?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Jakobmen on September 21, 2013, 11:51:59 PM
any new pre-builts i can play with=) ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 22, 2013, 12:35:15 AM
I think with the latest commits I can only use the bigger modules on DigicV.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 22, 2013, 01:22:33 AM
i updated the MLV player again - see main post.
now it supports resizing and different debayering algorithms.

parts of it are faster now, others are slower.
in the end its about the same speed as the version before ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 22, 2013, 12:08:32 PM
i dont know what is wrong on 5D2. there were just reports of "hanging" module.
can you explain the problem in detail?

I can try again, can you point me to the latest build of the module that should work?
(I tried one before, and I was the one saying it froze, gave me speeds of 6-7MB/s instead of 60-70 when it works)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Thomas Worth on September 22, 2013, 02:28:12 PM
now it supports resizing and different debayering algorithms.
Where did you get the debayer code? Did you write it yourself or are you using code from say, dcraw?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 22, 2013, 04:20:54 PM
Wrote it on my own.  It isn't much code.  Just simple math
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on September 23, 2013, 06:54:38 PM
I'm having really nice success with the latest updates found. Just have an issue with indicators roling over at 45GB and freezing on card full.

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 23, 2013, 07:25:35 PM
When will the format be available in the nightly builds? :) If it will.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: baldand on September 24, 2013, 08:40:58 AM
I've also made a simple player for both MLV and RAW file sets. It uses OpenGL shaders for the demosaicing to keep CPU usage lower. It's written in python, so it works at least on Linux and Mac (command line only though). I haven't tried on Windows but it might be possible to make it work there also with little or no changes.

Link and info about the tool can be found in my blog post here: http://thndl.com/?20
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: ted ramasola on September 24, 2013, 08:46:01 AM
I've also made a simple player for both MLV and RAW file sets. It uses OpenGL shaders for the demosaicing to keep CPU usage lower. It's written in python, so it works at least on Linux and Mac (command line only though). I haven't tried on Windows but it might be possible to make it work there also with little or no changes.

Link and info about the tool can be found in my blog post here: http://thndl.com/?20

that's great news! I do look forward to it being available for windows.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: baldand on September 24, 2013, 10:43:37 AM
that's great news! I do look forward to it being available for windows.

It turns out it also works fine on Windows (at least Win7) as-is using a python distro like WinPython 2.7 (See https://code.google.com/p/winpython/ )

If you install that and use it to compile the included bitunpack module it will decode a bit faster.
E.g. from the source directory something like:
c:/WinPython/python/python setup.py build_ext -c mingw32 --inplace
(though I had to remove -mno_cygwin calls from python/lib/distutils/cygwinccompiler.py first)
Then copy bitunpack.pyd from the build subsubdirectory
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on September 25, 2013, 06:45:21 AM
hi,

mlv really sounds interesting and i think we all hope that audio will make it to the final verison,
can someone post a link to a actual build with mlv for 5d mark3... have no skills to build my own one. thx.



thx.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Malex on September 25, 2013, 10:00:06 AM
Hi, I am following from far what's happening with the MLV format and I think it's pretty exiting, so I was wondering, do you think or hope that in the future, softwares like Davinci Resolve, After effects, Premiere, and others will read that format ? Will they debayer proxy directly!?
If that happens, it would be crazy for Canon, they'd have to hire you guys! (to make you stop that nonsense!) or sue you.
What are you thoughts on that?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on September 25, 2013, 01:07:17 PM
hi,

mlv really sounds interesting and i think we all hope that audio will make it to the final verison,
can someone post a link to a actual build with mlv for 5d mark3... have no skills to build my own one. thx.



thx.

Likewise but for 5D2, cant find a newer one that should work besides the old 50D one =)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 25, 2013, 07:01:38 PM

If that happens, it would be crazy for Canon, they'd have to hire you guys! (to make you stop that nonsense!) or sue you.
What are you thoughts on that?
we dont think about any of these options
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Midphase on September 25, 2013, 07:44:57 PM
I was wondering, do you think or hope that in the future, softwares like Davinci Resolve, After effects, Premiere, and others will read that format ? Will they debayer proxy directly!?

I think there's a pretty good chance that Premiere Pro and Aftereffects will once the .mlv format is fully finalized. Adobe has hinted at it and I don't think they'd have anything to lose by adding that format in much the same way they deal with RED's proprietary format.

Regarding Canon, I don't think they care. They continue to sell cameras, and I'm sure they've seen a spike in sales of the 5D3 since the ML raw came out. Given the recent offerings from Blackmagic, I don't think anyone who is serious about video would consider buying a DSLR anymore if it wasn't for ML raw, so for that I think Canon must be pretty thankful to these guys (whether they'd admit it or not).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: enliten on September 26, 2013, 12:07:41 PM
hey guys, what's the CPU overhead whilst recording raws?

I was just wondering if it was possible to compress a frame every now and then to "improve" bandwidth. With the meta data included in MLV does it matter if the frames are written out of order?

-Ben
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 26, 2013, 12:11:03 PM
compression -> no

there is nearly no cpu involved, thats the reason why its possible.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: enliten on September 26, 2013, 12:19:37 PM
no compression because it won't work or because it's too hard or because it isn't a priority?

(I know that this would have to be a wholly software implementation, thus why i was suggesting only a frame here or there)

EDIT: i missed a letter
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 26, 2013, 12:34:08 PM
the cpu is too slow to compress 100MiB/s in realtime
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 26, 2013, 03:33:48 PM
@Enliten  Compression takes time.  A lot more time than de-compression.  No matter what your source of data.   So why would you want to add the time to compress onto the time needed to read the data stream in the first place? 

I think what you're asking is why can't you throw out a bunch of data you don't need while showing it, to improve speed.  That sounds logical.  But if you think about for a bit you'll realize that's just another type of compression (which in a sense is already done when the software translates the bytes into data that can be used by the display). 

What the devs are probably thinking about is taking the time to compress the stream for easy playback (either at once, or buffering).  Realize however, that it would take their time and then you'd have people complaining about load times.  (Again, no free lunch). 

Some people on the forum get upset if they feel you haven't done your research first, or if you ask a question you could have answered yourself if you thought about it for a bit.  Some are here on the forum to get things done and others to learn.  So you'll get mixed responses.  I recommend going very slowly.  Try not to ask any question like "it's not a priority?" that may suggest you think they're blowing certain issues off.  Everyone is working on this for free, wishes they had more time, and is already upset that they can't do all they want to do.  Everyone sets priorities for themselves.  There is no central command.

Also, g3gg0 has put his MLV viewer up on Bitbucket.  You're welcome to clone and push any ideas you have!

Hope this helps! 

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 26, 2013, 05:49:52 PM
Everyone is working on this for free, wishes they had more time, and is already upset that they can't do all they want to do.

thanks, wise words :)
(at work atm and painting powerpoint slides. fu*k, yeah, hardcore lowelevel developer)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Greg on September 26, 2013, 06:03:21 PM
the cpu is too slow to compress 100MiB/s in realtime

This result for the DIGIC 5+?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: enliten on September 26, 2013, 06:42:04 PM
@maxotics hey, yes i did very little research going into that question.

please be aware that it is never my intent to offend, i just like problems and see a challenge behind a "no". which is why I wanted elaboration.

I did know that realtime compression isn't viable which is why I asked what the CPU overhead is and made the comment about frame numbers.

Let me throw you a hypothetical.

Let's say the devs are able to take a frame that is about to be dumped onto the card, and instead threw it onto camera memory for processing. The camera then compresses this frame and then dumps it onto the card when it is able to next (after having compressed it).

Lets say the 5th frame is the one that gets grabbed for compression. The frames would be written to card 1,2,3,4,6,7,8,9... and once compression has finished then 5,10,11. So it would be a pseudo-realtime compression. Taking a frame here or there and compressing, rather than every frame. The thing that would allow this to be completed is the new MLV format with has frame metadata, allowing the frame to be uncompressed by whatever tool supports MLV.

Of course this is only theoretical, which is why I asked the question in the first place.

Apologies for not elaborating better in the first place.

-Ben
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: maxotics on September 26, 2013, 06:46:14 PM
( painting powerpoint slides. fu*k, yeah, hardcore lowelevel developer)

PPT work?  You're going to seriously depress everyone!

BTW, I have tons of VBA code in Office (mostly Excel, Access and Word). If you ever need some VBA to do something stupid (so you can spend more time with C), PM me.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 26, 2013, 08:11:43 PM
if you have a brain-to-ppt converter, that outputs manager-compatible output, let me know.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on September 26, 2013, 08:36:38 PM
Lets say the 5th frame is the one that gets grabbed for compression. The frames would be written to card 1,2,3,4,6,7,8,9... and once compression has finished then 5,10,11. So it would be a pseudo-realtime compression. Taking a frame here or there and compressing, rather than every frame. The thing that would allow this to be completed is the new MLV format with has frame metadata, allowing the frame to be uncompressed by whatever tool supports MLV.

good point, but i guess it will not work either.

points against:
 - increasing cpu load / memory accesses will slow down CF DMA (concurrent SDRAM accesses)
 - locking pages for the frame to compress reduces writing efficiency (one buffer less)
 - the real benefit is low, as compressing is slow
 - buffer sizes will vary and thus we might see a writing slowdown due to this too

but reducing bit depth to 10 bits could help a bit. on the other hand, i guess the slowdowns cannot be
compensated using some frames that are 5/7th of their original size.

but i like the idea ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: enliten on September 27, 2013, 07:27:44 AM

points against:
 - increasing cpu load / memory accesses will slow down CF DMA (concurrent SDRAM accesses)


thanks for the feedback, out of interest does the above point mean that any further reduction of CPU / memory usage will potentially increase write speeds?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: JCBEos on September 29, 2013, 10:22:37 PM
Keep your personal stuff off this board!

If you read this thread you will find this on reply 218:

and a bit lower you will find this:

So try it and see.  If it is not compatible with current build then wait until dev makes it available.

Thanks for this post Renato, it helped me a lot
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Shield on October 01, 2013, 01:39:30 PM
Just popping my head in here to find out if any progress has been made getting the audio working / back in the raw builds / synched.
Thanks!
Shawn
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on October 03, 2013, 01:32:39 AM
Can someone please give me a link to the latest build?  I've been out of the loop for a long while now... :-/ 

Is there some central place where it will be easiest to find the module?

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 03, 2013, 02:32:46 AM
Can someone please give me a link to the latest build?  I've been out of the loop for a long while now... :-/ 
Unfortunately MLV has not made it yet to the unified builds so it is not available through nightly builds.  You will have to compile it yourself or find a complete build in this thread that you can use.
1% has incorporated the MLV into some of his builds including 7D, and others but dont know which.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: AnotherDave on October 03, 2013, 02:35:35 AM
Unfortunately MLV has not made it yet to the unified builds so it is not available through nightly builds.  You will have to compile it yourself or find a complete build in this thread that you can use.
1% has incorporated the MLV into some of his builds including 7D, and others but dont know which.

I know.  I'm looking for the most current build that works for the 5D3... i have no idea how to compile something...  I have a few days to mess around with any-all new features and I wanted to see MLV's progress on the mac.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: moroz on October 04, 2013, 06:03:05 PM
Hello,
i intend to switch from to 5D2 to 5D3. (never used a 5D3 yet)
i read about file spanning  and got confused.
i know it's a silly question but does it mean that the raw files will be recorded on both CF and SD card?
or the SD card is only used as an additional buffer?
if i  buy a 1000x 64GB CF , will it be necessary to buy a 64GB SD card as well ?
thank you for your help
 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 04, 2013, 07:32:56 PM
i know it's a silly question but does it mean that the raw files will be recorded on both CF and SD card?
...
if i  buy a 1000x 64GB CF , will it be necessary to buy a 64GB SD card as well ?

correct. when enabled, you get some 10MiB/s extra.
this is optional, it can be enabled if you wish.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Midphase on October 04, 2013, 07:37:05 PM
Hold your horses there.

I was under the impression that 64gig SD cards are not preferred size-wise if you install ML on them...is that right?

I assume that the CF and SD cards can be different sizes, they don't have to match....right?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 04, 2013, 07:42:25 PM
I assume that the CF and SD cards can be different sizes, they don't have to match....right?

in this case, size doesn't matter.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: gtunglaub on October 04, 2013, 09:33:02 PM
Is this build worth installing? Or are the nightlies better?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: moroz on October 04, 2013, 09:59:24 PM
correct. when enabled, you get some 10MiB/s extra.
this is optional, it can be enabled if you wish.

sorry but i don't understand. how we can  record the same file on 2 separate cards??
meaning that we'll have to empty both CF and SD when they are full?
meaning additional data transfer time?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 04, 2013, 10:05:43 PM
MLV format supports multiple chunks, parallel or sequential.
dont care about that - when enabled you get multiple files on both cards.
thats your video footage to process using mlv2dng or any other supported tool.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: moroz on October 04, 2013, 11:08:16 PM
g3gg0 thank you for your answers. but  i have the impression that it might slow down the pace on the field (double  data transfer rate, extra SD cards = extra budget...)
what do you think ?  do you recommend MLV for a short film production? (considering that we have limited time for shooting )
 
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 04, 2013, 11:27:19 PM
then use one card only... it is an OPTION
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: moroz on October 05, 2013, 01:34:09 AM
understood!
i apologize if this question has been asked before:
concerning the stability,  what do you recommend for a low budget short film? newest nightly build or the MLV v2.0 ?
thank you
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Midphase on October 05, 2013, 07:22:40 PM
Considering that .mlv is still being worked on, I think the answer is pretty clear.  ;D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 05, 2013, 11:27:41 PM
i updated the module.
http://upload.g3gg0.de/pub_files/1610c3f4dba6ad5dc12e84c626ddfd32/raw_rec_mlv.zip

it has some improved writing/opening/closing code, so it may show increased write performance, especially with longer takes.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 06, 2013, 12:44:12 AM
source?

New raw2dng.exe is in there if anyone needs to update :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 01:38:05 AM
updated the mlv fork

https://bitbucket.org/g3gg0/magic-lantern-mlv

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on October 06, 2013, 11:35:17 AM
New raw2dng.exe is in there if anyone needs to update :)

What's new?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: sebamax on October 06, 2013, 12:41:18 PM
Before yesterday updating raw_rec_mlv.zip the App MLV converter for MAC OSX worked fine, now, as I convert the file it creates an empty folder.
In the camera as I use preview, it doesn't work and I see a black screen.
Does anyone have the same problem?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 06, 2013, 01:52:15 PM
updated the mlv fork

https://bitbucket.org/g3gg0/magic-lantern-mlv
I have been compiling and testing mlv for 5D3 form this source:
https://bitbucket.org/g3gg0/magic-lantern/commits/all

Now you update:
https://bitbucket.org/g3gg0/magic-lantern-mlv/commits/all

Should I setup to compile from this new fork?  Why did you not update the other fork?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on October 06, 2013, 01:56:37 PM
Any news regarding 5D2? =)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:07:17 PM
https://bitbucket.org/g3gg0/magic-lantern/ is my main working fork.
it contains more than just the mlv format, but also some other stuff.

https://bitbucket.org/g3gg0/magic-lantern-mlv is an excerpt of the mlv related changes,
intended for merging into main repository. there is an pull request for this fork.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:09:31 PM
https://bitbucket.org/g3gg0/magic-lantern/ is my main working fork.
it contains more than just the mlv format, but also some other stuff.

Is it possible to compile mlv_rec from your source to cameras other than 5d3?
I've tried it for 650D but without success. Got error 1 at the end.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:13:51 PM
Before yesterday updating raw_rec_mlv.zip the App MLV converter for MAC OSX worked fine, now, as I convert the file it creates an empty folder.
In the camera as I use preview, it doesn't work and I see a black screen.
Does anyone have the same problem?

can you open these files with the MLV viewer?

what do you mean with preview and black screen?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:18:46 PM
Any news regarding 5D2? =)
I can try again, can you point me to the latest build of the module that should work?
(I tried one before, and I was the one saying it froze, gave me speeds of 6-7MB/s instead of 60-70 when it works)

you wanted to try it again, you said a few weeks ago ;)

Is it possible to compile mlv_rec from your source to cameras other than 5d3?

the module is not camera-dependent, you can use it for any model.

it should work with any camera on which the original raw_rec works too.
but it may require some symbols in the ML/MODULES/<model>.SYM, so you may get an error.

which error do you get? which model?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:24:16 PM
which error do you get? which model?

Just updated that message: 650D, Error 1.
.sym compiles fine, but when i try to make raw_rec, it gives me this:
(http://ipic.su/img/img7/fs/kiss_34kb.1381061946.jpg)

If you have a couple of mins, would you mind to describe me, what i have to do exactly to try mlv on 650d?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:35:15 PM
can you run hg update and retry?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:38:20 PM
can you run hg update and retry?

still the same
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:42:57 PM
hg pull
hg update
make clean
make

:)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 06, 2013, 02:44:58 PM
Just updated that message: 650D, Error 1.
.sym compiles fine, but when i try to make raw_rec, it gives me this:
(http://ipic.su/img/img7/fs/kiss_34kb.1381061946.jpg)

I get this kind of error if I do not compile the autoexec.bin first.

So if I compile the make 5D3 and then then compile the modules, i do not get the dep error.
I tried to compile the 650D but got an error, so I compiled for 5D3 and then do the module.
Maybe that will fix it.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:46:53 PM
hg pull
hg update
make clean
make

I've just deleted all the folders, cloned the whole copy of this branch https://bitbucket.org/g3gg0/magic-lantern/
.sym make works, but raw_rec make gives this:
(http://ipic.su/img/img7/fs/kiss_36kb.1381063246.jpg)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 02:47:44 PM
yeah right, you have to recompile the ml binary too, because of the stubs.
sorry, didnt tell you that that what i wrote was meant to be executed in the root directory of the clone :)

so now it works?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:51:25 PM
yeah right, you have to recompile the ml binary too, because of the stubs.

Of course, first i went to the platform/650d.104 and make. It makes ok, and gives me autoexec and .sym
But then, when trying to make raw_rec module, the error still the same
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Rewind on October 06, 2013, 02:55:47 PM
wait, i think i've got it )
Thank you!
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 03:52:39 PM
please can you add this to stubs.S for 600D

Code: [Select]
NSTUB(0xFF1F5C80, msg_queue_count)
Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 03:59:59 PM
done, thanks
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 04:04:57 PM
I cannot compile mlv_dump

Code: [Select]
[ GCC      ]   mlv_dump
In file included from ../../src/raw.h:203:0,
                 from ../lv_rec/lv_rec.h:24,
                 from mlv_dump.c:39:
../../src/menu.h:170:22: warning: padding struct to align ‘help’ [-Wpadded]
../../src/menu.h:259:13: warning: padding struct to align ‘split_pos’ [-Wpadded]
../../src/menu.h:261:1: warning: padding struct size to alignment boundary [-Wpadded]
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaLib.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(Alloc.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaDec.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaEnc.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzFind.host.o)' is incompatible with i386 output
collect2: error: ld returned 1 exit status
make: *** [mlv_dump] Fehler 1

If I change the makefile like this, I can compile

Code: [Select]
# large file support
# get them with $(shell getconf LFS_CFLAGS)
LFS_FLAGS=-DLARGEFILES -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
# MLV_CFLAGS=-DMLV_USE_LZMA -Wpadded -mno-ms-bitfields -std=c99 -m32 -O2 -Wall $(LFS_FLAGS) -I$(SRC_DIR)
MLV_CFLAGS=-DMLV_USE_LZMA -Wpadded -mno-ms-bitfields -std=c99 -O2 -Wall $(LFS_FLAGS) -I$(SRC_DIR)

Where is my fault ( I´m running OpenSuse 64Bit)

Edgar

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 04:18:03 PM
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.

can you check if compression still works?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 04:19:31 PM
Sorry, but next problem with my 600D

If I record a mlv-file, all is fine. Now I stop the recording.
If I want to start a new record, the camera crashes.

assert.log
Code: [Select]
ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:20


Magic Lantern version : v2.3.NEXT.2013Oct06.600D102
Mercurial changeset   : db0460be09c1+ (unified) tip
Built on 2013-10-06 08:47:30 UTC by [email protected]
Free Memory  : 199K + 942K
crash.log
Code: [Select]
ASSERT: IsChunkSignature( hChunk )
at PackMemory\PackMem.c:264, task RscMgr
lv:0 mode:20


Magic Lantern version : v2.3.NEXT.2013Oct06.600D102
Mercurial changeset   : db0460be09c1+ (unified) tip
Built on 2013-10-06 08:47:30 UTC by [email protected]
Free Memory  : 198K + 943K

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 04:26:04 PM
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.

can you check if compression still works?

Not at the moment, because I get an sigsegfault, using mlv_dump

Code: [Select]
[email protected]:~/test> ./mlv_dump -r -o out.raw 1.MLV

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: '1.MLV'
   - Convert to legacy RAW
   - Output into 'out.raw'
[i] File 1.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
Speicherzugriffsfehler

The recorded file can be seen in your player

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 04:36:08 PM
Sorry, but next problem with my 600D

If I record a mlv-file, all is fine. Now I stop the recording.
If I want to start a new record, the camera crashes.


can you hg pull/hg update and recompile the module?
please also use the trace.mo and enable tracing in raw rec menu.
then send me the raw_rec.txt content.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 04:38:06 PM
Unknown Block: NULL, skipping
Speicherzugriffsfehler[/code]

can you open raw.h in src/ directory and replace the raw_info struct with this one?

Code: [Select]
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
    int32_t api_version;            // increase this when changing the structure
    uint32_t buffer;               // points to image data
   
    int32_t height, width, pitch;
    int32_t frame_size;
    int32_t bits_per_pixel;         // 14

    int32_t black_level;            // autodetected
    int32_t white_level;            // somewhere around 13000 - 16000, varies with camera, settings etc
                                // would be best to autodetect it, but we can't do this reliably yet
    union                       // DNG JPEG info
    {
        struct
        {
            int32_t x, y;           // DNG JPEG top left corner
            int32_t width, height;  // DNG JPEG size
        } jpeg;
        struct
        {
            int32_t origin[2];
            int32_t size[2];
        } crop;
    };
    union                       // DNG active sensor area (Y1, X1, Y2, X2)
    {
        struct
        {
            int32_t y1, x1, y2, x2;
        } active_area;
        int32_t dng_active_area[4];
    };
    int32_t exposure_bias[2];       // DNG Exposure Bias (idk what's that)
    int32_t cfa_pattern;            // stick to 0x02010100 (RGBG) if you can
    int32_t calibration_illuminant1;
    int32_t color_matrix1[18];      // DNG Color Matrix
   
    int32_t dynamic_range;          // EV x100, from analyzing black level and noise (very close to DxO)
};

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 04:56:25 PM
First the raw_rec.txt

Code: [Select]
/* 00:00:35.948984  */ Resolution: 896x504 @ 25.000 FPS
/* 00:00:36.097453  */ frame size = 0xC0F00
/* 00:00:36.097725  */ slot   0: pre_align = 0x00000180, edmac_size_align =  3552, write_size_align =   256, data_start = 0x42185000, size = 0xC1E00
/* 00:00:36.097999  */ slot   1: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42247000, size = 0xC2000
/* 00:00:36.098255  */ slot   2: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42309000, size = 0xC2000
/* 00:00:36.098519  */ slot   3: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x423CB000, size = 0xC2000
/* 00:00:36.098778  */ slot   4: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4248D000, size = 0xC2000
/* 00:00:36.099031  */ slot   5: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4254F000, size = 0xC2000
/* 00:00:36.099285  */ slot   6: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42611000, size = 0xC2000
/* 00:00:36.099549  */ slot   7: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x426D3000, size = 0xC2000
/* 00:00:36.099810  */ slot   8: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42795000, size = 0xC2000
/* 00:00:36.100068  */ slot   9: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42857000, size = 0xC2000
/* 00:00:36.100330  */ slot  10: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42919000, size = 0xC2000
/* 00:00:36.100590  */ slot  11: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x429DB000, size = 0xC2000
/* 00:00:36.100856  */ slot  12: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42A9D000, size = 0xC2000
/* 00:00:36.101115  */ slot  13: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42B5F000, size = 0xC2000
/* 00:00:36.101379  */ slot  14: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42C21000, size = 0xC2000
/* 00:00:36.101640  */ slot  15: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42CE3000, size = 0xC2000
/* 00:00:36.101909  */ slot  16: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42DA5000, size = 0xC2000
/* 00:00:36.102160  */ slot  17: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42E67000, size = 0xC2000
/* 00:00:36.102421  */ slot  18: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42F29000, size = 0xC2000
/* 00:00:36.102938  */ slot  19: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x42FEB000, size = 0xC2000
/* 00:00:36.103243  */ slot  20: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x430AD000, size = 0xC2000
/* 00:00:36.103507  */ slot  21: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4316F000, size = 0xC2000
/* 00:00:36.103775  */ slot  22: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43231000, size = 0xC2000
/* 00:00:36.104039  */ slot  23: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x432F3000, size = 0xC2000
/* 00:00:36.104304  */ slot  24: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x433B5000, size = 0xC2000
/* 00:00:36.104563  */ slot  25: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43477000, size = 0xC2000
/* 00:00:36.104917  */ slot  26: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43539000, size = 0xC2000
/* 00:00:36.105186  */ slot  27: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x435FB000, size = 0xC2000
/* 00:00:36.105471  */ slot  28: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x436BD000, size = 0xC2000
/* 00:00:36.105751  */ slot  29: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4377F000, size = 0xC2000
/* 00:00:36.106049  */ slot  30: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43841000, size = 0xC2000
/* 00:00:36.106342  */ slot  31: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43903000, size = 0xC2000
/* 00:00:36.106630  */ slot  32: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x439C5000, size = 0xC2000
/* 00:00:36.106904  */ slot  33: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43A87000, size = 0xC2000
/* 00:00:36.107194  */ slot  34: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43B49000, size = 0xC2000
/* 00:00:36.107477  */ slot  35: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43C0B000, size = 0xC2000
/* 00:00:36.107764  */ slot  36: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43CCD000, size = 0xC2000
/* 00:00:36.108038  */ slot  37: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43D8F000, size = 0xC2000
/* 00:00:36.108335  */ slot  38: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43E51000, size = 0xC2000
/* 00:00:36.108621  */ slot  39: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x43F13000, size = 0xC2000
/* 00:00:36.108924  */ slot  40: pre_align = 0x00000190, edmac_size_align =  3552, write_size_align =   256, data_start = 0x4A001000, size = 0xC1E00
/* 00:00:36.109218  */ slot  41: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A0C3000, size = 0xC2000
/* 00:00:36.109515  */ slot  42: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A185000, size = 0xC2000
/* 00:00:36.109803  */ slot  43: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A247000, size = 0xC2000
/* 00:00:36.110085  */ slot  44: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A309000, size = 0xC2000
/* 00:00:36.110362  */ slot  45: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A3CB000, size = 0xC2000
/* 00:00:36.110643  */ slot  46: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A48D000, size = 0xC2000
/* 00:00:36.110927  */ slot  47: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A54F000, size = 0xC2000
/* 00:00:36.111207  */ slot  48: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A611000, size = 0xC2000
/* 00:00:36.111475  */ slot  49: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A6D3000, size = 0xC2000
/* 00:00:36.111762  */ slot  50: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A795000, size = 0xC2000
/* 00:00:36.112048  */ slot  51: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A857000, size = 0xC2000
/* 00:00:36.112338  */ slot  52: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A919000, size = 0xC2000
/* 00:00:36.112605  */ slot  53: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4A9DB000, size = 0xC2000
/* 00:00:36.112896  */ slot  54: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AA9D000, size = 0xC2000
/* 00:00:36.113182  */ slot  55: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AB5F000, size = 0xC2000
/* 00:00:36.113465  */ slot  56: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AC21000, size = 0xC2000
/* 00:00:36.113736  */ slot  57: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4ACE3000, size = 0xC2000
/* 00:00:36.114018  */ slot  58: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4ADA5000, size = 0xC2000
/* 00:00:36.114305  */ slot  59: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AE67000, size = 0xC2000
/* 00:00:36.114651  */ slot  60: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AF29000, size = 0xC2000
/* 00:00:36.114958  */ slot  61: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4AFEB000, size = 0xC2000
/* 00:00:36.115249  */ slot  62: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B0AD000, size = 0xC2000
/* 00:00:36.115535  */ slot  63: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B16F000, size = 0xC2000
/* 00:00:36.115823  */ slot  64: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B231000, size = 0xC2000
/* 00:00:36.116094  */ slot  65: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B2F3000, size = 0xC2000
/* 00:00:36.116381  */ slot  66: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B3B5000, size = 0xC2000
/* 00:00:36.116662  */ slot  67: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B477000, size = 0xC2000
/* 00:00:36.117006  */ slot  68: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B539000, size = 0xC2000
/* 00:00:36.117299  */ slot  69: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B5FB000, size = 0xC2000
/* 00:00:36.117589  */ slot  70: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B6BD000, size = 0xC2000
/* 00:00:36.117873  */ slot  71: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B77F000, size = 0xC2000
/* 00:00:36.118155  */ slot  72: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B841000, size = 0xC2000
/* 00:00:36.118427  */ slot  73: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B903000, size = 0xC2000
/* 00:00:36.118708  */ slot  74: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4B9C5000, size = 0xC2000
/* 00:00:36.118997  */ slot  75: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4BA87000, size = 0xC2000
/* 00:00:36.119282  */ slot  76: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4BB49000, size = 0xC2000
/* 00:00:36.119563  */ slot  77: pre_align = 0x00000000, edmac_size_align =  4064, write_size_align =   256, data_start = 0x4BC0B000, size = 0xC2000
/* 00:00:36.120830  */ Building a group list...
/* 00:00:36.121105  */ group: 0 block_len: 40 block_start: 0
/* 00:00:36.121360  */ group: 1 block_len: 38 block_start: 40
/* 00:00:36.121589  */ Sorting group list...
/* 00:00:36.121811  */ group: 0 length: 40 slot: 0
/* 00:00:36.122031  */ group: 1 length: 38 slot: 40
/* 00:00:36.154540  */ WARNING: This model doesn't have 'PROPAD_GetPropertyData' defined. Reading properties not possible.
/* 00:00:36.154890  */ [IDNT] err: 1 model_data: 0x00000000 model_len: 0
/* 00:00:36.256603  */ <-- WRITER#0: write took:    61386 µs ( 12633 KiB/s),    794112 bytes,  1 blocks, slot  0, mgmt      0 µs
/* 00:00:36.317562  */ <-- WRITER#0: write took:    60839 µs ( 12754 KiB/s),    794624 bytes,  1 blocks, slot  1, mgmt     64 µs
/* 00:00:36.393092  */ <-- WRITER#0: write took:    75448 µs ( 10285 KiB/s),    794624 bytes,  1 blocks, slot  2, mgmt     70 µs
/* 00:00:36.450494  */ <-- WRITER#0: write took:    57301 µs ( 13533 KiB/s),    794112 bytes,  1 blocks, slot  0, mgmt     70 µs
/* 00:00:36.505611  */ <-- WRITER#0: write took:    54873 µs ( 14141 KiB/s),    794624 bytes,  1 blocks, slot  1, mgmt     73 µs
/* 00:00:36.715451  */ <-- WRITER#0: write took:   209972 µs ( 11087 KiB/s),   2383872 bytes,  3 blocks, slot  2, mgmt     72 µs
/* 00:00:36.849533  */ <-- WRITER#0: write took:   133968 µs (  5788 KiB/s),    794112 bytes,  1 blocks, slot  0, mgmt     62 µs
/* 00:00:37.174537  */ <-- WRITER#0: write took:   325014 µs (  9550 KiB/s),   3178496 bytes,  4 blocks, slot  5, mgmt     75 µs
/* 00:00:37.591594  */ <-- WRITER#0: write took:   416950 µs (  7444 KiB/s),   3178496 bytes,  4 blocks, slot  1, mgmt     62 µs
/* 00:00:38.241110  */ <-- WRITER#0: write took:   649460 µs (  9558 KiB/s),   6356992 bytes,  8 blocks, slot  9, mgmt     71 µs
/* 00:00:38.770845  */ <-- WRITER#0: write took:   529614 µs (  8791 KiB/s),   4767744 bytes,  6 blocks, slot 17, mgmt     69 µs
/* 00:00:39.715733  */ <-- WRITER#0: write took:   944816 µs (  9855 KiB/s),   9535488 bytes, 12 blocks, slot 23, mgmt    131 µs
/* 00:00:41.145812  */ <-- WRITER#0: write took:  1429778 µs (  9226 KiB/s),  13508096 bytes, 17 blocks, slot  0, mgmt     65 µs
/* 00:00:42.582296  */ <-- WRITER#0: write took:  1436464 µs (  9723 KiB/s),  14302720 bytes, 18 blocks, slot 40, mgmt     67 µs
/* 00:00:44.429078  */ <-- WRITER#0: write took:  1846656 µs (  9665 KiB/s),  18276352 bytes, 23 blocks, slot 17, mgmt     69 µs
/* 00:00:46.218938  */ <-- WRITER#0: write took:  1790003 µs (  8670 KiB/s),  15892480 bytes, 20 blocks, slot 58, mgmt     74 µs
/* 00:00:46.865935  */ <-- WRITER#0: write took:   647156 µs ( 20383 KiB/s),  13508096 bytes, 17 blocks, slot  0, mgmt     74 µs
/* 00:00:47.064886  */    --> WRITER#0: expected to terminate
/* 00:00:56.468994  */ Resolution: 896x504 @ 25.000 FPS
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 05:29:29 PM
can you open raw.h in src/ directory and replace the raw_info struct with this one?



Done, bur now raw2dng doesn´t compile anymore

Code: [Select]
[ GCC      ]   raw2dng
In file included from ../../src/chdk-dng.c:57:0:
../../src/raw.h:200:5: error: unknown type name ‘uint32_t’
../../src/chdk-dng.c: In function ‘write_dng’:
../../src/chdk-dng.c:628:20: warning: initialization makes pointer from integer without a cast [enabled by default]
make: *** [raw2dng] Fehler 1

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 05:38:50 PM
add a
#include <stdint.h>
to the c file as first line
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 05:51:21 PM
to raw.c or to chdk-dng.c ?

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 06:07:51 PM
to raw.c or to chdk-dng.c ?

Edgar

OK, to raw.h  :)

I compiled the whole stuff new with this changes and now the sigsegfault has gone. mlv_dump runs and I can decode the file with raw2dng

ThankYou


Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 06:14:26 PM
ok thats the reason why i always use stdint types "uint32_t" etc instead of the random width types "int, long, short".
this saves you from architecture trouble.

(ok i rarely care for endianess conversion in my code and always assume we work on little endian, but thats an other issue)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 06:22:23 PM
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.

can you check if compression still works?

Compression and decompression seems to work

Code: [Select]
[email protected]:~/test> ./mlv_dump -c -l 9 -o compress.mlv 1.MLV

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: '1.MLV'
   - Rewrite MLV
   - Compress frame data
   - Output into 'compress.mlv'
[i] File 1.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
  saving: 790272 -> 740650  (93.72%)
[i] Unknown Block: NULL, skipping
  saving: 790272 -> 748475  (94.71%)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 06:39:00 PM
The camera-name looks a bit strange  :)

[email protected]:~/test> ./mlv_dump -v 1.MLV >test
Code: [Select]
...
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 28.726000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
...

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 06:57:34 PM
thats because of:
"/* 00:00:36.154540  */ WARNING: This model doesn't have 'PROPAD_GetPropertyData' defined. Reading properties not possible."

they usually can be found by searching for the string and checking who is being called.
PROP_GetMulticastProperty is usually named prop_get_value, thats the reason why it has the same address.

e.g. 7D:
/** corectly named canon property reading functions **/
NSTUB(0xFF0687C8, PROP_GetMulticastProperty)
NSTUB(0xFF06934C, PROPAD_GetPropertyData)

5D3:
/** corectly named canon property reading functions **/
NSTUB(0xFF123DE4, PROP_GetMulticastProperty)
NSTUB(0xFF12523C, PROPAD_GetPropertyData)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 07:18:08 PM
can you hg pull and recompile magic lantern again?
no need to recompile the modules.
it may be necessary to undo the stdint.h and raw.h changes you made

you then should see your camera information in the recorded movie.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on October 06, 2013, 07:43:19 PM
you wanted to try it again, you said a few weeks ago ;)

the module is not camera-dependent, you can use it for any model.

it should work with any camera on which the original raw_rec works too.
but it may require some symbols in the ML/MODULES/<model>.SYM, so you may get an error.

which error do you get? which model?

Which I did, but i noticed ppl with 5d2 posted logs and videos showing the same results.
Tried again  with the same results, lock-up.

Thought there was another recent build to try :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 07:57:25 PM
can you hg pull and recompile magic lantern again?
...
you then should see your camera information in the recorded movie.

I added the prop... per hand to stubs.S, because your repos are a little bit amazed at the moment  ;) . Now the IDNT-Block with camera-name is ok.

ThankYou

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 06, 2013, 08:38:11 PM
I'm still getting hsuite error on 7D, second record

Replaced the raw struct too.

Quote
ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:3
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 08:48:32 PM
hmm so on 600D and 7D there are such errors?
i tried a bunch of videos on 7D with some older version and had no such errors. damn.

which settings?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 06, 2013, 09:15:30 PM
Just normal stuff/ gd off + hacks.

I'm trying the binary and it says no info structure found...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mk11174 on October 06, 2013, 09:17:05 PM
Cool, I see MLV works on 550D, just made my first MLV clip.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 06, 2013, 09:37:00 PM
Code: [Select]
6D freezes when stopping 2nd record

ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:3

Have i f'ed something up?
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/0ae870f7dbd161440955320583c5eff381fb8b60/modules/mlv_rec/mlv_rec.c
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 06, 2013, 09:44:09 PM
Some other hints for this camera-crash on my 600d, maybe

I can record as many videos I want, if I switch off the camera after every video.

I record a video and stop it and than switch away from videomode. Now, if i switch back to videomode, the camera tells me: busy, please wait (beschäftigt, bitte warten)

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: mk11174 on October 06, 2013, 09:54:07 PM
6D freezes when stopping 2nd record


Same freeze on 550D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 06, 2013, 10:23:37 PM
ok there seems to be some weird shoot mem issue... hmm
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 12, 2013, 05:19:56 PM
Tested todays build on 5D3.

At the moment the spanning function records both files on CF card and SD is not being used.
Recorded files are ok.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 12, 2013, 06:58:24 PM
Ok, I found the commits, thank you for fixing it.. I see 83 -85MB/s instant writes but seems a bit inconsistent.

When I do the buffer test:

7D FULL 1x

M 0 B 3 79.1 5GB
M 0 B 3 80.1 6GB
M 0 B 3 81.9 14GB
M 0 B 3 82.1 14GB

As card becomes more full...

M1 81.2 8GB
M1  80.1 7GB
M1  72.1 1.2GB
M1  77.4 3.2GB

M2 78.1 2.5GB
M2 79.2 4.3GB
M2 70.4 1.5GB

M3 60.5 800MB
M3 73 1.5GB
M3 74.4 1.7GB

M4 71.1 1.3GB
M4 74.5 1.9GB
M4 72.2 1.4GB
M4 75.2 2GB

Raw_rec writes  a full card 2nd time recording.

Maybe this test needs to be done on 2.5K in zoom mode?



Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 12, 2013, 08:16:17 PM
raw_rec doesn´t compile with latest sources (magic-lantern-mlv) on my 600d. I have to change raw_rec.c like this:

Code: [Select]
[email protected]:~/MagicLantern/magic-lantern-mlv/modules/raw_rec> diff raw_rec_original.c raw_rec.c
68,69c68,69
< #include <math.h
< #include <cropmarks.h>>
---
> #include <math.h>
> #include <cropmarks.h>
342c342
< static unsigned int raw_rec_should_preview(unsigned int ctx);
---
> static uint32_t raw_rec_should_preview(uint32_t ctx);

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: pravdomil on October 12, 2013, 08:26:50 PM
https://bitbucket.org/g3gg0/magic-lantern-mlv/pull-request/2/5d2-fix
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 12, 2013, 09:06:20 PM
Didn´t see the pull request.  Great

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 12, 2013, 09:57:13 PM
thanks for the support!
merged in the necessary stuff for 5D2.

i still have the write performance drops every few seconds and no idea where they come from.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 12, 2013, 10:12:34 PM
600D:

Any reasons not to append these changes to the mlv-sources?  mlv_dump doesnt run wifhout for me

can you open raw.h in src/ directory and replace the raw_info struct with this one?

Code: [Select]
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
    int32_t api_version;            // increase this when changing the structure
    uint32_t buffer;               // points to image data
   
    int32_t height, width, pitch;
    int32_t frame_size;
    int32_t bits_per_pixel;         // 14

    int32_t black_level;            // autodetected
    int32_t white_level;            // somewhere around 13000 - 16000, varies with camera, settings etc
                                // would be best to autodetect it, but we can't do this reliably yet
    union                       // DNG JPEG info
    {
        struct
        {
            int32_t x, y;           // DNG JPEG top left corner
            int32_t width, height;  // DNG JPEG size
        } jpeg;
        struct
        {
            int32_t origin[2];
            int32_t size[2];
        } crop;
    };
    union                       // DNG active sensor area (Y1, X1, Y2, X2)
    {
        struct
        {
            int32_t y1, x1, y2, x2;
        } active_area;
        int32_t dng_active_area[4];
    };
    int32_t exposure_bias[2];       // DNG Exposure Bias (idk what's that)
    int32_t cfa_pattern;            // stick to 0x02010100 (RGBG) if you can
    int32_t calibration_illuminant1;
    int32_t color_matrix1[18];      // DNG Color Matrix
   
    int32_t dynamic_range;          // EV x100, from analyzing black level and noise (very close to DxO)
};



you can keep the -m32 removed maybe.


add a
#include <stdint.h>
to the c file as first line

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 12, 2013, 10:26:46 PM
can you enable the buffer graphs and check if you also get some write rate drop every now and then?

green line = buffer fill level
red line = write rate
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 01:44:20 AM
Yes, I get a drop in rate every 4 seconds approximately.
It is a repeatable drop, sometimes it skips on cycle and drops at 8 seconds, etc
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 01:55:01 AM
Also I noticed it is related to idle % calculation.  When the idle shows, the drop in rate becomes regular.  If you push the speed so the idle % disappears also the drop in rate disappears.

For some reason I am also getting these images as if I was pressing screen capture.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 13, 2013, 02:00:40 AM
wait, i didnt understand the last post.
you mean CPU idle calculation?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 02:22:02 AM
While recording RAW there is a speed indicator in MB/s and next to it appear an idle calculation.  If I start recording at low speeds where I show a big idle then the drop becomes regular every approximately 4 seconds.  If I speed up recording (by increasing frame size or fps) until idle is gone then I get less drops in speed.  If I continue to get close to the maximum speed I get no drops in speed.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 03:06:37 AM
Have you enabled some automatic screen capture?  I get bmp files and I never pressed Screen Capture.

I get best speed results with buffer fill method = 0 and  CF-only buffers= 0  At high speed like 95 to 103 MB/s.  (Sometimes zero drops)

Also at the 4GB mark always get drops in speed.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on October 13, 2013, 09:26:23 AM
Can anyone test mlv format with 48fps?

On a 5d mk3 i have problems with raw and highest resolution. The writing speed is much lower than in 24fps mode. So it is not possible to record more than 500frames.
Greets re
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 13, 2013, 10:04:04 AM
Higher fps -> more dma bandwidth needed -> reduced write speed
this is inavoidable.
you can try to switch preview mode to hacked, which will disable the live view functionality.
this squeezes some MiB/s out
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: swinxx on October 13, 2013, 10:55:01 AM
thx, will try that out.

is there a possibility to record unsqueezed one day or is this not possible?
thx.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 02:21:16 PM
is there a possibility to record unsqueezed one day or is this not possible?
Not possible, line skipping is required to get high fps.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: vstrglv on October 13, 2013, 10:04:00 PM
Tested todays build on 5D3.
Can you upload mlv module for latest build please? (5D3). I can not compile it myself, sorry.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 13, 2013, 10:18:43 PM
Can you upload mlv module for latest build please? (5D3). I can not compile it myself, sorry.
You can get it from first page look for pre built
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: vstrglv on October 13, 2013, 11:36:19 PM
Thank you very much! I tried this link before - it was very old.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 14, 2013, 12:02:23 AM
Also I noticed it is related to idle % calculation.  When the idle shows, the drop in rate becomes regular.  If you push the speed so the idle % disappears also the drop in rate disappears.

For some reason I am also getting these images as if I was pressing screen capture.

screenshots are nothing new. they are made on stop so you can upload your buffer graph.

the idle time and similar stuff is calculated nearly every write and has no correlation to the performance drop.
see the logs: http://upload.g3gg0.de/pub_files/1ccce7e3382250447649960595a0343e/RAW_REC.TXT
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 14, 2013, 12:03:16 AM
Also at the 4GB mark always get drops in speed.

thats normal and its better than in the original raw_rec versions.
file open/close operations take their time.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 14, 2013, 01:25:14 AM
the idle time and similar stuff is calculated nearly every write and has no correlation to the performance drop.

Ok, but I have seen some kind of cycle on 5D3 tests.  The drop sometimes repeats every 4 seconds, sometimes every 2 seconds, sometimes every 8 seconds.  It is not totally repeatable by I see some correlation.  The drop disappears with high demanding rates and appears more frequent when there is some idle % showing.

I hope you can find the couse
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 14, 2013, 08:25:57 AM
it could be vice versa - as soon the drop rate goes down (due to some other stuff being calcuated or so) the idle time rises.
hard to tell. i guess i will have to start a tskmon trace.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 15, 2013, 09:26:51 PM
600D:

Any reasons not to append these changes to the mlv-sources?  mlv_dump doesnt run wifhout for me

Edgar

Shall I create a pull request to ML with the changes in raw.h or will this conflict with existing stuff?

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 15, 2013, 10:13:09 PM
Shall I create a pull request to ML with the changes in raw.h or will this conflict with existing stuff?

Edgar

ah sorry, forgot to reply.
i am afraid this shotgun approach could raise some other problems :)
can you create a pull request for the main repository and check if the original raw_rec still works fine?

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 15, 2013, 11:29:25 PM
...
can you create a pull request for the main repository and check if the original raw_rec still works fine?

I changed it in my clone of ML-Repo. Raw-Video seems to run fine. I found no problems at the moment.

But

Compiling this whole stuff shows a bunch of compiler warnings like that:

Code: [Select]
../../src/raw.c:858:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_red_pixel_bright':
../../src/raw.c:866:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_green_pixel_bright':

So, regarding these warnings, I don´t have the courage to create a pull request

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: 1% on October 15, 2013, 11:50:40 PM
It just makes warnings about comparing those uint_32 to int in other parts of ML source.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 18, 2013, 01:55:10 PM
Congratulations g3goo and Alex,  I see you merged the new format RAW 2.0 to the main branch!  I have done a quick test on the 5D3 with and without the spanning function and all seems to work properly.  I will be doing more detailed testing and reporting of the new mlv_rec.mo!

This gives me a very positive feeling for the power that the ML team has, and hope that this serves as a very good example for others to follow.  The cooperation of all the players to bring their knowledge merging their code (and knowledge) back to the main branch is essential to the benefit of ML Team as a whole.   ;D
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 18, 2013, 04:32:54 PM
Only bug so far under cropped mode with preview MLGray or Hacked the LV sometimes remains frozen and requires a re-boot to enable LV.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: kgv5 on October 18, 2013, 07:55:18 PM
Congratulations g3goo and Alex,  I see you merged the new format RAW 2.0 to the main branch!  I have done a quick test on the 5D3 with and without the spanning function and all seems to work properly.  I will be doing more detailed testing and reporting of the new mlv_rec.mo!

What does it mean exactly? Is mlv in nightly builds now?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 18, 2013, 08:28:20 PM
Probably soon.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: xNiNELiVES on October 19, 2013, 12:03:22 AM
Have the bugs really been sorted out for cameras other than the 5D3? Like the 7D, and 5D2.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 19, 2013, 02:27:06 AM
Probably not!  Will have to test... Do you want to help?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: PressureFM on October 19, 2013, 10:34:15 AM
Probably not!  Will have to test... Do you want to help?

I can help test the 5D Mark II, if needed.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on October 19, 2013, 11:24:20 AM
It's there any version for the 7D of the new mlv?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 19, 2013, 11:54:01 AM
its the same .mo as for the other models.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on October 19, 2013, 12:43:46 PM
It's need to be compile? In the the first page i have download the pre_build but just find raw_rec.mo  no mlv_rec.mo

Thanks =))
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: marekk on October 19, 2013, 01:02:04 PM
I would like to test mlv on 60D. How should I look for a msg_queue_count stub ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 19, 2013, 02:29:34 PM
Let me check the missing stubs tonight
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dariSSight on October 19, 2013, 04:50:58 PM
Let me check the missing stubs tonight

So does it work for 5D Mark II?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 19, 2013, 08:55:47 PM
You know this sigsegfault:

Code: [Select]
[email protected]:~/test> ./mlv_dump -r -o out.raw *.MLV

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: 'M18-2052.MLV'
   - Convert to legacy RAW
   - Output into 'out.raw'
[i] File M18-2052.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
Speicherzugriffsfehler

This error seems to appeare, if I compile mlv_rec on a linux-platform and try to run mlv_dump -r (I cannot test windows, because I don´t have windows).

A solution, that runs for me, is to change the void pointer in the raw_info struct in raw.h to uint:


Code: [Select]
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
   ...
    unsigned int buffer;               // points to image data
   ...
};

Now I have three questions:

1. Why do I have this sigsegfault, I told above, if I use the default pointer from raw.h and compile this whole stuff in Linux?

Code: [Select]
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
   ...
    void* buffer;               // points to image data
   ...
};

2. What can be changed in mlv_rec to avoid this problem? The solution with the unsigned int buffer gives a bunch of compiler warnings like this:

Code: [Select]
../../src/raw.c:842:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_green_pixel_dark':

3. Can we use my solution until we find a better way?, If yes, I`ll create a pull request.

Sorry, if I have to ask a little bit stupid, but I´m no programmer (but I try to learn a bit  :) )

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 20, 2013, 12:41:10 AM
You know this sigsegfault:
64 bit compiler?
can you update and try again?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 20, 2013, 01:29:59 AM
So does it work for 5D Mark II?
yes
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 20, 2013, 02:21:07 AM
1. Why do I have this sigsegfault, I told above, if I use the default pointer from raw.h and compile this whole stuff in Linux?

Code: [Select]
    unsigned int buffer;               // points to image data
    void* buffer;               // points to image data;

on 32 bit systems, unsigned int is *usually* 32 bits, on 64 bit systems it is usually 64 bits.
memory pointers, no matter of which type behave the same way.

this is based on
 - native register width
 - memory address bus width
and compilers may do anything as long C standard is followed.
this even can result in unsigned int being 16 bit and char * being 64 bit.

thus developers are encouraged to use stdint.h where possible.
it defines uint32_t (8/16/32/64) types that are reliably of desired width.
as soon you write e.g. structs that should be read by other systems, you can be sure that
the other system doesnt get into trouble and reads crap.

well, still we all only think of little endian devices, so people with big endian ARM systems will
still have trouble in getting mlv_dump etc running.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: escho on October 20, 2013, 10:47:55 AM
64 bit compiler?
can you update and try again?

Yes, 64 bit compiler. Now mlv_dump run without sigsegfault. Great!

And ThankYou for your detailed explanation.  :)

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: dariSSight on October 21, 2013, 10:15:08 AM
How do I use it for with my 5D Mark II?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on October 21, 2013, 04:50:11 PM
Are there any pre-built modules working for the 5d2 that is newer than 10-3?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 21, 2013, 05:03:01 PM
the latest pre built module is from 13.10.2013.
you still have to compile the core binary for your model.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: oddname on October 21, 2013, 05:11:36 PM
Guess Ill wait for the nightlies then =P

Me and compiling stuff... nah :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: taccca on October 22, 2013, 01:38:55 PM
Hey guys, long time follower and have loved using magic lantern on my 50D :)

I really know nothing about compiling, code etc but ive got my camera and are willing to test what ever you want if you point me in the right direction (ie, what/how exactly to load the module?? and how you want me to test it)

Ive got plenty of time (just had my knee reco last week) would love to help out wherever possible.

Cheers
Damien
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: RenatoPhoto on October 22, 2013, 01:53:45 PM
A small change is needed so that "make zip" command will include the mlv_rec.mo in the nightly builds.  Should this be done so that mlv_rec.mo is tested across the different builds?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on October 25, 2013, 04:17:07 PM
Still getting issues with random corrupt frames on the 50D and just wondering if the buffering method choices are different than those of raw_rec which I don't have any problems with?

It's frustratingly close to being useable but I can't risk dropping frames and there seems to be no pattern to where and when they appear in the footage. I'm using a KB 1000x card and tried even small frame sizes but the corrupt frames still make an appearance. I also have had shots with no crap frames.

I would love to try the 50D specific DPC to Log from Visioncolor with ACR but my mlv->raw->dng files (using freshly compiled mlv_dump.exe and raw2dng.exe) lose the metadata needed and it defaults to Canikon :( How do you convert to legacy raw without losing metadata?

Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: loconetter on October 25, 2013, 04:25:41 PM
Just tried the new MLV format.  However, the "mlv2dng" is not working, likely due to being on the new Mac OS 10.9.  Any suggestions?

Thanks!

Nelson

Exception! Header not recognized: 'STYL'
dyld: lazy symbol binding failed: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Volumes/Media1/MLV/./mlv2dng
  Expected in: /usr/lib/libstdc++.6.dylib

dyld: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Volumes/Media1/MLV/./mlv2dng
  Expected in: /usr/lib/libstdc++.6.dylib

Trace/BPT trap: 5
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 25, 2013, 04:46:34 PM
Just tried the new MLV format.  However, the "mlv2dng" is not working, likely due to being on the new Mac OS 10.9.  Any suggestions?

i dont develop mlv2dng.
the developer was told to NOT THROW errors when an unknown block is seen, because this will happen any time.
dont know why this still happens.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 25, 2013, 04:58:16 PM
Still getting issues with random corrupt frames on the 50D and just wondering if the buffering method choices are different than those of raw_rec which I don't have any problems with?

is it better with lower frame rates?


I would love to try the 50D specific DPC to Log from Visioncolor with ACR but my mlv->raw->dng files (using freshly compiled mlv_dump.exe and raw2dng.exe) lose the metadata needed and it defaults to Canikon :( How do you convert to legacy raw without losing metadata?

use mlv2dng

Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?

didnt try, will check.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: Andy600 on October 25, 2013, 05:08:21 PM
is it better with lower frame rates?

No. I've tried down to 1fps and still get occasional bad frames


Quote
use mlv2dng

I just tried and it converts but model shows an error in the Exif info:   ERR:1 md:0x  0 ml:0 (this is needed for the Visioncolor Log DCP)

Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: core_32 on October 25, 2013, 08:29:50 PM
I've put CF and SD cards in my 5d3 and installed today's build - everything is working fine but I can't see the file spanning switch in the menus, my SD card seems not to be in use.
Any hint? Thanks.

upd: oops) I'm actually using raw_rec module) Am I right to think that mlv module isn't ready yet?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: heniloux on October 26, 2013, 10:59:45 PM
got two clips today on new build of 5d3 raw that came out green, looks like the other channels weren't captured, very odd...
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 26, 2013, 11:06:21 PM
No. I've tried down to 1fps and still get occasional bad frames

does this also happen (even if just rarely) with the old raw_rec module?
i would be surprised, because the share the same copy code.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 26, 2013, 11:07:22 PM
got two clips today on new build of 5d3 raw that came out green, looks like the other channels weren't captured, very odd...

can you upload one of these clips?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 26, 2013, 11:08:49 PM
Am I right to think that mlv module isn't ready yet?
it is ready. just compile and install it.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 26, 2013, 11:09:42 PM
Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?

just committed some fixes. that should work now.
can you test it?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 26, 2013, 11:22:14 PM
i added mlv_play.mo to the pre-built binaries.
if you install file_man.mo, you can browse your card's file system and open .mlv and .raw files for playback
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on October 26, 2013, 11:47:20 PM
 Im getting also random pink/green clips on footage with MLV
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: heniloux on October 27, 2013, 06:43:32 AM
(http://s18.postimg.org/8oadya3x1/Screen_Shot_2013_10_26_at_9_16_08_PM.jpg)
 (http://postimg.org/image/8oadya3x1/)
can you upload one of these clips?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 27, 2013, 09:32:06 AM
the whole footage looks like that?
can you use mlv_dump to write only one frame?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on October 27, 2013, 09:36:07 AM
Change the black level.

exiftool -BlackLevel=2048 *.DNG

(or 1024 or just adjust it until you get rid of the color cast)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 27, 2013, 09:57:30 AM
Im getting also random pink/green clips on footage with MLV

tried playing in MLV_Viewer ?
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: heniloux on October 27, 2013, 10:18:42 AM
Thanks alex, after some reading seems like changing the black level is my forte

G3gg0- the whole clip comes out with the green color cast, gonna get the HDD from the editor this afternoon to check MLV and a couple of other stuff, ill report back as soon as i have more info.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 27, 2013, 10:37:52 AM
yeah, please also check the footage with MLV-Viewer.
if you can run mlv_dump, run it with -v to dump RAWI content
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: heniloux on October 28, 2013, 08:49:11 AM
yeah, please also check the footage with MLV-Viewer.
if you can run mlv_dump, run it with -v to dump RAWI content
I checked it on MLV viewer droped the .RAW in... still green.
raw2dng.exe=green
raw2dng.app=green
rawmagic=green
blacklevel 2048 - works
Im not to familiar with mlv_dump etc
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on October 28, 2013, 09:59:48 AM
tried playing in MLV_Viewer ?

No, i just use mlv2dng and then open in after effects. All my clips have this pink glitch error in all footage (like in the begging of raw video) I recorded this files showing in Lv  "Continuos Recording OK".  But why the question if i use MLV_Viewer?

Today will try the new mlv module update (already in the nights builds and see if the pink thing show up again. With the antepenult i already got the right name of my camera in ACR. Thanks for that =))

Thanks  :)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 28, 2013, 11:32:36 AM
But why the question if i use MLV_Viewer?

because i want you to try it and tell me if it is better there ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 28, 2013, 11:33:51 AM
for all with green cast problems:

mlv_dump -v file.mlv > log.txt

and please send me the log (http://upload.g3gg0.de/)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on October 28, 2013, 11:36:42 AM
It's a side effect of broken pink frames (black level is computed from the optical black bars, but if that bar contains corrupted image data in the moment when black level is updated, you'll get green images). Nothing to do with front-ends like MLV or raw_rec; it's in the raw backend (vsync or EDMAC).
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: g3gg0 on October 28, 2013, 11:46:50 AM
yep and mlv_dump will print the RAWI frame and thus the stored black level which comes from autocalculation ;)
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: a1ex on October 28, 2013, 11:50:49 AM
Yes, but the autocalculation is only done before recording (at least in raw_rec). So if it starts with a wrong estimation, it will use that for the entire movie.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mrnv45 on October 30, 2013, 12:59:59 AM
i cant find a working .mlv2dng  i always get an error that says error 11...logout? on shell.

i am on a mac.  i wanna compare .raw to .mlv on my 7d.  sometimes i try to record on .mlv and it ends up saving as .mov? any reason why?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on October 30, 2013, 08:36:04 AM
i cant find a working .mlv2dng  i always get an error that says error 11...logout? on shell.

i am on a mac.  i wanna compare .raw to .mlv on my 7d.  sometimes i try to record on .mlv and it ends up saving as .mov? any reason why?

> mlv2dng: please post in the mlv2dng thread (see first post)

> .mov: well, i guess you forgot to enable the mlv_rec module, or you forgot to enable raw recording in menu
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mrnv45 on October 30, 2013, 09:03:11 PM
they cant answer me they always seem to skim pass my questions or cant suggest anything for mac.
this being a post about mov_rec i figured i would ask.

and no i have it enabled.  thats why i said it records one or 2 videos then the rest are .mov files.  is it cause i might hit the 5x zoom?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on October 31, 2013, 02:30:24 AM
added dng saving code to mlv_dump.
so if you can compile it yourself, call it with --dng for saving the frames as dng
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on October 31, 2013, 06:02:19 PM
cannot compile mlv_rec anymore due to mlv_dump-error:

Code: [Select]
lzma/LzmaEnc.c: In function ‘LzmaEnc_Alloc’:
lzma/LzmaEnc.c:1908:8: warning: variable ‘btMode’ set but not used [-Wunused-but-set-variable]
ar: creating lzma/lib7z.a
mlv_dump.c:78:1: warning: padding struct size to alignment boundary [-Wpadded]
mlv_dump.c: In function ‘main’:
mlv_dump.c:635:9: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘long unsigned int’ [-Wformat]
mlv_dump.c:1211:33: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘size_t’ [-Wformat]
mlv_dump.c:1211:33: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘size_t’ [-Wformat]
mlv_dump.c:1456:33: error: ‘struct raw_info’ has no member named ‘buffer’
mlv_dump.c:1537:37: warning: format ‘%d’ expects argument of type ‘int’, but argument 2 has type ‘size_t’ [-Wformat]
make: *** [mlv_dump] Fehler 1

Tried with latest sources

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on October 31, 2013, 06:19:20 PM
I've changed it to prevent wrong code from being emitted on 64-bit systems (you were casting a 32-bit integer to a 64-bit pointer, which has undefined behavior). I think it's better for the compilation to fail, rather than having to hunt for random bugs with apparently no explanation.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on October 31, 2013, 06:22:25 PM
So, it´s the best to setup a 32bit machine with vmware, right?

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on October 31, 2013, 06:30:22 PM
Aren't 32-bit executables compatible with 64-bit systems? I remember running them on 64-bit Windows a few years ago, and on Linux they just work (I think I've installed some 32-bit libraries and that was it). No idea about Mac.
Title: Re: Magic Lantern (RAW) Video format v2.0
Post by: arrinkiiii on October 31, 2013, 06:59:08 PM
because i want you to try it and tell me if it is better there ;)

Hi,

Today i recorded again some MLV with the latest tragic lantern and i have a lot of pink/corrupt frames ...and this time i have try with mlv_viewer  :)   


[7D]

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on October 31, 2013, 07:20:33 PM
Aren't 32-bit executables compatible with 64-bit systems? I remember running them on 64-bit Windows a few years ago, and on Linux they just work (I think I've installed some 32-bit libraries and that was it). No idea about Mac.

Not sure, which  32-bit libraries for linux you mean.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on October 31, 2013, 07:25:31 PM
RAW format 2.0 specification has info about mlv_wbal_hdr_t. As far I understand we can read values of wbgain_r and wbgain_b. I think we can save white balance info to DNG (AsShotNeutral) like this:


WB RGGB Levels As Shot          : 1967 1024 1024 1796

1024/wbgain_r = 1024/1967 = 0,52059
1
1024/wbgain_b = 1024/1796 = 0,57015


static int cam_AsShotNeutral[]          = {100000,100000,100000,100000,100000,100000};
cam_AsShotNeutral[0] = 52059;
cam_AsShotNeutral[4] = 57015;

Am I mistaken ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on October 31, 2013, 10:49:22 PM
> http://www.magiclantern.fm/forum/index.php?topic=3409.msg17485#msg17485

i dont really understand how to provide either the RGB multipliers or kelvin value in an useful format for DNG.
any try causes totally weird color values, because it is unclear how canon calculates the WB stuff :(
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 01, 2013, 10:27:11 AM
I've found this paper: http://users.soe.ucsc.edu/~rcsumner/papers/RAWguide.pdf

look at page 8, 4.3 White Balancing


> http://www.magiclantern.fm/forum/index.php?topic=3409.msg17485#msg17485

i dont really understand how to provide either the RGB multipliers or kelvin value in an useful format for DNG.
any try causes totally weird color values, because it is unclear how canon calculates the WB stuff :(
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on November 01, 2013, 11:00:39 AM
It doesn't answer our questions. These are:

1) what is the relationship between WBGain multipliers and RGB multipliers? (only for Custom WB)
2) what RGB multipliers should we save in the DNG when the EXIF says 2500K or any other Kelvin value? (of course, only for Kelvin WB)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 01, 2013, 08:26:19 PM
there is one thing i do not understand.
when i go to downloads and see the changelog from e.g. 5d3.. i can see that there are some changes in mlv.rec
but when i download the latest nightly (e.g. 1.nov) the module is not there..
so why? i have downloaded the latest prebuild in this thread.. but all in all the changelog displays changes that are not made within that build (because the module is not included)

so is it useable?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rsterran on November 01, 2013, 08:46:12 PM
Please forgive me if I misunderstand the question, but I think A1ex's question 1) asks about the relationship between the various arrays of "WB RGGB" (e.g. four ints) values and the actual values to scale each color channel by to white balance?

If so, then marekk is right but has the values inverted. The ratio of the WB RGGB integers is the same as what should be used for the color channel scaling values. Thus, as in marekk's example and setting the green multiplier to be 1 (since only the ratio of them matters), we get

WB RGGB Levels As Shot          : 1967 1024 1024 1796
becomes
wbgain_r = 1967/1024 = 1.9209
wbgain_g = 1
wbgain_b = 1796/1024 = 1.7539

These directly applied to the color channels will neutralize the color of the light, but not correct for being in the camera's color space.

The second question is much more complicated, and I'm not sure if there's any standard way to do it. It might be different for every camera, requiring some calibration. If the "WB RGGB Levels" EXIF tag or the "Red/Blue Multiplier" tags are available those should be used instead of the Kelvin. Are there cameras where only Kelvin is available?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on November 01, 2013, 09:06:40 PM
If you click on the link above, posted by g3gg0, you will notice this answer is not the one I'm looking for. WBGain are Canon multipliers used for custom WB, and they are different from RGGB multipliers (they look like 1/x at first sight, but they are not).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 02, 2013, 03:03:43 PM
I finally compiled mlv_dump on mac but when I'm trying to convert do dng files it ends with Bus error 10 (recorded on 60D), on linux - segmentation fault.

small mlv file:
https://drive.google.com/file/d/0B-HdscXfsKpgWW5aeXF3OXdvSlk/edit?usp=sharing


Quote
mlv_dump -v --no-cs --dng M02-1733.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M02-1733.MLV'
   - Verbose messages
   - Verify file structure
   - Dump all block information
File M02-1733.MLV opened
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 4679971695627853972
    FPS         : 24.000000
    File        : 0 / 0
    Frames Video: 29
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 12.857000 ms
    Res:  1600x680
    raw_info:
      api_version      0x00000001
      height           1182
      width            1888
      pitch            3304
      frame_size       0x003B9730
      bits_per_pixel   14
      black_level      2048
      white_level      15000
      active_area.y1   26
      active_area.x1   152
      active_area.y2   1182
      active_area.x2   1886
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: INFO
  Offset: 0x000000e8
    Size: 16
    Time: 21.395000 ms
Block: RTCI
  Offset: 0x000000f8
    Size: 44
    Time: 24.897000 ms
     Date:        02.10.2013
     Time:        17:33:40 (GMT+0)
     Zone:        ''
     Day of week: 0
     Day of year: 0
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000124
    Size: 40
    Time: 25.451000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    21318 µs (1/46.91)
Block: LENS
  Offset: 0x0000014c
    Size: 96
    Time: 25.488000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/1.40
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 25.501000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Block: WBAL
  Offset: 0x00000200
    Size: 44
    Time: 25.547000 ms
     Mode:   0
     Kelvin:   6500
     Gain R:   489
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 25.559000 ms
     picStyle:   33
     contrast:   -4
     sharpness:  1
     saturation: -2
     colortone:  0
Block: NULL
  Offset: 0x00000260
    Size: 36
Unknown Block: NULL, skipping

Block: EXPO
  Offset: 0x00000284
    Size: 40
    Time: 4.878000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    21318 µs (1/46.91)
Block: LENS
  Offset: 0x000002ac
    Size: 96
    Time: 4.915000 ms
     Name:        '1-65535mm'
     Serial:      ''
     Focal Len:   50 mm
     Focus Dist:  0 mm
     Aperture:    f/1.40
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x0000001B
     Flags:       0x00000000
Block: WBAL
  Offset: 0x0000030c
    Size: 44
    Time: 5.187000 ms
     Mode:   0
     Kelvin:   6500
     Gain R:   489
     Gain G:   1024
     Gain B:   1024
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x00000338
    Size: 1906344
    Time: 35.575000 ms
   Frame: #0
    Crop: 216x264
     Pan: 216x264
   Space: 2312
Bus error: 10
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 04:06:24 PM
does it only happen with one file or with anything you record?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 04:09:16 PM
just saw that you updated your post after i read it.
thanks for the sample footage, converts fine on linux and windows.

can you tell me which compiler you use? is it a 64 bit system or anything weird? (old m68k mac?)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 04:11:58 PM
ah forget about it.
you didnt specify any output file with -o - this causes an error.

didnt catch this case. sorry, will fix it.
it should use a default file pattern of course.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 04:22:54 PM
fixed, can you pull and check?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 02, 2013, 05:05:04 PM
Now it works, thx

... But somenthing is wrong with camera identification:

Quote
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 25.501000 ms
     Camera Name:   'ERR:1 md:0x       0 ml:0'
     Camera Serial: ''
     Camera Model:  0x00000000
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 02, 2013, 05:07:35 PM
Quote
exiftool -a test1000005.dng
ExifTool Version Number         : 9.33
File Name                       : test1000005.dng
Directory                       : .
File Size                       : 1892 kB
File Modification Date/Time     : 2013:11:02 16:57:08+01:00
File Access Date/Time           : 2013:11:02 16:57:57+01:00
File Inode Change Date/Time     : 2013:11:02 16:57:08+01:00
File Permissions                : rw-r--r--
File Type                       : DNG
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Reduced-resolution image
Image Width                     : 128
Image Height                    : 84
Bits Per Sample                 : 8 8 8
Compression                     : Uncompressed
Photometric Interpretation      : RGB
Image Description               : (MLV Video without INFO blocks)
Make                            : Canon
Camera Model Name               : ERR:1 md:0x       0 ml:0
Strip Offsets                   : 1536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 3
Rows Per Strip                  : 84
Strip Byte Counts               : 32256
Planar Configuration            : Chunky
Software                        : Magic Lantern
Modify Date                     :
Artist                          :
Subfile Type                    : Full-resolution Image
Image Width                     : 1600
Image Height                    : 680
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 680
Strip Byte Counts               : 1904000
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 2048
White Level                     : 15000
Default Crop Origin             : 0 0
Default Crop Size               : 1600 680
Active Area                     : 0 0 680 1600
Opcode List 1                   : (Binary data 28 bytes, use -b option to extract)
Copyright                       :
Exposure Time                   : 1/46
F Number                        : 1.4
Exposure Program                : Not Defined
ISO                             : 100
Exif Version                    : 0221
Date/Time Original              :
Shutter Speed Value             : 1
Aperture Value                  : 1.0
Exposure Compensation           : 0
Max Aperture Value              : 1.0
Metering Mode                   : Unknown
Flash                           : No Flash
Focal Length                    : 50.0 mm
Sub Sec Time                    :
Sub Sec Time Original           :
Focal Length In 35mm Format     : 0 mm
TIFF-EP Standard ID             : 1 0 0 0
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : ERR:1 md:0x       0 ml:0
Color Matrix 1                  : 0.6719 -0.0994 -0.0925 -0.4408 1.2426 0.2211 -0.0887 0.2129 0.6051
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : Standard Light A
Calibration Illuminant 2        : D65
Frame Rate                      : 24
Serial Number                   :
Lens Model                      : 1-65535mm
Aperture                        : 1.4
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 1600x680
Shutter Speed                   : 1/46
Focal Length                    : 50.0 mm
Light Value                     : 6.5
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 05:15:29 PM
Now it works, thx

... But somenthing is wrong with camera identification:

this is most likely due to property handlers not made public in your camera model.
which model is it?

this must be done in ML Core.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 02, 2013, 05:17:23 PM
60D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 05:26:30 PM
solution 1:
can you add these two lines to your stubs.S in platforms/60D... ?

/** corectly named canon property reading functions **/
NSTUB( 0xff05b210, PROP_GetMulticastProperty)
NSTUB( 0xff05be34, PROPAD_GetPropertyData)

then rebuild ML core and try.

solution 2:
add those two lines to your .sym file on your CF in ML/MODULES/

ff05b210 PROP_GetMulticastProperty
ff05be34 PROPAD_GetPropertyData

and then try again without recompiling.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: marekk on November 02, 2013, 05:44:00 PM
now it works, thx

solution 1:
can you add these two lines to your stubs.S in platforms/60D... ?

/** corectly named canon property reading functions **/
NSTUB( 0xff05b210, PROP_GetMulticastProperty)
NSTUB( 0xff05be34, PROPAD_GetPropertyData)

then rebuild ML core and try.

solution 2:
add those two lines to your .sym file on your CF in ML/MODULES/

ff05b210 PROP_GetMulticastProperty
ff05be34 PROPAD_GetPropertyData

and then try again without recompiling.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 02, 2013, 11:11:31 PM
ok pushed this change. the next nightly will contain it then.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 02, 2013, 11:15:32 PM
what do you mean by the next nightly..
5dmk3 had no mlv rec module added til now.. so this means that its added now? am i right?
thx.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 02, 2013, 11:37:46 PM
what do you mean by the next nightly..
5dmk3 had no mlv rec module added til now.. so this means that its added now? am i right?
thx.

They talked about 60D, not about 5DIII  ;)

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 03, 2013, 12:47:02 AM
I know but why can i see all the changes in the changelog ??
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on November 03, 2013, 07:22:39 AM
The changes you see were not hand-picked.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: enliten on November 03, 2013, 07:05:30 PM
Hey guys,

Hope you're all having a good weekend.

Is it possible to move the crop area inbetween frames whilst capturing video?
If so then would it be possible to alternate frame locations every second frame so they can be stitched in post.

I'm just thinking that this method could allow a lower frame rate to have a greater aspect ratio.

Cheers,

Ben
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 03, 2013, 11:48:00 PM
possible and would be possible to stitch in post processing simple.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Toffifee on November 07, 2013, 08:16:16 AM
What's the news on audio implementation?  :)
Haven't seen any comments about it for a while
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 07, 2013, 12:54:27 PM
nothing new yet.
i am still waiting for someone who is going to implement it. ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ChadMuffin on November 07, 2013, 02:31:54 PM
What if it is just put in how it is or was previously? At least just for now, because shorter clips can work for dialogue or give a starting point for scratch audio for a music video. Even if they are slightly off. It may also inspire people with more knowledge to implement it properly. I sure have no idea.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: maxotics on November 07, 2013, 03:09:46 PM
What's the news on audio implementation?  :)
Haven't seen any comments about it for a while
I think it's because no one wants to make the sacrifice to make MLV a success.

It doesn't matter how good MLV is as a technical spec, and how much potential it has, it requires politics (buy-ins) from other devs and users to be a success.   I don't see it happening any time soon because the simple fact is MLV requires precious bandwidth from the camera.  It's a trade-off.  Do I think it's worth it?  Yes.  But even my few attempts to help MLV gain traction have met with failure because the devs don't want to spend time on anything else but work on some new, cool feature (even devs who should want MLV to succeed).  Users keep asking for new things and the devs can't say no, or group together and prioritize.  Because they work for free I can't say anything without attracting a lot of sycophantic support for the devs--which is misdirected. 

On a bright note, ML is a hack that should be dead (because it's a steep development learning curve), yet it lives on, and grows.  So who knows, maybe someone will implement an audio solution tomorrow.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on November 07, 2013, 03:09:54 PM
What if it is just put in how it is or was previously?

That would make me VERY, VERY happy!  ;D ;D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on November 07, 2013, 03:25:42 PM
heh, 5DIII is the only cam that can even close to pull off audio. Maybe could get away with it a little if it wrote at the same time.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: maxotics on November 07, 2013, 03:58:11 PM
I did some quick Googling and see someone has done some minor hacks to a Teac.   An Arduino project is possible, though the sample rate would probably be very low.  Maybe some other 16/32bit DIY board? I think the ultimate solution is not to record camera audio, but to link up an audio recorder and the camera through some sort of time-code or click-track.  Either

A.) Hacked Audio recorder spits out timecode that is burned into the MLV while shooting through USB, the mic input, flash connector...? (thoughts anyone?)
B.) Camera passes a click track to the audio recorder that maybe starts with an easy to synch pattern, like da, da, dum, dum, da dum, da dum. etc. 
C.) MLV software on the backend synchs up both streams using timecode or clicktrack, etc.

Any of the master devs here could do this, if time and money were no object :)

For it to happen, someone would have to project manage the thing.  Get a time commitment from g3gg0 on the MLV stuff, then one of the camera devs, Alex or 1% or both, and then another dev to work on the audio recorder part and finally, we can't forget about $$$ so each dev can have a complete setup for testing.  Also, you want testers, documentation, and some junior devs to assist the master devs.



Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on November 07, 2013, 04:04:28 PM
Well right now clapper board works for me and syncs up perfectly. I can shoot raw on 7D or 50D and then record audio on 600D or 6D, etc.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: maxotics on November 07, 2013, 04:22:10 PM
Well right now clapper board works for me and syncs up perfectly. I can shoot raw on 7D or 50D and then record audio on 600D or 6D, etc.

Me too!

Maybe my point is though the devs are maxed out, stuff like this can get done, IMHO, if people show enough interest by putting in the time to organize it and make it happen.  You don't need to be a dev to do project management, or raise money, or administrate bug reporting software, etc.  If a user asks you for something you can get done in a day or two, and it seems interesting, you do it.  But if they ask you to do something that would take weeks, why do it?  You get the same thanks for the first feature as the second.   And we can't forget, the dev is also doing stuff HE wants.

So if users want audio in this way they need to band together so there is more reward, challenge AND they need to make sure the dev isn't doing all the work. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on November 07, 2013, 04:27:09 PM
but the problem is you're asking near the impossible, most cameras can't deal with writing the video alone, much less an audio stream with it. g3gg0 did this with MLV and it has issues just from the meta data being written.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: maxotics on November 07, 2013, 04:43:43 PM
but the problem is you're asking near the impossible, most cameras can't deal with writing the video alone, much less an audio stream with it. g3gg0 did this with MLV and it has issues just from the meta data being written.

Sorry, I'm not making myself clear.  If you could read a simple voltage spike from the mic input say, or flash, etc., then you can read 0s and 1s, right.  I don't know how high a frequency you need with, but lets say it's 10 times the frame rate for easy sampling, so you'd need to read at 240bits per second.  Okay, so you're reading 0s and 1s from whatever and in the mlv file you're writing either a 0 or 1 for each frame, which gives you a 24bit signature for each second.

At the same time, the recorder is also recording those 0s and 1s (click track) to say channel 4, or the channel 2, if you can live with mono.

Now, in the recorder you hack it to plug in a cable that sends out these spikes every 240 seconds (which are sampled down to 1 bit per frame).  You also program the recorder to send out a specific "smoke signal" once it's started, so it sends batches of 12 0s and 1s for the first few seconds, but in a way that you can match up later.

So when you load the MLV file, it builds a signature from the encoded click track.  It builds a signature from the audio file and then matches them up.  Once they're matched up other software would know where the audio starts and cut it to that point and put the start point in the MLV file which the NLE (thinking big here) would ultimately use to match up the video and audio track.

I'm extemporizing here, but that's the general idea.  You mount the hacked audio recorder on a cold-shoe, and maybe, if the dev is a real genius, the camera sends a signal to start and stop the audio recorder and then we're really farting through silk ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on November 07, 2013, 05:22:46 PM
It might work, it might cause pink frames for non digicV :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 08, 2013, 11:02:51 PM
i updated the pre built binaries.

seems to work fine with 60 fps video modes now.
i had some pink frames in this mode before.

can you check it, if it the writing rate is better? getting ~95 MiB/s most of the time now with 24 fps.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: KurtAugust on November 09, 2013, 09:51:15 AM
For now I'm recording the audio beeps from the headphone output on my 5d3 to one channel of an audio recorder. Together with the time stamp of the raw file, this syncs up pretty well. Snap fingers in front of the lens when I can't use the clapper. Doable and you have a long audio recording which is nice on documentary shoots.

Still, would kill for in camera recording as well.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Toffifee on November 09, 2013, 10:58:17 PM
Yeah, running dual system is of course a good solution, but it is less ideal for us (very) guerrilla filmmakers needs to have minimal gear.
Often when I film people while travelling they freeze up because of the size of my camera and sometimes results in awkward looking shots.
And I didn't even have that much, running a 5DM3 + 24-105, small handheld rig, shotgun mic, pre-amp.
I would love to try out a bare minimum gear set, with a z-finder + videomic pro. But it kind of depends on having in-camera audio.

I can't code, but I'd be very willing to test with my 5DM3!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 10, 2013, 01:57:52 AM
this time i added a lot of stuff to mlv_play, the module to view raw files in camera.

i will update the mlv_play post (http://www.magiclantern.fm/forum/index.php?topic=9062.0).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 10, 2013, 09:25:13 PM
Then the last problem with MLV is batch tools...

using mlv_dump you can convert .mlv to .dng
anything else missing?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 10, 2013, 10:45:56 PM
Hi!

I have. 3 questions:
1. Can i convert a mlv with mlv_dump with a mac system?

2. canon 5d mk3 - When will mlv be available in the nightly..?
Cause it would be great to integrate it.. And the user can decide he will enabled the module or not...

3. can i playback mlv within a mac?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Midphase on November 10, 2013, 11:33:52 PM
Hi!

I have. 3 questions:
1. Can i convert a mlv with mlv_dump with a mac system?

2. canon 5d mk3 - When will mlv be available in the nightly..?
Cause it would be great to integrate it.. And the user can decide he will enabled the module or not...

3. can i playback mlv within a mac?

1.  http://www.magiclantern.fm/forum/index.php?topic=9130.0

2. I think it might be time for mlv_rec to be included in all the nightly builds. Without it, users will continue not to use it as much and post-production development will continue to be behind.

3. I think we need a Mac version of this:  http://www.magiclantern.fm/forum/index.php?topic=8447.0   

But really 2. is key here, until .mlv isn't in the hands of most ML users, there won't be a whole lot of reason for 3rd party devs (including Adobe) to support it. On the Mac, the best app by far is RAWMagic, and unfortunately it ain't getting updated so it might be time to make some noise!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on November 10, 2013, 11:40:11 PM
I've been including it for a long while now.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Stedda on November 10, 2013, 11:48:56 PM
MLV has been available in Nightly builds for weeks if not longer.... and was available even longer as an extra download.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 10, 2013, 11:50:02 PM
hmm i cant find it..

when i download todays nightly for canon5dmk3 there is only a mlv_play module, but no mlv_rec module.

thx.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Stedda on November 11, 2013, 12:01:28 AM
Hmmm.... maybe I added mine through this thread a while back then...

Go to the first page download the prebuilt files and you're off....
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 08:24:12 AM
I know that i can dl it here. But i just wanted to mention that it would be a great idea to add it to the nightlys
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 11, 2013, 09:24:12 AM
should be in nightly soon
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 10:47:10 AM
very nice.
mlv2dng is not working here with mac 10.9 and the latest mlv_rec module for 5dmk3
thx.

link: http://www.magiclantern.fm/forum/index.php?topic=7802.msg87994;topicseen#msg87994
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 11, 2013, 10:49:56 AM
use ''mlv_dump --dng'' for splitting mlv files into DNGs
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 11:34:47 AM
great worked. thank you.  :)

as i am not a developer or a terminal guy, a gui would be so great
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 11, 2013, 01:41:18 PM
i am planning to make the commandline parameters similar to raw2dng, so you simply can replace the .exe for GUIs that call native tools.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 11, 2013, 05:29:52 PM
Hi!
I,m such an illiterate when it comes to terminal in mac. Please help me out. I drag my file to the terminal which creates a directory? Then I type in "mlv_dump --dng"
Nothing gets converted? Check the picture

(http://s10.postimg.org/5n77kpfo5/Ska_rmavbild_2013_11_11_kl_17_52_30.jpg) (http://postimg.org/image/5n77kpfo5/)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 09:08:33 PM
hello

first i wanna say that spanned recording works great and i can record 1920x1290 continuously.
i also successfully converted those spanned files with the available mlv_dump version for the mac.

some things that would be great for future gui versions on macs:

- preview of the files - also spanned files(player)
- to be able to batch convert a bunch of files at once into different directories named after the file names (also for spanned files)
- to be able to convert dualiso shots too:)


it would also be great to implement a in camera function to delete bad takes really quick after the shot (also when recorded in spanned mode) perhaps with only one button.  that would really be handy..

thx so far - really great stuff and i hope that the progress will go on
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 09:10:18 PM
@danne

everthing great mlv_dump --dng and then take the file which should be converted from the finder and drop it in the terminal.
worked for me..;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 11, 2013, 09:57:02 PM
Hi Swinxx! Could you upload a screenshot of your terminal when you get this going? Or maybe be more specific on the mlv_dump --dng. Do you doubleclick the mlv_dump file or what? Sorry but I can,t get this rigth yet.
Thanks
//D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 11, 2013, 10:13:29 PM
no,

put the dump file into the dir where you want to convert the files

open terminal
to go to the directory where you have the mlv_dump file and the file to convert type "cd" into the terminal and drag and drop the dir from the findet into the terminal.
then type "mlv_dump --dng" and take the file which you want to convert and drag and drop it into the findet.
then press enter.. this is how it works here.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 11, 2013, 10:39:39 PM
Hi Swinxx. Sorry but I must be doing something wrong. It goes like this.
1 - I open terminal then type "cd".
2 - Then I drag my directory with the mlv file and the mlv_dump.
3 - write mlv_dump --dng
4 - drag the .mlv file I want to convert and hit enter.

Do I miss space or hitting enter somewhere?

I get this

(http://s10.postimg.org/glsmknted/image.jpg) (http://postimg.org/image/glsmknted/)

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 12:06:41 AM
1. don't use spaces
2. first cd into a directory ("cd  <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 12, 2013, 12:42:40 AM
Thanks for helping. Not quite there though.
Got something about illegal instruction, then tried typing "sudo" which created segmentation fault? Such a noob night :)
Nice work on the browser related to the viewer by the way

pics of terminal fail

(http://s9.postimg.org/j5tn29u3v/image.png) (http://postimg.org/image/j5tn29u3v/)

(http://s9.postimg.org/faq8zpay3/image.png) (http://postimg.org/image/faq8zpay3/)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on November 12, 2013, 01:09:17 AM
I am trying to switch from raw to mlv back again ao i just added mlv_rec.mo to MODULES folder.
Right now i am testing mlv file spanning at 2,5k 2,35:1 24fps  (5d3, CF: KB64 1000x, SD sandisk 45MB/s). I dont know, if this was reported but i tried many times and expected frames number doesnt work properly, for example it shows 1200 and skips after 600 (buffer dots are somewhere in half of the scale and it suddenly stops). Couple months ago just when mlv starts i remember this was very reliable.

Next thing is "beep" sound - could we have a beep when recording starts and stops? Now there is only when stops. THanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Scheme2Cream on November 12, 2013, 04:50:48 AM
1. don't use spaces
2. first cd into a directory ("cd  <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.

I am new to the forums, I've done a lot of searching on the web looking for quick and easy tutorials to convert my MLV files to RAW.
I've hit a road block with the "MLV_DUMP" via terminal solution.

Just wanted to say thank you "g3gg0" for posting very clear step by step directions on how to do this. I'm obviously doing it wrong..


Quick overview of my situation.

My Camera
5D Mark II

My Computer system
Macintosh OS X 10.9 "Mavericks"

I downloaded the Alpha_One_5D Mark II Magic lantern file from Nightly build website.

I've tried looking through the magic lantern settings via my 5D Mark II to see if there is a way to just shoot in RAW or DNG format directly, because I read in one forum someone suggested changing the file format from MLV to RAW. I was unable to find the setting to do just that..

I need help, and I've recorded my desktop typing in all the tips that everyone provided in this thread and I keep getting errors. I figured recording exactly what I'm doing in terminal should help everyone narrow down what I'm doing wrong in this process.
Please help!

Thank you!!


http://www.youtube.com/watch?v=A4ogodHSHww&feature=youtu.be (http://www.youtube.com/watch?v=A4ogodHSHww&feature=youtu.be)

Please set video quality to 1080P so you can read exactly what I'm typing in terminal...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: sebamax on November 12, 2013, 09:12:24 AM
1. don't use spaces
2. first cd into a directory ("cd  <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Thanks, work fine,
I finally did
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 09:25:12 AM
Thanks for helping. Not quite there though.
Got something about illegal instruction, then tried typing "sudo" which created segmentation fault? Such a noob night :)
Nice work on the browser related to the viewer by the way

pics of terminal fail

(http://s9.postimg.org/j5tn29u3v/image.png) (http://postimg.org/image/j5tn29u3v/)

(http://s9.postimg.org/faq8zpay3/image.png) (http://postimg.org/image/faq8zpay3/)

is the binary without any parameters running fine?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 09:27:33 AM
Thanks for all the feedback!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: frenchps49 on November 12, 2013, 10:56:14 AM
1. don't use spaces
2. first cd into a directory ("cd  <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.

Worked for me too. Maybe it is useful to say that mlv_dump should be in the same folder as the .mlv files.
Thanks for the great work. Hope for sound soon.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 12, 2013, 01:13:53 PM
is the binary without any parameters running fine?

Don,t know how to check binary but tried doubleclicking the mlv_dum file which resulted in "illegal instruction"?

(http://s23.postimg.org/8rs441j3r/Ska_rmavbild_2013_11_12_kl_13_37_06.jpg) (http://postimg.org/image/8rs441j3r/)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 01:19:59 PM
This means that the binary is not for your processor
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on November 12, 2013, 01:39:01 PM
aha, thanks for claryfing this. I,ll stick to my virtual box for now. Thanks for helping out
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 09:23:56 PM
for the card spanning users:
can you reduce CF only buffers to 1 or 2 and play with the buffering method?
maybe this will help. method 3 or 4 was the most reliable for my setup.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on November 12, 2013, 09:53:00 PM
Tried CF only buffers 1 and 2 (also combined with all buffer fill methods), 1 seems to be a little bit better (10-15 sec more) than 3 or 4 but still far from performance from the beginning. With 2,5k 2,35:1 i am getting 35-45 sec (23,976). Ah.. and it seems that "expected frames" and color buffer graph also doesnt work properly  - because it skips frames when theoretically recording should last longer.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 12, 2013, 10:25:35 PM
yeah it's possible that the estimation isnt that acurate.

how are your data rates when you enable spanning and the camera barely shows a continuos OK?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on November 12, 2013, 10:34:34 PM
108-109 MB/s in total. When it shows about 108,5 it estimates 2500-3500 frames but after 35-45sec suddenly decreases and frame skips when shows 800-1200.

EDIT: Could we have a 2240 horizontal resolution as it is in raw_rec? I have just noticed, that in raw rec we have 1920,2048, 2240, 2560, 2880 etc, mlv_rec  has  1920, 2048, 2560, 2880.
2240 is missing and this res could give decent frame number in 16:9 with good margins for post stabilization. thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Scheme2Cream on November 13, 2013, 02:45:07 AM
This means that the binary is not for your processor

Thanks for all the help by the way.
I'm getting the same message "cannot execute binary file"
would the MLV_dump work if I use terminal on say, another macintosh running an older version of OS X?

or would I have to use a windows machine?

are there any alternatives to converting MLV files? I've seen many videos online that show how people convert and edit their files, but it seems they leave out the mlv_dumping part and their files are already RAW..

Again, thanks for the insight and I appreciate all the tips I can get from you. I'm just eager to start editing the raw footage or "MLV" footage I shot..

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: nigel on November 13, 2013, 05:42:21 AM

Link :Watch on youtube (http://www.youtube.com/watch?v=PFaiU7rv1pg)

its working on a 2008 mac book pro running OSX Mavericks

cd desktop was to navigate to my desktop
cd mlv was to bring it to the mlv folder inside desktop

do note mlv_dump is in the same folder (idk if its necessary)

You should probably try it on a different PC. I had the binary error while in the VM.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 17, 2013, 09:10:58 PM
mlv_dump has this useful option:
Code: [Select]
-f frames           stop after that number of frames
Is it possible to integrate another option:
Code: [Select]
begin with frame number x and stop after frame number y
Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 17, 2013, 11:27:36 PM
good point, will add.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on November 21, 2013, 05:03:07 AM
What information do you need in order to diagnose an issue?  I have found a frame with a weird border and it's happened a couple of times now while using MLV_REC.MO.  Want to ensure any bugs I submit have all relevant information so as to not waste any time.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 21, 2013, 09:53:02 AM
what do you mean with weird border?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Cypressentgrp.com on November 22, 2013, 07:03:47 AM
I need help converting my MLV's, I was blindsided by this, I Bought a 7d and spent hours digging through threads trying to find the latest nightly build so i could go shoot raw, well just got back and a discovered I shot everything in this MLV format, which sounds great, but I can't process the files :( Im on a mac and I was hoping for a little guidance, the info is speed all over this form, I gather what MLV is and I have the mlv_dumper and that i need to use terminal but just not working for me :(
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on November 22, 2013, 01:36:54 PM
Did you see the video above?

And also this?

http://www.youtube.com/watch?v=A4ogodHSHww
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 22, 2013, 01:58:26 PM
in this video he blindly entered "<filename.mlv>".. correct would be "filename.mlv" of course

i didnt expect that someone really doesnt understand the common computer syntax for obligatory parameters :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Cypressentgrp.com on November 23, 2013, 02:53:34 AM
Did you see the video above?

And also this?

http://www.youtube.com/watch?v=A4ogodHSHww

Yes I did, I did not work, " invalid Instruction 4 " or something to that effect, It seems the binary doesn't work on my mac. I've tried both the dumper and the mlv2dng. Nether worked on this mac. It seems I have no way to process these files. Total bummer. Even tried that app, want work on 10.7.5
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on November 23, 2013, 01:11:58 PM
Get yourself a virtualmachine for windows
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 23, 2013, 02:30:48 PM
mlv_dump has this useful option:
Code: [Select]
-f frames           stop after that number of frames
Is it possible to integrate another option:
Code: [Select]
begin with frame number x and stop after frame number y
Edgar

I see your commit. ThankYou for adding  :)

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 23, 2013, 06:14:34 PM
you are welcome.
can you test it thoroughly?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 23, 2013, 06:49:05 PM
I tested all possibilities. It runs fine.

But I noticed a 0 Byte-file:
Code: [Select]
[email protected]:~/test> dir
insgesamt 158688
-rwxr-xr-x 1 edgar users 154351712 19. Okt 14:36 1.MLV
-rwxr-xr-x 1 edgar users    193028 23. Nov 18:14 mlv_dump
-rw-r--r-- 1 edgar users         0 23. Nov 18:28 test
-rw-r--r-- 1 edgar users   1985280 23. Nov 18:28 test000000.dng
-rw-r--r-- 1 edgar users   1985280 23. Nov 18:28 test000001.dng
-rw-r--r-- 1 edgar users   1985280 23. Nov 18:28 test000002.dng
-rw-r--r-- 1 edgar users   1985280 23. Nov 18:28 test000003.dng
What is this file for? This file appeares without the f-option, too.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on November 23, 2013, 07:09:07 PM
Question: do we have any way of extracting just the metadata per frame to a file so that my shot logs can leave out the ISO/shutter/aperture information, and follow focus/zoom etc?

I've tried extracting just the metadata but it comes out as that 0 byte file.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 23, 2013, 09:01:28 PM
./mlv_dump -m -o meta.mlv in.mlv
after that you can dump all metadata with
./mlv_dump -v meta.mlv

doesnt that work?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 23, 2013, 09:05:55 PM
I tested all possibilities. It runs fine.

But I noticed a 0 Byte-file:
What is this file for? This file appeares without the f-option, too.

Edgar

thanks a lot.
that sparse file should be gone now.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 23, 2013, 09:30:42 PM
OK! Has gone.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on November 23, 2013, 09:34:50 PM
Another thing: bad pixel remooving

I tested the bad-pixel remooving from cr2hdr with normal dngs, extracted from a raw-video-clip with raw2dng. To be able in doing so, I fooled the hdr_check in cr2hdr and looked at the debug-output-dngs: All the bad pixels are remooved! That´s really great!

Can this function be ported to mlv_dump in some way?

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on November 23, 2013, 11:31:52 PM
can anyone please post a link with the current mlv_dump version for mac
thx
 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on November 23, 2013, 11:48:42 PM
Seems that after moving thread to other place download link from the first page doesnt work anymore, i have tried many times and no luck. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: chmee on November 24, 2013, 10:30:04 AM
btw. raw2cdng is now able to use pre-converting to mlv->raw ;) v1.4.5

http://www.phreekz.de/wordpress/2013/06/magiclantern-raw2cdng-cinema-dng/
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 24, 2013, 10:18:20 PM
ah nice, the first step :)
reading mlv isnt that hard.

which programming language do you use?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: chmee on November 24, 2013, 10:42:28 PM
g3gg0 c# - lol, i use your mlv_dump.exe as preprocessor :)  in german: "das rad nicht neu erfinden" - but at least, yes, i'll rewrite it natively.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 24, 2013, 11:25:47 PM
raw2cdng is written in C#?
then you/we can easily add MLV support

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dia3olik on November 27, 2013, 02:25:51 PM
great!!!!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danson Delta-40 on November 28, 2013, 07:31:20 AM
Ok not gonna lie, tried to find this info elsewhere, none of the converters worked for me....

How the hell do I convert MLV to DNG/CDNG on a mac? Does the MLV dump work? And if so is there a way to batch process them in to 16 bit so I can use speedgrade cs6?

not trying to be a troll I literally cannot find any working converter and it is annoying the hell out of me.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: budafilms on November 28, 2013, 07:35:16 AM
Ok not gonna lie, tried to find this info elsewhere, none of the converters worked for me....

How the hell do I convert MLV to DNG/CDNG on a mac? Does the MLV dump work? And if so is there a way to batch process them in to 16 bit so I can use speedgrade cs6?

not trying to be a troll I literally cannot find any working converter and it is annoying the hell out of me.

Try ¨Son of the Batch¨ to open Raw and import to DNG.
Open DNG with Da Vinci Resolve lite or Adobe Photoshop ACR.

Forget the bits. There are a lot.

Good Lantern!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danson Delta-40 on November 28, 2013, 08:18:29 AM
Try ¨Son of the Batch¨ to open Raw and import to DNG.
Open DNG with Da Vinci Resolve lite or Adobe Photoshop ACR.

Forget the bits. There are a lot.

Good Lantern!

I've been converting RAW for a while now, its MLV I've been having a hard time with.
AGAIN: I want to convert MLV(EDIT: Convert on MACINTOSH), I already know how to convert RAW
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 28, 2013, 09:40:34 AM
i didnt upload binaries, as i cannot compile it for mac.
can you ask there? http://www.magiclantern.fm/forum/index.php?topic=9130.msg89058#msg89058
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on November 28, 2013, 09:05:19 PM
I downloaded the latest pre-built binary (was at http://upload.g3gg0.de/pub_files/27a867063ed87d2f079cde5f8b577a88/mlv_rec.zip), placed its autoexec.bin at the root level of the SD card and the other files in modules, and am now experiencing a problem with the magiclantern menus in my 5D3 (as well as text displayed on the liveview screen when shooting). It's now as if my camera's screen is cropping its input (looks almost like overscan), as if the buffer for the screen is larger than the screen's resolution and it's just displaying a center extraction. Even the boot screen features text cropped at the edges. And the explanatory text for ML menu options runs on top of other menu items and off the screen.

I've removed my config files and restored defaults, but that didn't help. Perhaps I was using a sufficiently-old set of ML files that I'm missing some new smaller font that would make everything fit? The text all looks huge compared to what it was with my last set of ML files (was recording raw, but didn't have MLV 2.0).

Thanks for any help or info. Here's a screenshot showing what menus look like:

(http://www.zachfine.com/forum_images/VRAM0.jpg)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Greg on November 28, 2013, 10:05:02 PM
You need fonts files.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on November 29, 2013, 03:39:49 AM
It's odd --I have fonts files, was recording raw just fine before, and everything looked fine. Can someone tell me which distribution of magic lantern has the files that are compatible with the mov 2.0 autoexec and modules? The latest? Or do I need to backtrack a few versions?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on November 29, 2013, 05:47:53 AM
Alright, got the latest 5D3 magic lantern files in there and it's all looking good now. Great to see the rapid pace of development!

I'm having  a bit of trouble, likely expected with these cutting edge builds. I'd love to have magic zoom but remove the text on the screen that tells me the names of the two files that are being written, as it's covering at least 20% of the video with those names and bitrates. All I want to see is magic zoom and a timer telling me the length of the recording. Haven't discovered the right option though I've explored every possible menu as far as I can tell.

I'm finding that if I set for spanning, I can only record about 10 seconds of 1080p23.976 raw video before it skips a frame and stops. But if I turn off spanning it records continuously just fine. Maybe it's due to the speed (or lack thereof) of my SD card? My CF is a Komputerbay 1050X 128Gb that's been working great for 1080p23.976 canon raw for the past couple of months.

One feature idea: I record second system audio using a r0de videomic plugged into a zoom H1. A pretty cheap, nice=sounding combo. It'd be neat if the camera had a way of outputting a series of beeps at the start of recording that would identify the name of the filename to which the audio recording would belong. Morse code perhaps? Maybe the name of the clip in morse over the first couple of seconds of the clip and then a beep that coincides with the exact frame on the 3 second mark to use to sync? Or a beep that coincides with a little dot in the corner of the frame, fire the flash, darken the frame, something that can be used visually to line up with the beep. Anyway, just throwing that out there in case it makes sense to anyone (or in case it's already in there and I haven't noticed).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on November 29, 2013, 11:28:53 AM
One feature idea: I record second system audio using a r0de videomic plugged into a zoom H1. A pretty cheap, nice=sounding combo. It'd be neat if the camera had a way of outputting a series of beeps at the start of recording that would identify the name of the filename to which the audio recording would belong. Morse code perhaps? Maybe the name of the clip in morse over the first couple of seconds of the clip and then a beep that coincides with the exact frame on the 3 second mark to use to sync? Or a beep that coincides with a little dot in the corner of the frame, fire the flash, darken the frame, something that can be used visually to line up with the beep. Anyway, just throwing that out there in case it makes sense to anyone (or in case it's already in there and I haven't noticed).

did you try the SMPTE module?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on November 30, 2013, 02:15:11 AM
I haven't yet tried the SMPTE module. I've got a 3.5mm dual-mono to stereo splitter cable ordered, and once it arrives I'll try running the camera's headphone jack into 1 channel of my zoom H1 and the videomic into the other. Maybe I can get Resolve or some other program to recognize the timecode track. I've read all the comments here and am not sure if anyone's gotten the module to work for syncing.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mhimhi on December 04, 2013, 11:39:16 AM
Hello! I can't get this working on 5d Markiii! How shall I do? I have copied the autoexect.bat  and other stuff on the sd card over the old one, but the camera won't start. Am I missing something?? Thank u!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 04, 2013, 11:50:31 AM
First, you make sure you're running the nighly off that SD card. Then you copy the autoexec.bin and the rest of the files from the mlv_rec.zip (http://upload.g3gg0.de/pub_files/27a867063ed87d2f079cde5f8b577a88/mlv_rec.zip). Next step, enable the new module and go with the flow :)
If your camera won't start... go back to step 1.

Hello! I can't get this working on 5d Markiii! How shall I do? I have copied the autoexect.bat  and other stuff on the sd card over the old one, but the camera won't start. Am I missing something?? Thank u!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 04, 2013, 12:05:39 PM
I'm getting some [email protected]#$d up frames when using Card spanning:
(http://www.quinox-std.com/ml/Comp1%280-00-01-13%29.jpg)

You can download the whole snippet: https://www.dropbox.com/s/rw7vewpyv4xc9u6/Comp%201.mp4 (https://www.dropbox.com/s/rw7vewpyv4xc9u6/Comp%201.mp4)
Here's the rundown:
1920x508 @ 50fps / 2.25:1 / Global Draw On / Auto Preview / Memory hack On / Extra Hacks On (KB 32GB 1000X CF & SanDisk 32GB Extreme Pro SD)
Same settings, Card spanning Off will give me almost perfect results (just a single frame gets "busy" somewhere in the first 10-20frames)...
Ideas?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 04, 2013, 04:46:10 PM
when you watch the video frame by frame in MLV Viewer, is there smth weird to see?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 05, 2013, 11:40:52 AM
@geggo, yeah... the same messed up frames :(
https://www.dropbox.com/s/36suy3x3h9w8u7e/Span.zip (https://www.dropbox.com/s/36suy3x3h9w8u7e/Span.zip)
if you wanna check for yourself(mlv's)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 11, 2013, 10:37:20 AM
Say Hi to mlv_rec in the nightlies:
Changelog for 2013-12-11 02:11:09 +0200
8881: Add mlv_rec to nightly builds
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 11, 2013, 10:54:47 AM
Say Hi to mlv_rec in the nightlies:
Changelog for 2013-12-11 02:11:09 +0200
8881: Add mlv_rec to nightly builds



;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ItsMeLenny on December 11, 2013, 01:00:01 PM
ooooh, i'll have to go get the nightly (on 550D too right?)

What is the current differences at this point in time between raw_rec and mlv_rec?
Is there some sort of comparison list for the now rather than for what its plans are?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 13, 2013, 01:06:25 AM
Any chance we'll soon have the option to move the recording information to the upper part of the screen like the legacy RAW format?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 13, 2013, 03:22:00 PM
yep, i am planning to port this.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 13, 2013, 08:32:23 PM
yep, i am planning to port this.

THANK YOU! =)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 17, 2013, 10:10:37 PM
Is there any audio support other than the sync beep for mlv_rec at this point?

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 17, 2013, 11:55:16 PM
sorry, there is no other audio support yet :(
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 18, 2013, 10:02:07 AM
What about the SMTPE module?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 18, 2013, 12:39:33 PM
What about the SMTPE module?

what do you want to know about it?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 18, 2013, 12:41:58 PM
Isn't it a good solution for audio sync (if your equipment can take advantage of it, of course)?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 18, 2013, 01:27:31 PM
Isn't it a good solution for audio sync (if your equipment can take advantage of it, of course)?

uhm, yes it is.
the module is available with nearly all functions people wanted, but noone really checked if it is working for their workflow.
it was even included in most mlv_rec builds i uploaded here.

see this thread: http://www.magiclantern.fm/forum/index.php?topic=7764.0

do you have real equipment to play with it?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 18, 2013, 01:33:21 PM
I'm waiting for a Tascam DR-60D to show up real soon & i'll be on it!!!

Offtopic: Is there a way to "follow" a thread, without having to post there? I want to watch this SMTPE thread in case of new development/new posts...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 01:02:45 AM
update:
 - added mlv_snd which adds audio recording to mlv_rec

HowTo:
 - unpack modules into ML/MODULES directory
 - enable mlv_snd module
 - go to "Audio" menu and enable "MLV Sound"
 - record a video using mlv_rec
 - test the video on your computer (mlv_dump or MLV Viewer latest revision)


when you call './mlv_dump -o <outname>', it will create a .wav
(thanks to Jarno Paananen (https://bitbucket.org/hudson/magic-lantern/pull-request/309/add-wav-header-to-audio-output-file-and/diff))



can you test it whether it works for you? tried on
5D3: works
7D: no audio recorded
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 22, 2013, 02:03:20 AM
update:
 - added mlv_snd which adds audio recording to mlv_rec

you can play it using
a) my MLV Viewer or
b) when you use 'mlv_dump -o <outname>' which will create a .wav (thanks to Jarno Paananen, https://bitbucket.org/hudson/magic-lantern/pull-request/309/add-wav-header-to-audio-output-file-and/diff)



can you test it whether it works for you? tried on
5D3: works
7D: no audio recorded

g3gg0,

Is mlv_snd part of the latest nightly build? I would love to test it out for my 5D3. Thanks so much for all the hard work and updates!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on December 22, 2013, 02:17:32 AM
I just tried on my 6D and i can see the meters but apparently i can't hear any audio recorded.
Am i doing it wrong?
I am viewing the MLV file in MLV viewer but can't hear anything.
Suggestions?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 02:24:07 AM
edit:
re-uploaded the package, the module was missing in the .zip.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on December 22, 2013, 02:30:20 AM
Oh, that's what i am missing.
I thought it was in the zip file but indeed it isn't.
Am i missing something again?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 02:54:20 AM
mea culpa, i didnt drop it onto the .zip, but next to it.
re-uploaded, please test again ;)
updated the post before with installation instruction.

-> enable mlv_snd, go to audio menu and enable "MLV Sound". then you can record.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on December 22, 2013, 03:18:21 AM
Still no luck.
I get FFT graphs but apparently no audio is recorded. I am playing MLV files on the viewer with everything on low so it's fast.
I also got a crash, i am trying to reproduce it.
Quote
ASSERT: FALSE
at ./ASIF/ASIF.c:444, task beep_task
lv:1 mode:3


Magic Lantern version : Tragic.2013Nov09.6D113
Mercurial changeset   : a315c155c985 (unified) tip
Built on 2013-11-09 17:19:15 UTC by [email protected]
Free Memory  : 302K + 1694K
Also i am getting "audio failed to stop state 4" in the middle of a recording sometimes.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 22, 2013, 05:13:36 AM
i hear audio in MLviewer but i don't understand how to output the DNG with audio.. any help please..  test on 5D3
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on December 22, 2013, 05:43:51 AM
What camera?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 22, 2013, 05:52:49 AM
5 d 3
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 08:36:08 AM
I also got a crash, i am trying to reproduce it.Also i am getting "audio failed to stop state 4" in the middle of a recording sometimes.
please use a recent version of magic lantern.
you are using an outdated custom build.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 08:37:41 AM
i hear audio in MLviewer but i don't understand how to output the DNG with audio.. any help please..  test on 5D3
did you try mlv_dump already?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: naturalsound on December 22, 2013, 10:30:11 AM
I can not believe you added AUDIO SUPPORT!  :D 8)
It really works for me on 5D3 (Komputerbay 128GB, Fill:3 Buffer:4 CF & SD: Mode S1)
(Edit: CF-only: 8 works more reliable)
This is a niiice x-mas present!


PS: Test-Mode freezes sometimes using the nightly build (Dec 22 2013). This happens after about 3-5 aquisitions stopped by a skipframe.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 22, 2013, 11:15:02 AM
there is no mlv_snd in my 550D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 22, 2013, 01:31:01 PM
did you try mlv_dump already?

it works but looking for easier way.. =p
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: NedB on December 22, 2013, 01:51:34 PM
@g3gg0, andy kh: sound recording works on 550D!

Steps I took:
- Downloaded latest (22Dec) nightly build for the 550D, also your mlv_rec.zip file from the first post of this thread
- Installed nightly build on my SD card by simply copying it over the existing files
- Extracted mlv_snd.mo from the .zip file and copied it into the MODULES folder on the card
- Enabled mlv_play, mlv_rec, and mlv_send, rebooted to load modules
- Enabled "MLV Sound" in Audio menu
- Recorded a short (18 sec) clip (using mlv_rec, obviously, and not raw_rec), which was called M22-1151.MLV
- Clip played back in camera, created a file M22-1151.IDX
- Tried to get mlv_dump (also from your mlv_rec.zip file) to work, but it kept giving error message about "Missing header"
- Renamed index file (created in camera) to M22-1151.IDX.bak and now mlv_dump works with following command (assumes a copy of mlv_dump.exe has been copied to C:\) :
        C:\>mlv_dump --dng "E:\raw tests\20131222\M22-1151.MLV"    ## replace the path with YOUR path to the .MLV file
- After this the typical processing with Photoshop CC (for ACR, synchronizing all .dng's to the selected settings), then After Effects CC -> Adobe Media Encoder to render as H.264 .mp4 file

I did notice that the sound file was a bit longer than the .dng sequence (I was shooting at 25fps): 19.00 seconds vs. 18 seconds, 20 frames for the video. I simply assumed that the starting points were in sync and the sound file cut off a bit later than the video, and that seems to have been a correct assumption, although a longer (few minutes) test is in order to check this thoroughly.

Very nice that we have sound with raw now, excellent work! Cheers...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 02:41:44 PM
I did notice that the sound file was a bit longer than the .dng sequence (I was shooting at 25fps): 19.00 seconds vs. 18 seconds, 20 frames for the video. I simply assumed that the starting points were in sync and the sound file cut off a bit later than the video, and that seems to have been a correct assumption, although a longer (few minutes) test is in order to check this thoroughly.

thanks for the feedback. you are right. recording starts with the first frame (to be verified)
and records about 0.5 sec longer because the buffers are already queued for filling when you stop recording.
as soon the buffers are done filling, audio recording stops
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: auxmar on December 22, 2013, 03:11:29 PM
Hello

When i try to extract the dng's on mac with mlv_dump, i got this message, and i can't see the audio files :


Code: [Select]
Vertical stripes correction:
  1.000  1.000  0.997  0.994  0.993  0.990  0.999  1.018
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Reached end of chunk 0/1 after 766 blocks
[i] Processed 265 video frames
[i] Done
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 04:06:19 PM
thats obvious because you dont use the latest version...
i added sound support last night, so needless to say that you need to get an updated version....
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: auxmar on December 22, 2013, 04:14:37 PM
Thank you for you response, but i only found on your file the linux and windows version ... any chance to have the osx mlv_dump ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on December 22, 2013, 04:25:04 PM
audio works on 600D too (if I have enabled audio-recording in canon-menu).
mlv_dump creates a wav-file which I can playback, using mplayer for example
mlv-viewer crashes:
Code: [Select]
[email protected]:~/Software/MagicLantern> mono MLVViewSharp.exe
KDE colorscheme read failure, using built-in colorscheme

Unhandled Exception:
System.IO.DirectoryNotFoundException: Could not find a part of the path "/HOME/EDGAR/AUDIOTEST/M22-1707.IDX".
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean isAsync, Boolean anonymous) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess)
  at mlv_view_sharp.MLVReader.SaveIndex () [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVReader.UpdateIndex () [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVReader..ctor (System.String fileName, mlv_view_sharp.MLVBlockHandler handler) [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVViewerForm.PlayFile () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.DirectoryNotFoundException: Could not find a part of the path "/HOME/EDGAR/AUDIOTEST/M22-1707.IDX".
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean isAsync, Boolean anonymous) [0x00000] in <filename unknown>:0
  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access) [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess)
  at mlv_view_sharp.MLVReader.SaveIndex () [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVReader.UpdateIndex () [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVReader..ctor (System.String fileName, mlv_view_sharp.MLVBlockHandler handler) [0x00000] in <filename unknown>:0
  at mlv_view_sharp.MLVViewerForm.PlayFile () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
[email protected]:~/Software/MagicLantern>
I don´t find a .idx-file anywhere.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 04:28:36 PM
thanks for testing.

about mono: is this directory writeable for your current user?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on December 22, 2013, 04:49:51 PM
which dir? mono is /usr/bin/mono on opensuse an not writeable.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 05:09:54 PM
/HOME/EDGAR/AUDIOTEST/M22-1707.IDX
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: escho on December 22, 2013, 05:16:01 PM
/HOME/EDGAR/AUDIOTEST/M22-1707.IDX
/HOME/EDGAR/AUDIOTEST is writeable.
 /HOME/EDGAR/AUDIOTEST/M22-1707.IDX does not exist.

Edgar
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 22, 2013, 05:37:11 PM
g3gg0, this is simply amazing  :D

After some initial test (5d3, KB64 1000x, 1920x818 25fps)

-some dropped frames after unpacking with the newest mlv_dump (first time - over 40 out of ~400, second time - 1 out of 600)
This was with card spanning ON, i didnt notice. Will check with spanning OFF.

EDIT: dropped frames were because of spanning, with OFF is OK.

-sound out of mlv_dump seems to be  slower than picture, it is pitched down a lot. Tried with 25fps, no fps override.

EDIT: after some further testing it is still an issue.

I hope that this long awaited feature will now trigger devs of RAWanizer, raw magic, batchelor and others to create some form of mlv_dump with GUI. Now it is time to switch to the new format  :D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kfprod on December 22, 2013, 06:09:27 PM
Thank you this is great!

Unfortunately I can't get the audio out yet as there is no new mlv_dump for mac. I also have the problem that I have to rename any IDX file (to IDX.bak for example) otherwise I get "Missing file header".
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 07:42:27 PM
-sound out of mlv_dump seems to be  slower than picture, it is pitched down a lot. Tried with 25fps, no fps override.

sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 22, 2013, 08:06:07 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)

I mean that after unpacking with mlv_dumb and importing dngs to NLE (after effects cs6 in my case) video stream has different lenght than sound stream. I checked import settings and it is 25 like in camera so this is not an issue. When i am playing back sound is "slower" - like 70-80 percent of video stream (which gives "lower" sound). But it can be corrected of course, have to figure out by how much. Maybe i am doing something wrong so i will make more testing.
But it is great, really, i am very happy, maybe i could keep my zoom H1 and not spend more money on H6 for now. Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: aaphotog on December 22, 2013, 08:06:12 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
44.1khz is audio for cds/mp3s... music
video has 48khz audio.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 08:55:57 PM
44.1khz is audio for cds/mp3s... music
video has 48khz audio.

nevertheless the .wav has set 44100 Hz and the audio data comes with 44100 samples per second.
i just double checked that. just play the .wav in any player and it plays correctly.

if you now import that and say "wow, hey ignore the 44100 Hz from WAV header and use 48000 Hz instead!" then its not my fault.

until there is a audio format setup screen, it is 44100 Hz.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: swinxx on December 22, 2013, 09:04:24 PM
hello.
soundrecording with canon 5d mk3.. whow.. great. thx so much.. will test it as soon as possilbe.

in the meantime, can someone please compile an updated mac version of the new mlv_dump..
thx. so much.

greets. sw
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 22, 2013, 10:25:36 PM
OSX Users:

i recently installed a OSX machine inside a virtual box.
after some struggle regarding installation of the necessary stuff i was able to compile mlv_dump in the VM.

can you try this (http://upload.g3gg0.de/pub_files/18ee8e20ac50501adcf36e07ef599279/mlv_dump.osx) mlv_dump?

update:
if you use OSX 10.6 or 10.7 does the link above work?
if not, can you try this (http://upload.g3gg0.de/pub_files/8705642032cded4e22a3051acb0ca870/mlv_dump.osx) mlv_dump?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on December 22, 2013, 10:41:10 PM
please use a recent version of magic lantern.
you are using an outdated custom build.
There is none in the nightly builds for the 6D anymore, as you might know.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 22, 2013, 10:51:13 PM
@NedB thank u so much
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 22, 2013, 11:10:28 PM
Audio, this is good. Thanks for all your work g3ggo.
Trying the mac binary but can,t get it to act the way the former binary acts. It says its a document file rather than a unixbased file like the former binary? Do I extract as usual? Only tried with an older .MLV file, not with recorded sound though.

By the way. If someone uploads a .MLV with audio @Baldand will start working on getting it extracted through his pythonbased rawviewer.
http://www.magiclantern.fm/forum/index.php?topic=9560.msg92659;topicseen#msg92659
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 01:07:53 AM
g3ggo! Amazing! The audio is great and is totally in sync for a full 30 second test I just completed! This is so exciting to see the audio being developed for mlv_rec. Thank you again!

Here is some feedback and questions for you g3ggo or anyone else. (I am new to mlv_rec so I apologize ahead of time if the questions are already common knowledge)
1. Like a few people mentioned above… I also get a "missing header" error message when using mlv_dump on pc unless I rename the extension of the .IDX file. It also works if I just move the .IDX file to another directory.
2. The OSX version of mlv_dump file that you just posted didn't work for me. I think Danne is right in the post above where he mentions the file is not a unixbased file and maybe that is causing some issue.
3. When recording I loose the ML overlays and I also loose the 16:9 black bar crop on the screen. This might be a known issue, but I wasn't sure if I needed to set a setting to fix it. Also, when I stop the recording the ML overlays don't come back until I hit the preview button twice.

Thanks again for all the hard work! I really appreciate it. The addition of audio is going to make my life so much easier!


Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 23, 2013, 01:12:46 AM
can you try to exec
chmod +x ./mlv_dump.osx
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 23, 2013, 01:22:23 AM
Thanks g3gg0, worked! Will try to extract audio now...
Worked, the regular command extracts both audio and dng,s with your mac binary. How wonderful!

Mac-instruction

1 - download g3gg0 binary from "Reply #891"
2 - Run this command in terminal in the directory of the binary(mlv_dump.osx)to "executes" chmod +x ./mlv_dump.osx


Then run the usual commands

1. don't use spaces
2. first cd into a directory ("cd  <directory>" and note the SPACE after cd)
3. then execute  ./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.

Thanks g3gg0!

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 01:27:21 AM
can you try to exec
chmod +x ./mlv_dump.osx
g3gg0,

Yes, that fixed it! I have it working on my OSX 10.8.5. I still had to rename the IDX extension so it would ignore it, but it has the same behavior on the pc.

Thanks for the tip!

Here was the command I used to create the dng's and audio file: ./mlv_dump.osx --dng M22-1413.MLV
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: crazypig on December 23, 2013, 03:07:32 AM
Dear g3gg0,

Thanks for your great surprise before this christmas.

I really want to test the mlv_snd.

But i can't find it in the updated nighty bulid.

Regards,

Sean
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: auxmar on December 23, 2013, 03:42:42 AM
Thanks,  The mlv_dump.osx work for me too, it extract the wav file and the audio is sync, but the converted dng are color shifted to ugly green.

I use 5d MKIII
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 05:50:08 AM
Dear g3gg0,

Thanks for your great surprise before this christmas.

I really want to test the mlv_snd.

But i can't find it in the updated nighty bulid.

Regards,

Sean

You can get mlv_snd from the first post in this thread. Just add it to your latest nightly build. Post #874 a few pages back has a good overview.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: crazypig on December 23, 2013, 05:52:29 AM
You can get mlv_snd from the first post in this thread. Just add it to your latest nightly build. Post #874 a few pages back has a good overview.
thank you so much
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 06:18:41 AM
Thanks,  The mlv_dump.osx work for me too, it extract the wav file and the audio is sync, but the converted dng are color shifted to ugly green.

I use 5d MKIII
I get the green too. Once I bring it into After Effects I can adjust the Temperature and Tint to make it look right. The sliders for Temperature and Tint default to their minimum value which makes the green.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 23, 2013, 12:53:21 PM
Tested the new audio function in MLV on a 6d.

Got a real audio wav file, normal size, normal specs.
But when I play the audio file, there's no audio at all ? Not even noise, it's like it's recording without any real audio input...

Tested several options in the ML audio menu, like sound record: separate wav, sound record: normal, Input source: internal mic, Input source auto...
And all these options where tested with MLV sound: ON.

Anybody else tested it on 6d ?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dadinio13 on December 23, 2013, 02:02:03 PM
someone can upload the osx compiled version of mlv_dump please ?
i can't compile it from where i'am
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mk11174 on December 23, 2013, 02:33:42 PM
Wow! sound record working on 550D so far, pretty cool!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ouuzi on December 23, 2013, 02:38:12 PM
Yep Audio working!!!!You are the best g3gg0.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mageye on December 23, 2013, 03:12:37 PM
OK I tested the MLV sound recording from this topic on 5DMKII. It appears to work and then I used the linked MLV converter (the latest one) for osx.

I managed to get it to spit out a some RAW files for video and some *.wav files for the audio. The files appear to be around the expected size.

I can't play these files though. I inspected the file with a hex editor and find that they are RIFF files. I don't seem to have any application on the Mac that can deal with RIFF. :(

Does anyone know of any Mac application that can either convert or directly deal with RIFF files?

Anyone else have any joy playing/editing the sound files on the Mac?

I'm on Mac OS X Mavericks 10.9.1
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 23, 2013, 03:32:53 PM
@g3ggo thank q so much g3gg0 n the magic lantern team for enabling audio recording
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: [email protected] on December 23, 2013, 04:06:23 PM
Thanks to g3gg0, sound works perfect with 5D3 on OS X Maverick.

For OS X users:
.MLV file and mlv_dump.osx have to be in the same directory, then convert with Terminal using this command:
./mlv_dump.osx --dng NameOfTheFile.MLV

Happy holidays shooting Raw with sound ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 23, 2013, 04:09:44 PM
Seems to work now on 6D!

I had to turn OFF the FPS override function (It said so in a message at the bottom of the display, FPS override turns off most sound function settings... ::)

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 07:08:17 PM
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know… Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mk11174 on December 23, 2013, 09:54:33 PM
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know… Thanks.
Turn the Global Draw option On inside MLV it will keep it on instead of turning it off for performance if the performance is not needed, it seems to be Off by Default in MLV
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on December 23, 2013, 10:07:57 PM
This is fantastic news!  Thank you.

I am shooting on a 5D3 and noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.

Do I need to copy/replace this autoexec.bin to the root directory of my SD card?

Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 23, 2013, 10:10:04 PM
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know… Thanks.


switch on "Disable G Draw" in Raw Video(MLV)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 23, 2013, 10:16:45 PM
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is

https://vimeo.com/82579655
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 23, 2013, 10:18:31 PM
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is

https://vimeo.com/82579655

>We couldn't find that page.
>Please check to make sure you've typed the URL correctly.
>You may also want to search for what you are looking for.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 23, 2013, 10:22:57 PM
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is

https://vimeo.com/82579655
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 23, 2013, 10:47:34 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)

I still have problem with sync - tried 24 and 25 fps. Sound sounds slower, i mean sloooooower.
I used goldwave and changed wav playback rate from 44,1 to 48kzh and it helped, still not perfect but much better. I have recorded 1 minute clip, in 56-58 sec the lag was about 0,5 sec. 5d3, kb64 1000x. To use wav with resolve 10 it is required to change 44,1 to 48khz anyway.

EDIT: i can fix this problem in premiere cs6 with using Rate stretch tool on the sound clip and stretch it down to match the video layer.
Is anybody have the same problem?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 23, 2013, 11:08:25 PM
I still have problem with sync - tried 24 and 25 fps. Sound sounds slower, i mean sloooooower.

can you upload such .wav?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: seku on December 23, 2013, 11:35:53 PM
Hi all,

been trying for a few hours now to get this to work, but I'm a bit clueless ... and feeling bad posting for help with my low postcount :/

- On my 5Dmk3, I've tried with installing yesterday's as well as today's nightly, and activated filemanager, mlv-rec, and mlv-play, then enabled mlv-rec. records just fine.
- downloaded mlv-snd from the first post in this thread, activated it as well, and enabled mlv-snd. i get the big reddish audiobars fine when recording.
- on OSX Mavericks, i installed mlv_dump.osx, made it executable and ran the --dng command: it extracts the DNG's just perfectly, but i just do not manage to get an assorted wav file.

Any pointers to where i missed an obvious step?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 11:42:29 PM
Hi all,

been trying for a few hours now to get this to work, but I'm a bit clueless ... and feeling bad posting for help with my low postcount :/

- On my 5Dmk3, I've tried with installing yesterday's as well as today's nightly, and activated filemanager, mlv-rec, and mlv-play, then enabled mlv-rec. records just fine.
- downloaded mlv-snd from the first post in this thread, activated it as well, and enabled mlv-snd. i get the big reddish audiobars fine when recording.
- on OSX Mavericks, i installed mlv_dump.osx, made it executable and ran the --dng command: it extracts the DNG's just perfectly, but i just do not manage to get an assorted wav file.

Any pointers to where i missed an obvious step?

I am obviously not an expert with my low post count either, but you might need the mlv_record module from the first post of this thread rather than the nightly version. I use the mlv_rec from this thread and I have it working.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 23, 2013, 11:47:02 PM
switch on "Disable G Draw" in Raw Video(MLV)

Thanks Andy kh! That worked.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: seku on December 24, 2013, 12:01:58 AM
Thanks rob_6 and andy kh :)

t'was the Disable G Draw, works perfectly now, thanks a lot !
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 24, 2013, 12:05:29 AM
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting.  The option in the MLV module is OFF too.

Any idea?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 12:07:16 AM
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting.  The option in the MLV module is OFF too.

Any idea?

it has to be ON
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 24, 2013, 12:14:00 AM
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting.  The option in the MLV module is OFF too.

Any idea?

it has to be ON

Sort of counterintuitive, but it worked!  Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 24, 2013, 12:24:35 AM
I'd just like to remind everyone that PLURALEYES WILL WORK AGAIN!!! * head explodes *
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 24, 2013, 01:08:45 AM
Been doing many tests and now I only get the MLV files recorded to the card. The sound is still working great, but the IDX files are no longer being created. Anyone know what could be the reason? I tried formatting my cards an reinstalling ML files and that didn't change it. I don't have a use for the IDX files at the moment so I am not complaining, just curious. :)

I am having so much fun with this sound recording development. Its like the best Xmas gift ever! Thanks again g3gg0 and the ML team!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 24, 2013, 01:32:33 AM
you are welcome ;)

i updated mlv_rec and mlv_snd.
those of you who had weird audio problems (e.g. not playable wav's) please try again.
that should be fixed now.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 24, 2013, 01:42:51 AM
update:

added OS X binary and updated to include this (https://bitbucket.org/hudson/magic-lantern/pull-request/312/add-time-metadata-to-dngs-exported-from/diff) fix for DNG datetime
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 24, 2013, 01:56:25 AM
can you upload such .wav?

Sure. Here it is:

wav straight from mlv_dump
https://www.dropbox.com/s/mst85gtpzutwd7o/m24-0223_frame_straight%20from%20mlv_dump.wav

the same wav from goldwave - playback changed from 44,1 to 48khz  - seems quite correct.
https://www.dropbox.com/s/wuihi0t2saondxp/m24-0223_frame_48khz%20from%20goldwave.wav
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: DJBFilmz on December 24, 2013, 03:40:27 AM
Hello g3gg0,

I downloading the recent nightly build of MLV Raw, but before I could get a chance to test the audio I ran into an error.

After recording a video in MLV it will always state "stopping..." and unless i shut off the camera and turn it back on, it will always state "stopping..."
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: daisermac on December 24, 2013, 10:08:16 AM
Thanks g3gg0, this is invaluable. And thank you also for advancing the whole file format stuff in the first place.

Haven't tried the new build, but already the first build worked great on 5Dm3. Sadly I am still plagued by random crashes and not shutting down until pulling the battery when using an external monitor. But this also happens with the traditional raw_rec.mo
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 24, 2013, 12:07:14 PM
Hi G3gg0,

If it won't take too much of your time, is it possible to put in some more horizontal video resolution options in the MLV module?
In some raw.rec builds there where options for 1792 pixels wide and 1664 pixels wide resolution.
The 1792 resolution is probably only used for 6d owners, but I happen to have a 6d and love the 1792 pixel wide clips. 

Further, THANKS for the audio recording in combination with raw video!
Works on my 6d (using the tragic lantern build of 30 November), audio clips are a little bit longer than video, but it seems in sync, it only records some more audio after video recording has stopped.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 12:19:20 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio. i wil be uploading a tutorial in an hour.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ouuzi on December 24, 2013, 12:38:33 PM
We are waiting!!Thanks a lot
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 01:31:32 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio. i wil be uploading a tutorial in an hour.

I'd waiting on pins and needles...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on December 24, 2013, 01:54:02 PM
MLV FTW!!! @g3gg0 4 President :) Amazing what u did on the audio side too man!!! Have a fine holidays session too!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ouuzi on December 24, 2013, 02:56:41 PM
How to monitor level with this?Can anybody help?I will post a workflow tutorial for Final Cut Pro.
It goes like this >mlv dump>convert dng's to Visionlog>compressor 4.1 to export Prores 4444 with audio>Final cut pro with LUT utility for grading.

Very good quality to my eyes
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MA Visuals on December 24, 2013, 03:05:56 PM
update:
 - added mlv_snd which adds audio recording to mlv_rec

Great work g3gg0!  I just tested this and it worked very well... I had perfect sync when post-processed.  I only recorded 25 seconds so I'll try to record something much longer to test that sync is maintained for an extended period.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 04:34:43 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio



https://vimeo.com/82623320
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 04:39:35 PM
OSX Users:

i recently installed a OSX machine inside a virtual box.
after some struggle regarding installation of the necessary stuff i was able to compile mlv_dump in the VM.

can you try this (http://upload.g3gg0.de/pub_files/18ee8e20ac50501adcf36e07ef599279/mlv_dump.osx) mlv_dump?

update:
if you use OSX 10.6 or 10.7 does the link above work?
if not, can you try this (http://upload.g3gg0.de/pub_files/8705642032cded4e22a3051acb0ca870/mlv_dump.osx) mlv_dump?

I've tried this in terminal on both 10.7 and 10.9 on both I am getting a response of "command not found".  Am I putting in the correct command?  I'm not much of a terminal user...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 04:52:50 PM
I've tried this in terminal on both 10.7 and 10.9 on both I am getting a response of "command not found".  Am I putting in the correct command?  I'm not much of a terminal user...

BINGO!  I read a little more and realized my mistake. chmod +x ./mlv_dump.osx needs to be executed first.  Second one works well in 10.7.  First one says 'illegal instruction 4'

Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 04:58:35 PM
When I attempt to open the dng in premiere pro cc, it says 'The file appears to have no media data'….
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 05:22:13 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio



https://vimeo.com/82623320
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 05:24:28 PM
When I attempt to open the dng in premiere pro cc, it says 'The file appears to have no media data

you can open cdng but not dng in premiere pro cc. watch my video how i open cdng in premire pro cc
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 06:03:21 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio



https://vimeo.com/82623320

this actually makes the process seem far more complex than just converting dngs in Resolve...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 24, 2013, 07:15:05 PM
this actually makes the process seem far more complex than just converting dngs in Resolve...

AnotherDave,

Do you know of a way to link up the audio in Resolve automatically based on filename? I can align the audio manually in the timeline, but I wonder if there is a way to align the audio for the clips automatically by name.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 08:24:07 PM
If the developers can find a way to encode the metadata the right way, Resolve should be able to recognize it right off the bat.  Unfortunately, when I open my clips they don't seem to have any of the metadata stored…  I'm processing through mlv.dump.osx for 10.7…
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 24, 2013, 09:08:43 PM
Has anyone tested a long clip of a more thane a couple minutes? I have only tried short clips so far and they seem in sync. Are longer clips still in sync?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 24, 2013, 10:11:26 PM
-m question…

What file extension should we be outputting the metadata to?  What can read this metadata?  Is the audio track information being encoded on the metadata track?  Is this information used when processing to .dng, or does additional software need to be written to re-encorporate it?

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 24, 2013, 10:26:14 PM
Excellent work! I just started testing the new mlv-sound.mo  and the MLV viewer.

So far the MLVBrowser exports (wav + dng) the dng and the wav.  The sound seem good but the dng have green cast.

If I export to RAW + wav and then use raw2dng then the dngs are OK!

Finally my movies will not be muted.  Thank you very much.

I will be testing this further as I have time....

Merry Chiristmas and thank you again.


Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 24, 2013, 11:16:10 PM
Has anyone tested a long clip of a more thane a couple minutes? I have only tried short clips so far and they seem in sync. Are longer clips still in sync?


yeah longer clips in sync
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 25, 2013, 12:55:44 AM
g3gg0, is there any possibility to implement 44,1 to 48khz conversion in MLV browser?
Resolve 10 dont accept 44,1, sure we can do this one by one by hand but the workflow would be much faster with ready 48khz wav out of mlv_dump. What do you think?
Cheers
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 25, 2013, 01:19:06 AM

yeah longer clips in sync

Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 25, 2013, 05:42:59 AM
Resolve 10 dont accept 44,1
Ah ha!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 25, 2013, 07:23:12 AM
Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks

yeah i hav done so many test. long clips short clips and everything works fine in premiere pro cc. setting in camera is same in all camera.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 25, 2013, 08:22:15 AM
yeah i hav done so many test. long clips short clips and everything works fine in premiere pro cc. setting in camera is same in all camera.

Got it. If you just click on wav and open it in say windows media player - does it play normal ?  Could you please check one of my previous post in this thread with dropbox links and see those two short clips  and the diffetence between them? Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 25, 2013, 08:48:04 AM
Got it. If you just click on wav and open it in say windows media player - does it play normal ?  Could you please check one of my previous post in this thread with dropbox links and see those two short clips  and the diffetence between them? Thanks

i got u. there used to be problems sometimes. that problem is no more with yesterday's nightly build. g3ggo has already fixed it.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 11:09:07 AM
Sampling rate selection will be added soon.
dont know if it is possible to select 48khz in camera.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 25, 2013, 01:34:41 PM
Sampling rate selection will be added soon.
dont know if it is possible to select 48khz in camera.

Thats great, i tested many times, also with yesterday nightly build and of course newest mlv rec pre built. Still having the same problem. Converting playback rate (not just resampling) from 44,1 to 48khz in goldwave makes wav sounds correct (it shortens it a little bit). Strange thing is that nobody else is having such a problem...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mageye on December 25, 2013, 02:20:18 PM
48K will be great if it's possible. I am impressed with the audio recording now. It now makes 'run and gun' type filming a much more realistic possibility. The audio thing is a game changer. Again. Thanks for all the work. It's much appreciated. :)

Can we have 24-bit too? (only joking! ;) :D)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 25, 2013, 07:52:50 PM
anyone recording more then 1min on mlv with audio ?

i keep getting frame skip around the sametime
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 25, 2013, 08:02:35 PM
Did you try setting still image quality to a low jpg setting and rawphoto off? WIth this I get continuous 25fps 1920x1080 with audio.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 08:18:59 PM
i keep getting frame skip around the sametime

video or audio frame skip?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 09:21:42 PM
Here's a quick bash script for anyone wishing to run the OS X edition... move MLV_DUMP.OSX to /usr/bin and rename to MLV_DUMP then run CHMOD +X /usr/bin/MLV_DUMP

Save the script as dumper.sh in the same folder as the MLV files ... I usually copy everything to my desktop.  Usage of the script: ./dumper.sh <input.mlv> <folder name>

mkdir $2
mv $1 ./$2
cd ./$2
/usr/bin/mlv_dump -o $1 --dng $1
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 09:27:26 PM
update:
 - mlv_snd now allows to select desired sampling rate (48000, 44100, 22050, 11025, 8000)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 09:29:11 PM
Here's a quick bash script for anyone wishing to run the OS X edition... move MLV_DUMP.OSX to /usr/bin and rename to MLV_DUMP then run CHMOD +X /usr/binMLV_DUMP.

Save the script as dumper.sh in the same folder as the MLV files ... I usually copy everything to my desktop.  Usage of the script: ./dumper.sh <input.mlv> <folder name>

mkdir $2
mv $1 ./$2
cd ./$2
/usr/bin/mlv_dump -o $1 --dng $1

not quite correct.

this:
> move 'mlv_dump.osx' to /usr/bin and rename to 'mlv_dump' then run 'chmod +x /usr/bin/mlv_dump'
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 09:32:23 PM
thanks dude, forgot the /.  i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"

will report back.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 25, 2013, 09:38:49 PM
thanks dude, forgot the /.  i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"

will report back.

i can do it in windows
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 09:42:35 PM
thanks dude, forgot the /.  i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"

will report back.


for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    /usr/bin/mlv_dump --dng -o ${BASE}/frame_;
done
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 25, 2013, 09:55:11 PM
update:
 - mlv_snd now allows to select desired sampling rate (48000, 44100, 22050, 11025, 8000)

Perfect!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 25, 2013, 10:01:53 PM
Any more settings to try I stop recording after 1 min from frame skip?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 25, 2013, 10:06:25 PM
what setting do you have now? Tried 24fps?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 25, 2013, 10:23:42 PM
Yes 23.976 fps.. I know I can record more then 1 min with normal raw.rec
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 25, 2013, 10:28:55 PM
g3gg0,

When I try to record in 3x crop mode and when I set the FPS override to 23.976 it currently disables the audio. Is there a way to have prevent the audio from being disabled now that we have have the mlv_snd module?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 25, 2013, 10:31:12 PM
g3gg0, 48khz solved my problem, now sound has a perfect sync  :D
I experienced some sound frame skips though, recorded over 1 minute clip 1920x1080 25fps and after 37sec sound lost its sync completelly. I will make some further testing.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: mageye on December 25, 2013, 10:43:59 PM
g3gg0,

When I try to record in 3x crop mode and when I set the FPS override to 23.976 it currently disables the audio. Is there a way to have prevent the audio from being disabled now that we have have the mlv_snd module?

Are you sure the audio is disabled?

The reason I say this is because when you have the FPS override turned on there is a message saying that the audio is disabled. I would record regardless of this message and just check and see if you have recorded an audio file when you extract it.

Just try it. You might be pleasantly surprised (as I was) to see that it actually does record it! ;)

If that's not the case then I really can't help you and sorry! but Merry Christmas anyway :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 10:54:08 PM

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    /usr/bin/mlv_dump --dng -o ${BASE}/frame_;
done


Not working ... do I need to feed in args or is it just a batch?  It makes the directory but nothing happens after that, just shows me the mlv_dump options.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 10:55:36 PM
sorry, missed a parameter. try this


for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    /usr/bin/mlv_dump --dng -o ${BASE}/frame_ $FILE;
done


Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:00:34 PM
I got it working like this ...

Code: [Select]
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./$FILE ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump --dng -o $BASE $FILE
    cd ..
done
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 25, 2013, 11:02:39 PM
BAH!! anything under 1920 x 960 records fine over 2mins bah.. lol
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 25, 2013, 11:09:09 PM
come on... cant you be ANY MORE DETAILED?!

"does not work bah" isnt something i could fix.
also i dont know how to reproduce that.
plus you could have tested it with different settings, with/without sound etc.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:12:27 PM
Last post, mad respect to g3gg0 for the help....

Do the following ...

1. Download mlv_dump.osx
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Make a file in the folder where you've moved the MLV files, save as recursive.sh, with contents:

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./$FILE ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump --dng -o frame_ $FILE
    /usr/bin/mlv_dump -m -o meta.mlv $FILE
    /usr/bin/mlv_dump -v meta.mlv >$FILE.txt
    rm ./meta.mlv
    rm ./meta.mlv.wav
    cd ..
done

6. Run recursive.sh from the Terminal.

This will extract all the frames from the MLV files and extract MLV metadata into a TXT file.

7. Profit!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 25, 2013, 11:17:45 PM
Are you sure the audio is disabled?

The reason I say this is because when you have the FPS override turned on there is a message saying that the audio is disabled. I would record regardless of this message and just check and see if you have recorded an audio file when you extract it.

Just try it. You might be pleasantly surprised (as I was) to see that it actually does record it! ;)

If that's not the case then I really can't help you and sorry! but Merry Christmas anyway :)

Actually you are right. It does record the sound. I just assumed that the sound wasn't coming through because the sound meters disappeared and in the menu it says the sound will be disable when FPS override is activated.

Maybe there is a way to prevent the sound meters from being disabled when FPS override is active? Not a big deal, but would be nice. :)

g3gg0,
48khz update works perfectly! I no longer need to convert the sound so that it can be read by Resolve. Thanks for the update! On Christmas even! Merry Christmas to you and everyone else.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:21:07 PM
Merry Christmas to you and everyone else.

Hear hear!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: rob_6 on December 25, 2013, 11:39:45 PM
Last post, mad respect to g3gg0 for the help....

Do the following ...

1. Download mlv_dump.osx
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Make a file in the folder where you've moved the MLV files, save as recursive.sh, with contents:

for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
    BASE=`echo $FILE | cut -d "." -f1`;
    mkdir $BASE;
    mv ./$FILE ./$BASE
    cd ./$BASE
    /usr/bin/mlv_dump --dng -o frame_ $FILE
    /usr/bin/mlv_dump -m -o meta.mlv $FILE
    /usr/bin/mlv_dump -v meta.mlv >$FILE.txt
    rm ./meta.mlv
    cd ..
done

6. Run recursive.sh from the Terminal.

This will extract all the frames from the MLV files and extract MLV metadata into a TXT file.

7. Profit!

dubzeebass!
Thanks for putting this together this great script. So great for converting a bunch of files. It works perfectly! Thanks so much. Thanks to g3gg0 again too for the help!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:40:50 PM
Cheers, glad to hear!  You could also add it to /usr/bin and always have it handy then since it's in the path.

:)

Glad to contribute!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 25, 2013, 11:49:15 PM
haha, that script from dubzeebass totally nailed it :)
Thanks a lot!

Strange though I get 2 wav files from each conversion? One called meta.mlv.wav and the other one simply called frame.wav

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 25, 2013, 11:51:51 PM
with out Snd it still skips frame at 1min
most settings are off
preview is auto
disable G draw is on

any Resolution under 1920 x 960 works fine with sound

i thought i had MLV recording doing full 1920x1080 before..

i have tried all kinds of number settings for "Buffer fell method" and "CF-only buffers"
this is on a KB 64gb 1000x card

any pointers?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:54:33 PM
haha, that script from dubzeebass totally nailed it :)
Thanks a lot!

Strange though I get 2 wav files from each conversion? One called meta.mlv.wav and the other one simply called frame.wav

    rm ./meta.mlv
    rm ./meta.mlv.wav
    cd ..

Add the bolded line into the script.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 25, 2013, 11:55:35 PM
Updated my post with the 2ndary WAV removal.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 26, 2013, 12:13:34 AM
Worked nicely :)
Can I ask you how to execute the recursive.sh script.
I copy pasted the scripttext to text editor and the saved it and changed the suffix to .sh
How do I execute this?
Thanks for helping
/D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 26, 2013, 12:33:47 AM
From the terminal... I'll make a video when I get home next week.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: naturalsound on December 26, 2013, 12:41:25 AM
Any more settings to try I stop recording after 1 min from frame skip?

I can repoduce the skip.
It seems to happen at 4096MB. Suddenly the buffer fills up and a frame is skipped. Before about 1/5 to 1/4 of the buffer is used. Nevertheless I  observe a similar behavior when the recording starts. First the buffer fills up to almost 100% then writig catches up and the buffer emptys. I assume there is something happening while beginning the second chunk and writing metadata?

5DMKIII (Fill Method: 3 CF-only buffers: 8)
Nightly build December 25th.
Nightly build December 26th.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 26, 2013, 12:46:20 AM
From the terminal... I'll make a video when I get home next week.

Ok.
I modified a command script from a member here,  jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.

Instruction would be:

1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip (http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip)
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command press enter
6. run the recursive.command by doubleclicking

You can doubleclick the recursive.command script a couple of times to have them work in parallell.

commandscript https://drive.google.com/file/d/0B4tCJMlOYfird0U5dTNnR0twNE0/edit?usp=sharing

Thanks
/D

*updated recursive.command 29 dec (g3gg0)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 26, 2013, 12:53:59 AM
I've re-uploaded it with the attributes fixed as some people may have had trouble due to ownership and administrative settings.  Cleared all attribs with Batchmod.

https://drive.google.com/file/d/0B7VaNSfNjUYOT2s0NXp3cXA0UVU/edit?usp=sharing
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 26, 2013, 01:01:55 AM
I recorded 1:30 and this time no dropped frames at all. Sound perfectly synced till the end of a clip. i dont know why i had dropped frames before, both times 1920x1080 25fps. This time however i formatted my card to exfat, before was FAT32. In both cases MLV file was splitted to 4 GB and the rest. I think i was able to record all card with single file with raw-rec. Does Mlv-rec records files over 4GB?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 26, 2013, 01:09:03 AM
I've re-uploaded it with the attributes fixed as some people may have had trouble due to ownership and administrative settings.  Cleared all attribs with Batchmod.

https://drive.google.com/file/d/0B7VaNSfNjUYOT2s0NXp3cXA0UVU/edit?usp=sharing

Tried the reuploaded fix but got problem with adminrights from that one too. Updated my modified instruction. Good work on this dubzeebass
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: naturalsound on December 26, 2013, 01:19:08 AM
I recorded 1:30 and this time no dropped frames at all. Sound perfectly synced till the end of a clip. i dont know why i had dropped frames before, both times 1920x1080 25fps. This time however i formatted my card to exfat, before was FAT32. In both cases MLV file was splitted to 4 GB and the rest. I think i was able to record all card with single file with raw-rec. Does Mlv-rec records files over 4GB?

Which buffer settings do you use for MLV 1080P?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 26, 2013, 01:24:04 AM
Which buffer settings do you use for MLV 1080P?

I made every tests with CF only buffers=0 and buffer fill method=0 so both OFF. Card spanning also OFF.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: naturalsound on December 26, 2013, 01:44:31 AM
I made every tests with CF only buffers=0 and buffer fill method=0 so both OFF. Card spanning also OFF.

skips again at 4095MB! buffers: 0 fill: 0, spanning OFF (Global Draw does not matter)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: auxmar on December 26, 2013, 01:46:10 AM
I have also problem with the mlv_rec, it does lot of skipping frames with or without sound, especially the first time i try to record. i have tried various setting but, still skipping, the max time a get is about 1 minute. with Raw_rec it record continuously without skip most of the time.
i have the KB 64G 1000X with 5d mk III, the bitrate when it record mlv is around 80, with Raw_rec it's around 90 .

can someone share his setting of MLV to record continuously on MK III ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Tai.Fighter on December 26, 2013, 05:42:21 AM
Hi and Merry Christmas! Running last night's nightly build on a 5D MkII.

However when I try to load the mlv_dump module, I get a "file err" and "ret 0x", not allowing it to load. Does anyone know what's going on and how I can fix it? Thanks!

Also, the first post of this thread is extremely confusing for a MagicLantern newbie like me... Is there a comprehensive full guide/post that could explain clearly the steps to get mlv_dump working from shooting to processing? I'd really appreciate it, even willing to pay through PayPal! (The bit-depth/file-size reduction is so tempting)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ouuzi on December 26, 2013, 09:02:50 AM
People one more question.Do I have to copy autoexec.bin from the first post aswell?I got it working with 25 dec build but i dod not copy the auto exec.bin.Am i missing something?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 26, 2013, 10:32:56 AM
I have all files from mlv-rec package on my card, including autoexec.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 26, 2013, 11:51:42 AM
On the 6d, every first clip, audio is recorded good.
Every clip after it, only has a very small audio file (just as long as the sync beep file would be), it contains normal audio though, but it's only 1 or 2 seconds long.
And it says "audio failed to stop, state 4".
When I turn the camera off and on, the first clip has normal audio, the next clips have the state 4 error message and a 1 or 2 seconds audio file.
Is there something in the ML menu that I need to turn on or off ?

I'm using the new MLV record and sound modules in tragic lantern build from 30 November, which is the most recent I can find for 6d.
There isn't a nightly build for 6d (does anyone know why there are no nightly builds for 6d ?)

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 26, 2013, 02:19:44 PM
I'm shooting on a 5D3 with 64gb 1000x KB cards.  I shot about 1:30 of mlv_raw w/audio and got some stuff I hadn't seen before…  On the card, I have 3 files M26-0833.mlv, M26-0833.M00, and M26-0833.M01.  Why did it span my shot across multiple clips?  How do I combine them?

I am using the latest MLV_REC.mo and have never seen this before...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 26, 2013, 02:30:11 PM
I'm shooting on a 5D3 with 64gb 1000x KB cards.  I shot about 1:30 of mlv_raw w/audio and got some stuff I hadn't seen before…  On the card, I have 3 files M26-0833.mlv, M26-0833.M00, and M26-0833.M01.  Why did it span my shot across multiple clips?  How do I combine them?

I am using the latest MLV_REC.mo and have never seen this before...

NVM it's automatic…  Why did it span though?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 26, 2013, 02:47:17 PM
some technical facts:
 - structured format
 - extensible layout
 - as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
- multi-file support (4 GiB splitting is enforced)
 - spanning suport (write to CF and SD in parallel to gain 20MiB/s)
 - out-of-order data support (frames are written some random order, depending on which memory slot is free)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 26, 2013, 03:37:38 PM
ok my problem is happening after the 4gb limit is reach it starts the next file .m00 it gives me a frame skip .. tried and few settings and it happens the same way all the time..
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 26, 2013, 04:09:31 PM
is the buffer already quite full when the next chunk is created?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 26, 2013, 04:18:54 PM
Right when the next chunk it starts to get full then skips
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 26, 2013, 04:25:54 PM
problem is, i canot create another chunk how it was done in raw_rec.
alex just wrote and check how much was written.
if the write failed because of a 4G limit, then he writes the rest into another chunk.

i cant do this because the file format doesnt allow that.
i must know in advance if your setup (camera + card) allows files >4G.

maybe i can add an option "Files >4G supported" for those who know that they have exfat and a 5D3.
but this will cause for sure 10 other people asking why it doesnt work and that the option is faulty etc etc.
then there is again a lot of crying here. oh dear...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 26, 2013, 04:52:29 PM
Yup my cards are exfat also
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: sergiocamara93 on December 26, 2013, 06:06:13 PM
maybe i can add an option "Files >4G supported" for those who know that they have exfat and a 5D3.
but this will cause for sure 10 other people asking why it doesnt work and that the option is faulty etc etc.
then there is again a lot of crying here. oh dear...

I understand your concerns, g3gg0, but I would really appreciate to have that option. For me it's much more convenient having one single .mlv file per shot. People using this module (or ML by extension) should know their cards file system and its limitations.

But that isn't important. We finally have synchronous audio and I cannot thank you enough. I've been working with the 'beep' and the slate method and it was quite painful. This makes things so much easier and quicker. For those doing their dailies in Resolve and struggling, there's an amazing tutorial from Abelcine  (http://www.youtube.com/watch?v=GRKoJuRjajE)(I know I should be posting this link in the postproduction section but it's much more likely to be seen here). Right now sync in Resolve can only be done manually, in order for it to do it automatically the video and audio need to have matching time codes. Anyway, manual sync it's a minor inconvenient for this awesome format. Thanks again!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MvM on December 26, 2013, 08:41:06 PM
Hi, how about, naming it 5dmk3-exfat-only-mode ? So everybody knows what and how and so on... Thanks for your work and reading.
Merry ... Matthias
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 26, 2013, 09:15:58 PM
One thing after such change is to remember not to format card in camera anymore. I didnt know that formatting in camera makes card FAT32 even if it was formatted as exfat before. So only deleting clips via browser. If someone forget this - well -  he can have much trouble unless has a laptop with him.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 26, 2013, 09:35:12 PM
a option to delete all in folder..
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: seku on December 26, 2013, 10:36:02 PM
agreed on the exFAT part. and ML, you're making dreams come true !

i'd never have thought to have the possibility, but i just got a few lexar cards, and a 1TB scratch SSD for the dng files for editing... i can try a RAW workflow ...this holiday trip shall be a lot funnier than i initially thought!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 26, 2013, 10:46:52 PM
I compiled myself for 5D3 including a "make clean" and make for the new modules.  For some reason my modules do not always get compiled when something new gets updated.

Tested "as is" with the camera settings as have been saved by g3goo.  Great results all audio is pretty well matched.  I get better alignment if I move the audio forward by 1 frame.

Changed setting to Preview=Hacked and also enabled Mrmory Hack, and card spanning ON.

Tested with Card spanning ON at 1920x1080 at 29.97 fps and the audio matched again!

Tested with Card spanning ON cropped video at 1920x1080 at 29.97 and had to slide audio forward by many frames and then to shrink the audio a bit.  After that it matched.

Tested with Card spanning ON 60 fps at 1600x560 the audio could not be synched by trimming or stretching or sliding, the audio was there but could not align.

Tested with Card spanning ON Tested 50 fps at 1856x650 the audio could not be synched by trimming or stretching or sliding, the audio was there but could not align.

For Post processing I always used MLVBrowseSharp and exported to DNG (with green cast) then I imported the dng and wav file to Vegas.

Noticed that the green cast on dng is not due to BlackLevels.  It appears that Color Temperature and Tint information is missing.  As soon as I set in ACR Temperature to 5000 and tint to 0 the colors seem appropriate.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 27, 2013, 02:04:10 PM
Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks

Filmed some clips during the holidays and now I am editing I hear that my voice and other male voices do sound lower and slower (than they should be) and my wife and kids voices actually sounds normal ?
Could it be that there is something going on with the lower frequencies ?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 27, 2013, 02:09:09 PM
I had the same problem. Newest build resolves this, just make sure sound recording is set to 48khz in the sound menu. This should help. Cheers
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Toffifee on December 27, 2013, 04:56:12 PM
Sound recording! Yes!

I tested 30 seconds with 48k + 23,976fps + 1920x1080p, perfect sync so far   :D

It was quite a struggle getting the .mlv files to work on a mac, but thanks to Danne's post I got it working.

Does anybody have a program or developing such as rawmagic with support for mlv?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 27, 2013, 05:19:00 PM
The newest MLV browser has ability to unpack mlv to dng with wav or raw with wav. Right click on the file icon.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Toffifee on December 27, 2013, 05:49:26 PM
Where can I download the mlv browser? Can't seem to find it.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 27, 2013, 06:00:50 PM
See reply 101-111
http://www.magiclantern.fm/forum/index.php?topic=8447.100
It,s a work in progress. g3gg0 hasn,t implemented audio export on mac yet I think.

Another work in progress is this fine viewer
http://www.magiclantern.fm/forum/index.php?topic=9560.msg91651;topicseen#msg91651
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 27, 2013, 08:11:11 PM
I had the same problem. Newest build resolves this, just make sure sound recording is set to 48khz in the sound menu. This should help. Cheers

Thanks will check the newest build.

I see at the bottom of your posts that you also have 6d, do you use the new sound module and new MLV-rec  module with it, or did you only test it on 5d mark 3?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 27, 2013, 08:55:20 PM
Thanks will check the newest build.

I see at the bottom of your posts that you also have 6d, do you use the new sound module and new MLV-rec  module with it, or did you only test it on 5d mark 3?

No, currently tested only on 5d3.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 28, 2013, 01:02:55 AM
Tested longer clips on 5D3.

1920x1080 at 29.97 fps
Preview: Hacked
Memory Hack: ON
Extra Hack: ON
Buffer: 4:3
Card Spanning: ON
Global Draw ON but nothing selected ( I use this to get proper framing without much speed lost)

Recorded for 3 min and audio was in synch.  about 20,7 GB  Recording stopped automatically on frame skip

Tried the same on cropped video.

Once recorded for 4 min and the audio was OK untill 2 min 10 sec.  After which point the audio skipped ahead, it seems that the ordering of the frames was lost.
Same thing happened on second recording for 3.5 minutes.  The audio skipped ahead at 1 min 50 sec.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Eran on December 28, 2013, 02:58:05 AM
The Sound recording is amazing!

Davinci Resolve have no problem to recognize 48khz, thanks g3gg0 !
My workflow: MLVBrowseSharp ---  raw2cdng.1.4.8 --- dropping each WAV into his DNG's file  --- premiere pro cc  ----  davinci resolve 10 lite.

But when I am exporting the final timeline as XML from premiere pro cc to davinci resolve 10, I'm not getting any signal on the sound tracks (I do get all the tracks though but mute). Anyone have same experience?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 28, 2013, 03:03:50 AM
More details of sound skipping during cropped video:

I counted the seconds on the recording information and then listened to hear how the numbers are scrambled:

From beginning to second 92 is OK, then it starts counting on second 152 until 211, then it skips to 231 and continues counting until the 240 (end of recording), then it picks ups from 93 till 151, then it picks from 132 until 151.

1....92
152...211
231....240
93...151

So yes definitely the audio frames are being mixed up in cropped recording.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on December 28, 2013, 07:37:59 AM
The Sound recording is amazing!

Davinci Resolve have no problem to recognize 48khz, thanks g3gg0 !
My workflow: MLVBrowseSharp ---  raw2cdng.1.4.8 --- dropping each WAV into his DNG's file  --- premiere pro cc  ----  davinci resolve 10 lite.

But when I am exporting the final timeline as XML from premiere pro cc to davinci resolve 10, I'm not getting any signal on the sound tracks (I do get all the tracks though but mute). Anyone have same experience?

everything works fine for me in resolve 10. make sure u dont mute ur sound track in premiere pro n resolve
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 28, 2013, 10:55:45 AM
So yes definitely the audio frames are being mixed up in cropped recording.

can you create an index before dumping?
(run mlv_dump with -x first)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 28, 2013, 01:51:24 PM
can you create an index before dumping?
(run mlv_dump with -x first)

Yes it creates, with MLVSharpBrowser it gives the same results.

With mlv_dump I get the following results:

Code: [Select]
F:\0-Videos\14-test\sound\cropped>mlv_dump -x M27

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: 'M27-1927.MLV'
   - Verify file structure
   - Output .idx file for faster processing
[i] File M27-1927.MLV opened
[i] File M27-1927.M00 opened
[i] File M27-1927.M01 opened
[i] File M27-1927.M02 opened
[i] File M27-1927.M03 opened
[i] File M27-1927.M04 opened
[i] File M27-1927.M05 opened
[i] Processing...
[i] Reached end of chunk 0/7 after 2422 blocks
[i] Reached end of chunk 1/7 after 3167 blocks
[i] Reached end of chunk 2/7 after 2393 blocks
[i] Reached end of chunk 3/7 after 2490 blocks
[i] Reached end of chunk 4/7 after 2484 blocks
[i] Reached end of chunk 5/7 after 2389 blocks
[i] Reached end of chunk 6/7 after 788 blocks
[i] Processed 7215 video frames
[i] XREF table contains 16133 entries
[i] File M27-1927.IDX opened for writing
[i] Done


F:\0-Videos\14-test\sound\cropped>mlv_dump --dng

 MLV Dumper v1.0
-----------------

[i] Mode of operation:
   - Input MLV file: 'M27-1927.MLV'
   - Convert to DNG frames
   - Output into 'M27-1927_frame_'
[i] File M27-1927.MLV opened
[i] File M27-1927.M00 opened
[i] File M27-1927.M01 opened
[i] File M27-1927.M02 opened
[i] File M27-1927.M03 opened
[i] File M27-1927.M04 opened
[i] File M27-1927.M05 opened
[i] File M27-1927.IDX opened (XREF)
[i] XREF table contains 16133 entries
[i] Processing...
Error: GUID within the file chunks mismatch!
[i] Processed 0 video frames
[i] Done
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: dubzeebass on December 28, 2013, 02:35:15 PM
What's the point in building the index first?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 28, 2013, 02:50:39 PM
all blocks are save in some random order, depending on the current buffer usage.
the recoding module picks the largest block to write so you get maximum write rate.

when saving .raw or single .dng with mlv_dump, it doesnt matter in which order they are written.
but the .wav must (currently) be written in the correct order.
for parsing the blocks in the correct order, you need an index file that can be created using the -x option.
all subsequent mlv_dump executions will use the index for correct processing (in terms of time order)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 28, 2013, 03:26:36 PM
In case you did not see it, I get the following error:

Error: GUID within the file chunks mismatch!

First I did the mlv_dump -x
Then I used mlv_dump --dng

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: NedB on December 28, 2013, 03:31:23 PM
@g3gg0: During one of my first recordings with mlv_snd, my 550D camera locked up (got a "busy" text, sorry I don't remember the exact wording) and apparently the sound file didn't get written properly: I get a "Received AUDF without WAVI, the .wav file might be corrupt" message after all the .dng's are (correctly, it seems) extracted.

Despite this, the .dng's look fine and the length of the resulting .wav file is roughly what one would expect for a 40 second (at 25fps) clip. Of course, it is corrupt, as its metadata indicates "MPEG AAC Audio at 7350 Hz, 28kb/s" instead of the correct "PCM S16 LE Stereo 44100 Hz, 16bps (from VLC player)" as on my other clips.

Question: is it possible to rescue this .wav clip by inserting a WAVI block into the .mlv file, or by hex-editing the .wav itself to correct the header? [Update: SOLVED! See two posts down]
In any case, thanks again for raw with sound, it's a quantum leap in usability and enjoyment! Cheers.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: NedB on December 28, 2013, 03:39:45 PM
To all ML users trying out raw with sound: I was shooting a scene with my 550D, using mlv_raw and mlv_snd. One of the clips turned out to have an ugly greenish-black tint over all the .dng's, even though I didn't consciously change any settings between the various clips of the same scene. After I tried and failed to correct this in ACR (Photoshop/After Effects), I thought of using a more scientific method. Using Phil Harvey's exiftool (Google it), I compared the metadata of a .dng from a "good" clip to the metadata of one of these "green-black" .dng's. The only difference I noticed was in the data tagged "Exif:BlackLevel", which had a value of 2046 in the "good" .dng but 2509 in the "bad" clip.

I used exiftoolGUI (Google it) to change the value of "Exif:BlackLevel" on all the "bad" .dng's to 2046, and the problem was solved: the .dng's now looked exactly like ones from the other clips I shot of this scene.

Again, I don't know whether or not I did something to cause this error, or whether it will just crop up from time to time. But if you have this symptom (.dng's with a crushed-black/greenish tint), give it a try. Cheers!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: NedB on December 28, 2013, 07:10:48 PM
@g3gg0 (again): Again the principle of "if you formulate a question clearly enough, you almost have your answer" applies. Pasted the header (Addresses 00000000 - 0000002B) of one of the "good" audio files over the same addresses in the "bad" audio file and it worked. FYI for anyone with a similar problem (shooting with mlv_rec and mlv_snd and you have a crash or skipped frames, resulting in an error message about a possible corrupt .wav file). Cheers and I hope TLDR didn't apply too much...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 28, 2013, 10:53:03 PM
the latest binaries should not need mlv_dump -x anymore.
(although it should work. can you check if you still get such GUID error?)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 29, 2013, 02:35:59 AM
Updated batchscript command for mac-users(30dec 2013) Thanks to dubzeebass, g3gg0, daisermac, jerrykil

Ok.
I modified a command script from a member here,  jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.

Instruction would be:

1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command0.2 press enter
6. run the recursive.command by doubleclicking

You can doubleclick the recursive.command0.2 script a couple of times to have them work in parallell.

commandscript https://drive.google.com/file/d/0B4tCJMlOYfirWDd2RldZVXdjck0/edit?usp=sharing

Thanks
/D

*updated recursive.command 29 dec (g3gg0)
*updated recursive.command0.2 30 dec (daisermac moving spanned files to folder)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 29, 2013, 04:32:00 AM
the latest binaries should not need mlv_dump -x anymore.
(although it should work. can you check if you still get such GUID error?)

Compiled new build for 5D3, and downloaded new MLVBrowser

I recorded 4 minutes 4 seconds of CROPPED video 1920x1080 at 29,97 fps = 24.5 GB  I stopped the recording.

The audio file is now in correct order, I counted all the way up and never missed any numbers during playback.

To aligned the video I had to shrink the audio file form 4:02,24  to 4:01,08 which is 0.48% 
In other words, I had to shrink the audio file by 1.16 seconds.  Everything was properly aligned at this point.
To check for alignment I hit a little crystal bottle every ten seconds and record the strike, once the file was shrunk the strike matched perfectly on every hit.

Congratulations, you definitely got the wav frame order correct!

Tomorrow I will check 50 and 60 fps.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: daisermac on December 29, 2013, 01:42:13 PM
Updated batchscript command for mac-users

Ok.
I modified a command script from a member here,  jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.

Instruction would be:

1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command press enter
6. run the recursive.command by doubleclicking

You can doubleclick the recursive.command script a couple of times to have them work in parallell.

commandscript https://drive.google.com/file/d/0B4tCJMlOYfird0U5dTNnR0twNE0/edit?usp=sharing

Thanks
/D

*updated recursive.command 29 dec (g3gg0)

Thanks Danne for your script.

I think it needs one modification in order to handle split mlv-files - the move command should be:

mv ./"$BASE".M* ./$BASE

This way not only the first .MLV gets moved but also .M00 and .M01 etc.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Arturo Sánchez on December 29, 2013, 02:58:05 PM
Dual ISO and 3X crop mode supported? And both together?
And what about 48-60fps ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 29, 2013, 03:48:12 PM
Thanks Danne for your script.

I think it needs one modification in order to handle split mlv-files - the move command should be:

mv ./"$BASE".M* ./$BASE

This way not only the first .MLV gets moved but also .M00 and .M01 etc.


Thanks. Updated the scriptinstruction, reply #1044, and put in your code. Please test it, havn,t got any spanned .mlv files at the moment.
D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 29, 2013, 03:53:30 PM
Reporting 5D3 testing with 50fps and 60fps, card spanning on.
Sound file needed to be moved 4 frames forward and bingo it is synchronized both for 50 and 60 fps

Only outstanding issue is Color Temperature and Tint for dngs.  They have green cast corrected on ACR: when ACR opens the dng it reads colortemperature 2000 and tint -150.  This is WB error. To correct set WB to 5000 and tint 0
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 29, 2013, 06:24:49 PM
Reporting 5D3 testing with sound module.

During cropped mode I cannot go above 2240x1260 pixels.  The LV has some sort of an error and quits working.  If I disable the sound module it works OK.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 06:34:40 PM
During cropped mode I cannot go above 2240x1260 pixels.  The LV has some sort of an error and quits working.  If I disable the sound module it works OK.

thanks! will check where this comes from.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 29, 2013, 07:14:02 PM
When I turn the camera on, MLV Sound & MLV Raw seem to be turned off by default.  Is this the case, or do I not have a setting flipped or something?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 29, 2013, 07:50:34 PM
When I turn the camera on, MLV Sound & MLV Raw seem to be turned off by default.  Is this the case, or do I not have a setting flipped or something?

yup for me too...its like i need to go to canon audio menu to make it come back on.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on December 29, 2013, 08:14:13 PM
7D doesn't seem to record any audio frames, dunno why I see the red buffers . Canon audio on or off. 6D I need to play with more but I did get some wavs out.

7D Log:

http://pastebin.com/8EjPPFbi


How do you stop the ADC? At first I was getting audio failed to stop 4->0. I added a stopasifdmaadc but not sure if it helps. At least on 6D it recorded more than once after that.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 08:36:34 PM
on 7D i guess it misses this part:

    /* experimental for 7D now, has to be made generic */
    /* Enable audio Device */
    void SoundDevActiveIn (uint32_t);
    SoundDevActiveIn(0);


this is something for the "next-gen" wave support in magic lantern.

sound recording is simply stopped by not further queueing any new audio buffers.
there are always two buffers (double buffered) and as soon one is filled, you get the callback
where you have to enqueue the second-to-next buffer.

if you dont queue a buffer during the callback, you will get another callback for the last buffer being filled.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 29, 2013, 08:47:45 PM
Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac

download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing

1 – Extract recursive0.3.command and mlv_dump.osx  in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx  according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders

I,m not a coder so feel free to critizise

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on December 29, 2013, 08:51:33 PM
That stuff is old news. I have full audio support for 7D (pel should port this back). Meters are going while its recording. Its just AK4646 with a different audio task you can't override. You have to re configure it after canon like on 600D.

I can record old style wav just fine but then I record with MLV_snd and get nothing... on 6D I'm getting 4GB wavs for some reason (with data), mlv dump just keeps going and going until that limit is reached. I can't run trace because it crashes from out of memory.

If you leave the asif going it crashes out on a bunch of cams or won't let you restart it until it times out. When I run the FFT test I also get action on it on both cameras.

Also: 6D with correct .raw file and wav (5MB) in MLV browser (first recorded clip, canon audio off, audio override)
Next clip is 4gb, 930kb resaved. At least all data seems to be there on 6D.
Code: [Select]
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 1380004378867732245
    FPS         : 23.970000
    File        : 0 / 0
    Frames Video: 0
    Frames Audio: 0

Ok, got 1 wav out of the 7D, correctly sized and both channels out of 5 files. Canon audio off, separate wav override. Maybe its a timing issue. All entries in the log look identical:

mlv_snd_flush_entries: 105 entries to free in queue
/* 00:01:27.710749  */ mlv_snd_flush_entries: 98 entries to free in queue
/* 00:01:27.711923  */ mlv_snd_flush_entries: entry is MLV slot
/* 00:01:27.712237  */ mlv_snd_flush_entries: NULL slot 0 entry
etc....

All clips show 0 audio frames.

*update... I hear some audio in MLV viewer but wavs are not extracted by mlv browser. Audio playback is jerky... like silence silence sound, silence silence sound, etc.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 29, 2013, 10:18:04 PM
I've run into three problems with the latest (er, yesterday's) build, I figured maybe documenting them here would help development or someone could tell me what I'm doing wrong.

1. I loaded the mlv-rec and mlv-snd modules and set it for 1920x1080p23.976 and tried recording some raw video to my Komputerbay 128Gb 1050X CF card. It gets about 48-70 frames in, reports a skipped frame or two, and stops recording. Every time with the same result. I think spanning is off, and there's only files recorded to CF, but I'm not sure (need to poke around and find that setting again). Recording at the same resolution and frame rate using the normal raw recording module works flawlessly to the same card.

2. I booted the camera with mlv-rec and mlv-snd enabled, and (without recording) hit the zoom button to see a close-up view for focusing. After focusing, I didn't hit the zoom button again to pop back to the normal view and instead just hit record thinking the view would pop back to normal and recording would commence. Instead the view froze, and remained on the screen after I shut down the camera. I had to remove the battery to get it to power off.

3. It's been noted earlier in the thread, but the mac version of mlv_dump claims the .mlv file lacks a header unless the .IDX file is moved, deleted, or renamed out of the application's view. If the IDX is present, no decoding happens. When I renamed the .IDX file to end in .IDX.bak, mlv_dump was able to see the file and I could view the metadata for its contents just fine.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 10:23:14 PM
Ok, got 1 wav out of the 7D, correctly sized and both channels out of 5 files. Canon audio off, separate wav override. Maybe its a timing issue. All entries in the log look identical:

[...}

*update... I hear some audio in MLV viewer but wavs are not extracted by mlv browser. Audio playback is jerky... like silence silence sound, silence silence sound, etc.


when adding the lines i mentioned the 7D can record 5/5 times.
maybe you missed the necessary
> SoundDevShutDownIn();

about jerky playback... well maybe it is because the video plays back with 1 fps instead of 25...?
MLVViewer is no media player, it is for previewing ;)

just export and decide by the .wav
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 10:30:38 PM
btw. added 7D support. see bitbucket, no binaries yet.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 29, 2013, 11:01:17 PM
1. I loaded the mlv-rec and mlv-snd modules and set it for 1920x1080p23.976 and tried recording some raw video to my Komputerbay 128Gb 1050X CF card. It gets about 48-70 frames in, reports a skipped frame or two, and stops recording. Every time with the same result. I think spanning is off, and there's only files recorded to CF, but I'm not sure (need to poke around and find that setting again). Recording at the same resolution and frame rate using the normal raw recording module works flawlessly to the same card.

I just looked at the SD and CF card and found the answer to this problem -- it was trying to record everything to the SD card. I had the preferred card set to CF, so I figured I had this set properly. Maybe there's some other mechanism to use to tell ML to just never, ever try to record anything to the SD card?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 29, 2013, 11:13:55 PM
OK, got the latest nightly build and the latest mlv_rec modules installed now and am ready to try again.

Is it wrong that in the ML menu, the descriptive text for mlv_snd.mo is "Adds SMPTE output functionality to mlv_rec"?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on December 29, 2013, 11:20:47 PM
I'll give the commands a go but have the feeling it will just reset the audio settings (to 23db internal mic)... look at what registers those touch.

It seems I'm getting wav 5/5 times too, just MLV browse isn't dumping it and I never thought to try MLV player.

Quote
Is it wrong that in the ML menu, the descriptive text for mlv_snd.mo is "Adds SMPTE output functionality to mlv_rec"?

Yep.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 11:23:49 PM
uhm just tried - it shows the correct description.
but i realized that i put an old version into it, not the most recent one.

reuploaded.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 29, 2013, 11:26:01 PM
It seems I'm getting wav 5/5 times too, just MLV browse isn't dumping it and I never thought to try MLV player.

MLVBrowseSharp doesnt dump for you?
which OS? which error? did you get the latest version?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 29, 2013, 11:31:48 PM
Got continuous recording working now, with possible skipped frames.

This may be something unrelated to mlv and simply a new (to me) way that the nightly ML is noting skipped frames, but I'm not sure what to make of this "71 frames / 4115 expected" text at the bottom of the frame while recording.

I've got a screenshot I'd be happy to post, but clicking the "Insert image" button in this form just inserts a couple of img tags into my post -- doesn't seem that there's an upload option. Maybe this'll work: (https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832)
Or just this link: https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832 (https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832)

It appears to be recording fine and the buffer graph at top is not filling past the end, but that text at the bottom alternates between saying things like "00:39.941 frames. continuous OK !" and "58 frames / 5102 expected". I'm not sure how to interpret this.

When I installed the latest nightly I blew away my settings. So I'm not sure what in my previous settings caused it to be recording to SD.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on December 29, 2013, 11:35:40 PM
Same 7D files I thought were 1/5 were 5/5 when played in MLV viewer.

Quote
which OS? which error? did you get the latest version?

Win7 x64
Downloaded again today from first post when I saw commits. Both dated 12/29

Both 6D and 7D files are coming out 0 audio frames a bunch in the log but audio is there and looks complete. The 6D files are 4GB after the first clip. Most of the 6D extracts even with 0 audio frames reported.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 30, 2013, 12:06:09 AM
Both 6D and 7D files are coming out 0 audio frames a bunch in the log but audio is there and looks complete. The 6D files are 4GB after the first clip. Most of the 6D extracts even with 0 audio frames reported.

just tried on 7D:

audio works fine, here a log.

Exec: 'mlv_dump.exe --batch --dng -v -o "d:\Users\g3gg0\Desktop\Neuer Ordner\M30-0051.MLV.frame_" "F:\DCIM\100EOS7D\M30-0051.MLV"'
 MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'F:\DCIM\100EOS7D\M30-0051.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into 'd:\Users\g3gg0\Desktop\Neuer Ordner\M30-0051.MLV.frame_'
File F:\DCIM\100EOS7D\M30-0051.MLV opened
File F:\DCIM\100EOS7D\M30-0051.IDX opened (XREF)
XREF table contains 99 entries
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 3806773218881307577
    FPS         : 25.000000
    File        : 0 / 0
    Frames Video: 73
    Frames Audio: 16


sure you use *my* release?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Yoshiyuki Blade on December 30, 2013, 12:57:03 AM
I've been playing with the MLV dump compression options and I have problems when using the -b parameter. When outputting files to DNG, the images are distorted in one form or another (I can't even recognize what's in the image). Most of the time it's all white with some random bits of noise. Tried playing around with different values, but with no luck. This parameter gives just a little more compression than without it, but even that's a nice thing to have. The files were recorded on a 5D2. All the other parameters (-e -l -z) appear to work fine in the short tests I've done.

edit: spoke a little too soon. There are a few artifacts coming from somewhere else, though not as severe.
edit2: Recompressed it and it looks fine now o_O I think having the MLV Player running while its compressing isn't a good idea XD Outputting the frames crashes after a while though.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on December 30, 2013, 05:10:59 PM
On 6d
working with the newest available build (11 december build for 6d). And the mlv-rec and mlv-snd module from yesterday (copied from the 5d build, from the download link in the 1st post.)
First clip is with audio and error free.
Every clip after the first one gives the error (audio failed to stop, state 4), in those files I can't extract a full wave file with MLV-dump.osx.

I'm I doing things right by using the 11 december build for the 6d and replacing the mlv-rec.mo and mlv-snd.mo file for the ones in available in the newest 5d build ?


Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 30, 2013, 06:00:40 PM
so with MLV_REC whats the best format the cards should be in ? exFAT or
FAT32.. all my cards are KB 64gb 1000 exFAT


still getting skip frame when it starts to roll over to the next 4GB file
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 30, 2013, 07:23:45 PM
Is anyone able to record long clips? I got a 1920x1080p23.976 clip with audio recorded to my Komputerbay 1050X 128Gb card, and it appeared to be running continuously ok (except for onscreen messages about "expected frames" that I don't really understand), but recording stopped soon after the 4Gb limit was hit. My card is formatted exFat, and I can record to it with the standard raw recording for 10 minutes and beyond without issue.

I'm processing the file now, it appears to have actual audio (I listened to the wav), but I haven't yet viewed the clip to see if there are any apparent skipped frames.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 30, 2013, 07:25:31 PM
BTW, here's another way to script the processing of MLV files, using tcsh:

% foreach foo ( *.MLV )
foreach? mlv_dump --dng -o $foo:r_ "$foo"
foreach? mlv_dump -m -o $foo:r_meta.mlv "$foo"
foreach? mlv_dump -v $foo:r_meta.mlv > $foo:r.txt
foreach? rm $foo:r_meta.mlv
foreach? rm $foo:r_meta.mlv.wav
foreach? end

I dropped my .MLV and .M00 file for one clip into a folder, then cd'd into that folder and typed the lines above (the "%" and later "foreach?" are prompts, no need to type those).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 30, 2013, 08:29:22 PM
FAT32 FTW!

I reformatted my CF card using OS X Disk Utility to FAT32, and now I can record long stretches with no skipped frames (where onscreen was previously displayed "X Frames Expected" now it just says "continuous frames OK"), and no skips at 4Gb split points.

So some issue must be getting in the way of recording MLV 2.0 to exFat formatted CF. But Fat32 seems to work a treat.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 30, 2013, 09:05:17 PM
And just like that, after I pulled the files from the CF card to hard drive and deleted them, it's skipping frames again, still formatted to FAT32.

I guess I'll shoot with the normal raw video and second system sound at a shoot this afternoon, then I'll get back to trying to make mlv2.0 work for me later. It's very exciting that sound recording has been made possible! I'm thinking even a scratch track at 8-bit 22kHz could be used to automatically sync up the second system audio later (pluraleyes/fcpx/etc).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 31, 2013, 02:38:03 PM
updated mlv_dump to skip WBGain information when writing to DNG
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 02:45:09 PM
I shot a full 12 mins @ 1920x1080 with sound on a 5D3 with the 64gb KomputerBay 1000x card in a test.  Maybe this isn't new information, but I was reading that people were having issues with skipping… I didn't lose sync once.

*It is important to note, however, in a previous attempt with the sync beep on (which didn't sound at rec start) there was NO recorded audio when I processed the clip.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 31, 2013, 02:47:35 PM
I shot a full 12 mins @ 1920x1080 with sound on a 5D3 with the 64gb KomputerBay 1000x card in a test.  Maybe this isn't new information, but I was reading that people were having issues with skipping… I didn't lose sync once.

*It is important to note, however, in a previous attempt with the sync beep on (which didn't sound at rec start) there was NO recorded audio when I processed the clip.

what are your settings on MLV_REC?
after about 60-75secs it skips and i tried all 8 of my KB cards.. lol
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 03:05:28 PM
what are your settings on MLV_REC?
after about 60-75secs it skips and i tried all 8 of my KB cards.. lol

Audio
1 Beep, test tones - off
2 MLV Sound - ON
   - trace off
   - Sample rate 48k

Movie
1 - REC indicator - Elapsed Time
2 - Gradual Expo - off
3 - FPS override - off
4 - HDR video - off
5 - Vignetting - off
6 - Image Finetunning - none
7 - RAW video (MLV) - on, 1920x1080
   - disable g draw - off
   - frame skip - off
   - preview - ML Grayscale (I'm using a zacuto evf so it isn't grayscale over hdmi)
   - start delay - off
   - digital dolly - off
   - card warm-up - off
   - memory hack -off
   - extra hacks - on
   - debug trace - off
   - show buffer graph - off
   - test mode - off
   - buffer fill method - 4
   - CF-only buffers - 3
   - Card spanning - off
   - tag:text - none
   - tag:take - off
8 - Movie Tweaks
   - movie logging - off
   - REC/stby - red crossout
   - dim rec led - on
   - force liveview - start& cpulens

Modules - file_man, mlv_rec, mlv_snd  *no other modules loaded.

Sound in canon menu is on auto.

I'm not sure if ALL of these settings make a difference… but maybe it'll help trace down the issue. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 03:07:37 PM
I do think it is important to note that with sync beep on, several times my first frame has been garbled and no audio was recorded, despite mlv sound being on.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on December 31, 2013, 03:10:48 PM
Warm up your card and set picture quality to jpg not raw.
I get continuous woth 24fps 1920x1080 with sound and continuous with 25fps after warmed up card
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 03:17:59 PM
Warm up your card and set picture quality to jpg not raw.
I get continuous woth 24fps 1920x1080 with sound and continuous with 25fps after warmed up card

I have picture quality set to raw not jpg.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on December 31, 2013, 03:19:25 PM
updated mlv_dump to skip WBGain information when writing to DNG
Ok, I compiled new mlv_dump.exe and now MLBrowser exports dng with WB Temperature 5500 and Tint +19.
Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 31, 2013, 04:33:49 PM
Audio
1 Beep, test tones - off
2 MLV Sound - ON
   - trace off
   - Sample rate 48k

Movie
1 - REC indicator - Elapsed Time
2 - Gradual Expo - off
3 - FPS override - off
4 - HDR video - off
5 - Vignetting - off
6 - Image Finetunning - none
7 - RAW video (MLV) - on, 1920x1080
   - disable g draw - off
   - frame skip - off
   - preview - ML Grayscale (I'm using a zacuto evf so it isn't grayscale over hdmi)
   - start delay - off
   - digital dolly - off
   - card warm-up - off
   - memory hack -off
   - extra hacks - on
   - debug trace - off
   - show buffer graph - off
   - test mode - off
   - buffer fill method - 4
   - CF-only buffers - 3
   - Card spanning - off
   - tag:text - none
   - tag:take - off
8 - Movie Tweaks
   - movie logging - off
   - REC/stby - red crossout
   - dim rec led - on
   - force liveview - start& cpulens

Modules - file_man, mlv_rec, mlv_snd  *no other modules loaded.

Sound in canon menu is on auto.

I'm not sure if ALL of these settings make a difference… but maybe it'll help trace down the issue.

tried 5 times I did get 1at 3mins also dave are you still getting .M00 files also?

everytime the file needs to break pass the 4GB size thats when it starts.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: zachnfine on December 31, 2013, 04:48:58 PM
Set picture quality to jpeg? You mean in the normal Canon menu? Does that affect video recording for some reason with mlv_rec when it doesn't seem to have an impact on raw_rec?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: kgv5 on December 31, 2013, 06:53:27 PM

   - memory hack -off
   - extra hacks - on
 

Are you sure, that memory hack shouldnt be ON?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 08:45:31 PM
tried 5 times I did get 1at 3mins also dave are you still getting .M00 files also?

everytime the file needs to break pass the 4GB size thats when it starts.

Yep, I'm still getting .m00 files, but I am not (nor did I ever) have an issue with them splitting. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 08:46:32 PM
Are you sure, that memory hack shouldnt be ON?

Should it be? I don't know.  It isn't in my setup and it is working. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 31, 2013, 08:53:02 PM
Yep, I'm still getting .m00 files, but I am not (nor did I ever) have an issue with them splitting.

anotherdave,
    whats your allocation unit size on your KB cards ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on December 31, 2013, 08:54:17 PM
the first 4GiB are saved in a .mlv, the next in a .m00, the next in .m01 and so on.

memory hack on 5D3 gives you some extra buffers
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: AnotherDave on December 31, 2013, 10:57:20 PM
anotherdave,
    whats your allocation unit size on your KB cards ?

I donno… how do I find that out?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on December 31, 2013, 10:58:08 PM
When you format card it should tell you
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on December 31, 2013, 11:15:05 PM
@g3ggo

used MLV dec 29 and ML dec 30.

I can achieve continuous ok MLV with Sound On with the ff resolutions;

1728 x 972 16:9

1856 x 928 2:1

they can record 15 min in a 64 gig 1000X KB card.
tested with clapper at regular intervals to test for audio sync. -No problems with sync.

1872 x 936 - starts to skip after 8 min.

tested FPS over ride with sound- I find this nice as it will work. MLV Sound is recorded in real time.
This can be used in certain situations.

I had a minor problem when the framing guides and other info turns OFF when recording and found out that

DISABLE G. DRAW needs to be ON to keep the frame guides and other overlays on Screen.

I suggest this be RENAMED as its description and name does the Opposite and got me misled.

Also, using certain screen layouts, the buffer graphics/elapsed time/frames/datarate info that comes on during record
clashes with the overlays such as vu meters, lens info framing guide etc.
Using 16:10 HDMI t/b layout scheme will help avoid this.

On using ext HDMI monitor.- I get short record times and coruppted frames with hdmi attached.

1728 x 972 GD.OFF + hdmi monitor = 28 sec only then skips and stops.
1856 x 928 GD.OFF + hdmi monitor = 35 sec only then skips and stops has corrupt frames.


Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on January 01, 2014, 03:42:12 AM
whats everyones Allocation unit size on there KB 64gb 1000 cf cards?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 01, 2014, 01:29:51 PM
@g3ggo

used MLV dec 29 and ML dec 30.

I can achieve continuous ok MLV with Sound On with the ff resolutions;
..

thanks for the feedback!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jason Montalvo on January 01, 2014, 03:15:58 PM
Finished a shoot yesterday and recorded using mlv_rec . Some of the takes were longer and over 4gb in size where ML split them into .m00 .m01 etc. Every time I open MLV browse sharp and navigate to the folder where the files are located it quits. Any pointers from anyone ?

I also noted that some of the clips are 4.29gb in size FYI card is exfat formatted.

Thanks !
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 01, 2014, 04:00:16 PM
Every time I open MLV browse sharp and navigate to the folder where the files are located it quits. Any pointers from anyone ?

wrong thread, go there:
http://www.magiclantern.fm/forum/index.php?topic=8447.0

and redownload the latest version and retry.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a.d. on January 01, 2014, 04:33:46 PM
Einen guten Rutsch!
I've just roughly fixed Issue #1773: global draw disable feature not reverts changes after stopping (https://bitbucket.org/hudson/magic-lantern/issue/1773/mlv_rec-global-draw-disable-feature-not)
Magic Lantern MLV (https://bitbucket.org/a_d_/magic-lantern-mlv/commits/all)

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 01, 2014, 04:58:26 PM
would commenting out line #1497 if (!PREVIEW_HACKED) return;
do the same?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a.d. on January 01, 2014, 05:14:34 PM
Confirm, it does the same thing!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 01, 2014, 06:43:32 PM
Quote
sure you use *my* release?

For dumping yes 100%, I can't compile the windows binaries. The best I can compile is MLV dump but still using yours from the pack.

So the functions are getting overrode by audio_configure (right before start asif) and no problems but have to check if anything has changed for 7D. 6D still isn't shutting down properly, stuck at state 4 (but message is shorter now). I tried with both shutdownin and stop asif. Maybe MLV dump needs to stop extracting if the wav is really done for 6D and others.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: reddeercity on January 02, 2014, 02:15:01 AM
Hi  :)
Just wanted to report my tests with MLV with Audio on the 5D2
I first have to Say Great Job Everyone! I all Cases audio was sync .
Now to the Test Numbers;
I did two different type of tests one with HDMI devices (Zacuto EVF & Atomos Ninja HDMI recorder)
And Without HDMI. Tested on Lexar 1000x 64GB.
Without > Largest Frame size Continuous record
1872x936 23.976p with Audio 19,931 frames/13:51 Min. 60GB 73.3MB/s 67° C.  :o  buffer method 2
Crop Mode > 2048x872 23.976p with Audio 19,930 Frames/13:50 Min 60GB about 74-75MB/s buffer method 1
With HDMI>Largest Frame size Continuous record,
1872x850 23.976p 2.20:1  with Audio 22,710/15:47 min 60GB 67.9MB/s buffer method 1
Crop Mode > 1856x938 23.976p with Audio 20,000 frames 68 MB/s buffer method 0
This on A.D. build for the 5D2 dec.30
Tried the nightly build from Jan-1 MLV, and could get larger frame size continuous with HDMI
1856x930 23.976p 2:1 19,000 frames 55GB buffer method 1
In all tests GD off and only the 5D2_212.sym, file_man.mo,  mlv_rec.mo, mlv_snd.mo
Crash 2 times with hdmi plugin when I tried to record 1856x1004 and or 1872x1012, the buffer just filled up with blue color then  lockup
Battery pull & reboot ok after that.
It seems on the 5D2 you are limited to about 70-73 MB/s without hdmi active
and with it active then that drops to about 65-68 MB/s & in crop mode to about 74MB/s normally that's about 76-79MB/s
and 75MB/s in non crop mode.
Its definitely usable without hdmi at this point, run & gun style  ;) 
 

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 02, 2014, 02:30:20 AM
The 4gb wav problem is from mlv_dump, its impossible to end it even from process manager.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 02, 2014, 02:31:53 AM
4 gib wave? how long did you record?!
and to which post did you refer?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Audionut on January 02, 2014, 02:32:35 AM
The 4gb wav problem is from mlv_dump, its impossible to end it even from process manager.

Hmmm, I think that is actually a Windows OS problem.

edit:  http://en.wikipedia.org/wiki/WAV#Limitations
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on January 02, 2014, 11:06:43 AM
finally with mlv sound i m able to make a film. thank u magic lantern so much. i was never interested in magic lantern raw since there was no audio recording. now i have made a film with mlv raw

http://www.youtube.com/watch?v=2hiFKQbQ6cc&feature=youtu.be
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 02, 2014, 05:23:44 PM
Quote
4 gib wave? how long did you record?!


Not long, actual data is a few MB. The problem is if you get an asif did not stop error in MLV sound the wav will end up being huge. So for 6D. 1st wav is fine, 2nd wav asif didn't stop and you get the giant wav. In MLVbrowse it hangs on the last frame or maybe 1 before it and keeps extracting until the 4gb limit is reached.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on January 02, 2014, 06:52:29 PM
This is driving me nuts..  :o  in rec.raw i can record fine 25gb+ raw file 5mins or more, no problem..
in rec.MLV i start getting skips when .m00 files start to be made.. tried alot of settings.

i love how multi-file support (4 GiB splitting is enforced) works but when it spilts for me it just starts to skip...

lower frame size does fix it.. lol
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on January 02, 2014, 09:34:39 PM
This is driving me nuts..  :o  in rec.raw i can record fine 25gb+ raw file 5mins or more, no problem..
in rec.MLV i start getting skips when .m00 files start to be made.. tried alot of settings.

I had some problems at first and realized that my mlv_sound.mo and mlv_rec.mo was getting updated but not re-compiled after the update was done.  First I used g3goo compile for 5D3 and then I  realized that my compiling machine was the cause.  I make sure I do a make clean and then make on the modules and check the date of the .mo to make sure it is the latest.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on January 02, 2014, 09:46:29 PM
Renatophoto can you send me your files mlv_rec & mlv_sound.. ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on January 02, 2014, 10:31:13 PM
I sent them to you but remember that MLV is slower.  For example I can record 1920x1080 at 29.97 fps on the CF card with raw_rec but with mlv_rec I have to enable card spanning to be able to record at this resolution with audio.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on January 02, 2014, 10:40:22 PM
I sent them to you but remember that MLV is slower.  For example I can record 1920x1080 at 29.97 fps on the CF card with raw_rec but with mlv_rec I have to enable card spanning to be able to record at this resolution with audio.

thanks

i guess we need faster cards..  8)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: RenatoPhoto on January 02, 2014, 10:58:02 PM
More processor speed would be better. 

@ g3g00: I noticed that the recording speed of the SD card is not showing and was wondering if it would be best to disable all unnecessary calculations to allow for the fastest recording speed?  For example these speed computations could be moved to the Show buffer graph option in case someone wants to look at it.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 02, 2014, 11:17:49 PM
I am shooting on a 5D3

I noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.

In addition to adding the mlv_snd module to the newest nightly build...

Do I need to copy/replace this autoexec.bin to the root directory of my SD card?

Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: michael84 on January 03, 2014, 12:27:06 AM
It seems that MLV.SND is finally working on the 7D. I just recorded 4 short Clips, all 4 MLVs extracted with MLVBrowseSharp in DNGs and WAV - and got 4 WAV files, each of them containing usable Sound.

My Settings:

1) For the first two files:
 
Sound Record: Normal
Beep, test tones: OFF
MLV Sound: ON
other Settings Default, Sound in Canon Menu on AUTO

2) Fot the other two files
Sound Record: Separate WAV
Analog Gain: 26 db
Rest unchanged

Tested with a quick DNG->JPEG convert in Lightroom and importing in Premiere, the Audio is 7 Frames longer than the Video at 25 fps (no FPS override, Canon Video set to 25p and PAL). If i Interpret the sequence as 24p, the Audio is 5 Frames too short.

Next things to test will be wether i can get continuous recording with MLV Sound on, and wether the audio stays in sync. I will shoot an analog clock with audible ticking for a Minute or two and use a flash at intervalls for easy sync checking and in case the ticking is not loud enough. This should also help to check the video timebase - perhaps i need to use fps override to get exact 25p.

Currently, the write performance lags a bit whena recording is started, display shows 200-300 Frames expected, but then catches up at up to 80 MB/s, the buffer graph goes down, and the writing Speed settles at about 70 MB/s. My card is a Transcend 64 GB, it definitely needs low Level formatting with autoformat, otherwise it really struggles to reach the necessary writing speed. buffer fill method is set at 4, extra hacks on. I did not test other settings, maybee i should to find out what works best.

Global draw and Digic Peaking are on, also the  Histogram.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 03, 2014, 12:37:16 AM
You'll gain a little speed with GD off. My continuous with audio is the largest 1x 16x9 size... the 5:3/3:5 are don't seem to be making it continuous + sound.

How is the alignment at 23.976? vs 24.00? Also harder to be continuous with external stereo vs the doubled internal mic. Sampling rate doesn't seem to be making a difference speed wise.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on January 03, 2014, 02:59:17 PM


Not long, actual data is a few MB. The problem is if you get an asif did not stop error in MLV sound the wav will end up being huge. So for 6D. 1st wav is fine, 2nd wav asif didn't stop and you get the giant wav. In MLVbrowse it hangs on the last frame or maybe 1 before it and keeps extracting until the 4gb limit is reached.

Experiencing the same problem with my 6d (MLV-dump.osx, so on OSX platform).
When I'm filming now, I turn my camera off and on after every single clip, just to have sound (raw video WITH sound, which is an huge improvement!)
Will there be a solution for the "sound failed to stop, state 4" error problem ?



Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 03, 2014, 05:50:26 PM
I hope so. There are only a couple of things to do:

1. Figure out the audio state machine and get rid of the patches
2. Stop asif properly between 6D and 7D
3. Make MLV_rec stop outputting 4gb wavs and ignore the error.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 03, 2014, 06:50:52 PM
3. Make MLV_rec stop outputting 4gb wavs and ignore the error.

4gig wavs are not possible, this is no fix for a bug.
although it should not hang at all.

i need such a .mlv or a full dump (-v) to check whats wrong.
with some vague "does hang" reports, i cannot fix anything.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 03, 2014, 10:33:52 PM
No problem, here is a log of a 2nd file... it hangs when it hits the first AUDF block and writes the wav till 4gb.

http://pastebin.com/henB4t1K

Properly ended 1st video.

http://pastebin.com/63PBASKd

From looking:
I think maybe the seconds are excessive and need to be written as ended even if audio is stopping ,etc.

I will also try to get rid of the message with SoundDevShutDownIn and SoundDevStopIn... 6D has those same functions.. on 7D it calls stopasifdmaADC (so it stops) and unpowers the internal mic, on 6D actrl functions so still state machine problems. all the beep.c wavs across cameras were endless like this and had to be re-saved... but the asif there is stopped or times out right away when you look at the DM log.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Embark on January 03, 2014, 11:31:17 PM
I am shooting on a 5D3

I noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.

In addition to adding the mlv_snd module to the newest nightly build...

Do I need to copy/replace this autoexec.bin to the root directory of my SD card?

Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?

Same question here. Also, will mlv_snd be an included module in the nightly builds anytime soon?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 01:10:08 AM
No problem, here is a log of a 2nd file... it hangs when it hits the first AUDF block and writes the wav till 4gb.

http://pastebin.com/henB4t1K

thanks.
nothing as simple as that.
see the log and you will notice there is one AUDF with frame #-1 (or 0xFFFFFFFF)

Block: AUDF
  Offset: 0x0cbb5a60
    Size: 38656
    Time: 18446744073709552.000000 ms
   Frame: #-1
   Space: 232



guess what this will cause?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ouuzi on January 04, 2014, 01:34:58 AM
Same question here. Also, will mlv_snd be an included module in the nightly builds anytime soon?
Same question here.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 01:49:05 AM
Do I need to copy/replace this autoexec.bin to the root directory of my SD card?
Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?

you don't have to, but its the latest version i tried with.
if there is any trouble with the nightly, use my version
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 01:52:13 AM
thanks.
nothing as simple as that.

should be fixed, can you try?
(re-record a new movie)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 03:29:20 AM
Better but wavs are coming out 1KB with these logs:

http://pastebin.com/KVTEhUwG

http://pastebin.com/4dL3FBSV

Ok, undid the fix 1/2 way and now they have sound + dump... that block never gets written for some reason otherwise.

http://nitrobits.com/file/1lvTdJN2EcZv0Og/magiclantern-Tragic.2014Jan03.6D113.zip
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 08:26:06 AM
Better but wavs are coming out 1KB with these logs:

that is absolutely correct as the mlv doesn't have any wave.
you said that the asif didn't shut down correctly.
so it doesn't start the second time and we get no audio....
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 05:23:06 PM
Yea but I fixed that, no more error message, correct # of frames with audio in the log but then 1KB wavs as the AUDF header was never written any time. Undoing 1/2 the overflow fix made it work and not produce 4gb wavs nor 1kb wavs.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: michael84 on January 04, 2014, 08:06:20 PM
Did some further tests with Sound recording on the 7D, this time with an external mic plugged in. So far, i can get continous recording at 1728x972 and 25 fps. When i reach 1 minute, the buffer suddenly begins to fill, and the recording stops at around 1:03 to 1:04 each time (no skip allowed). This coincides with reaching the 4 GB limit and file splitting. Either my card is too slow to catch up after the momentary writing speed drop at file split, or there is still the same bug others have been reporting.

Both with and without fps override (set at 25 fps exact) the timebase of the video is 100% correct compared to a clock, at least for the 1 minute+few seconds intervall i am able to record. I imported the files in Resolve lite, and set the project at 25 fps. No worries here.

The audio is perfectly in sync, both checked against the ticking of the clock an several flashes fired next to the mic throughout the recording. I got only one video with a 6 second longer audio file than the video, this was one i recorded until the camera stopped. The other files, where i stopped just before the buffer was full, i only get a few frames to much audio. Because the sync is ok, i guess the audio recording does not stop exactly simultaneous with the video. However, compared to the hassle external sound recording witout any in camera audio for easy matching in post, i can perfectly live with that little imperfection.

Now i need to develop a workflow, i guess i will try a conversion in ACR with the VisionLog Profile into DNxHD 10 bit, and grade the footage in AE or Premiere, perhaps using VisionLog LUTs. My machine is too slow to work directly with DNGs, and i want to use ACR for conversion, as Resolve Lite has no noise reduction and feels rather non intuitive to work with. Too bad that ACR has these flicker issues, it would be easy to get a nice look in it quickly, without having to use any LUTs later.

Next thing to test will be the 24p mode, although i probably wont use it here anyway. In PAL land, 25p is the way to go, i think, unless you plan to print your work to cine film.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 08:34:41 PM
Yea but I fixed that, no more error message, correct # of frames with audio in the log but then 1KB wavs as the AUDF header was never written any time. Undoing 1/2 the overflow fix made it work and not produce 4gb wavs nor 1kb wavs.

fixed that. you should now be able to pull my version.

@all:
updated the mlv_snd module and mlv_dump which had problems since the last change to allow long path names.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 08:49:52 PM
Either my card is too slow to catch up after the momentary writing speed drop at file split, or there is still the same bug others have been reporting.
yeah, i am aware of this. its due to the filesystem catalog being locked etc.
i am experimenting to work around this.

The other files, where i stopped just before the buffer was full, i only get a few frames to much audio. Because the sync is ok, i guess the audio recording does not stop exactly simultaneous with the video.
yes, thats the reason - audio and video cannot stop perfectly in sync.
well it could - then i would have to configure one audio buffer per frame duration. (40 ms @ 25 fps)
and this would cause a little more cpu load what i tried to prevent. (i use 200 ms buffers)

right now you have between 200 ms and 400 ms of additional audio data, which is 8 to 16 frames with 25fps.
one thing i could to is dropping the last audio frame, which would produce 200ms less audio data.
so we end up with 0 ms to 200ms of additional data. (0 to 8 frames)
but to be honest, i prefer having a bit too much data than too few ;)


hmm i have read reports that said their audio start is 2-3 frames off.
so you are sure that the audio is perfectly in sync on 7d?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 09:01:28 PM
I doubt any are in sync perfectly.... would be interesting to see a black frame + FFFF or something in the audio injected at the same time, begining + end.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 09:04:18 PM
why do you doubt?

we can not inject black and FF hardware-synced.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 09:33:41 PM
I doubt because all of the wavs I've had from any canon always seemed like they started at a similar time, not necessarily frame 0/1 and have been the wrong length... if slated though and sometimes stretched it would line up however. Interestingly too if one camera is recording wav and another records videos the files were much easier to sync and didn't seem to have any stretch....

I guess if its not hardware synced any kind of built in slate would be useless, probably why I had no luck going that route.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 10:08:14 PM
I doubt because all of the wavs I've had from any canon always seemed like they started at a similar time, not necessarily frame 0/1 and have been the wrong length... if slated though and sometimes stretched it would line up however. Interestingly too if one camera is recording wav and another records videos the files were much easier to sync and didn't seem to have any

the reason why audio is recorded longer than the video is, i already explained there (http://www.magiclantern.fm/forum/index.php?topic=7122.msg94443#msg94443)
this is no bug or problem, mlv_snd just recording 200-400ms longer. there is no need to stretch.

about the start time, i dont quite understand what you wanted to explain.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 11:03:49 PM
When I tried to inject frames to sync with and beep into the audio, it would start at the same time but it wasn't frame 0.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 11:06:22 PM
i still dont understand what you wanted to say, nevermind :)

well, at least the people report that wav is in sync.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 04, 2014, 11:11:39 PM
lol. just that the start is consistent but not at the exact begining of the video.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 04, 2014, 11:12:28 PM
here  (http://upload.g3gg0.de/pub_files/b10437c5bc8658866a51ef9a7dfb1e6c/mlv_rec.mo)is an experimental version that *could* be better regarding buffer drops on file switching.
can some of you test it?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 05, 2014, 01:36:03 AM
Recorded for 2 mins with the previous incarnation on 6D, didn't get a speed drop when the file split. Still split on an exfat card.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: michael84 on January 05, 2014, 05:05:16 AM
hmm i have read reports that said their audio start is 2-3 frames off.
so you are sure that the audio is perfectly in sync on 7d?

At least so far i can tell with my method. I fired an EX-430 remotely while recording, and looked frame by frame wether the Flash in the picture lined up with the discharge pop in the audio track. Every time it matched exactly. So unless the pop and the visual flash did not occur at the same time (will check this later with my HDR-FX1, just to be sure), the sync has to be ok, otherwise i would certainly be able to see it. Flash duration is only one frame. There is no sync drift over 1 minute recording.

However, what does happen is that the first few frames (2-3 could be just right) are not usable due to some kind of high level noise (waveform starts with a spike then drops to normal level). This is consistent through multiple recordings. So no offset, but some few frames of unusable audio. Note that this observation is only for 25p (selected in Canon menu /with and without 25 fps override selected, not 24p test yet), so for other modes there might be an offset/sync error.

I will try the experimental mlv.rec later. Amazing to see how much progress has been made recently, with updates rolling in on a daily basis. Keep up the good work!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Jakobmen on January 05, 2014, 06:54:36 AM
here  (http://upload.g3gg0.de/pub_files/b10437c5bc8658866a51ef9a7dfb1e6c/mlv_rec.mo)is an experimental version that *could* be better regarding buffer drops on file switching.
can some of you test it?

16:9
1920x1080 after 1min in it starts the buffer bars go crazy then it skips. once with GD on and off

1856x1944 samething like above.. once with GD on and off

1728x972 recorded 5mins+ fine.. with everything on

Thanks g3ggo
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: michael84 on January 06, 2014, 03:11:14 PM
Tested the new mlv.rec with my 7D, i can now record past 1 minute without frame drops. This time, sync was off by 2 frames, the audio was 2 frames before the picture, while the audio start (waveform peak) appeared to be 3-4 frames late. Now i will try again using the old mlv.rec, to see wether the sync issue is caused by the new mvl.rec, or just some inconsistency by the camera.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: budafilms on January 06, 2014, 06:13:59 PM
Help: How can open MLV file in Apple Macintosh?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Walter Schulz on January 06, 2014, 06:19:56 PM
http://www.magiclantern.fm/forum/index.php?topic=8447.0
Title: system out of memory exemption error
Post by: Steve Kahn on January 06, 2014, 09:53:15 PM
"Please contact g3ggo with this info:

Exception of type "System out of memory Exeption was thrown
at mlv_view_sharpVLVHandlerBlock(Sting type, mlv_rawi_har_t header,
Byte(} raw_data, Int32 raw_pos, Int32 raw_lwngth)



???


Also, on this same day shoot about half of the mlv files had no sound for some reason.  ???
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 06, 2014, 11:37:26 PM
7D, 1728x1036 + internal mic is continuous for me now. Nothing is happening when going over the 4gb hump. GD off though
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 01:16:31 AM
Sorry I haven't read this entire thread (planning to tonight) - Using the 5d3 + 1000CF KomputerBay + 32GB Sandisk I'm able to extract files fine and I get raw and audio.  My only question is this - when I set it to 1920x1080 in raw and have the MLV option enabled, I lose global draw (my sweet focus peaking is no more).  If I toggle the MLV recording off and restart the camera everything is fine.  Before you all yell at me, is there something simple I'm missing?
Thanks!
Shawn
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 07, 2014, 02:21:17 AM
if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu  and turn DISABLE G.DRAW to ON.

I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON.

Hope a developer can change this.  :) I already reported this in an earlier post #1093.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 07, 2014, 02:57:07 AM
I found this:

this
 uint32_t dmaFlags = 0x40001000;
to
   uint32_t dmaFlags = 0x20001000;

improved write speed on 7D+6D... couldn't find what the flags stand for in the wiki , the previous also caused edmac timeouts at certain 2.35:1 resolutions.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on January 07, 2014, 03:14:54 AM
Is the latest TL build working with sound?
Improved in that aspect ratio you mean?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 07, 2014, 03:25:26 AM
Yep the sound is working..

0x40001000 was what g3gg0 changed it to, I changed it back because of above problems, wondering
what the flags are about. maybe there is a better setting because there was a definite effect on speed
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on January 07, 2014, 08:39:16 AM
Yep the sound is working..

0x40001000 was what g3gg0 changed it to, I changed it back because of above problems, wondering
what the flags are about. maybe there is a better setting because there was a definite effect on speed
How much?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Fukurou on January 07, 2014, 08:47:22 AM
Sorry about the stupid question but I looked a bit and find nothing related, so here it is : how do yout extract sound when you film in the .mlv format ?
Just tried for the first time and I get a .mlv and a .idx. I can extract the vid with mlv2dng but no sound.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 07, 2014, 09:25:48 AM
You can convert with mlvviewer from g3gg0 http://www.magiclantern.fm/forum/index.php?topic=8447.0
or with dubzebass script here http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 12:46:16 PM
Sorry about the stupid question but I looked a bit and find nothing related, so here it is : how do yout extract sound when you film in the .mlv format ?
Just tried for the first time and I get a .mlv and a .idx. I can extract the vid with mlv2dng but no sound.

You did turn sound on too right?  It's not enough to just enable the module; you have to go to the first page and enable it too.  After I did this when I did the extraction to a raw file the .wav file appeared as well magically - meaning there's no special "switch" to extract audio.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 12:48:22 PM
if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu  and turn DISABLE G.DRAW to ON.

I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON.

Hope a developer can change this.  :) I already reported this in an earlier post #1093.

Thanks Ted - I'm sure that will be it.


On another note, I'm going to do some long recordings today with audio as well as have my Zoom H1n recording simultaneously mounted on the shoe.  Then I'm going to pull it into Vegas and run Pluraleyes - if I get a sync I'll be super happy - that's all I've wanted was a reference audio with my raw....

Will report back in.
Shawn
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 02:28:31 PM
Looks good - took an empty card and let it run until the end.  No problems and the audio synched up perfectly.  Now to just fine tune my settings.

Thanks guys for the wonderful update!  Shawn
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 08:23:17 PM
Quick question for the meta-data - I ran mlv-dump (-m -o) and successfully exported it to another MLV file that is meta-data only (no video) and I was able to read the information in MLV Browser.  However, if I try to read that metadata in notepad it's garbled.

How can I dump the following information into a text file?  Would be very handy for quick reference.
(http://www.goldcomics.com/incoming/ML.jpg)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 07, 2014, 08:33:05 PM
How can I dump the following information into a text file?  Would be very handy for quick reference.

mlv_dump -v
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 07, 2014, 08:56:44 PM
mlv_dump -v

Still not getting a file that is readable in notepad.  I'm going to setup a batch for exiftool to just grab the info from the dng's extracted from the MLV browser.  What would really be slick is an option in MLV browser to "Export selected files as .dng +.wav +.txt"...I realize you're busy, but wow, things are really close.  The audio sync is perfect now.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 07, 2014, 09:19:21 PM
Finding that if I use ATTR (half-shutter press) it disables the mlv_snd for some reason
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 07, 2014, 09:24:27 PM
Still not getting a file that is readable in notepad.

uhm the output of mlv_dump is readable, right? ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 07, 2014, 09:25:06 PM
Finding that if I use ATTR (half-shutter press) it disables the mlv_snd for some reason
it disables the module?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on January 07, 2014, 09:28:43 PM
It's most likely a conflict with the beeps (turn them off).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 07, 2014, 09:52:16 PM
It's most likely a conflict with the beeps (turn them off).

That was it!  With beeps on after you ATTR audio doesn't register or record. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 07, 2014, 09:59:22 PM
what do you mean with ATTR? ETTR?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 07, 2014, 10:25:48 PM
sorry not to be clear.

yes.  the ettr module (or Auto ETTR)  when I initiate it with a half shutter press (HalfS DBC) and if the BEEP is on it disables sound meters in mlv_snd and no sound is recorded.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: bnvm on January 07, 2014, 11:14:36 PM
Still not getting a file that is readable in notepad.  I'm going to setup a batch for exiftool to just grab the info from the dng's extracted from the MLV browser.  What would really be slick is an option in MLV browser to "Export selected files as .dng +.wav +.txt"...I realize you're busy, but wow, things are really close.  The audio sync is perfect now.

try mlv_dump -v file.mlv > file.txt
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 08, 2014, 12:41:43 AM
uhm the output of mlv_dump is readable, right? ;)

It's mostly gibberish with a few words that are relevant, and not formatted in any easy to use way. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 08, 2014, 12:48:15 AM
try mlv_dump -v file.mlv > file.txt

That works, but far too verbose.  Just looking for exposure, lens, camera info.  Don't need hundreds of lines that have this stuff:
Block: RTCI
  Offset: 0x000002a4
    Size: 44
    Time: 41.330000 ms
     Date:        07.01.2014
     Time:        11:21:22 (GMT+0)
     Zone:        ''
     Day of week: 2
     Day of year: 6
     Daylight s.: 0

Here's what I get by right clicking on my .dng file with my exiftool registry hack:

File Name                       : M07-1017.MLV.frame_000000.dng
Camera Model Name               : Canon EOS 5D Mark III
Lens Model                      : EF24-70mm f/2.8L II USM
F Number                        : 2.8
Shutter Speed                   : 1/40
ISO                             : 2000
Focal Length                    : 50.0 mm
Light Value                     : 4.0
Exposure Compensation           : 0
Frame Rate                      : 23.976
Date/Time Original              : 2014:01:07 10:17:38
Orientation                     : Horizontal (normal)
Image Size                      : 1920x1080
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 08, 2014, 12:51:23 AM
Quote
How much?

A bunch... like 3-5 MB/s and won't let you record certain sizes. Wonder if it had an effect on pink frames because I saw way less.

Quote
It's most likely a conflict with the beeps (turn them off).

Even with full audio controls beep turns off the input... sucks I don't think the IC can record and play at the same time. I blocked beeps while recording.

So with today's commits (pre-buffering):

1728x1152 is continuous with audio if using fill method 3. 4 won't do it for some reason.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Fukurou on January 08, 2014, 02:35:35 AM
Thanks Danne and Shield. Both of you were right and it works now.  :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ilguercio on January 08, 2014, 02:40:32 AM
A bunch... like 3-5 MB/s and won't let you record certain sizes. Wonder if it had an effect on pink frames because I saw way less.

Even with full audio controls beep turns off the input... sucks I don't think the IC can record and play at the same time. I blocked beeps while recording.

So with today's commits (pre-buffering):

1728x1152 is continuous with audio if using fill method 3. 4 won't do it for some reason.
This is for 7D right?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 08, 2014, 03:03:13 AM
yep, 7D.

Audio doesn't seem in sync though... on a 10 min clip the wav was a little short, not long. Another one was a few frames too long only. 6D was longer when I tested but I don't think 100% in sync.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 08, 2014, 03:11:51 AM
Just a heads up from my experience - I have been testing and diddling with this all day.  I'm the geek type that will change just one setting and exhaustively see what difference it makes - anyway, I was fooling around with the resolution settings for the 3x crop mode, and forgot that I set it above 1920x1080 - shooting normally it was fine, but when I went above that it would lock the camera up.
Not a big deal - pull the battery and start over.

But...inexplicably this lockup disabled audio recording in the Canon menu.  If you go off the wrong path as I did please be sure to verify the audio is still enabled.  Got some fantastic footage of the kids tonight, and the audio file (while there) was dead silent.  Went through all the menu settings and sure enough, the Canon menu had it disabled instead of the usual "auto".

Just wanted to share my experience.  I would like to thank all the developers and testers; the MLV format is amazing and the ability to have accurate, synch able reference audio is a huge benefit.  Thanks again guys - think how far you've come since last May.

Another side note - I let it record until the CF card filled up - it shut off "automagically" and everything was recoverable in post too.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 08, 2014, 06:49:59 AM
On 6D/7D it works fine with canon audio off as long as the IC is powered (wav mode), tap on the mic and if the meters move you're good

Quote
but when I went above that it would lock the camera up.

Maybe related to the dma flags. Which resolutions did it?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on January 08, 2014, 12:57:47 PM
i think so many people have audio sync proplem. why i dont have any problem in my 550d. i hve been shooting short films since mlv enable audio n i dont have any (repeat) any problem. everything works fine. audio sync is also very easy but i m so bad in color grading

http://www.youtube.com/watch?v=CVDQf91YmOM

atleast i shoot 50 clips a day and all audio sycn with video very well
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 08, 2014, 02:24:20 PM
On 6D/7D it works fine with canon audio off as long as the IC is powered (wav mode), tap on the mic and if the meters move you're good

Maybe related to the dma flags. Which resolutions did it?

Oh something stupid like 2.5k - I was just bored and curious to see how long I could get it to go.  I'll be sure to keep my eye on the meters.
The problem is the Magic Lantern software is too smart - it remembered that in 3x mode I was higher than 1920x1080.  I'm used to software being much more stupid.  I also periodically get a "Error -1" at the top, then I realized it was set to write to the SD card (Card 2).  You guys have thought of just about everything; well done!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on January 08, 2014, 02:29:06 PM
Sound was probably disabled by FPS override (and it couldn't be restored because of abnormal shutdown). It was important when recording H.264, but with raw it no longer needs to be disabled.

It gets pretty complex to handle all these tricks.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 08, 2014, 02:58:19 PM
right...
so mlv_snd should enable audio again when it is enabled.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on January 08, 2014, 02:59:46 PM
I guess this will result in a fight between the two codes.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 08, 2014, 05:33:14 PM
Should just add audio to 5DIII, its AK4646 with slightly different command method.

I should also try audio on/audio off and see what happens to sync.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 08, 2014, 11:30:42 PM
any idea of when you will include mlv_snd into the nightly builds?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: thehadgi on January 09, 2014, 03:09:40 AM
Subbing. Thanks once again :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 09, 2014, 01:01:20 PM
Hey everyone, hoping for a bit of help here, I'm really interested in trying out the new MLV raw with audio for the 5d mk III but looking for a bit of help with installing the files. 
I know there's a few others also having difficulty figuring out how to get the latest MLV developments working on their 5d Mk III's as I've seen step by step instructions for the mkII,
would anyone who's gone through the process be able to give us a step by step guide to installing these latest developments.  I've got the October ML with Raw working at the moment,
and I've downloaded the modules kindly provided by G3ggo, do I just drag all the .mo files into the modules folder I currently have? Or is that totally wrong.  Huge apologies in advance for asking but hopefully someone with a bit more ML experience out there won't mind walking me through it.  Thanks guys!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 09, 2014, 01:30:25 PM
Just to clarify, I presume I'm overwriting all the .mo files from the first post here into the 5d mk iii latest build (from 9th Jan)? If so, where does the 5d3_113.sym file go? MLV dump is the post processing program (ie. not for the SD card in the camera) is that about right? Sorry for the stupidly basic questions :-)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 09, 2014, 01:51:32 PM
I think you can download the regular nightly build for your camera here http://builds.magiclantern.fm/#/
Then you just add the mlv_rec.mo and mlv_snd.mo from g3gg0 build from the first page in this thread. Conflict occurs if you load both raw and mlv recording in camera. At least this is what I have done lately.
Alternative is to download and install g3gg0 build removing the other nighly build.

Correct me if I,m wrong here
//D
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 09, 2014, 04:33:03 PM
I think you can download the regular nightly build for your camera here http://builds.magiclantern.fm/#/
Then you just add the mlv_rec.mo and mlv_snd.mo from g3gg0 build from the first page in this thread. Conflict occurs if you load both raw and mlv recording in camera. At least this is what I have done lately.
Alternative is to download and install g3gg0 build removing the other nighly build.

Correct me if I,m wrong here
//D

That's pretty much what I did too.  Seemed to work out fine for me.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 09, 2014, 04:40:48 PM
Hey Danne,
Thanks for that info.  Great I seem to have it all up and running.  My only issue is that I have MLV audio running, but aren't seeing any options for audio under the audio tab.  I'm seeing audio bars though on the main screen, but in the menus there's just three options, speaker volume, beep test and MLV sound - is this normal? on the 5d 2 mlv i know i've seen a screen with more audio options. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 09, 2014, 05:34:48 PM
enable "MLV Sound" and enter the menu using Q, if you want to see the options.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 09, 2014, 07:15:55 PM
Hey g3ggo!
Thanks for the help :)
I do have mlv_snd enabled in the modules, in the audio tab I have mlv sound switched to ON, when I press Q on MLV sound I only get three options, debug, trace output and sampling rate.  I wasn't sure whether I should be expecting to see digital and analog output, headphone level etc as per the usual magic lantern or not? Is this right? Thanks again for all your hardwork, it's MASSIVELY appreciated!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 09, 2014, 07:48:18 PM
no there are no further options to mlv_snd. it is just recording, usually from the internal mic.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 09, 2014, 07:51:24 PM
Thanks for your help guys :-) I'm up and running :-) I'll let you know if I come up against any problems :-)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: HrRaiko on January 10, 2014, 03:56:59 AM
Sry but i cant still understand how to get mlv files to dng and that audio file too?

Edit:
Ok, found it. Bad in english so its hard to find that things.
(http://www.magiclantern.fm/forum/index.php?topic=7802.0 and http://www.magiclantern.fm/forum/index.php?topic=8447.0)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 10, 2014, 08:09:57 AM
also here http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Sthirasukha on January 10, 2014, 09:58:46 AM
Here I'm again on this topic g3gg0. I installed your build and removed the other nighly build.
Now I can see the mlv_snd module but when I record (the global draw is ON), the global draw turn to off on the LV and when I stop recording I have a "error -1" on the top left. Did I miss something?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Sthirasukha on January 10, 2014, 10:11:18 AM
I found an answer that resolve my problem:
As Ted Ramasola said:
"if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu  and turn DISABLE G.DRAW to ON. I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON."
Now it seems to work fine. Thanks again!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: gravitatemediagroup on January 11, 2014, 02:59:35 AM
why isn't mlv snd included in the nightly builds?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 11, 2014, 04:45:40 AM
Okay a little bit of my experiences from testing today.  I tested the mlv with sound, both plugged into a monitor and not on a 5d mk iii with 1050x 128gb cards.  In the older RAW module I can record constant 1920x1080 @ 25fps.  In this mlv module the camera only runs for a few hundred frames then cuts out, I have to record at a lower resolution 1728x972, and sometimes even then it isn't continuous.  The sound however is perfectly in sync.  The sound cuts out a second or so before the cam starts to skip frames.  The other issue I noticed is that when connecting a monitor sometimes the monitor screen freezes up, when pressing record when this happens you see nothing but a frozen external monitor screen, but when processing the file you get a strobe type effect where you're seeing a constant flicker between the frozen screen and the thing you're recording. 
The biggest issue I have is it not being able to run 1920x1080 @ 25fps.  Any advice on settings that people recommend for getting the best bang for the buck on the 5d 3?
Hope the above info is useful to someone? :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: bluewater on January 11, 2014, 09:53:09 AM
Hi, I'm wondering there's way to convert MLVtoDNG+WAV in one commnad file by Drag and Drop.
I know MLV2DNG can do it. But not convert WAV file from MLV.

Anybody know about the development?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 11, 2014, 10:13:59 AM
this works http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: bluewater on January 11, 2014, 10:49:10 AM
Thanx Danne ;D

I use windows, though. ㅜ.ㅜ
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 11, 2014, 12:26:28 PM
Then use g3gg0 viewer/browser to convert http://www.magiclantern.fm/forum/index.php?topic=8447.0
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: timbytheriver on January 11, 2014, 03:48:44 PM
I have a Camera model ID error showing up in ACR 8.3/CC/OSX 10.7.5 using the most recent mlv modules.

I've been discussing it in this post http://www.magiclantern.fm/forum/index.php?topic=9731.25 but was advised by Danne to post it here also.

Dngs created are coming into ACR with the ID/tag: 'Camera Raw 8.3 - ERR:1 md:0x   0 ml:0' and all I get offered [in ACR Camera profiles] is the embedded profile. I'd like to apply VisionLOG here.

Exiftool on the dng shows me the line:

Unique Camera Model             : ERR:1 md:0x       0 ml:0

All other extraction methods on my normal raw > dng produce usable camera ID such as 'Canon 5D Mk 2' or 'Canikon'. I've updated to latest mlv_dump also.

Any help gladly received. Many thanks.

tim
5D MK2
a.d. build 24.10.2012 with added mlv modules.
OSX


UPDATE: This has now been fixed by me doing a clean install of ML on the CF card. Not sure where the problem was… :o
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: crazypig on January 11, 2014, 07:48:08 PM
Dear g3gg0,

MLV is GREAT~~~~~ I am start using it now, very stable.

Just one request...

Can you make the recording information smaller like raw_rec.mo

Because I would like to use MLV for my new project.

Best regards,

Sean
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 11, 2014, 08:00:12 PM
Adding in the raw rec stuff like the little green camera, etc is in the works. MLV_REC just got to the point of being a viable replacement.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: naturalsound on January 11, 2014, 11:12:15 PM
The biggest issue I have is it not being able to run 1920x1080 @ 25fps.  Any advice on settings that people recommend for getting the best bang for the buck on the 5d 3?
Hope the above info is useful to someone? :)

I struggled with se same problem. Back then i was able to record contineously and suddenly only one in a while a cont. recording worked.
Yesterday I realized that I changed the audio sample rate to 48 kHz recently. I switched back to 44.1kHz and viola:

With 44.1kHz audio sampling rate MLV is contineous for me, while with 48kHz it skips after about 100-500frames.
Build: Jan-11 5DIII 128GB KB
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 11, 2014, 11:28:02 PM
You should try some of the other buffering methods too. 4 isn't always the best.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 11, 2014, 11:56:36 PM
And also the small size jpg settings in picture quality
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on January 12, 2014, 12:07:04 AM
And also the small size jpg settings in picture quality
I believe this is not the case with 5DM3...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 12, 2014, 12:08:02 AM
Try it, It makes a difference
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on January 12, 2014, 12:28:24 AM
Which one of the 4 (on 5DM3) ?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 12, 2014, 12:47:13 AM
I use 3 and 4 on 7D and 6D. on 4 the 7D wouldn't record 1728x1152 continuously with audio, on 3 it did. Seemed to blast through the 1038 though.

So try each and see because I don't have a 5DIII.

5DIII + 6D also support the memory hack which should give you all of the memory that is possible. The hack only works when pic quality is raw. One of the jpegs may give you slightly more memory than raw + nothing but not too likely on these 2 bodies. Every camera has a different trick for maxing out the buffer size.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 12, 2014, 05:28:50 PM
Hey everyone,
Thanks for the advice ill test that tomorrow.  On another test last night I noticed that the connection of my smallhd ac7 via hdmi was causing regular issues, both in RAW rec and MLV, it certainly seemed to be slowing things down and locking things up.  It's more stable in RAW only creating slight issues but totally unusable in MLV, every time I hit record the screen would lock up and I'd have to turn the cam on and off to reset. 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: gravitatemediagroup on January 12, 2014, 06:22:08 PM
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess.  Shouldn't be "this" difficult right? 
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: andy kh on January 12, 2014, 07:19:57 PM
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess.  Shouldn't be "this" difficult right? 

use MLVBrowsesharp to convert mlv raw into dng + audio or raw + audio
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: reddeercity on January 12, 2014, 09:54:05 PM
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess.  Shouldn't be "this" difficult right?
I did a short tutorial for Windows with mlv+audio but on a 5d2 it should help.
Link below
https://vimeo.com/83449475
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: gravitatemediagroup on January 13, 2014, 08:40:57 AM
delete
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 13, 2014, 09:15:31 AM
I was doing some tests on sound sync and I loaded SND.mo and saw the meters come on upon checking my footage I was surprised not to see a WAV file,
I checked my cam and saw I missed turning MLV sound to ON in ML menu, - I see potential for trouble here when using this in a shoot.

Can devs tweak this to put safety nets?

Like, If MLV sound is not ON then Vu meters should not be active?

Or, you should not hear anything on your headphones when MLV sound is OFF.

or, when SND module is loaded, this should automatically turn ON MLV Sound by Default.

There is just too many "buttons to push" that the probability for error is high.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Sthirasukha on January 13, 2014, 09:36:59 AM
I just tried to use the whole MLV (shoot and dev) this week end. I have to say that everything works quite well! Hudge thanks to the whole team!!!!
I have a 5D MKIII with MLV_rec, 25 fps, 1920x1080, continuous recoding ok, CF card lexar 64Gb x1000 (+SD card), sound sync without any issue.
The MLVBrowseSharp.app works well too (except the preview), specially when the file is 10 Gb (split in .mlv (4.29Gb), .m00(4.29Gb), m01(1.8Gb) in the CF Card but seen as one file in MLVBrowseSharp.app) and the wav file is created as one file that is sync to the dng folder once put together in AE (with ACR)! Really great job.
The MLVViewer in the 5D is fantastic, too. My only asking (and I know it has been asked already) is: it woud be nice to have a smaller information area in the LV when recording as it was for the Raw_rec module...
Many many thanks again!!!! Your an amazing team!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on January 13, 2014, 11:17:16 AM
Can devs tweak this to put safety nets?

Like, If MLV sound is not ON then Vu meters should not be active?

Yes. In audio-common.c, one has to modify SOUND_RECORDING_ENABLED (turn it into a function first) and make it aware of mlv_sound.

I'm not sure whether sound in Canon menu has to be enabled for mlv_sound, and whether this behavior is consistent on all cameras.

This kind of integration is tricky, a very large part of the audio code has to be rewritten.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: gravitatemediagroup on January 13, 2014, 04:36:59 PM
in my ML audio settings I only have 3 menu items, but In the tutorial video I seen tons of other options/extra options that I don't see?  Maybe these are settings that got placed or built into other settings in newer builds?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: daisermac on January 13, 2014, 04:39:10 PM
Hey everyone,
Thanks for the advice ill test that tomorrow.  On another test last night I noticed that the connection of my smallhd ac7 via hdmi was causing regular issues, both in RAW rec and MLV, it certainly seemed to be slowing things down and locking things up.  It's more stable in RAW only creating slight issues but totally unusable in MLV, every time I hit record the screen would lock up and I'd have to turn the cam on and off to reset.
I am using an EVF and it locks up from time to time too or gives me corrupt frames. Too bad, as my shots are so much more stable with this extra-point of contact. I wished the devs could invest some love into this, but I fear with an external monitor, we are a small minority.

Most of my shots turn out very usable. You have to make sure to switch off the camera first and wait for five seconds before switching off the EVF.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: MRozier on January 13, 2014, 08:04:44 PM
Hi there
Would anyone getting continuous shooting at 25fps at 1920x1080 on the 5d mkIII mind sharing their settings, I'm getting that no problem with raw_rec but not managing to get anywhere close with mlv.
It'd be good to know if others have had a bit of success with those settings.  I'm not card spanning at the moment as I'm recording to CF cards and keeping magic lantern on the SD.  Any other tips or advice from MLV users? Thanks in advance :)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 13, 2014, 09:34:18 PM
I got myself a transcend 128gb 1000x card today and it actually is a little bit faster than my 64gb 1000x transcend.
On my 5d mark 3 I get continuous with audio on following

1920x1080 24p
I can actually use canon gui and set manual audio while recording. Cool!
1920x1080 25p
Continuous without audio. With audio on it,s actually continuous until it splits the 4gb chunk. Here it,s kin of borderline. Sometimes it continues sometimes it stops, skips frames.

*update. Audio in auto mode gives me full continuous 1080x1920 25p *tried again and audio auto stopped around 10gb. No luck in 25p and audio continuous

For 24p I don,t use extra hacks on for audio meter in canon gui. G.draw off but I get it continous with global draw on with focus peak on as well in 24p. Also use pictue quality set to s2
for 25p I use extra hacks on g.draw off

Tried the modified mlv.rec.mo but made no difference when 4gb split.



Thanks g3gg0
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 14, 2014, 12:42:06 AM
Quote
I'm not sure whether sound in Canon menu has to be enabled for mlv_sound

If you have audio controls and power the IC no, if you don't yes. Otherwise its muted/missing.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Sthirasukha on January 15, 2014, 12:14:37 AM
To MRozier
As I'm able to have a continuous recording with 25fps at 1920x1080 on the 5d mkIII, the settings that works for me are:
In MLV-Rec -press(Q)- Aspect ratio 16:9 / Disable G. Draw ON / Frame skipping OFF/ Preview Auto/ Start delay.... to Memory hack OFF / Extra Hacks ON / debug trace OFF (if this debug is ON, sometimes I can't have continuous recording) / the others OFF / Buffer  3 / CF buffers 3 or 4
For the sound, don't forget to turn off Beep, test tones
In Canon menu, sound is on manuel (I have no issue with sound for the moment even with 10 GB file)
As Danne already said, use only the g3gg0 builds (don't try to mix it with other module)
I use a CF card Lexar x1000 64GB (for mlv files) and SD SanDisk 95MB/s 32GB (for ML Firmware)
I don't know if it could help or not as I have no special settings indeed.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: budafilms on January 16, 2014, 06:10:26 AM
To MRozier
As I'm able to have a continuous recording with 25fps at 1920x1080 on the 5d mkIII, the settings that works for me are:
In MLV-Rec -press(Q)- Aspect ratio 16:9 / Disable G. Draw ON / Frame skipping OFF/ Preview Auto/ Start delay.... to Memory hack OFF / Extra Hacks ON / debug trace OFF (if this debug is ON, sometimes I can't have continuous recording) / the others OFF / Buffer  3 / CF buffers 3 or 4
For the sound, don't forget to turn off Beep, test tones
In Canon menu, sound is on manuel (I have no issue with sound for the moment even with 10 GB file)
As Danne already said, use only the g3gg0 builds (don't try to mix it with other module)
I use a CF card Lexar x1000 64GB (for mlv files) and SD SanDisk 95MB/s 32GB (for ML Firmware)
I don't know if it could help or not as I have no special settings indeed.

How can you get audio? (wich build are you using, and the settings... thank you).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: arturochu on January 16, 2014, 06:59:33 AM
can the sound recording option work with old raw rec?

it would be great for me because its way easier to process the old format than the new one. (and you can record a little faster)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Audionut on January 16, 2014, 07:08:24 AM
How can you get audio? (wich build are you using, and the settings... thank you).

You just quoted the settings being used  ???

Reading the OP is probably a good step to getting started.

can the sound recording option work with old raw rec?

AFAIK, if raw_rec is kept after mlv_rec becomes more stable, it will only be in a stripped down maximum speed version.  In other words, don't expect audio.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 16, 2014, 08:26:24 AM
While we, re on the maximum speed subject. I, ve always been curious if the spanning feature applied to raw_rec would give that final extra mb to be able to record close to continuous in 50fps in full hd  :P
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 16, 2014, 12:13:30 PM
Testing the last 2 MLV nightlies including one from a.d. for the mkII, is it a current limitation for MLV raw with sound to stop at a little past 10 minutes?

I notice the buffer graphs seem to be managing it until past 10 minutes when it would say skip and recording would stop. tested different buffer methods at 1728 x 972 16:9.

35.9 gig seem to be the limit in my tests (40859MB on the camera lcd )

Tested another resolution, 1728 x 864 and it went past 11 min but again it stops at 40918MB as indicated on the LV screen.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: arturochu on January 16, 2014, 03:28:10 PM
While we, re on the maximum speed subject. I, ve always been curious if the spanning feature applied to raw_rec would give that final extra mb to be able to record close to continuous in 50fps in full hd  :P

i dont think 50fps in fullHD can be posible, but that would be awesome!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: jose_ugs on January 16, 2014, 04:12:14 PM
50fps is very possible, just not continuous ;)
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 16, 2014, 06:05:47 PM
I'm suprised you guys are getting stops with the sound, I've gotten none. If it gets continuous, it stays.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 16, 2014, 08:06:16 PM
What camera and build are you using? 25fps full hd?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Sthirasukha on January 17, 2014, 12:19:31 AM
To Budafilms:
I'm using the g3gg0 build (mlv_rec) from the Jan 10th 2014 (my birthday... nice gift g3gg0 and thanks again!!!)
I tried to put the mlv_snd.mo in the "classical" module folder (with raw_rec.mo, Arkanoid....) but it crashed (as Danne said)
So, my module folder has exactly (no raw_rec included):
5D3_113.sym
autoexpo.mo
dot_tune.mo
file_man.mo
mlv_play.mo
mlv_rec.mo
mlv_snd.mo
pic_view.mo
silent.mo
smpte.mo
trace.mo

To (amazing) developers:
I hope you won't stop the sound module. I'have being waiting this for such a long long time! Pleaaaaase!!! Don't go back. And tell me there are a lot of people that are able to use the sound.

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 17, 2014, 01:16:28 AM
Heh, I've tried this on most cameras now. Even EOSM works for a while... afterwards tho it hangs :( At least I got an error message this time and it said it dumped the log. Not the fault of mlv_snd, mainly the camera.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 17, 2014, 08:54:56 AM
Ok, so here's my tests in terms of filling up a 64gig card.
All tests run at 1728 x 972 16:9 all tests run with MLV SOUND ON.

5D mkII tests

Jan 16 nightly
   skip/stop at 10'7"  40,800MB (41 GB)

Jan 14 by a.d.
   skips/stops approx 10 min 40,859MB

Jan 8 nightly
   skip/stop at 10'6"  40,721 (41.4 GB)

Jan 6 nightly - MLV_REC + Jan 14 a.d. -MLV_SND
   OK CONTINUOUS FILLS UP CARD.

Jan 6 a.d. all modules.
  OK CONTINUOUS FILLS UP CARD.

Jan 6 a.d -MLV_REC + Jan 14 a.d. - MLV_SND, FILE_MAN, MLV_PLAY
   OK CONTINUOUS FILLS UP CARD.

7D tests.

Jan 16 TL
   skips/stop at 10'11" 40,929MB (39.9 GB)

Jan 14 TL
   skips/stops 10'10"  40,907MB (39.9GB)

Jan 9 TL
   skips/stops 10'11" 40,929MB (39.9GB)

Jan 8 TL (incomplete audio features in Audio tab ML menu - did not run test)

Jan 7 TL
   OK CONTINUOUS FILLS UP CARD.

Jan 6 TL
   OK CONTINUOUS FILLS UP CARD.



So I guess something was wrong starting from Jan 8 builds?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 17, 2014, 10:11:13 AM
you say from jan 8th the recording is continuous and that is a bug?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 17, 2014, 10:20:12 AM
you say from jan 8th the recording is continuous and that is a bug?
I wrote starting with the 8th there must be something wrong.
I listed the tests from recent to older.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 17, 2014, 03:14:14 PM
Tried the build from first post here and it gives me 40gb file than stops. Gonna try nightly build in a while.
5d mark 3
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 17, 2014, 04:32:02 PM
I wrote starting with the 8th there must be something wrong.
I listed the tests from recent to older.

ah okay thanks!
now i understand.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: budafilms on January 19, 2014, 04:12:42 AM
I can't fin the module for sound. Someone can place the link? thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 19, 2014, 05:00:08 AM
I can't fin the module for sound. Someone can place the link? thanks!

First post. If your using Tragic lantern, its in the TL NBs. A.D also included it in his builds for the 5D2.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 19, 2014, 08:13:09 AM
The pre allocation commit was the 7th. I am seeing it stop early too, no message or anything.  It would have ended up in the nightly on the 8th.

I'm only getting like 20 - 25gb of 1728x1152 now.  Skips with no warning.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 19, 2014, 09:42:36 AM
Yeah, already nailed down.
there are only a few file handlers that are possible being left open.
on 5d3 its ten files max.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: EuAcelasi on January 19, 2014, 12:36:16 PM
I tried replacing the files of the nightly builds with the files of the mlv folder, but it doesn`t work... Could anybody, please, tell me what is the exact way to make it run... I used the 19 Jan 2014 build and then tried to add the files with replacing the existing... and tried to delete all and then placing the mlv files...  Thanks.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on January 19, 2014, 09:07:20 PM
Playing with the tragic lantern builds for 6d form 3 and 18 Januari.
Sound recording issues with mlv_rec are fixed, I can record multiple clips with sound now, THANKS !!!
BUT
Is it me or is the sound recording volume really low (compared to previous builds), I have to crank up the volume on my mac to hear the wave files.
In Canon menu I have the wind filter and Attenuator OFF, so that's not it.
In ML menu I tried analog gain on 23 db and on 32 db, not much different.
digital gain is all off.

Also the meters from ML aren't moving that much, in canon menu, the sound meters are moving a lot further on the scale...

What's happening here ?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 19, 2014, 09:16:42 PM
The audio wasn't touched, just mlv_snd. Canon always uses a little digital gain with it. If you add a tiny bit just to 1 channel it will get hot pretty fast.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Levas on January 19, 2014, 10:26:48 PM
Probably need to do some more testing with it, and check audio settings on the previous builds I used, maybe there was some digital gain there.
(or maybe I touched some of the many many settings in the ML or Canon menu)

Digital gain in camera has no benefits on digital gain in post on computer (I guess ?), and with digital gain in camera there's more risk of "clipping" audio, so probably better to leave it off.




Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 19, 2014, 11:18:07 PM
I was under the impression audio was recorded through canon audio menu settings? At least it is when I change audio levels manually while fiming mlv raw.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: feureau on January 20, 2014, 07:21:49 PM
I've been looking for an mlv to dng batch converter for windows but there doesn't seem to be any.  Found the mlv2dng.exe but it's not batchable. Did I miss a thread or nobody hasn't build one yet?

Thanks
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: gary2013 on January 20, 2014, 07:27:21 PM
I've been looking for an mlv to dng batch converter for windows but there doesn't seem to be any.  Found the mlv2dng.exe but it's not batchable. Did I miss a thread or nobody hasn't build one yet?

Thanks
I use the MLVBrowserSharp.exe to do batch MLV extractions.

Gary
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 20, 2014, 07:48:08 PM
I use the MLVBrowserSharp.exe to do batch MLV extractions.

Gary

What is your workflow?  I have the same issues.  I have to manually create a DNG folder for each MLV file.  Is there an easier way to batch several MLV files? 

Thanks!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 20, 2014, 07:59:51 PM
Steve,

If you use resolve after converting MLV to DNGs, just let the MLVbrowsersharp extract all your MLVs to one folder as resolve can automatically see them as belonging to one clip when you point resolve to the folder containing all the dngs. At first I thought I needed them to be in separate folders but I got use to it later on. I guess it's more organized if they are in separate folders though but its more convenient that theyre not in folders so resolve can show you all the Thumbnails of your clips in one window rather than when they are in individual folders.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 20, 2014, 08:32:30 PM
Quote
I was under the impression audio was recorded through canon audio menu settings?

It configures audio, asif is what records it. Canon stuff runs when the menu is accessed and record start, etc. After that ML configures the settings you've set.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: feureau on January 20, 2014, 10:51:20 PM
I use the MLVBrowserSharp.exe to do batch MLV extractions.

Gary

Thank you \o/
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Canon eos m on January 21, 2014, 02:12:15 AM
I have a big issue. On MLV mode, the camera skips frames (even with skip frames option off). There are about 3-4 pink frames in the whole video of about 250 frames. The video is choppy. Video shot at 1920X1080 (Canon 25fps)

Workflow: MLV Browser; After Effects CC.

Other programs: VLC;Windows 8 on PC.

Equipment: Canon 5D3 (Sigma 35mm F1.4), Lexar CF 1000X 32GB; Scandisk SD 95kpbs Extreme Series.

I am using the 20 January 2014 build (no separate SND files imported from g3gg0's module on first page).

What am I doing wrong?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 21, 2014, 02:25:00 AM
Global draw on or off? Raw_rec?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Canon eos m on January 21, 2014, 04:42:37 AM
Rec_rec off. Global draw on.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 21, 2014, 04:48:30 AM
Global draw can cause pink frames.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Canon eos m on January 21, 2014, 06:11:14 AM
Ok.

Will try w/o.

Another thing - how can I prevent jerky video in post. 25fps looks jerky on vlc which it shouldn't. Format QuickTime.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 1% on January 21, 2014, 07:54:01 AM
depends how you encoded it, ae defaults to 30P for image sequences, etc
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Steve Kahn on January 21, 2014, 08:05:04 PM
Steve,

If you use resolve after converting MLV to DNGs, just let the MLVbrowsersharp extract all your MLVs to one folder as resolve can automatically see them as belonging to one clip when you point resolve to the folder containing all the dngs. At first I thought I needed them to be in separate folders but I got use to it later on. I guess it's more organized if they are in separate folders though but its more convenient that theyre not in folders so resolve can show you all the Thumbnails of your clips in one window rather than when they are in individual folders.

Thanks Ted.  This works well for a few mlv files but if I have many mlv files MLVbrowsersharp throw a lot of excemption errors and ultimately crashes.   

For a few files though Resolve does indeed import them as individual files even though all of the DNGs are in the same folder.  After Effects imports the DNGs in a similar way as Resolve.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: EuAcelasi on January 21, 2014, 08:59:17 PM
Is there any kind of rawanizer to batch convert either into raw or dng creating subfolders for each shot? I tried to convert first with BrowseSharp to RAW, and then import them in RAWanizer, but it crashes... Also, the MLV.RAW.WAV extension make RAWanizer treat it as a RAW file...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 22, 2014, 05:46:43 AM
Started a new thread on this, but perhaps it just belongs here.

Recording typical 1920x1080 video tonight; nothing too crazy.  For some reason ML locked up, which is no big deal, but after the restart the Canon menu's audio became disabled so I lost some good audio tonight.

I normally notice the audio meters but tonight I didn't see them.

Here's my question - back in the day in order to use the 3x mode one had to set "FPS override" (in my case I set it to 23.976).  However if you do this now, one cannot see the audio meters at the top.  I have the "Raw Histobar" at the top.  Once I disable the FPS override the audio meters come back.

So, do I still need to do the FPS override trick?  All I care about shooting is 1920x1080 @ 24FPS in normal mode as well as 3x crop/zoom mode.  In the past it was recommended to set the Canon menu for 30 FPS then dial down the FPS override to 23.976.

Also, if it's still required, is there any way to get the audio meters back with it on?  When shooting handheld stuff I just set the Canon audio menu to "auto"..

Thanks guys.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: looyod on January 22, 2014, 11:56:16 AM
Hi,

we've been using ML a lot during a large (and important) production, it has been amazing and we've been able to get footage that would otherwise not have been possible. big thanks here to everybody who contributed. this is such a game changer!

we're a swiss production company shooting with two cameras running a build from 07.jan with the snd module from 4th jan. we convert using the mlv dump on os x with the script by dubzeebass (also thanks for that) We shot about 4 TB of footage. All documentary style with sound.

two problems occured.

1) out of about 200 shots about two had bad first frames. I could not open the first dng file in photoshop and davinci would therefore not recognise the sequence.

I saw one thing, instead of the normal vertical stripes correction:

Vertical stripes correction:
  1.000  1.000  1.001  0.994  1.003  0.998  1.001  1.003

MLV Dump gave me those:

Vertical stripes correction:
  1.000  1.000    1      1      1      1      1      1

I tried running the script on different locations but it seems to be an issue with the MLV file.

dublicating the second frame solves the issue with the sequence of course... but the problem is still a bit weired.


2) In about 3 shots out of over 400 we had the issue of skipped frames, they are just missing. The Sound is fine.
I can not pinpoint any logic behind it. The main Problem is, that the camera won't stop and we're therefore not aware of the issue on the shoot.

I could provide the MLV Files for analysis if it would help. Non of the files with a problem was spanned)
(but I don't want to post them publicly on the forum)

We're using the Sandisk 160MB/S  128GB Cards.


Thanks!

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 22, 2014, 12:03:50 PM
thanks for the feedback!
very nice to see it is actually being used and people are statisfied.

i am sorry that there are dropped frames :(
sure i will have a look into this - maybe i can find something that helps in tracking down the bug.
can you upload the files to a google drive?
of course i will not disclose any content.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: bnvm on January 22, 2014, 09:41:26 PM
2) In about 3 shots out of over 400 we had the issue of skipped frames, they are just missing. The Sound is fine.

I have ran into this a few times my self, seems totally random and rather rare. I am using a 5D2, I have also had some issues where the camera will not stop recording when the button is pushed. Going into the menu shows stopping... but never does (Well is does stop recording frames but never returns to normal function camera must be turned off and on). This issue is just as rare as the other.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: 2020 on January 23, 2014, 11:23:24 AM
I'm new to this so this might sound stupid to most but I am having a issue with just getting the new nightly build on my 5d3 so I can use the MLV, I dragged and drop on my SD card and load modules but I am getting a error. How do I fix this or install the new build,  I'm on a Mac and I am using a Canon 5d3 that has the ML Raw on there already..
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: shinokubo on January 25, 2014, 11:42:12 PM
Hello to everyone.

I'd like to tell you that we shot a shotfilm in .mlv the last two days. We were shooting 1080p25 with Global Draw on and mlv_snd on a couple of Komputerbay 64GB 1000x Cards. We did produce around 400GB of footage.

We had almost no problems, only had to warm up cards everytime we switched oherwise we had some frameskipping.
Also the remaining free space on the card was incorrect for some reason.

So far I did convert all the .mlv files to .dng+wav with the batch script on osx and now I am trying to figure out how to produce proxies with audio in a proper manner.

Thanks so far!
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 26, 2014, 12:32:49 AM
You could use this MLVbrowsesharp to convert your mlv to raw: http://www.magiclantern.fm/forum/index.php?topic=8447.0

Then use RAWanizer to export proxies and the DNGs. http://www.magiclantern.fm/forum/index.php?topic=5557.0



Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 26, 2014, 03:30:21 AM
Tested the Jan 25 Nightly on the 5DmkII.

With MOVIE RESTART=ON MLV raw video will not stop recording. If you press SET to stop, it will start by itself.
This issue is not present with older builds like the Jan 6th NB and by a.d.
This is suppose to be active in H264 recording and should not have an effect in Raw recording.

HDMI MONITORING TEST
Tested playback functionality.
Immediately after recording a raw clip, I went to File manager> view mlv file.
Playback of COLOR or FAST was possible and it showed on HDMI monitor.
However after EXIT of mlv playback screen went Black and no function possible.
Had to remove battery to reboot camera.
On Older build using Jan 6 nB or A.D. Playback was possible and EXIT from playback was OK,
liveview came back and additional recording was possible.

CAPACITY BEYOND 40GIG TEST
Jan 25 Nightly build will still skip and stop approx, 39GB (Canon menu), 35.9GB (PC properties, 38,582,760,448 bytes)
The Jan 6 NB and by A.D. does not have this issue and will record till your CF card is full.
NBs after Jan 6 up to the 25th will Stop at approx that size or around 10minutes. This also applies to the 7D.


...
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Shield on January 26, 2014, 05:45:38 AM
Out today shooting some 2-4 min clips.  One clip was 3:16 and the associated .wav file was 3:07.  It just didn't record the last 10 seconds for some reason.  I was able to sync it up with my external source so I was able to determine that was the problem.  Clips just before and after this were fine, and there was no lockup or any other problems.
Would anyone like me to upload the mlv file somewhere?
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 26, 2014, 06:55:20 AM
Out today shooting some 2-4 min clips.  One clip was 3:16 and the associated .wav file was 3:07.  It just didn't record the last 10 seconds for some reason.  I was able to sync it up with my external source so I was able to determine that was the problem.  Clips just before and after this were fine, and there was no lockup or any other problems.
Would anyone like me to upload the mlv file somewhere?

on my tests using 7D and 5DmkII using various resolutions the results are always as follows.

when recording MLV+SND, the wav file would always be
LONGER by 7-10 frames.
It would be ahead by 1-2 frames from the video so it needs to be nudged back to sync.

Knowing this and you won't have problems syncing in post. H264 was also off by 1-2 frames so this is not an ML issue.

However, if your recording stops ABRUPTLY, due to card full for instance, the VIDEO would be LONGER and the WAV file much shorter.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: g3gg0 on January 26, 2014, 10:22:33 AM
However, if your recording stops ABRUPTLY, due to card full for instance, the VIDEO would be LONGER and the WAV file much shorter.

yeah thanks for the feedback.when card gets full, I cannot clear buffers anymore.
then you lose some megabytes of audio.
not good, right. But I don't know a quick solution.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Audionut on January 26, 2014, 10:27:25 AM
Can you monitor the file size and end recording before card full, giving a chance to clear the buffers?

Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: a1ex on January 26, 2014, 10:30:28 AM
With raw_rec, I create a backup.raw file for this. Whenever the card gets full, I delete that backup file and try to save the buffers again.

Last time I checked (months ago) it was working quite well.

@Audionut: this requires keeping track of all file writes since the last time Canon code reported free space. Possible, but requires some major refactoring (maybe a file i/o backend that wraps around Canon calls in a way similar to memory backend).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Danne on January 26, 2014, 12:15:03 PM
I filled my 64gb card three weeks ago and with Alex wizardry (I think backupsolution) I managed to extract my files. There was even a messaged asking if my card was full :).
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: Canon eos m on January 27, 2014, 01:26:25 AM
Posted here as advised by Ted:

Ted, also to add if the raw video is taken when the sd card is the preferred card, the 5D3 hangs. This is technically a glitch.

While I understand you can set your preference and all but I am sure there is a programming way to prevent the sd card from leading when it clearly is not meant to.

My take is that it will discourage mass adoption of the ML platform small things like these are not addressed. Pretty much like texting changed for ever when predictive texting was introduced.
Title: Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
Post by: ted ramasola on January 27, 2014, 02:01:33 AM
To add my test on the 7D, testing Jan 25 TL.

MLV Raw+Snd

It still skips and stops at approx 10 min or 40,929 MB ( 39.9GB )
Even though it recording constantly at 77.3/80/76 MB/s 9%idle.

The last Tragic Lantern version that does not stop at 10 minutes and will fill up card is Jan 7.
(This probably coincides with the Jan 6 a.d build for the 5d2-last build with no skip/stop problems)

after that up to jan 25 will skip and stop at 10min or 40gig.

Also to add an observation which I hope will help, I reverted back to Jan 7 TL and did a capacity test again.
I observed that the LV info keeps on updating constantly and the CF card activity led light keeps flashing but upon reaching approx 10min
or 40gig which is the threshold for other builds before stopping, I notice every info stop updating and the CF activity led stops flashing, like the cam froze for a few seconds,
but after maybe 2 and 1/2 seconds the activity and led flashing continuous and so does recording.
Title: Re: Magic Lantern (R