Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[chmee]

thanks for info. will play with that as well.

3 days ago, ffmpeg released a new major version 2.3. This release supports mlv-demuxing. I´m waiting for this version to find the way to the repos of my OpenSuse (packman).  I don´t know, what is possible with this new feature in ffmpeg, but I will find it out

Edgar

[chmee]

found a block named "BKUP". whats that? should be mentioned in the google-docs on 1st thread-entry.

[g3gg0]

this one was a temporary solution and should not happen anymore.
just ignore that block (as any unknown block)

[chmee]

ok, found it in a todays recording. forgotten right now.

[g3gg0]

this shouldnt be in mlv files for months now

[ayshih]

@g3gg0
I've been trying to improve the performance of exact playback of MLV files (on my 50D), which struggles on >~1 GB files due to out-of-sequence blocks.  Canon FIO has poor performance with reading backwards, with increasing delays the further into a file (presumably it's loaded from the beginning).  Consequently, increasingly more frames need to be skipped the further past >~1 GB, and eventually, around 3 GB, it's no longer possible to read block headers fast enough to catch up with the FPS timer and playback stalls.

The issue is that mlv_play reads the header of every block during playback, and all of those FIO_ReadFile calls start becoming too expensive.  I've tried some sub-optimal solutions – skipping over blocks, but that skips metadata blocks too, or flushing the timer queue when too far behind, but then playback isn't exact anymore – but I think the "right" solution would be to add information to the XREF block in the generated IDX file so that it's possible to know whether a block can be skipped (VIDF) in playback without needing to read the block header (again).

Here's my implementation: https://bitbucket.org/ayshih/magic-lantern-50d/branch/mlv_play_fix3
https://bitbucket.org/hudson/magic-lantern/branch/mlv_play

I split off a byte from the unused variable empty in the structure mlv_xref_t, and this new frameType variable is set to 1 if the block is VIDF and thus skippable.  This changes the format specification, but I'm hoping the ramifications will be minor because the structure variable was unused and the size of the structure hasn't changed.  (Existing on-camera IDX files would need to be re-generated.)  Playback starts to drag deep into a 4 GB file, but now playback completes and maintains >2 FPS.

What do you think?  This frameType variable could be extended to discriminate all of the different block types, if useful.

[g3gg0]

yep, like that change
just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible

[stevefal]

Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?

[ayshih]

just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible

Are you recommending an actual enum?  I'm not sure how that would work since I can't include an enum as part of the structure due to size, and it'd be clunky as a separate global.  I'll use preprocessor constants to remove the magic numbers, and then I'll issue the pull request.

Edit: pull request

[g3gg0]

Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?

16 bit hardware limit as far as i can tell

[AWPStar]

Hi!

First of all, i am sorry for my bad english.
I have an idea about the noise reduction in mlv_dump or raw2dng.
1 Write the sample of video without a light.
2 Averages the noise(frames). Subtract noise by 2 or more frames
3 Subtract from the normal video.
It mostly constant noise.
It is convenient to do during unpacking, right in the DNG.
You can add a key to the command line.

I can not do this, because I don't know C. I understand the sources, but i can't change it
I did this in Visual Basic, but with 16-bit Tiff files  It's not comfortable, and I could not to convert them back into DNG 
I think it would be useful

This link can be usefull.
http://www.photo.net/learn/dark_noise/

Thank you for your atention!

[g3gg0]

this is not related to the MLV file format. you can do this in post with any tool.

[AWPStar]

this is not related to the MLV file format. you can do this in post with any tool.

it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.

[dmilligan]

You can already do this with mlv_dump (see the -s option), and like g3gg0 says, it's not related to the thread topic which is about the MLV file format itself, NOT about how to best post process those files (there are many other threads for that: http://www.magiclantern.fm/forum/index.php?board=54.0)

[g3gg0]

it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.

before raw: modifying analog or optical path => hardware mod, not ML
after raw: DNG files => can be done in postprocessing
special case: modifying ADC parameters like black level or CMOS registers => not related to your request.

and as dmilligan said:
not here, please (mlv_dump -a averages frames,  mld_dump -s subtracts a reference frame)

[Levas]

Is there a way to do this option:
MLV_dump -a                  average all frames in <inputfile> and output a single-frame MLV from it.
When you already got rid of the MLV file and only have some dng's left ?

I'd like to average out a lot of DNG's for dark frame usage, but I don't have the original MLV files anymore.
Any chance to average out some dng's with mlv_dump or maybe other software 
MacUser...

[g3gg0]

unfortunately not easily possible without rewriting it totally
you could export DNG to some other raw format and use e.g. octave to average

[Audionut]

You can use ImageJ (free), probably darktable (also free). or even photoshop.

[dmilligan]

Also DeepSkyStacker

[Levas]

Ah  thanks for the other software options.

Just taking a look at some high iso clips I shot for fun a while ago, at the same time I also did some clips with lens cap on for dark frames.
Trying to figure out what is possible with the extreme iso's.

[kfprod]

Regarding MLV large file support is there a reason you want to have all variables within mlv_rec.c? I'm thinking it would be better to set a global variable in bootflags.c where the system actually knows if it is an exFAT card- line 171 in bootflags.c ((p.type == 7) // ExFAT)

Right now mlv_rec.c just knows what camera models have support for exFAT but doesn't check.

Code: [Select]

    /* not all models support exFAT filesystem */
    uint32_t exFAT = 1;
    if(cam_5d2 || cam_50d || cam_7d)
    {
        exFAT = 0;
        large_file_support = 0;
    }
I've updated the long unique exFAT file names that I've been using and will upload it soon for you to consider. File name examples are: 20140927_102314_ n4yl.MLV and if you've enabled camera number and take number: 20140927_102314_cam1_take3_7pbi.MLV (The last 4 characters are just randomized "0123456789abcdefghijklmnopqrstuvwxyz"). I know some people are shooting multi camera and have problems when they get the same file name.

[mothaibaphoto]

I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv  -v -e -c -b 16 -l 9 -x big.mlv
and I get about 4 GB file, good.
Than, i try to extract dng from that file:
mlv_dump.exe --dng --no-cs compress.mlv  -o
 I get about 2 GB of dngs and mlv_dump.exe crashes having about 2 GB memory allocated according to Task manager. My PC is 64 bit with 24 GB memory.

[diogo.fagundes]

The file MLV_REC.TMP is there only to reserve space, so I don't need it to make my .dng files neither to proxy them. So I can delete before decompressing the raw files. Am I right?

[dmilligan]

I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv  -v -e -c -b 16 -l 9 -x big.mlv

You might want to try this: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92

[mothaibaphoto]

@dmilligan: Thanks, looks interesting, but... the main thing I can understand from comments  - this is something to compress in camera (as long, as it looks like modifications of mlv_rec module). But I actually want to find a best way to backup video. I can't believe, that compressing is possible in camera without a big overhead. Anyway, I can't try this till someone will make compiled modules available.