Author Topic: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)  (Read 1072769 times)

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1900 on: July 24, 2014, 11:29:30 PM »
thanks for info. will play with that as well.
3 days ago, ffmpeg released a new major version 2.3. This release supports mlv-demuxing. I´m waiting for this version to find the way to the repos of my OpenSuse (packman).  I don´t know, what is possible with this new feature in ffmpeg, but I will find it out :)

Edgar

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1901 on: August 06, 2014, 10:07:18 PM »
found a block named "BKUP". whats that? should be mentioned in the google-docs on 1st thread-entry.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1902 on: August 06, 2014, 11:16:25 PM »
this one was a temporary solution and should not happen anymore.
just ignore that block (as any unknown block)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1903 on: August 06, 2014, 11:54:02 PM »
:) ok, found it in a todays recording. forgotten right now.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1904 on: August 07, 2014, 12:22:14 AM »
this shouldnt be in mlv files for months now :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

ayshih

  • Contributor
  • Senior
  • *****
  • Posts: 266
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1905 on: August 07, 2014, 01:27:47 AM »
@g3gg0
I've been trying to improve the performance of exact playback of MLV files (on my 50D), which struggles on >~1 GB files due to out-of-sequence blocks.  Canon FIO has poor performance with reading backwards, with increasing delays the further into a file (presumably it's loaded from the beginning).  Consequently, increasingly more frames need to be skipped the further past >~1 GB, and eventually, around 3 GB, it's no longer possible to read block headers fast enough to catch up with the FPS timer and playback stalls.

The issue is that mlv_play reads the header of every block during playback, and all of those FIO_ReadFile calls start becoming too expensive.  I've tried some sub-optimal solutions – skipping over blocks, but that skips metadata blocks too, or flushing the timer queue when too far behind, but then playback isn't exact anymore – but I think the "right" solution would be to add information to the XREF block in the generated IDX file so that it's possible to know whether a block can be skipped (VIDF) in playback without needing to read the block header (again).

Here's my implementation: https://bitbucket.org/ayshih/magic-lantern-50d/branch/mlv_play_fix3
https://bitbucket.org/hudson/magic-lantern/branch/mlv_play

I split off a byte from the unused variable empty in the structure mlv_xref_t, and this new frameType variable is set to 1 if the block is VIDF and thus skippable.  This changes the format specification, but I'm hoping the ramifications will be minor because the structure variable was unused and the size of the structure hasn't changed.  (Existing on-camera IDX files would need to be re-generated.)  Playback starts to drag deep into a 4 GB file, but now playback completes and maintains >2 FPS.

What do you think?  This frameType variable could be extended to discriminate all of the different block types, if useful.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1906 on: August 07, 2014, 06:27:19 PM »
yep, like that change :)
just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

stevefal

  • Contributor
  • Hero Member
  • *****
  • Posts: 528
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1907 on: August 07, 2014, 10:23:29 PM »
Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?
Steve Falcon

ayshih

  • Contributor
  • Senior
  • *****
  • Posts: 266
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1908 on: August 08, 2014, 05:23:54 PM »
just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible
Are you recommending an actual enum?  I'm not sure how that would work since I can't include an enum as part of the structure due to size, and it'd be clunky as a separate global.  I'll use preprocessor constants to remove the magic numbers, and then I'll issue the pull request.

Edit: pull request
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1909 on: August 09, 2014, 03:03:25 PM »
Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?

16 bit hardware limit as far as i can tell
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

AWPStar

  • Senior
  • ****
  • Posts: 411
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1910 on: August 30, 2014, 07:27:50 AM »
Hi!

First of all, i am sorry for my bad english.
I have an idea about the noise reduction in mlv_dump or raw2dng.
1 Write the sample of video without a light.
2 Averages the noise(frames). Subtract noise by 2 or more frames
3 Subtract from the normal video.
It mostly constant noise.
It is convenient to do during unpacking, right in the DNG.
You can add a key to the command line.

I can not do this, because I don't know C. I understand the sources, but i can't change it ::)
I did this in Visual Basic, but with 16-bit Tiff files  :) It's not comfortable, and I could not to convert them back into DNG  :(
I think it would be useful

This link can be usefull.
http://www.photo.net/learn/dark_noise/

Thank you for your atention!
MLVProducer. p.s. sorry for my bad english.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1911 on: August 30, 2014, 11:09:04 AM »
this is not related to the MLV file format. you can do this in post with any tool.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

AWPStar

  • Senior
  • ****
  • Posts: 411
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1912 on: August 30, 2014, 02:58:50 PM »
this is not related to the MLV file format. you can do this in post with any tool.
it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.
MLVProducer. p.s. sorry for my bad english.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1913 on: August 30, 2014, 11:57:17 PM »
You can already do this with mlv_dump (see the -s option), and like g3gg0 says, it's not related to the thread topic which is about the MLV file format itself, NOT about how to best post process those files (there are many other threads for that: http://www.magiclantern.fm/forum/index.php?board=54.0)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1914 on: August 31, 2014, 12:48:49 PM »
it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.

before raw: modifying analog or optical path => hardware mod, not ML
after raw: DNG files => can be done in postprocessing
special case: modifying ADC parameters like black level or CMOS registers => not related to your request.

and as dmilligan said:
not here, please (mlv_dump -a averages frames,  mld_dump -s subtracts a reference frame)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1634
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1915 on: September 16, 2014, 12:49:06 AM »
Is there a way to do this option:
MLV_dump -a                  average all frames in <inputfile> and output a single-frame MLV from it.
When you already got rid of the MLV file and only have some dng's left ?

I'd like to average out a lot of DNG's for dark frame usage, but I don't have the original MLV files anymore.
Any chance to average out some dng's with mlv_dump or maybe other software  :-\
MacUser...

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3155
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1916 on: September 16, 2014, 01:46:02 AM »
unfortunately not easily possible without rewriting it totally :(
you could export DNG to some other raw format and use e.g. octave to average
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Audionut

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3606
  • Blunt and to the point
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1917 on: September 16, 2014, 03:29:20 AM »
You can use ImageJ (free), probably darktable (also free). or even photoshop.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1918 on: September 16, 2014, 03:47:27 AM »

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1634
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1919 on: September 16, 2014, 11:06:56 AM »
Ah  :) thanks for the other software options.

Just taking a look at some high iso clips I shot for fun a while ago, at the same time I also did some clips with lens cap on for dark frames.
Trying to figure out what is possible with the extreme iso's.

kfprod

  • New to the forum
  • *
  • Posts: 39
  • 5d Mk3 1.2.3
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1920 on: September 27, 2014, 01:14:21 AM »
Regarding MLV large file support is there a reason you want to have all variables within mlv_rec.c? I'm thinking it would be better to set a global variable in bootflags.c where the system actually knows if it is an exFAT card- line 171 in bootflags.c ((p.type == 7) // ExFAT)

Right now mlv_rec.c just knows what camera models have support for exFAT but doesn't check.

Code: [Select]
    /* not all models support exFAT filesystem */
    uint32_t exFAT = 1;
    if(cam_5d2 || cam_50d || cam_7d)
    {
        exFAT = 0;
        large_file_support = 0;
    }

I've updated the long unique exFAT file names that I've been using and will upload it soon for you to consider. File name examples are: 20140927_102314_ n4yl.MLV and if you've enabled camera number and take number: 20140927_102314_cam1_take3_7pbi.MLV (The last 4 characters are just randomized "0123456789abcdefghijklmnopqrstuvwxyz"). I know some people are shooting multi camera and have problems when they get the same file name.

mothaibaphoto

  • Senior
  • ****
  • Posts: 392
  • pesky kid
Memory leak possible in mlv_dump.exe extracting dng from compressed MLV
« Reply #1921 on: February 25, 2015, 06:55:05 AM »
I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv  -v -e -c -b 16 -l 9 -x big.mlv
and I get about 4 GB file, good.
Than, i try to extract dng from that file:
mlv_dump.exe --dng --no-cs compress.mlv  -o
 I get about 2 GB of dngs and mlv_dump.exe crashes having about 2 GB memory allocated according to Task manager. My PC is 64 bit with 24 GB memory.

diogo.fagundes

  • New to the forum
  • *
  • Posts: 16
  • 5D2 and 5D3
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1922 on: February 26, 2015, 10:44:45 PM »
The file MLV_REC.TMP is there only to reserve space, so I don't need it to make my .dng files neither to proxy them. So I can delete before decompressing the raw files. Am I right?
Premiere cs6, LightRoom 4, LR Time-Lapse 3, Mac OSX 10.8.5

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1923 on: February 26, 2015, 10:54:40 PM »
I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv  -v -e -c -b 16 -l 9 -x big.mlv
You might want to try this: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92

mothaibaphoto

  • Senior
  • ****
  • Posts: 392
  • pesky kid
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1924 on: February 27, 2015, 07:16:04 AM »
@dmilligan: Thanks, looks interesting, but... the main thing I can understand from comments  - this is something to compress in camera (as long, as it looks like modifications of mlv_rec module). But I actually want to find a best way to backup video. I can't believe, that compressing is possible in camera without a big overhead. Anyway, I can't try this till someone will make compiled modules available.