Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

[dmilligan]

It's not for in camera compression, that's impossible. mlv_dump shares some code with mlv_rec and they live in the same directory, that doesn't mean these changes are to support in-camera compression.

This new compression method is much faster than the current one. I implemented it in MLVFS and getting pretty close to real-time playback.

It shouldn't be that hard to compile using mingw or whatever. You don't have to setup the whole ARM toolchain to compile it.

[KMikhail]

Howdy!
First off - great work, much appreciated.

So, what I have is an image of CF card, after in-camera recording failure. Unfortunately it was a new year celebration, crossing the 00:00 mark, so I'd really like to recover it. Right after the failure I took the CF card out, to prevent overwriting. Thus, this exact image contains fragments of all files recorded earlier and not yet overwritten.

The goal: scan it for all possible mlv file headers, then scan for all frame blocks following definitions of found mlv files, then save all of them sorted according to time stamp into corresponding mlv files (with option of filling gaps - if frame is missing, repeat the previous one with correct TS, option of inserting bright/loud dummy frame to make sure I will notice it, option of not inserting anything but what was found). Would be nice to make a check for blocks (i.e. partial / erroneous should be skipped).

Surely, some of them will not workout well, but my interest is primarily focused on the recovery of the last one. I checked definition of C structs, and probably can make a parser, but it should be definitely much easier and faster to get done by Author.

Is this possible/could be made?

Thank you,
Mikhail

[g3gg0]

FIRST: make an image of the CF card using some disk imaging tools (there are some open source tools)

some weeks ago i chatted with the developer of PhotoRec who was so kind to add MLV support.
so: get photorec, it will do its best to recover your footage.

[moodlover]

If one didnt need sound, should we just stick to .raw instead?

[g3gg0]

if your desire is higher speeds, yes. thats the big advantage of .raw.
if you want metadata with your footage, choose MLV.

i personally do not need high speeds but prefer the files to contain all possible information.

it might be interesting to convert .raw->.mlv as .mlv seems to reach a broad tool suite.
so people who need to work around the mlv-deficit "writing speed" can still post process and archive it as .mlv

[Danne]

+10 for raw to mlv

[KMikhail]

Yes, I of course did make image - to secure data and free-up the CF card for usage. Will think over, Thanks!

[DeafEyeJedi]

+10 for raw to mlv

Agreed!

[jose_ugs]

Wasn't there already a tool for that, RAW2MLV?

[Levas]

Does somebody know if mlv_dump can average all images in an Full-res-silent-picture MLV ?
I can't get the -a, option to work on full res silent MLV ?

This is what I type in terminal
mlv_dump -a /Users/wij/Documents/TEST/72160000.MLV

This is the output:
MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Verify file structure
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Reached end of chunk 1/1 after 205 blocks
Processed 50 video frames
Floating point exception: 8

[g3gg0]

uhm i the number of averaged frames is exactly zero.
thats why there is an exception.

reason: you forgot to add an output file

[Levas]

He G3gg0,

Tried to put an output file in the command line...I get a segmentation fault 11 and a zero byte sized averaged.mlv

mlv_dump -o averaged.mlv -a /Users/wij/Documents/TEST/72160000.MLV

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Rewrite MLV
   - Output only one frame with averaged pixel values
   - Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Segmentation fault: 11


EDIT:
The reason I'm trying this, is because I want to have a long exposure picture in daylight without needing ND filters.
Nice landscape shots, with soft tree leaves and soft clouds 

[DeafEyeJedi]

@g3gg0:

Just wanted to provide feedbacks regarding 70D footage. Apparently either the MLV_Dump isn't responding to 70D files nor there aren't proper color matrixes ( it thinks its a 6D when it isn't? lol ) It looks like it may be lacking either one of these....

Here are the 70D111B files in case you need to dig through them...

Short Clips:
https://mega.co.nz/#F!HtUx0CgS!jQslzxzJTC19VVwGGN75lA

Varies of Files including Dual-ISO:
https://mega.co.nz/#F!yoNQWBTa!3ohyAu9UggiyreAiFjo8nA

I've also posted some videos of what the footage looks like via Vimeo:
https://vimeo.com/126099527

Hope this helps!

*UPDATE*

After busting an all nighter with an fellow ML buddy (Special Thanks to @Danne!) he finally made it possible for me to get my converted DNG's from 70D's Dual-ISO CR2's to work again (Previously wouldn't be able to read from PS) with the help of a '70D matrix command' provided by @Danne. (Hope it's OK w me sharing this link - if not let me know!)

Another good news is the fact that now I am able to convert both MLV & RAW files with Full color matrice 6D/70D by using @Danne's latest cr2hdr-r which works well on Mac's.

Will still need to investigate further on 70D111B Dual-ISO MLV/RAW though...

[g3gg0]

@Levas:
can you run mlv_dump with parameter -v and upload the dump?
or can you put the MLV file in my btsync folder with key AUUKDJIEUOPLOPFN5NECA2NEVCLRHPHFO?

@DeafEyeJedi:
mlv_dump didnt produce clean DNG files?

[DeafEyeJedi]

Not for Dual-ISO MLV/RAW files but for regular MLV/RAW files it works fine from a 70D111B w latest nightly.

[Danne]

Did you put up a dual iso sample in the folder? I could check later.

[DeafEyeJedi]

Yes, I believe I did.

[Danne]

Deafeye. Could you retest a non corrupted dual iso mlv/raw and extract with mlv_dump, raw2dng? Or upload a new sample file, dual iso from 70D cam. Again, check that it isn,t any corruption issue.

[DeafEyeJedi]

What are exactly the corruption issues do you recall from 70D dual ISO files? Because all of them comes out the same weird colors w flickering effect.

I'll do another new dual ISO MLV/RAW tonight for you...

[Danne]

Thanks.
Downloaded the file M01-1819.MLV and the file itself shows corruption. It is not mlv_dump that is the problem. Maybe share a link to the file here or in the 70D thread for the experts to look at?

[Levas]

@Geggo

Used the -v option, here's the output:

mlv_dump -o averaged.mlv -a -v /Users/wij/Documents/TEST/72160000.MLV

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Verbose messages
   - Rewrite MLV
   - Output only one frame with averaged pixel values
   - Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 10242427998058437803
    FPS         : 0.500000
    File        : 0 / 0
    Frames Video: 50
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 500.314000 ms
    Res:  5568x3723
    raw_info:
      api_version      0x00000001
      height           3723
      width            5568
      pitch            9744
      frame_size       0x02298AB0
      bits_per_pixel   14
      black_level      2047
      white_level      14783
      active_area.y1   52
      active_area.x1   72
      active_area.y2   3723
      active_area.x2   5568
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 503.485000 ms
     Camera Name:   'Canon EOS 6D'
     Camera Serial: '1354ACF6DB'
     Camera Model:  0x80000302
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 503.574000 ms
     Mode:   2
     Kelvin:   2500
     Gain R:   496
     Gain G:   1024
     Gain B:   624
     Shift GM:   -1
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 503.585000 ms
     picStyle:   132
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: RTCI
  Offset: 0x0000019c
    Size: 44
    Time: 513.114000 ms
     Date:        27.04.2015
     Time:        14:33:55 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 116
     Daylight s.: 0
Block: EXPO
  Offset: 0x000001c8
    Size: 40
    Time: 513.761000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    445000 microseconds (1/2.25)
Block: LENS
  Offset: 0x000001f0
    Size: 96
    Time: 513.785000 ms
     Name:        'EF35mm f/2 IS USM'
     Serial:      ''
     Focal Len:   35 mm
     Focus Dist:  65535 mm
     Aperture:    f/11.30
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x000000F9
     Flags:       0x00000000
Block: VIDF
  Offset: 0x00000250
    Size: 36276944
    Time: 525.316000 ms
   Frame: #0000
    Crop: 0x0
     Pan: 0x0
   Space: 0
Segmentation fault: 11

[DeafEyeJedi]

@g3gg0:

Here's the link to 70D files including Dual-ISO MLV/RAW from 70D111B (April 23)...

https://mega.co.nz/#F!6hchkCbD!zScVqL68khdbY6MylpzhNg

[g3gg0]

@levas:
fixed. try the latest version:

@DeafEyeJedi:
i think i dont understand the problem.
so mlv_dump creates buggy matrices for dual iso mlv files?

[DeafEyeJedi]

Regular MLV/RAW seems responding to mlv_dump well when using @Danne's cr2hdr-r latest beta but then when I record Dual-ISO RAW/MLV videos then it shows all kinds of strange pinkish effects as you can see in these files I uploaded.

MLV_dump won't respond to those files maybe because it's already corrupted from inside camera while recording?

Just thought it was worth noting... That's all.

However, Dual-ISO CR2's from 70D seems to respond well but just not the videos.

Just wanted to share that with you since I wasn't sure of @A1ex hasn't updated this binary yet for the 70D files.

[Levas]

@g3gg0

Averaging with the newest MLV_dump works!

Multiple Full-res-silent-pictures -> Recording in MLV container -> averaging out to one single photo with MLV_dump = The most extreme and variable ND filters one can have 

Edit: oh yeah I forgot something, THANKS