Author Topic: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)  (Read 1101913 times)

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1925 on: February 27, 2015, 03:42:46 PM »
It's not for in camera compression, that's impossible. mlv_dump shares some code with mlv_rec and they live in the same directory, that doesn't mean these changes are to support in-camera compression.

This new compression method is much faster than the current one. I implemented it in MLVFS and getting pretty close to real-time playback.

It shouldn't be that hard to compile using mingw or whatever. You don't have to setup the whole ARM toolchain to compile it.

KMikhail

  • Freshman
  • **
  • Posts: 75
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1926 on: March 25, 2015, 11:25:24 PM »
Howdy!
First off - great work, much appreciated.

So, what I have is an image of CF card, after in-camera recording failure. Unfortunately it was a new year celebration, crossing the 00:00 mark, so I'd really like to recover it. Right after the failure I took the CF card out, to prevent overwriting. Thus, this exact image contains fragments of all files recorded earlier and not yet overwritten.

The goal: scan it for all possible mlv file headers, then scan for all frame blocks following definitions of found mlv files, then save all of them sorted according to time stamp into corresponding mlv files (with option of filling gaps - if frame is missing, repeat the previous one with correct TS, option of inserting bright/loud dummy frame to make sure I will notice it, option of not inserting anything but what was found). Would be nice to make a check for blocks (i.e. partial / erroneous should be skipped).

Surely, some of them will not workout well, but my interest is primarily focused on the recovery of the last one. I checked definition of C structs, and probably can make a parser, but it should be definitely much easier and faster to get done by Author.

Is this possible/could be made?

Thank you,
Mikhail

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3157
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1927 on: March 27, 2015, 10:04:01 AM »
FIRST: make an image of the CF card using some disk imaging tools (there are some open source tools)

some weeks ago i chatted with the developer of PhotoRec who was so kind to add MLV support.
so: get photorec, it will do its best to recover your footage.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

moodlover

  • New to the forum
  • *
  • Posts: 10
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1928 on: March 28, 2015, 04:38:26 AM »
If one didnt need sound, should we just stick to .raw instead?

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3157
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1929 on: March 28, 2015, 10:44:20 AM »
if your desire is higher speeds, yes. thats the big advantage of .raw.
if you want metadata with your footage, choose MLV.

i personally do not need high speeds but prefer the files to contain all possible information.

it might be interesting to convert .raw->.mlv as .mlv seems to reach a broad tool suite.
so people who need to work around the mlv-deficit "writing speed" can still post process and archive it as .mlv
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6937
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1930 on: March 28, 2015, 01:24:07 PM »
+10 for raw to mlv

KMikhail

  • Freshman
  • **
  • Posts: 75
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1931 on: March 28, 2015, 10:59:48 PM »
Yes, I of course did make image - to secure data and free-up the CF card for usage. Will think over, Thanks!

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1932 on: March 29, 2015, 12:11:42 AM »
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

jose_ugs

  • Member
  • ***
  • Posts: 243
  • 5DM3
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1933 on: April 09, 2015, 11:05:42 AM »
Wasn't there already a tool for that, RAW2MLV?

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1668
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1934 on: April 27, 2015, 04:04:51 PM »
Does somebody know if mlv_dump can average all images in an Full-res-silent-picture MLV ?
I can't get the -a, option to work on full res silent MLV ?

This is what I type in terminal
mlv_dump -a /Users/wij/Documents/TEST/72160000.MLV

This is the output:
 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Verify file structure
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Reached end of chunk 1/1 after 205 blocks
Processed 50 video frames
Floating point exception: 8

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3157
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1935 on: April 27, 2015, 09:19:03 PM »
uhm i the number of averaged frames is exactly zero.
thats why there is an exception.

reason: you forgot to add an output file
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1668
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1936 on: April 28, 2015, 10:49:36 AM »
He G3gg0,

Tried to put an output file in the command line...I get a segmentation fault 11 and a zero byte sized averaged.mlv

mlv_dump -o averaged.mlv -a /Users/wij/Documents/TEST/72160000.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Rewrite MLV
   - Output only one frame with averaged pixel values
   - Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Segmentation fault: 11


EDIT:
The reason I'm trying this, is because I want to have a long exposure picture in daylight without needing ND filters.
Nice landscape shots, with soft tree leaves and soft clouds  8)

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1937 on: April 29, 2015, 03:56:06 AM »
@g3gg0:

Just wanted to provide feedbacks regarding 70D footage. Apparently either the MLV_Dump isn't responding to 70D files nor there aren't proper color matrixes ( it thinks its a 6D when it isn't? lol ) It looks like it may be lacking either one of these....

Here are the 70D111B files in case you need to dig through them...

Short Clips:
https://mega.co.nz/#F!HtUx0CgS!jQslzxzJTC19VVwGGN75lA

Varies of Files including Dual-ISO:
https://mega.co.nz/#F!yoNQWBTa!3ohyAu9UggiyreAiFjo8nA

I've also posted some videos of what the footage looks like via Vimeo:
https://vimeo.com/126099527

Hope this helps!

*UPDATE*

After busting an all nighter with an fellow ML buddy (Special Thanks to @Danne!) he finally made it possible for me to get my converted DNG's from 70D's Dual-ISO CR2's to work again (Previously wouldn't be able to read from PS) with the help of a '70D matrix command' provided by @Danne. (Hope it's OK w me sharing this link - if not let me know!)

Another good news is the fact that now I am able to convert both MLV & RAW files with Full color matrice 6D/70D by using @Danne's latest cr2hdr-r which works well on Mac's.

Will still need to investigate further on 70D111B Dual-ISO MLV/RAW though...
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3157
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1938 on: May 01, 2015, 09:02:14 PM »
@Levas:
can you run mlv_dump with parameter -v and upload the dump?
or can you put the MLV file in my btsync folder with key AUUKDJIEUOPLOPFN5NECA2NEVCLRHPHFO?

@DeafEyeJedi:
mlv_dump didnt produce clean DNG files?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1939 on: May 01, 2015, 09:28:21 PM »
Not for Dual-ISO MLV/RAW files but for regular MLV/RAW files it works fine from a 70D111B w latest nightly.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6937
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1940 on: May 01, 2015, 10:16:12 PM »
Did you put up a dual iso sample in the folder? I could check later.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1941 on: May 02, 2015, 12:25:38 AM »
Yes, I believe I did.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6937
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1942 on: May 02, 2015, 01:06:54 AM »
Deafeye. Could you retest a non corrupted dual iso mlv/raw and extract with mlv_dump, raw2dng? Or upload a new sample file, dual iso from 70D cam. Again, check that it isn,t any corruption issue.

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1943 on: May 02, 2015, 03:30:51 AM »
What are exactly the corruption issues do you recall from 70D dual ISO files? Because all of them comes out the same weird colors w flickering effect.

I'll do another new dual ISO MLV/RAW tonight for you...
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6937
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1944 on: May 02, 2015, 07:49:11 AM »
Thanks.
Downloaded the file M01-1819.MLV and the file itself shows corruption. It is not mlv_dump that is the problem. Maybe share a link to the file here or in the 70D thread for the experts to look at?

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1668
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1945 on: May 02, 2015, 02:54:33 PM »
@Geggo

Used the -v option, here's the output:

mlv_dump -o averaged.mlv -a -v /Users/wij/Documents/TEST/72160000.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
   - Verbose messages
   - Rewrite MLV
   - Output only one frame with averaged pixel values
   - Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 10242427998058437803
    FPS         : 0.500000
    File        : 0 / 0
    Frames Video: 50
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 500.314000 ms
    Res:  5568x3723
    raw_info:
      api_version      0x00000001
      height           3723
      width            5568
      pitch            9744
      frame_size       0x02298AB0
      bits_per_pixel   14
      black_level      2047
      white_level      14783
      active_area.y1   52
      active_area.x1   72
      active_area.y2   3723
      active_area.x2   5568
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: IDNT
  Offset: 0x000000e8
    Size: 84
    Time: 503.485000 ms
     Camera Name:   'Canon EOS 6D'
     Camera Serial: '1354ACF6DB'
     Camera Model:  0x80000302
Block: WBAL
  Offset: 0x0000013c
    Size: 44
    Time: 503.574000 ms
     Mode:   2
     Kelvin:   2500
     Gain R:   496
     Gain G:   1024
     Gain B:   624
     Shift GM:   -1
     Shift BA:   0
Block: STYL
  Offset: 0x00000168
    Size: 52
    Time: 503.585000 ms
     picStyle:   132
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: RTCI
  Offset: 0x0000019c
    Size: 44
    Time: 513.114000 ms
     Date:        27.04.2015
     Time:        14:33:55 (GMT+0)
     Zone:        ''
     Day of week: 1
     Day of year: 116
     Daylight s.: 0
Block: EXPO
  Offset: 0x000001c8
    Size: 40
    Time: 513.761000 ms
     ISO Mode:   0
     ISO:        100
     ISO Analog: 72
     ISO DGain:  0/1024 EV
     Shutter:    445000 microseconds (1/2.25)
Block: LENS
  Offset: 0x000001f0
    Size: 96
    Time: 513.785000 ms
     Name:        'EF35mm f/2 IS USM'
     Serial:      ''
     Focal Len:   35 mm
     Focus Dist:  65535 mm
     Aperture:    f/11.30
     IS Mode:     0
     AF Mode:     3
     Lens ID:     0x000000F9
     Flags:       0x00000000
Block: VIDF
  Offset: 0x00000250
    Size: 36276944
    Time: 525.316000 ms
   Frame: #0000
    Crop: 0x0
     Pan: 0x0
   Space: 0
Segmentation fault: 11

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1946 on: May 02, 2015, 05:54:09 PM »
@g3gg0:

Here's the link to 70D files including Dual-ISO MLV/RAW from 70D111B (April 23)...

https://mega.co.nz/#F!6hchkCbD!zScVqL68khdbY6MylpzhNg
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3157
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1947 on: May 02, 2015, 11:09:12 PM »
@levas:
fixed. try the latest version:

@DeafEyeJedi:
i think i dont understand the problem.
so mlv_dump creates buggy matrices for dual iso mlv files?
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1948 on: May 03, 2015, 03:47:56 AM »
Regular MLV/RAW seems responding to mlv_dump well when using @Danne's cr2hdr-r latest beta but then when I record Dual-ISO RAW/MLV videos then it shows all kinds of strange pinkish effects as you can see in these files I uploaded.

MLV_dump won't respond to those files maybe because it's already corrupted from inside camera while recording?

Just thought it was worth noting... That's all.

However, Dual-ISO CR2's from 70D seems to respond well but just not the videos.

Just wanted to share that with you since I wasn't sure of @A1ex hasn't updated this binary yet for the 70D files.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Levas

  • Contributor
  • Hero Member
  • *****
  • Posts: 1668
  • 6d - Nightly build user
Re: Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)
« Reply #1949 on: May 03, 2015, 10:40:21 AM »
@g3gg0

Averaging with the newest MLV_dump works! :D

Multiple Full-res-silent-pictures -> Recording in MLV container -> averaging out to one single photo with MLV_dump = The most extreme and variable ND filters one can have  8) :D

Edit: oh yeah I forgot something, THANKS