Magic Lantern Forum

Using Magic Lantern => Raw Video => Raw Video Postprocessing => Topic started by: Rolfe Klement on March 31, 2018, 12:47:01 AM

Title: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Rolfe Klement on March 31, 2018, 12:47:01 AM
Hi - I searched the forums for bit but couldn't find anything on it// Apologies if it has been covered before.

I get the following message in the MLV app (none of the other apps work at all..)

"File Header is missing, invalid for file xxx.mlv"

I am using the latest experimental build - but did play back during the day and it worked fine but now it doesn't play in the camera either... I tried the old HEX edit on the footer but the issue is the header. footage was 1980 3:2 at 25fps. I set the crop mode to 1920.

Unfortunately it was great footage :-(

I include a link to it - can anyone check?

https://www.sendspace.com/filegroup/7MmDo5BCjMvRJbqNRewD01Bhsk0cp3CsVeKun44jUm4S8%2BAkiDfisxO8W1%2BPAYZv9tdstiPHLlM

Fingers crossed :-)

HUGE THANKS!
Rolfe

Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: a1ex on March 31, 2018, 01:37:57 AM
The data before the first VIDF looks corrupted (no obvious block headers). First VIDF is at 0x212800, second at 0x430400, third at 0x64E200 and so on. Frame size is slightly variable, which suggests lossless compression. Before first VIDF there's too much data (nearly the size of one frame; a header is not that big). Entropy analysis suggests it might be video data. Hypothesis: file starts from the middle.

First VIDF (after replacing that big chunk before it with a header from a good MLV):

Block: VIDF
  Offset: 0x00000600
  Number: 13
    Size: 2219008
    Time: 2485.695000 ms
   Frame: #0059
    Crop: 152x32
     Pan: 152x33
   Space: 32


That frame #0059 does not look good. At the end, it prints "Processed 131 video frames" and hints that there may be some more frames after that.

If the size of the .IDX is to be believed, (2104 - 4) / 20 = 105 frames (not good). Data from IDX is not valid (likely video data). Possible filesystem corruption?

Crash logs seem related to H.264 (proxy, I assume).

How long was the clip? (I assume longer than 5 seconds).

Assuming you did not write to the card after the error happened, the first step would be to copy the raw card contents (https://www.magiclantern.fm/forum/index.php?topic=21699.msg197980#msg197980) to a regular file (a card image); after that, you may experiment with data recovery tools. I can try to look into it if you upload the card image, but cannot guarantee success.

BTW - I did not try to decode any frames.
Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Rolfe Klement on March 31, 2018, 10:56:31 AM
Great info- thanks Alex

I am running some deep scan (Recuva) recovery software on the disk now... I'll let you know when it is done..

thanks again!!!
Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: a1ex on March 31, 2018, 11:15:11 AM
With some knowledge of the file format (each frame tagged with VIDF and some metadata like frame number, timestamp and block size), it should be possible to scan the entire card image for missing frames, then dump individual VIDF blocks to files, then assemble them by concatenation, then prepend a header from a good MLV recorded at the same setting.

I've recovered some CR2 (https://www.magiclantern.fm/forum/index.php?topic=18513.0) files with a similar method (brute force search for EXIF info containing some specific date).

There is MLV support in PhotoRec (https://www.magiclantern.fm/forum/index.php?topic=13602.0), but I did not try it and don't know how far it goes with MLV analysis.
Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Rolfe Klement on April 01, 2018, 09:34:32 AM
Dear Alex

Thanks - I did a recover and it looks bad. Even though the kb count of the MLV files is exactly the same as the version on my Windows machine copy - the last sections of the recuva recovered files are all just 00 in HEXEditor...

I think the first dump I made of the files was the best -  it seems full of data. I m still running a VDIF content search on the disk but it will take the whole day..

I bought another card yesterday and installed the same version and same setup so I can get a header.

I notice I did the 1920 1:1 - and then filmed 1980 at 3:2 - so it has a bar at the bottom but that is fine...

My question - what is the header in this working file below (Myairbrdige link)  and where should I paste / append it into the damaged files? I tried copying everything from the beginning down to the first VDIF line and Paste inserting into the damaged file but says corrupted header...

https://mab.to/flzSUbRq1

Again - huge thanks!

Rolfe

Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: a1ex on April 03, 2018, 01:35:26 AM
Just take everything until the first VIDF, as header. Then, from the bad MLV, take everything starting from first VIDF.


dd if=M01-0704.MLV of=header.mlv bs=$((0x600)) count=1
dd if=M25-1721.MLV of=frames.mlv bs=$((0x212800)) skip=1
cat header.mlv frames.mlv > recovered.mlv
mlv_dump --dng recovered.mlv


If running from a Windows command prompt: install dd for windows, replace hex offsets with decimal, find the equivalent of "cat".

Please keep me posted regarding VIDF content search - what method did you use for that?
Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Walter Schulz on April 03, 2018, 01:56:06 AM
Quote from: a1ex on April 03, 2018, 01:35:26 AMfind the equivalent of "cat"
copy /b header.mlv+frames.mlv recovered.mlv
Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Rolfe Klement on April 03, 2018, 11:50:12 PM
Awesome thanks - I am approaching the limits of my windows knowledge :-)

So I should

install DD -
Open command
replace HEX with decimal - not sure how to do this..
then run cat - "copy /b header.mlv+frames.mlv recovered.mlv"
then run the following code
"dd if=M01-0704.MLV of=header.mlv bs=$((0x600)) count=1
dd if=M25-1721.MLV of=frames.mlv bs=$((0x212800)) skip=1
cat header.mlv frames.mlv > recovered.mlv
mlv_dump --dng recovered.mlv""


:-) - is this correct?

Huge thanks!!

Rolfe

Title: Re: MLV - Missing Header issue - on new 4k / crop build - help needed
Post by: Walter Schulz on April 04, 2018, 09:40:21 AM
dd if=M01-0704.MLV of=header.mlv bs=1536 count=1
dd if=M25-1721.MLV of=frames.mlv bs=2172928 skip=1
copy /b header.mlv+frames.mlv recovered.mlv
mlv_dump --dng recovered.mlv


In case dd is not found you have to add the path to dd (replace with your location!).
d:\cygwin64\bin\dd if=M01-0704.MLV of=header.mlv bs=1536 count=1
d:\cygwin64\bin\dd if=M25-1721.MLV of=frames.mlv bs=2172928 skip=1
copy /b header.mlv+frames.mlv recovered.mlv
mlv_dump --dng recovered.mlv