Author Topic: 5DM3 MLV Problem can't convert to DNG, nothing works  (Read 7948 times)

Nick2333

  • New to the forum
  • *
  • Posts: 10
5DM3 MLV Problem can't convert to DNG, nothing works
« on: October 20, 2014, 05:41:33 PM »
Hey there, i've got a problem with some .mlv data. The files were made with a 5D Mark 3, firmware 1.2.3, i used the build 2014August07.5D3123.
Nearly 20% of all the .mlv files seems to be corrupted.  I've tried to convert the files with MLV Converterv1.9.2, but no luck... I also tried to open the files with MLRawViewer_1_2_3, but mlrawviewer doesn't even load the mlv files. Same effect with mlv2dng and with MLVViewSharp...

this is the only info i get with MLV Converter :

Does anybody know wether there is a another way how i can open the .mlv container and get the dng sequence out? I would appriciate avery advice to solve this problem.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #1 on: October 20, 2014, 10:56:17 PM »
Open a bad MLV file with a hex editor and post a screen shot (of the beginning of the file). You can also try mlv_dump and MLVFS (if the files are actually corrupted this probably won't help)

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #2 on: October 21, 2014, 12:21:20 AM »
i've tried mlv_dump, but it doesn't work either....  but i got this message from MLVBrowseSharp



i also opend a .mlv file with a hex editor




dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #3 on: October 21, 2014, 12:52:22 AM »
That doesn't look good. The file should start with string "MLVI". At least the beginning of the file is corrupted. Do a search for VIDF, we'll see if there are any intact frames.

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #4 on: October 21, 2014, 01:09:21 AM »
VIDF....


g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3166
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #5 on: October 21, 2014, 01:17:49 AM »
> goog-badbinurl-shavar.cache

that looks as if your MLV has been overwritten by some bad guy... filesystem error?
RCRD is the header of the $LogFile NTFS logging records.
bad mojo happening here...
definitely nothing MLV related, as MLV creation only happens on (ex)FAT devices.

please describe how the files came from the card to where they are now.
(very detailed, mind different drive letters, filesystems, readers, etc)
i would definitely check my backups now if i were you. (no, do not overwrite old ones!)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #6 on: October 21, 2014, 01:47:18 AM »
out of the5dm3, the cf card( formatted in exFat) went into a usb3 card reader, then i connected the cardreader  to a usb3 expresscard 54mm that is in my laptop win8.1 pro. 
some footage went directly to the local SSD inside the laptop, after it was full, i transfered the footage directly from the CF Card reader to a external HDD drive which was also conncted to the usb3 expresscard... the external HDD first was a local system  drive with windows 7 on it, i formatted it with the windows internal HDD Manager.
i put the HDD into a 2,5 usb3 case and use it as a external drive, although the settings of this HDD still say its a local drive....

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3166
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #7 on: October 21, 2014, 01:54:11 AM »
can you make out if the defect files are perhaps those that you transferred from CF to the external drive, both over USB3?

to me it really looks as if you had some hardware issues.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #8 on: October 21, 2014, 02:16:44 AM »
yes it really could be like this, all the broken .mlv's were transferd over the usb3 expresscard to the external drive... (i'm not 100% sure but it looks like..) is there sth. i can do in this case?

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #9 on: October 21, 2014, 11:55:53 AM »
but where is the connection between my usb3 hardware and this 'goog-badbinurl-shavar.cache' in my .mlv file?

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3166
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #10 on: October 21, 2014, 12:48:41 PM »
theory:
that file name was somewhere in a sector on your HDD that has NTFS filesystem.
it could have been anything, in this case it was the first sector of a NTFS transaction log for your browser's URL blacklist.

during transfer something really bad happened that caused mixing up sectors - or - their content.
might have been bad luck or (what i expect) some race condition when having both devices under heavy load on your PCIe USB3 card.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #11 on: October 21, 2014, 01:21:10 PM »
thank you very much for clearing this up! your expectrations could be true, it makes sense now, when the usb3 card was used, i realized a overheating of my cpu, also the usb3 expresscard was pretty hot, my cpu was running on full power... ,  i could never think of that this could be the conciquences of a usb3 expresscard.

i seems like this situation is the worst case ever that can happen to files?... so i guess i can say au revoir to my files?


dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #12 on: October 21, 2014, 01:38:24 PM »
It looked like there was still some data there (the VIDF header for frame #1 appeared to be valid and intact). So perhaps only the very beginning of the file was overwritten. If so, it should be possible to recover the data, but it will take some manual tweaking of the file to fix it. Continue searching through the file for "VIDF" and see if there is one about every ~3.5MB or so.

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #13 on: October 21, 2014, 02:01:14 PM »
some more screenshots of the broken file...













Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #14 on: October 22, 2014, 11:29:02 AM »
it seems like there are some dng data in it, or am i wrong? is it possible to copy the beginnig of a working .mlv and replace it with the corrupted in te hex editor? or am i thinking to simple? dmilligan if you know how to prozess with the broken files, please give me a advice how i can start.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #15 on: October 22, 2014, 01:28:06 PM »
It's hard to tell, from the screen shots some of it looks okay and some looks like garbage. For example you shouldn't be having VIDF blocks close together like that, they should be ~3.5MB apart.

Replacing the beginning of the file may or may not work. MLV is a block based format, each block indicates the offset to the next block, so if any offsets are messed up a parser is going to have trouble (which might be the case if you copy in the beginning of a different MLV file).

You can read about the format here: http://www.magiclantern.fm/forum/index.php?topic=7122.0 and the specific fields for each block here: https://docs.google.com/spreadsheet/ccc?key=0AgQ2MOkAZTFHdHJraTVTOEpmNEIwTVlKd0dHVi1ULUE#gid=0

The blocks always start with 4 letters (these are easy to see and search for in the hex) followed by a "size" (4 byte integer) that basically tells you the relative offset to the next block (also keep in mind this is little endian, so the bytes are in reverse order, a 4 byte integer: A1 B2 C3 D4 in the file would actually be D4C3B2A1 in hex).

Basically what you need to do is copy in the required blocks (at least the MLVI and RAWI blocks) from a good MLV file (with the exact same settings, resolution, etc. that you were using with this clip) these blocks should be at the beginning, and then have the last block you copy in, point to the first good VIDF block in your file. You may have to trim some out of the file or adjust the offsets, or insert "NULL" blocks to achieve this.

Nick2333

  • New to the forum
  • *
  • Posts: 10
Re: 5DM3 MLV Problem can't convert to DNG, nothing works
« Reply #16 on: October 29, 2014, 04:23:17 PM »
thank you very much dmilligan, i love you!
I could rescue a lot of data, just by copying the beginning of  a working mlv into a corrupt, even audio was still there and synchronized... i had to cut more or less of the beginning of the corrupt files, but it worked out!
fotunately it seems like most of the files were only damaged at the begining...
but i'm still struggeling with some files. one corrupt 13gb .mlv file is tricky for me, with the copy paste technique i just can get parts of the file.
when i paste the working mlv code at the beginning of the 13gb .mlv, i just get the first 37 seconds of the file, then it goes black. second time i delted half of the file in HxD and pasted the beginning of a working .mlv, i also get 37 seconds footage, but now the middle parts of the file... its strange... because for some reason i got as a result, 37secs or 18 secs of footage on several files...   i guess it's not a coincidence, but my lack of knowledge about what i'm copy/pasting... i used on all files the same .mlv code of a 20GB working .mlv footage, on most files it worked out... but there are still some that just work for exactly 37 secs then it goes black... 
i have three different cases, first case it works fine (data rescuded!), second case i just get 37 secs, and third one i get about half of the file, in MlRawViwer it looks like this:



it just stops loading the other half, in mlrawviewer i can not preview it, but i can convert the one half into dng...

if you got some more tips i can try, it would be nice... nevertheless you helped out a lot! big thx!