Author Topic: Problem with playing back whole raw card  (Read 1861 times)

dfgh

  • New to the forum
  • *
  • Posts: 33
Problem with playing back whole raw card
« on: November 08, 2017, 08:41:41 PM »
Hello!

I've used magic lantern a fair bit for lots of different shoots but today I've come across a new problem. I have a whole card of footage that will not playback or export. MLrawviewer just will not open any of the files from that card.






Can anyone shed light on this, as I obviously really need this footage, or I will have reschedule a reshoot.

bpv5P

  • Member
  • ***
  • Posts: 167
Re: Problem with playing back whole raw card
« Reply #1 on: November 08, 2017, 09:19:38 PM »
The data is probably there, since the file is not empty.
Try another software, so you can confirm that it's not a MLrawViewer issue. If none of them can open it, try to put the card back in camera, turn on the file_man module and try to playback the .RAW file inside the camera. If the preview works, then you're data is actually there, but for some reason you can't read it. In that case, it would be needed more information about your system (maybe the filesystem is read-only?)...

Kharak

  • Hero Member
  • *****
  • Posts: 722
Re: Problem with playing back whole raw card
« Reply #2 on: November 08, 2017, 09:30:08 PM »
It is most likely because your card is Full to the brim. MLrawviewer creates MRX (metadata) files and makes new WAV-"sidecar" files on the side of each MLV (2x audio files). If your card is Maxed out, there is no space for MLRawViewer to playback/export the files.

Copy/cut one MLV to your computer and the that should be enough space to play the rest of the files on the card.
once you go raw you never go back

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #3 on: November 08, 2017, 09:46:10 PM »
I've tried other software and tried rewriting the hex data with another hexcode I've found on the forum. The former ( other software ) doesn't work and the latter opens the file, but in a scrambled mess.

I'm viewing these on my computer, they've been copied over

I've had this happen before on files at the end of space on a card, but never on every file on that card that played fine in the camera

I viewed the files in ML on the camera after I recorded them.

bpv5P

  • Member
  • ***
  • Posts: 167
Re: Problem with playing back whole raw card
« Reply #4 on: November 08, 2017, 10:18:08 PM »
So, you can play the file inside the camera, but cannot extract the dng from the .raw file?
If that's the case, then the data corruption hypothesis is not the problem. If you make sure you really can't process it with the latest raw2dng, then the problem is also not on the processing software. You also said you copied the files to the computer, so it's read-write.
Considering your hardware is performing well and the software (operating system) is also OK (could try in another computer?), I can't see the solution, sorry. Maybe some developer have the answer...

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #5 on: November 08, 2017, 10:20:41 PM »


I have a lot of other files from other cards today and they're all playing fine. It's just every one this particular card dump, doesn't open. If I alter the hex footer then they open, but scrambled, with various frames from what I shot blended in.

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #6 on: November 08, 2017, 10:23:16 PM »
You can try transcoding them in program Switch. In there you can convert to MLV and see if that works better. Or just transcode your RAW files directly.

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #7 on: November 08, 2017, 10:27:41 PM »
Thanks Danne, I'll give this a go

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #8 on: November 08, 2017, 10:32:41 PM »
You are still shooting "RAW" version 1.0 files? There have been so many updates and changes since that format was retired. You should really update your Magic Lantern workflow.

The only way anyone can help you is if you could upload one of these files. Yeah, they're huge so try to pick the smallest .RAW file. Maybe convert to MLV and trim before uploading? You can also do that in Switch. Don't know if that will work.
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #9 on: November 08, 2017, 10:40:53 PM »
You can try transcoding them in program Switch. In there you can convert to MLV and see if that works better. Or just transcode your RAW files directly.



Doesn't seem to want to work
Doesn't produce any files.


Code: [Select]
M06-1238.RAW
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
/tmp/DUALISO/RAW_demolish
ls: /tmp/DUALISO/RAW_demolish: No such file or directory
/tmp/DUALISO/RAW
01_legacy_RAW_to_MLV
M06-1238.RAW
M06-1241.RAW
M06-1243.RAW
M06-1248.RAW
M06-1250.RAW
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1238.RAW
--- End of sequence ---
M06-1241.RAW
M06-1243.RAW
M06-1248.RAW
M06-1250.RAW
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1241.RAW
--- End of sequence ---
M06-1243.RAW
M06-1248.RAW
M06-1250.RAW
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1243.RAW
--- End of sequence ---
M06-1248.RAW
M06-1250.RAW
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1248.RAW
--- End of sequence ---
M06-1250.RAW
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1250.RAW
Found file M06-1250.R00
--- End of sequence ---
M06-1255.RAW
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1255.RAW
--- End of sequence ---
M06-1256.RAW
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1256.RAW
--- End of sequence ---
M06-1300.RAW
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1300.RAW
--- End of sequence ---
M06-1301.RAW
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1301.RAW
--- End of sequence ---
M06-1307.RAW
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1307.RAW
--- End of sequence ---
M06-1308.RAW
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1308.RAW
--- End of sequence ---
M06-1337.RAW
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1337.RAW
Found file M06-1337.R00
--- End of sequence ---
M06-1338.RAW
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1338.RAW
--- End of sequence ---
M06-1339.RAW
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1339.RAW
--- End of sequence ---
M06-1340.RAW
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1340.RAW
--- End of sequence ---
M06-1342.RAW
Error: This ain't a lv_rec RAW file


Found file M06-1342.RAW
--- End of sequence ---

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #10 on: November 08, 2017, 10:45:07 PM »
Seems something corrupted as you said.
You seem to be one of tha last RAW format shooters around  :P

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #11 on: November 08, 2017, 10:47:02 PM »
Yeah, I only shot it because I find MLV doesn't perform long takes on my camera very well for some reason.

The question is, can I reverse this corruption

bpv5P

  • Member
  • ***
  • Posts: 167
Re: Problem with playing back whole raw card
« Reply #12 on: November 08, 2017, 10:50:48 PM »
I don't think that's corruption, else it wouldn't decode inside the camera... have you tried to check if your copy is exactly what's on the card (using a hashing algorithm, for example)? Maybe while you were doing the copy it just corrupted. The 7zip has a simple checksum tool, try it, please.

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #13 on: November 08, 2017, 10:56:36 PM »

The card is now formatted and was used later in the day unfortunately, so I can't check it but it was playing the files as I viewed them back
How do I perform the 7zip checksum?

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #14 on: November 08, 2017, 11:02:46 PM »
Quote
Yeah, I only shot it because I find MLV doesn't perform long takes on my camera very well for some reason.
Tried mlv_lite and 12bit mode for instance?

bpv5P

  • Member
  • ***
  • Posts: 167
Re: Problem with playing back whole raw card
« Reply #15 on: November 08, 2017, 11:04:19 PM »
You can't now, since the original file is gone (hash algorithms create a unique number for a file, so you can make sure it's not alterated)...

tl;dr unless someone appear with a magical solution or you have the skills to fix the file yourself (it would take too much time and effort to even consider), the files can't be converted. Better reschedule a reshoot. Make sure to use MLV module instead now. Magic Lantern is experimental, sometimes things can go bad.

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #16 on: November 08, 2017, 11:25:16 PM »
The only way anyone can help you is if you could upload one of these files.

Still applies.
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #17 on: November 08, 2017, 11:28:37 PM »

Ok I'll do that

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #18 on: November 08, 2017, 11:51:17 PM »
 This is what I get from raw2dng

raw2dng converter GUI for OsX
Beta ver.0.13

M06-1342 File Supported
Error: This ain't a lv_rec RAW file

Generating ProResHQ 4444 with FPS:
*.dng: No such file or directory
ffmpeg version 1.2.1-tessus Copyright (c) 2000-2013 the FFmpeg developers
  built on May  9 2013 21:58:14 with llvm-gcc 4.2.1 (LLVM build 2336.1.00)
  configuration: --prefix=/Users/tessus/data/ext/ffmpeg/sw --as=yasm --extra-version=tessus --disable-shared --enable-static --disable-ffplay --enable-gpl --enable-pthreads --enable-postproc --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-libspeex --enable-bzlib --enable-zlib --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libxavs --enable-version3 --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvpx --enable-libgsm --enable-libopus --enable-fontconfig --enable-libfreetype --enable-libass --enable-filters --enable-runtime-cpudetect
  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
[image2pipe demuxer @ 0x10281f400] Could not parse framerate: .
pipe:0: Invalid argument
Done

bpv5P

  • Member
  • ***
  • Posts: 167
Re: Problem with playing back whole raw card
« Reply #19 on: November 09, 2017, 12:18:12 AM »
The raw2dng can't read it. The header is probably fucked.
Can you reproduce this error? Other files recorded with this camera gets the same error?
You could send the file as @dford suggested, if it's not something that require strict privacy. Compression with LZMA2 can be done with 7zip (again), so it will take less time to upload.

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #20 on: November 09, 2017, 12:40:26 AM »
I am currently uploading but it's going to take a few hours. I have copied and pasted hex footer data from a working clip into the footer of a clip that doesn't work and I got it to load up albeit pink fuzzy and distorted, so I have some hope there's a solution

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #21 on: November 09, 2017, 01:32:18 AM »
Have you tried this yet?

try using chmee's raw2cdng, it seems to have a better tolerance when i have a corrupt first frame from my file and the raw to dng converters, browse sharp and tonybeccar's windows dng converter and rawanizer can't convert them, the cdng converter always get past the corruption and convert the file, including the corrupt frames.
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #22 on: November 09, 2017, 05:24:35 AM »

 I haven't tried those dfort thanks, I'll give them a go.

In the meantime here is a RAW to download https://wetransfer.com/downloads/4331bcf5c6beb4abb4553b8ca57633c520171108223510/caf5e097adf10e503ea522c0804a3f2920171108223510/37fe82

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #23 on: November 09, 2017, 05:33:43 AM »
Downloading. Meanwhile, chew on this:
http://www.magiclantern.fm/forum/index.php?topic=5732.msg41099#msg41099


*Update
Seems corrupted. Just for fun I commented out following in raw2dng.c to see what comes out of raw2dng:
Code: [Select]
    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
        FAIL("This ain't a lv_rec RAW file\n");
   
    if (raw_info.api_version != 1)
        FAIL("API version mismatch: %d\n", raw_info.api_version);

Gives the following info. Not even exiftool could fix this one :P:
Code: [Select]
Resolution  : 44486 x 38704
Frames      : 1252570185
Frame size  : 340438548 bytes
FPS         : 3182729.412
Black level : -1592480841
White level : 67810884

Processing frame 1 of 1252570185 Segmentation fault: 11
dans-MBP:lv_rec dan$

You should try the replace footer solution.

*Update
http://www.magiclantern.fm/forum/index.php?topic=5732.msg41179#msg41179
Added the 8gb footer above and out comes dng files however all are corrupted:

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #24 on: November 09, 2017, 06:01:44 AM »
 By copying this from and pasting it in the footer
Code: [Select]
AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAA
AAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA555
5AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA55555555AAAA5555AAAA5555AAAAAAAA555555555241574D80073804006037004A
02000001000000A8610000D8470400000000000100000000A0314D2605000020080000380E0000503449000E000000FE070000983A000000000000
000000008C0700000A0500001C00000092000000260500001E08000000000000000000000001010201000000421A00001027000085FDFFFF102700
003DFCFFFF1027000041EFFFFF10270000AC30000010270000EC0700001027000074FCFFFF1027000072080000102700002416000010270000F4030
000

I can get MLviewer to play it, but it's distorted and sectioned into pink bits and a bit of normal colour, so the footage is certainly there

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #25 on: November 09, 2017, 06:05:20 AM »
Yes, tried as well but all corruption. Updated my answer above.

Not sure if it´s possible to patch raw2dng to fix the files.

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #26 on: November 09, 2017, 08:52:41 AM »
By copying this from and pasting it in the footer

You can't insert just anything in the footer. What you inserted was:

Code: [Select]
RAW footer:
6d73c15c: 52 41 57 4d 80 07 38 04 00 60 37 00 4a 02 00 00  RAWM..8..`7.J...
6d73c16c: 01 00 00 00 a8 61 00 00 d8 47 04 00 00 00 00 00  .....a...G......
6d73c17c: 01 00 00 00 00 a0 31 4d 26 05 00 00 20 08 00 00  ......1M&... ...
6d73c18c: 38 0e 00 00 50 34 49 00 0e 00 00 00 fe 07 00 00  8...P4I.........
6d73c19c: 98 3a 00 00 00 00 00 00 00 00 00 00 8c 07 00 00  .:..............
6d73c1ac: 0a 05 00 00 1c 00 00 00 92 00 00 00 26 05 00 00  ............&...
6d73c1bc: 1e 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02  ................
6d73c1cc: 01 00 00 00 42 1a 00 00 10 27 00 00 85 fd ff ff  ....B....'......
6d73c1dc: 10 27 00 00 3d fc ff ff 10 27 00 00 41 ef ff ff  .'..=....'..A...
6d73c1ec: 10 27 00 00 ac 30 00 00 10 27 00 00 ec 07 00 00  .'...0...'......
6d73c1fc: 10 27 00 00 74 fc ff ff 10 27 00 00 72 08 00 00  .'..t....'..r...
6d73c20c: 10 27 00 00 24 16 00 00 10 27 00 00 f4 03 00 00  .'..$....'......

Width in hexidecimal:
8007

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
32775

Converted from little endian encoding:
0780

Image width taken from the raw footer:
1920

Image height taken from the raw footer:
1080
a861
61a8
25000
25.000

So if this wasn't the settings you used when you shot it you are going to get corrupt footage. The above output is from a little script I wrote a while back when trying to figure out RAW footers. I never took the time to complete the script but it does give you an idea what is in the footer. You can read how this came about in the Dealing with Focus Pixels topic starting here.

You need to take one of your uncorrupted RAW files and run it through this script to figure out what you should be pasting into those corrupted files.

The corrupted file you uploaded is missing the footer. What the script read from your file is pretty much just gibberish:

Code: [Select]
RAW footer:
6d73c000: ab 8a ad 4a c6 ad 30 97 14 ae 4a 14 49 b4 a8 4a  ...J..0...J.I..J
6d73c010: a5 49 ab a9 c4 98 b4 bd 0a b4 49 44 aa ca 9f 49  .I........ID...I
6d73c020: bc a9 a8 a1 a4 98 4a 88 4a 54 ab 8a 95 4a bc ab  ......J.JT...J..
6d73c030: 68 a5 64 df 8a f4 49 24 ab 8a ad 4a b7 ab 14 a1  h.d...I$...J....
6d73c040: 44 b6 0a 04 4a f4 ab ca a3 49 da aa f0 9f f4 ba  D...J....I......
6d73c050: ca 74 49 d4 a9 4a 9c 49 a4 ab 64 a9 74 ac 8a 84  .tI..J.I..d.t...
6d73c060: 4a e4 aa ca a3 49 cd ab a8 ae 54 a9 0a e8 4a 94  J....I....T...J.
6d73c070: ac ca 99 49 bf a9 a4 9b 14 be ca b0 49 a4 ab 8a  ...I........I...
6d73c080: 9d 49 c9 a9 68 92 24 ce 4a 38 49 04 ab 0a a6 4a  .I..h.$.J8I....J
6d73c090: c0 aa 10 9b 34 ae 0a 3c 4a 74 ab 8a a3 4a c5 ab  ....4..<Jt...J..
6d73c0a0: 5c a1 44 c7 8a 84 4a 54 ac 0a a8 4a c5 aa a4 9d  \.D...JT...J....
6d73c0b0: 74 e3 ca 44 4a 54 ac 8a a2 4a da ad 68 a8 54 b7  t..DJT...J..h.T.

Width in hexidecimal:
c6ad

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
50861

Converted from little endian encoding:
adc6

Image width taken from the raw footer:
44486

Image height taken from the raw footer:
38704
c498
98c4
39108
39.108

Not sure what else is missing but I was able to extract 586 frames out of it. Now how about posting the footer from one of your uncorrupted files shot at the same settings?
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #27 on: November 09, 2017, 05:10:45 PM »

If I were to uplaod a working file with the same fps and resolution and roughly the same size/length would you be able use the script to find the footer ? 'cuase I don't really understand the finer details of most of what you said.

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #28 on: November 09, 2017, 05:24:34 PM »
Yes, just film a second or two with the exact same settings and upload it here.

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #29 on: November 09, 2017, 07:18:21 PM »
Right, the shortest RAW clip will still have the footer information. Note that it does have a field for the number of frames. I'm not sure if it is necessary to change that in order to extract all the frames. BTW-"magic" is "RAWM" marking the start of the footer and you can easily find with a hex editor.

Code: [Select]
/* file footer data */
typedef struct
{
    unsigned char magic[4];
    unsigned short xRes;
    unsigned short yRes;
    unsigned int frameSize;
    unsigned int frameCount;
    unsigned int frameSkip;
    unsigned int sourceFpsx1000;
    unsigned int reserved3;
    unsigned int reserved4;
    struct raw_info raw_info;
} lv_rec_file_footer_t;
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #30 on: November 09, 2017, 07:48:19 PM »

Here is the link to a piece of footage I recorded which is the same resolution and fps https://we.tl/hN34CqWHQK

Also do you guys know decent hex editor for MAC. 0xed won't let me copy and paste the hex data here properly

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #31 on: November 09, 2017, 08:20:34 PM »
I use hex fiend.

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #32 on: November 09, 2017, 09:29:30 PM »
I use iHex.

Will check it out when I get home.


Sent from my iPhone using Tapatalk
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

Roberto Mena

  • Freshman
  • **
  • Posts: 68
Re: Problem with playing back whole raw card
« Reply #33 on: November 09, 2017, 11:14:26 PM »
When this happens to me I just copy the MLV file to another hard drive (which I need to do anyways to back up) and that lets me play the MLV file via Mlviewer. I freaked out the first time this happened to me... haha.

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #34 on: November 10, 2017, 01:40:19 AM »

Much appreciated, Thanks

When this happens to me I just copy the MLV file to another hard drive (which I need to do anyways to back up) and that lets me play the MLV file via Mlviewer. I freaked out the first time this happened to me... haha.
Tried that, unfortunately it doesn't work.

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #35 on: November 10, 2017, 04:57:38 AM »
@Roberto Mena - That might work with MLV files but the problem is with RAW files.

I worked on it some more. Modified my script to look at some more fields. Chewed on the old topic that Danne pointed out - How to replace file footer, aka fix the Error: This ain't a lv_rec RAW file but haven't had any luck getting it looking any better than what was already posted.


5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10715
  • 5D Mark Free
Re: Problem with playing back whole raw card
« Reply #36 on: November 10, 2017, 01:51:29 PM »
From the file size (1836302528 = 0x6d73c0c0), your large file has a footer (size = 0xC0); however, as others noted, the metadata from this footer does not make any sense.

Since you were able to play back the file in the camera, my best guess was that something happened while copying the file (but the file size was correct), or some sort of filesystem corruption.

With the footer from M07-1720 (file size 547987648 = 0x20a9a0c0), the first frame is split in two, similar to this, but the bottom half of the frame was good. The shift amount was 580 lines x 512 columns = 1920*14/8*580 + 512*14/8 = 1949696 bytes.

Code: [Select]
dd if=M07-1720.RAW of=footer.raw bs=1 skip=$((0x20A9A000))
cat M06-1301.RAW footer.raw > M06-1301f.RAW
dd if=M06-1301f.RAW of=recovered.raw skip=1 bs=$((1920*14/8*580+512*14/8))

=> first half of the clip recovered.

The image breaks after a few seconds - raw data is now off by 1 byte (!):
Code: [Select]
dd if=M06-1301f.RAW of=recovered2.raw skip=1 bs=$((1920*14/8*580+512*14/8+1))

=> second half of the clip recovered.

BTW - when the image data is broken like this, you can try offsets from 1 to 13 bytes - one of them should give clean image.

Why 13? The 14-bit raw format packs 8 pixels in 14 bytes. That means, the offset until the next group of pixels (aligned at byte boundary) will be from 0 to 13. Since 0 is not the right one, you'll have to try the offsets from 1 to 13.

ML version used? (just curious)

dfgh

  • New to the forum
  • *
  • Posts: 33
Re: Problem with playing back whole raw card
« Reply #37 on: November 14, 2017, 09:35:24 AM »
Hi a1ex thanks for looking into this. I was using quite an old version of ML on that one card that I had forgot to update, I can certainly find out which one it was.


So are you saying that there's a fix a for this and if so, how would I go about it? I'm not saying that this is beyond my level of understanding but I've just never had to fix a clip like this before, so if you could run me through how to do it then I'd appreciate it, presumably the code supplied goes into terminal? Thanks!

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #38 on: November 14, 2017, 08:33:00 PM »
...presumably the code supplied goes into terminal?

Yes. Put the two files you supplied into a directory, "cd" into that directory with the terminal and run those commands. Here is what you should see. On the first pass which results in a file named "recovered.raw" frames 0 - 96 should be fine.



It breaks on frame 97.



Everything after that is garbled but running that next command that creates "recovered2.raw" frames 98 -150 will look something like this.



I'm noticing some vertical stripes on those frames so there's some more work to be done--or maybe it is because of the way my system is configured (Mac 10.13.1). Here's a closer look:



In addition, the footer on your M07-1720.RAW file has 151 frames in it and your M06-1301.RAW has more frames in it, though we don't know exactly how many more. I did some more work on my simple raw_footer.sh script so it prints out some additional information. Note there is a lot more in that footer that the script doesn't show.

Code: [Select]
./raw_footer.sh footer.raw

RAW footer:
00000000: 52 41 57 4d 80 07 38 04 00 60 37 00 97 00 00 00  RAWM..8..`7.....
00000010: 01 00 00 00 a8 61 00 00 d8 47 04 00 00 00 00 00  .....a...G......
00000020: 01 00 00 00 00 a0 31 4d 26 05 00 00 20 08 00 00  ......1M&... ...
00000030: 38 0e 00 00 50 34 49 00 0e 00 00 00 fe 07 00 00  8...P4I.........
00000040: 98 3a 00 00 00 00 00 00 00 00 00 00 8c 07 00 00  .:..............
00000050: 0a 05 00 00 1c 00 00 00 92 00 00 00 26 05 00 00  ............&...
00000060: 1e 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02  ................
00000070: 01 00 00 00 42 1a 00 00 10 27 00 00 85 fd ff ff  ....B....'......
00000080: 10 27 00 00 3d fc ff ff 10 27 00 00 41 ef ff ff  .'..=....'..A...
00000090: 10 27 00 00 ac 30 00 00 10 27 00 00 ec 07 00 00  .'...0...'......
000000a0: 10 27 00 00 74 fc ff ff 10 27 00 00 72 08 00 00  .'..t....'..r...
000000b0: 10 27 00 00 24 16 00 00 10 27 00 00 20 04 00 00  .'..$....'.. ...

Magic:
0x5241574d
00000000: 52  R
00000001: 41  A
00000002: 57  W
00000003: 4d  M

Width in hexidecimal:
0x8007

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
32775

Converted from little endian encoding:
0x0780

Image width taken from the raw footer
from original hex through conversion to decimal:
0x8007
0x0780
1920

Image height taken from the raw footer:
0x3804
1080

Frame size:
0x00603700
3629056

Frame count:
0x9700
151

Frame skip:
0x000001000000
65536

Frames per second:
0xa861
25.000
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10715
  • 5D Mark Free
Re: Problem with playing back whole raw card
« Reply #39 on: November 14, 2017, 08:59:41 PM »
If you use raw2dng to convert recovered2.raw, comment out fix_vertical_stripes(). It analyzes the first frame, which is not valid, so the correction code ends up actually introducing the defect.

Better yet - just delete the extra byte. Its location is 97*3629056 + 534*1920*14/8 - 110*14/8 = 0x1516c000 (first magic number is frame size as printed by raw2dng, second is the number of good lines from frame 97 - count the pixels - and third is number of bad columns on the right after fixing an integer number of lines, again found by counting the pixels).

Easiest way: open recovered.raw in a hex editor, navigate to offset 0x1516c000 and delete one byte (it will be a B7 in this particular file). Also change the number of frames in the footer to something very high (e.g. replace 97 00 with 97 FF) Save, run raw2dng (unmodified) and all the 505 frames will be correct.

What's puzzling me: the offset is multiple of 16384, but not a multiple of line size (1920*14/8) and ends up in the middle of the frame. Even though there's an extra byte introduced, the total file size is correct (integer number of frames + footer). What exactly happened is a mystery to me.

For other files, you'll need to count the pixels and do the math, as the magic numbers will be different.

BTW - have you found what ML version was used?

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #40 on: November 14, 2017, 09:26:45 PM »
Another method is using mlv_dump to convert the RAW file to MLV and process the DNG files with the "--no-stripes" option. For those unfamiliar with the command line it can also be done in Danne's Switch app.

Pick option "01":


Then run it again and choose "m" for the mlv_dump settings:


There's the "do not fix vertical stripes in highlights" option:


Works like magic.
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103

Danne

  • Hero Member
  • *****
  • Posts: 3612
Re: Problem with playing back whole raw card
« Reply #41 on: November 14, 2017, 10:52:41 PM »
Cool  8)

dfort

  • Hero Member
  • *****
  • Posts: 2316
Re: Problem with playing back whole raw card
« Reply #42 on: November 15, 2017, 06:41:13 AM »
Easiest way: open recovered.raw in a hex editor, navigate to offset 0x1516c000 and delete one byte (it will be a B7 in this particular file).

Amazing - this fixed that one broken frame and there's no need to mess around with vertical stripes.

Also change the number of frames in the footer to something very high (e.g. replace 97 00 with 97 FF) Save, run raw2dng (unmodified) and all the 505 frames will be correct.

Well that filled up my drive in a hurry. Changed it to F9 01 (little endian encoding for 505 decimal) and that garbled up RAW file is as good as new.

https://www.dropbox.com/sh/sfo9mf2bu3n4ffb/AACD1ChpyR27FU9-2cdCLCTJa?dl=0

@dfgh - You should have all you need to fix those corrupted files.

One more thing--

BTW - have you found what ML version was used?
5D3.* 7D.* 700D.115 EOSM.* EOSM2.103