Problem with playing back whole raw card

[Danne]

Yes, tried as well but all corruption. Updated my answer above.

Not sure if it´s possible to patch raw2dng to fix the files.

[dfort]

By copying this from and pasting it in the footer

You can't insert just anything in the footer. What you inserted was:

Code Select Expand

RAW footer:
6d73c15c: 52 41 57 4d 80 07 38 04 00 60 37 00 4a 02 00 00  RAWM..8..`7.J...
6d73c16c: 01 00 00 00 a8 61 00 00 d8 47 04 00 00 00 00 00  .....a...G......
6d73c17c: 01 00 00 00 00 a0 31 4d 26 05 00 00 20 08 00 00  ......1M&... ...
6d73c18c: 38 0e 00 00 50 34 49 00 0e 00 00 00 fe 07 00 00  8...P4I.........
6d73c19c: 98 3a 00 00 00 00 00 00 00 00 00 00 8c 07 00 00  .:..............
6d73c1ac: 0a 05 00 00 1c 00 00 00 92 00 00 00 26 05 00 00  ............&...
6d73c1bc: 1e 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02  ................
6d73c1cc: 01 00 00 00 42 1a 00 00 10 27 00 00 85 fd ff ff  ....B....'......
6d73c1dc: 10 27 00 00 3d fc ff ff 10 27 00 00 41 ef ff ff  .'..=....'..A...
6d73c1ec: 10 27 00 00 ac 30 00 00 10 27 00 00 ec 07 00 00  .'...0...'......
6d73c1fc: 10 27 00 00 74 fc ff ff 10 27 00 00 72 08 00 00  .'..t....'..r...
6d73c20c: 10 27 00 00 24 16 00 00 10 27 00 00 f4 03 00 00  .'..$....'......

Width in hexidecimal:
8007

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
32775

Converted from little endian encoding:
0780

Image width taken from the raw footer:
1920

Image height taken from the raw footer:
1080
a861
61a8
25000
25.000


So if this wasn't the settings you used when you shot it you are going to get corrupt footage. The above output is from a little script I wrote a while back when trying to figure out RAW footers. I never took the time to complete the script but it does give you an idea what is in the footer. You can read how this came about in the Dealing with Focus Pixels topic starting here.

You need to take one of your uncorrupted RAW files and run it through this script to figure out what you should be pasting into those corrupted files.

The corrupted file you uploaded is missing the footer. What the script read from your file is pretty much just gibberish:

Code Select Expand

RAW footer:
6d73c000: ab 8a ad 4a c6 ad 30 97 14 ae 4a 14 49 b4 a8 4a  ...J..0...J.I..J
6d73c010: a5 49 ab a9 c4 98 b4 bd 0a b4 49 44 aa ca 9f 49  .I........ID...I
6d73c020: bc a9 a8 a1 a4 98 4a 88 4a 54 ab 8a 95 4a bc ab  ......J.JT...J..
6d73c030: 68 a5 64 df 8a f4 49 24 ab 8a ad 4a b7 ab 14 a1  h.d...I$...J....
6d73c040: 44 b6 0a 04 4a f4 ab ca a3 49 da aa f0 9f f4 ba  D...J....I......
6d73c050: ca 74 49 d4 a9 4a 9c 49 a4 ab 64 a9 74 ac 8a 84  .tI..J.I..d.t...
6d73c060: 4a e4 aa ca a3 49 cd ab a8 ae 54 a9 0a e8 4a 94  J....I....T...J.
6d73c070: ac ca 99 49 bf a9 a4 9b 14 be ca b0 49 a4 ab 8a  ...I........I...
6d73c080: 9d 49 c9 a9 68 92 24 ce 4a 38 49 04 ab 0a a6 4a  .I..h.$.J8I....J
6d73c090: c0 aa 10 9b 34 ae 0a 3c 4a 74 ab 8a a3 4a c5 ab  ....4..<Jt...J..
6d73c0a0: 5c a1 44 c7 8a 84 4a 54 ac 0a a8 4a c5 aa a4 9d  \.D...JT...J....
6d73c0b0: 74 e3 ca 44 4a 54 ac 8a a2 4a da ad 68 a8 54 b7  t..DJT...J..h.T.

Width in hexidecimal:
c6ad

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
50861

Converted from little endian encoding:
adc6

Image width taken from the raw footer:
44486

Image height taken from the raw footer:
38704
c498
98c4
39108
39.108


Not sure what else is missing but I was able to extract 586 frames out of it. Now how about posting the footer from one of your uncorrupted files shot at the same settings?

[dfgh]

If I were to uplaod a working file with the same fps and resolution and roughly the same size/length would you be able use the script to find the footer ? 'cuase I don't really understand the finer details of most of what you said.

[Danne]

Yes, just film a second or two with the exact same settings and upload it here.

[dfort]

Right, the shortest RAW clip will still have the footer information. Note that it does have a field for the number of frames. I'm not sure if it is necessary to change that in order to extract all the frames. BTW-"magic" is "RAWM" marking the start of the footer and you can easily find with a hex editor.

Code Select Expand

/* file footer data */
typedef struct
{
    unsigned char magic[4];
    unsigned short xRes;
    unsigned short yRes;
    unsigned int frameSize;
    unsigned int frameCount;
    unsigned int frameSkip;
    unsigned int sourceFpsx1000;
    unsigned int reserved3;
    unsigned int reserved4;
    struct raw_info raw_info;
} lv_rec_file_footer_t;


[dfgh]

Here is the link to a piece of footage I recorded which is the same resolution and fps https://we.tl/hN34CqWHQK

Also do you guys know decent hex editor for MAC. 0xed won't let me copy and paste the hex data here properly

[Danne]

I use hex fiend.

[dfort]

I use iHex.

Will check it out when I get home.


Sent from my iPhone using Tapatalk

[Roberto Mena]

When this happens to me I just copy the MLV file to another hard drive (which I need to do anyways to back up) and that lets me play the MLV file via Mlviewer. I freaked out the first time this happened to me... haha.

[dfgh]

Much appreciated, Thanks

When this happens to me I just copy the MLV file to another hard drive (which I need to do anyways to back up) and that lets me play the MLV file via Mlviewer. I freaked out the first time this happened to me... haha.

Tried that, unfortunately it doesn't work.

[dfort]

@Roberto Mena - That might work with MLV files but the problem is with RAW files.

I worked on it some more. Modified my script to look at some more fields. Chewed on the old topic that Danne pointed out - How to replace file footer, aka fix the Error: This ain't a lv_rec RAW file but haven't had any luck getting it looking any better than what was already posted.

[a1ex]

From the file size (1836302528 = 0x6d73c0c0), your large file has a footer (size = 0xC0); however, as others noted, the metadata from this footer does not make any sense.

Since you were able to play back the file in the camera, my best guess was that something happened while copying the file (but the file size was correct), or some sort of filesystem corruption.

With the footer from M07-1720 (file size 547987648 = 0x20a9a0c0), the first frame is split in two, similar to this, but the bottom half of the frame was good. The shift amount was 580 lines x 512 columns = 1920*14/8*580 + 512*14/8 = 1949696 bytes.

Code Select Expand


dd if=M07-1720.RAW of=footer.raw bs=1 skip=$((0x20A9A000))
cat M06-1301.RAW footer.raw > M06-1301f.RAW
dd if=M06-1301f.RAW of=recovered.raw skip=1 bs=$((1920*14/8*580+512*14/8))


=> first half of the clip recovered.

The image breaks after a few seconds - raw data is now off by 1 byte (!):

Code Select Expand


dd if=M06-1301f.RAW of=recovered2.raw skip=1 bs=$((1920*14/8*580+512*14/8+1))


=> second half of the clip recovered.

BTW - when the image data is broken like this, you can try offsets from 1 to 13 bytes - one of them should give clean image.

Why 13? The 14-bit raw format packs 8 pixels in 14 bytes. That means, the offset until the next group of pixels (aligned at byte boundary) will be from 0 to 13. Since 0 is not the right one, you'll have to try the offsets from 1 to 13.

ML version used? (just curious)

[dfgh]

Hi a1ex thanks for looking into this. I was using quite an old version of ML on that one card that I had forgot to update, I can certainly find out which one it was.


So are you saying that there's a fix a for this and if so, how would I go about it? I'm not saying that this is beyond my level of understanding but I've just never had to fix a clip like this before, so if you could run me through how to do it then I'd appreciate it, presumably the code supplied goes into terminal? Thanks!

[dfort]

...presumably the code supplied goes into terminal?

Yes. Put the two files you supplied into a directory, "cd" into that directory with the terminal and run those commands. Here is what you should see. On the first pass which results in a file named "recovered.raw" frames 0 - 96 should be fine.



It breaks on frame 97.



Everything after that is garbled but running that next command that creates "recovered2.raw" frames 98 -150 will look something like this.



I'm noticing some vertical stripes on those frames so there's some more work to be done--or maybe it is because of the way my system is configured (Mac 10.13.1). Here's a closer look:



In addition, the footer on your M07-1720.RAW file has 151 frames in it and your M06-1301.RAW has more frames in it, though we don't know exactly how many more. I did some more work on my simple raw_footer.sh script so it prints out some additional information. Note there is a lot more in that footer that the script doesn't show.

Code Select Expand

./raw_footer.sh footer.raw

RAW footer:
00000000: 52 41 57 4d 80 07 38 04 00 60 37 00 97 00 00 00  RAWM..8..`7.....
00000010: 01 00 00 00 a8 61 00 00 d8 47 04 00 00 00 00 00  .....a...G......
00000020: 01 00 00 00 00 a0 31 4d 26 05 00 00 20 08 00 00  ......1M&... ...
00000030: 38 0e 00 00 50 34 49 00 0e 00 00 00 fe 07 00 00  8...P4I.........
00000040: 98 3a 00 00 00 00 00 00 00 00 00 00 8c 07 00 00  .:..............
00000050: 0a 05 00 00 1c 00 00 00 92 00 00 00 26 05 00 00  ............&...
00000060: 1e 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02  ................
00000070: 01 00 00 00 42 1a 00 00 10 27 00 00 85 fd ff ff  ....B....'......
00000080: 10 27 00 00 3d fc ff ff 10 27 00 00 41 ef ff ff  .'..=....'..A...
00000090: 10 27 00 00 ac 30 00 00 10 27 00 00 ec 07 00 00  .'...0...'......
000000a0: 10 27 00 00 74 fc ff ff 10 27 00 00 72 08 00 00  .'..t....'..r...
000000b0: 10 27 00 00 24 16 00 00 10 27 00 00 20 04 00 00  .'..$....'.. ...

Magic:
0x5241574d
00000000: 52  R
00000001: 41  A
00000002: 57  W
00000003: 4d  M

Width in hexidecimal:
0x8007

Normal hexadecimal to decimal conversion doesn't work on little endian encoding:
32775

Converted from little endian encoding:
0x0780

Image width taken from the raw footer
from original hex through conversion to decimal:
0x8007
0x0780
1920

Image height taken from the raw footer:
0x3804
1080

Frame size:
0x00603700
3629056

Frame count:
0x9700
151

Frame skip:
0x000001000000
65536

Frames per second:
0xa861
25.000


[a1ex]

If you use raw2dng to convert recovered2.raw, comment out fix_vertical_stripes(). It analyzes the first frame, which is not valid, so the correction code ends up actually introducing the defect.

Better yet - just delete the extra byte. Its location is 97*3629056 + 534*1920*14/8 - 110*14/8 = 0x1516c000 (first magic number is frame size as printed by raw2dng, second is the number of good lines from frame 97 - count the pixels - and third is number of bad columns on the right after fixing an integer number of lines, again found by counting the pixels).

Easiest way: open recovered.raw in a hex editor, navigate to offset 0x1516c000 and delete one byte (it will be a B7 in this particular file). Also change the number of frames in the footer to something very high (e.g. replace 97 00 with 97 FF) Save, run raw2dng (unmodified) and all the 505 frames will be correct.

What's puzzling me: the offset is multiple of 16384, but not a multiple of line size (1920*14/8) and ends up in the middle of the frame. Even though there's an extra byte introduced, the total file size is correct (integer number of frames + footer). What exactly happened is a mystery to me.

For other files, you'll need to count the pixels and do the math, as the magic numbers will be different.

BTW - have you found what ML version was used?

[dfort]

Another method is using mlv_dump to convert the RAW file to MLV and process the DNG files with the "--no-stripes" option. For those unfamiliar with the command line it can also be done in Danne's Switch app.

Pick option "01":


Then run it again and choose "m" for the mlv_dump settings:


There's the "do not fix vertical stripes in highlights" option:


Works like magic.

[Danne]

Cool 

[dfort]

Easiest way: open recovered.raw in a hex editor, navigate to offset 0x1516c000 and delete one byte (it will be a B7 in this particular file).

Amazing - this fixed that one broken frame and there's no need to mess around with vertical stripes.

Also change the number of frames in the footer to something very high (e.g. replace 97 00 with 97 FF) Save, run raw2dng (unmodified) and all the 505 frames will be correct.

Well that filled up my drive in a hurry. Changed it to F9 01 (little endian encoding for 505 decimal) and that garbled up RAW file is as good as new.

https://www.dropbox.com/sh/sfo9mf2bu3n4ffb/AACD1ChpyR27FU9-2cdCLCTJa?dl=0

@dfgh - You should have all you need to fix those corrupted files.

One more thing--

BTW - have you found what ML version was used?