Unknown Block errors PLEASE HELP

Started by sax100, September 19, 2018, 04:54:55 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

sax100

I used 3x different KomputerBay CF cards 128GB 1000x cards and also tested problem with 3x different 64GB 1000x cards. Nearly every clip would only convert up to around 45-55 seconds and spit out 1-4 frames at the end of clip.

Is there something wrong with my CF cards?

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M15-0904.MLV'
   - Verify file structure
   - Output .idx file for faster processing
File M15-0904.MLV opened
File M15-0904.M00 opened
Processing...
Unknown Block: %%, skipping
Unknown Block: ;?H, skipping
Reached end of chunk 1/2 after 2059 blocks
Unknown Block: ????, skipping
Reached end of chunk 2/2 after 10 blocks
Processed 781 video frames
XREF table contains 1286 entries
File M15-0904.IDX opened for writing
Done
Canon 5D Mark III / KomputerBay 1000x 128GB CF
ML for Canon 5d3 Build - Nightly.2014May22.5D3113
Firmware Version 1.1.3-ml-v2.3

Danne

Please upload a file that won't convert.

sax100

Uploading 4GB affected file to Google Drive - another 1.5 hrs left. I'll provide link shortly.
Canon 5D Mark III / KomputerBay 1000x 128GB CF
ML for Canon 5d3 Build - Nightly.2014May22.5D3113
Firmware Version 1.1.3-ml-v2.3

sax100

Canon 5D Mark III / KomputerBay 1000x 128GB CF
ML for Canon 5d3 Build - Nightly.2014May22.5D3113
Firmware Version 1.1.3-ml-v2.3

dfort

Seeing the same issue with your file:

mlv_dump --dng --relaxed M15-0904.MLV

MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M15-0904.MLV'
   - Using output path 'M15-0904_' for DNGs
   - Convert to DNG frames
   - Output into 'M15-0904_'
File M15-0904.MLV opened
File M15-0904.M00 not existing.
Processing...
Unknown Block: %%, skipping
[ERROR] Invalid block size at position 0x7dd46afe
Processed 777 video frames at 24.00 FPS (32.38 s)
Done


Let's keep the help discussion going here and leave the intro topic for--well, introductions.

Taking a closer look:

Block: VIDF
  Offset: 0x7d8d2290
  Number: 2056
    Size: 2705488
    Time: 32341.457000 ms
   Frame: #0764
    Crop: 152x272
     Pan: 146x271
   Space: 4016
Block: %%
  Offset: 0x7db66ae0
  Number: 2057
    Size: 1966110
    Time: 6192543978356.757812 ms
Unknown Block: %%, skipping
[ERROR] Invalid block size at position 0x7dd46afe
Processed 777 video frames at 24.00 FPS (32.38 s)
Done


0x7dd46afe (hex) is 2111073022 (dec) so does that mean that the problems start here?



I'm not very knowledgable on saving corrupted MLV files so maybe someone else could give it a shot?

BTW--the video and audio frames don't match, that might be a problem.

File Header (MLVI)
...
    Frames Video: 1586
    Frames Audio: 330

Danne

Can´t download the file over here, too big but what happen if ou select no audio and relaxed setting in mlv_dump?
You can probably test these settings in Switch(mac), batch_mlv(windows) and/or Mlv App.

dfort

Same issue:

mlv_dump --dng --relaxed --no-audio M15-0904.MLV

...
Skipping 'AUDF' block
Skipping 'AUDF' block
Skipping 'AUDF' block
Skipping '%%' block
[ERROR] Invalid block size at position 0x7dd46afe
Processed 777 video frames at 24.00 FPS (32.38 s)
Done

reddeercity

I'm downloading now , I'll have a look -- I've repaired a few file before with HxD hex editor & mlvdiag.1.0.1 (creator of Raw2Cdng)

reddeercity

Ok , so what's the problem with the file ?
I see no problem with the mlv I downloaded , Audio & Cdng export fine here with MLVProducer .
1920x804 , 777 Cdng frames & 32 second of audio



I quickly ran the file thought MLVProducer with a lite grad in sRGB color space and saved a ffmpeg prores 4444
No problem extracting audio also , same length as video file .
Just tested MLVFS on quick mount in wins 7 all fine here , every thing exports as it should .

Edit: Tried the file with mlv_dump , did get a error message

I:\2018-9-19_problem_mlv_5d3\cdngs>mlv_dump --dng M15-0904.mlv
MLV Dumper
-----------------
Mode of operation:
   - Input MLV file: 'M15-0904.mlv'
   - Using output path 'M15-0904_' for DNGs
   - Convert to DNG frames
   - Output into 'M15-0904_'
File M15-0904.mlv opened
File M15-0904.m00 not existing.
Processing...
Unknown Block: % % , skipping
[ERROR] Invalid block size at position 0x7dd46afe
Processed 777 video frames at 24.00 FPS (32.38 s)
Done

cdng's extracted normally with audio , thou it was 4 second shorter then mlvfs & mlvproducer those where 32 second .
Can't see any other problems , maybe a bad block on card or the  trim function on the CF card didn't erase the block fully .
https://en.wikipedia.org/wiki/CompactFlash
QuoteCompactFlash Revision 6.0 (November 2010) added UltraDMA Mode 7 (167 MByte/s), ATA-8/ACS-2 sanitize command, TRIM and an optional card capability to report the operating temperature range of the card
QuoteThe TRIM command enables an operating system to notify the SSD of pages which no longer contain valid data. For a file deletion operation, the operating system will mark the file's sectors as free for new data  .... 

dfort

The problem is that there are 1586 frames in that file.

reddeercity

ok I see it now ,
Edit: in MLV autopsy when I run this
mlv_dump --dng --skip-block 0x7dd46afe M15-0904.mlv

I get this
Processing...
[ERROR] Missing file header
Processed 0 video frames at -1.#J FPS (-1.#J s)
Done

with out skip block it converts the 777 frames , strange
Damage file header ?
I don't have a 5d3 to make mlv file for the header & replace it with a different header .

Danne

Here are two versions of mlv_dump which I removed the "goto abort" code. Could you test and process the file with one of these?
Mac:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump

Windows:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump.exe

reddeercity

No , tried windows version still same issue -- There one thing that puzzle me there a Alignment fill after every frame block .





Is this one of those null bugs I remember hearing about about half year ago ?
I don't see this on 5d2


Danne

Tried some more, downloaded the file but seems corrupted. Checking output in MlRawViewer shows black image after dng 777:

a1ex

When all else fails, brute force prevails. Put this at the end of this script:


def read_block_bf(f):
    data = ""
    count = 0
    while True:
        chunk = f.read(1024 * 1024);
        if len(chunk) == 0: return
        data += chunk

        next_vidf = data.find("VIDF")
        if next_vidf >= 0:
            f.seek(next_vidf - len(data), 1)
            data = ""
            block_header = f.read(mlv_block_hdr.size)
            blocktype, blocksize, timestamp = mlv_block_hdr.unpack_from(block_header)
            f.seek(-mlv_block_hdr.size, 1);
            count += 1
            print hex(f.tell()) + (" [%d]:" % count),
            block = f.read(blocksize)
            yield block

with open(sys.argv[1], 'rb') as f:
    [parse_block(block) for block in read_block_bf(f)]



0x7c6bf260 [770]: Block VIDF (2705536 bytes)
0x7c954260 [771]: Block VIDF (2705536 bytes)
0x7cbe9280 [772]: Block VIDF (2705504 bytes)
0x7ce7e260 [773]: Block VIDF (2705536 bytes)
0x7d113280 [774]: Block VIDF (2705504 bytes)
0x7d3a8260 [775]: Block VIDF (2705536 bytes)
0x7d63d260 [776]: Block VIDF (2705536 bytes)
0x7d8d2290 [777]: Block VIDF (2705488 bytes)
0x7db67280 [778]: Block VIDF (2705504 bytes)
0x7ddfc260 [779]: Block VIDF (2705536 bytes)
0x7e091260 [780]: Block VIDF (2705536 bytes)
0x7e326280 [781]: Block VIDF (2705504 bytes)
0x7e5bb260 [782]: Block VIDF (2705536 bytes)
0x7e850280 [783]: Block VIDF (2705504 bytes)
0x7eae5260 [784]: Block VIDF (2705536 bytes)
0x7ed7a280 [785]: Block VIDF (2705504 bytes)
...
0xffc85280 [1582]: Block VIDF (2705504 bytes)


So, it looks like the video frames are there. Adding this into mlv_dump (diff from crop_rec_4k branch):

diff -r 0d39d995cb65 modules/mlv_rec/mlv_dump.c
--- a/modules/mlv_rec/mlv_dump.c Tue Sep 18 07:54:06 2018 +0200
+++ b/modules/mlv_rec/mlv_dump.c Thu Sep 20 10:41:34 2018 +0200
@@ -1239,6 +1239,33 @@
     }
}

+static int seek_to_next_known_block(FILE * in_file)
+{
+    int read_ahead_size = 128 * 1024 * 1024;
+    uint8_t * ahead = malloc(read_ahead_size);
+    assert(ahead);
+
+    int read = fread(ahead, 1, read_ahead_size, in_file);
+    /* fixme: file_set_pos doesn't work */
+    fseek(in_file, -read, SEEK_CUR);
+    for (int i = 0; i < read; i++)
+    {
+        if (memcmp(ahead + i, "VIDF", 4) == 0 ||
+            memcmp(ahead + i, "AUDF", 4) == 0 ||
+            memcmp(ahead + i, "NULL", 4) == 0)
+        {
+            print_msg(MSG_INFO, "Next known block: %c%c%c%c at 0x%"PRIx64"+0x%x = ", ahead[i], ahead[i+1], ahead[i+2], ahead[i+3], file_get_pos(in_file), i);
+            fseek(in_file, i, SEEK_CUR);
+            print_msg(MSG_INFO, "0x%"PRIx64"\n", file_get_pos(in_file));
+            free(ahead);
+            return 1;
+        }
+    }
+
+    print_msg(MSG_ERROR, "Could not find any known block from 0x"PRIx64".\n", file_get_pos(in_file));
+    free(ahead);
+    return 0;
+}

int main (int argc, char *argv[])
{
@@ -2129,7 +2156,13 @@
             else
             {
                 print_msg(MSG_ERROR, "Invalid block size at position 0x%08" PRIx64 "\n", position);
-                goto abort;
+
+                /* attempt to seek to next valid block */
+                if (seek_to_next_known_block(in_file)) {
+                    goto read_headers;
+                } else {
+                    goto abort;
+                }
             }
         }
         
@@ -4080,6 +4113,11 @@
                 else
                 {
                     print_msg(MSG_INFO, "Unknown Block: %c%c%c%c, skipping\n", mlv_block->blockType[0], mlv_block->blockType[1], mlv_block->blockType[2], mlv_block->blockType[3]);
+
+                    /* this block might be bad; seek to next valid one */
+                    fseek(in_file, -mlv_block->blockSize, SEEK_CUR);
+                    seek_to_next_known_block(in_file);
+                    goto read_headers;
                 }

                 lua_handle_hdr(lua_state, mlv_block->blockType, "", 0);


Appears to work, but have to go, didn't check the DNGs.

cc @g3gg0 for cleanup and commit, possibly as an option.

Danne

Cool. Tried hg import but fail. Need to go now but a1ex provided the needed fix so...
Also pm:d g3gg0.
hg import /Users/dan/Desktop/patch.diff
applying /Users/dan/Desktop/patch.diff
patching file modules/mlv_rec/mlv_dump.c
Hunk #1 FAILED at 1238
Hunk #2 FAILED at 2128
Hunk #3 FAILED at 4079
3 out of 3 hunks FAILED -- saving rejects to file modules/mlv_rec/mlv_dump.c.rej
abort: patch failed to apply
dans-MacBook-Pro:magic-lantern dan$

a1ex

@Danne: "hg up crop_rec_4k -C" (or crop_rec_4k_mlv_snd) before "hg import". Or, for temporary patching:

patch -p1 < foo.patch


I've got your error message (with your line numbers minus 1) after trying to apply the patch twice. Guess: it either worked the first time, or you've got some incompatible changes in your repository.

Quote from: sax100 on September 17, 2018, 06:50:43 PM
ML for 5d3 2014.01.08 Build

Pretty sure it was this bug. You should have mentioned all the relevant info in this thread.

The DNGs near frame 777 look OK to me. The file seems to contain some more bad blocks, but this brute-forcing trick appears to jump over them.


Block: VIDF
  Offset: 0x7d8d2290
  Number: 2056
    Size: 2705488
    Time: 32341.457000 ms
   Frame: #0764
    Crop: 152x272
     Pan: 146x271
   Space: 4016
Block: %%
  Offset: 0x7db66ae0
  Number: 2057
    Size: 1966110
    Time: 6192543978356.757812 ms
Unknown Block: %%, skipping
Next known block: NULL at 0x7db66ae0+0x280 = 0x7db66d60
Block: NULL
  Offset: 0x7db66d60
  Number: 2057
    Size: 1280
    Time: 18446744073709552.000000 ms
Block: ELVL
  Offset: 0x7db67260
  Number: 2058
    Size: 32
    Time: 32884.321000 ms
     Roll:    1.00
     Pitch:   3.50
Block: VIDF
  Offset: 0x7db67280
  Number: 2059
    Size: 2705504
    Time: 32883.162000 ms
   Frame: #0777
    Crop: 152x272
     Pan: 146x271
   Space: 4032
...
[ERROR] Invalid block size at position 0xfff19ae0
Next known block: NULL at 0xfff19af0+0x270 = 0xfff19d60
Block: NULL
  Offset: 0xfff19d60
  Number: 4168
    Size: 1280
    Time: 18446744073709552.000000 ms


Reached end of chunk 1/1 after 4169 blocks
Processed 1582 video frames at 24.00 FPS (65.92 s)
Done

bouncyball

Here is repaired MLV. More than 100 errors showed up. Mostly VIDF block size mismatches (VIDF blocks had ~600bytes more at the end). One NULL block size mismatch.

Plays fine in MLV App. Some async/glitches with audio at the end of the clip (BTW audio level is very low).

Unfortunately MLV has no VERS info block. Hence can not identify which ML build and versions of modules were used.

ELVL blocks are all over there (after each VIDF/NULL pair).

Danne

Cool to see solutions.

@a1ex
No idea what´s up with hg here. Redownloaded magic-lantern repo and got into the mentioned branch:
dans-MacBook-Pro:magic-lantern dan$ hg branch
crop_rec_4k


First try:
patch -p1 < /Users/dan/Desktop/patch.diff
patching file modules/mlv_rec/mlv_dump.c
Hunk #1 FAILED at 1239.
Hunk #2 FAILED at 2156.
patch unexpectedly ends in middle of line
Hunk #3 FAILED at 4113.
3 out of 3 hunks FAILED -- saving rejects to file modules/mlv_rec/mlv_dump.c.rej


Erasing, importing vanilla magic-lantern repo, entering crop_rec_4k branch:
Second try:
dans-MacBook-Pro:magic-lantern dan$ hg import /Users/dan/Desktop/patch.diff
applying /Users/dan/Desktop/patch.diff
patching file modules/mlv_rec/mlv_dump.c
Hunk #1 FAILED at 1238
Hunk #2 FAILED at 2128
Hunk #3 FAILED at 4079
3 out of 3 hunks FAILED -- saving rejects to file modules/mlv_rec/mlv_dump.c.rej
abort: patch failed to apply
dans-MacBook-Pro:magic-lantern dan$


Will check and see if I can find an answer to this...

a1ex

Just to exclude copy/paste errors:

mlv_dump_bruteforce.patch (buggy, updated below)

Danne

Your patch file works! Successfully patched and processed the file.
Tried copy pasting from the forum post but gives errors:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/patch.diff

Copy pasting your content into the above file works. Some formatting issue? No clean answers checking mercurial forums.

Danne

Ok, here we go. Formatting issue. Run this in terminal:
unix2dos /Users/dan/Desktop/Timer_A/Mercurial/patch.diff
unix2dos: converting file /Users/dan/Desktop/Timer_A/Mercurial/patch.diff to DOS format...

Now it´s possible to add the patch on mac...sigh

hg import /Users/dan/Desktop/Timer_A/Mercurial/patch.diff --edit danne

At least kind of worked. I probably missed a line at the bottom:
dans-MacBook-Pro:magic-lantern dan$ hg import /Users/dan/Desktop/Timer_A/Mercurial/patch.diff --edit danne
applying /Users/dan/Desktop/Timer_A/Mercurial/patch.diff
patching file modules/mlv_rec/mlv_dump.c
Hunk #3 succeeded at 4114 with fuzz 1 (offset 0 lines).
abort: no username supplied
(use 'hg config --edit' to set your username)
dans-MacBook-Pro:magic-lantern dan$

sax100

Thanks for all of your diligent investigating of this problem!

Bare with me, I'm still learning my ABC's...

I tried to run the patches @a1ex and @Danne provided. I'm unsure if this the correct command?

mlv_dump patch < patch.patch M15-0904.MLV

Cause I got this error...

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'patch'
   - Verify file structure
[ERROR] Failed to open file 'patch'


Then got message from @Danne and since I'm on a MAC, I ran unix2dos command, but...

My mac (OSX 10.85) doesn't recognize unix2dos. Researched online and found [url https://brew.sh/]Homebrew[/url] to allow me to use unix2dos, but wouldn't install. Does anyone know why? Or is there another way of doing this?

==> Downloading and installing Homebrew...
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
Failed during: /bin/bash -o pipefail -c '/usr/bin/curl -fsSL https://github.com/Homebrew/brew/tarball/master | /usr/bin/tar xz -m --strip 1'


Canon 5D Mark III / KomputerBay 1000x 128GB CF
ML for Canon 5d3 Build - Nightly.2014May22.5D3113
Firmware Version 1.1.3-ml-v2.3

Danne