Is there a way to easily repair corrupted frames ?

[Rojiraan]

Hi all !

So I just received some .MLV files from a short. While creating proxies under Resolve for the editing, I saw that Resolve was stopping the process only with certain files, showing a unknown error.
After looking a little bit deeper, I understood that some files were corrupted.
Also, this is a mess, because some files seems to be only "slightly" corrupted (like this or this) (so Resolve can continue the convert process without crashing), while others are completely corrupted (so Resolve need to stop, showing a "Unknown error" message). Which means that this is really complicated for me to find where the corrupted frames are (for sure I can't stay in front of my computer during the 6 hours needed to create the proxies ).
I didn't see anything that could help me inside the MLVFS settings, but I found a topic showing that it's possible to edit the HEX to find the problem (I found mlv_dump here, but I don't really understand how to locate the HEX line).

Do you know if there is anything available to easily repair these corrupted frames on multiple files ? And what could be the exact cause ?
Thanks a lot !!

PS: Shooting with 5D3 firmware 1.2.3 | magiclantern-Nightly.2015Dec20.5D3123 | Global Draw On (used for 4/3 cropmarks, I imagine this could be the cause...?) | SanDisk Extreme Pro 64 Go 160mbps

[a1ex]

The second example ("pinky.jpg") should be recoverable. Can you upload a DNG?

[Rojiraan]

Thanks for your fast answer !

Yup, here is a sample file : https://drive.google.com/file/d/0B4ljLoZMs0moOGlUaGEybEI2Nzg/view?usp=sharing

And I finally succeed to use mlv_dump.exe on a mlv file, but using this command : "mlv_dump -F 1 --dng name_of_the_mlv.mlv", I don't have any message indicating some HEX values.

Edit : Sorry I uploaded the wrong file (the 2 "slightly corrupted" examples weren't from mine), I'm actually looking for a file corrupted like the "pinky.jpg" one, I saw something like this during the convert process on Resolve, but I don't remember exactly which file it is. By the way, you mean that this isn't possible to recover a file looking like the one I uploaded ? Thanks again !)

[a1ex]

Indeed, I'm afraid the file you uploaded isn't recoverable. Here's what I've got:

[Rojiraan]

Yes, here it is : https://drive.google.com/file/d/0B4ljLoZMs0moYl91WkY2VnlBY1U/view?usp=sharing
The command was : mlv_dump.exe -v -b 14 --dng M02-0353.MLV
And here is a screenshot of the concerned frame (is this the log you wanted or do I need to export something else ?) : http://i.imgur.com/EvirECD.jpg

Thanks a lot for the help !

Edit : Argh    By the way, which is the exact way to modify the hexa values to get these results ? Is there any tutorial available ? And what could be the cause of this ? Thanks !

[a1ex]

I've edited mlv_dump to add a constant offset to the buffer address; here's what I've changed:

Code Select Expand


--- a/modules/mlv_rec/mlv_dump.c
+++ b/modules/mlv_rec/mlv_dump.c
@@ -1081,6 +1081,7 @@
     /* long options */
     int chroma_smooth_method = 0;
     int black_fix = 0;
+    int frame_offset = 0;
     enum bug_id fix_bug = BUG_ID_NONE;
     int fix_bug_1_offset = 0;
     int fix_bug_2_offset = 0;
@@ -1091,6 +1092,7 @@
     struct option long_options[] = {
         {"lua",    required_argument, NULL,  'L' },
         {"black-fix",  optional_argument, NULL,  'B' },
+        {"raw-offset", required_argument, NULL, 'O' },
         {"fix-bug",  required_argument, NULL,  'F' },
         {"batch",  no_argument, &batch_mode,  1 },
         {"dump-xrefs",   no_argument, &dump_xrefs,  1 },
@@ -1116,7 +1118,7 @@
     }

     int index = 0;
-    while ((opt = getopt_long(argc, argv, "A:F:B:L:txz:emnas:X:I:uvrcdo:l:b:f:", long_options, &index)) != -1)
+    while ((opt = getopt_long(argc, argv, "A:F:B:O:L:txz:emnas:X:I:uvrcdo:l:b:f:", long_options, &index)) != -1)
     {
         switch (opt)
         {
@@ -1155,7 +1157,11 @@
                     black_fix = MIN(16384, MAX(1, atoi(optarg)));
                 }
                 break;
-               
+           
+            case 'O':
+                frame_offset = atoi(optarg);
+                break;
+           
             case 'A':
                 if(!optarg)
                 {
@@ -2398,6 +2404,12 @@
                             raw_info = lv_rec_footer.raw_info;
                             raw_info.frame_size = frame_size;
                             raw_info.buffer = frame_buffer;
+                           
+                            if (frame_offset > 0)
+                            {
+                                printf("Raw offset: %d\n", frame_offset);
+                                memcpy(raw_info.buffer, raw_info.buffer + frame_offset, frame_size - frame_offset);
+                            }

                             /* override the resolution from raw_info with the one from lv_rec_footer, if they don't match */
                             if (lv_rec_footer.xRes != raw_info.width)
@@ -2420,11 +2432,11 @@
                             }

                             /* call raw2dng code */
-                            fix_vertical_stripes();
-                            find_and_fix_cold_pixels(fix_cold_pixels, block_hdr.frameNumber);
+                            //~ fix_vertical_stripes();
+                            //~ find_and_fix_cold_pixels(fix_cold_pixels, block_hdr.frameNumber);

                             /* this is internal again */
-                            chroma_smooth(chroma_smooth_method, &raw_info);
+                            //~ chroma_smooth(chroma_smooth_method, &raw_info);

                             /* set MLV metadata into DNG tags */
                             dng_set_framerate_rational(main_header.sourceFpsNom, main_header.sourceFpsDenom);
@@ -2482,6 +2494,7 @@
                             }

                             /* finally save the DNG */
+                            printf("pitch=%d\n", raw_info.pitch);
                             if(!save_dng(frame_filename, &raw_info))
                             {
                                 print_msg(MSG_ERROR, "VIDF: Failed writing into .DNG file\n");


Then I tried different numbers, knowing the pattern repeats every 14 bytes, and I ended up with offset 4 for the top section, and offset 5326 for the middle one, by trial and error.

[Rojiraan]

Wow, thanks a lot for posting this ! Seems complicated
So, this means I need the source code & Visual Studio to add/delete the lines you give me, then modify the "0" value (at the line "int frame_offset = 0;" if I understood well), then compile the .exe and re-convert the desired frame (with something like "mlv_dump.exe -f 2792-2793 -b 14 --dng M02-0353.MLV" for my case), if I'm correct.

Thanks again !

-----

Edit : After some searches and the help from a friend I understood that the code you gave me is a patch file. I tried to download the source code available at the link provided in my first post (by the way, is this the "real"/up to date source code ? I tried to find a g3gg0 (the mlvdump dev if I'm correct) Git, but found nothing. Or maybe that the project isn't open source anymore ?), but the mlv_dump.c file don't match with your patch file, so I edited the file manually. By the way, I can't use the Makefile (I tried using GnuWin32 Make.exe, didn't work).

[a1ex]

The patch is for mlv_dump.c from the current ML source tree. You need to download the entire repository in order to compile it.

https://bitbucket.org/hudson/magic-lantern/src/tip/modules/mlv_rec/

[Rojiraan]

Okay perfect ! Thanks a lot ! Sorry for the noobism, I'm not really into C development, but I start to understand little by little. I had downloaded WinGnu32 Make.exe, but I didn't succeed to compile the soft with it. So I'm actually getting Ubuntu, and I'll try to get all the libs needed to compile without problems.

Also, is there a difference between correcting the corrupted frame with a mlv_dump modification and correcting using a Hex modifier software like g3gg0 did before ?
And what could be the cause of these corrupted frames ? Is this linked to the Magic Lantern build, or something else ?

[a1ex]

The thread you linked shows a different issue - invalid MLV headers, caused by a bug in mlv_rec.

For this one, I think you could achieve the same thing by editing frameSpace in the VIDF header of the affected frame (see the definition of mlv_vidf_hdr_t in mlv.h). Didn't try this method though.

This artifact seems to be caused by the raw buffer becoming corrupted by skipping some data every now and then. Cause is unknown, I can reproduce it on 60D with this method, but that's pretty much it. There are lots of threads about it, but no definitive answer yet.

The latest 1.1.3 builds contain a possible fix, and the reports were good so far, though it's not clear to me why that "fix" appears to work.

[Rojiraan]

Okay, so if I understand correctly, this is representing invalid MLV headers, and this is a totally corrupted file.

I will check mlv.h & this last Nightly.2016May07 build asap, thanks again for the help !!

[a1ex]

Right - the first file had invalid MLV headers, so at some point, mlv_dump thought that frame begins somewhere in the middle of the real frame.

On the second file, I think it's still worth trying the offset trick. While playing with the offsets, some of them actually gave that kind of noise.

[Rojiraan]

Okay I see, I will try playing with the "int frame_offset = 0;" line when finishing to install Ubuntu and getting all the libs needed.
By the way, do you think it's possible to call this line directly inside the command window, while launching mlv_dump.exe ?

I was thinking that maybe adding something like this could work :

Code Select Expand

case 'w':
{
frame_offset = MIN(99999999999, MAX(0, atoi(optarg)));
}
break;


(I don't even know if this code is correct, 'just playing around )

[a1ex]

+            case 'O':
+                frame_offset = atoi(optarg);
+                break;

[Rojiraan]

!!! Awesome, I'll try this code asap, thank you very much !!

[Pastorino]

I had similar issues, especially on a project now that I shot 60fps and had very little time to test. my build is dec 20 and I shot on the limit of buffers because the takes were fast enough. It lasted more or less 30 secs before dropping frames and I was careless enough to not check global draw..

Anyways, from what I am seeing on this topic, there's no hope for it. Anyone confirms that? heres a ss and both corrupted dog's:

http://imgur.com/zU1js39
https://drive.google.com/open?id=0B88qiln0uIziWGZ4QTFHRjZla2M

Thanks a bunch!