Problem with MLV in 5DMarkiii (PLEASE HELPME!!!)

[Jared]

Hi people, Today I was recording MLV with 5dMarkiii (BUILD= magiclantern-Nightly.2014Aug07.5D3123)
everything worked perfect, when I convert to dng, get these corrupt frames
I'm using MLV Converter 1.9.1, and Mlrawviewer 1.4.2
could someone help me, PLEASE?

[DeafEyeJedi]

PM me one of your MLV files and I'll see if I can duplicate your "issues"...

I can only speak for Mac's...

[Jared]

@DeafEyeJedi Ok, now upload 1 file, thanx for response

[Jared]

share one of the dng
all help will be appreciated

https://www.dropbox.com/s/4kavri9vaxhl87d/MLV_A001_101233_C00477.000000.dng?dl=0

[ali farahani]

Same problem with me sometimes that when CF card has a little space untill will be full. last shot usualy

[Jared]

there any chance of recovering those files???

[budafilms]

Those files, as you uploaded, are lost.
I had a couple of that.


Edited: continuing reading, there's a possible solution if you have this problem!

[Jared]

@budafilms really? I CAN'T BELIEVE IT!!!!!!!!!!!   
and this because it happens? I have over 1 year using MagicLantern and I had never happened this

[g3gg0]

tried mlv_dump ?

[Jared]

@g3gg0, as should use the mlv_dump? like to BlackFix?
if you could guide me or help, I appreciate

[Jared]

This is one of the damaged files
https://www.dropbox.com/s/zalvkps6va5dskt/M10-1234.MLV?dl=0

[dmilligan]

Yeah, that entire file is just garbage data. There is nothing to recover.

[budafilms]

@Jared I really sorry.

It happens as other say when you have very few space in your card. And is not often. In 2500 shots I get 2 or 3 times, in two years. Nothing.
The good is when this happened to me, I saw in the view finder that something goes wrong, and I have the luck to reproduce the take.

Good takes always back.

Edited: Under this post are a possible solution, continuing reading!

[g3gg0]

sorry to say that dmilligan, but we can recover that

[g3gg0]

get mlv_dump and execute this:

mlv_dump -F2,-2 -o fixed.mlv defect.mlv

make sure:
- you execute it according to your OS's requirements (./ on unix systems)
- pick the binary name for your OS (linux/exe/osx)
- you replace <defect.mlv> with the name of the file to fix
- dont have a typo

[Jared]

WOOOOOOOOOOOOOOOWWWW!!!!!!! @g3gg0 u rock friend!!! u save my life!!!!! THAAANNNNNXXXX!!!!!!! 

[DeafEyeJedi]

@Jared:

See all you had to do was keep on believing and have faith in what the guys do here in this incredible community... Congrats!

*cheers* to everyone!

[Jared]

For me it is a pride and honor to have the cooperation of all of you ...
you too thank you very much for being outstanding @DeafEyeJedi

[a1ex]

For those who experience this problem, and are running 5D3.123: can you downgrade to 1.1.3 and see if the issue persists?

I had major trouble with corrupted frames when porting ML to 1.2.3 (problems that could not be reproduced on 1.1.3), but I'm not sure if the issue is still valid.

[Jared]

@a1ex I change the firmware because I'm using HDMI monitor for some recordings, right now I 'm remembering , the problem may be when using zoom to focus ( I think it was the only thing I did differently ) ( when I zoom to focus on the screen of the camera some errors were seen )

[g3gg0]

okay back and have some time.
about the problem that your footage had:

looking at the raw VIDF frame payload, it didnt show a totally random pattern as you would expect from the footage.
checking the byte value distribution, there were spikes at the usual positions 0x08, 0x20, 0x48, 0x82, 0x88, 0xC8 and lower spikes at 0x02..0xF2 in step widths of 0x10.
thats what ive seen before in a lot of other footage. depending on overexposure you get a lot more noise, but the general distribution is the same.
you can see that as a fingerprint of the raw data. (no matter if .raw or .mlv)



your footage had exactly the fingerprint i would expect, so it had to be good data.
checking the image you posted, you will notice a "scratch" in the top area, a bit left of the center.
that also indicates that there is some useful data in your footage.

so i remembered back that MlvViewerSharp had a byte-offset trick that allowed to skip some bytes of the raw payload.
playing with that revealed that the payload simply "misses" two bytes at its start, so all footage data is being interpreted incorrectly.
the LSB area, which represents "noise" now suddenly is at the MSB position.
so the thermal/iso/etc noise is amplified and the real information is reduced, resulting in "pure noise" making all of us think that the footage only contains garbage.


yet the reason of this fault is unknown.
can someone reproduce that? it only happens with v1.2.3 ?

[Jared]

in my case happened with firmware 1.2.3 ( magiclantern-Nightly.2015Mar16.5D3123.zip )
if it serves as a guide for the " damaged " shots , I zoom to focus (I see problems like pink frames on the screen of the camera )

[a1ex]

Magic Zoom? What settings?

[Jared]

@a1ex no magic zoom, canon zoom + focus peaking (Balanced, Low Res, 1%, Local Focus) Cropmark (cinesco2.bmp)
Always use this configuration for video. to focus the scene.

[luchobaz]

Hi guys! I am with the same problem than Jared experimented, but in my case with a RAW file. I have tried mlv-dump but it doesnt works, I think because it is not a MLV file.
Could you please give me an advice? I have an amazing shots blocked 

Thanks a lot! every thing is a lot for me now!

[g3gg0]

if it is plain .raw you can use a hex editor to paste some bytes at the beginning of the file.
PLEASE WORK WITH BACKUP FILES

as most hex editors do not allow editing such large files, you can concatenate files:
e.g. get HxD and create 7 files. one file with only one byte data (its ok to use 0x00 or 0xFF as data).
and the others with each one byte more.
name them dummy_1.bin, dummy_2.bin, etc according to the number of bytes it has.

and then use the copy command to concatenate either dummy_1.bin and your faulty.raw

windows:
copy /b dummy_1.bin faulty.raw try_1.raw
copy /b dummy_2.bin faulty.raw try_2.raw
etc.

and for every file check if it contains valid data.

[luchobaz]

Thank you so much for the quick answer!!
I have tried concatenating files with command, but it doesnt work, because I didnt use Hxd to change de beginning of the files. I have tried concatenante those in HEX but as you said, is not possible because are to large.

About your instructions, I understand that I need to create o new HEX with only 0x00 or 0xFF (that I will name "dummy_1.bin"). Then to concatenate this one with the RAW file (backup) in command. Its right? Or I am missing some step?
I dont understand when you say the others files with one byte more. What do you mean? (sorry I am totally new in HEX).

Thank you! Now I feel that I do not lose hope!

[g3gg0]

dummy_1.bin: 00
dummy_2.bin: 00 00
dummy_3.bin: 00 00 00
...
dummy_7.bin: 00 00 00 00 00 00 00

then concatenate those files so you will get 7 files:
00 <original raw data>
00 00 <original raw data>
00 00 00 <original raw data>
...
00 00 00 00 00 00 00 <original raw data>

and try which file will work properly

[luchobaz]

Ok! Thx! I will tray it and report the results!
(I am uploading an example, RAW failed file, right now)

[luchobaz]

  ]
Ok, I have tried mergging
copy/b dummy_1.bin+failedfile.raw test1.raw
copy/b dummy_2.bin+failedfile.raw test2.raw
....
(from 1 to 7 unsuccessfully ) 

Print screen raw2cdng / HxD:
https://s29.postimg.org/6yph0zfdz/print_screen.jpg

Just Uploaded one raw failed file (the smaller)
http://www.mediafire.com/?ftetuiv7z0lp5

[g3gg0]

"The owner of this folder has not added any files."

[g3gg0]

can you please explain what modules you used for recording, which settings and how this could be happened?

[luchobaz]

"Mediafire is verifying the file" right now :/ I ll tell you when veryfing get done!

[luchobaz]

So Sorry for the delay!

Here is the file:
https://www.mediafire.com/folder/ftetuiv7z0lp5/Raw_Failed_files

[luchobaz]

I use ML raw all the time, without any problem.
This time I think the error was to record with the cards formatted in FAT32. I always do it in ExFAT. Configuration was 5DMKIII at 50fps in PAL (1920x490 -> 1920x818). Override 50.000 (for50)

[g3gg0]

when checking the raw data, most of its content (>80%) is 0x55 or 0xAA
they are stored in repeating patterns like
55 55 AA AA 55 55 55 55 55 55 AA AA AA AA AA AA
or
55 55 AA AA AA AA AA AA 55 55 AA AA 55 55 55 55
patterns alternate after 0xD20 bytes, and various bits are randomly flipped.

there is also data that looks a bit like proper RAW data.


best guess: your CF card is corrupted or the FAT table was defect

[a1ex]

If the filesystem was corrupted, you could give this a try, using a CF card image file as argument.

[luchobaz]

Ok.. but.. How could I do that?

[luchobaz]

In other hand, it is strange, because I am using different CF cards, And It happened in different two. 

[luchobaz]

Trying with onther raw file. CMD "dummy_1.bin+failedfile.raw" I have got a result where I can see the image of the plane. If you can see, there is the stairt that I have shooted.

Here is the image:
https://s23.postimg.org/gchwcumnv/trying_recovery_RAW.jpg

[g3gg0]

if that happened to two different cards, that is weird.

a) both formatted FAT?
b) do you have a proper CF reader that has 3.3V support? (some cards are troublesome with 5V of ancient CF card readers)
c) did you already erase the files from the original CF cards?

aside of that, i am a bit puzzled where the 55 55 AA AA pattern comes from.
i also tend to use dummy values 0x55 and 0xAA but the values seen are groups of two and six bytes. (which is very uncommon for things i know of)
could also come from an error in the CF controller?

[luchobaz]

Well, it happened in 3 cards. Two in FAT and One in ExFAT. In all the cards I have good files as failed files.

I used to use this cards all the time in Raw. (2 cards are Lexar 1000x 32gb - formatted in FAT-, and the other is a Komputer Bay 128GB 1066x-formatted in ExFAT-)

a) two in FAT, 1 in ExFat.
b) Card reader is a Vantec usb 3.0 That I used several times with Raw shooting. (I will get another one and try just in case)
c) No, I dont erased or changed any file.

It's desperate. I can see the images in strange colors
Should I upload another raw file?

crossing fingers

[g3gg0]

sorry, there is nothing i can recover from this file.
if there are traces of something that looks like an image, then it is still less than 20% of your footage.

[luchobaz]

 
Is there any possibility to convert raw to mlv and try mlv_dump? As Jared did (-F2,-2 -o fixed.mlv defect.mlv)?

[a1ex]

Your issue is different from OP's; there's no point in trying what worked for him.

Code Select Expand


Resolution  : 1920 x 490
Frames      : 571
Frame size  : 1646592 bytes
FPS         : 50.000
Black level : 6957
White level : 15000


Black level is wrong -> this indicates frame corruption before recording started.

File size should have been 1646592*571 = 940204032 bytes. The footer is present, so I'm not sure why the file is 785809600 bytes; filesystem corruption?

Also:

Code Select Expand


raw2dng M05-1900.RAW
md5sum *.dng
7e64871ef5012c82f2c730203538f69f  000000.dng
36ff0070a2cf6e9a8ff4d879b0177d09  000001.dng
7e64871ef5012c82f2c730203538f69f  000002.dng
36ff0070a2cf6e9a8ff4d879b0177d09  000003.dng
7e64871ef5012c82f2c730203538f69f  000004.dng
36ff0070a2cf6e9a8ff4d879b0177d09  000005.dng
...


Same MD5 checksums repeated over and over.

This probably means the raw image capture didn't work for some reason, and the raw recorder simply saved what happened to be in camera memory at that time.

raw_rec does some checking on the image copy process (see FRAME_SENTINEL in the source code), and these checks worked fine (the sentinel is present after each frame). That means, the transfer from Canon's image processor to main memory got stuck for some reason, but raw_rec's edmac_memcpy worked fine.

In this case, I would expect any ML build newer than Aug02 2013 (changeset a6b4519) to print an error in this situation.

What version did you use?

Do you remember what happened after recording this clip, before turning off the camera? (crash? another clip successfully recorded? another failed clip?)

[luchobaz]

Thanks a1lex!

Iam using ML Ver: nightly 2015 apr. 28 5D3133.

I have (in total -3 cards-) recorded 82 files. 70 successfully recorded, and 12 failed.
Camera never crashed. It was stopping without error, with some exceptions that the camera stopped by itself while I was using FAT format.
It gives me 3 ".r00" files of those files recorded in FAT format. But when I concatenate the r00 with raw, it gives me the same strange color error.

Concatenatting dummy:1.bin [(00 in HEX) or (00 00 00) or others odd files] + failedfile.raw It gives me dngs where I can clearly recognize the image.

[a1ex]

Can you upload a file where you can recognize some image?

[luchobaz]

Yes  I am uploading the file right now, it´s 12gbs heavy, so.. It will be uploaded in some hours.

This is the comparision print screen of raw2cdng

https://s29.postimg.org/e6fqjnut3/trying_recovery_RAW.jpg

*First image is fail file.
*Second image is failed file concatenated with 00 bin file
*Third one is an image of a successfully raw taken in the same place.

I will send the link of the file when it gets uploaded.

[luchobaz]

Here it is the uploaded failed file:
https://mab.to/1jL5Og0hj