Salvaging Pink Crazy DNGs

[scarluuk]

Okay, so here's my problem.
I shot a videoclip on my 5D mark III in 1920p 2.35 with 48fps (FPS override in 720p60 setting)
The card went corrupt (camera was showing the card was not compatible with the camera or cannot be used) and when i connected it to my iMac it said the card had to be formatted or ejected.
So with the help of a few recovery programs i got some files of the card but they were .SWF files and with another program i got one big .gz file.
Now i tried using 0xED to add the footer but i'm just doing something i have 0 knowledge of.
I tried it and renamed the file to end with .RAW and while RAWmagic now converts the file i only get the pink crazy DNG files.
Can anybody explain to me what it is that i really need to replace or add to the file?
I'm using one .SWF file tot try it out on and that file is 354,1 MB.
If anybody could help me i would be really happy!

[baldand]

I have a python script that attempts to extract usable .RAW files from this kind of corrupted/recovered partial RAW files.

You can get it here, but you'll need a python runtime to run it (built in on mac, get from http://www.python.org/ for windows).

Script: https://bitbucket.org/baldand/mlrawviewer/raw/master/tools/rawcover.py

Do:

python rawcover.py <dir_containing_raw_files_or_one_raw_file> [<dir_for_recovered_raw_file>]

It will recover everything it can find, including sometimes partial frames that have been chopped in half.

Good luck!

[fatpig]

This script is VERY useful for my corrupted card.
It saved all my data, except for 2 of 52 videos, which came out pink.

Great script!

[casuso]

I'm having the same problem on a 50D i recorded some videos and three of them came out with pink crazyness. I can join the files and extract the dngs but some of the dngs are pink noise with recognizable shapes. I have tried the footer hex fix in several different ways and nothing. Always the same dngs came out corrupted. Im going to try the python script and see the results....

[casuso]

Yes  your script worked perfectly. It made some files and inside them the good files. Some of them came out with bandin but at least i could recover most of them. Thanks a lot!

[Jip-Hop]

I have a python script that attempts to extract usable .RAW files from this kind of corrupted/recovered partial RAW files.

You can get it here, but you'll need a python runtime to run it (built in on mac, get from http://www.python.org/ for windows).

Script: https://bitbucket.org/baldand/mlrawviewer/raw/master/tools/rawcover.py

Do:

python rawcover.py <dir_containing_raw_files_or_one_raw_file> [<dir_for_recovered_raw_file>]

It will recover everything it can find, including sometimes partial frames that have been chopped in half.

Good luck!

I successfully recovered a RAW file consisting of multiple parts using the above script. The .RAW file and the .R01 file recovered fine. However, the .R00 file threw an error. This is the output in the command line window:

 e:\Users\Me>C:\Python27\python.exe e:\Users\Me\Desktop\rawcover.py E:\Users\Me\Videos\ZG\100CANON\M14-1911.R00
Source: E:\Users\Me\Videos\ZG\100CANON\M14-1911.R00 Target: E:\Users\Me\Videos\ZG\100CANON
Attempting to recover: E:\Users\Me\Videos\ZG\100CANON\M14-1911.R00
File length: 4294967295
Scanning file for Magic Lantern RAW headers.
0%.. 0%.. 0%.. 1%.. 1%.. 1%.. 2%.. 2%.. 3%.. 3%.. 3%.. 4%.. 4%.. 5%.. 5%.. 5%..
6%.. 6%.. 7%.. 7%.. 7%.. 8%.. 8%.. 8%.. 9%.. 9%.. 10%.. 10%.. 10%.. 11%.. 11%..
12%.. 12%.. 12%.. 13%.. 13%.. 14%.. 14%.. 14%.. 15%.. 15%.. 16%.. 16%.. 16%.. 17
%.. 17%.. 17%.. 18%.. 18%.. 19%.. 19%.. 19%.. 20%.. 20%.. 21%.. 21%.. 21%.. 22%.
. 22%.. 23%.. 23%.. 23%.. 24%.. 24%.. 25%.. 25%.. 25%.. 26%.. 26%.. 26%.. 27%..
27%.. 28%.. 28%.. 28%.. 29%.. 29%.. 30%.. 30%.. 30%.. 31%.. 31%.. 32%.. 32%.. 32
%.. 33%.. 33%.. 33%.. 34%.. 34%.. 35%.. 35%.. 35%.. 36%.. 36%.. 37%.. 37%.. 37%.
. 38%.. 38%.. 39%.. 39%.. 39%.. 40%.. 40%.. 41%.. 41%.. 41%.. 42%.. 42%.. 42%..
43%.. 43%.. 44%.. 44%.. 44%.. 45%.. 45%.. 46%.. 46%.. 46%.. 47%.. 47%.. 48%.. 48
%.. 48%.. 49%.. 49%.. 50%.. 50%.. 50%.. 51%.. 51%.. 51%.. 52%.. 52%.. 53%.. 53%.
. 53%.. 54%.. 54%.. 55%.. 55%.. 55%.. 56%.. 56%.. 57%.. 57%.. 57%.. 58%.. 58%..
58%.. 59%.. 59%.. 60%.. 60%.. 60%.. 61%.. 61%.. 62%.. 62%.. 62%.. 63%.. 63%.. 64
%.. 64%.. 64%.. 65%.. 65%.. 66%.. 66%.. 66%.. 67%.. 67%.. 67%.. 68%.. 68%.. 69%.
. 69%.. 69%.. 70%.. 70%.. 71%.. 71%.. 71%.. 72%.. 72%.. 73%.. 73%.. 73%.. 74%..
74%.. 75%.. 75%.. 75%.. 76%.. 76%.. 76%.. 77%.. 77%.. 78%.. 78%.. 78%.. 79%.. 79
%.. 80%.. 80%.. 80%.. 81%.. 81%.. 82%.. 82%.. 82%.. 83%.. 83%.. 83%.. 84%.. 84%.
. 85%.. 85%.. 85%.. 86%.. 86%.. 87%.. 87%.. 87%.. 88%.. 88%.. 89%.. 89%.. 89%..
90%.. 90%.. 91%.. 91%.. 91%.. 92%.. 92%.. 92%.. 93%.. 93%.. 94%.. 94%.. 94%.. 95
%.. 95%.. 96%.. 96%.. 96%.. 97%.. 97%.. 98%.. 98%.. 98%.. 99%.. 99%..
No Magic Lantern RAW headers found.
Unknown section from 0 to 4294967295
start 0
Found possible bayer frame
Traceback (most recent call last):
  File "e:\Users\Me\Desktop\rawcover.py", line 489, in <module>
    sys.exit(main())
  File "e:\Users\Me\Desktop\rawcover.py", line 486, in main
    recover_file(source,target)
  File "e:\Users\Me\Desktop\rawcover.py", line 458, in recover_file
    rescue_norawm(f,us,ue,fn,rn)
  File "e:\Users\Me\Desktop\rawcover.py", line 307, in rescue_norawm
    f.seek(check)
IOError: [Errno 22] Invalid argument

Does anyone know what this error means? It seems like a program crash.. however the error is consistent with the .R00 file and the script worked with the other files so I have little reason to doubt the code of the script.

[casuso]

@Jip-Hop try merging the files together and rescuing the merged file.

[Jip-Hop]

@Jip-Hop try merging the files together and rescuing the merged file.

Thanks for the great tip! I merged the files with the following command in a command line window:

copy /b "C:\...\blah.RAW"+"C:\...\blah.R00"+"C:\...\blah.R01"+...+"C:\...\blah.RXX" "C:\...\blah-merged.RAW"

And after that I was able to recover the raw file with rawcover.py

[EC Film]

I have several files capped at 4.29 GB, and I've tried merging them and then using the Python script to rescue them, but only some of them work.  When I run the python script the files that don't work have an incorrect estimated frame height (less than 1080).  Should I run the hex code on these files before merging them? Any help would be greatly appreciated!!!

[Robot Jungle Films]

I recently finished shooting a short film with a 5d mark iii only to realize all shots (around 30) that ran until the camera stops, or to 3.99 gbs were unable to convert to cdng. At the time we filmed this, I didn't realize shooting in exFat would take care of the stoppage problem. Raw2Cdng, which I'm running through a windows OS, recognized the frame rate as an outrageously large number and would quit working every time I tried to convert it. I then grabbed a .R00 file from a previous shoot in which used a mark ii and attached it to the .RAW file and sure enough Raw2Cdng recognized it as a 23.976 frame rate. I was then able to convert them and thought I solved my problem. Turns out the all of the frames were crazy pink dngs. Using hexfiend and realizing that our hex code was 2 digits short as "pedrosuarez" had mentioned, we tried the workflow suggested, by calculating a footer and inserting it into hex fiend.

We calculated the footer for only the 3.99 gb RAW file. 3.99 * 1024 = 4,085.76/2.82 mb frames = 1,448.85106 or 1,449. After putting it into the demical to hex converter we were given 5a9 or 09 5A in place of XX XX.

Using the code suggested by "pedrosuarez" we used this as our footer:

52 41 57 4D 80 07 38 04 00 60 37 00 09 5A  00 00 01 00 00 00 A8 5D 00 00 48 48 04 00 00 00 00 00 01 00 00 00 00 20 15 0B 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FF 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 88 07 00 00 08 05 00 00 1E 00 00 00 92 00 00 00 26 05 00 00 1A 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 EB 03 00 00

Raw2Cdng recognized the file now as the correct frame rate and converted it without having to attach a .R00 file as we did previously. Raw2Cdng said that it was around 27,000 frames, which obviously wasn't right, but since we noticed ifleeter had this problem we were just going to cut it off around the correct frame amount as he did.

Our problem, is the files still remained pink and fuzzy. Some frames were a little more decipherable than they were previously, but not where they needed to be.

This makes me think I am on the right track, but somewhere I went wrong in the conversion or calculating process.

I was hoping someone could shed light on my problem or suggest any solutions. I could also share the 3.99 gb with you via google drive if that would make things easier. I know nothing of code and have tried figuring it out thru forums and my own skills, but have consistently failed.

Anything would be greatly appreciated.

[Dareios]

Also shooting a film and had the same pink problem, that´s what I did to solve it, it´s quite simple:

You need:
- Any non-corrupt file shot with the same settings
-  A calculator


1) First, open the non-corrupt file with the hex editor and copy the footer. At least in my case, it was about the same length of the one posted here. I recognized it because, as the example posted here, it started with: "52 41 57 4D"

2) Second, extract the DNGs from the non-corrupt file to know how big that files are.

3) Then you have to calculate how many frames (approx.) you want extracted from the clip.
I did the following math:

( (4.29 * number of split-files in GB) + size of the corrupt file in GB ) * 1024 * 1.1463 / size of a single DNG file

Explanation:
- (4.29 * number of split-files in GB) = if you have your files split because you shot in a FAT card, you ´ll end having FILE.R00, FILE.R01, FILE.RAW etc. This is just to avoid the step of merging the files, as some people suggest.

- Size of the corrupt file in GB:   The .RAW is the first file, and the highest .R0 is the last (so, that´s the corrupt file and the one you need).

- * 1024 (from GB to MB duh!)

- * 1.1463 . This is how much bigger my files are after conversion from RAW to DNG. I get about 14% bigger files after the extraction, I would advise you do your own calculation with the non-corrupt file, as it may vary if you use different settings (I have no idea)

- size of a single DNG file: the size of a single frame.

4) Convert the number of frames you got for that file size to Hexadecimal
http://easycalculation.com/decimal-converter.php

5) Replace the position in the footer with the correct hex number you now have and paste the footer in the end of your corrupt file.
The previous posts explain the right place to put it, the 13th position (there must be a better term for this) in the footer.
Remember it´s Little Endian, so you have to invert the position of the numbers in the footer!
 
For example:
You got A3 you write A3 00
you got FD4A you write 4A FD
you got 6C2 you write C2 06

No idea what to do if you have more than FFFF frames (45 min at 24 fps). I guess you could test writing it after or before.

6) Done, extract with your favorite raw to dng converter.
If you feel like you are missing some frames because the math isn´t exact, you can always add frames to the footer (haven´t tested yet, but I guess it will extract some garbage after the good frames finish.



Thanks everybody for the previous posts, I wouldn´t have no idea what to do without them!

[BrigantineFilms]

totally saved my footage on a rollout. this totally works. thanks pedrosuarez.

[tpwilkerson]

I feel like I'm close to fixing my clips but am getting hung up on a couple things....

1) First, open the non-corrupt file with the hex editor and copy the footer. At least in my case, it was about the same length of the one posted here. I recognized it because, as the example posted here, it started with: "52 41 57 4D"

How do I find the footer if it doesn't start with "52 41 57 4D"?

5) Replace the position in the footer with the correct hex number you now have and paste the footer in the end of your corrupt file.
The previous posts explain the right place to put it, the 13th position (there must be a better term for this) in the footer.

Am I alway replacing the 13th position with the Hexadecimal determined by Step 3 and 4 in Dareios post?

[Jared]

Greetings people ...
I have a problem with multiple MLV files

using the build (BUILD = magiclantern-Nightly.2014Aug07.5D3123)

I'm using MLV Converter 1.9.1, 1.4.2 and Mlrawviewer

there any chance of recovering those files?
Could someone help me, PLEASE?

screenshoot:


DNG: https://www.dropbox.com/s/4kavri9vaxhl87d/MLV_A001_101233_C00477.000000.dng?dl=0
MLV FILE: https://www.dropbox.com/s/zalvkps6va5dskt/M10-1234.MLV?dl=0

[sephiroth]

Hi guys.

First of all I'm not sure if I have the same issue as the rest of you, but too seeing crazy pink corrupted frames.
I recorded this on the latest Nightly (magiclantern-Nightly.2015May03.7D203) on my Canon 7D on a Transcend 1000x 64GB.
For more details on settings check the images. The RAW files with corrupted frames are mostly in the beginning, and the later recordings the same shoot are all free from pink frames.
To recover the files I've tried the rawcover.py script, holding my thumbs, however the file came out as corrupted as the input.
I can see that most of the information seem to be in the image, although offsetted and pink, I can imagine it would be possible to restore frame-by-frame....
Any ideas what can be done to restore? And further to avoid the same issue from happning again (i didnt see any warning about dropped or corrupt frames while recording).

http://i.imgur.com/34PwBYB.jpg
http://i.imgur.com/MhgOS1w.jpg
http://i.imgur.com/CM3EeGy.jpg
http://i.imgur.com/BEs9CLr.jpg

Thankful for any input here!

Cheers,

[MagicEclipse]

Any update on this?