Salvaging Pink Crazy DNGs

[dangerh33@mac.com]

Hello everyone, I understand we may have accidentally recorded some RAW video to the SD card instead of the CF card.
We may also have done Load Modules wrong.

My big question is this:
"Can we salvage this footage?"

We have a few dozen RAW files ranging from 300MB to 1.5GB's that are 1920x1080, 29.97.
Losing this footage would be epic. I have Hex Fiend installed for Mac and have changed the footer as instructed.
All I get is 1 frame that's good and the rest are Pink crazy dng's.

Can anyone please help?

thanks
Dan

[dangerh33@mac.com]

Let me add this to the "help" list.. I will gladly PayPal someone who can help me salvage these files tonight.
I will pay for your time.

[pedrosuarez]

+1 on this..

I have a 42 gig file that ran off the end of my card, so the footer is missing. Stitched together the .raw + .r00, etc. in Hex Fiend, applied the 64 gig footer, (thanks to Shield: http://www.magiclantern.fm/forum/index.php?topic=5732.0 ). I have tried to run this new file through both Raw2DNG and Son of a Batch (running on a mac), but am having the same problem as dangerh33, where every DNG is a pink mess, aside from the first frame.

As I understand it, this issue is only occurring on Mac hardware. Correct? Has anyone been able to process files like these? I believe the issue of 2gig+ files has been resolved, so could it be the footer or the way that they're stitched? I won't pretend to understand Hex code, so pardon my ignorance. I just assume that since RAW2DNG and Son of a Batch work fine with smaller files, the error is likely due to something on my end.

At this point, I'm saving the .RAW + .R00s to an external drive until there's a solution, but I would also be happy to PayPal someone who can help with this situation.

I would be happy to go into further detail if it would be of any help. Many thanks to all of the Devs and those assisting, your work is greatly appreciated.

[1%]

From the other thread it looks like your aligment is off... ie the first frame is pushed down and the next one is pushed down some more and by the end you just end up with pink noise.

If you have something like 100MB I can try to fix it but 42gb

[pedrosuarez]

@ 1% ...

thanks for the reply, man. That makes sense, because the pink files are often split in the preview frame for Son of a Batch, or in ACR.  I'm not sure if this is applicable, but when I'm stitching the frames in hex fiend, I noticed that broken files are consistently two digits "short" from filling up the coding line completely (and are obviously missing a footer):
 
i.e. {52 83 08 0D / 20 3B 83 FC / 0C 72 3D 88 / F8 20 B2 84
       1E 21 3B 85 / B4 1C 42 35 / 48 BC 21 B2 / 84 08 1A __ }

Would it be possible to cut a 50MB sample from the 4 gig .RAW file in HF for you to examine? I can send it through WeTransfer, or whichever method you prefer. And, of course, compensate you for your time.

If you have the link for the thread you mentioned, that would be great as well. I can't seem to find it.

[1%]

Yea, cut 50MB of it and send me the link. Lets see what happens.

[1%]

Also can you record a small file like 20 frames with the exact same settings you used?

[pedrosuarez]

@1%

Greatly appreciate your help with this. I PM'd you a dropbox link as well as some sample code with a footer from a similar, working sequence. Let me know if there's anything else that would help you.

[1%]

BTW, get

https://www.macupdate.com/app/mac/22750/0xed

It looks like it shows the ascii on the side so you can see the rawM text at the start of the header.

[pedrosuarez]

Just wanted to update this workflow with the fix, thanks to @1%...

Please correct me if you spot an error.

This is a Mac workflow, and is intended to fix the problem with the pink DNGs caused by a corrupted file sequence. Corrupted files often occur when the video sequence takes up more room on the CF card than is available or when there is a problem saving the sequence to the card. In order to recover the salvageable data, you need to apply the correct footer in the hex code for processing in Raw2DNG, Son of a Batch, etc. This also covers how to stitch together 4+ gig files, how to calculate your total frame count, and how to covert that number to hexa decimal code.

 - import the desired .raw +.r00, etc. files from your cf card to your drive (I prefer a new folder on my desktop that I can relocate upon completion)

 - open the .raw file first using 0xED to view the hex code .. available @ https://www.macupdate.com/app/mac/22750/0xed

 - open the sequential .r00 +.r01, etc. files in 0xED and copy and paste the hex code (located on the left side) below the hex code for the .raw file in order to create a sequential batch

 - apply the appropriate footer to the end of the batched hex code. To determine this you will need your total frame count and the corresponding hex code for that value.

       - to determine frame count, multiply the gb of the total file by 1024 and divide by your DNG file size in MB (i.e. 40.25 gb * 1024 / 3.7 MB = 11139.45 -> 11140 frames)

       - to determine the corresponding hex code, use a decimal to hexa decimal converter: http://easycalculation.com/decimal-converter.php

       - the hex code is applied backwards in the footer where "XX XX" is: 11140 = 2B84 or "84 2B"

52 41 57 4D 80 07 38 04 00 60 37 00 XX XX 00 00 01 00 00 00 A8 5D 00 00 48 48 04 00 00 00 00 00 01 00 00 00 00 20 15 0B 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FF 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 88 07 00 00 08 05 00 00 1E 00 00 00 92 00 00 00 26 05 00 00 1A 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 EB 03 00 00

---------------------------
completed footer for 40.25 gig file:

52 41 57 4D 80 07 38 04 00 60 37 00 84 2b 00 00 01 00 00 00 A8 5D 00 00 48 48 04 00 00 00 00 00 01 00 00 00 00 20 15 0B 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FF 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 88 07 00 00 08 05 00 00 1E 00 00 00 92 00 00 00 26 05 00 00 1A 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 EB 03 00 00

 - save this new, completed batch from 0xED to your desktop with a new component in the name that you'll recognize.. i.e. M09-0858.RAW -->  M09-0858full0xED.RAW

       - (if you need space on your drive at this point, you can move the original files to an external drive. deletion is possible, but it is recommended that you retain the originals in some form.)

 - open this file with your preferred batch processor, i.e. Son of a Batch ( http://www.magiclantern.fm/forum/index.php?topic=7266.50 ), or Raw2DNG ( http://www.magiclantern.fm/forum/index.php?topic=5508.0 ), and convert the .RAW files to .DNGs ... 000000.dng -> 011139.dng

 - save these DNGs to a folder on your desktop

 - once completed, select the folder containing all of your applicable DNGs and, with all of the files highlighted, double click them to open in Adobe Camera Raw (Photoshop)

 - adjust the settings of your file in ACR, click "Select All > Synchronize > Save Images". Save these new DNGs to a subfolder with a new component in their name: 000000.dng > 000000ACR.dng

 - once completed, import the DNG image sequence to Adobe After Effects, make sure all of your settings are applicable/as desired (i.e. frame rate, color correction, lossless format, etc.), and export the files as a video sequence for further processing in Premiere Pro, Final Cut Pro, etc.

(There are other methods that can be used for processing, such as GingerHDR and DaVinci Resolve, but I don't have much experience with these programs.)

Hopefully this workflow will be of assistance to you when trying to recover corrupted data off of your CF card. I was able to successfully pull all but the last twenty frames from my 11140 frame sequence. Again, please let me know if I missed anything or if more detail is needed.

Many thanks to all of the Developers. Your work is greatly appreciated.

[adityakapur]

I followed this workflow.

I could not salvage my frames. Can someone please help? My best shot was on that file. All the frames come out pink except the first.

I have two files, a .RAW (4.29 GB) and a .R00 (2.73 GB). Both are spitting out pink frames through RawMagic.

I used Hex Fiend as for some reason with 0xED I am unable to copy paste 2.73 GB of data onto the clipboard. I'm using a MAC.

[ifleeter]

This thread was very, very helpful to me. Thank you all!

I recently just shot four 12hr days with the 5d mark 3 ML hack. We used it to shoot a student short film. Going into it I really had no idea how to work everything with the hack, so it was a major learning process for the crew and I. Many of which have never even shot a short narrative. We had an amazing DP who does a lot of professional work, helping us out at a lower than normal charge.

Well the first day we shot everything and it went great. Day two came around and while we were prepping for it I came across this thread. I was reading about the combining of the .R00 files. Immediately I'm like F**K! I told the DIT that those extra .R00 files weren't needed. Day one being an intense dialogue driven day, shitty for us losing all that extra footage. Now who knows how much of the stuff was cut. And all those files that stop at 4gb are corrupt as well. They won't open in any Raw2Dng converter Mac or Windows.

So, I read the thread about replacing the file footer, seems easy, I do it and yes! Now the files are converting to Dng's. Once they finished I opened them in Photoshop and they are all pink crazy dng's. FML. All that footage ruined. I come back to this thread and follow the instructions and do the math and create my own footer.

52 41 57 4D 80 07 38 04 00 60 37 00 01 45 00 00 01 00 00 00 A8 5D 00 00 48 48 04 00 00 00 00 00 01 00 00 00 00 20 15 0B 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FF 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 88 07 00 00 08 05 00 00 1E 00 00 00 92 00 00 00 26 05 00 00 1A 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 EB 03 00 00

I save the .RAW file as a copy and run this copy with replaced footer through the RAW2DNG converter. It says it is going to convert over 17,000 frames for 4GB of footage. FML. I know that's not right. So I give a rough estimate and stop it at 1200 frames. When I open the DNG's the pink stuff is all gone and there is my footage! I was beyond happy. I did many more converts like this and kept cutting it off at around 1200 frames. I started to notice that the Pink noise comes back in at exactly frame number 1182 every time.

I'm not good at any of this hex code stuff so using this workflow is fine with me, I just have to cut every corrupt file off at around 1200 frames using the above footer. When using the exact 4gb corrupt file footer from the "How to replace file footer thread" it gives me pink frames but when I use the footer above it creates perfect frames. I'm not sure why this is but I know that footer above works if anyone is having trouble or deleted the .R00 files on accident and still want to use the 3.99Gb file.

Here are some stills from the student film we shot. Thanks to Magic Lantern and everyone working on this! It has been awesome to get to learn and use this Hack.

RAW No CC straight from camera, Film Grain Added.

[ifleeter]

I followed this workflow.

I could not salvage my frames. Can someone please help? My best shot was on that file. All the frames come out pink except the first.

I have two files, a .RAW (4.29 GB) and a .R00 (2.73 GB). Both are spitting out pink frames through RawMagic.

I used Hex Fiend as for some reason with 0xED I am unable to copy paste 2.73 GB of data onto the clipboard. I'm using a MAC.

@adityakapur

Use the program called terminal. It is the equivalent to command prompt for windows. Use terminal to merge the .RAW file with the .R00 file using this code "cat Blah.RAW Blah.R00 > Blah-Merge.RAW" Terminal should already be installed on your Mac. Replace "Blah" with whatever the file names are. Also, don't use the " in the code. This should work like a charm and merge your files. Once they are merged it should run through any RAW to DNG converter. If not replace the file footer using @pedrosuarez method by creating your own footer.

[Spring Pictures]

I've tried the workflow suggested by pedrosuarez a number of times with no success.

First I tried with already merged .RAW files.  I merged these using the Terminal method as outlined here:

I then tried using the original .RAW, .R00, .R01, .R02 files as described by pedrosuarez. 

All I get is crazy pink DNGs!!!

Any help would be deeply appreciated, I would also be willing to Paypal someone to work this through with me, very very important clips.

[Spring Pictures]

I should also mention that I never had the "Error: This ain't a lv_rec RAW file" for my clips. I drag them to raw2dng (on MAC) and they create a folder for the DNGs but nothing is created within there, no ProRes either.
The files were cut short when the camera automatically shut off at 4mins.  They are 15.64GB in size, I worked this out to be 6160 frames.
Thanks

[Spring Pictures]

One final thought is that maybe I'm replacing the wrong code with the footer?
I'm using HexFiend and replacing the exact number of lines that the footer pedrosuarez mentioned has.  So I paste the footer from this thread over the last few lines of the my file's hex.  Should I maybe be just be pasting pedrosuarez's footer after the existing hex, not replacing it?

[ifleeter]

Don't replace the Hex. I did that at first too, it was confusing because it does say "replace file footer" But adding it to the end of the Hex is what you do.

Hope this works for you.

EDIT* I would use this method on the merged file you created in Terminal. Just create your own code for your file size and paste it at the end of the HEX. Don't replace it.

[Spring Pictures]

Thanks ifleeter.
I tried this but alas, no difference!

The bottom of my hex in some cases seems to be very damaged, with row after row looking like this:
FF FF FF FF FF FF DF FF FD FF FF FF FF FF FF FF FF FD FF FF FF FF FF FF FF FF 7F FF FF FD FE FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF DF FF FF FF FF FF FF FF FF F7 FF FF FF FF FF FF F7 FF FF FF FF FF FD FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FD FF FF FF FF FF FF FF FF FF FF FF 7F 7F FF FF FF FF FF FF FF FF FF FF FF FF FF FF FD FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 7F FF FF FF FF FF FF FD

The last part with a variety of hex (before the above part) looks like this:
C1 71 C7 CD 1C 17 71 D0 CA 61 12 C7 80 1C 81 70 87 C7 1C 0E 71 58 CB 81 09 47 E0 1C 31 71 07 CE 1C 1D 71 DC CB 61 13 47 54 1D 41 73 07 E4 1E 3A 75 7C E7 11 54 C7 B0 1E 11 75 47 E4 1E 44 74 38 DF 21 3C 87 C4 1D F1 72 07 E3 1D 33 72 98 DB 91 29 C7 B0 1D 61 72 47 E0 1D 31 72 80 D5 81 27 87 90 1D 01 73 07 D5 1D 16 71 78 D1 F1 27 87 84 1D 61 71 C7 D7 1D 1B 71 40 D3 A1 1A 07 2C 1D E1 71 47 D4 1D 19 72 48 D8 21 1F 47 B0 1D 11 72 47 D9 1D 23 71 80 DD B1 2C 47 C8 1D 61 73 C7 D9 1E 42 73 08 E1 31 44 07 1C 1E A1 73 47 E7 1E 50 73 48 DE B1 30 C7 FC 1D 31 73 07 DE 1E 28 73 84 E9 41 2B C7 A0 1E 01 73 47 E6 1D 28 71 D4 D9 D1 1E 47 A5 A5 A5 A5

The code I'm pasting in for a 15.64GB file (6160 frames, hex 1810) is this:
52 41 57 4D 80 07 38 04 00 60 37 00 10 18 00 00 01 00 00 00 A8 5D 00 00 48 48 04 00 00 00 00 00 01 00 00 00 00 20 15 0B 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FF 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 88 07 00 00 08 05 00 00 1E 00 00 00 92 00 00 00 26 05 00 00 1A 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 EB 03 00 00

Any more pointers?  These pink DNGs are driving me crazy!!!! Thanks again.

[Spring Pictures]

Oh, also I'm shooting on 5D2 in 2.35:1 so my DNGs seem to all be 2.6MB.  Although the pink DNGs are 3.7MB.

[ifleeter]

Spring Pictures

Is there anyway you could send me your original .RAW file so I could work on it? just give me the 4GB one to work on for now, if you have gmail use googledrive and share it with me, my email is RyanJFleet@gmail.com

[ciardi]

can you paste a code for long 1920x1038 (1.85) files? it would be really nice! thanks a lot.

[zach915m]

Shot a bunch of Magic Lantern Raw the other night and am getting the Pink DNG's after the first raw file in all the "spanned" files.  So the first 4GB is fine and then the rest I have the pink dots.  I have tried to edit the footers on my PC and couldn't find a program that would copy all the footers as the memory was not large enough and the crappy Mac I had froze up when I used the program above. 

Questions:

1.  Do I just need a more powerful mac to be able to combine the HEX files?  Why would it freeze up 0Xed?

2.  Is there a PC program that would let me copy over all the files without saying I don't have enough memory to copy over?

3.  Is this an issue that was created by me shooting in the cold (20 degrees F) or is Magic Lantern just not correctly installed?  Have shot using Raw before with no issues...

[zach915m]

I was able to correct the hex files as listed above, but the image is still not fixed, it did get SLIGHTLY better, but not to anywhere near the extent needed.  Anyone have any idea how to fix the pink DNG files if the above method doesn't work?  The first 4GB of each file is perfect so I have to think it has something to do with the spanning....

[ifleeter]

@zach915m Instead of using 0xED(MAC) or HXD (PC) to combine .RAW with .R00 etc.. use command prompt (PC) or Terminal (MAC)

Command Prompt Code: copy /b "F:M18-0340.RAW"+"F:M18-0340.R00" "F:M18-0340-merged.RAW"

Terminal Code: cat M18-0340.RAW M18-0340.R00 > M18-0340-merged.RAW

***(Change code to match your information.)***
Hope this help.

-Ryan

[kennethsperling]

Thank you for this thread.

I have the same problem, and I'm trying to use the Terminal solution which seems to work.