SOLVED! - Need to extract DNG from RAW multi files that comes up as invalid

[KenshirouX]

Hi all,

I wonder if you can help me on this one as this is pretty important for a client.  I've shoot several scenes in RAW format and on one of the scenes, I have shot over 31gb worth of RAW video (not MLV) that split into 8 files (M18-1157. RAW, M18-1157.R00, M18-1157.R01 ..... M18-1157.R06).  When I went to post, all the other RAW files (including multi files) worked perfectly fine but this one refuse to open for extraction.

Here are the methods I have tried:

1) Used MlRawViewer_1_2_2 to view the files for extraction:  I dropped the M18-1157.RAW file into the execute  mlrawviewer.exe file but nothing happen other than the program providing the following log info:

(c) Andrew Baldwin & contributors 2013-2014 Using GLFW wavnames [] wavfilename E:\****\*****\Video 2 Idashi\M18-1157a.WAV Traceback (most recent call last):   File "<string>", line 1874, in <module>   File "<string>", line 1867, in main   File "<string>", line 702, in __init__   File "c:\Projects\mlrawviewer\build\mlrawviewer_win\out00-PYZ.pyz\GLComputeGLFW", line 146, in __init__   File "c:\Projects\mlrawviewer\build\mlrawviewer_win\out00-PYZ.pyz\GLComputeGLFW", line 165, in installCallbacks   File "<string>", line 8, in glfwSetWindowRefreshCallback WindowsError: exception: access violation reading 0x000001E0

2) Put the M18-1157.RAW directly into RAW2DNG:  The MSDOS flash on screen which indicated the program attempted to open the file but failed.

3) Used File Splitter & Joiner to join all the files together then open it again by using either RAW2DNG or mlrawviewer:  The files joined together perfectly but the same result occurred as point 2.

4) Tried renaming one of the multi files filename extension to ".RAW" then see if I can extract the files using RAW2DNG or MLRAWviewer.  No luck.

5) Tried to only open the original RAW file using any of the programs and attempted to hide the rest of the data within another folder so the program can only recognize the one file. Still no luck.

I've ran out of ideas and I urgently need to solve this...  I know that the hundreds of DNG files are still alive inside the RAW files itself, but I don't know how to extract them, and all the files baring the last one are 4,194,304kb file size, which is the correct size. Maybe there is an header error? I have no idea.

Can anyone help me, please? I would be most grateful for any amazing help to solve this issue.

To help further , here are my system and equipment info that I use for all my work:

Camera:  Canon 5D MKIII, Magic Lantern firmware v1.2.3-ml Nightly.2
CPU:  Windows 7, i7 OC 4.8ghz 3930k, 32gb memory, Asus 680 Direct CU II Top,
Extraction programs used:  MLRawViewer 1.2.2, MLRawviewer, RAW2DNG

Thanks!

[chmee]

please open the last split-file of the recording with a hex-editor (fi HxD) and look if theres the footer-data. the last 192bytes are important. you should see near the end RAWM. is it there?

[KenshirouX]

HI Chmee,

I am not 100% sure what to look for but here is a screen grab of what the HxD provided.  Can you kindly decipher it, please?




I hope this is solvable...  :S

Additional information about the recording:

FPS: 23.976
Res: 1920x1080
Shutter: 1/50, 180'
Aspect: 16:9
Total RAW file size according to properties (grouping files and selecting properties in windows): 29.7 GB (31,991,857,145 bytes)

[chmee]

if this is the end of the last file (.R06? thats only 7 files), there's no footer. you should create/merge as described in the link i wrote above.

regards chmee

[KenshirouX]

Thanks Chmee.  I will try to give it a shot. I am not good at this but I'll do my best.

I'll keep you all updated. Meanwhile, if anyone have any suggestions please let me know...

[KenshirouX]

Update -

As instructed, I've copied and pasted the hex code that is suitable for a 32gb file from the other forum and I was happy to see that I was able to extract all the files which is fantastic!  However, the only problem is only the first file/frame is perfect, all the other files/frames got tons of strong pink noise that makes everything unusable (see below):




Does anyone have any idea on how to solve this?  It seems like the final hurdle to solve everything!

[chmee]

it seems, there's kind of offset, some wrong startpoint on second picture. look on the last lines/pixels from your first picture. is all fine there? i dont remember, if raw is working with an alignment. did you read the thread i linked?

[KenshirouX]

The first frame looks perfect with no pixel lost or indication of pink anywhere. It is all the other frames that came up with this issue with one frame appearing completely scrambled with pink, the next is the lines as you've noticed, then the next almost look like an invert but dominated with pink, then the next goes back to completely scrambled.  Every several frames afterwards there does appear to look like an perfect picture but it actually appear like an offset picture (like a bad sync picture from the old 1960s movies or tv or something when you trying to tune the image to the screen perfectly) as the image steer to the left with a pink dominant image appearing on the opposite side. This actually repeats all throughout the sequence with no other appearance of a perfect picture frame.

I've read the whole thread that you've linked and it appears that some people did mentioned the pink issue and the first frame looking perfect (though no one highlighted the issue with detail like I've described it) but there was no mention of any solution. The thread did go on for 2 pages only so maybe it was further discussed elsewhere, I have no idea...

Note that all the other videos that I've taken in the same setting came out perfectly...  it is just this batch. Just extremely painful that it is an important batch.

[chmee]

Did you tried to copy the footer from a working raw to this one?

[KenshirouX]

Yes, I tried that too.  I copied the last 12 lines hex code from a working RAW and put it into the one with the issue but it didn't work at all.   *sigh...

I have been browsing around and there appear to be some solutions but only for MAC.  I really don't know, I just not knowledgeable enough, I'm afraid...

[baldand]

I have a python script that can try and recover usable data from corrupted .RAW files. It's called "rawcover.py". Here's a thread where people have used it with different amounts of success:

http://magiclantern.fm/forum/index.php?topic=8237.msg102821#msg102821

It will work on Windows if you get a python runtime (link in that thread). You will need to run it from the command line.

[KenshirouX]

Thank you so much.  I'll give that a try and keep you updated.

[KenshirouX]

Baldand, I'm experiencing some trouble getting your script to work. Please let me inform you what I did so you can correct where I have gone wrong.

1) I've downloaded Python for Windows 3.4.1 from the official site (https://www.python.org/downloads/) and installed it into my program files <x86> directory.

2) I've downloaded your script and saved it within the Python script directory:
(C:\Program Files (x86)\Python34\Scripts)

3) I opened MS DOS, went to the Python directory and typed in the word "Python" to double check that it is recognised within MS DOS and to get  to Python mode. Happily, I received the following info:
(Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.)

4) I typed in the following command in attempt to activate your script to do its works but it came with a syntax error:
>>> python rawcover.py <E:\CONVERSION\a> [<E:\CONVERSION\b>]

  File "<stdin>", line 1
    python rawcover.py <E:\CONVERSION\a> [<E:\CONVERSION\b>]
                  ^
SyntaxError: invalid syntax


5) I tried adjusting the command to the info below but it came up with a similar error:
>>> python rawcover.py E:\CONVERSION\a [E:\CONVERSION\b]

  File "<stdin>", line 1
    python rawcover.py E:\CONVERSION\a [E:\CONVERSION\b]
                  ^
SyntaxError: invalid syntax


Honestly, I don't know what I am doing wrong. Can you help me out, please?  Much appreciated.

[baldand]

You need python 2.7, not 3.xx

[baldand]

Don't put the square brackets. Those just mean that part is optional

[KenshirouX]

Ahh.  Oki

----
Update - Alright. I have uninstall 3.0 and install Python 2.xx, re-downloaded and inserted the Script into the Script directory within Python, then I re-took the steps again making sure I don't include the square bracket but the same error occurred. The error occurred when I tried both trying to execute anything in the "a" directory, and when I request for a specific RAW file:

>>> python rawcover.py <E:\CONVERSION\a> <E:\CONVERSION\b>

  File "<stdin>", line 1
    python rawcover.py <E:\CONVERSION\a> <E:\CONVERSION\b>
                  ^
SyntaxError: invalid syntax


>>> python rawcover.py <E:\CONVERSION\a\M18-1157.RAW>
  File "<stdin>", line 1
    python rawcover.py <E:\CONVERSION\a\M18-1157.RAW>
                  ^
SyntaxError: invalid syntax


I double checked to see if I am running the correct Python and this is the message I received:
Python 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.


Am I doing anything wrong with the commands? It must be something from my end because other people was successful and I can't seem to move beyond the first stage, which at this point is becoming embarassing. Please let me know...

[baldand]

I think I see the problem. I think you were trying to type the command from inside an interactive python session.

Don't run "python" first. Instead, from the basic command line, do:

python rawcover.py e:\conversion\a e:\conversion\b

(no angle brackets or square bracket)

[KenshirouX]

Thanks Baldand.  The command line you suggested work and I have managed to extract the files.  The only problem is all of them baring is suffering from Pink madness as shown in my earlier post.  ....  It seems like I really do need to fix the footer or something, I really don't know and now I am afraid. 

Any help from anyone to solve this will be greatly appreciated.

[baldand]

Did it guess the correct frame size, or was it the wrong size?
Did you use an unusual frame size for these files?
Did the first frame look the same as you got previously with the footer fix?

It would be good to see the output of the script when it processed the files - you can send it to me in PM.

[KenshirouX]

Hi Baldand and Chmee.  Good news, I DID IT!!!    It's solved. No pink scrabbles, just over 8000 frames of clean image!  Thank goodness...

Ok. Here is a step by step of how I did it:

1) I recorded nearly 6 minutes of RAW recording on my Canon 5D MK III using the exact settings that I used for the video shoot I did for my client. I took 6 minutes worth in attempt to record close to 31gb as possible as that was the approximate size of the last recording.

2) I took the last final recorded multi-file (M28-1750.R06) and opened it using the HEX Editor (HxD).

3) On the HEX editor I moved to the bottom of the screen and look for the text RAWM which is on the right hand column.  It was located 13 row from the bottom.  When I found it, I clicked on the left of the letter "R" which highlighted a number on the central hex code, which is the number "52".

4) From the highlighted number 52 which is in the center column, I copied all the code to the end (all 13 rows starting FROM 52 and not any number before it). When pasted onto notepad it came out as the code below:
52 41 57 4D 80 07 38 04 00 60 37 00 5D 21 00 00 01 00 00 00 A8 5D 00 00 D8 47 04 00 00 00 00 00 01 00 00 00 00 A0 31 4D 26 05 00 00 20 08 00 00 38 0E 00 00 50 34 49 00 0E 00 00 00 FE 07 00 00 98 3A 00 00 00 00 00 00 00 00 00 00 8C 07 00 00 0A 05 00 00 1C 00 00 00 92 00 00 00 26 05 00 00 1E 08 00 00 00 00 00 00 00 00 00 00 00 01 01 02 01 00 00 00 42 1A 00 00 10 27 00 00 85 FD FF FF 10 27 00 00 3D FC FF FF 10 27 00 00 41 EF FF FF 10 27 00 00 AC 30 00 00 10 27 00 00 EC 07 00 00 10 27 00 00 74 FC FF FF 10 27 00 00 72 08 00 00 10 27 00 00 24 16 00 00 10 27 00 00 F7 03 00 00

5) I loaded the last multi-file into the hex editor (M18-1157.R06). This is the muti RAW file that I was trying to solve.

6) I went to the bottom of the file and pasted the copied hex code at the very end of the "center column" of codes.

7) I saved the hex code which updated the file M18-1157.R06.

8 ) I loaded the original RAW file (M18-1157.RAW) into the MLRawViewer program.

9) The MLRawViewer program loaded and showed the first frame perfectly as expected. I held my breath with hope that all other files after the first are perfect too as I moved on to press the PLAY button on the program.  Happily, all the other files showed perfect!!!

10) I exacted the files then looked at all of them using Adobe Bridge....  with happiness all the files are fine.


Thank you guys. If it weren't for you  I would have been stuck or screwed but your kindness really helped! I am most grateful.

Kindest regards


P.S - If anyone would like a little more detail to the steps I've highlighted above, please let me know in this thread and I'll do my best to elaborate.

[KenshirouX]

Sorry Badard, for your own curiousity, do you still want to see the output? If you do, please let me know where to retrieve it so I can send it to you via PM.

[baldand]

Sorry Badard, for your own curiousity, do you still want to see the output? If you do, please let me know where to retrieve it so I can send it to you via PM.

No need. Glad to hear you recovered the files.

(BTW I really suggest you move to use MLV if that works for you. It's designed so that this kind of failure doesn't happen so easily.)

[KenshirouX]

That sounds like a plan, Baldand.  MLV is new to me so I'll start researching. Is there any other distinct advantages MLV have over normal RAW recording? I'm sure there is a page that describe the differences in detail somewhere...

Again, thank you for helping me. So much appreciated.

[ds.trias]

Hey guys,
Interesting discussion in here.

I've recently moved to .MLV and I seem to be experiencing exactly the same problem KenshirouX did with RAW.

Always when my recording ends because card gets full - the last file to be recorded is unusable via MLRawViewer.. I try to drag the MLV on the MlRawViewer.exe and nothing happens. Also, I find that the card contains some extra files for the last MLV to be recorded (i.e for M22-1534.MLV I get M22-1534.M01, M22-1534.M02.... M22-1534.M08). Is there a reason for this? Any way to retrieve the last MLV and convert it to DNGs?

Any help will be appreciated!
Daniel S.

[dmilligan]

Always when my recording ends because card gets full - the last file to be recorded is unusable via MLRawViewer..

Have you tried another converter like mlv_dump? Could just be an issue with MLRawViewer.

Is there a reason for this?

http://www.magiclantern.fm/forum/index.php?topic=12466.msg121960#msg121960