RAW Video Post processing (raw2dng) [Mac/Win/Linux]

[LB]

Hey folks,

I do not have after fx and I failed to open the DNGs with Davinci resolve software, FCX and premiere pro, any other software that could work?

thanks for help

o.

Photoshop opens them too, you can process them in batch.

GoPro CineForm Studio also handles them (quite speedily) but it doesn't seem to understand the embedded profile and the colors are beyond horrendous so it's not really usable at this point.

[scrax]

Photoshop opens them too, you can process them in batch.

Can you confirm that also dng made with last nightly can be opened?

[marten]

Just finished making a simple batch conversion app for windows to make mp4 proxy files.

http://www.magiclantern.fm/forum/index.php?topic=5557.0

[hjfilmspeed]

YES!!! sweet! Nice work marten! This is very useful! I wish i could help you dev but i know nothing about programming but i will test once i get my Komputerbay card which is looking delayed.

[sicetime]

I tried opening the dng's in Assimilate's scratch and had no luck as well, just wanted to give a heads up about that, I guess they only support cinemadng's womp womp.

[peoplemerge]

I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

[Danne]

I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

This is a sweet solution but I have no idea how to hack my software, is it hard?

[peoplemerge]

This is a sweet solution but I have no idea how to hack my software, is it hard?

You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.

[Danne]

You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.

I see, thanks . When and if the occasion arrives I know what I have to do for a week

[peoplemerge]

Yeah, I think when enough people have last-frame corruption a permanent solution will materialize.  I would need to study the format (https://bitbucket.org/hudson/magic-lantern/src/tip/src/raw.h) and brush up on my C, but there's probably a way to extract the image metadata so you don't need another file as reference.

[cube]

Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

[Andy600]

Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something 

[cube]

1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something 

DaVinci Resolve 9.1 has CinemaDNG IDT.
The question is: Is RAW2DNG transcoding correct taking into consideration the ACES concept?

[mageye]

Just a general tip here for people that may be trying to get a more 'natural' skin tone when using ACR. I am sure there are plenty more definitive guides but this one helped me. I do intend on going through more tutorials on the subject of skin tones and colour cast.

http://www.iheartfaces.com/2012/09/how-to-correct-green-color-cast-in-photoshop/

[mageye]

Question:

Does anybody know if, once you have used ACR to import your DNG (into After Effects), you can change ACR settings for the imported video?

It seems to me that once imported these are the settings that you have to stick with. It's really not ideal.

What are other peoples thoughts on this? or perhaps you have a better workflow that allows more control?.

[squig]

There's an "edit original" command in the edit menu.

[CarVac]

I may be the only Linux user asking for help but...

I couldn't get the bash script to run; it complained that there was no "raw2dng".

So I realized I needed that program, but when I tried to compile it it needed two headers.

So I grabbed both headers and the makefile and modified the .c file to not look two directories up for the other header, and it still didn't compile.

Then I decided to clone the whole ML Unified branch and build that, but then I realized I needed the gcc-arm-none thing and got that.

Then, it started to compile, but it stopped:

boot-hack.o: In function `my_init_task':
boot-hack.c:(.text+0x544): undefined reference to `_bss_end'
boot-hack.c:(.text+0x548): undefined reference to `_text_start'
boot-hack.o: In function `my_task_dispatch_hook':
boot-hack.c:(.text+0x5f8): undefined reference to `_task_overrides_start'
boot-hack.c:(.text+0x5fc): undefined reference to `_task_overrides_end'
boot-hack.o: In function `copy_and_restart':
boot-hack.c:(.text+0x7e4): undefined reference to `_bss_start'
boot-hack.c:(.text+0x7e8): undefined reference to `_bss_end'
boot-hack.o: In function `my_big_init_task':
boot-hack.c:(.text+0x8d0): undefined reference to `_init_funcs_start'
boot-hack.c:(.text+0x8d4): undefined reference to `_init_funcs_end'
boot-hack.c:(.text+0x8e8): undefined reference to `_tasks_start'
boot-hack.c:(.text+0x8ec): undefined reference to `_tasks_end'
config.o: In function `config_parse':
config.c:(.text+0x2b8): undefined reference to `_config_vars_start'
config.c:(.text+0x2cc): undefined reference to `_config_vars_end'
config.o: In function `config_save_file':
config.c:(.text+0x440): undefined reference to `_config_vars_start'
config.c:(.text+0x444): undefined reference to `_config_vars_end'
config.o: In function `get_config_vars_start':
config.c:(.text+0x538): undefined reference to `_config_vars_start'
config.o: In function `get_config_vars_end':
config.c:(.text+0x544): undefined reference to `_config_vars_end'
config.o: In function `config_var_was_changed':
config.c:(.text+0x590): undefined reference to `_config_vars_start'
config.c:(.text+0x594): undefined reference to `_config_vars_end'
config.o: In function `config_var_restore_default':
config.c:(.text+0x5dc): undefined reference to `_config_vars_start'
config.c:(.text+0x5e0): undefined reference to `_config_vars_end'
property.o: In function `prop_add_internal_handlers':
property.c:(.text+0x1d0): undefined reference to `_prop_handlers_start'
property.c:(.text+0x1d4): undefined reference to `_prop_handlers_end'
collect2: error: ld returned 1 exit status
make[1]: *** [magiclantern] Error 1


Did I try the right things?

How can I get raw2avi to work on linux?

[mageye]

Thank you squig very much. That really helps. Lots

[RenatoPhoto]

@CarVac

Right question wrong area.  Please post you question here:

RAW Video Module raw_rec [Compiling/Installation]
http://www.magiclantern.fm/forum/index.php?topic=5405.0

[Shield]

I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

[budafilms]

Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!

[Danne]

Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!

I,d say if you are satisfied with your dng-files and you don,t need to much work after exporting from dng you should be fine. It,s still a heck lot better than the h264 codec from the camera moviemode

[peoplemerge]

I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

Much better!  Thanks for this.

[Mirazimov]

Hi there! i have a one problem, when i'm open dmg file in PS CS 6 a see broken image. Any ideas?

[3pointedit]

raw2dng.exe for windows, from 1st post seems to work, however it doesn't generate any DNG files in the folder. It processes the frame count (only 112 frames).