RAW Video Post processing (raw2dng) [Mac/Win/Linux]

Started by g3gg0, May 14, 2013, 11:42:07 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

LB

Quote from: olik on May 14, 2013, 02:07:32 PM
Hey folks,

I do not have after fx and I failed to open the DNGs with Davinci resolve software, FCX and premiere pro, any other software that could work?

thanks for help

o.

Photoshop opens them too, you can process them in batch.

GoPro CineForm Studio also handles them (quite speedily) but it doesn't seem to understand the embedded profile and the colors are beyond horrendous so it's not really usable at this point.

scrax

Quote from: LB on May 21, 2013, 10:25:57 AM
Photoshop opens them too, you can process them in batch.
Can you confirm that also dng made with last nightly can be opened?
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

marten

Just finished making a simple batch conversion app for windows to make mp4 proxy files.

http://www.magiclantern.fm/forum/index.php?topic=5557.0

hjfilmspeed

YES!!! sweet! Nice work marten! This is very useful! I wish i could help you dev but i know nothing about programming but i will test once i get my Komputerbay card which is looking delayed.

sicetime

I tried opening the dng's in Assimilate's scratch and had no luck as well, just wanted to give a heads up about that, I guess they only support cinemadng's womp womp.

peoplemerge

I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);

-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

Danne

Quote from: peoplemerge on May 24, 2013, 12:16:29 AM
I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);

-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

This is a sweet solution but I have no idea how to hack my software, is it hard?

peoplemerge

Quote from: Danne on May 24, 2013, 02:30:12 PM
This is a sweet solution but I have no idea how to hack my software, is it hard?

You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.

Danne

Quote from: peoplemerge on May 24, 2013, 06:44:12 PM
You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.
I see, thanks :). When and if the occasion arrives I know what I have to do for a week ;)

peoplemerge

Yeah, I think when enough people have last-frame corruption a permanent solution will materialize.  I would need to study the format (https://bitbucket.org/hudson/magic-lantern/src/tip/src/raw.h) and brush up on my C, but there's probably a way to extract the image metadata so you don't need another file as reference.

cube

Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

Andy600

Quote from: cube on May 25, 2013, 03:11:51 PM
Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something  ;D
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

cube

Quote from: Andy600 on May 25, 2013, 03:16:04 PM
1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something  ;D

DaVinci Resolve 9.1 has CinemaDNG IDT.
The question is: Is RAW2DNG transcoding correct taking into consideration the ACES concept?

mageye

Just a general tip here for people that may be trying to get a more 'natural' skin tone when using ACR. I am sure there are plenty more definitive guides but this one helped me. I do intend on going through more tutorials on the subject of skin tones and colour cast.

http://www.iheartfaces.com/2012/09/how-to-correct-green-color-cast-in-photoshop/
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

mageye

Question:

Does anybody know if, once you have used ACR to import your DNG (into After Effects), you can change ACR settings for the imported video?

It seems to me that once imported these are the settings that you have to stick with. It's really not ideal.

What are other peoples thoughts on this? or perhaps you have a better workflow that allows more control?. :-\
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

squig

There's an "edit original" command in the edit menu.

CarVac

I may be the only Linux user asking for help but...

I couldn't get the bash script to run; it complained that there was no "raw2dng".

So I realized I needed that program, but when I tried to compile it it needed two headers.

So I grabbed both headers and the makefile and modified the .c file to not look two directories up for the other header, and it still didn't compile.

Then I decided to clone the whole ML Unified branch and build that, but then I realized I needed the gcc-arm-none thing and got that.

Then, it started to compile, but it stopped:

boot-hack.o: In function `my_init_task':
boot-hack.c:(.text+0x544): undefined reference to `_bss_end'
boot-hack.c:(.text+0x548): undefined reference to `_text_start'
boot-hack.o: In function `my_task_dispatch_hook':
boot-hack.c:(.text+0x5f8): undefined reference to `_task_overrides_start'
boot-hack.c:(.text+0x5fc): undefined reference to `_task_overrides_end'
boot-hack.o: In function `copy_and_restart':
boot-hack.c:(.text+0x7e4): undefined reference to `_bss_start'
boot-hack.c:(.text+0x7e8): undefined reference to `_bss_end'
boot-hack.o: In function `my_big_init_task':
boot-hack.c:(.text+0x8d0): undefined reference to `_init_funcs_start'
boot-hack.c:(.text+0x8d4): undefined reference to `_init_funcs_end'
boot-hack.c:(.text+0x8e8): undefined reference to `_tasks_start'
boot-hack.c:(.text+0x8ec): undefined reference to `_tasks_end'
config.o: In function `config_parse':
config.c:(.text+0x2b8): undefined reference to `_config_vars_start'
config.c:(.text+0x2cc): undefined reference to `_config_vars_end'
config.o: In function `config_save_file':
config.c:(.text+0x440): undefined reference to `_config_vars_start'
config.c:(.text+0x444): undefined reference to `_config_vars_end'
config.o: In function `get_config_vars_start':
config.c:(.text+0x538): undefined reference to `_config_vars_start'
config.o: In function `get_config_vars_end':
config.c:(.text+0x544): undefined reference to `_config_vars_end'
config.o: In function `config_var_was_changed':
config.c:(.text+0x590): undefined reference to `_config_vars_start'
config.c:(.text+0x594): undefined reference to `_config_vars_end'
config.o: In function `config_var_restore_default':
config.c:(.text+0x5dc): undefined reference to `_config_vars_start'
config.c:(.text+0x5e0): undefined reference to `_config_vars_end'
property.o: In function `prop_add_internal_handlers':
property.c:(.text+0x1d0): undefined reference to `_prop_handlers_start'
property.c:(.text+0x1d4): undefined reference to `_prop_handlers_end'
collect2: error: ld returned 1 exit status
make[1]: *** [magiclantern] Error 1


Did I try the right things?

How can I get raw2avi to work on linux?

mageye

Thank you squig very much. That really helps. Lots ;D
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

RenatoPhoto

@CarVac

Right question wrong area.  Please post you question here:

RAW Video Module raw_rec [Compiling/Installation]
http://www.magiclantern.fm/forum/index.php?topic=5405.0
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Shield

Quote from: peoplemerge on May 24, 2013, 12:16:29 AM
I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);

-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!


I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

budafilms

Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!

Danne

Quote from: budafilms on May 27, 2013, 08:26:12 PM
Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!
I,d say if you are satisfied with your dng-files and you don,t need to much work after exporting from dng you should be fine. It,s still a heck lot better than the h264 codec from the camera moviemode

peoplemerge

Quote from: Shield on May 27, 2013, 08:28:19 AM

I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

Much better!  Thanks for this.

Mirazimov

Hi there! i have a one problem, when i'm open dmg file in PS CS 6 a see broken image. Any ideas?
Mirazimov Photographer http://www.mirazimov.com
Canon RebelT2i, Canon 6D and a lot of optics.

3pointedit

raw2dng.exe for windows, from 1st post seems to work, however it doesn't generate any DNG files in the folder. It processes the frame count (only 112 frames).
550D on ML-roids