Author Topic: RAW Video Post processing (raw2dng) [Mac/Win/Linux]  (Read 608439 times)

LB

  • New to the forum
  • *
  • Posts: 9
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #150 on: May 21, 2013, 10:25:57 AM »
Hey folks,

I do not have after fx and I failed to open the DNGs with Davinci resolve software, FCX and premiere pro, any other software that could work?

thanks for help

o.

Photoshop opens them too, you can process them in batch.

GoPro CineForm Studio also handles them (quite speedily) but it doesn't seem to understand the embedded profile and the colors are beyond horrendous so it's not really usable at this point.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #151 on: May 21, 2013, 10:32:40 AM »
Photoshop opens them too, you can process them in batch.
Can you confirm that also dng made with last nightly can be opened?
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

marten

  • Freshman
  • **
  • Posts: 98
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #152 on: May 21, 2013, 03:40:37 PM »
Just finished making a simple batch conversion app for windows to make mp4 proxy files.

http://www.magiclantern.fm/forum/index.php?topic=5557.0

hjfilmspeed

  • Senior
  • ****
  • Posts: 479
  • 5D III and IV
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #153 on: May 21, 2013, 05:22:45 PM »
YES!!! sweet! Nice work marten! This is very useful! I wish i could help you dev but i know nothing about programming but i will test once i get my Komputerbay card which is looking delayed.

sicetime

  • Freshman
  • **
  • Posts: 53
  • 5D III
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #154 on: May 22, 2013, 09:45:23 PM »
I tried opening the dng's in Assimilate's scratch and had no luck as well, just wanted to give a heads up about that, I guess they only support cinemadng's womp womp.

peoplemerge

  • Freshman
  • **
  • Posts: 50
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #155 on: May 24, 2013, 12:16:29 AM »
I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7177
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #156 on: May 24, 2013, 02:30:12 PM »
I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!

This is a sweet solution but I have no idea how to hack my software, is it hard?

peoplemerge

  • Freshman
  • **
  • Posts: 50
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #157 on: May 24, 2013, 06:44:12 PM »
This is a sweet solution but I have no idea how to hack my software, is it hard?

You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7177
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #158 on: May 24, 2013, 08:32:16 PM »
You need to have a software development environment set up, look for the wiki pages on that.  Then you need to change the source code (written in C) and build the tool.
I see, thanks :). When and if the occasion arrives I know what I have to do for a week ;)

peoplemerge

  • Freshman
  • **
  • Posts: 50
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #159 on: May 24, 2013, 11:46:11 PM »
Yeah, I think when enough people have last-frame corruption a permanent solution will materialize.  I would need to study the format (https://bitbucket.org/hudson/magic-lantern/src/tip/src/raw.h) and brush up on my C, but there's probably a way to extract the image metadata so you don't need another file as reference.

cube

  • New to the forum
  • *
  • Posts: 2
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #160 on: May 25, 2013, 03:11:51 PM »
Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

Andy600

  • Moderator
  • Hero Member
  • *****
  • Posts: 1863
  • Have you tried turning it off and on again?
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #161 on: May 25, 2013, 03:16:04 PM »
Hi,

Does anyone know if there is any proper workflow for Magic Lantern RAW recording and post production in ACES Color Space? Does it require any specific IDT?

1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something  ;D
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

cube

  • New to the forum
  • *
  • Posts: 2
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #162 on: May 26, 2013, 12:36:36 AM »
1) No not yet,  and 2) I'm wondering the same thing.

Good choice of name BTW lol. I was thinking of using Lutty or something  ;D

DaVinci Resolve 9.1 has CinemaDNG IDT.
The question is: Is RAW2DNG transcoding correct taking into consideration the ACES concept?

mageye

  • Senior
  • ****
  • Posts: 296
  • Atheist
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #163 on: May 26, 2013, 03:37:14 PM »
Just a general tip here for people that may be trying to get a more 'natural' skin tone when using ACR. I am sure there are plenty more definitive guides but this one helped me. I do intend on going through more tutorials on the subject of skin tones and colour cast.

http://www.iheartfaces.com/2012/09/how-to-correct-green-color-cast-in-photoshop/
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

mageye

  • Senior
  • ****
  • Posts: 296
  • Atheist
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #164 on: May 26, 2013, 03:53:57 PM »
Question:

Does anybody know if, once you have used ACR to import your DNG (into After Effects), you can change ACR settings for the imported video?

It seems to me that once imported these are the settings that you have to stick with. It's really not ideal.

What are other peoples thoughts on this? or perhaps you have a better workflow that allows more control?. :-\
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #165 on: May 26, 2013, 04:45:14 PM »
There's an "edit original" command in the edit menu.

CarVac

  • New to the forum
  • *
  • Posts: 12
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #166 on: May 26, 2013, 06:19:58 PM »
I may be the only Linux user asking for help but...

I couldn't get the bash script to run; it complained that there was no "raw2dng".

So I realized I needed that program, but when I tried to compile it it needed two headers.

So I grabbed both headers and the makefile and modified the .c file to not look two directories up for the other header, and it still didn't compile.

Then I decided to clone the whole ML Unified branch and build that, but then I realized I needed the gcc-arm-none thing and got that.

Then, it started to compile, but it stopped:

boot-hack.o: In function `my_init_task':
boot-hack.c:(.text+0x544): undefined reference to `_bss_end'
boot-hack.c:(.text+0x548): undefined reference to `_text_start'
boot-hack.o: In function `my_task_dispatch_hook':
boot-hack.c:(.text+0x5f8): undefined reference to `_task_overrides_start'
boot-hack.c:(.text+0x5fc): undefined reference to `_task_overrides_end'
boot-hack.o: In function `copy_and_restart':
boot-hack.c:(.text+0x7e4): undefined reference to `_bss_start'
boot-hack.c:(.text+0x7e8): undefined reference to `_bss_end'
boot-hack.o: In function `my_big_init_task':
boot-hack.c:(.text+0x8d0): undefined reference to `_init_funcs_start'
boot-hack.c:(.text+0x8d4): undefined reference to `_init_funcs_end'
boot-hack.c:(.text+0x8e8): undefined reference to `_tasks_start'
boot-hack.c:(.text+0x8ec): undefined reference to `_tasks_end'
config.o: In function `config_parse':
config.c:(.text+0x2b8): undefined reference to `_config_vars_start'
config.c:(.text+0x2cc): undefined reference to `_config_vars_end'
config.o: In function `config_save_file':
config.c:(.text+0x440): undefined reference to `_config_vars_start'
config.c:(.text+0x444): undefined reference to `_config_vars_end'
config.o: In function `get_config_vars_start':
config.c:(.text+0x538): undefined reference to `_config_vars_start'
config.o: In function `get_config_vars_end':
config.c:(.text+0x544): undefined reference to `_config_vars_end'
config.o: In function `config_var_was_changed':
config.c:(.text+0x590): undefined reference to `_config_vars_start'
config.c:(.text+0x594): undefined reference to `_config_vars_end'
config.o: In function `config_var_restore_default':
config.c:(.text+0x5dc): undefined reference to `_config_vars_start'
config.c:(.text+0x5e0): undefined reference to `_config_vars_end'
property.o: In function `prop_add_internal_handlers':
property.c:(.text+0x1d0): undefined reference to `_prop_handlers_start'
property.c:(.text+0x1d4): undefined reference to `_prop_handlers_end'
collect2: error: ld returned 1 exit status
make[1]: *** [magiclantern] Error 1


Did I try the right things?

How can I get raw2avi to work on linux?

mageye

  • Senior
  • ****
  • Posts: 296
  • Atheist
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #167 on: May 26, 2013, 06:25:51 PM »
Thank you squig very much. That really helps. Lots ;D
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #168 on: May 26, 2013, 10:52:16 PM »
@CarVac

Right question wrong area.  Please post you question here:

RAW Video Module raw_rec [Compiling/Installation]
http://www.magiclantern.fm/forum/index.php?topic=5405.0
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

Shield

  • Senior
  • ****
  • Posts: 254
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #169 on: May 27, 2013, 08:28:19 AM »
I figured out how to parse my remaining 3Gb file showing "This ain't a lv_rec RAW file" using a1ex's hack suggestion early this thread.  With the code changes allowing >2Gb but this file still failing, I realized that the footer must not have been written.  There are a number of reasons that could have caused this, like a battery dying, and I had footage I wanted to keep.  Here's what I did to recover the file, in case anyone finds themselves in this boat.

When I recorded the .RAW, I used the same settings as the previous file in the series.
The previous file (which was only 34M) outputted the following when I processed it with raw2dng
$ /path/to/raw2dng M0000008.RAW
Resolution  : 1720 x 720
Frames      : 16
Frame size  : 2167200 bytes
Frame skip  : 1
FPS         : 23.976
Processing frame 16 of 16...
Done.

So first, you need to calculate the # frames of the file you want to process.
$ ls -al M0000009.RAW
-rwxrwxrwx  1 david.thomas  staff  3101263200 May 22 19:02 M0000009.RAW

Mine is 3101263200 bytes.  Then you need a calculator. Divide that file size by the frame size.  In my case 3101263200/2167200 = 1430 frames.

Next you need to read the last frame from the working file onto the tail of the broken one.
$ tail -c 2167200 M0000008.RAW >> M0000009.RAW

Next, hack the raw2dng.c file and build:
$ diff -Naur raw2dng.c.orig raw2dng.c
--- raw2dng.c.orig   2013-05-22 18:50:44.000000000 -0700
+++ raw2dng.c   2013-05-22 19:05:44.000000000 -0700
@@ -38,12 +38,8 @@
     raw_info = lv_rec_footer.raw_info;
     fseek(fi, 0, SEEK_SET);
 
-    if (strncmp((char*)lv_rec_footer.magic, "RAWM", 4))
-        FAIL("This ain't a lv_rec RAW file\n");
-   
-    if (raw_info.api_version != 1)
-        FAIL("API version mismatch: %d\n", raw_info.api_version);
-   
+lv_rec_footer.frameCount=1430;
+
     printf("Resolution  : %d x %d\n", lv_rec_footer.xRes, lv_rec_footer.yRes);
     printf("Frames      : %d\n", lv_rec_footer.frameCount);
     printf("Frame size  : %d bytes\n", lv_rec_footer.frameSize);

As you can see, I've taken out the checks and hardcoded the frame count.

Yep, it's a hack, but I got 59 seconds of video from my little brother's med school graduation out of it!


I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #170 on: May 27, 2013, 08:26:12 PM »
Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7177
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #171 on: May 27, 2013, 10:27:29 PM »
Importan Question:
If I use JPG instead TIFF to save image from Adobe Raw, I lost to much quality in the video created from that files? It´s a huge difference in size for me, and I´m not the person who knows about quality, compression, etc. Thanks!
I,d say if you are satisfied with your dng-files and you don,t need to much work after exporting from dng you should be fine. It,s still a heck lot better than the h264 codec from the camera moviemode

peoplemerge

  • Freshman
  • **
  • Posts: 50
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #172 on: May 28, 2013, 07:50:17 AM »

I'm clearly not as smart as you, but I fumbled my way through creating this, which should allow you to get back all of your "corrput" mov files:

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

Enjoy,
Shawn

Much better!  Thanks for this.

Mirazimov

  • New to the forum
  • *
  • Posts: 33
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #173 on: May 28, 2013, 01:49:30 PM »
Hi there! i have a one problem, when i'm open dmg file in PS CS 6 a see broken image. Any ideas?
Mirazimov Photographer http://www.mirazimov.com
Canon RebelT2i, Canon 6D and a lot of optics.

3pointedit

  • Senior
  • ****
  • Posts: 263
  • 550D
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #174 on: May 29, 2013, 03:21:24 AM »
raw2dng.exe for windows, from 1st post seems to work, however it doesn't generate any DNG files in the folder. It processes the frame count (only 112 frames).
550D on ML-roids