Headline news

An announcement regarding the forking of Magic Lantern, and the support that can be provided for these forks on this forum.

You can find our live webchat here: [email protected]

Author Topic: RAW Video Post processing (raw2dng) [Mac/Win/Linux]  (Read 111640 times)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2141
RAW Video Post processing (raw2dng) [Mac/Win/Linux]
« on: May 14, 2013, 11:42:07 AM »
Please ask questions about how to post process RAW-Video here.

Keywords: raw2dng, 14-bit raw, dng, video

All questions regarding RAW VIDEO POST PROCESSING on any OS
HERE

All questions regarding RAW_REC MODULE COMPILATION/INSTALLATION/USAGE
HERE: http://www.magiclantern.fm/forum/index.php?topic=5405.0



raw2dng (reference implementation, command-line tool)

Source code: raw2dng.c and chdk-dng.c
Windows executable: raw2dng.exe (drag and drop the .raw file over the executable)
Mac executable: raw2dngOSX.zip (run in terminal, don't forget chmod +x)
Windows executable for cameras with pink dots (650D, 700D, EOS-M): raw2dng_cs2x2.exe (it does some chroma smoothing, which happens to remove the pink dots too; use it when PinkDotRemover doesn't work)





GUI application for MAC (scrax):
Thread
Just drag a raw file on it and it will convert to dng and save the file in a subfolder inside the .raw folder




Linux scripts
Bash script for conversion to mjpeg (needs ffmpeg and ufraw-batch): raw2avi.sh
« Last Edit: January 25, 2014, 08:26:25 PM by a1ex »

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
RAW Video Post processing [Mac/Win/Linux]
« Reply #1 on: May 14, 2013, 11:46:12 AM »
raw2dgn.app
More info on the GUI version discussion topic

raw2dng
Osx command line

Know issues
- Can't convert file bigger than 2Gb.
« Last Edit: May 19, 2013, 03:17:02 PM by scrax »
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2141
RAW Video Post processing [Mac/Win/Linux]
« Reply #2 on: May 14, 2013, 11:48:25 AM »
by the way:where are the raw files saved on the card? on root or in DCIM?

should be DCIM folder. my lv_rec modules placed them in the root which was not ideal, so alex placed it in DCIM\xxx\

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6742
  • 5D Mark Free
RAW Video Post processing [Mac/Win/Linux]
« Reply #3 on: May 14, 2013, 11:48:36 AM »
raw2dng (reference implementation, command-line tool)

Source code: raw2dng.c
Windows executable: raw2dng.exe (also runs in Wine, for Mac and Linux)

Usage: drag and drop the .raw file over the executable, or use the command line.

Linux shell script for conversion to mjpeg: raw2avi.sh

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
RAW Video Post processing [Mac/Win/Linux]
« Reply #4 on: May 14, 2013, 11:52:18 AM »
should be DCIM folder. my lv_rec modules placed them in the root which was not ideal, so alex placed it in DCIM\xxx\
good that's perfect since MLTools checks only the DCIM folder, since there should there the captured stuff.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
RAW Video Post processing [Mac/Win/Linux]
« Reply #5 on: May 14, 2013, 11:53:57 AM »
raw2dng (reference implementation, command-line tool)

Source code: raw2dng.c
Windows executable: raw2dng.exe (also runs in Wine, for Mac and Linux)

Usage: drag and drop the .raw file over the executable, or use the command line.

Linux shell script for conversion to mjpeg: raw2avi.sh

here a link to the raw2dng compiled for mac (no wine needed)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
RAW Video Post processing [Mac/Win/Linux]
« Reply #6 on: May 14, 2013, 12:04:34 PM »
Mac executable: raw2dngOSX.zip
Usage: drag and drop the .raw file over the executable, or use the command line.

just a small note, for osx drag and drop a .raw on the raw2dng executable don't works and double clicking it will run it without any raw file selected
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

HD Cam Team

  • Supporter
  • New to the forum
  • *****
  • Posts: 7
RAW Video Post processing [Mac/Win/Linux]
« Reply #7 on: May 14, 2013, 12:14:45 PM »
Hi,

Great idea about this thread.

Is it possible to somehow use FFmpeg to get video directly from the current .raw file?

http://www.ffmpeg.org/index.html

Thanks
HD Cam Team
Web: www.HDCamTeam.com | Twitter: @HDCamTeam | Facebook: HDCamTeam

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6742
  • 5D Mark Free
RAW Video Post processing [Mac/Win/Linux]
« Reply #8 on: May 14, 2013, 12:16:34 PM »
Yes, the Linux script does just that.

electronicwomble

  • New to the forum
  • *
  • Posts: 11
RAW Video Post processing [Mac/Win/Linux]
« Reply #9 on: May 14, 2013, 12:34:42 PM »
Thanks so much for all your work on this! Your turnaround times for all this dev work are amazing :)

Any ideas about where to look for converting DNGs to CinemaDNGs ready for Resolve?

HD Cam Team

  • Supporter
  • New to the forum
  • *****
  • Posts: 7
RAW Video Post processing [Mac/Win/Linux]
« Reply #10 on: May 14, 2013, 12:53:41 PM »
Yes, the Linux script does just that.

Something for Windows?
HD Cam Team
Web: www.HDCamTeam.com | Twitter: @HDCamTeam | Facebook: HDCamTeam

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2141
RAW Video Post processing [Mac/Win/Linux]
« Reply #11 on: May 14, 2013, 12:55:21 PM »
Something for Windows?
no all-in-one tool yet.

Yoshiyuki Blade

  • Contributor
  • Member
  • *****
  • Posts: 103
  • 5D2
RAW Video Post processing [Mac/Win/Linux]
« Reply #12 on: May 14, 2013, 01:12:41 PM »
Regarding audio recording with the raw footage, is there some kind of nonstandard sample rate that the audio recorder goes by? I could not properly sync the audio with the video no matter what at 48KHz or 44.1KHz, but if I set it to something weird like 46.6KHz, it was kinda close to sync. The footage in general is kinda jumpy and has weird stutters (not frameskipping, the module reported no dropped frames and the buffer was never exceeded) so idk if the weird jumps could cause a problem with the audio sync or if the audio is messed up too.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2141
RAW Video Post processing [Mac/Win/Linux]
« Reply #13 on: May 14, 2013, 01:22:59 PM »
yet audio recording is not scope of this module, i (and i bet alex too) didnt try too much.

Danne

  • Supporter
  • Hero Member
  • *****
  • Posts: 767
RAW Video Post processing [Mac/Win/Linux]
« Reply #14 on: May 14, 2013, 01:39:01 PM »
just a small note, for osx drag and drop a .raw on the raw2dng executable don't works and double clicking it will run it without any raw file selected

Nice to have the files in a subfolder. Next step would be to be able to throw the on the mac-app script all at once. Is it hard to do Scrax? Now I,m extracting them one by one.
Thanks!
/D

Yoshiyuki Blade

  • Contributor
  • Member
  • *****
  • Posts: 103
  • 5D2
RAW Video Post processing [Mac/Win/Linux]
« Reply #15 on: May 14, 2013, 01:46:55 PM »
yet audio recording is not scope of this module, i (and i bet alex too) didnt try too much.

Ah indeed. I understand that audio is not the focus your efforts atm so it's not a big issue for me either. I was curious to see how much "overhead" the audio recorder would add while recording RAW and it turns out to be virtually negligible. The syncing issues might've existed before RAW recording too, but I never explored it. After fooling around a little more, I needed to adjust it to about 46.8KHz to look reasonably synced in the ~1.2 mins my card does in RAW.

olik

  • Freshman
  • **
  • Posts: 81
RAW Video Post processing [Mac/Win/Linux]
« Reply #16 on: May 14, 2013, 02:07:32 PM »
Hey folks,

I do not have after fx and I failed to open the DNGs with Davinci resolve software, FCX and premiere pro, any other software that could work?

thanks for help

o.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6742
  • 5D Mark Free
RAW Video Post processing [Mac/Win/Linux]
« Reply #17 on: May 14, 2013, 02:14:11 PM »
I use ufraw.

cinema5D

  • Contributor
  • New to the forum
  • *****
  • Posts: 11
RAW Video Post processing [Mac/Win/Linux]
« Reply #18 on: May 14, 2013, 02:24:25 PM »
As posted in the other article we put together a small GUIDE covering the basic steps to get the 5D recording RAW: http://c5d.at/1o4
Please give us feedback if anything in there is wrong or easier to achieve or outdated and we will update the guide.

Also we would like to let you know that the DNG format is extremely unwelcome to non-photographers. If there is any way to get cinemaDNG files instead, that would be very helpful to many.
But hey, it's amazing to be at this point here, so we will all be patient for this to evolve.

Thanks and good light!

ch_d

  • Supporter
  • Freshman
  • *****
  • Posts: 79
  • 5D MII
RAW Video Post processing [Mac/Win/Linux]
« Reply #19 on: May 14, 2013, 02:31:18 PM »
All questions regarding RAW VIDEO POST PROCESSING on any OS
HERE

What would be really great: Something like nice workflow for proxy editing:
a) convert the raw to dng (allready here, called raw2dng)
b) ! in the same step as raw2dng make a proxy for editing (like mjpeg or even prores proxy)

so you can edit the files and reconnect them with the raw dng afterwards. like online/offline editing.
5D MII

olik

  • Freshman
  • **
  • Posts: 81
RAW Video Post processing [Mac/Win/Linux]
« Reply #20 on: May 14, 2013, 02:33:41 PM »
I agree with cinemaDNG, if it helps, I've found some CinemaDNG white-papers http://download.macromedia.com/pub/labs/cinemadng/cinemadng_p1_spec_091009.pdf

Colemar

  • Freshman
  • **
  • Posts: 52
RAW Video Post processing [Mac/Win/Linux]
« Reply #21 on: May 14, 2013, 03:36:25 PM »
scrax,
   raw2dng for Mac is not able to resolve filepaths with name spaces.  Once I copy the .raw dataset to a contiguous filepath name, converter works great.

EDIT:  Unrelatedly, has anyone noticed .raw files that record to 4GB or larger won't convert in raw2dng?  On Mac OSX I get an "this ain't a lv_rec file" error.

 thanks.
« Last Edit: May 14, 2013, 03:43:38 PM by Colemar »

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6742
  • 5D Mark Free
RAW Video Post processing [Mac/Win/Linux]
« Reply #22 on: May 14, 2013, 03:50:11 PM »
Are these files larger than 4GB or they are cropped at 4GB?

The metadata is at the end of the file, so if it's missing, all you can do is to fill in the metadata yourself or hack the source code of the converter.

Colemar

  • Freshman
  • **
  • Posts: 52
RAW Video Post processing [Mac/Win/Linux]
« Reply #23 on: May 14, 2013, 03:58:16 PM »
Are these files larger than 4GB or they are cropped at 4GB?

The metadata is at the end of the file, so if it's missing, all you can do is to fill in the metadata yourself or hack the source code of the converter.

they are larger, usually ~4.28GB according to OSX Finder properties.  It seems like a corrupt header because I also have a couple ~2.07GB .raw files that are not recognised in the same manner.  Is this a quick fix or do you want a smaller file Alex?


mucher

  • Member
  • ***
  • Posts: 118
RAW Video Post processing [Mac/Win/Linux]
« Reply #24 on: May 14, 2013, 03:58:39 PM »
Thanks for the white paper. Good exercise for the brain.

Here I've found a link to convert DNG to tiff first, saidly, Resolve can open the tiff sequence.

http://www.completedigitalphotography.com/?p=521