RAW Video Post processing (raw2dng) [Mac/Win/Linux]

[Redrocks]

Touch wood, it looks like ML RAW video is here to stay and most of us will be using the ML variety to some degree for the forseeable future. The many workflow guides are a godsend, but I searched the thread and found no real mention of it: is anyone interested in conducting some experiments into RAW workflow?

ML RAW video is still bleeding edge and there could be changes to the actual files we are transfering to our desktops, but for now we have 14 bit rough diamonds to polish up and it seems there are many ways to do this. It would be handy if we had a standard set of files that we could use to a) nail down the process for getting the best out of our footage b) establish benchmarks for the constantly updating software we use.

We agree on one set of shots covering the ranges we encounter (low light, sunny, charts...) we host them and each of us has access to the same files. We then run them through our own personal workflow and deliver in the agreed format with info about what we did.

Obviously open to all thoughts on this.

[a1ex]

Sounds good, and having some reference footage is great for fine-tuning the convertors.

For example, it was quite hard to find a banding correction algorithm that covers pretty much all practical cases, and I could use this to verify any future changes in this algorithm.

I'd say you can open a separate thread for this; it may become quite large.

[Redrocks]

I think a set of constants will help all of us in different ways. Do I start the thread on the workflow or RAW board?

[tob]

I want to start by giving a huge thanks to everyone involved with Magic Lantern and this forum. It's impressing to see the tool progress.

I shot some footage that looks like it's sped up after post-processing it. It gives the impression that it was recorded in another, lower frame rate than 25, when looking at it after it's been post-processed. I did not have FPS override to ON in magic lantern, could this have affected it?

I used the june 3rd-version with 5D MKIII. Camera settings i Canons menu set to PAL and video set to 1080p 25fps ALL-I.

In post when converting to dng's with raw2dng.app, it says that it was shot with 25 fps. That's problably just camera metadata though? I tried the workflow with both After Effects and using Quicktime 7 via Photoshop. When importing the image sequence inside After Effects it came in as 30 fps. I interpreted the image sequence from 30 to 25 and created a comp from that. That's when I get that sped up look. Anyone else experienced this problem?

I have exactly the same setup/camera as you have and the same problem, but new version of ML 10 June. I have exactly the same issue. I used the raw2dng.exe - 10 june. 

[random]

If I want to get the highest result using a workflow with exporting a TIFF 16bit sequence from AFX, how can I make Premiere CS6 able to handle it? It's just working until 8bit, but isn't there a workaround?

If not, there's no chance for getting more than 8bit if I want to edit my footage in Premiere, right?
And what happens if I export to DNxHD 10bit? Shouldn't be any advantage compared to 8bit then.

[deleted.account]

Anyone know if raw2dng can use standard output, or if not can the option be added a1ex?

[mkrjf]

raw workflow thread would be nice

I also figured out why my video had magenta cast - it was just preview issue - not using correct LUT

But looking at magicraw created dng in Adobe - image looks fine but is listed by Adobe as 8bit

Now for what is captured anyway (should be in another thread?):
Since 2Mpixel frame is taking up 4Mbytes - that would imply 16bit per pixel (14/16 of which is data?) - can someone outline the detailed mapping of sensor bayer elements to data in the DNG file?
There is a claim that Canon 5Dmk3 'photosites' for 1920x1080 recording use 9 bayer elements (as below) to get one 'pixel'
GBG
RGR
GBG
http://www.eoshd.com/comments/topic/2696-big-news-hands-on-with-continuous-raw-recording-on-canon-5d-mark-iii/page-6
But their selection of 9 elements does not seem right in their diagram (some would be almost only red info).

what exactly does magic lantern code do to read the sensor and generate 14bits of what per pixel?

Or is magic lantern just reading alternating bayer elements - so that we actually have 14 bits of information in a 16 bit datum at 1024K of green, 512K of red and 512K of blue - 960x540x2 true green resolution, 960x540 true blue resolution, 960x540 true red resolution.
So spatial resolution should be 1K by 1K?

Where Adobe CS4 (for example) is debayering (what algorithm) and interpolating to 1920x1080 at 24bpp RGB (all derived not actual)
and preview just dumped to screen - not sure how that added up to magenta

So using 9 'bayer elements' to get 1 'pixel' seems like it would be higher spatial and color resolution than simple alternating R/B rows and */G columns.

Any Canon / ML experts that can explain what goes into a ML raw video pixel for 5DMK3? Is it using Canon algorithm to derive pixels or bypassing it? I am guessing bypassing it.
And is it different algorithm for crop mode versus subsample of full sensor? Some people are saying crop is sharper - is that accurate?

Guess it is impossible to read whole sensor and debayer on the fly - that would be a weighted convolution operation and too much cpu on camera?
Higher end Canon for cinema does something like that, right?

[budafilms]

Any news or link about work directly in Final Cut X? Thanks!

[frspp]

Could the opening post include also commandline howto (taken from http://www.magiclantern.fm/forum/index.php?topic=5533.msg37819#msg37819 ):

Joining files
WIN
    Start dialog: cmd
    1. cd C:\Users\my_destination_folder
    2. copy /B xyz.RAW+xyz.R00+xyz.R01 xyz.RAW

OSX/LINUX
    Start Terminal
    1. cd /Volumes/my_destination_folder
    2. cat xyz.RAW xyz.R00 xyz.R01 > xyz.RAW

    Tips: cmd and Terminal support drag & drop => Don't need to type!

[mkrjf]

Okay so no response on internal read algorithm for ML
So I will hunt through the source when I get a chance and try to reverse engineer and document
If sensor could be read efficiently from alternating pixels - perfect interpolation would be possible in post processing resulting in high quite 4K video!!

[Sthirasukha]

I don't want to be boring but can someone tell me where I can find a Raw2dng able to convert a 10 GB raw file (I mean bigger than 4GB)?
I tried with the Mac and PC version but none of them can do it.

[camilitox]

here a link to the raw2dng compiled for mac (no wine needed)

Has anyone the compiled version for linux? Thank you. (Sorry for my bad English)

[pavelpp]

Am I stupid or OSX version just does not work? I run in terminal and get

Code: [Select]

-bash: raw2dng: command not found

I did chmod a+x for the executable, does not seem to help...

[meditant]

aw2dng stop working !

On my mac 10.8.4, its generate dng and .mov but the .mov files can be read by quicktime X. and the .dng files, can't be opened by resolve, a strange lowres picture with flat blue filter. i can open the dng with photoshop ?

I use resolve 9.1.4 lite, and a 5D Mark III

3 day ago thats work fine (but maybe with resolve 9.1.3 i dont remember) ! (I reinstall 9.1.3 but the same bug, i reinstall 9.1.4)

The same raw file converted by magicraw is ok and can be open by resolve.

I uninstall raw2dng, and reinstall, reboot and do standard administration tash but no change.

The same problem with the GUI os x app raw2dng.

a capture of resolve :


Do you have idea ??

Regards

[camilitox]

Hi there, I want to share, the script, which I use to convert RAW to PNG in Linux. with aspect ratio 2.35:1, with the generation of the .toc file for editing with cinelerra =)

Code: [Select]

#!/bin/bash
# Converts RAW videos to PNG sequence for use with cinelerra
# usage: raw2png foo.raw
fbname=`basename "$1" .RAW`
wine raw2dng $1
ufraw-batch --out-type=tif *.dng
rm *.dng
mkdir "$fbname"
avconv -f image2 -r 24.0 -i %06d.tif -c:v png -s 1920x840 -r 24.0 ./"$fbname"/%06d.png
echo PNGLIST >> "$fbname".toc
echo 24 >> "$fbname".toc
echo 1920 >> "$fbname".toc
echo 840 >> "$fbname".toc
ls $PWD/"$fbname"/*.png >> "$fbname".toc
rm *.tif


[Sthirasukha]

Just for your information, it's possible to convert the DNG files (from Raw2dng) to Tif with Capture One (6.4 and 7). ACR is not the only one.
It might help some of us even if the workflow is better with Rawmagic + Resolve.

[MarkvC]

Not to keep bringing this up but is anyone having success converting RAW's to DNG's on files larger than 4GB. I have tried both RAW2DNG and RawMagic on my mac and had no success. Smaller clips than 4GB's work great.

[tin2tin]

Have you tried this:
http://www.magiclantern.fm/forum/index.php?topic=5404.msg51806#msg51806

[kshoban]

Hello everyone, I think I'm posting in the correct location (at least I hope so). I am incredibly frustrated. I have been searching these forums for the past 2 days and all I see are people saying "I uprezzed the video." Well, that's great, but What the heck? Can someone please explain to me how to uprez raw video for the the 600d in adobe premiere cs5? Right now I am only able to record at 640x369 and 704x416. I'm trying to uprez to at least 720p. An explanation on how to do this in either premeiere cs5 or after effects would be amazingly helpful. thank you so much. If this is posted in the wrong forum can someone please help direct me to the proper place to post this? thansk again.

just FYI:

I have already changed the footage to tiff and made it into a .mov file to edit in premiere.

-Kevin

[rue]

Dear all, think I'm doing sth. stupid regarding the framerates in my first little 50D raw workflow, so hints and help appreciated:

- Recorded with the May 18th build, FPS Override 23,968
- raw2dng > LightRoom 3 (color & exposure correction) > Export to 1920 x 1080 JPEG
- Import as Image Sequence into PPro CS6 always gives me clips interpreted with 25 fps

How can I convice PPro to interpret the sequence in 23,968 to avoid the ugly artifacts when exporting it?

Sorry, I'm sure I'm missing sth. obvious here 

THX

[rue]

Ok - found the reason for my framerate issue: when you import still-image sequences PPro interprets them according to the settings under Preferences > Media > Intermediate Timebase and not according to the project/sequence settings. This setting defaults according to your locale (so in my case 25 fps). Unfortunately you can't change it for the image sequence (interpret footage) once it is imported.

Hence - make sure that the intermediate timebase is set according to the framerate you wish your image sequence to be interpreted BEFORE you import it.

[yanone]

Hi everyone, I’m new here. You people rock!

I have two feature requests for the converter tools.

Problem: Combining several >4GB files into one via "cat * > file.raw" takes forever because it happens in place on the same disk (and I already have a speedy SSD. Don't even want to think about what happens on an HDD).

Related: What I started to do with <4GB files is to extract DNGs directly from the CF card to the hard disk with RawMagic, effectively saving one step of processing, because it's the bus speed in my system (USB2 and cheap card reader) that is the speed bottleneck. So instead of copying the raw file first and then extracting the DNGs in a second step, I load the raw files directly from the CF card into RawMagic.
In other words: Extracting the files directly from CF takes exactly as long as only copying them.
This way I also save half of the required disk space because the raw files never even make it to the hard disk.
By the way, the same bus speed issue from above applies here, because extracting DNGs from hard disk to the same hard disk requires simultaneous reading/writing.

Request 1: Please implement that the raw converters read all individual "filename.*" files (.RAW, .R00, .R01, ...) consecutively. The minimalistic "cat * > file.raw" command makes me believe that this should be possible. And the files’ relation is unambiguous. So one would only insert the first (.RAW) file into the process and the converters would look up existing .R** files in the same directory.

Request 2: Please implement an output path option for the command line tool. As far as I understand it it works in place directory-structure-wise.
This way one could use the direct CF->HDD conversion outlined above with the command line tool. I'm a big fan of command line tools as I expect to implement some sort of automatized copying/sorting with scene numbers and comments and stuff.

I will use ML/RAW for shooting a dance art film in about a month, whether you guys approve of that (stability-wise) or not 

Thank you.

[slingerland]

My work-flow is at this moment 1) Have photoshop raw convert  the dng filies in to either tiff, and then I go the quiktime 7 route (not so happy about that route because of the way it comes out) have photoshop make a video file of it, mp4 or a quiiktime movie made of "animation". Last one gives big files. I want to get to finalcutpro so i also used compressor to convert the mp4 to prores. There is got to be a better route?

[SoulState]

My 2 workflows:
First (Slow)
1. Convert from raw to 16-bit dng with raw2cdng (16 bits give me best results in terms of color reproduction and and "filmic look")
2. Import dng's into After Effects with ACR plugin, some tweaking
3. Apply log lut ( PHOTOGS LOG LUT http://www.megafileupload.com/en/file/431990/index.php ) as starting point for filmic look and further color correction (Not really needed, but i like flat log "filmic" look, you can make this in ACR, but using lut for that is faster).
4. Export video with HuffYUV codec (losless) (i use huffyuv from ffdshow) or any codec like x264 if you want final compressed file.

Second (Fastest)
1. Convert from raw to 16-bit dng with raw2cdng
2. Import dng's into Photoshop with ACR plugin, some tweaking, synchronize and export as 100% jpegs (i use jpeg to save some space, but you can save stills in some losless format)
3. Import jpegs to VirtualDub, set the framerate, and resize (if needed) from 1728x972 (60d footage) to 1920x1080 (it's 111.111% upsizing), for upsize i use Lancoz3 mode, it's little sharper than precise biqubic (a=-1.00). I usually don't resize, if i want just to look at resulting video clip.
4. Export video with HuffYUV codec (losless) (i use huffyuv from ffdshow) or any codec like x264 if you want final compressed file.

Second method about 2-3 times faster if you don't need heavy color correction or effects in AE.
In first time i used Lightroom for second method, but with using just ACR plugin in Photoshop i have much faster workflow.

I have only one question - how to convert Silent Picture DNG to normal DNG to open it in ACR? My silent pics dng won't opens nor AE with ACR plugin, nor Photoshop's ACR nor with Lightroom... ( Camera is 60d). I shoot some timelapses, but just can't open dng raws...

[fatpig]

Is somebody working on automatic processing of .R00, .R01, etc. Files in RAW2DNG?

Would be huge for my Software BATCHelor.