Author Topic: RAW Video Post processing (raw2dng) [Mac/Win/Linux]  (Read 632384 times)

buildbyflying

  • New to the forum
  • *
  • Posts: 6
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #225 on: June 07, 2013, 06:39:47 AM »
Pouring through the forum on post=processing 2gb+ files.
Are users getting raw2dng to work on windows?
It seems I'm reading about users doing this with Batchelor on mac (but I've found nothing to confirm about windows)

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #226 on: June 07, 2013, 01:01:56 PM »
Pouring through the forum on post=processing 2gb+ files.
Are users getting raw2dng to work on windows?
It seems I'm reading about users doing this with Batchelor on mac (but I've found nothing to confirm about windows)
I don't think batchelor works on mac
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

tin2tin

  • Member
  • ***
  • Posts: 158
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #227 on: June 07, 2013, 02:31:23 PM »
Would it be possible to add -o "output dir" to the command line options of raw2dng?

Then it wouldn't be necessary(takes time) to copy the source RAW files to the same folder as raw2dng before processing them.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #228 on: June 07, 2013, 02:33:56 PM »
Just type it without the -o ;)

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #229 on: June 07, 2013, 02:39:28 PM »
Would it be possible to add -o "output dir" to the command line options of raw2dng?

Then it wouldn't be necessary(takes time) to copy the source RAW files to the same folder as raw2dng before processing them.
to avoid that on osx I start raw2dng from the folder where I want the dng with correct addresses to raw2dng and RAW file

like raw2dng is in ~/User/App/raw2dng
RAW is in ~/User/Video/M000001.RAW
dng need to go in ~/User/Video/M000001/000000.dng...

"go to ~/User/Video/M000001/"
cd ~/User/Video/M000001/
"start raw2dng from there"
~/User/App/raw2dng ../M000001.RAW
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #230 on: June 07, 2013, 02:40:02 PM »
Just type it without the -o ;)
is that new?
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

tin2tin

  • Member
  • ***
  • Posts: 158
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #231 on: June 07, 2013, 02:44:01 PM »
I'm not sure I understand. Without -o I'm only specifying the import folder/file(which also is the output folder/file).

What I'm asking is the option to specify a different output folder than the source folder.

raw2dng "sourcefolder" "outoutfolder" - isn't that read as two source folders?

EDIT:
Just tested that with the latest raw2dng:
Code: [Select]
raw2dng myraw.raw myfolderMyfolder was already created, but the resulting dng files ended up in the same folder as the myraw.raw file.

EDIT2:
raw2dng "C:\Users\user\Desktop\myraw.raw"
Places the files in the location of raw2dng.  :)

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #232 on: June 07, 2013, 02:57:23 PM »
I'm not sure I understand. Without -o I'm only specifying the import folder/file(which also is the output folder/file).

What I'm asking is the option to specify a different output folder than the source folder.

raw2dng "sourcefolder" "outoutfolder" - isn't that read as two source folders?

Just tested that with the latest raw2dng:
Code: [Select]
raw2dng myraw.raw myfolderMyfolder was already created, but the resulting dng files ended up in the same folder as the myraw.raw file.

when yu do raw2dng "source" you are in the .raw folder, that is causally the same of the destination folder. Destination folder is the one you are when starting raw2dng.exe so just start it from myfolder:
Code: [Select]
dir myfolder
..\raw2dng.exe ..\myraw.raw
not really sure that the code is right for win but sort of
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #233 on: June 07, 2013, 03:00:25 PM »
Try this: raw2dng myraw.raw myfolder/

tin2tin

  • Member
  • ***
  • Posts: 158
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #234 on: June 07, 2013, 03:05:46 PM »
He he, this works!  8) 

Thanks!!

Steadycamer

  • New to the forum
  • *
  • Posts: 11
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #235 on: June 07, 2013, 06:50:43 PM »
I made a script where you can drag & drop multiple raw file for batch processing

it will create as many directory as raw files (directory will be named with raw filename)

http://www.steadycamer.com/batch2dng.rar


* raw2dng.exe,   batch2dng.bat  and raw files have to be in the same directory!

random

  • New to the forum
  • *
  • Posts: 46
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #236 on: June 07, 2013, 08:52:15 PM »
Random, H264 is not a good option to export your footage for the codec limitation. Forget it excepto upload a final video to internet. Before, for editing and color post, use a codec with more information as pro res, avi uncompressed to edit IN YOUR software like premire or better, final cut...

Alright thanks, so I can do it like DNG->losless AVI->Premiere->DNxHD->h264 by Stremclip.
And how about workflow with TIFFs or DPX? DNG->TIFF16 bit>losless AVI (?)>Premiere>DNxHD 10bit (?)>h264?
EOS 6D  -  EOS 650D  -  EF 24-105mm/4 IS L  -  Sigma 50mm/1,4 EX  -  Kenko 1,4x MC4 DGX

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #237 on: June 08, 2013, 02:52:02 AM »
Alright thanks, so I can do it like DNG->losless AVI->Premiere->DNxHD->h264 by Stremclip.
And how about workflow with TIFFs or DPX? DNG->TIFF16 bit>losless AVI (?)>Premiere>DNxHD 10bit (?)>h264?

Appears to be fine for me.
I prefer Pro Res, because I used to edit in Final Cut X.

johanhenrikssn

  • New to the forum
  • *
  • Posts: 3
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #238 on: June 08, 2013, 09:34:15 AM »
I want to start by giving a huge thanks to everyone involved with Magic Lantern and this forum. It's impressing to see the tool progress.

I shot some footage that looks like it's sped up after post-processing it. It gives the impression that it was recorded in another, lower frame rate than 25, when looking at it after it's been post-processed. I did not have FPS override to ON in magic lantern, could this have affected it?

I used the june 3rd-version with 5D MKIII. Camera settings i Canons menu set to PAL and video set to 1080p 25fps ALL-I.

In post when converting to dng's with raw2dng.app, it says that it was shot with 25 fps. That's problably just camera metadata though? I tried the workflow with both After Effects and using Quicktime 7 via Photoshop. When importing the image sequence inside After Effects it came in as 30 fps. I interpreted the image sequence from 30 to 25 and created a comp from that. That's when I get that sped up look. Anyone else experienced this problem?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #239 on: June 08, 2013, 06:24:12 PM »
Just uploaded a new raw2dng.exe. If anybody is still having banding issues with it, please upload the first DNG from the video.

Regarding file format, starting from today, frame size might be a bit higher than w * h * 14/9 (it's rounded to 4096 bytes). If your third party converter uses the same file reading loop as raw2dng, it should be fine. If not... you'll have trouble at certain resolutions.

pavelpp

  • Freshman
  • **
  • Posts: 92
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #240 on: June 08, 2013, 07:48:30 PM »
Guys, just wanted to share my workflow on Mac here, if anyone is interested:

1) use raw2dng.app to export frames
2) import dng's to Adobe Lightroom
3) white balance / tint correction, if needed
4) export to tif (frames become 4 times the size of dng!)
5) Next you need to make sure you have ffmpeg installed. In Terminal cd to the tiff directory and run command:

ffmpeg -f image2 -r 24 -i %6d.tif -vcodec libx264 -preset veryslow -qp 0 -pix_fmt yuv420p -vf unsharp=5:5:0.5 movie.mkv

This is supposed to produce almost lossless output, however I am not sure about that. Maybe it's better to use yuv422p or yuv444p ? Unsharp filter is used here, because by default the output is quite blurry.

fatpig

  • Member
  • ***
  • Posts: 163
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #241 on: June 09, 2013, 01:29:48 PM »
hi guys,
I am looking to implement spanning support in BATCHelor.

Can someone tell me if the latest raw2dng supports spanned files without having to join them first?
this is another step in the chain i would like to avoid.

is it possible to use raw2dng on the .RAW , the .R00 and the .R01 separately, or does it only work for .RAW that is joined beforehand?
I would like to avoid joining files on harddisk first..

PS: if someone could send me R00 and RAW i would appreciate it. :)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #242 on: June 09, 2013, 01:34:31 PM »
No, but you are welcome to implement it :D

John Hable

  • New to the forum
  • *
  • Posts: 27
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #243 on: June 09, 2013, 09:51:49 PM »
hi guys,
I am looking to implement spanning support in BATCHelor.

Can someone tell me if the latest raw2dng supports spanned files without having to join them first?
this is another step in the chain i would like to avoid.

is it possible to use raw2dng on the .RAW , the .R00 and the .R01 separately, or does it only work for .RAW that is joined beforehand?
I would like to avoid joining files on harddisk first..

PS: if someone could send me R00 and RAW i would appreciate it. :)

IIRC this isn't possible.  In some cases the frames get split between between the files, so there is no way to process just one at a time.  Also, the footer is only at the end of the last file.  If you want to extract frames without merging you have to:

1.  Take the .RAW file as input.
2.  Look in the directory structure, counting upwards (R00, R01, R02, etc) to find the last file in the chain.
3.  Extract the footer from the last file in the chain.

Then to extract the frame data you have to figure out how many bytes it is from the start, and read that section from the data knowing that it might be split across two files.  So yes, you can extract the files without joining them first, but no, it's not possible to handle each of the sub-files independently.



random

  • New to the forum
  • *
  • Posts: 46
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #244 on: June 10, 2013, 08:58:41 PM »
Just uploaded a new raw2dng.exe. If anybody is still having banding issues with it, please upload the first DNG from the video.

Regarding file format, starting from today, frame size might be a bit higher than w * h * 14/9 (it's rounded to 4096 bytes). If your third party converter uses the same file reading loop as raw2dng, it should be fine. If not... you'll have trouble at certain resolutions.

Is there a reason why my bitdefender detects a Gen:Variant Kazy.171435 virus in raw2dng.exe?
EOS 6D  -  EOS 650D  -  EF 24-105mm/4 IS L  -  Sigma 50mm/1,4 EX  -  Kenko 1,4x MC4 DGX

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #245 on: June 10, 2013, 09:34:30 PM »
Uploaded new raw2dng, with bad pixel fix from marekk. Very cool trick, didn't know about it.

dlrpgmsvc

  • Senior
  • ****
  • Posts: 398
  • The 7D and 50D Robin Hood
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #246 on: June 10, 2013, 11:19:38 PM »
Do there is a postprocessing solution that takes a ml raw video file, extract the dng files and fills the dropped frames automatically with twixtor ?
If you think it's impossible, you have lost beforehand

random

  • New to the forum
  • *
  • Posts: 46
Re: RAW Video Post processing [Mac/Win/Linux]
« Reply #247 on: June 12, 2013, 01:10:11 AM »
Where can I get the old raw2dng.exe without detecting this virus?
Doesn't work for me anymore, neither with rawanizer.
EOS 6D  -  EOS 650D  -  EF 24-105mm/4 IS L  -  Sigma 50mm/1,4 EX  -  Kenko 1,4x MC4 DGX

mkrjf

  • Freshman
  • **
  • Posts: 53
Re: RAW Video Post processing [Mac/Win/Linux] can't convert to DNG
« Reply #248 on: June 12, 2013, 12:56:27 PM »
I installed ML on my MkIII and generated just under a second or so of video for several clips (don't have 1000x cards yet just 400x).
Using RawMagic I get deep magenta for a clip of a white wall illuminated with 5600K LED. Photos with ML active for same scene are normal - looks like white wall with range from bright to dark shadow. (and yes I posted that to the RawMagic thread ;)

Also tried raw2dngapp for OSX - segmentation fault with some reference to a library
Tried raw2dng in terminal - also segmentation fault with no error messages

So not super productive although I greatly appreciate all the development work that has been done.
Do other mac users (latest SnowLeopard updates on macbookpro17) not have issues?
I saw someone is working on FFMBC tool - which is great - not sure what status of that work is.
I have cinelerra - but the tool for that is .exe (sorry I am not getting pc just to run your program no matter how great :(

Would be great to have ffmbc command line that would take raw and output prores444 or some other mastering quality standard codec.
I guess the author of ffmbc does not have a canon 5dmk3 or else the raw codec would exist ;)

BTW what is the actual dynamic range captured in the raw file output by ML - and is the debayering done by raw2dng? and is it using a standard algorithm? If technical spec.s are documented please share the link. I have searched but maybe am not using the right tags.

THX!
Mike

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: RAW Video Post processing [Mac/Win/Linux] can't convert to DNG
« Reply #249 on: June 12, 2013, 01:58:05 PM »
..BTW what is the actual dynamic range captured in the raw file output by ML - and is the debayering done by raw2dng? and is it using a standard algorithm? If technical spec.s are documented please share the link. I have searched but maybe am not using the right tags.
raw2dng has a simple dabayering-calculator for the preview-picture - the main picture (in a subIFD) isn't changed. its still bayered raw data.