Messages

So I did some stuff with rawtherapee batching a while back. At that point you needed to create a saved profile via the GUI and then run it via command line. These were the cmd line options rawtherapee -t -Y -p profilepath/profile.pp3 -o outputdir -c inputdir

-t = 16 bit tiff
-Y = overwrite
-p = profile
-o = output dir
-c = input dir to convert

The rawpedia entry (not sure how long the IP is valid for, but it sure is interesting to see a website run on an IP these days.)
http://50.87.144.65/~rt/w/index.php?title=Command-Line_Options

My archaic batch file VBS and BAT combined, including multiple command line batch commands for different raw programs and compiling into movies with ffmpg based on the old raw format (before mlv)

https://drive.google.com/file/d/0B2sc-xLyUSKqQWdPdlFHZjlPaU0/edit?usp=sharing

Chris,

Ever since the article from National Geographic I have been wanting to see a more down to earth perspective of Son Doong. I like more behind the scenes style imagery but still done with professional equipment and the with same care as the magazine cover. You managed to give me just that with pictures which could still be on the front page and a story behind the scenes too!

Thanks!

I just got my D50 last week and am trying to catch up with all the new stuff. I tried to find some tutorials on Raw Therapee, either on the command line or otherwise but couldn't find much. I rarely buy software for anything but I'm sensing that many people on these forums has spent a fair bit of money on something from Adobe. The only other free alternative I've heard of is UFRaw but again finding good tutorials is something that's eluding me. Please let us know if you find anything useful.

ADJ and also to trueindigo there are a couple more options in the open source world that I can put out there since I like to experiment to.

Dcraw all platforms - insanely fast minimal options all command line

Photivo all platforms - very detailed and lots of options (eg. 5 types of denoise, 5 types of sharpening, both bayered and debayered methods). When I want the most out of a raw file I use this

Darktable linux mac - great with color with stackable filters (eg. apply the same filter 5x differently each time), my favorite interface besting even LR after setting a few hotkeys, but it does not cache anything kinda slows things down

RawTherapee all platforms - as already mentioned this one is pretty nice

I have been able to do batch processing from command line with all of these except I have not tried with darktable yet. ADJ can you tell me what you use for video editing.

Hi True Indigo

Thanks for taking the time to reply. I saw the documentation before your post. I searched for "sequence" and "dng" in their 105 page document but it didn't return anything useful. I've just looked again and realise "batch" and "command" would have been a better searches.  I thought I'd be more likely to get a sensible answer on here rather than on their forums since everyone shooting raw video is doing much the same thing albeit with different software. I didn't think many people shooting stills would be trying to do what we're doing. Anyway for the record their documentation is better than I first thought so I'll have a read at p91 on batch processing and p95 on the command line. I think I read somewhere that the batch operations are more stable.

Anyway I'm not in a rush. I have dng files but won't do anything serious until my adapters arrive. I'm going to take a leaf out of goldenchild9to5's book and use some old pentax prime lenses I have which are much faster than my canon zooms. One's an M42 and the other is a PK mount.  I took my first real pictures on a pentax s1a many years ago and that didn't even have a light meter of any kind on it. I had to use and old Weston handheld meter. I never thought I'd be using the lenses again but I found them in an old kit bag the other day while I was looking for something else.

So I have traveled this road already RawTherapee works nicely for batch processing. It is slower than ACR but I like the color handling that can be done with it. I built a batch program which allows you to build a preset in RawTherapee and then select the a preset to use with your raw files. It will allow your to drag and drop a whole directory of raw video, expand them to dng, and then apply presets to each one, after processing it uses ffmbc to make a prores444/yuv444p10 .mov file. Its all batch so feel free to borrow any of the command line code.

The biggest drawback is for some reason in batch mode RawTherapee seems to process white balance, sharpening, and microcontrast on the first file in the sequence horribly wrong your first frame will be unusable.

Your actual batch command might look like
rawtherapee -t -Y -p "%_presetFile%" -o "%CD%" -c "%CD%"

The batch file is going to look for dcraw.exe ffmbc.exe libiconv2.dll libintl3.dll raw2dng.exe regex2.dll and sed.exe in the folder with it. The zips in the linked thread provide those files. Raw2dng.exe is horribly out of date though.

Latest version of the windows based .bat file only I have
https://docs.google.com/file/d/0B2sc-xLyUSKqQWdPdlFHZjlPaU0/edit?usp=sharing

http://www.magiclantern.fm/forum/index.php?topic=5837

Found the error and working on fixing it. It does not handle spaces in the filepath. Thanks!

For windows ... Download dcraw http://www.heliconsoft.com/heliconsoft-products/dcraw/

Add this line to a batch file for the most basic conversion. Execute in the directory you want to make .tiff files.
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2.132483 1.000000 1.480864 1 -H 2 -T -b 3.5 -4 %%f

For .RAW to tiff Make a batch file and add the spoiler below. Should give you even basic white balance.

[spoiler]
@echo off
setlocal enabledelayedexpansion

:again

set thefile="%~1"
REM make directory for extraction of
if "%~1" == "" goto done
mkdir %n%
move %~1 %n%
cd %n%
for %%F in (%thefile%) Do Set _RawName=%%~nxF
..\raw2dng.exe %cd%\%_RawName%
move %_RawName% ..


REM get user choice of white balance
ECHO WScript.Echo InputBox( "What preset would you like to use?"  ^& vbnewline ^& "1 Daylight"  ^& vbnewline ^& "2 Tungsten"  ^& vbnewline ^& "3 Cloudy"  ^& vbnewline ^& "4 Shade"  ^& vbnewline ^& "5 Flourescent"  ^& vbnewline ^& "6 Flash or LED"  ^& vbnewline ^& "7 Default", "Preset", "1" ) > usermessage.vbs
FOR /F "tokens=*" %%A IN ('CSCRIPT.EXE //NoLogo usermessage.vbs') DO SET wb=%%A
del usermessage.vbs

if "%wb%"=="1" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2.132483 1.000000 1.480864 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)
if "%wb%"=="2" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 1.392498 1.000000 2.375114 1 -H 2 -T -b 4.5 -4 %%f
goto settif
)
if "%wb%"=="3" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2.336605 1.000000 1.334642 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)
if "%wb%"=="4" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2.531894 1.000000 1.223749 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)
if "%wb%"=="5" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 1.783446 1.000000 1.997113 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)
if "%wb%"=="6" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r  2.429833 1 1.284593 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)
if "%wb%"=="7" (
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2.395443 1.000000 1.253807 1 -H 2 -T -b 3.5 -4 %%f
goto settif
)

:settif

cd..
set /a n+=1
shift
goto again
:done

del *.vbs
pause
exit
[/spoiler]

This is an excerpt from my more full featured batch maker http://www.magiclantern.fm/forum/index.php?topic=5837 which will get you all the way to .mov files.

I finished my batch processor to convert the .raw file to .mov prores4444 ! Uses the power of portable app Photivo (open source alternative to lightroom/ACR) to make presets for processing and run jobs. Or it can also do a quick and dirty conversion using dcraw.

Processor v0.2 - RAWTherapee support
https://docs.google.com/file/d/0B2sc-xLyUSKqVU5UNFFZeFpyam8/edit?usp=sharing

Processor v0.1 - fixed spaces in files + bug fixes
https://docs.google.com/file/d/0B2sc-xLyUSKqcG9ka1dDWlpjelU/edit?usp=sharing

Processor v0.0
https://docs.google.com/file/d/0B2sc-xLyUSKqTDd4aUtuQXg3cEk/edit?usp=sharing


After dragging and drop RAW files onto the processor.bat and follow the prompts. A preview of the first image from the first raw to be processed will be opened in windows preview. Next select to use photivo or dcraw.

If you select photivo then choose if you want to open the file in photivo and do custom editing for the sequence. If you open a file in photivo right click each subsection header to append it to a new preset file. The output file type must stay png if you want it to work. Remember where you put the preset file to load the it later. Next you will be asked to select the preset file select the one you want. Photivo will create a batch job and do the rest.

If dcraw was selected, next you will be asked to select a white balance. These are the defaults based on a 350D. Once selected dcraw will do the rest.

The .mov is made with ffmbc and can be found in the original directory.

Learn about using Photivo.
http://photivo.org/cast

Processor v0.0
Initial release

Processor v0.1
Changes
[spoiler]
added support for infinite number of dng files eliminating the character limit from the batch variable
added support for spaces in file and directory names
updated raw2dng to build from 2013-05-21 to support 4Gb+ files
added sed.exe and needed dll files for text processing
cleared all photivo .ini settings and set it to portable mode

Known issues
Not sure how to set photivo file type from batch file. Asking developer. If you know how to fix this please share.
[/spoiler]

Processor v0.2
Changes
[spoiler]
Initial RAWTherapee support for creating a style and processing
Note needs rawtherapee installed and a path in environment variables
No longer including photivo with download
[/spoiler]

TODO :
#̶1̶ ̶A̶d̶d̶ ̶R̶A̶W̶T̶h̶e̶r̶a̶p̶e̶e̶ ̶s̶u̶p̶p̶o̶r̶t̶
Maybe pick a better name. I was really tired and uncreative.
Add support for more video types
Find a way to start more batches at once
Add %resize for dcraw
Add 8bit options for proxy editing
Learn visual C#
Reprogram in C#

DCRaw is small portable and reads the DNG files nicely

haysuess I was having the same issue. Take a look back at

I was also able clear the 145 frame issue using the files from below. And starting a zoom raw rec from any C1, C2 ,C3 modes but not the M mode.

No clue what causes this, but this voodoo process cleared it for me.
Made a ML card using the files in that post.

Downloaded the program and it looks quite useful. What settings are you using?

Are you opening the .raw file or the sequence of .dng files?

I would like an option to export say the (first, middle and last frame) or (every ##th frame) so I can make a preview before extracting all the frames. Im working on a little something with UFRaw and batch processing. Before starting UFRaw from the management tool I would like to build a preview so I can decide on which clips to use. I don't want to process every frame in a clip just to throw away B-reel. I have a SSD so processing a ~3 min clip takes up the whole thing Space is precious.

Just an example of the auto white balance at work and then the color shift at the end.

http://www.youtube.com/watch?v=oNcem045MkI

Enjoy.

Ok so it was a really dense idea to try to manually do white balance with dcraw from the command line. The prores is never close enough to use without extensive grading. I have altered my script to use auto wb. Works like a charm.

from this (-r 2 1 1.75 1)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f

to this (-a)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -a -H 2 -T -b 3.5 -4 %%f

The drawback is it is a frame by frame thing so wb can shift throughout the clip. 

I worked on a batch file. Similar thing but outputs prores4444. You could make a prores4444 or DNxHD file from your tiffs as an option aside from mp4 using ffmbc. It would make a great check-box option for those who feel like editing prores. I also found encoding a prores4444 file was not significantly more intensive than mp4.

I call ffmbc for prores4444 it using
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 movie.mov
I have not tried DNxHD yet.

Are you using color correction in your dcraw call? If so what gain and brightness are you using?

Are you using the multithreaded dcraw? http://www.heliconsoft.com/heliconsoft-products/dcraw/

Does your program issue multiple decoding processes (raw2dng -> dcraw -> ffmpeg) at simultaneously, or does it process them sequentially? My batch file is limited to sequential and I never got the hang of programming .

FFMBC main site.
http://code.google.com/p/ffmbc/

The two key commands from the batch file I worked on
[spoiler]
/f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 %~n1.mov
[/spoiler]
or see the thread
http://www.magiclantern.fm/forum/index.php?topic=5528.0

Showing 168 M in both Manual and C modes for both 006d890 and e91e2bc.

Trying to pin down how this error is made can't seem to get it yet 60+ tests and configurations. It was repeatable from the memory card with 006d890 but then I rebuilt the ML on that card and it works...

I was also able clear the 145 frame issue using the files from below. And starting a zoom raw rec from any C1, C2 ,C3 modes but not the M mode.

5D3 update files. Replace existing files. 
https://drive.google.com/folderview?id=0B7QlH_BH2m32QjRhRDhUeVBwZmM&usp=sharing
includes cf_acc: module as noted on http://www.magiclantern.fm/forum/index.php?topic=5471.msg36588#msg36588

Then as any good tester I was able to repeat the issue from the 006d890 build. And repeat the fix!

After installing 006d890 no previous build works in zoom not sure if there are newer builds to test .

Should I try pull the backup battery or is that pointless?

Same settings Global draw, zebras, spotmeter, histogram all off. Sound set as sync beep video set 1280x720 using a x400 card with 36Mb write (tested in camera). When I start a raw record in zoom it only shows (...) for ram instead of (....) when in non zoom.

Found the arrow key multiple press thanks ! Now if I just had smoother hands .

After installing build 006d890 May 19th Nightly for 5D mark III I have been infected with the 145 frame limit (at 720p) 5x zoom issue. After starting the build I enabled hack3d to test it. Next I tried the 5x zoom which stopped at 145 frames. I tested 4 previous builds (e91e2bc, f686b8c, and builds from may 15, 13) and the 145 frame zoom issue persists. I also changed sd cards and used eoscard to rebuild. Before using build 006d890, I can confirm working (centered only) 5x zoom on builds e91e2bc, f686b8c. This all lead me to believe that it changes a setting on the camera. Did I miss a fix for this? Side note digital dolly is great wouldn't mind being able to set the movement speed from a menu.

Mayo thanks for the idea I have changes the script I use! If anyone has AE I would love to see an actual time comparison between the methods.

I was messing around with ImageMagick to do the .tiff conversion and it comes with a command line utility called mogrify which works nicely as well. I cannot tell a speed difference. It does an auto brightness and an okay job of it but does not have color correction. The video comes out a beautiful shade of red. But it has a convenient resizing option if you want to upscale your images from 720 before importing into to your NLE.

Install ImageMagick http://www.imagemagick.org/script/binary-releases.php#windows
ImageMagick will install mogrify and link paths to it. Mogrify does not need to be moved to the same folder as the .bat

In the batch file replace
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
with
mogrify -format tiff -resize 150%% *.dng

You can change 150% 720 -> 1080 for any scaling ie 66% for 1080 -> 720

Thank you kgv5 for helping me try this out on another machine. I apologize for it not working I got a good nights sleep and checked it again. I found and fixed the issue. The code to call dcraw.exe was not looking for the program in the right directory. I changed the bat file to look for dcraw in the correct directory...  dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng -> ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng.

Here is the new code
[spoiler]
@echo off
set n=0
:again
if "%~1" == "" goto done
raw2dng.exe "%~1"
mkdir ^(%n%^)
move *.dng ^(%n%^)
cd ^(%n%^)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
..\ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 ^(%n%^).mov
del *.tiff
del *.dng
cd..
set /a n+=1
shift
goto again
:done
pause
exit
[/spoiler]

The files needed before dragging and dropping on the bat file
[spoiler]
C:\my pics>dir
Volume in drive C has no label.
Volume Serial Number is 24BC-5917

Directory of C:\my pics

05/20/2013  05:18 PM    <DIR>          .
05/20/2013  05:18 PM    <DIR>          ..
08/10/2012  12:33 PM           354,304 dcraw.exe
05/20/2013  05:19 PM               402 raw to prores.bat
03/20/2013  10:24 AM        14,316,544 ffmbc.exe
05/19/2013  05:14 AM       195,148,992 M0000017.RAW
05/20/2013  05:18 PM       127,411,392 M0000028.RAW
05/19/2013  05:55 PM       206,438,592 M0000031.RAW
05/16/2013  04:28 PM           243,938 raw2dng.exe
               7 File(s)    543,914,161 bytes
               2 Dir(s)  162,463,543,296 bytes free

C:\my pics>
[/spoiler]

What it should look like afterwards. Directories (0) (1) and (2) will have the videos in them
[spoiler]
C:\my pics>dir
Volume in drive C has no label.
Volume Serial Number is 24BC-5917

Directory of C:\my pics

05/20/2013  05:21 PM    <DIR>          .
05/20/2013  05:21 PM    <DIR>          ..
05/20/2013  05:20 PM    <DIR>          (0)
05/20/2013  05:21 PM    <DIR>          (1)
05/20/2013  05:23 PM    <DIR>          (2)
08/10/2012  12:33 PM           354,304 dcraw.exe
05/20/2013  05:19 PM               402 raw to prores.bat
03/20/2013  10:24 AM        14,316,544 ffmbc.exe
05/19/2013  05:14 AM       195,148,992 M0000017.RAW
05/19/2013  05:53 PM       127,411,392 M0000028.RAW
05/19/2013  05:55 PM       206,438,592 M0000031.RAW
05/16/2013  04:28 PM           243,938 raw2dng.exe
               7 File(s)    543,914,164 bytes
               5 Dir(s)  161,767,456,768 bytes free

C:\my pics>
[/spoiler]

I concocted a a batch file as well to do roughly the same thing. You need ffmbc and dcraw in the folder with the script for it to work. Don't have a clue how to set metadata on the dng to fix WB. The script converts the dng files to 16 bit tiff files and applies some color correction via  dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng. The color correction is the -r 2 1 1.75 1 portion, pick a single dng and fiddle with the numbers for perfection. Next it compiles the .tiff files into a prores4444 .mov using FFMBC and cleans up the .dng and .tiff files. Just drag and drop the .raw files on the .bat and it does the rest. The only issue is bat files are just as bad as single threaded and lots of HD access. Performance on a 2.4Ghz sandy bridge i7 laptop. 6 min/16 Gb of 720p footage converted to prores4444 in 1 hr 10 min with average ~15% cpu usage.

[spoiler]
@echo off
set n=0
:again
if "%~1" == "" goto done
raw2dng.exe "%~1"
mkdir ^(%n%^)
move *.dng ^(%n%^)
cd ^(%n%^)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
..\ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 ^(%n%^).mov
del *.tiff
del *.dng
cd..
set /a n+=1
shift
goto again
:done
pause
exit
[/spoiler]

Executables -- these are multi-OS hacking together a batch script for another OS is certainly possible.
Dcraw multi-threaded edition http://www.heliconsoft.com/heliconsoft-products/dcraw/
-- Open source RAW command line processor that handles just about every image format on earth
FFMBC http://code.google.com/p/ffmbc/downloads/list
-- Open source FFMPEG but customized for broadcasting codecs
-- more about FFMBC http://code.google.com/p/ffmbc/

On a side note anyone with Lightworks community edition ie not compatible with prores/DNxHD will play nice with .tiff sqeuences. So if you have an absurd amount of hard drive space (5x dng size)... remove the del *.tiff line and the ..\ffmbc..... line and this .bat will generate a sequence of tiff files.

Finished playing around and decided to do something. Here is my expose on leaves, flowers, and slow focus pulls. https://www.youtube.com/watch?v=FYFTaIslaSo

Using a transcend x400 32gb which tested at 35 Mb/s and getting continuous 1280x720 with a may 13 build. All the newer builds seem to be dropping many more frames.

I found a way to get prores4444/ap4h using free tools ... without adobe software... It is quite quick. I do not have any editing software to test the video out on. Opening it in kmplayer confirms ap4h but it looks a bit over exposed. Give me a heads up if this is not right or feel free to alter it and write a working version.

Setup
1. Download AnotherGUI 1.6 and prores4444 preset from http://www.authorityfx.com/encoding-videos-in-prores-4444-on-windows/
2. Optional enjoy the youtube tutorial or read the page on using AnotherGUI
3. Download AnotherGUI 1.7 beta from http://doom10.org/index.php?topic=1943.0. This one actually works with win 7
4. Extract both versions and copy the "AnotherGUI Presets.xml" from the directory with AnotherGUI 1.6 to the directory of AnotherGUI 1.7 beta and overwrite it. This has the prores4444 preset in it unless you wish to write you own ffmbc command line code.
5. Grab ffmpeg http://ffmpeg.zeranoe.com/builds/ I used ffmpeg-20130514-git-56ba331-win64-static. AnotherGUI 1.7 beta will be looking for ffmpeg when it starts.
6. Extract ffmpeg with your choice of compression software.
7. Copy all "ffmpeg.exe", "ffplay.exe", and "ffprobe.exe" to the directory with AnotherGUI 1.7 beta
8. Grab ffmbc from http://code.google.com/p/ffmbc/ I used FFmbc-0.7rc8-win64
9. Extract ffmbc with your choice of compression software.
10. Copy the "ffmbc.exe" into the folder with AnotherGUI 1.7 beta

Process
1. Extract dng from raw using raw2dng.exe. ~ 30s / 1000 frames
2. Use ImageMagick. Open a command prompt and change directory to the where the dng files are execute "mogrify -format tiff *.dng" to convert the dng to a sequence of high bit depth tiff files. ~5min 40s / 1000 frames
3. Open AnotherGUI 1.7 beta. Select the first file in the tiff sequence. Select the prores4444 preset. Press Go to convert the sequence of tiff to a .mov. ~1min 45s / 1000 frames

Testing done on a SandyBridge i7 laptop with 1003 frames of 1280x540 raws

Visit the AnotherGUI developers preset gallery for other video formats
http://www.stuudio.ee/anothergui/presets.html