DNGextractor script-makes folders with DNG's from *.raw files (for windows)

[kgv5]

To make my live easier I asked my brother to write a script for me.

When you put it to a folder with any number of *.raw files it will extract every one of them to a separate folder named the same as raw and with added date and time of raw file creation (to differentiate them because after every cam shutdown ML starts counting from the beginning).
There is a need to have Scrax's raw2dng app in a folder with *.raw files and of course this extractor script.

Short instruction:

1. Create a folder for  *.raw files.
2. Copy all *.raw files from you camera to this folder
3. Put dngextractor script to this folder

download:
http://dfiles.eu/files/9e451vjjf

4. Put raw2dng app to this folder
5. click dngextractor1.0.bat

To save HDD space script deletes original *.raw files after extracting leaving only DNGs in separate folders.

Enjoy and please report any problems.

[Stedda]

Very cool, thanks

[RenatoPhoto]

Sounds good, Thanks.

Can any of these extractors write to the DNG files and correct the WB and Tint to more neutral states?

[kgv5]

Sounds good, Thanks.

Can any of these extractors write to the DNG files and correct the WB and Tint to more neutral states?

I don't know, I just asked my bro to help me a little bit with all the clicking/making folders and he made this script. It's very simple, took him just a half an hour to make this.

[mjneubrander]

I concocted a a batch file as well to do roughly the same thing. You need ffmbc and dcraw in the folder with the script for it to work. Don't have a clue how to set metadata on the dng to fix WB. The script converts the dng files to 16 bit tiff files and applies some color correction via  dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng. The color correction is the -r 2 1 1.75 1 portion, pick a single dng and fiddle with the numbers for perfection. Next it compiles the .tiff files into a prores4444 .mov using FFMBC and cleans up the .dng and .tiff files. Just drag and drop the .raw files on the .bat and it does the rest. The only issue is bat files are just as bad as single threaded and lots of HD access. Performance on a 2.4Ghz sandy bridge i7 laptop. 6 min/16 Gb of 720p footage converted to prores4444 in 1 hr 10 min with average ~15% cpu usage.

[spoiler]
@echo off
set n=0
:again
if "%~1" == "" goto done
raw2dng.exe "%~1"
mkdir ^(%n%^)
move *.dng ^(%n%^)
cd ^(%n%^)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
..\ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 ^(%n%^).mov
del *.tiff
del *.dng
cd..
set /a n+=1
shift
goto again
:done
pause
exit
[/spoiler]

Executables -- these are multi-OS hacking together a batch script for another OS is certainly possible.
Dcraw multi-threaded edition http://www.heliconsoft.com/heliconsoft-products/dcraw/
 -- Open source RAW command line processor that handles just about every image format on earth
FFMBC http://code.google.com/p/ffmbc/downloads/list
 -- Open source FFMPEG but customized for broadcasting codecs
 -- more about FFMBC http://code.google.com/p/ffmbc/

On a side note anyone with Lightworks community edition ie not compatible with prores/DNxHD will play nice with .tiff sqeuences. So if you have an absurd amount of hard drive space (5x dng size)... remove the del *.tiff line and the ..\ffmbc..... line and this .bat will generate a sequence of tiff files.

[kgv5]

mjneubrander could you pls add links for all the resources needed?

[kgv5]

In AE rendering of ~1 minute of 1504x640 raw upscaled to 1920x1080p DNxHD 444 10 bit took me about 15 minutes. 6 minute would take more than an hour I believe so it's not so different... desktop, fast i7, 16GB ram

[kgv5]

mjneubrander  - I wanted to try your script, I downloaded the stuff from the links above but it doesn't work, I mean I got many errors. Could you please explain, how exactly files and folders should be organized to make it work?

Correct me if I am wrong:

1. Make a folder named say"raw"
2. put MCRAW folder to "raw" folder or
    put files from MCRAW folder to "raw" folder
3 put FFmbc-0.7rc8-win64 folder to "raw"folder or
   put files from FFmbc-0.7rc8-win64 folder to "raw" folder
4.Put raw2dng app to "raw" folder
5. put *.raw files to "raw" folder
6. select all *.raw files and drag and drop them onto your script.bat

is that right? But It doesn't work, dng's are created but there are errors with mcraw.

[Andy600]

Huge time saver. Thankyou! 

[ted ramasola]

I tested with 2   *.raw files however it extracted them in the folder and did not create separate folders.

[kgv5]

About which script are you talking about? kgv5 or mjneubrander?
Script which I posted creates separate folders for every file.

[ted ramasola]

About which script are you talking about? kgv5 or mjneubrander?
Script which I posted creates separate folders for every file.

The one you posted.

I created a folder,

Copied 2 *.raw files into this folder,

Copied raw2dng.exe and your bat file.

Clicked your bat file.

ONLY THE FIRST FILE extracted in this folder

2ND *.RAW FILE DID NOT EXTRACT

BOTH *.RAW files DELETED.

raw2dng.exe also deleted.

[kgv5]

hmmm, i don't know, I checked this on windows XP and windows 7, both were ok. I checked with 3 *.raw files and script generated 3 folders.

EDIT: one thing which comes to my mind... did you change af file name by hand? There can be a problem if name has space in it. Original raw files from camera doesn't have space in the name.

[ted ramasola]

hmmm, i don't know, I checked this on windows XP and windows 7, both were ok. I checked with 3 *.raw files and script generated 3 folders.

the 2 raw files I used were from different dates.

my system is windows 7  64bit

[kgv5]

There can be a problem if name has space in it. Please check this out, original raw files from camera doesn't have space in the name.

[ted ramasola]

There can be a problem if name has space in it. Please check this out, original raw files from camera doesn't have space in the name.

no, I did not rename the *.raw files.

[Andy600]

A word of warning. Make sure the raw file has data in it. Sometimes I get files of 1kb and if you don't remove these before running the batch convert it goes into an endless loop. I went to make a cup of tea and came back to one folder of 13gb empty DNGs 

Other than that, this is a great workflow! 

[mjneubrander]

Thank you kgv5 for helping me try this out on another machine. I apologize for it not working I got a good nights sleep and checked it again. I found and fixed the issue. The code to call dcraw.exe was not looking for the program in the right directory. I changed the bat file to look for dcraw in the correct directory...  dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng -> ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 file.dng.

Here is the new code
[spoiler]
@echo off
set n=0
:again
if "%~1" == "" goto done
raw2dng.exe "%~1"
mkdir ^(%n%^)
move *.dng ^(%n%^)
cd ^(%n%^)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
..\ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 ^(%n%^).mov
del *.tiff
del *.dng
cd..
set /a n+=1
shift
goto again
:done
pause
exit
[/spoiler]

The files needed before dragging and dropping on the bat file
[spoiler]
C:\my pics>dir
 Volume in drive C has no label.
 Volume Serial Number is 24BC-5917

 Directory of C:\my pics

05/20/2013  05:18 PM    <DIR>          .
05/20/2013  05:18 PM    <DIR>          ..
08/10/2012  12:33 PM           354,304 dcraw.exe
05/20/2013  05:19 PM               402 raw to prores.bat
03/20/2013  10:24 AM        14,316,544 ffmbc.exe
05/19/2013  05:14 AM       195,148,992 M0000017.RAW
05/20/2013  05:18 PM       127,411,392 M0000028.RAW
05/19/2013  05:55 PM       206,438,592 M0000031.RAW
05/16/2013  04:28 PM           243,938 raw2dng.exe
               7 File(s)    543,914,161 bytes
               2 Dir(s)  162,463,543,296 bytes free

C:\my pics>
[/spoiler]

What it should look like afterwards. Directories (0) (1) and (2) will have the videos in them
[spoiler]
C:\my pics>dir
 Volume in drive C has no label.
 Volume Serial Number is 24BC-5917

 Directory of C:\my pics

05/20/2013  05:21 PM    <DIR>          .
05/20/2013  05:21 PM    <DIR>          ..
05/20/2013  05:20 PM    <DIR>          (0)
05/20/2013  05:21 PM    <DIR>          (1)
05/20/2013  05:23 PM    <DIR>          (2)
08/10/2012  12:33 PM           354,304 dcraw.exe
05/20/2013  05:19 PM               402 raw to prores.bat
03/20/2013  10:24 AM        14,316,544 ffmbc.exe
05/19/2013  05:14 AM       195,148,992 M0000017.RAW
05/19/2013  05:53 PM       127,411,392 M0000028.RAW
05/19/2013  05:55 PM       206,438,592 M0000031.RAW
05/16/2013  04:28 PM           243,938 raw2dng.exe
               7 File(s)    543,914,164 bytes
               5 Dir(s)  161,767,456,768 bytes free

C:\my pics>
[/spoiler]

[Felixlgr]

The one you posted.

I created a folder,

Copied 2 *.raw files into this folder,

Copied raw2dng.exe and your bat file.

Clicked your bat file.

ONLY THE FIRST FILE extracted in this folder

2ND *.RAW FILE DID NOT EXTRACT

BOTH *.RAW files DELETED.

raw2dng.exe also deleted.

i have the exact same problem here ( running Windows8 64bit on OSX via parallel )

[Mayo]

mjneubrander your script is great.
I changed the ^(%n%^) for directories/output files to %~n1
I find it more practical to figure out which directory/mov corresponds to which RAW.
Great stuff

[mjneubrander]

Mayo thanks for the idea I have changes the script I use! If anyone has AE I would love to see an actual time comparison between the methods.

I was messing around with ImageMagick to do the .tiff conversion and it comes with a command line utility called mogrify which works nicely as well. I cannot tell a speed difference. It does an auto brightness and an okay job of it but does not have color correction. The video comes out a beautiful shade of red. But it has a convenient resizing option if you want to upscale your images from 720 before importing into to your NLE.

Install ImageMagick http://www.imagemagick.org/script/binary-releases.php#windows
ImageMagick will install mogrify and link paths to it. Mogrify does not need to be moved to the same folder as the .bat

In the batch file replace
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do ..\dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
with
mogrify -format tiff -resize 150%% *.dng

You can change 150% 720 -> 1080 for any scaling ie 66% for 1080 -> 720

[mjneubrander]

Ok so it was a really dense idea to try to manually do white balance with dcraw from the command line. The prores is never close enough to use without extensive grading. I have altered my script to use auto wb. Works like a charm.

from this (-r 2 1 1.75 1)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f

to this (-a)
for /f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -a -H 2 -T -b 3.5 -4 %%f

The drawback is it is a frame by frame thing so wb can shift throughout the clip. 

[mjneubrander]

Just an example of the auto white balance at work and then the color shift at the end.

Enjoy.

[vipmask]

You can play videos on your computer? Founded on something but it opens nepravelno RAW file. www.offminor.de/changelog.html
It is necessary to quickly see that shot. Even in black and white. 

[mjneubrander]

Downloaded the program and it looks quite useful. What settings are you using?

Are you opening the .raw file or the sequence of .dng files?