Maxotics - Windows RAW Processing Scripts

Started by maxotics, August 26, 2013, 03:05:54 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

maxotics

Although I love RAWanizer, it sometimes stops working for me and I can't figure out why.  So I created a script that will do what RAWanizer does, and which I (or you) could customize.  Benefits of this script is that I can control which version of dcraw I use, or could put in any command line exe (maybe ufraw, etc).  It is written in VBscript (Visual Basic) and runs on Windows.  I chose VBscript because I already have some VBscript imagemagick and  ffmpeg processing scripts.  If I had the time, and hope to in the future, I'd write this in Python. 

PLEASE, DO NOT THINK OF USING THIS SCRIPT UNLESS YOU HAVE PROGRAMMING EXPERIENCE OF SOME KIND!  Sorry, this script needs software-development type configuration.  If in trouble, you may need to go into task-manager and kill a process or two.

Scary warning boxes will come up that you'll need to understand.

What this script does
o. After copying the script file into folder with RAW files, will...
1. Extracts DNGs from ML video RAW file
2. Create TIFFs from DNGs
3. Create highest quality MP4 from TIFFs, sharpening added

Eventually I will put this script, and more info up on my site.  I'm starting a thread here, however, because if you're like me, you may need this NOW ;)

If you make improvements, please send them to me at [email protected]


' Created by Max Rottersman / [email protected]
' Date: 8/30/2013 v 1.1
' USE AT OWN RISK!  Feel free to copy
' Yes, I could make more elegant, but perfect enemy of good
' or what I can do in free time

' PURPOSE
' 1. Extract DNG files out of Magic Lantern video RAW files
' 2. Process DNG files into video-editor readable TIFFs
' 3. Process TIFFs into VIDEO through ffmpeg

' ****************** CONFIG ME *******************************
' >>> WHAT TO PROCESS
' CHANGE TO PROCESS ONLY 1 RAW FILE, or extract only a few DNGS
NumberOfRAWFilesToProcess = 4 ' make, say 3, for testing, or 1000000 for all
NumberOfDNGsToProcess = 1000000 ' or 10000000 for all

flagRunRAWtoDNGProcess = True
flagRunDNGtoTIFFProcess = True
flagRunTIFFStoFFMPEGProcess = True
' *************************************************************

'
' What folder is this script in, from there I start
Dim Prefix_dcraw, Suffix_dcraw
ThisFolder = left(WScript.ScriptFullName,(Len(WScript.ScriptFullName))-(len(WScript.ScriptName)))
' FILE NAMES AND FOLDER LOCATIONS
FolderRAWs = ThisFolder ' Script should be put with RAW files
FolderDNGs = FolderRAWs & "\DNGs"   ' DNGs put sub-folder here, you can change
FolderTIFFs = FolderDNGs & "\TIFFs"  ' assume TIFFs sub-folder off DNGs

' **** CONFIG dcRAW Params! *********************************
' MUST MUST MUST HAVE!
' Make this any location you want.  MUST HAVE!
MyRootFolderForRAW = "C:\Files2013_VidPhotoSoft\RAWCinema"
' WARNING: If these files aren't here, script will NOT work
raw2dng_exe = MyRootFolderForRAW & "\RAWtoDNG\raw2dng.exe"
dcraw_exe = MyRootFolderForRAW & "\DCRawLatest\dcraw.exe"
ffmpeg_exe = MyRootFolderForRAW & "\ffmpeg_latest\ffmpeg.exe"

' ARGS: raw2dng
Suffix_raw2dng = "" ' nothing needed currently
' ARGS: dcraw
Suffix_dcraw = "-w -6 -T" ' CHANGE HERE TO SUITE YOUR NEEDS
' ARGS: ffmpeg
Suffix_ffmpeg = " -i <FILEPATH>\000%3d.tiff -vcodec libx264 -r 24 -pix_fmt yuv420p -q:v 1 -b:v 20M -vf unsharp=5:5:1.5:5:5:1.5 "


'*************************************************************

ReDim xFilesRAW(0)
' Get names of RAW files in this folder that the VBS file has
' been COPIED TO!
rtn = GetFilesRAW(ThisFolder, NumberOfRAWFilesToProcess)

Set objFso = CreateObject("Scripting.FileSystemObject")

' Cycle through RAWs, Create target folder for DNGs and TIFFS
for i = 0 to ubound(xFilesRAW) -1
' STEP 1
' Come up with folder name for DNGs like RAWanizer
sFileNameRAW = xFilesRAW(i+1)
'msgbox sFileNameRAW
sFolderNameDNGs = replace(sFileNameRAW,".RAW","") & "-dng"
'msgbox sFolderNameDNGs
sFolderNameDNGs_FullPath = ThisFolder & sFolderNameDNGs ' \
'msgbox sFolderNameDNGs_FullPath
sFolderNameDNGS_FullPath_RAWFileName = sFolderNameDNGs_FullPath & "\" & sFileNameRaw
' TIFF folders
' Not calling proxy TIFFs like RAWanizer because assume
' these will be final
sFolderNameTIFFs_FullPath = sFolderNameDNGs_FullPath & "\TIFFs"

' Create DNGs folder!
If Not objFSO.FolderExists(sFolderNameDNGs_FullPath) Then
objFSO.CreateFolder(sFolderNameDNGs_FullPath)
End If

' Create TIFFs folder!
If Not objFSO.FolderExists(sFolderNameTIFFs_FullPath) Then
objFSO.CreateFolder(sFolderNameTIFFs_FullPath)
End If

' ***
' RAW BLOCK RUN?
' ***
if flagRunRAWtoDNGProcess = True then

' STEP 2
'Copy RAW file int DNG folder, we will delete copy later
objFso.CopyFile ThisFolder & "\" & sFileNameRAW, _
sFolderNameDNGs_FullPath & "\" & sFileNameRAW

' STEP 3
' Copy raw2dng to folder
' Copy RAW2DNG.exe to folder
objFso.CopyFile raw2dng_exe, sFolderNameDNGs_FullPath & "\raw2dng.exe"

' Step 4
'Run RAW2DNG (and wait for process to end)
Set objShell = WScript.CreateObject( "WScript.Shell" )
' Change Shell path to current DNG folder
objShell.CurrentDirectory = sFolderNameDNGs_FullPath
objShell.Run "raw2dng.exe " & sFileNameRAW, 1, True 'chr(34) double quotes if needed
Set objShell = Nothing

' Step 5 Clean up, remove RAW and raw2dng.exe
' Check if file exists to prevent error
If objFSO.FileExists(sFolderNameDNGS_FullPath_RAWFileName) Then
objFSO.DeleteFile sFolderNameDNGS_FullPath_RAWFileName
End If

' Step 6 Clean up, remove raw2dng.exe
' Check if file exists to prevent error
If objFSO.FileExists(sFolderNameDNGs_FullPath & "\raw2dng.exe") Then
objFSO.DeleteFile sFolderNameDNGs_FullPath & "\raw2dng.exe"
End If

' ************************************
' DNGS SHOULD HAVE NOW BEEN EXTRACTED
' ************************************

' Step 1 Copy DNGs for TIFFs folder for processing
' Create another shell, killed one above for safety
' use xcopy for speed
Set objShell = WScript.CreateObject( "WScript.Shell" )
sRun = "%comspec% /c xcopy "
sRun = sRun & chr(34) & sFolderNameDNGs_FullPath & "\*.dng" & chr(34)
sRun = sRun & " " ' space before target
sRun = sRun & chr(34) & sFolderNameTIFFs_FullPath & "\" & chr(34)
objShell.Run sRun ,1,True
Set objShell = Nothing

End if ' RAW BLOCK RUN?

' ***
' DNGs to TIFFs
' ***
If flagRunDNGtoTIFFProcess = True then

' Step 2 COPY dcraw
objFso.CopyFile dcraw_exe, sFolderNameTIFFs_FullPath & "\dcraw.exe"

' Step 3 run dcraw
' going to use create BAT file and run method so can use
' bat file for testing/configuring dcraw if necessary
ReDim xFilesDNG(0)
rtn = GetFilesDNG(sFolderNameTIFFs_FullPath, NumberOfDNGsToProcess)
' Write files to bat file
Set objBATFile = objFso.CreateTextFile(sFolderNameTIFFs_FullPath & "\convertDNG.bat")
For j = 1 to ubound(xFilesDNG) -1
objBATFile.WriteLine "dcraw " & Suffix_dcraw & " " & xFilesDNG(j)
Next
objBATFile.close()

' Okay, now run bat file
Set objShell = WScript.CreateObject( "WScript.Shell" )
objShell.CurrentDirectory = sFolderNameTIFFs_FullPath
sRun = sFolderNameTIFFs_FullPath & "\convertDNG.bat"
'msgbox sRun
objShell.Run sRun, 1, True
Set objShell = Nothing

' Finally, clean up by deleting DNGs in TIFF folder
' and the dcraw.exe file
Set objShell = WScript.CreateObject( "WScript.Shell" )
objShell.CurrentDirectory = sFolderNameTIFFs_FullPath
sRun = "%comspec% /c del "
sRun = sRun & chr(34) & sFolderNameTIFFs_FullPath & "\*.dng" & chr(34)
objShell.Run sRun ,1,True
Set objShell = Nothing

If objFSO.FileExists(sFolderNameTIFFs_FullPath & "\dcraw.exe") Then
objFSO.DeleteFile sFolderNameTIFFs_FullPath & "\dcraw.exe"
End If

' If you want to delete BAT file, for now I'll leave it
if false then
If objFSO.FileExists(sFolderNameTIFFs_FullPath & "\convertDNG.bat") Then
objFSO.DeleteFile sFolderNameTIFFs_FullPath & "\convertDNG.bat"
End If
end if


' run bat file
end if ' TIFF BLOCK RUN?

' ***
' run ffmpeg
' ***
If flagRunTIFFStoFFMPEGProcess = True then

sRun = ffmpeg_exe & replace(Suffix_ffmpeg, "<FILEPATH>",sFolderNameTIFFs_FullPath)
'sFolderNameTIFFs_FullPath
sOutputFile = replace(sFileNameRAW,".RAW",".mp4")
sOutputFileWithPath = ThisFolder & sOutputFile

sRun = sRun & " " & sOutputFileWithPath
msgbox sRun

' Finally, clean up by deleting DNGs in TIFF folder
' and the dcraw.exe file
Set objShell = WScript.CreateObject( "WScript.Shell" )
'objShell.CurrentDirectory = sFolderNameTIFFs_FullPath
'sRun = "%comspec% /c del "
'sRun = sRun & chr(34) & sFolderNameTIFFs_FullPath & "\*.dng" & chr(34)
objShell.Run sRun ,1,True
Set objShell = Nothing

'WScript.quit
' ThisFolder & "\" &

end if


next

' >>>>>>>>>>
' FUNCTIONS
' >>>>>>>>>>

'**************************************
' GetFilesRAW: Load RAW file names into array
'**************************************
Function GetFilesRAW(argDir, argNumberRAWFilesToProcess)
Dim fso, folder, files, NewsFile,sFolder
Set fso = CreateObject("Scripting.FileSystemObject")
Set folder = fso.GetFolder(argDir) 
Set files = folder.Files   
vCounter = 0
For each folderIdx In files
'msgbox folderIdx.Name
if right(ucase(folderIdx.Name),4) = ".RAW" Then
Redim Preserve xFilesRAW(Ubound(xFilesRAW)+1)
xFilesRAW(Ubound(xFilesRAW)) = folderIdx.Name
vCounter = vCounter + 1

' For resting, only run a few
if vCounter > argNumberRAWFilesToProcess then
exit function
end if

' *** Write file name, testing
'NewFile.WriteLine(folderIdx.Name)
End if
Next
End Function

'**************************************
' GetFilesDNG: Load DNG file names into array
'**************************************
Function GetFilesDNG(argDir, argNumberDNGFilesToProcess)
Dim fso, folder, files, NewsFile,sFolder
Set fso = CreateObject("Scripting.FileSystemObject")
Set folder = fso.GetFolder(argDir) 
Set files = folder.Files   
vCounter = 0
For each folderIdx In files
'msgbox folderIdx.Name
if right(ucase(folderIdx.Name),4) = ".DNG" Then
Redim Preserve xFilesDNG(Ubound(xFilesDNG)+1)
xFilesDNG(Ubound(xFilesDNG)) = folderIdx.Name
vCounter = vCounter + 1

' For resting, only run a few
if vCounter > argNumberDNGFilesToProcess then
exit function
end if

' *** Write file name, testing
'NewFile.WriteLine(folderIdx.Name)
End if
Next
End Function

'*********************************
' WriteBATFile
'*********************************
Function WriteBATFile(argDir, argFileNameAndPath)
Set WshShell = CreateObject("WScript.Shell")
Set objFso = CreateObject("Scripting.FileSystemObject")
Set objFile = objFso.CreateTextFile(argFileNameAndPath)
objFile.Close
End Function

'
' MISC NOTES
' About VBS batch job security issues
':: Tutorial: http://www.sevenforums.com/tutorials/182353-open-file-security-warning-enable-disable.html
' DISABLE SECURITY
'REG ADD "HKCU\Environment" /V SEE_MASK_NOZONECHECKS /T REG_SZ /D 1 /F
' ENABLE SECURITY
'REG Delete "HKCU\Environment" /V SEE_MASK_NOZONECHECKS /F




I'm think I'm getting TIFF to video conversions that are very close to TIFF quality.  Any insight (improvement to below), please  let me know!  These are the commands I'm using currently


*** DNxHD ***
ffmbc -r 24 -i 000%
3d.tiff -vcodec dnxhd -b 185M -r 24 -vf scale=1920:1080 outputDNxHD.mov

*** Proress ***
ffmbc -r 24 -i 000%
3d.tiff -vcodec prores -profile hq -r 24 -vf scale=1920:1080 outputProre
s444.mov

*** MP4 ***
ffmpeg -i 000%3d.tiff -vcodec libx264 -r 24 -pix_fmt yuv420p -qscale 2 -vf unsharp=5:5:1.5:5:5:1.5 output.mp4