RAWanizer - windows batch generation of video files from RAW

Started by marten, May 21, 2013, 03:32:46 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

marten

To make a better workflow for myself I created this windows app. Thought I should share it in case anyone else find it useful.

It's just i quick dev. No extensive testing done. No guarantee of anything. Use at your own risk! Only works on windows and is using raw2dng, dcraw and ffmpeg.

My workflow
As of version 0.3.0 I have tried to get this app to help me with my whole workflow which centers around Lightroom as RAW converter. Here is how I do it:
-Copy raw files to folder on local computer (SSD disk will improve conversion speed alot)
-Open RAWanizer, select folder, choose video settings and press start. This will extract dng from raw file, create proxy tiff files and a proxy video file.
-For a fast preview of the sequence, I choose tiff size half and mp4 option. This will give me a preview that I can use to decide which files to do real conversions for.
-Once I know which files I want to do deal with I have two options:

---Go to Lightroom or other raw converter and do conversion there. Export files as tiff and put them in 'tiff' sub folder to the dng folder (in Lightroom there is a nice option to do this easy)
---Go back to RAWanizer and choose 'video' instead of 'proxy video', select a higher quality video codec and hit start. This will give me a video file to work in my NLE.
---If I want to go back and change some settings in Lightroom I can do that and reexport to the same folder as before. Then open RAW anizer and just hit start again and it will replace the previously created video file. This will automatically have the clip in your NLE  get updated.

---OR

---Instead of lightroom, in RAWanizer I hit custom button which brings up a dialog for making settings for dcraw conversion. Tweak the settings and hit save. This is now the settings use for conversions. If I'm satisfied with the quality I choose to use a high quality codec and hit start. Now I have a file ready for my NLE

Important The 'video' settings to the left of start button has these meanings:
-Proxy video: Extracts the dng from raw file. Then creates tiff files from dng and finally creates a video file from the tiffs.
-Video: Looks for tiff files in 'tiff' sub folder in the dng subfolder and creates video file from them. Use this option if you have developed the dng in other converter.
-No video: Only extracts the dng from raw file. No tiffs created and no video file created.


Open the folder with raw files. Choose frame rate (for example 23.976) and size for proxy video. Click start and wait.

RAWanizer will create a sub folder for each raw file and put the dng files there. When proxy is created a sub folder in the dng folder will be created with tiffs. The final proxy video is created from the tiffs and put in the same folder as the original raw file. If there is a wav file with the same name as the raw file it will be included in the proxy video.

The proxy file generated can be either mp4 (5mbit) or DNxHD (185mbit). The mp4 file is created in the resolution of the tiff files.  Since DNxHD only supports two resolutions, the image is upscaled to 1920x1080 with black bars if not a perfect fit.

If people find it useful I might put some more time on it. What improvements would you like?

Download full install 0.6.0
Patch 0.6.1 Save .exe in previous installation folder.
If you have installed a previous version, when using full install, you have to use windows 'add and remove progam' to uninstall the previous version first. You may also be prompted to download and install .net 4.5 when you start the program for the first time.


version 0.6.1
-dng subfolder is named -dng instead of .dng so Resolve can recognize the files.

version 0.6.0
-fixed processing of multiple spanned files
-added option to save all dng files in same folder
-included the latest raw2dng

version 0.5.5
-added setting to delete dng files after video creation. For safety reasons this only works if original is a RAW file, not a Dng folder.
-added button to restore default presets
-some file handling bugfixes

version 0.5.4
-the settings are applied when processing starts and not when added to queue, this has confused some people so I hope this will help some, but now you can't get different settings for different files in the queue.

version 0.5.3
-bug fixes

version 0.5.2
-bug fixes
-use framrate from raw file when creating video

version 0.5.1
-automatic merging of split files (>4Gb) (there is a setting under file tab where you can choose to write to the original RAW file. This is faster, cleaner and the result is just one single RAW file. But if something fails along the way you probably end up with at corrupt RAW file, so use with caution, preferably a backup. If not checked it will create a new RAW file with prefix 'M-' and the originals will still be there, safer if you have no backup)
-some more options for dcraw conversions
-now possible to save more than one custom preset for dcraw, Cineform and video output.

version 0.5.0
-raw file meta data shown in file list.
-option to show first, middle and last thumbnail from the raw file before processing.
-clean up of UI.
-latest version of raw2dng included.

version 0.4.3
-New install with all needed files and some fixes. Hopefully the ffmpeg problem will be gone now.

version 0.4.2
-Fixed problem with path for Cineform tools. This folder could appear either under 'Cineform' or 'Gopro'. You can now also set custom path under 'folder settings' if you have it installed in a different folder.

version 0.4.1
-For the Cineform batch processing I'm now throwing up console window so the progress can be examined and aborted by closing the console window. This is temporary as I hope to find another way to deal with it.

version 0.4.0
-Batch processing for Gopro Cineform. You have to download and install Gopro Cineform studio + a small utility. Follow the instructions here: http://www.magiclantern.fm/forum/index.php?topic=5479.msg41378#msg41378
-For the Cineform batch processing, the top progressbar is not working (detailed progress for each clip). You have to wait and see...

version 0.3.3
-Possible to open a single dng folder
-Fixed bug when starting program and last folder used had been removed
-Increased responsiveness when previewing dcraw settings

version 0.3.2
-tried to get rid of some users program crash

version 0.3.1
-Added possibility to set custom folders to override default structure
-Initial attempt for 'watch folder'. When this is activated, new raw files copied to that folder will automatically start processing.

version 0.3.0
-Add date time prefix to raw file name
-UI much more responsive and cancel works almost all the time
-More options for managing files and processing
-Now possible to use custom dcraw converter settings
-UI to help create custom dcraw settings
-Use custom ffmpeg settings for video file
-Create video file from tiff sub folder that can be populated by your favorite raw converter (lightroom, ps)

version 0.2.1
-added DNxHD as an option for the proxy file

version 0.2.0
-using multiple instances of dcraw to increase speed of dng to tiff conversion. You can change no of instances to find optimal for your computer.


Known Issues:
- Does not work with file names containing spaces or other special characters

Stedda

5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

Jakobmen

wow thank you ) can we have it make DNxHD files =) ?
Canon 5D3

hjfilmspeed

Thank you soooo much!!!! Havent tried it yet but sounds awesome

Andy600

Great work! it half works for me (which is still ok). I get as far as TIFF conversion but it doesn't create a proxy (Win7 64bit)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com


marten

Quote from: Andy600 on May 21, 2013, 06:08:20 PM
Great work! it half works for me (which is still ok). I get as far as TIFF conversion but it doesn't create a proxy (Win7 64bit)

Hmm. I run it on win7 64-bit myself. So can you see the created TIFF files in the tiff sub folder in the dng sub folder?

marten

Quote from: Andy600 on May 21, 2013, 06:08:20 PM
Great work! it half works for me (which is still ok). I get as far as TIFF conversion but it doesn't create a proxy (Win7 64bit)

Did you rename the RAW file to something other then M000000x.RAW ? Looked at my code and could see this would break it. Sorry. Will try to fix it soon.

Andy600

@marten - Yes, I can see the TIFFs. Everything seems to work apart from making the proxy. I haven't altered the raw name. BTW, are these uncompressed TIFFs, if yes, this is probably the fastest way to get files into Davinci Resolve  :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

marten

Quote from: Andy600 on May 21, 2013, 06:50:47 PM
@marten - Yes, I can see the TIFFs. Everything seems to work apart from making the proxy. I haven't altered the raw name. BTW, are these uncompressed TIFFs, if yes, this is probably the fastest way to get files into Davinci Resolve  :)

Yes, they are uncompressed tiffs. BUT, the dng converter is not making good quality conversions. Haven't tested with so many different settings yet, but I would go with a real raw converter for the final footage. I see these more as proxies for viewing and cutting not to do grading from.

To bad it wasn't the naming of the file. Will see if I can find something else.

Andy600

An error did come up when I first ran RAWanize but hasn't since. I'll reboot to see if I can cause it to happen again.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

mjneubrander

I worked on a batch file. Similar thing but outputs prores4444. You could make a prores4444 or DNxHD file from your tiffs as an option aside from mp4 using ffmbc. It would make a great check-box option for those who feel like editing prores. I also found encoding a prores4444 file was not significantly more intensive than mp4.

I call ffmbc for prores4444 it using
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 movie.mov
I have not tried DNxHD yet.

Are you using color correction in your dcraw call? If so what gain and brightness are you using?

Are you using the multithreaded dcraw? http://www.heliconsoft.com/heliconsoft-products/dcraw/

Does your program issue multiple decoding processes (raw2dng -> dcraw -> ffmpeg) at simultaneously, or does it process them sequentially? My batch file is limited to sequential and I never got the hang of programming :(.

FFMBC main site.
http://code.google.com/p/ffmbc/

The two key commands from the batch file I worked on
[spoiler]
/f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 %~n1.mov
[/spoiler]
or see the thread
http://www.magiclantern.fm/forum/index.php?topic=5528.0

marten

Quote from: mjneubrander on May 21, 2013, 07:30:49 PM
I worked on a batch file. Similar thing but outputs prores4444. You could make a prores4444 or DNxHD file from your tiffs as an option aside from mp4 using ffmbc. It would make a great check-box option for those who feel like editing prores. I also found encoding a prores4444 file was not significantly more intensive than mp4.

I call ffmbc for prores4444 it using
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 movie.mov
I have not tried DNxHD yet.

Are you using color correction in your dcraw call? If so what gain and brightness are you using?

Are you using the multithreaded dcraw? http://www.heliconsoft.com/heliconsoft-products/dcraw/

Does your program issue multiple decoding processes (raw2dng -> dcraw -> ffmpeg) at simultaneously, or does it process them sequentially? My batch file is limited to sequential and I never got the hang of programming :(.

FFMBC main site.
http://code.google.com/p/ffmbc/

The two key commands from the batch file I worked on
[spoiler]
/f "usebackq delims=|" %%f in (`dir /b "%CD%\*.dng"`) do dcraw -v -r 2 1 1.75 1 -H 2 -T -b 3.5 -4 %%f
ffmbc -threads 16 -r 23.976 -i 0%%05d.tiff -vcodec prores -pix_fmt yuv444p10 %~n1.mov
[/spoiler]
or see the thread
http://www.magiclantern.fm/forum/index.php?topic=5528.0

It was my plan to expand the available codecs once I got the conversion process stable. So they will be coming.

No, I'm not using multithreaded dcraw. I tried it, but it didn't give me much better results. However, I'm currently working on another solution to speed it up. Coming soon I hope.

Thanks for your tips and feed back. Hopefully most of them will be done soon. :)

marten

New version uploaded.

Uses multiple instances of dcraw to increase speed. On my machine the total time improved approx 400%  8)

The downside of using multiple instances is the the progress bar will not report updates during this conversion :( But for me that is an easy choice.

marten

Quote from: Andy600 on May 21, 2013, 07:23:41 PM
An error did come up when I first ran RAWanize but hasn't since. I'll reboot to see if I can cause it to happen again.

Can you try the new version and see if you still have the same problem. I have done some small fixes.

Andy600

Quote from: marten on May 21, 2013, 10:24:27 PM
Can you try the new version and see if you still have the same problem. I have done some small fixes.

Yes but the link isn't working for me  ???

Update - The link works but you have to copy/paste it
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Andy600

@marten - It's working great now!! 8) Many thanks for this excellent app
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

marten

Added DNxHD as an option for the proxy file.

Download the patch from the original post.

marten


tferradans

awesome work, @marten! thanks a lot!

I don't mean to be stupid, but is this thing working with 4GB+ clips already? :P

marten

Quote from: tferradans on May 21, 2013, 11:51:04 PM
awesome work, @marten! thanks a lot!

I don't mean to be stupid, but is this thing working with 4GB+ clips already? :P

On some early test versions I tried it on a 5GB file and it worked fine. Haven't tested it on the latest versions though, but hope it's still working.

Jakobmen

Canon 5D3

1%

You can update raw2dng binary in program files if necessary.

JulianH

This is great! Seems to be the easiest way to preview the shots right now. Thanks for the app!

marten

New version with quite a lot of new features.

Read the first post for download links.

The conversion from dng to tiff is hitting my disk quite hard. I had massive improvement when using an SSD. Don't know why dcraw is so hard on the disk, shouldn't be that much to move around.