OSX Canon Video Import Script

Started by Murphy, May 28, 2013, 07:57:49 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Murphy

Hey all. Just wrote a Mac OSX bash shell script, for importing video off CF cards from the canon.

http://www.basetheory.com.au/osx-canon-video-import-script/

I thought it was getting a bit silly, with the amount of messing around required to get all the different footage formats off one single camera (h264, h264 slowmo, cr2 timelapse, 14bit raw video) so i decided to write a script that would sort and convert it all for me at once, regardless of what combination of footage I'd shot that day.

Its a shell script which is daunting for some (and could probably stand to have a simple gui makeover) but it seems to do the job pretty well once the environmental variables are set.

Basically just plug in your CF card. Run the script. Go make yourself a coffee and wait.

It will:
1) create a folder in your scratch location based on the date of ingest.
2) convert 14bit raw video to DNG sequences
3) create mjpeg proxy videos of those DNG sequences for editing
4) determine the framerates of the h264 video
5) copy the standard framerate files straight over, and conform the slowmotion footage.
6) create mjpeg proxy videos of the h264 footage for editing
7) convert CR2 files to a DNG sequence (treating it as a timelapse)
8) create mjpeg proxy video of the DNG timelapse sequence for editing.

Let me know what you think. If you have any suggestions for improving it, comments and criticisms are welcome.

http://www.basetheory.com.au/wp-content/uploads/2013/05/canonimport.zip

Oedipax

Awesome, thanks for making this. I'll have a go at it tomorrow.

Murphy

Thanks Oedipax. I hope you get some good use out of it.

I've made a few minor updates to the script. As well as a bit of code cleanup and organisation, I've slightly changed the folder layout. The script now separates the slow motion shots and standard video shots into separate folders.

I've also added a few lines of code to analyse and sort the CR2 files. In the previous script, all the CR2 files were basically treated as one long timelapse sequence which would be stitched together. The script now analyses the unix time stamp of each CR2 file, and if the gap between the current and previous frames is longer than the TLBREAK variable (35 seconds by default), it will treat the current frame as the start of a new timelapse, and organise the files accordingly.

Also, if there is a long timestamp gap on either side of a CR2 file, it will be treated as a standalone photo rather than part of a timelapse sequence, and will be moved to the Stills folder.

http://www.basetheory.com.au/wp-content/uploads/2013/05/canonimport-2013-5-29.zip

cineblah

Thanks Murphy, this works like a charm! The proxy-movies that it generates are especially handy, because it offers an easy way to preview the movies. I tested it, and it also works with SD-cards (in my case a Canon 6D)

For anyone who isn't experienced with the OSX Terminal (I wasn't before I tried this), here's a quick "How to install this script":

1. Install raw2dng (http://www.magiclantern.fm/forum/index.php?topic=5508.0) and Adobe DNG Converter (http://www.adobe.com/support/downloads/detail.jsp?ftpID=5568)
2. install Macports from macports.org:

Mountain Lion: https://distfiles.macports.org/MacPorts/MacPorts-2.1.3-10.8-MountainLion.pkg
Lion: https://distfiles.macports.org/MacPorts/MacPorts-2.1.3-10.7-Lion.pkg
Snow Leopard: https://distfiles.macports.org/MacPorts/MacPorts-2.1.3-10.6-SnowLeopard.pkg

3. install xcode https://itunes.apple.com/us/app/xcode/id497799835?ls=1&mt=12 (it's free)
4. open Xcode, go to preferences ->downloads -> install "command line tools"
5. Open Terminal.app (it's in your Applications folder)
6. in the terminal window type: "sudo port install ufraw" and press enter
7. when asked, enter your password (you will not be able to see what you type) and press enter
8. be patient while ufraw gets installed, it may take a while ;) Sometimes it doesn't seem to do anything for a while, but wait until it says "No broken files found".
9. in the terminal window type: "sudo port install ffmpeg" and press enter
10. again, wait till it's finished (this happens much quicker this time)
11. Ok, now all the needed software is installed on your computer, all you need to do is make sure you edit the paths in canonimport.sh to suit your situation. (see Murphy's explanation: http://www.basetheory.com.au/osx-canon-video-import-script/)
12. Once you've done that, in the terminal window type: sh /location/of/the/.sh/script and press enter

If you don't know the path to your .sh script, just drag the file onto the terminal-window and it will show you the path

For example, if it's in the downloads-folder on your mac, type:

sh /Users/Your_username/Downloads/canonimport.sh

That's it! The script will now batch-process the files on your CF (or SDcard)

Hope this helps someone!

scrax

Quote from: Murphy on May 28, 2013, 07:57:49 AM
Hey all. Just wrote a Mac OSX bash shell script, for importing video off CF cards from the canon.

http://www.basetheory.com.au/osx-canon-video-import-script/

I thought it was getting a bit silly, with the amount of messing around required to get all the different footage formats off one single camera (h264, h264 slowmo, cr2 timelapse, 14bit raw video) so i decided to write a script that would sort and convert it all for me at once, regardless of what combination of footage I'd shot that day.

Its a shell script which is daunting for some (and could probably stand to have a simple gui makeover) but it seems to do the job pretty well once the environmental variables are set.

Basically just plug in your CF card. Run the script. Go make yourself a coffee and wait.

It will:
1) create a folder in your scratch location based on the date of ingest.
2) convert 14bit raw video to DNG sequences
3) create mjpeg proxy videos of those DNG sequences for editing
4) determine the framerates of the h264 video
5) copy the standard framerate files straight over, and conform the slowmotion footage.
6) create mjpeg proxy videos of the h264 footage for editing
7) convert CR2 files to a DNG sequence (treating it as a timelapse)
8) create mjpeg proxy video of the DNG timelapse sequence for editing.

Let me know what you think. If you have any suggestions for improving it, comments and criticisms are welcome.

http://www.basetheory.com.au/wp-content/uploads/2013/05/canonimport.zip

Take a look at MLTools I suppose that lot of those things could be integrated into it
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

Oedipax

Hey Murphy, just had a chance to try out the script and overall I think it's great!

A couple requests - what about adding multicore support for the ufraw-batch process? Ufraw uses 100% of one core, but the proxy conversions could be way faster if it did multiple frames (per number of cores) rather than one at a time.

GNU Parallel looks like one way to achieve this. I'm not sure if there would be additional code needed to manage which processes were converting which frames.

Also, instead of writing all the DNGs out as 000001.dng, 000002.dng, etc, what about unique filenames derived from creation date or something similar?

Murphy

Hi guys. I've done some more work on the scrip, but some telco technicians cut my ADSL line when installing a phone into the apartment next door, so i've been without interwebs for a few days (still using my phone as a 3G modem right now) so I haven't been able to upload changes I've made.

@cineblah Thanks for the step by step. The script is kindof a pain, because its really just an OS instruction set which tells a bunch of other programs what to do, and those other programs can be a bit of work to install and set up properly.

@Scrax, I'll take a look at MLTools when i have full internet again. I'm kindof a bit pedantic about how my files are organised, and haven't really found many automatically organising apps that i'm comfortable with (one of the reasons i hate AVID is its database system) so i tend to write my own scripts to specifically fit my personal workflow wherever possible. As for integration into MLTools, i'm assuming that its cross platform? A lot of the things in this script are quite POSIX specific. I'm a Linux/OSX guy, and it just made sense to write it in bash given that its a set of instructions to run other programs, and requires a lot of OS specific tasks like mkdir and ls.

Windows natively lacks a lot of the terminal scripting functionality that POSIX systems like OSX and Linux have. You might be able to find other ways of achieving it by rewriting it to something more cross platform like python, or maybe getting it working with cygwin, but i don't think it would be easy to get this script ported to windows.

@Oedipax. I'll look at multicore support, but multicore is a fairly new thing program wise. If it was written from scratch in c++ i could use pthread or boost, but the script is in bash (which makes sense given its a relatively simple set of instructions to run other programs) and thus limited to what a POSIX terminal, and the programs it launches, are capable off.

The easiest way to multithread would probably be to create a loop that loads each DNG frame as a separate ufraw background process using &, but the problem is that if you have a sequence of 1000 frames, it could be horribly unstable to load 1000 parallel processes all at once. It would need to have a wait loop to check that it doesn't add too many processes at once. Maybe piping ps -ef into grep and counting the number of results would work. It will be a couple of weeks before i have time to try implementing that though (i'm out of town on a job next week).

For the numbering system, as far as i understand it, Magic Lantern RAW is basically still in pre-alpha stages of development. With any luck, when the raw module is finally merged into the mainline firmware, it will generate unique file names like regular MOV or CR2 files have, rather than starting at M000000 every time you format the CF card (which currently causes the script to overwrite files being put in the same scratch location).

In the meantime using the creation timestamp is a good idea. In the latest version i've actually gotten the DNG timelapses to be named based on their first and last frame, rather than a generated number.

<---------->

which brings us to the next part. The latest version of the script:

http://www.basetheory.com.au/wp-content/uploads/2013/06/canonimport-2013-06-02.zip

1) Cleaned up the code a little. Added more options at the beginning of the script for your choice of full res and proxy codecs. Also significantly cleaned up the terminal output (FFMPEG and UFRAW generate a lot of junk)

2) Changed the folder layout a little. Got rid of the MOV and Proxy folders as the Mac version of FFMPEG can apparently transcode to Apple ProRes422 which negates the need for lower res offline proxy files (at least for me anyway. If you think they should still be created, let me know), so now it just transcodes everything once, to ProRes (though you can change the FULLRES variable to your editing codec of choice).

3) Added support for HDR timelapse creation. This is dependent on the ML's bracketing though, as the script searches for the HDR .SH files generated by ML firmware in camera. If you just use standard Canon bracketing, the script will not be able to tell the difference between a timelapse and a HDR timelapse. It also requires enfuse to be installed.

4) Better algorithm for analysing timelapse and still shots. Also names timelapses based on the first and last frame in the sequence, rather than a generated ID number.

5) -h help and -v verbose options

Also, i've created a tweaked version of the script so that it works for my GoPro Hero3. Hopefully i'm going to eventually merge the scripts so that its one master script that imports and organises media from a variety of different cameras.

Thanks again guys. The feed back is greatly appreciated. Its nice to know other people find it useful, and i'm not just talking to myself on here :)