HDSLR Offloader (Windows utility to backup footage & transcode HDR videos)

Started by Malcolm Debono, June 24, 2012, 11:53:53 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Malcolm Debono

Thought I'd share with you a small utility for Windows I've written a few months ago in C# in case it's useful to you guys  :)

This is a simple light-weight app which allows you to offload footage from your memory card to your PC. I've found it quite useful as a backup for on-location shooting, and carry it on a small netbook.

Here's a break-down of the features:
- Copy files from memory card to a primary & secondary (optional) location of your choice
- Filter file types to copy (currently MOV, THM, JPG, CR2, WAV, MP3)
- Renaming video files to creation date & time (as a timecode reference)
- Verification of data after copying (by comparing exact file sizes)
- Storing up to 3 profiles (video, audio & photo profiles)
- Quick profile switching using CTRL+1/2/3 keyboard shortcuts
- Copying all Magic Lantern files, only the configuration file (magic.cfg) or the config and hidden.cfg (introduced in ML 2.3) from a pre-defined location
- Generating a shot list based on data obtained from LOG files generated by Magic Lantern, which includes the date & time, lens used, shutter speed, aperture, ISO, focal length, white balance, picture style, FPS, HDR and bitrate
- Exporting shot list to Excel spreadsheet
- Transcoding of HDR videos captured through ML
- Argument passing for running through scripts or batch files

Screenshot:


Download latest version (see posts below for changelog):
http://malcolmdebono.com/apps/HDSLRoffloader_0.6.1.zip

Let me know if you have any ideas for new features or encounter any bugs.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Schloime

Hi.

I just found your tool while strolling through the forum, can't test it now because I'm on work.
It sounds definitely useful, especially all the options to treat the files.
I already have a feature request (you asked for it): Do you know the tool "QTChange"? I use it to get real TC into the files. Is there a possibility to have that functionality in your tool?

Greetings,
  Schloime
600D, EF-S 17-85, EF-S 55-200, SNS-HDR, Photoshop7, Lightroom, Lightwave, Lightworks, Light everywhere

Malcolm Debono

Hey Schloime,

I will definitely look into it. Apparently QtChange doesn't support command-line interface, so for now I don't think I can integrate it. I will check if there are any other libraries which can do the same thing. So far the closest I found is a modified version of ffmpeg (with added broadcasting features), and according to its documentation, it can write quicktime files containing timecode tracks.

I'll see if I can work something out in the coming days  :)

Malcolm

Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

ecek2

EOS 550D 18-55mm IS - 50mm F/1.8

Malcolm Debono

MediaInfo only shows you timecode and other video properties. It cannot modify the TC track like QTchange can.

I've done some tests with FFmpeg and it can add a TC to the video file itself, but only by converting the video. What I really like in QTchange is that it saves the TC track in the MOV file without any conversions, so I'm still looking for a better solution.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

3pointedit

Wow, cool tool, very useful!

Feature pester request: Any chance of adding a transcode video HDR files (at transfer time) into seperate light/dark files? If using ffmpeg would be nice to choose between frame sequences or duped media at frame rate = half.
550D on ML-roids

GershberG

looks very usefull. TnX alot!

some questions:
is verification time\size based or CRC?
are both targets verified?
7D[ML], 60D[ML], 5Dc[ML]
8-15L, 16-35L II, 24-70L II, 70-200L, 50L f:1.2

Malcolm Debono

Quote from: 3pointedit on July 04, 2012, 02:58:05 AM
Feature pester request: Any chance of adding a transcode video HDR files (at transfer time) into seperate light/dark files? If using ffmpeg would be nice to choose between frame sequences or duped media at frame rate = half.

I think I can do that with ffmpeg (convert video to image sequence, and split images based on even/odd frames). Do HDR videos have a different name (in comparison with normal videos)? This can be used to determine whether a video is HDR or not.

Just checked and HDR videos are named the same format, so I don't know how I can check if a video is HDR or not automatically. The only other option I can think of is for the user to select which videos are HDR before copying starts (so that transcoding starts straight afterwards without any user intervention).

Update: Another (better) option is to use movie logging, but that can only be used if movie logging was enabled (obviously). The HDR settings used are logged when an HDR video is captured, otherwise this field is missing. I simply have to search for HDR in this file to determine if the video is HDR or not  :)

Quote from: GershberG on July 27, 2012, 09:56:20 PM
some questions:
is verification time\size based or CRC?
are both targets verified?

Verification works by checking that the byte size of the copied file matches that of the original file.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Malcolm Debono

Version 0.4
HDRSLR Offloader now supports transcoding HDR videos while importing  :)

The following steps can be done automatically:
- Extract the audio track from file (done through ffmpeg - included)
- Extract and sort even and odd frames from HDR videos (also done through ffmpeg)
- Process HDR frames with Enfuse (included) or with Photomatix (user must specify location of PhotomatixCL.exe from Options window)
- Save processed frames back to video (framerate and size can also be customized; done through ffmpeg)

Note: HDR videos are detected through the log file generated by ML, so please enable this option from ML menu before capturing




Download link:
http://malcolmdebono.com/apps/HDSLRoffloader_0.4.zip
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Michael Zöller

Now thats a nice new feature. Will try this tomorrow. One feature I was just thinking about would be a command-line option so that one could auto-run this whenever a card is inserted.
neoluxx.de
EOS 5D Mark II | EOS 600D | EF 24-70mm f/2.8 | Tascam DR-40

ilguercio

Nice application but movie log isn't available for the 50D.
Can you add an option for logless files? :)
Canon EOS 6D, 60D, 50D.
Sigma 70-200 EX OS HSM, Sigma 70-200 Apo EX HSM, Samyang 14 2.8, Samyang 35 1.4, Samyang 85 1.4.
Proud supporter of Magic Lantern.

screamer

Wow Malcom, this is amazing, i will give it immediately a try!! great work indeed
always trying to use the 100% of magic lantern..
Gear:
Canon 60D, all the samyang lenses, Canon 50 mm 1.4, Canon 60mm macro, Canon 70-300 usm, Sigma 4.5mm fisheye, Sigma 17-70 2.8, Canon 40mm f2.8 pancake, all the Lensbabies and a lot of other pieces, Flash metz 58 af2

Malcolm Debono

Quote from: Michael Zöller on July 30, 2012, 11:18:32 PM
Now thats a nice new feature. Will try this tomorrow. One feature I was just thinking about would be a command-line option so that one could auto-run this whenever a card is inserted.

That's a cool feature! I'll implement argument passing so that all the required info can be passed through cmd.

Quote from: ilguercio on July 30, 2012, 11:41:42 PM
Nice application but movie log isn't available for the 50D.
Can you add an option for logless files? :)

I'll try to find some other way  :) Only other option I can think of is opening a file chooser dialogue so that you can select which videos are HDR manually. This can also be used to process HDR videos without the importing part.

Quote from: screamer on July 31, 2012, 12:07:15 AM
Wow Malcom, this is amazing, i will give it immediately a try!! great work indeed

Thanks  :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Malcolm Debono

Version 0.5
- Manual selection of HDR videos (can be used both during importing stage, or from the Tools menu)
- Argument passing

Arguments supported are:
-p1 / -p2 / -p3            Specify which profile will be loaded automatically when the app starts
-start                         Automatically starts copying when app starts

Example:
"HDSLR Offloader" -p1 -start
This command starts the app, loads the Video profile, and starts the copying process automatically

To use the profiles, you will first need to have them saved by starting the app, specify which options will be used and the paths, and save the profile to one of the 3 slots through the File menu. This only needs to be done once, as you can then load the profile either manually through the File menu or the shortcut keys (CTRL + 1/2/3), or through the arguments.

Download link:
http://malcolmdebono.com/apps/HDSLRoffloader_0.5.zip

Enjoy  :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

ilguercio

Thanks a lot Malcolm ;)
I really appreciate your effort and contribution to the community.
Canon EOS 6D, 60D, 50D.
Sigma 70-200 EX OS HSM, Sigma 70-200 Apo EX HSM, Samyang 14 2.8, Samyang 35 1.4, Samyang 85 1.4.
Proud supporter of Magic Lantern.

Malcolm Debono

Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Michael Zöller

Thanks for sharing your work and even implementing ideas from the community (I think the ml community is really maturing :p)!
neoluxx.de
EOS 5D Mark II | EOS 600D | EF 24-70mm f/2.8 | Tascam DR-40

Malcolm Debono

A healthy community is all about sharing & helping. I wanted to build this app mainly for personal use, but if it can help others too, why not? :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

cosmo.lv

Quote from: Malcolm Debono on July 31, 2012, 03:55:16 PM
A healthy community is all about sharing & helping. I wanted to build this app mainly for personal use, but if it can help others too, why not? :)

Looks great tool :) i'm always impressed by peoples with knowledge of C and other languages - ehh.. :)
can i hope for Mac OS X application - or this stays only for Windows?  ??? :)
5D Mark II

Malcolm Debono

Quote from: cosmo.lv on August 01, 2012, 02:50:17 AM
Looks great tool :) i'm always impressed by peoples with knowledge of C and other languages - ehh.. :)
can i hope for Mac OS X application - or this stays only for Windows?  ??? :)

Thanks!

This is written in C#, which is a Windows language. There might be some solutions to compile this for Mac using tools like Mono. I'll see if I can manage to try it out when I find some free time  :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

cosmo.lv

Quote from: Malcolm Debono on August 01, 2012, 12:41:28 PM
Thanks!

This is written in C#, which is a Windows language. There might be some solutions to compile this for Mac using tools like Mono. I'll see if I can manage to try it out when I find some free time  :)

That's will be great! :)
5D Mark II

Malcolm Debono

Unfortunately it is next to impossible to port it for Mac (would be better to just write it from ground up in another language). The way HDR transcoding works is by passing parameters to external apps such as ffmpeg, enfuse and photomatix through command-line, which would need to be changed completely to work on Macs. There are also some other dependencies such as Microsoft Office, which is used for exporting a spreadsheet file of all the data in the LOG files.

One more thing I wanted to add is that the ML features within the app (mainly copying config file or all ML folder to card) currently do not support ML 2.3 (since the file structure was changed). An updated version will be released soon which offers support for both ML 2.3 and previous version (2.2).
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Malcolm Debono

Version 0.6
- Support for ML 2.3
- Fixed shot list generator (now fully compatible with log files generated by ML 2.3)
- Minor bug fixes

It is now easier than ever to keep your cards synchronised. You can now keep a copy of all your ML files (including the standard config and the hidden items config) on your PC, and copy either all the ML files or only the configs to your memory card with the click of a button - useful when you have many different memory cards and need to have the same configs on all cards!

The shot list generator has also been updated and is fully compatible with log files generated by ML 2.3, which now includes HDR and bitrate information. The list can also be exported as a spreadsheet.


Download link:
http://malcolmdebono.com/apps/HDSLRoffloader_0.6.zip

Enjoy  :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

hd1roam

I really appreciate your development efforts and support.
Do you ever get any sleep?

I think I need to tr this tool. Looks very interesting.
ML ver. 2.3: mostly still photos, HDR, pano, and some video. Targets of choice; Art, landscape, wildlife, collector vehicles, events. Equipment; EOS 550D/T2i | EF 100-400mm L USM - EF-S 18-55mm f4-5.6 IS - EF-S 55-250mm IS - Win7Ult., Lightroom, Photomatix, Topaz Labs, Hugin, GIMP, Smugmug

Malcolm Debono

Quote from: hd1roam on August 11, 2012, 06:20:55 PM
I really appreciate your development efforts and support.
Do you ever get any sleep?

Rest assured that I do  :D

This app isn't something that I spend a lot of work on. I only see what needs to be done (both stuff that I require, and user requests that can be implemented) and try to get it done in my free time (which I currently have plenty of since I'm on holidays!).
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

scrax

Quote from: Malcolm Debono on August 02, 2012, 03:27:49 PM
Unfortunately it is next to impossible to port it for Mac (would be better to just write it from ground up in another language). The way HDR transcoding works is by passing parameters to external apps such as ffmpeg, enfuse and photomatix through command-line, which would need to be changed completely to work on Macs.
all those three works on mac too  :)

Osx user, MLTools is for mac but it is still just a script because I had no feedback so far, so i've stopped adding things since for me it works and starts automatically when card is inserted.
If feedback are positive i'll bundle it in an app, but without testers i'll not make something that maybe works only on my particular configuration.

@ Malcolm if you want we can try to work together on an osx app
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

Malcolm Debono

@ scrax
I'll be more than happy to help you out, but I don't have any knowledge in mac programming (as far as I know it uses objective c). Had I known that this app would become 'popular', I would have based it on a more universal language (maybe java), however as it is it requires a complete re-work for it to work on Mac.

If you need to have a look at the code, let me know (it's in C#).
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

scrax

For now i'm just interested in the command line code you use for you conversion, so to try it on mac and compare result with what i have.

So far I can have only an automated process without a lot of user interaction to test the basic functions, but i'll add more options when those thing are tested working ok.

In those day I'm really busy so don't worry if have ont time right now.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

1%


Malcolm Debono

Hey guys, I'll get back to you next week as I'm currently abroad!
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

jordancolburn

This is awesome!  I'm currently working on a similar utility in python to provide a nice front end to some of the timelapse scripts I've seen around here for deflickr and  also to add a GUI to set keyframes for some simple ken burns style motion during timelapses.  It too uses ffmpeg, imagemagick and other tools that should be very portable.

I think It would be extremely beneficial to develop a GPL'd magic lantern companion software that would support multiple OS, and allow users to simply log footage, create timelapses, simple HDR video and provide extensisibility to take advantage of new ML features as they come out.  I'd be more than willing to help support a project like that in any way.

ilguercio

Quote from: jordancolburn on September 13, 2012, 06:14:07 PM
This is awesome!  I'm currently working on a similar utility in python to provide a nice front end to some of the timelapse scripts I've seen around here for deflickr and  also to add a GUI to set keyframes for some simple ken burns style motion during timelapses.  It too uses ffmpeg, imagemagick and other tools that should be very portable.

I think It would be extremely beneficial to develop a GPL'd magic lantern companion software that would support multiple OS, and allow users to simply log footage, create timelapses, simple HDR video and provide extensisibility to take advantage of new ML features as they come out.  I'd be more than willing to help support a project like that in any way.
+1 for this, at least all the basic stuff thata ML user should have on his PC.
Canon EOS 6D, 60D, 50D.
Sigma 70-200 EX OS HSM, Sigma 70-200 Apo EX HSM, Samyang 14 2.8, Samyang 35 1.4, Samyang 85 1.4.
Proud supporter of Magic Lantern.

Malcolm Debono

Quote from: jordancolburn on September 13, 2012, 06:14:07 PM
This is awesome!  I'm currently working on a similar utility in python to provide a nice front end to some of the timelapse scripts I've seen around here for deflickr and  also to add a GUI to set keyframes for some simple ken burns style motion during timelapses.  It too uses ffmpeg, imagemagick and other tools that should be very portable.

I think It would be extremely beneficial to develop a GPL'd magic lantern companion software that would support multiple OS, and allow users to simply log footage, create timelapses, simple HDR video and provide extensisibility to take advantage of new ML features as they come out.  I'd be more than willing to help support a project like that in any way.

Such a tool would definitely be useful! Unfortunately I'm not quite familiar with any cross-platform language (except for Java), which is why I started this app (which was very basic at first, and was only intended as a simple personal project!) in C#. If it can be useful, I'd be more than willing to share any code from this app if anyone wants to work on such a tool :)
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

jordancolburn

Quote from: Malcolm Debono on September 13, 2012, 06:36:03 PM
Such a tool would definitely be useful! Unfortunately I'm not quite familiar with any cross-platform language (except for Java), which is why I started this app (which was very basic at first, and was only intended as a simple personal project!) in C#. If it can be useful, I'd be more than willing to share any code from this app if anyone wants to work on such a tool :)
That would be great! I'm an electrical engineer by degree but working as a newbie software developer so I'm still learning a lot. 

AFAIK Python tends to be pretty portable (AKA, I can run my scripts on any linux/mac/win computer with all the library dependencies), I'm just not sure how to go about including all dependencies and packing it up really nicely for the different platforms.

It seems like most of the real complicated functions in your utility are calling other programs and the c# just provides the GUI and setting up batches/file operations, right?

Malcolm Debono

Yeah, I'm only using it for the GUI interface and batch operations, which should be easier to convert to another language. However we still need to make sure that all the libraries used are supported by multiple OSs.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

jordancolburn

Cool, well, I didn't mean to completely derail your original thread because the current app really useful! so should we start a separate thread to gauge potential interest and talk about how best to get started?

Malcolm Debono

No worries! Having a cross-platform app is the way to go since many users use Macs too. Feel free to open a new topic since it's cleaner that way.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

miyake

@Malcolm

Thankyou for creating this software.
I just test it, and I want to set encoding options.
Because, default output.mov is very heavy when I edit it.
So I want to choose mpeg2-hd or some low compression encoding format for low PC CPU usage.

I think,
Add one more edit box in HDR option tab , then describe full system() argument .
Of course , add new event handler in another edit box , and  the new one is realtime update by added event handler.
Finally, user is able to customize new editbox.
It's better to make a  full GUI encoding options support, but I know it's a really hard work. So it's a landing point, I think.
Please consider it.

P.S. I'm only using a VisualC++ for my work. I never use  C# .......

Malcolm Debono

Thanks miyake! I'll see what I can do in the coming days! Shouldn't be too much work.
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

Malcolm Debono

Version 0.6.1
- Added support for customizing ffmpeg parameters for output video, so you can use different output formats such as MP4 or MPG

The PDF includes a usage example and an external link with more info on ffmpeg commands!


Download link:
http://malcolmdebono.com/apps/HDSLRoffloader_0.6.1.zip
Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

miyake

Thanks Malcolm.
It's very good for me.

my Extra param is
-q:v 1 -vcodec mpeg2video

And multiplexing container is
m2v

Mpeg2 HD and ALL-I is really right wait for CPU.

Malcolm Debono

Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

scrax

Malcolm I've worked on MLtools those days and I've managed to add some GUI interaction to the script.
It does most of the thing HDSLR does but in sort of wizard way. So far I have problem to generate FST script with align_image_stack but maybe is the ML script that don't works, more test needed.
I'm still curious to take a look at the script that you use, maybe i can copy something ;)
thanks  :)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

Malcolm Debono

Wedding & event cinematographer
C100 & 6D shooter
New here?  Check out the FAQs here!

tin2tin

If you like to add timecode and reelname to the files then ffmbc can do that with command line switches:
-timecode <Timecode> -metadata reel_name=<ReelName>
The <Timecode> is in this format 00:00:00:00 and the <ReelName> is letters and numbers. I've just added this functionality to EyeFrame Converter. I believe that it is possible to just add the metadata without an actual transcoding.

taapo

Very nice program, but it proved to be (almost) catastrohic for me today in a certain situation. Maybe this feedback could improve it?

We work with two camera's. When we have the program rewrite the filename, it is possible that movies that were made on the same moment overwrite eachother (same minute and second - it's possible, and becomes worse with the addition of extra cameras. A random hash after the filename would solve it. Or take a combination of time, serial number of card and folder name (and hash).

Secondly, one of the camera's had a wrong timestamp (the time was 3 hours off), so the the filesorting was completelt thrown off. I'd rather see the timestamp based on the moment we are offloading. This solves the double camera overwriting also.

Is there a possibility to have this situation resolved? I have to admit - of all (commercial) programs I have tried, this freebie was the best - but proved unsufficient for production work at the moment.

I'd even consider donating money to improve it, so it could become the swiss offloading knife for windows user. Let me know if this would be an incentive.

donjames150

Quote from: Malcolm Debono on July 30, 2012, 04:54:37 PM
Version 0.4
HDRSLR Offloader now supports transcoding HDR videos while importing  :)

The following steps can be done automatically:
- Extract the audio track from file (done through ffmpeg - included)
- Extract and sort even and odd frames from HDR videos (also done through ffmpeg)
- Process HDR frames with Enfuse (included) or with Photomatix (user must specify location of PhotomatixCL.exe from Options window)
- Save processed frames back to video (framerate and size can also be customized; done through ffmpeg)

Note: HDR videos are detected through the log file generated by ML, so please enable this option from ML menu before capturing




Download link:
http://malcolmdebono.com/apps/HDSLRoffloader_0.4.zip

I'd like to keep this thread alive as it's turned out to be a good solution for any of you Real Estate video shooters out there who have multiple HDR inside window shots. This program (ver 0.6.1) does the job nicely.  I found the enfuse option quality too poor, but the Photomatix works well. It's about 3 times faster than the interframe script option plus it batches unlimited number of files. Sadly it's not available for the mac, pc only.

EOS 760D + 55-250mm + Tokina 11-16mm

donjames150

EOS 760D + 55-250mm + Tokina 11-16mm

Forssux

Hi There,

Hi There I made a test movie and noticed that ...
I get a Charlie Chaplin effect .....although frame rate is correct at 25
The sound isn't embedded in the output file.

I'm probably doing something wrong

Can somebody help me...
Guy