Author Topic: MLTools (OsX utility to manage ML install, files import and scripts conversions)  (Read 42469 times)

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Here is the first test version for MLTools bundled with last version of ML 2.3 (for simple install on mac).
Maybe better to have termina off and run it twice first time should be needed...
MLTools is a droppable app that helps with file and script made with ML.
At first start will check and install dependencies, then will show a simple dropdown menu with some options.
First time be sure to check MLTools Preferences button for setting up your working folder (where file will be imported and converted), your import preferences and if you want to rename jpg file with date taken.
File transfer makes a checksum verification before deleting imported files (if you chose to move them).

OPTIONS
If autoimport is off you can use MLTools simply by selecting an option and then pressing next button.
Options are:
-Write Magic Lantern settings on card: It will write previously saved settings from MLcard on a new card (to have same MLsettings on multiplecard, works also for cropmarks)
-Save Magic Lantern settings from card: Saves to your working folder ML settings and cropmarks from the actually mounted ML card
-Delete  saved Magic Lantern settings: deletes ML settings and cropmarks from the disk
-Toggle Autoimport feat.: Activate or deactivate autoimport feat. When active autoimport will automatically check ML card for file to import and convert, if no card preset it will show an import dialog asking for files or folder to process
-Install ML on card with saved settings: Installs ML 2.3 and saved settings on a new card (it needs to be formatted on camera before this step)
-Scan card for ML generated file (default): Automatically scans ML card for files, scripts to import and convert into the working folder.
-Toggle Unmount after import: Option to unmount card after file import (useful with autoimport option)

You can use manual mode for preparing multiple card with same ML settings and cropmarks,
After setting up all your cards, turn autoimport on, unmount on and with Image Capture set MLTools to start when the card is inserted.
That way MLTools will start and import your file in the working folder you chose and the it will convert MLscripts, 422 file and so on if needed.

TIP:
If you want to convert scripts or files from a folder you can just drop them on the MLTools icon. MLTools can convert RAW, HDR script, HDR video (if video log is on in ML), 422 file, Focus stack script (to be tested more).

PREFERENCES:
From here you can chose to set up various options of MLTools like:

Working Folder: where to save your imported and converted files (default ~/MLTools/),

File Import: import files by copy, move or nothing (default copy).

Rename by date: if you want to rename file by date taken (default no renaming)

WARNING
It's tested so far only on my system with osx Lion 10.7.5 so report all your experiences what works and what not please. Make backup of your data on card or use test shot before using it for important files.
Some functions like script conversion may not work right cause recent changes in ML, they should be updated maybe.

DOWNLOAD

MLTools 0.8 fixes: manually close to debug better, smaller size.

MLTools 0.7 fixes: added video RAW import and conversion, fixes to md5 check and folders checks.
MLTools 0.6 Alpha test: fixed some error and folder scan not always working.
MLTools 0.5 Alpha test: more cleanup, waiting feedback
MLTools 0.3 Alpha test: clean up a little, feedback needed).
MLTools 0.2 Alpha test: New version with most of dependencies integrated to reduce install problems and errors, only PIL needs to be installed (seems there are some problem with that, more feedback needed).
MLTools 0.1 Alpha test: First test version to test, waiting for feedback.
REQUIRED: command_line_tools_for_xcode_june_2012.dmg (154,6MB) from Apple here (is free but needs registration first) or you can install 1.83GB for XCode
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10077
  • 5D Mark Free
Rename JPG from EXIF date => +10 for this :)

Hopefully these scripts can be ported to Linux too. In future I might add my scripts for HDR from a single RAW, or for correcting fisheye photos.

I'm also researching how suitable is to apply exposure compensation automatically, so all shots look consistent in exposure (based on the idea from the timelapse deflickering script).

G3gg0 had a nice suggestion, but only for 60D/7D/5D3 users for now: auto-rotate pictures based on level sensor data.

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Rename JPG from EXIF date => +10 for this :)

Hopefully these scripts can be ported to Linux too. In future I might add my scripts for HDR from a single RAW, or for correcting fisheye photos.

There is something mac only but for installing dependencies so not needed on linux, most of the script use linux tools installed on osx with homebrew so it should be easily portable. I hope there will be more people who want to contribute on optimize and work on it.

For example rename from exif is done with:
exiv2 -r '%Y-%m-%d_%H-%M-%S_:basename:' rename "$file_is"
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10077
  • 5D Mark Free
The GUI toolkit you have used is specific to Mac, or is something more portable?

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
The GUI toolkit you have used is specific to Mac, or is something more portable?
Is specific for mac, I don't know if there is something similar for linux.
The app for the GUI input is CocoaDialog.app, for making the bundle so to have an app that is droppable, with icons and so on I used Platypus also only for mac probably.
Also settings are stored in a file using mac .plist system, maybe saving MLTools setting in plain text will be better for portability.
The main file is script, it makes all the works and can work from command line, other files already working for linux are make_bootable.sh and vid2hdr.sh
prepare.Lion.sh has some basic linux support that can be used for making a universal script maybe.
prepare.Lion.scpt is an apple script to install PIL on osx cause sudo is needed.
startMENU.pl and settings_menu are for showing the options that can be activated on the script. and probably need to be rewritten for a linux GUI
AppSettings.plist MainManu.nib appIcon.icns are generated by Platypus to crate the bundle app.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
added link to required xcode command line tools to first post
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10077
  • 5D Mark Free
This sounds interesting: http://www.gnustep.org/

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
For me this is more easy at first look: http://www.gnustep.it/Renaissance/index.html
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
For me this is more easy at first look: http://www.gnustep.it/Renaissance/index.html
but don't know if can use scripts like cocoadialog
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

collbeanzfilms

  • New to the forum
  • *
  • Posts: 5
I keep getting this when I try to import the files from my card...

/Volumes/EOS_DIGITAL/ML/CROPMKS/*.BMP: No such file or directory pip pip already installed enfuse enfuse is missing convert convert is missing wget wget is missing ufraw-batch ufraw-batch is missing mplayer mplayer is missing mypip: /usr/local/bin/pip numpy numpy already installed PIL PIL is missing prepare.Lion.scpt: execution error: Finder got an error: The command exited with a non-zero status. (1)

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
I keep getting this when I try to import the files from my card...
 prepare.Lion.scpt: execution error: Finder got an error: The command exited with a non-zero status. (1)

You don't have some tools installed but the Apple script for installing them is failing, which os version are you using?

To find out what's wrong try to start this Applescript it will install PIL or give us error messages.

Also do you have installed XCode or command line xcode tools?

Basically to work MLTools needs some command line applications installed to run, those are:
Python      (already on Osx Lion)
and those python modules:
pip             (already on Osx Lion): to install other python modules
numpy      (already on Osx Lion): needed to convert 422 files
PIL            (need to be installed): needed to convert 422 files
watchdog  (need to be installed): not used yet
Ruby         (already on Osx Lion)
Other tools needed that are installed by MLTools are:
Hugin (for enfuse and align_image_stack): to merge hdr and focus stack script generated by ML. I think in future version to integrate that in the bundle so to be sure that they are there without need to install them at first start.done check next post
mercurial: for downloading last ML code (not used for now);
convert (ImageMagick): to convert RAW to jpg;
wget: to download and install what's missing;
ufraw-batch (ufraw): to convert CR2 to jpg;
mplayer: to render HDR video and in future for timelapse video too.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Here is a new 0.2 version with most of the tools needed integrated into MLTools, to reduce problems with dependencies.

Only thing that I don't know how to bundle is PIL, here integrated are:
align_image stack
enblend
enfuse
convert
mencoder
mplayer
ufraw-batch
wget

this should work also without xcode or command lines tools installed.

If you have problem try to remove the .MLTools folder in your user home folder (it's hidden by default on osx)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
0.3 clean up a little version.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
0.5 posted, with more fixes. (0.4 was used for internal test  :P )

https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.5.app.zip
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

jplxpto

  • Developer
  • Hero Member
  • *****
  • Posts: 506

And why not ipython ?

http://ipython.org/

"
IPython provides a rich toolkit to help you make the most out of using Python, with:

Powerful Python shells (terminal and Qt-based).
Support for interactive data visualization and use of GUI toolkits.
"

jplxpto

  • Developer
  • Hero Member
  • *****
  • Posts: 506
I do not know but there is more that possibility.

http://en.wikipedia.org/wiki/PyGTK

http://www.pygtk.org/

"
PyGTK lets you to easily create programs with a graphical user interface using the Python programming language. The underlying GTK+ library provides all kind of visual elements and utilities for it and, if needed, you can develop full featured applications for the GNOME Desktop.

PyGTK applications are truly multiplatform and they're able to run, unmodified, on Linux, Windows, MacOS X and other platforms.

Other distinctive features of PyGTK are, besides its ease of use and rapid prototyping, its first class accessibility support or the capability to deal with complex multilingual or bidirectional text for fully localized applications.
"

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Updated first download link with some fixes to MLTools
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
New version out with some fixes from raw2dng.app and also the option to import and convert raw files.
More bugs reported more robust we can make this so try carefully. At first make copies of your card and files, it uses md5 check before removing any file copied but test it, it's out for testing for now...

MLTools latest zip

also maybe to make it more compatible 422 conversion could be removed now that ML makes dng (that isn't yet added btw)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

budafilms

  • Hero Member
  • *****
  • Posts: 705
  • 5D Mark a1ex
I am using Lion Mountine 10.8.2. Appears a horrible voice trying to sing. I can´t undersatnd what I have to do. Maybe is not for 10.8.2. Any suggestion?

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
I am using Lion Mountine 10.8.2. Appears a horrible voice trying to sing. I can´t undersatnd what I have to do. Maybe is not for 10.8.2. Any suggestion?
:) It's the finished conversion/import message.

Basic use should be open it once to set the working folder (default is ~/MLTools) and enable autoimport, next connect your card and set Image Acquisition.app to automatically start MLTools when the card is inserted. Also in MLTools preferences you can enable options to remove file from card and rename jpg with date taken from exif. Once all set when you insert the card IA should start MLT and it will copy to the working folder all the files on your card, once copied/moved the files (in a subfolder of the working folder with name based on the import date) it should convert them if needed.
If you have problem post what you see in the windows with the text.

I'll update a new version without the sound and some different setting so it will be easier to debug.

for now if you double click it what you get?
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
New version uploaded, now it has to be closed manually so the text can be seen, we can restore auto close when finished beta version.
I've made small changes to make card just to make the app smaller.
also removed the voice
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

AnotherDave

  • Senior
  • ****
  • Posts: 348
Link seems to be broken.

Any reason why?

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Link seems to be broken.

Any reason why?
Should be fixed now
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools

guentergunter

  • Member
  • ***
  • Posts: 144
  • 5D2
I have some problems using the app. (OSX 10.8.3)

What I want:
While testing the new RAW-feature on my 5D2, I realized, that the cards fill up that fast, that there's now the need to transfer them to a hard drive to gain again free recording space.

So, I'm searching for an On-Set-solution. Which means to me a program with the following specs:
- automated start after card insertion
- transferring the RAW-files from the card to a specified output-folder (while on the fly assembling separated files!)
- (optional; but of more sense to me) only reading the RAW-files and saving the dng-files.
- an addition could be a little jpg-preview just from the first frames of every RAW-file for checking porposes.

It's good news and I'm very glad to see, that MLTools is going to be exactly what I'm waiting for.
So I tried it.

First of all I installed PIL ;-)
Simple X-Code-Guide

...and recorded the following files for testing purposes:
M11-1448.RAW
M11-1449.RAW
M11-1449.R00

Autostart works: it detects the card and starts the import process into the specified folder (GREAT!!).

But:
- it just imports the .RAW-files without the adjacent .R00-file
- it imports the bmp's from the 'CROPMKS'-folder inside the ML-folder. (?)
- it does not convert to DNG at all (also I can't find any options on that)
- there's no progress bar - so I can only see it's activity by the blinking of the card reader's LED
- it does not unmount the card after import (altough feature activated)


What I found out about the 'cat'-command:
If there's only one input file given, it simply copies the file.
___

Some log-data...

When starting the app without cardreader attached, it says:
Quote
pippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn’t understand the «event panSdlog» message. (-1708)

When the programm is started by an attached CF-card (auto-import: off) and (to my surprise) with Dropbox running in background:
Quote
pippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpyImportError: Bad magic number in /Users/User/Library/Application Support/Dropbox/Dropbox.app/Contents/Resources/site.pyc numpy is missing PILImportError: Bad magic number in /Users/User/Library/Application Support/Dropbox/Dropbox.app/Contents/Resources/site.pyc PIL is missing prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn’t understand the «event panSdlog» message. (-1708)

The same, but without dropbox:
Quote
pippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn’t understand the «event panSdlog» message. (-1708)

With both AutoImport and Unmount activated and autostarted by an attached CF-card it does import, but not unmount (besides the problem described above):
Quote
pippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn’t understand the «event panSdlog» message. (-1708) Searching… Magic Lantern on /Volumes/EOS_DIGITAL/:FOUND cp: /Volumes/EOS_DIGITAL/ML/SETTINGS/*.CFG: No such file or directory import file: filename: M11-1448.RAW File Transfer Check: ok import file: filename: M11-1449.RAW File Transfer Check: ok imported: 0 Started : 130611T164151 Finished: 130611T164841



Great app so far! Many Thanks!
5D2 ML RAW + VAF-5D2b: http://vimeo.com/69350650

scrax

  • Developer
  • Hero Member
  • *****
  • Posts: 1417
  • Code monkey
Thank's for the report, I'll work on it, PIL was needed for 422 conversion, but I'll remove that and it should fix the dropbox error.

Also seems that it can't find bundled stuff, so will try in another way.

.R00 is not yet implemented, will look at it too.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus - EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX Lion, Photoshop & Lightroom -no video experience- MLTools