Magic Lantern Forum

Using Magic Lantern => Post-processing Workflow => Topic started by: scrax on October 15, 2012, 11:53:41 AM

Title: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 11:53:41 AM
Here is the first test version for MLTools bundled with last version of ML 2.3 (for simple install on mac).
Maybe better to have termina off and run it twice first time should be needed...
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_menu.png)
MLTools is a droppable app that helps with file and script made with ML.
At first start will check and install dependencies, then will show a simple dropdown menu with some options.
First time be sure to check MLTools Preferences button for setting up your working folder (where file will be imported and converted), your import preferences and if you want to rename jpg file with date taken.
File transfer makes a checksum verification before deleting imported files (if you chose to move them).

OPTIONS
If autoimport is off you can use MLTools simply by selecting an option and then pressing next button.
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_menu_option.png)
Options are:
-Write Magic Lantern settings on card: It will write previously saved settings from MLcard on a new card (to have same MLsettings on multiplecard, works also for cropmarks)
-Save Magic Lantern settings from card: Saves to your working folder ML settings and cropmarks from the actually mounted ML card
-Delete  saved Magic Lantern settings: deletes ML settings and cropmarks from the disk
-Toggle Autoimport feat.: Activate or deactivate autoimport feat. When active autoimport will automatically check ML card for file to import and convert, if no card preset it will show an import dialog asking for files or folder to process
-Install ML on card with saved settings: Installs ML 2.3 and saved settings on a new card (it needs to be formatted on camera before this step)
-Scan card for ML generated file (default): Automatically scans ML card for files, scripts to import and convert into the working folder.
-Toggle Unmount after import: Option to unmount card after file import (useful with autoimport option)

You can use manual mode for preparing multiple card with same ML settings and cropmarks,
After setting up all your cards, turn autoimport on, unmount on and with Image Capture (http://en.wikipedia.org/wiki/Image_Capture) set MLTools to start when the card is inserted.
That way MLTools will start and import your file in the working folder you chose and the it will convert MLscripts, 422 file and so on if needed.

TIP:
If you want to convert scripts or files from a folder you can just drop them on the MLTools icon. MLTools can convert RAW, HDR script, HDR video (if video log is on in ML), 422 file, Focus stack script (to be tested more).

PREFERENCES:
From here you can chose to set up various options of MLTools like:
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_pref.png)
Working Folder: where to save your imported and converted files (default ~/MLTools/),
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_workingfolder.png)
File Import: import files by copy, move or nothing (default copy).
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_import.png)
Rename by date: if you want to rename file by date taken (default no renaming)
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLTools_rename.png)

WARNING
It's tested so far only on my system with osx Lion 10.7.5 so report all your experiences what works and what not please. Make backup of your data on card or use test shot before using it for important files.
Some functions like script conversion may not work right cause recent changes in ML, they should be updated maybe.

DOWNLOAD

MLTools 0.8 fixes (https://dl.dropboxusercontent.com/u/123918/MagicLantern/MLTools/MLTools.last.app.zip): manually close to debug better, smaller size.

MLTools 0.7 fixes (https://dl.dropboxusercontent.com/u/123918/MagicLantern/MLTools/MLTools.0.7.app.zip): added video RAW import and conversion, fixes to md5 check and folders checks.
MLTools 0.6 Alpha test (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.6.app.zip): fixed some error and folder scan not always working.
MLTools 0.5 Alpha test (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.5.app.zip): more cleanup, waiting feedback
MLTools 0.3 Alpha test (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.3.app.zip): clean up a little, feedback needed).
MLTools 0.2 Alpha test (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.2.app.zip): New version with most of dependencies integrated to reduce install problems and errors, only PIL needs to be installed (seems there are some problem with that, more feedback needed).
MLTools 0.1 Alpha test (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.1.app.zip): First test version to test, waiting for feedback.
REQUIRED: command_line_tools_for_xcode_june_2012.dmg (154,6MB) from Apple here (http://developer.apple.com/downloads) (is free but needs registration first) or you can install 1.83GB for XCode
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: a1ex on October 15, 2012, 12:14:41 PM
Rename JPG from EXIF date => +10 for this :)

Hopefully these scripts can be ported to Linux too. In future I might add my scripts for HDR from a single RAW, or for correcting fisheye photos.

I'm also researching how suitable is to apply exposure compensation automatically, so all shots look consistent in exposure (based on the idea from the timelapse deflickering script).

G3gg0 had a nice suggestion, but only for 60D/7D/5D3 users for now: auto-rotate pictures based on level sensor data.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 12:24:53 PM
Quote from: a1ex on October 15, 2012, 12:14:41 PM
Rename JPG from EXIF date => +10 for this :)

Hopefully these scripts can be ported to Linux too. In future I might add my scripts for HDR from a single RAW, or for correcting fisheye photos.

There is something mac only but for installing dependencies so not needed on linux, most of the script use linux tools installed on osx with homebrew so it should be easily portable. I hope there will be more people who want to contribute on optimize and work on it.

For example rename from exif is done with:
exiv2 -r '%Y-%m-%d_%H-%M-%S_:basename:' rename "$file_is"
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: a1ex on October 15, 2012, 12:29:30 PM
The GUI toolkit you have used is specific to Mac, or is something more portable?
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 12:36:10 PM
Quote from: a1ex on October 15, 2012, 12:29:30 PM
The GUI toolkit you have used is specific to Mac, or is something more portable?
Is specific for mac, I don't know if there is something similar for linux.
The app for the GUI input is CocoaDialog.app, for making the bundle so to have an app that is droppable, with icons and so on I used Platypus also only for mac probably.
Also settings are stored in a file using mac .plist system, maybe saving MLTools setting in plain text will be better for portability.
The main file is script, it makes all the works and can work from command line, other files already working for linux are make_bootable.sh and vid2hdr.sh
prepare.Lion.sh has some basic linux support that can be used for making a universal script maybe.
prepare.Lion.scpt is an apple script to install PIL on osx cause sudo is needed.
startMENU.pl and settings_menu are for showing the options that can be activated on the script. and probably need to be rewritten for a linux GUI
AppSettings.plist MainManu.nib appIcon.icns are generated by Platypus to crate the bundle app.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 12:53:29 PM
added link to required xcode command line tools to first post
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: a1ex on October 15, 2012, 01:01:01 PM
This sounds interesting: http://www.gnustep.org/
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 01:16:26 PM
For me this is more easy at first look: http://www.gnustep.it/Renaissance/index.html
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 01:21:25 PM
Quote from: scrax on October 15, 2012, 01:16:26 PM
For me this is more easy at first look: http://www.gnustep.it/Renaissance/index.html
but don't know if can use scripts like cocoadialog
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: collbeanzfilms on October 15, 2012, 02:11:13 PM
I keep getting this when I try to import the files from my card...

/Volumes/EOS_DIGITAL/ML/CROPMKS/*.BMP: No such file or directory pip pip already installed enfuse enfuse is missing convert convert is missing wget wget is missing ufraw-batch ufraw-batch is missing mplayer mplayer is missing mypip: /usr/local/bin/pip numpy numpy already installed PIL PIL is missing prepare.Lion.scpt: execution error: Finder got an error: The command exited with a non-zero status. (1)
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 07:03:38 PM
Quote from: collbeanzfilms on October 15, 2012, 02:11:13 PM
I keep getting this when I try to import the files from my card...
prepare.Lion.scpt: execution error: Finder got an error: The command exited with a non-zero status. (1)

You don't have some tools installed but the Apple script for installing them is failing, which os version are you using?

To find out what's wrong try to start this Applescript (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/prepare.Lion.scpt) it will install PIL or give us error messages.

Also do you have installed XCode or command line xcode tools?

Basically to work MLTools needs some command line applications installed to run, those are:
Python      (already on Osx Lion)
and those python modules:
pip             (already on Osx Lion): to install other python modules
numpy      (already on Osx Lion): needed to convert 422 files
PIL            (need to be installed): needed to convert 422 files
watchdog  (need to be installed): not used yet
Ruby         (already on Osx Lion)
Other tools needed that are installed by MLTools are:
Hugin (for enfuse and align_image_stack): to merge hdr and focus stack script generated by ML. I think in future version to integrate that in the bundle so to be sure that they are there without need to install them at first start.done check next post
mercurial: for downloading last ML code (not used for now);
convert (ImageMagick): to convert RAW to jpg;
wget: to download and install what's missing;
ufraw-batch (ufraw): to convert CR2 to jpg;
mplayer: to render HDR video and in future for timelapse video too.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 15, 2012, 08:41:34 PM
Here is a new 0.2  (https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.2.app.zip)version with most of the tools needed integrated into MLTools, to reduce problems with dependencies.

Only thing that I don't know how to bundle is PIL, here integrated are:
align_image stack
enblend
enfuse
convert
mencoder
mplayer
ufraw-batch
wget

this should work also without xcode or command lines tools installed.

If you have problem try to remove the .MLTools folder in your user home folder (it's hidden by default on osx)
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on October 16, 2012, 12:38:59 AM
0.3 clean up a little version.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on November 27, 2012, 10:18:39 PM
0.5 posted, with more fixes. (0.4 was used for internal test  :P )

https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLTools.0.5.app.zip
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: jplxpto on November 28, 2012, 12:07:35 AM

And why not ipython ?

http://ipython.org/

"
IPython provides a rich toolkit to help you make the most out of using Python, with:

Powerful Python shells (terminal and Qt-based).
Support for interactive data visualization and use of GUI toolkits.
"
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: jplxpto on November 28, 2012, 12:25:23 AM
I do not know but there is more that possibility.

http://en.wikipedia.org/wiki/PyGTK

http://www.pygtk.org/

"
PyGTK lets you to easily create programs with a graphical user interface using the Python programming language. The underlying GTK+ library provides all kind of visual elements and utilities for it and, if needed, you can develop full featured applications for the GNOME Desktop.

PyGTK applications are truly multiplatform and they're able to run, unmodified, on Linux, Windows, MacOS X and other platforms.

Other distinctive features of PyGTK are, besides its ease of use and rapid prototyping, its first class accessibility support or the capability to deal with complex multilingual or bidirectional text for fully localized applications.
"
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on May 13, 2013, 11:51:02 PM
Updated first download link with some fixes to MLTools
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on May 27, 2013, 02:18:43 PM
New version out with some fixes from raw2dng.app and also the option to import and convert raw files.
More bugs reported more robust we can make this so try carefully. At first make copies of your card and files, it uses md5 check before removing any file copied but test it, it's out for testing for now...

MLTools latest zip (https://dl.dropboxusercontent.com/u/123918/MagicLantern/MLTools/MLTools.last.app.zip)

also maybe to make it more compatible 422 conversion could be removed now that ML makes dng (that isn't yet added btw)
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: budafilms on May 29, 2013, 09:40:31 AM
I am using Lion Mountine 10.8.2. Appears a horrible voice trying to sing. I can´t undersatnd what I have to do. Maybe is not for 10.8.2. Any suggestion?
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on June 02, 2013, 05:13:50 AM
Quote from: budafilms on May 29, 2013, 09:40:31 AM
I am using Lion Mountine 10.8.2. Appears a horrible voice trying to sing. I can´t undersatnd what I have to do. Maybe is not for 10.8.2. Any suggestion?
:) It's the finished conversion/import message.

Basic use should be open it once to set the working folder (default is ~/MLTools) and enable autoimport, next connect your card and set Image Acquisition.app to automatically start MLTools when the card is inserted. Also in MLTools preferences you can enable options to remove file from card and rename jpg with date taken from exif. Once all set when you insert the card IA should start MLT and it will copy to the working folder all the files on your card, once copied/moved the files (in a subfolder of the working folder with name based on the import date) it should convert them if needed.
If you have problem post what you see in the windows with the text.

I'll update a new version without the sound and some different setting so it will be easier to debug.

for now if you double click it what you get?
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on June 02, 2013, 06:39:31 PM
New version (https://dl.dropboxusercontent.com/u/123918/MagicLantern/MLTools/MLTools.last.app.zip) uploaded, now it has to be closed manually so the text can be seen, we can restore auto close when finished beta version.
I've made small changes to make card just to make the app smaller.
also removed the voice
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: AnotherDave on June 02, 2013, 10:52:46 PM
Link seems to be broken.

Any reason why?
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on June 03, 2013, 05:14:48 AM
Quote from: AnotherDave on June 02, 2013, 10:52:46 PM
Link seems to be broken.

Any reason why?
Should be fixed now
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: guentergunter on June 11, 2013, 05:16:37 PM
I have some problems using the app. (OSX 10.8.3)

What I want:
While testing the new RAW-feature on my 5D2, I realized, that the cards fill up that fast, that there's now the need to transfer them to a hard drive to gain again free recording space.

So, I'm searching for an On-Set-solution. Which means to me a program with the following specs:
- automated start after card insertion
- transferring the RAW-files from the card to a specified output-folder (while on the fly assembling separated files!)
- (optional; but of more sense to me) only reading the RAW-files and saving the dng-files.
- an addition could be a little jpg-preview just from the first frames of every RAW-file for checking porposes.

It's good news and I'm very glad to see, that MLTools is going to be exactly what I'm waiting for.
So I tried it.

First of all I installed PIL ;-)
Simple X-Code-Guide (http://blog.artooro.com/2013/01/04/how-to-install-pil-python-imaging-library-on-mac-os-x-10-8/)

...and recorded the following files for testing purposes:
M11-1448.RAW
M11-1449.RAW
M11-1449.R00

Autostart works: it detects the card and starts the import process into the specified folder (GREAT!!).

But:
- it just imports the .RAW-files without the adjacent .R00-file
- it imports the bmp's from the 'CROPMKS'-folder inside the ML-folder. (?)
- it does not convert to DNG at all (also I can't find any options on that)
- there's no progress bar - so I can only see it's activity by the blinking of the card reader's LED
- it does not unmount the card after import (altough feature activated)


What I found out about the 'cat'-command:
If there's only one input file given, it simply copies the file.
___

Some log-data...

When starting the app without cardreader attached, it says:
Quotepippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn't understand the «event panSdlog» message. (-1708)

When the programm is started by an attached CF-card (auto-import: off) and (to my surprise) with Dropbox running in background:
Quotepippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpyImportError: Bad magic number in /Users/User/Library/Application Support/Dropbox/Dropbox.app/Contents/Resources/site.pyc numpy is missing PILImportError: Bad magic number in /Users/User/Library/Application Support/Dropbox/Dropbox.app/Contents/Resources/site.pyc PIL is missing prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn't understand the «event panSdlog» message. (-1708)

The same, but without dropbox:
Quotepippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn't understand the «event panSdlog» message. (-1708)

With both AutoImport and Unmount activated and autostarted by an attached CF-card it does import, but not unmount (besides the problem described above):
Quotepippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert is missingwgetwget is missingufraw-batchufraw-batch is missingmplayermplayer is missingnumpynumpy already installed PILPIL already installed prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn't understand the «event panSdlog» message. (-1708) Searching... Magic Lantern on /Volumes/EOS_DIGITAL/:FOUND cp: /Volumes/EOS_DIGITAL/ML/SETTINGS/*.CFG: No such file or directory import file: filename: M11-1448.RAW File Transfer Check: ok import file: filename: M11-1449.RAW File Transfer Check: ok imported: 0 Started : 130611T164151 Finished: 130611T164841



Great app so far! Many Thanks!
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: scrax on June 13, 2013, 01:51:21 PM
Thank's for the report, I'll work on it, PIL was needed for 422 conversion, but I'll remove that and it should fix the dropbox error.

Also seems that it can't find bundled stuff, so will try in another way.

.R00 is not yet implemented, will look at it too.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: guentergunter on June 14, 2013, 03:43:42 PM
Quote from: scrax on June 13, 2013, 01:51:21 PM
.R00 is not yet implemented, will look at it too.
You may want to take a look at THIS (http://www.magiclantern.fm/forum/index.php?topic=6218.msg50682#msg50682) (and the following posts. Since the problem about spanned files has been solved there.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: leebeau on July 04, 2013, 02:00:33 AM
i am having trouble using the .sh script files created by ML when doing HDR
osx 10.8 macbook pro2012
xcode installed
command line tools installed

when i run them in terminal i get :

Last login: Wed Jul  3 18:46:24 on ttys001
/Users/macbook/Desktop/jpeg_backyard_6_24_13\ 3/HDR_0496.SH ; exit;
FullSail-W89240P8644:~ macbook$ /Users/macbook/Desktop/jpeg_backyard_6_24_13\ 3/HDR_0496.SH ; exit;
/Users/macbook/Desktop/jpeg_backyard_6_24_13 3/HDR_0496.SH: line 5: align_image_stack: command not found
/Users/macbook/Desktop/jpeg_backyard_6_24_13 3/HDR_0496.SH: line 6: enfuse: command not found
rm: HDR_AIS_0496*: No such file or directory
logout

[Process completed]



and when i open MLTools:

pippip already installedalign_image_stackalign_image_stack is missingenfuseenfuse is missingconvertconvert already installedwgetwget already installedufraw-batchufraw-batch already installedmplayermplayer already installednumpynumpy already installed PILPIL is missing prepare.Lion.scpt: execution error: AppleScript Runner got an error: "username" doesn't understand the «event panSdlog» message. (-1708) Import file: nothing




do i misunderstand what MLTools does? i thought i would make the .sh files run correctly?

please help
thanks a bunch i know everyone working on ML and MLTools is doing it for free on their own time any help you can give me is going above and beyond
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: leebeau on July 12, 2013, 06:53:03 AM
I think i've found the easy solution to my question at least for anyone running 10.8 or in my case 10.8.4 . sorry if somewhat unrelated to ML tools original post but in my search of how to make to bash scripts for hdr from ml work this is the only thread that even got close to dealing with the subject.

Big ups to my boy suds who is a wizzard with many programming things that make this look like nothing :

What you need to do is first set your environment variables.

1) Open Terminal
2) type `cd` and press enter, this will take you to your users home directory
3) type `sudo open -t .bash_profile`, this may ask you for your password
4) Copy in the text below

export PATH="<path_to_hugin_tools>/bin:$PATH"

5) Save the file and close it

That should be all you have to do. You will need to restart terminal if you want to run the scripts directly from terminal, if you run the scripts from finder using terminal, you should be good.

the hugin tools referenced here come from the latest distro of hugin for osx and it is referring to the hugin folder that is the command line tools.

i saw another post before mine with similar issues specifically with a library that is resident in osx that isn't up the the same version as what hugin uses if you set the environment variables correctly it will reference the lib that is included in the hugin tools.
hope this helps someone.
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: timerickson on April 30, 2014, 11:38:38 AM
Ahh, thank you leebeau!!

That was the last missing piece.

For the record, you need to
1. Download Hugin: http://hugin.sourceforge.net/download/
2. Set your terminal environment variables to include the path to HuginTools folder
    I have everything from the 2013 Hugin download in a folder called "Tools" in my Applications folder, so I added the following to my .bash_profile

export PATH="/Applications/Tools/HuginTools:$PATH"


This should enable the use of align_image_stack, enfuse, ... everything you need to run the shell scripts

Also, for the record you can run the shell scripts by cd'ing to the folder you have then and running
sh HDR_4331.sh

or whatever the file is named obviously
Title: Re: MLTools (OsX utility to manage ML install, files import and scripts conversions)
Post by: MortenSJ on January 17, 2016, 05:20:20 PM
I have little trouble using the MLP.

I've copied all my RAW files to my LACIE disk, but then i right click i don't get the Services tab. When i right click on a random folder on my desktop, i do however get the Services tab.

I only see the Services if i right click a folder from the Desktop. If i go inside that folder and right click i don't get Services.

Why is that?