Since the # of people that can't install install ML properly is amazing, I'd like to know if anyone is interested in collaborating with me on an universal MagicLantern installer.
This is inspired by the latest release of Pel's EOSCard utility (which sadly works on Windows only)
Goals:
- Runs on Windows (7+), Linux (.deb and source code install) and OSX
- Has a consistent look & feel across the three OS
- Should be able to format and make the card bootable
- Can uninstall ML (including making the card non-bootable)
- Can install either the latest stable version (default) or one of the nightly
- Allows the user to manually select a .zip file to install (for new ports or other stuff)
What do I need:
- a developer with a Mac PC
- (not mandatory) a developer with a good knowledge of windows (low level disk access is very important, Pel?)
- a suggestion on what programming language (I think C++ is the only choice here) and toolkit to use (QT probably)
what about C#/.NET as cross-plattform language/runtime environment?
windows has support for that by MS and linux/max through the mono project
OK, I'm in. :)
Great!
How does the disk access work in windows? On linux (and OSX too) it's as simple as opening /dev/something
Not like that. I'm using RAWdiskaccess component for Delphi.
g3gg0, I had only bad experiences using mono for gui stuff on linux. Did you ever use it successfully? I suggest Qt, too. I can do the mac part.
Great, I'll download the QT Windows SDK and set up a repository so we can work on it
nanomad: what are the low-level operations that we need?
I think this should be implemented in a language easily portable.
The python was not enough? In this case C # does not seem a good option!
Could this be extended to support other projects (400plus, for example), please? AFAIK, the process is exactly the same, only difference is what files are copied to the card. Thanks!
i have little experience on linux with .net. it worked all like a charm.
just installed mono and all windows forms and non-native code worked as expected.
native code (directly calling a DLLs) was intercepted (exception) and executed in managed code.
except DirectX (using SlimDX) - that didnt work of course :)
and i have experience with Qt on Windows+Linux and its too annoying for me.
you need a huge SDK that needed some manual hacks on windows.
then the usage of Qt Designer and integrating in some IDE takes it time.
and then i had the struggle with the right libraries being delivered/available.
so i switched my next tools to managed code in .NET.
C# is (imho) a much cleaner language than C++.
and if you build a eg .NET 3.0 application, there is no fuzz with libraries.
i dont see any advantage in using C or C++ if it is not about number crunching or low-latency code.
especially when you build a UI application, C#/.NET will save overhead as e.g. the free Visual Studio Express
will provide you all features in a nice IDE.
accessing devices directly is in C# as complex as in C++.
for accessing i would create a DiskIO interface that decides on which platform we run and
instanciates the correct class that only wraps to the OS specific calls.
this is also necessary in C/C++
on windows i would use P/Invoke like that to et the drives:
http://www.pinvoke.net/default.aspx/kernel32.QueryDosDevice
and then open PhysicalDriveX for accessing like in e.g.
http://www.pinvoke.net/default.aspx/kernel32/DeviceIoControl.html
on linux i would try all /dev/sd[0-255], check their size etc.
all that via fopen/fseek/ftell
not sure about MacOS X
For Linux and Mac, this autodetection routine seems to work:
if [[ $OSTYPE == darwin* ]]; then # Existing: OS X
dev=/dev/$(diskutil list | grep EOS_DIGITAL | awk '{print $6}' )
echo $dev
diskutil unmount $dev
elif [[ $OSTYPE == linux* ]]; then # New: Linux
dev=$(mount | grep EOS_DIGITAL | awk '{print $1}' )
if [ "x$dev" = "x" ]; then
echo "The EOS_DIGITAL card should be mounted before running the script."
exit
fi
echo "Found $dev"
if [ $(id -u) != 0 ]; then
echo "dd operations require you to have access to the device, run script as root to be sure"
exit
fi
umount $dev
fi
Well, I was on the train with no internet access and I figured I could write something in python.
Here's the Windows IO manager
class W32IOManager(IOManager):
def is_removable(self, drive_num):
return win32file.GetDriveType(self.get_mount_point(drive_num)) == win32file.DRIVE_REMOVABLE
def get_mount_point(self, drive_num):
return r'%c:\\' % self.drive_letter(drive_num)
def drive_letter(self, drive_num):
return chr(ord('A')+drive_num)
def enum_removable_drives(self):
drivebits = win32file.GetLogicalDrives()
return [d for d in range(1,26) if drivebits&(1<<d) and self.is_removable(d)]
def get_raw_device(self, drive_num):
return r'\\.\%c:' % self.drive_letter(drive_num)
def get_label(self, drive_num):
return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[0]
def get_fs_type(self, drive_num):
return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[-1]
Quote from: nanomad on October 30, 2012, 04:34:36 PM
Well, I was on the train with no internet access and I figured I could write something in python.
Here's the Windows IO manager
class W32IOManager(IOManager):
def is_removable(self, drive_num):
return win32file.GetDriveType(self.get_mount_point(drive_num)) == win32file.DRIVE_REMOVABLE
def get_mount_point(self, drive_num):
return r'%c:\\' % self.drive_letter(drive_num)
def drive_letter(self, drive_num):
return chr(ord('A')+drive_num)
def enum_removable_drives(self):
drivebits = win32file.GetLogicalDrives()
return [d for d in range(1,26) if drivebits&(1<<d) and self.is_removable(d)]
def get_raw_device(self, drive_num):
return r'\\.\%c:' % self.drive_letter(drive_num)
def get_label(self, drive_num):
return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[0]
def get_fs_type(self, drive_num):
return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[-1]
Now we can implement up something similar for Linux and Mac
I think there is another possibility: http://www.mingw.org/
The Linux IOManager, thanks to the awesomeness of hal and dbus ;D
import IOManager
import dbus
class LinuxIOManager(IOManager):
def __init__(self):
self.bus = dbus.SystemBus()
self.hal_manager = self.bus.get_object("org.freedesktop.Hal",
"/org/freedesktop/Hal/Manager")
self.hal_iface = dbus.Interface(self.hal_manager, "org.freedesktop.Hal.Manager")
def get_device_with_udi(self, udi):
obj = self.bus.get_object('org.freedesktop.Hal', udi)
if obj != None:
device = dbus.Interface(obj, 'org.freedesktop.Hal.Device')
return device
return None
def get_mount_point(self, drive_num):
return drive_num.getProperty('volume.mount_point')
def enum_removable_drives(self):
udis = self.hal_iface.FindDeviceByCapability('volume')
result = []
for udi in udis:
# get a hal object for the volume referenced by udi
volume = self.get_device_with_udi(udi)
parent_uri = volume.GetProperty('info.parent')
parent = self.get_device_with_udi(parent_uri)
if(parent.GetProperty('storage.removable')):
result = result + [volume]
return result
def get_raw_device(self, drive_num):
return drive_num.getProperty('block.device')
def get_label(self, drive_num):
return drive_num.getProperty('volume.label')
def get_fs_type(self, drive_num):
return drive_num.getProperty('volume.fsversion')
And the IOManager itself
import os
class IOManager(object):
def __init__(self):
pass
def get_mount_point(self, drive_num):
raise NotImplementedError("IOManager is an interface")
def enum_removable_drives(self):
raise NotImplementedError("IOManager is an interface")
def get_raw_device(self, drive_num):
raise NotImplementedError("IOManager is an interface")
def get_label(self, drive_num):
raise NotImplementedError("IOManager is an interface")
def get_fs_type(self, drive_num):
raise NotImplementedError("IOManager is an interface")
def get_offset(self, fs, is_bootflag):
offsets = {'FAT16': [43, 64],
'FAT32': [71, 92],
'EXFAT': [130, 122]
}
if(is_bootflag):
return offsets[fs][1]
else:
return offsets[fs][0]
def write_to_disk(self, drive_num, what, sector=0, offset=0):
raw_device = self.get_raw_device(drive_num)
fd = open(raw_device, 'rb+')
fd.seek(sector * 512, os.SEEK_SET)
olddata = fd.read(512)
newdata = olddata[0:offset] + what + olddata[(offset + len(what)):]
fd.seek(sector * 512, os.SEEK_SET)
fd.write(newdata)
fd.close()
def write_bootflag(self, drive_num, sector):
fs = self.get_fs_type(drive_num)
offset = self.get_offset(fs, bootflag=True)
self.write_to_disk(drive_num, "BOOTDISK", sector, offset)
def write_eosdevelop(self, drive_num, sector):
fs = self.get_fs_type(drive_num)
offset = self.get_offset(fs, bootflag=False)
self.write_to_disk(drive_num, "EOS_DEVELOP", sector, offset)
def write_vbr_checksum(self, drive_num):
pass
def write_flags(self, drive_num):
fs = self.get_fs_type(drive_num)
self.write_bootflag(drive_num, 0)
self.write_eosdevelop(drive_num, 0)
if(fs == "EXFAT"):
self.write_bootflag(drive_num, 12)
self.write_eosdevelop(drive_num, 12)
self.write_vbr_checksum(drive_num)
Now, who's willing to write the OSXIOManager class?
Source code available under contrib/ml-installer :)
Quote from: nanomad on October 31, 2012, 02:52:37 PM
The Linux IOManager, thanks to the awesomeness of hal and dbus ;D
@nanomad
I can't even understand the code you posted since I have no idea how phyton works but I can do test or learn a little maybe, do you think it will be possible to use the linux IO manager for osx also?
I can compile dbus on osx but i've not found a way to have also dbus-phyton (not sure that this is correct even), and also no hal for osx.
reading about hal there is (http://apple.stackexchange.com/questions/46951/is-there-a-mac-equivalent-for-udev-folder-on-linux) an alternative that i'll try to add to MLTools.
I want to do something to help out but no idea what, I can learn how to make something with bash script using osx command but no idea if that can be "ported" to a mac IO manager, but if it's possible why not use the already working make_bootable?I suppose so that it's not possible.
With MLTools I use it to install ML with MLTools but is a bundeld osx app that detects the card and starts the script, that's is what i want to avoid and that could also help if that is confirmed (http://www.magiclantern.fm/forum/index.php?topic=3638.0)
unzip "$MLrelease" -d "$EOS_CARD"
bash make_bootable.sh
Yeah, we can use the make_bootable script without too many issues. The python version currently works on windows and linux without modifications (the full version is not in the repo since it has some bugs).
I'm still looking for someone who can design the UI
What will it do, first?
at start it will check for a card and then just copies bundled ML release to the card after making it bootable?
Or will it have more options like macboot or eosutil?
I'll suggest to keep it as simple as possible, something like just two button (or three with "cancel/close" too):
"Install ML 2.4"
"Install custom zip" (for nightly, forks and other project like 400Dplus support - zip file only, or also bin and fir?, i think better zip only?)
"Close ML Installer"
That way users can, with minimal effort, install ML quickly (just two click).
Maybe we can think about posting on headline news an advice like "MLTeam is looking for UI designer for ML installer"
For non universal installer, after stripping out MLTools this now works for Osx:
(https://dl.dropbox.com/u/123918/MagicLantern/MLTools/img/MLInstaller_Osx.png)
I think that the only thing I can change is text and maybe add an icon to the description part.
here the download for 2.3 osx MLInstaller: https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLInstaller.0.1.app.zip
this is the script:
#!/bin/bash -l
OSX_INIT(){
MLrelease=magiclantern-v2.3.550D.60D.600D.50D.500D.5D2.zip
EOS_CARD=/Volumes/EOS_DIGITAL/
#used for gui interaction
CD="CocoaDialog.app/Contents/MacOS/CocoaDialog"
#ML installer window
rv=`$CD msgbox --no-newline \
--text "Magic Lantern Installer" \
--informative-text "Magic Lantern v2.3 - 13/08/2012" \
--button1 "Install ML" --button2 "Install custom zip" --button3 "Cancel"`
}
startINST() {
if [ "$rv" == "1" ]; then
checkCARD
elif [ "$rv" == "2" ]; then
selectZIP
fi
}
checkCARD() {
if [ $(diskutil list $EOS_CARD >>/dev/null) ]; then
installML
else
$CD bubble --title "MLInstaller error" --text "Card not found, have you formatted it on card?"
fi
}
installML() {
unzip "$MLrelease" -d "$EOS_CARD"
bash make_bootable.sh
}
selectZIP() { # show a window for selecting files
rv=`$CD fileselect \
--text "Select another release release package from disk (.zip only)" \
--with-directory $HOME/ \
--with-extensions .ZIP .zip/* \
--select-directories \
--select-multiple*/`
if [ -f "$rv" ]; then
MLrelease="$rv"
installML
else
$CD bubble --title "MLInstaller error" --text "No files chosen"
fi
}
############## START SCRIPT PROCESS ##############
OSX_INIT
startINST