ML Universal installer

Started by nanomad, October 26, 2012, 03:32:47 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

nanomad

Since the # of people that can't install install ML properly is amazing, I'd like to know if anyone is interested in collaborating with me on an universal MagicLantern installer.
This is inspired by the latest release of Pel's EOSCard utility (which sadly works on Windows only)

Goals:
- Runs on Windows (7+), Linux (.deb and source code install) and OSX
- Has a consistent look & feel across the three OS
- Should be able to format and make the card bootable
- Can uninstall ML  (including making the card non-bootable)
- Can install either the latest stable version (default) or one of the nightly
- Allows the user to manually select a .zip file to install (for new ports or other stuff)

What do I need:
- a developer with a Mac PC
- (not mandatory) a developer with a good knowledge of windows (low level disk access is very important, Pel?)
- a suggestion on what programming language (I think C++ is the only choice here) and toolkit  to use (QT probably)
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

g3gg0

what about C#/.NET as cross-plattform language/runtime environment?

windows has support for that by MS and linux/max through the mono project
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Pelican

EOS 7D Mark II, EOS 7D, EOS 5, EOS 100 + lenses (10mm to 300mm), 600EX, 550EX, YN600EX x 3
EOScard, EOS DSLR firmwares, ARMu, NiControl, etc.: http://pel.hu/down

nanomad

Great!
How does the disk access work in windows? On linux (and OSX too) it's as simple as opening /dev/something
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

Pelican

Not like that. I'm using RAWdiskaccess component for Delphi.
EOS 7D Mark II, EOS 7D, EOS 5, EOS 100 + lenses (10mm to 300mm), 600EX, 550EX, YN600EX x 3
EOScard, EOS DSLR firmwares, ARMu, NiControl, etc.: http://pel.hu/down

Michael Zöller

g3gg0, I had only bad experiences using mono for gui stuff on linux. Did you ever use it successfully? I suggest Qt, too. I can do the mac part.
neoluxx.de
EOS 5D Mark II | EOS 600D | EF 24-70mm f/2.8 | Tascam DR-40

nanomad

Great, I'll download the QT Windows SDK and set up a repository so we can work on it
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

jplxpto

nanomad: what are the low-level operations that we need?

I think this should be implemented in a language easily portable.
The python was not enough? In this case C # does not seem a good option!


eduperez

Could this be extended to support other projects (400plus, for example), please? AFAIK, the process is exactly the same, only difference is what files are copied to the card. Thanks!

g3gg0

i have little experience on linux with .net. it worked all like a charm.
just installed mono and all windows forms and non-native code worked as expected.
native code (directly calling a DLLs) was intercepted (exception) and executed in managed code.
except DirectX (using SlimDX) - that didnt work of course :)

and i have experience with Qt on Windows+Linux and its too annoying for me.
you need a huge SDK that needed some manual hacks on windows.
then the usage of Qt Designer and integrating in some IDE takes it time.
and then i had the struggle with the right libraries being delivered/available.

so i switched my next tools to managed code in .NET.
C# is (imho) a much cleaner language than C++.
and if you build a eg .NET 3.0 application, there is no fuzz with libraries.

i dont see any advantage in using C or C++ if it is not about number crunching or low-latency code.
especially when you build a UI application, C#/.NET will save overhead as e.g. the free Visual Studio Express
will provide you all features in a nice IDE.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

g3gg0

accessing devices directly is in C# as complex as in C++.
for accessing i would create a DiskIO interface that decides on which platform we run and
instanciates the correct class that only wraps to the OS specific calls.
this is also necessary in C/C++

on windows i would use P/Invoke like that to et the drives:
http://www.pinvoke.net/default.aspx/kernel32.QueryDosDevice
and then open PhysicalDriveX for accessing like in e.g.
http://www.pinvoke.net/default.aspx/kernel32/DeviceIoControl.html

on linux i would try all /dev/sd[0-255], check their size etc.
all that via fopen/fseek/ftell

not sure about MacOS X
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

a1ex

For Linux and Mac, this autodetection routine seems to work:


if [[ $OSTYPE == darwin* ]]; then   # Existing: OS X
  dev=/dev/$(diskutil list | grep EOS_DIGITAL | awk '{print $6}' )
  echo $dev
  diskutil unmount $dev
elif [[ $OSTYPE == linux* ]]; then   # New: Linux
  dev=$(mount | grep EOS_DIGITAL | awk '{print $1}' )
  if [ "x$dev" = "x" ]; then
    echo "The EOS_DIGITAL card should be mounted before running the script."
    exit
  fi
  echo "Found $dev"
  if [ $(id -u) != 0 ]; then
    echo "dd operations require you to have access to the device, run script as root to be sure"
    exit
  fi
  umount $dev
fi

nanomad

Well, I was on the train with no internet access and I figured I could write something in python.
Here's the Windows IO manager


class W32IOManager(IOManager):

    def is_removable(self, drive_num):
        return win32file.GetDriveType(self.get_mount_point(drive_num)) == win32file.DRIVE_REMOVABLE

    def get_mount_point(self, drive_num):
        return r'%c:\\' % self.drive_letter(drive_num)
   
    def drive_letter(self, drive_num):
        return chr(ord('A')+drive_num)

    def enum_removable_drives(self):
            drivebits = win32file.GetLogicalDrives()
            return [d for d in range(1,26) if drivebits&(1<<d) and self.is_removable(d)]

    def get_raw_device(self, drive_num):
        return r'\\.\%c:' % self.drive_letter(drive_num)

    def get_label(self, drive_num):
        return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[0]

    def get_fs_type(self, drive_num):
        return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[-1]
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

jplxpto

Quote from: nanomad on October 30, 2012, 04:34:36 PM
Well, I was on the train with no internet access and I figured I could write something in python.
Here's the Windows IO manager


class W32IOManager(IOManager):

    def is_removable(self, drive_num):
        return win32file.GetDriveType(self.get_mount_point(drive_num)) == win32file.DRIVE_REMOVABLE

    def get_mount_point(self, drive_num):
        return r'%c:\\' % self.drive_letter(drive_num)
   
    def drive_letter(self, drive_num):
        return chr(ord('A')+drive_num)

    def enum_removable_drives(self):
            drivebits = win32file.GetLogicalDrives()
            return [d for d in range(1,26) if drivebits&(1<<d) and self.is_removable(d)]

    def get_raw_device(self, drive_num):
        return r'\\.\%c:' % self.drive_letter(drive_num)

    def get_label(self, drive_num):
        return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[0]

    def get_fs_type(self, drive_num):
        return win32api.GetVolumeInformation(self.get_mount_point(drive_num))[-1]


Now we can implement up something similar for Linux and Mac

jplxpto


nanomad

The Linux IOManager, thanks to the awesomeness of hal and dbus ;D


import IOManager
import dbus


class LinuxIOManager(IOManager):
    def __init__(self):
        self.bus = dbus.SystemBus()
        self.hal_manager = self.bus.get_object("org.freedesktop.Hal",
                                                            "/org/freedesktop/Hal/Manager")
        self.hal_iface = dbus.Interface(self.hal_manager, "org.freedesktop.Hal.Manager")

    def get_device_with_udi(self,  udi):
        obj = self.bus.get_object('org.freedesktop.Hal', udi)
        if obj != None:
            device = dbus.Interface(obj, 'org.freedesktop.Hal.Device')
            return device
        return None

    def get_mount_point(self, drive_num):
        return drive_num.getProperty('volume.mount_point')

    def enum_removable_drives(self):
        udis = self.hal_iface.FindDeviceByCapability('volume')
        result = []
        for udi in udis:
            # get a hal object for the volume referenced by udi
            volume = self.get_device_with_udi(udi)
            parent_uri = volume.GetProperty('info.parent')
            parent = self.get_device_with_udi(parent_uri)
            if(parent.GetProperty('storage.removable')):
                result = result + [volume]
        return result

    def get_raw_device(self, drive_num):
        return drive_num.getProperty('block.device')

    def get_label(self, drive_num):
        return drive_num.getProperty('volume.label')

    def get_fs_type(self, drive_num):
        return drive_num.getProperty('volume.fsversion')


EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

nanomad

And the IOManager itself


import os


class IOManager(object):

    def __init__(self):
        pass

    def get_mount_point(self, drive_num):
        raise NotImplementedError("IOManager is an interface")

    def enum_removable_drives(self):
        raise NotImplementedError("IOManager is an interface")

    def get_raw_device(self, drive_num):
        raise NotImplementedError("IOManager is an interface")

    def get_label(self, drive_num):
        raise NotImplementedError("IOManager is an interface")

    def get_fs_type(self, drive_num):
        raise NotImplementedError("IOManager is an interface")

    def get_offset(self,  fs,  is_bootflag):
        offsets = {'FAT16':   [43,  64],
                        'FAT32':  [71, 92],
                        'EXFAT':  [130, 122]
                    }
        if(is_bootflag):
            return offsets[fs][1]
        else:
            return offsets[fs][0]

    def write_to_disk(self,  drive_num,  what,  sector=0, offset=0):
        raw_device = self.get_raw_device(drive_num)
        fd = open(raw_device,  'rb+')
        fd.seek(sector * 512,  os.SEEK_SET)
        olddata = fd.read(512)
        newdata = olddata[0:offset] + what + olddata[(offset + len(what)):]
        fd.seek(sector * 512,  os.SEEK_SET)
        fd.write(newdata)
        fd.close()

    def write_bootflag(self,  drive_num,  sector):
        fs = self.get_fs_type(drive_num)
        offset = self.get_offset(fs,  bootflag=True)
        self.write_to_disk(drive_num,  "BOOTDISK",  sector,  offset)

    def write_eosdevelop(self,  drive_num,  sector):
        fs = self.get_fs_type(drive_num)
        offset = self.get_offset(fs,  bootflag=False)
        self.write_to_disk(drive_num,  "EOS_DEVELOP",  sector,  offset)

    def write_vbr_checksum(self,  drive_num):
        pass

    def write_flags(self,  drive_num):
        fs = self.get_fs_type(drive_num)
        self.write_bootflag(drive_num,  0)
        self.write_eosdevelop(drive_num,  0)
        if(fs == "EXFAT"):
            self.write_bootflag(drive_num,  12)
            self.write_eosdevelop(drive_num,  12)
            self.write_vbr_checksum(drive_num)


Now, who's willing to write the OSXIOManager class?
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

nanomad

Source code available under contrib/ml-installer :)
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

scrax

Quote from: nanomad on October 31, 2012, 02:52:37 PM
The Linux IOManager, thanks to the awesomeness of hal and dbus ;D

@nanomad
I can't even understand the code you posted since I have no idea how phyton works but I can do test or learn a little maybe, do you think it will be possible to use the linux IO manager for osx also?
I can compile dbus on osx but i've not found a way to have also dbus-phyton (not sure that this is correct even), and also no hal for osx.

reading about hal there is an alternative that i'll try to add to MLTools.

I want to do something to help out but no idea what, I can learn how to make something with bash script using osx command but no idea if that can be "ported" to a mac IO manager, but if it's possible why not use the already working make_bootable?I suppose so that it's not possible.
With MLTools I use it to install ML with MLTools but is a bundeld osx app that detects the card and starts the script, that's is what i want to avoid and that could also help if that is confirmed
unzip "$MLrelease" -d "$EOS_CARD"
bash make_bootable.sh
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

nanomad

Yeah, we can use the make_bootable script without too many issues. The python version currently works on windows and linux without modifications (the full version is not in the repo since it has some bugs).

I'm still looking for someone who can design the UI
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

scrax

What will it do, first?
at start it will check for a card and then just copies bundled ML release to the card after making it bootable?
Or will it have more options like macboot or eosutil?
I'll suggest to keep it as simple as possible, something like just two button (or three with "cancel/close" too):

"Install ML 2.4"
"Install custom zip" (for nightly, forks and other project like 400Dplus support - zip file only, or also bin and fir?, i think better zip only?)
"Close ML Installer"

That way users can, with minimal effort, install ML quickly (just two click).

Maybe we can think about posting on headline news an advice like "MLTeam is looking for UI designer for ML installer"

For non universal installer, after stripping out MLTools this now works for Osx:
 
I think that the only thing I can change is text and maybe add an icon to the description part.

here the download for 2.3 osx MLInstaller: https://dl.dropbox.com/u/123918/MagicLantern/MLTools/MLInstaller.0.1.app.zip

this is the script:

#!/bin/bash -l

OSX_INIT(){
MLrelease=magiclantern-v2.3.550D.60D.600D.50D.500D.5D2.zip
EOS_CARD=/Volumes/EOS_DIGITAL/

#used for gui interaction
CD="CocoaDialog.app/Contents/MacOS/CocoaDialog"
#ML installer window
rv=`$CD msgbox --no-newline \
--text "Magic Lantern Installer" \
--informative-text "Magic Lantern v2.3 - 13/08/2012" \
--button1 "Install ML" --button2 "Install custom zip" --button3 "Cancel"`
}

startINST() {
if [ "$rv" == "1" ]; then
checkCARD
elif [ "$rv" == "2" ]; then
selectZIP
fi
}

checkCARD() {
if [ $(diskutil list $EOS_CARD >>/dev/null) ]; then
installML
else
$CD bubble --title "MLInstaller error" --text "Card not found, have you formatted it on card?"
fi
}

installML() {
unzip "$MLrelease" -d "$EOS_CARD"
bash make_bootable.sh
}

selectZIP() { # show a window for selecting files
rv=`$CD fileselect \
--text "Select another release release package from disk (.zip only)" \
--with-directory $HOME/ \
--with-extensions .ZIP .zip/* \
--select-directories \
--select-multiple*/`
if [ -f "$rv" ]; then
MLrelease="$rv"
installML
else
$CD bubble --title "MLInstaller error" --text "No files chosen"
fi
}

############## START SCRIPT PROCESS ##############
OSX_INIT
startINST
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-