Author Topic: Compiling Magic Lantern on a Macintosh  (Read 83624 times)

kichetof

  • Contributor
  • Senior
  • *****
  • Posts: 469
  • Take a beer and enjoy it!
Re: Compiling Magic Lantern on a Macintosh
« Reply #125 on: September 29, 2017, 07:30:07 AM »
Does the Mac run 32-bit binaries? If yes, can you try compiling a 32-bit GDB from source and place that one in PATH?

Until the next (2018? 2019?) release of MacOS it can run 32 bit without “compromise” https://developer.apple.com/news/?id=06282017a

@DeafEyeJedi could you copy past the commands and output from terminal, hard to read :) and could you edit manually contrib/qemu/install.sh and look at the end the grep -Po and if you don’t have ‘ggrep’ the quick script failed to edit it. On my MacOS High Sierra VM freshly installed no problem with all commands ( try to copy past manually)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #126 on: September 29, 2017, 07:32:37 AM »
What does this print?

Code: [Select]
arm-none-eabi-gdb -v

Best guess: if both the 64-bit and the 32-bit toolchain are in PATH, the first one will be used.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #127 on: September 29, 2017, 07:43:00 AM »
Deafeyejedi´s problem isn´t grep related. He doesn´t get passed make -j cause processor info isn´t found(ubuntu command). I changed install.sh to:

echo "1) Compile QEMU"
echo
echo "   cd `pwd`/${QEMU_NAME}"
echo "   ./configure --target-list=arm-softmmu --disable-docs --enable-sdl"
echo "   make -j $(sysctl -n hw.physicalcpu)"

then running configure in qemu-1.6.0:
Code: [Select]
./configure --target-list=arm-softmmu --disable-docs --enable-sdl
   make -j 2

Compiles a lot but stops here:
Code: [Select]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(generated-tracers.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(host-utils.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(cache-utils.o) has no symbols
  LINK  qemu-ga
  LINK  qemu-nbd
Undefined symbols for architecture x86_64:
  "_use_rt_clock", referenced from:
      _bdrv_acct_start in block.o
      _bdrv_acct_done in block.o
      _qemu_get_clock_ns in qemu-timer.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [qemu-nbd] Error 1
make: *** Waiting for unfinished jobs....
dans-MacBook-Pro:qemu-1.6.0 dan$

*Error above seems related to qemu itself:
https://github.com/espes/xqemu/issues/85

Do we need version qemu1.6?
Seems latest version is qemu2.10.0


Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #128 on: September 29, 2017, 08:09:36 AM »
Also note that dfort didn´t update the script with this line from kichetof although I´m not sure it´s needed:
Code: [Select]
sed -i '' 's/grep/ggrep/g' install.sh
Also tried path(my gcc in my path):
Code: [Select]
PATH=~/gcc-arm-none-eabi-4_8-2013q4/bin:$PATHand ran configure again but no difference.

kichetof

  • Contributor
  • Senior
  • *****
  • Posts: 469
  • Take a beer and enjoy it!
Re: Compiling Magic Lantern on a Macintosh
« Reply #129 on: September 29, 2017, 08:25:59 AM »
echo "   make -j $(sysctl -n hw.physicalcpu)"

Code: [Select]
echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"Works well on my MBP (Sierra and High Sierra), I don't have it with me to paste output

ohhh... update your local repo!! :)

Code: [Select]
sed -i '' 's/grep/ggrep/g' install.sh

That's the workarround, and don't forget:

Code: [Select]
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
Everywhere grep run with -P argument, we need to replace it with ggrep (not implemented in macOS grep)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #130 on: September 29, 2017, 08:30:40 AM »
dfort script needs updating to fit sed replacements to ggrep.

This is in install.sh, doesn´t play well with mac:
Code: [Select]
./configure --target-list=arm-softmmu --disable-docs --enable-sdl"
echo "   make -j`grep -c processor /proc/cpuinfo`"

Your line probably works for both linux and mac. Works on mac at least, just tried:
Code: [Select]
echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"
I also wonder if qemu needs update? I get the following error:
https://github.com/espes/xqemu/issues/85

Do we need version qemu1.6?
Seems latest version is qemu2.10.0

Are you running qemu fully working on your mac kichetof?


@dfort
Could you update bottom part of the script to:
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Code: [Select]
Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu
cd contrib/qemu/
sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
./install.sh
fi

Kichetof. Why even $GREP? Empty variable. This will do:
Code: [Select]
echo "   make -j`-c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"
And then this will also work:
Code: [Select]
echo "   make -j`sysctl -n hw.ncpu`"

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3751
Re: Compiling Magic Lantern on a Macintosh
« Reply #131 on: September 29, 2017, 08:50:16 AM »
Updated the script but didn't have time to test it yet. I trust you have.

It would be nice if your prompts could accept carriage returns. If you type "Y [return]" on the cross compiler installation you'll never get to the QEMU option.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

kichetof

  • Contributor
  • Senior
  • *****
  • Posts: 469
  • Take a beer and enjoy it!
Re: Compiling Magic Lantern on a Macintosh
« Reply #132 on: September 29, 2017, 09:12:26 AM »
@Danne 50D works well and 5D3 only show SD card test (32 bit bug)

Don't forget to update your repo to qemu branch

Code: [Select]
hg update qemu -C
And the install.sh look like that

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #133 on: September 29, 2017, 09:44:33 AM »
Oops. THanks Kichetof for the latest  :P.
Anyway. Stops at the same place as dfort
Code: [Select]
*** Installing dependencies for Mac...


*** WARNING: arm-none-eabi-gdb is not installed.
*** Downloading gcc-arm-embedded toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download the recommended 5.4-mac from gcc-arm-embedded.

*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up arm-none-eabi-gdb before continuing.
dans-MacBook-Pro:

I guess if we´re updating to hg update qemu -C which seems needed there´s no need for these?:
Code: [Select]
sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
?

As a matter of fact after running sed replace brew complains when running the install.sh scrip.
Code: [Select]
Error: No available formula with the name "ggrep"

kichetof

  • Contributor
  • Senior
  • *****
  • Posts: 469
  • Take a beer and enjoy it!
Re: Compiling Magic Lantern on a Macintosh
« Reply #134 on: September 29, 2017, 09:51:32 AM »
Oh I forgot, my workarround with sed grep to ggrep was solved with latest update from a1ex.

I guess if we´re updating to hg update qemu -C which seems needed there´s no need for these?:
Code: [Select]
sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
?

Yes, say good by to my workarround :)

Now we use $GREP and set it to grep for linux and ggrep for macOS

In the script you could try to do that before running install.sh:

Code: [Select]
echo 'export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH' >> ~/.bash_profile
. ~/.bash_profile

As a matter of fact after running sed replace brew complains when running the install.sh scrip.
Code: [Select]
Error: No available formula with the name "ggrep"

Code: [Select]
brew list grep
which ggrep
grep -V

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #135 on: September 29, 2017, 09:53:49 AM »
Great.
Latest take dfort:
Code: [Select]
Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
./install.sh
fi

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #136 on: September 29, 2017, 09:55:08 AM »
Just updated the script to use a 32-bit gdb for Mac (untested) and a little more verbosity.

It also allows using a 64-bit GDB if you don't mind the bug (but asks for confirmation).

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #137 on: September 29, 2017, 09:59:57 AM »
Tested latest:
Code: [Select]
Continue? [y/n] y

*** Installing dependencies for Mac...


*** WARNING: a valid arm-none-eabi-gdb could not be found.
*** Downloading a toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download macx_i386_gcc_arm_none_eabi_4.8.2 from:
    https://acroname.com/software/arm-eabi-gcc-toolchain-mac-osx-macos-x-32bit

--2017-09-29 09:58:48--  https://acroname.com/system/files/software/macx_i386_gcc_arm_none_eabi_4.8.2.tgz
Resolving acroname.com... 192.155.81.18
Connecting to acroname.com|192.155.81.18|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24839733 (24M) [application/x-gtar]
Saving to: ‘macx_i386_gcc_arm_none_eabi_4.8.2.tgz’

macx_i386_gcc_arm_n 100%[===================>]  23.69M  2.49MB/s    in 17s     

2017-09-29 09:59:07 (1.42 MB/s) - ‘macx_i386_gcc_arm_none_eabi_4.8.2.tgz’ saved [24839733/24839733]

/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up a valid arm-none-eabi-gdb before continuing.
dans-MBP:qemu dan$

Where is gcc versions downloaded? Before it went straight to my user folder but not now.
Noticed it downloads but is deleted before extracted.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #138 on: September 29, 2017, 10:33:06 AM »
Committed a fix - the printed PATH command was incorrect.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #139 on: September 29, 2017, 10:48:42 AM »
Did the trick. However, there´s no gdb to be found in the bin like with gcc 5.4:
Code: [Select]
macx_i386_gcc_ar 100%[==========>]  23.69M  2.32MB/s    in 14s     

2017-09-29 10:47:28 (1.72 MB/s) - ‘macx_i386_gcc_arm_none_eabi_4.8.2.tgz’ saved [24839733/24839733]

/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~/cortex/4.8.2/bin:$PATH

*** Please set up a valid arm-none-eabi-gdb before continuing.
dans-MBP:qemu dan$

While we´re at it I renamed my version gcc-arm-none-eabi-4_8-2013q4 to gcc-arm-none-eabi-4_8-2013q3 which has gdb and the follow up to that is:

Code: [Select]
*** Installing dependencies for Mac...

*** WARNING: 64-bit GDB is known not to work.

*** WARNING: a valid arm-none-eabi-gdb could not be found.
*** Downloading a toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download macx_i386_gcc_arm_none_eabi_4.8.2 from:
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~//bin:$PATH

*** WARNING: 64-bit GDB is known not to work.
Continue anyway? [y/N] y


*** Using GDB: /Users/dan/gcc-arm-none-eabi-4_8-2013q3/bin/arm-none-eabi-gdb
GNU gdb (GNU Tools for ARM Embedded Processors) 7.6.0.20131129-cvs

*** Using GCC: /Users/dan/gcc-arm-none-eabi-4_8-2013q3/bin/arm-none-eabi-gcc
/Users/dan/magic-lantern/contrib/qemu/install.sh: line 286: syntax error near unexpected token `&'
dans-MBP:qemu dan$

It´s this line that has to be changed:
Code: [Select]
arm-none-eabi-gcc -v |& grep "gcc version"I erase that line(why do we need it?

Script finishes and then:
Code: [Select]
cd /Users/dan/qemu/qemu-2.5.0
   ../configure_eos.sh
   make -j4
'
Succes!
Ending lines.
Code: [Select]
CC    arm-softmmu/target-arm/gdbstub.o
  CC    arm-softmmu/target-arm/psci.o
6 warnings generated.
  CC    arm-softmmu/target-arm/crypto_helper.o
  GEN   trace/generated-helpers.c
  CC    arm-softmmu/gdbstub-xml.o
  CC    arm-softmmu/trace/generated-helpers.o
  LINK  arm-softmmu/qemu-system-arm
  REZ   arm-softmmu/qemu-system-arm
  SETFILE arm-softmmu/qemu-system-arm
dans-MBP:qemu-2.5.0 dan$

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #140 on: September 29, 2017, 11:06:37 AM »
Got it, back to the 64-bit one for now...

If you are able to compile a 32-bit GDB, would be nice to use that one.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #141 on: September 29, 2017, 11:07:37 AM »
Got it working. Check updated post above :).

We should use this version:
gcc-arm-none-eabi-4_8-2013q4
https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2

erase or rewrite this line:
Code: [Select]
arm-none-eabi-gcc -v |& grep "gcc version"

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #142 on: September 29, 2017, 11:14:59 AM »
Already done - any issues with 5.4? (4.8 is also 64-bit on Mac)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #143 on: September 29, 2017, 11:18:51 AM »
I don´t see no reason for it not to work with 5.4. I´ve been focused on reported errors.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #144 on: September 29, 2017, 12:16:48 PM »
The issue why it´s not working is simply because after the new PATH is set it will not find gdb since it scripted like this:
Code: [Select]
arm-none-eabi-gdb -vOur Path i set so now it has to be this:
Code: [Select]
gdb
e.g:
Code: [Select]
function valid_arm_gdb {
    if ! gdb -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

At least it seems to work here to get past all the brakes.

Full script:

Code: [Select]
#!/usr/bin/env bash

set -e

QEMU_NAME=${QEMU_NAME:=qemu-2.5.0}
ML=${ML:=magic-lantern}
GREP=${GREP:=grep}

echo
echo "This will setup QEMU for emulating Magic Lantern."
echo "Thou shalt not be afraid of compiling stuff on Linux ;)"
echo -n "Continue? [y/n] "
read answer
if test "$answer" != "Y" -a "$answer" != "y"; then exit 0; fi
echo

function install_gdb {
    echo
    echo "*** Will download gcc-arm-none-eabi-5_4-2016q3 from:"
    echo "    https://developer.arm.com/open-source/gnu-toolchain/gnu-rm"
    echo

    TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
    DOWNLOAD=https://developer.arm.com/-/media/Files/downloads/gnu-rm/5_4-2016q3/
    UNTAR="tar -jxf"

    if [ $(uname) == "Darwin" ]; then
        TARBALL=gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
    else
        TARBALL=gcc-arm-none-eabi-5_4-2016q3-20160926-linux.tar.bz2
    fi

    if [ ! -f ~/$TOOLCHAIN/bin/arm-none-eabi-gdb ]; then
        cd ~
        wget -c $DOWNLOAD$TARBALL && $UNTAR $TARBALL && rm $TARBALL
        cd -
    else
        echo "*** Toolchain already installed in:"
        echo "    ~/$TOOLCHAIN"
        echo
    fi

    echo "*** Please add gcc binaries to your executable PATH:"
    echo "    PATH=~/$TOOLCHAIN/bin:\$PATH"
    echo
}


function valid_arm_gdb {
    if ! gdb -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

    if gdb -v | grep -q "host=x86_64"; then
        # 64-bit version - doesn't work
        # fixme: this may get printed more than once
        echo "*** WARNING: 64-bit GDB is known not to work."
        return 1
    fi

    # assume it's OK
    # todo: check version number
    return 0
}

function valid_arm_gcc {
    if ! gcc -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

    echo "#include <stdlib.h>" > arm-gcc-test.c
    if ! gcc -c arm-gcc-test.c; then
        echo "*** WARNING: your gcc is unable to compile a simple program."
        rm arm-gcc-test.c
        return 1
    fi

    rm arm-gcc-test.c
    return 0
}

if [ $(uname) == "Darwin" ]; then
    echo "*** Installing dependencies for Mac..."
    echo
    # fixme: don't these require sudo?
    # can we check whether they are already installed, as on Ubuntu?
    if ! xcode-select -p &> /dev/null; then
        xcode-select --install
    fi
    # brew is "The missing package manager for macOS"
    # https://brew.sh
    if ! brew -v &> /dev/null; then
        ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    fi
   
    packages="python wget mercurial xz grep pkg-config glib automake libtool pixman mtools"
    for pkg in $packages; do
        brew list $pkg &> /dev/null || brew install $pkg
    done
   
    GREP=ggrep
fi

if apt-get -v &> /dev/null; then
    # apt-based system?
    # install these packages, if not already
    # only request sudo if any of them is missing
    # instead of GTK (libgtk2.0-dev), you may prefer SDL (libsdl1.2-dev)
    packages="
        build-essential mercurial pkg-config libtool
        git libglib2.0-dev libpixman-1-dev zlib1g-dev
        libgtk2.0-dev xz-utils mtools netcat-openbsd
        python python-pip python-docutils"

    # if a valid arm-none-eabi-gcc/gdb is already in PATH, try to use that
    # otherwise, we'll try to install something
    if ! valid_arm_gdb || ! valid_arm_gcc; then
        echo "*** You do not seem to have an usable arm-none-eabi-gcc and/or gdb installed."
        echo "*** 64-bit GDB is known not to work, so you'll have to install a 32-bit one for now."
        echo
        echo "*** You have a few options:"
        echo
        echo "1 - Install gdb-arm-none-eabi:i386 and gcc-arm-none-eabi from Ubuntu repo (recommended)"
        echo "    This will install 32-bit binaries - will not work under Windows Subsystem for Linux."
        echo
        echo "2 - Download a 32-bit gcc-arm-embedded and install it without the package manager."
        echo "    Will be installed in your home directory; to move it, you must edit the Makefiles."
        echo "    This will install 32-bit binaries - will not work under Windows Subsystem for Linux."
        echo
        echo "3 - Install gdb-arm-none-eabi from Ubuntu repository (64-bit)"
        echo "    WARNING: this will not be able to run all our GDB scripts."
        echo
        if dpkg -l binutils-arm-none-eabi 2>/dev/null | grep -q '^.i'; then
            echo "4 - Remove Ubuntu toolchain and install the one from gcc-arm-embedded PPA (gcc 6.x)"
            echo "    This will:"
            echo "    - sudo apt-get remove gcc-arm-none-eabi gdb-arm-none-eabi \\"
            echo "           binutils-arm-none-eabi libnewlib-arm-none-eabi"
        else
            echo "4 - Install the toolchain from gcc-arm-embedded PPA (gcc 6.x)"
            echo "    This will:"
        fi
        echo "    - sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa"
        echo "    - install the gcc-arm-embedded package."
        echo "    WARNING: this will not be able to run all our GDB scripts."
        echo
        echo "5 - Manually install arm-none-eabi-gdb from https://launchpad.net/gcc-arm-embedded"
        echo "    or any other source, make sure it is in PATH, then run this script again."

        if ! gdb -v &> /dev/null; then
            echo
        else
            echo
            echo "6 - Just use the current 64-bit toolchain."
            echo "    WARNING: this will not be able to run all our GDB scripts."
        fi

        echo
        echo -n "Your choice? "
        read answer
        echo
        case $answer in
            1)
                # Ubuntu's 32-bit arm-none-eabi-gdb works fine
                # gcc-arm-none-eabi:i386 does not include libnewlib - Ubuntu bug?
                packages="$packages gdb-arm-none-eabi:i386 "
                packages="$packages gcc-arm-none-eabi libnewlib-arm-none-eabi"
                ;;
            2)
                # 32-bit gdb will be downloaded after installing these packages
                packages="$packages libc6:i386 libncurses5:i386"
                ;;
            3)
                # Ubuntu's 64-bit arm-none-eabi-gdb works... sort of
                # it's unable to run 5D3 1.1.3 GUI and maybe others
                packages="$packages gdb-arm-none-eabi:amd64"
                packages="$packages gcc-arm-none-eabi:amd64 libnewlib-arm-none-eabi:amd64"
                ;;
            4)
                # gcc-arm-embedded conflicts with gcc-arm-none-eabi
                # but the dependencies are not configured properly
                # so we have to fix the conflict manually...
                if dpkg -l binutils-arm-none-eabi 2>/dev/null | grep -q '^.i'; then
                    echo
                    echo "*** Please double-check - the following might remove additional packages!"
                    echo
                    sudo apt-get remove gcc-arm-none-eabi gdb-arm-none-eabi \
                         binutils-arm-none-eabi libnewlib-arm-none-eabi
                fi
                packages="$packages gcc-arm-embedded"
                echo
                echo "*** Adding the team-gcc-arm-embedded PPA repository..."
                echo "    sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa"
                echo
                sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa
                ;;
            5)
                # user will install arm-none-eabi-gdb and run the script again
                exit 0
                ;;
            6)
                # use the installed version, even though it's known not to work
                ;;
            *)
                # invalid choice
                exit 1
                ;;
        esac
    else
        echo "*** You have a valid ARM GCC/GDB already installed - using that one."
    fi

    echo "*** Checking dependencies for Ubuntu..."
    echo
    # https://wiki.debian.org/ListInstalledPackages
    # dpkg -l also returns packages that are not installed
    deps_installed=yes
    for package in $packages; do
        if ! dpkg -l $package 2>/dev/null | grep -q '^.i'; then
            echo Not installed: $package
            deps_installed=no
        fi
    done

    if [ "$deps_installed" == "no" ]; then
        echo
        echo "*** Installing dependencies for Ubuntu..."
        echo
        sudo apt-get update
        sudo apt-get install $packages
        echo
    fi

    deps_installed=yes
    for package in $packages; do
        if ! dpkg -l $package 2>/dev/null | grep -q '^.i'; then
            echo Not installed: $package
            deps_installed=no
        fi
    done

    if [ "$deps_installed" == "no" ]; then
        echo
        echo "*** Error: Ubuntu dependencies could not be installed."
        echo
        exit 1
    fi
fi

# all systems (including Mac, or Ubuntu if the installation from repositories failed)
# this one works on old systems as well, but it won't work under WSL
if ! valid_arm_gdb; then
    echo
    echo "*** WARNING: a valid arm-none-eabi-gdb could not be found."
    echo "*** Downloading a toolchain and installing it without the package manager."
    echo "*** Will be installed in your home directory (Makefile.user.default expects it there)."
    echo
    install_gdb
fi

# make sure we have a valid arm-none-eabi-gdb (regardless of operating system)
if ! valid_arm_gdb; then
    if ! arm-none-eabi-gdb -v &> /dev/null; then
        echo "*** Please set up a valid arm-none-eabi-gdb before continuing."
        exit 1
    else
        # valid_arm_gdb will print why the current one is not good
        echo -n "Continue anyway? [y/N] "
        read answer
        if test "$answer" != "Y" -a "$answer" != "y"; then exit 1; fi
        echo
    fi
fi

# same for arm-none-eabi-gcc
if ! valid_arm_gcc; then
    echo "*** Please set up a valid arm-none-eabi-gcc before continuing."
    exit 1
fi

echo
echo -n "*** Using GDB: "
command -v arm-none-eabi-gdb
arm-none-eabi-gdb -v | head -n1
echo
echo -n "*** Using GCC: "
command -v arm-none-eabi-gcc
arm-none-eabi-gcc -v 2>&1 | grep "gcc version"
echo

# install docutils (for compiling ML modules) and vncdotool (for test suite)
# only install if any of them is missing
pip2 list | grep docutils  || rst2html -h  > /dev/null || pip2 install docutils
pip2 list | grep vncdotool || vncdotool -h > /dev/null || pip2 install vncdotool

function die { echo "${1:-"Unknown Error"}" 1>&2 ; exit 1; }

pwd | grep $ML/contrib/qemu > /dev/null || die "error: we should be in $ML/contrib/qemu"

# go to the parent of magic-lantern folder
cd ../../..
ls | $GREP $ML > /dev/null || die "error: expecting to find $ML here"

mkdir -p qemu
cd qemu

echo
echo "*** Setting up QEMU in `pwd`..."
echo

# get qemu
wget -q --show-progress --progress=dot:giga -c http://wiki.qemu-project.org/download/$QEMU_NAME.tar.bz2
echo
tar jxf $QEMU_NAME.tar.bz2
echo

# initialize a git repo, to make it easy to track changes to QEMU source
cd $QEMU_NAME
if [ ! -d .git ]; then
  git init
  # git requires a valid email; if not setup, add one for this directory only
  git config user.email || git config user.email qemu@magiclantern.fm
  git add .
  git commit -q -m "$QEMU_NAME vanilla"
fi
cd ..

echo "Copying files..."

# copy our helper scripts
cp -r ../$ML/contrib/qemu/scripts/* .
chmod +x *.sh

# copy our testing scripts
mkdir -p tests
cp -r ../$ML/contrib/qemu/tests/* tests/
chmod +x tests/*.sh

# apply our patch
cd ${QEMU_NAME}
mkdir -p hw/eos
cp -r ../../$ML/contrib/qemu/eos/* hw/eos/
cp -r ../../$ML/src/backtrace.[ch] hw/eos/dbi/
patch -N -p1 < ../../$ML/contrib/qemu/$QEMU_NAME.patch
cd ..

# setup the card image
if [ ! -f "sd.img" ]; then
    echo "Setting up SD card image..."
    cp -v ../$ML/contrib/qemu/sd.img.xz .
    unxz -v sd.img.xz
else
    echo "SD image already exists, skipping."
fi

if [ ! -f "cf.img" ]; then
    echo "Setting up CF card image..."
    cp -v sd.img cf.img
else
    echo "CF image already exists, skipping."
fi

echo ""
echo "Next steps:"
echo "==========="
echo
echo "1) Compile QEMU"
echo
echo "   cd `pwd`/${QEMU_NAME}"
echo "   ../configure_eos.sh"
echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"
echo
echo "2) Grab a copy of the Canon firmware from your own camera"
echo "   (don't request one and don't share it online - it's copyrighted)"
echo
echo "   Look on your SD card - you should find ML/LOGS/ROM0.BIN and ROM1.BIN"
echo "   Copy them under your camera model's subdirectory, for example:"
echo "   `pwd`/60D/"
echo
echo "   For models that use a serial flash, you may have to dump its contents"
echo "   using the sf_dump module, then copy SFDATA.BIN as well."
echo
echo "3) Mount the included SD (or CF) image (you may use mount.sh)"
echo "   and install ML on it, as usual. The card image must be bootable as well."
echo
echo "   The included card image is bootable and contains a small autoexec.bin"
echo "   that runs on all DIGIC 4/5 cameras and prints some basic info."
echo
echo "   To create your own SD/CF image, you need to copy the raw contents"
echo "   of the entire card, not just one partition. For example:"
echo "   dd if=/dev/mmcblk0 of=sd.img"
echo
echo "4) Start emulation with:"
echo
echo "   cd `pwd`/"
echo "   ./run_canon_fw.sh 60D"
echo
echo "   This will recompile QEMU, but not ML."
echo
echo "   Note: Canon GUI emulation (menu navigation, no LiveView) only works on"
echo -n "   "; $GREP --color=never -oP "(?<=GUI_CAMS=\( ).*(?=\))" tests/run_tests.sh;
echo
echo "5) Tips & tricks:"
echo "   - to enable or disable the boot flag in ROM, use something like:"
echo "     ./run_canon_fw.sh 60D,firmware=\"boot=1\""
echo "   - to use multiple firmware versions, place the ROMs under e.g. 5D3/113/ROM*.BIN and run:"
echo "     ./run_canon_fw.sh 5D3,firmware=\"113;boot=1\""
echo "   - to show MMIO activity (registers) and interrupts, use:"
echo "     ./run_canon_fw.sh 60D -d io,int"
echo "   - to show the executed ASM code, step by step, use:"
echo "     ./run_canon_fw.sh 60D -d exec,int -singlestep"
echo "   - to trace debug messages and various functions in the firmware, use:"
echo "     ./run_canon_fw.sh 60D -s -S & arm-none-eabi-gdb -x 60D/debugmsg.gdb"
echo "   - if the above is too slow, compile the dm-spy-experiments branch "
echo "     with CONFIG_QEMU=y and CONFIG_DEBUG_INTERCEPT_STARTUP=y and try:"
echo "     ./run_canon_fw.sh 60D,firmware=\"boot=1\" -d io,int"
echo "   - some camera models require GDB patches to bypass tricky code sequences:"
echo "     ./run_canon_fw.sh 700D -s -S & arm-none-eabi-gdb -x 700D/patches.gdb"
echo "   - to trace all function calls and export them to IDA:"
echo "     ./run_canon_fw.sh 60D -d calls -singlestep"
echo "   - you may enable additional debug code (such as printing to QEMU console)"
echo "     by compiling ML with CONFIG_QEMU=y in your Makefile.user (also run make clean)."
echo "   - caveat: you cannot run autoexec.bin compiled with CONFIG_QEMU on the camera."

echo
echo "Enjoy!"


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #145 on: September 29, 2017, 12:26:21 PM »
arm-none-eabi-gdb is correct - double-check the path. It should be:

Code: [Select]
PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH


Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #146 on: September 29, 2017, 12:30:28 PM »
You´re right. I was calling system specific gdb. However. The issue is that the checking command doesn´t return that it finds gdb. Will check more.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #147 on: September 29, 2017, 01:18:40 PM »
Ok, been through a few rounds now but it seems to boil down to the path not being properly set:
Code: [Select]
    echo "   PATH=~/$TOOLCHAIN/bin:\$PATH"
Only echoing the path won´t work. The backslash breaks it as well.

Usually I would do:
Code: [Select]
export PATH=~/$TOOLCHAIN/bin/:$PATH
This works:
Code: [Select]
    echo "   PATH=~/$TOOLCHAIN/bin/:$PATH"
    PATH=~/$TOOLCHAIN/bin/:$PATH

And this:
Code: [Select]
    echo "   PATH=~/$TOOLCHAIN/bin/:$PATH"
    export PATH=~/$TOOLCHAIN/bin/:$PATH


Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6696
Re: Compiling Magic Lantern on a Macintosh
« Reply #148 on: September 29, 2017, 02:51:37 PM »
Here is dfort script. Should get you right into qemu after all else is installed:
Code: [Select]
cat <<'EOF' > mac_ml.sh
#!/bin/sh

#  mac_ml.sh
#
#  Script to almost automatically install
#  and configure a Magic Lantern development
#  environment on a Macintosh.
#
#  Reference:
#  http://magiclantern.fm/forum/index.php?topic=16012.0
#
#  Daniel Fort (dfort) - 2015-10-15
#  Modified 2016-03-17 - took out ugly gcc hack
#  Modified 2017-06-27 - added mingw-w64 for cross compiling Windows binaries
#  Modified 2017-06-28 - check if magic-lantern already exists
#  Modified 2017-07-12 - write a Makefile.user file
#  Modified 2017-09-23 - made mingw-w64 (Windows cross compiler) optional
#  Modified 2017-09-24 - added interactive options for mingw-w64 and QEMU
#  Modified 2017-09-26 - QEMU install.sh now installs dependencies so no need to do it here
#                                   now using gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
#
#  Thanks to Danne and kichetof and of course a1ex for their input

if ! test -d /usr/local; then
  echo "/usr/local directory not found."
  echo "Follow the steps at the beginning of"
  echo "tutorial to create /usr/local and"
  echo "then re-run this script."
  exit 1
fi

echo "/usr/local found -- "
echo "continuing with installation."

cd ~
if [ ! -d "/Library/Developer/CommandLineTools" ]; then xcode-select --install; fi
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install python wget mercurial

#
# Uncomment the following line if you want to cross compile Windows binaries.
# brew install mingw-w64
#

pip2 install docutils

brew tap homebrew/dupes; brew install homebrew/versions/gcc5

cd ~ && \
    [ ! -f gcc-arm-none-eabi-4_8-2013q4 ] &&
    wget -c https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2

if ! test -d ~/magic-lantern; then
    hg clone https://bitbucket.org/hudson/magic-lantern
    cd magic-lantern
    hg update unified
    cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT
    cd ~
fi

#OPTIONAL 1
clear
read -p $(tput bold)"Would you also like to install the windows cross compiler mingw-w64?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
clear
read -p $(tput bold)"Would you also like to compile QEMU?$(tput setaf 1)
Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
clear
echo "
installing mingw-w64"
brew install mingw-w64 || :
#compiling qemu
clear
echo "
compiling qemu"
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
export PATH=~/$TOOLCHAIN/bin/:$PATH
./install.sh
else
clear
echo "
installing mingw-w64"
brew install mingw-w64 || :
fi
else
#OPTIONAL 2
clear
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
export PATH=~/$TOOLCHAIN/bin/:$PATH
./install.sh
fi
fi

cd ~
exit 0
EOF
bash mac_ml.sh && rm mac_ml.sh

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12422
Re: Compiling Magic Lantern on a Macintosh
« Reply #149 on: September 29, 2017, 04:01:10 PM »
Of course, echoing the path won't work, you need to copy/paste that command and run it outside the script (or place it into your .profile if you want). And I don't want to echo the entire path, that's the point of "\$".

The script itself is not able to set the path outside it (it won't change the environment of its parent shell). That's why I'm echoing the command.

The script itself has no need to run arm-none-eabi-gdb/gcc - it only makes sure *you* are going to be able to run them, as described in the QEMU guide.

See my screenshots above and this animation.