Compiling Magic Lantern on a Macintosh

[kichetof]

Does the Mac run 32-bit binaries? If yes, can you try compiling a 32-bit GDB from source and place that one in PATH?

Until the next (2018? 2019?) release of MacOS it can run 32 bit without "compromise" https://developer.apple.com/news/?id=06282017a

@DeafEyeJedi could you copy past the commands and output from terminal, hard to read and could you edit manually contrib/qemu/install.sh and look at the end the grep -Po and if you don't have 'ggrep' the quick script failed to edit it. On my MacOS High Sierra VM freshly installed no problem with all commands ( try to copy past manually)

[a1ex]

What does this print?

Code Select Expand


arm-none-eabi-gdb -v


Best guess: if both the 64-bit and the 32-bit toolchain are in PATH, the first one will be used.

[Danne]

Deafeyejedi´s problem isn´t grep related. He doesn´t get passed make -j cause processor info isn´t found(ubuntu command). I changed install.sh to:

echo "1) Compile QEMU"
echo
echo "   cd `pwd`/${QEMU_NAME}"
echo "   ./configure --target-list=arm-softmmu --disable-docs --enable-sdl"
echo "   make -j $(sysctl -n hw.physicalcpu)"

then running configure in qemu-1.6.0:

Code Select Expand

./configure --target-list=arm-softmmu --disable-docs --enable-sdl
   make -j 2

Compiles a lot but stops here:

Code Select Expand

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(generated-tracers.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(host-utils.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libqemuutil.a(cache-utils.o) has no symbols
  LINK  qemu-ga
  LINK  qemu-nbd
Undefined symbols for architecture x86_64:
  "_use_rt_clock", referenced from:
      _bdrv_acct_start in block.o
      _bdrv_acct_done in block.o
      _qemu_get_clock_ns in qemu-timer.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [qemu-nbd] Error 1
make: *** Waiting for unfinished jobs....
dans-MacBook-Pro:qemu-1.6.0 dan$

*Error above seems related to qemu itself:
https://github.com/espes/xqemu/issues/85

Do we need version qemu1.6?
Seems latest version is qemu2.10.0

[Danne]

Also note that dfort didn´t update the script with this line from kichetof although I´m not sure it´s needed:

Code Select Expand

sed -i '' 's/grep/ggrep/g' install.sh

Also tried path(my gcc in my path):

Code Select Expand

PATH=~/gcc-arm-none-eabi-4_8-2013q4/bin:$PATH
and ran configure again but no difference.

[kichetof]

echo "   make -j $(sysctl -n hw.physicalcpu)"

Code Select Expand

echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"
Works well on my MBP (Sierra and High Sierra), I don't have it with me to paste output

ohhh... update your local repo!!

Code Select Expand

sed -i '' 's/grep/ggrep/g' install.sh

That's the workarround, and don't forget:

Code Select Expand

sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh

Everywhere grep run with -P argument, we need to replace it with ggrep (not implemented in macOS grep)

[Danne]

dfort script needs updating to fit sed replacements to ggrep.

This is in install.sh, doesn´t play well with mac:

Code Select Expand

./configure --target-list=arm-softmmu --disable-docs --enable-sdl"
echo "   make -j`grep -c processor /proc/cpuinfo`"

Your line probably works for both linux and mac. Works on mac at least, just tried:

Code Select Expand

echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"

I also wonder if qemu needs update? I get the following error:
https://github.com/espes/xqemu/issues/85

Do we need version qemu1.6?
Seems latest version is qemu2.10.0

Are you running qemu fully working on your mac kichetof?


@dfort
Could you update bottom part of the script to:
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Code Select Expand

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu
cd contrib/qemu/
sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
./install.sh
fi

Kichetof. Why even $GREP? Empty variable. This will do:

Code Select Expand

echo "   make -j`-c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"

And then this will also work:

Code Select Expand

echo "   make -j`sysctl -n hw.ncpu`"

[dfort]

Updated the script but didn't have time to test it yet. I trust you have.

It would be nice if your prompts could accept carriage returns. If you type "Y [return]" on the cross compiler installation you'll never get to the QEMU option.

[kichetof]

@Danne 50D works well and 5D3 only show SD card test (32 bit bug)

Don't forget to update your repo to qemu branch

Code Select Expand

hg update qemu -C

And the install.sh look like that

[Danne]

Oops. THanks Kichetof for the latest  .
Anyway. Stops at the same place as dfort

Code Select Expand

*** Installing dependencies for Mac...


*** WARNING: arm-none-eabi-gdb is not installed.
*** Downloading gcc-arm-embedded toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download the recommended 5.4-mac from gcc-arm-embedded.

*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up arm-none-eabi-gdb before continuing.
dans-MacBook-Pro:

I guess if we´re updating to hg update qemu -C which seems needed there´s no need for these?:

Code Select Expand

sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
?

As a matter of fact after running sed replace brew complains when running the install.sh scrip.

Code Select Expand

Error: No available formula with the name "ggrep"

[kichetof]

Oh I forgot, my workarround with sed grep to ggrep was solved with latest update from a1ex.

I guess if we´re updating to hg update qemu -C which seems needed there´s no need for these?:

Code Select Expand

sed -i '' 's/grep/ggrep/g' install.sh
sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
?

Yes, say good by to my workarround

Now we use $GREP and set it to grep for linux and ggrep for macOS

In the script you could try to do that before running install.sh:

Code Select Expand


echo 'export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH' >> ~/.bash_profile
. ~/.bash_profile


As a matter of fact after running sed replace brew complains when running the install.sh scrip.

Code Select Expand

Error: No available formula with the name "ggrep"

Code Select Expand

brew list grep
which ggrep
grep -V

[Danne]

Great.
Latest take dfort:

Code Select Expand

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
./install.sh
fi

[a1ex]

Just updated the script to use a 32-bit gdb for Mac (untested) and a little more verbosity.

It also allows using a 64-bit GDB if you don't mind the bug (but asks for confirmation).

[Danne]

Tested latest:

Code Select Expand

Continue? [y/n] y

*** Installing dependencies for Mac...


*** WARNING: a valid arm-none-eabi-gdb could not be found.
*** Downloading a toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download macx_i386_gcc_arm_none_eabi_4.8.2 from:
    https://acroname.com/software/arm-eabi-gcc-toolchain-mac-osx-macos-x-32bit

--2017-09-29 09:58:48--  https://acroname.com/system/files/software/macx_i386_gcc_arm_none_eabi_4.8.2.tgz
Resolving acroname.com... 192.155.81.18
Connecting to acroname.com|192.155.81.18|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24839733 (24M) [application/x-gtar]
Saving to: 'macx_i386_gcc_arm_none_eabi_4.8.2.tgz'

macx_i386_gcc_arm_n 100%[===================>]  23.69M  2.49MB/s    in 17s     

2017-09-29 09:59:07 (1.42 MB/s) - 'macx_i386_gcc_arm_none_eabi_4.8.2.tgz' saved [24839733/24839733]

/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up a valid arm-none-eabi-gdb before continuing.
dans-MBP:qemu dan$

Where is gcc versions downloaded? Before it went straight to my user folder but not now.
Noticed it downloads but is deleted before extracted.

[a1ex]

Committed a fix - the printed PATH command was incorrect.

[Danne]

Did the trick. However, there´s no gdb to be found in the bin like with gcc 5.4:

Code Select Expand

macx_i386_gcc_ar 100%[==========>]  23.69M  2.32MB/s    in 14s     

2017-09-29 10:47:28 (1.72 MB/s) - 'macx_i386_gcc_arm_none_eabi_4.8.2.tgz' saved [24839733/24839733]

/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~/cortex/4.8.2/bin:$PATH

*** Please set up a valid arm-none-eabi-gdb before continuing.
dans-MBP:qemu dan$

While we´re at it I renamed my version gcc-arm-none-eabi-4_8-2013q4 to gcc-arm-none-eabi-4_8-2013q3 which has gdb and the follow up to that is:

Code Select Expand

*** Installing dependencies for Mac...

*** WARNING: 64-bit GDB is known not to work.

*** WARNING: a valid arm-none-eabi-gdb could not be found.
*** Downloading a toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download macx_i386_gcc_arm_none_eabi_4.8.2 from:
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
/Users/dan/magic-lantern/contrib/qemu
*** Please add gcc binaries to your executable PATH:
    PATH=~//bin:$PATH

*** WARNING: 64-bit GDB is known not to work.
Continue anyway? [y/N] y


*** Using GDB: /Users/dan/gcc-arm-none-eabi-4_8-2013q3/bin/arm-none-eabi-gdb
GNU gdb (GNU Tools for ARM Embedded Processors) 7.6.0.20131129-cvs

*** Using GCC: /Users/dan/gcc-arm-none-eabi-4_8-2013q3/bin/arm-none-eabi-gcc
/Users/dan/magic-lantern/contrib/qemu/install.sh: line 286: syntax error near unexpected token `&'
dans-MBP:qemu dan$

It´s this line that has to be changed:

Code Select Expand

arm-none-eabi-gcc -v |& grep "gcc version"
I erase that line(why do we need it?

Script finishes and then:

Code Select Expand

cd /Users/dan/qemu/qemu-2.5.0
   ../configure_eos.sh
   make -j4
'
Succes!
Ending lines.

Code Select Expand

CC    arm-softmmu/target-arm/gdbstub.o
  CC    arm-softmmu/target-arm/psci.o
6 warnings generated.
  CC    arm-softmmu/target-arm/crypto_helper.o
  GEN   trace/generated-helpers.c
  CC    arm-softmmu/gdbstub-xml.o
  CC    arm-softmmu/trace/generated-helpers.o
  LINK  arm-softmmu/qemu-system-arm
  REZ   arm-softmmu/qemu-system-arm
  SETFILE arm-softmmu/qemu-system-arm
dans-MBP:qemu-2.5.0 dan$

[a1ex]

Got it, back to the 64-bit one for now...

If you are able to compile a 32-bit GDB, would be nice to use that one.

[Danne]

Got it working. Check updated post above .

We should use this version:
gcc-arm-none-eabi-4_8-2013q4
https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2

erase or rewrite this line:

Code Select Expand

arm-none-eabi-gcc -v |& grep "gcc version"

[a1ex]

Already done - any issues with 5.4? (4.8 is also 64-bit on Mac)

[Danne]

I don´t see no reason for it not to work with 5.4. I´ve been focused on reported errors.

[Danne]

The issue why it´s not working is simply because after the new PATH is set it will not find gdb since it scripted like this:

Code Select Expand

arm-none-eabi-gdb -v
Our Path i set so now it has to be this:

Code Select Expand

gdb

e.g:

Code Select Expand

function valid_arm_gdb {
    if ! gdb -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

At least it seems to work here to get past all the brakes.

Full script:

Code Select Expand

#!/usr/bin/env bash

set -e

QEMU_NAME=${QEMU_NAME:=qemu-2.5.0}
ML=${ML:=magic-lantern}
GREP=${GREP:=grep}

echo
echo "This will setup QEMU for emulating Magic Lantern."
echo "Thou shalt not be afraid of compiling stuff on Linux ;)"
echo -n "Continue? [y/n] "
read answer
if test "$answer" != "Y" -a "$answer" != "y"; then exit 0; fi
echo

function install_gdb {
    echo
    echo "*** Will download gcc-arm-none-eabi-5_4-2016q3 from:"
    echo "    https://developer.arm.com/open-source/gnu-toolchain/gnu-rm"
    echo

    TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
    DOWNLOAD=https://developer.arm.com/-/media/Files/downloads/gnu-rm/5_4-2016q3/
    UNTAR="tar -jxf"

    if [ $(uname) == "Darwin" ]; then
        TARBALL=gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
    else
        TARBALL=gcc-arm-none-eabi-5_4-2016q3-20160926-linux.tar.bz2
    fi

    if [ ! -f ~/$TOOLCHAIN/bin/arm-none-eabi-gdb ]; then
        cd ~
        wget -c $DOWNLOAD$TARBALL && $UNTAR $TARBALL && rm $TARBALL
        cd -
    else
        echo "*** Toolchain already installed in:"
        echo "    ~/$TOOLCHAIN"
        echo
    fi

    echo "*** Please add gcc binaries to your executable PATH:"
    echo "    PATH=~/$TOOLCHAIN/bin:\$PATH"
    echo
}


function valid_arm_gdb {
    if ! gdb -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

    if gdb -v | grep -q "host=x86_64"; then
        # 64-bit version - doesn't work
        # fixme: this may get printed more than once
        echo "*** WARNING: 64-bit GDB is known not to work."
        return 1
    fi

    # assume it's OK
    # todo: check version number
    return 0
}

function valid_arm_gcc {
    if ! gcc -v &> /dev/null; then
        # not installed, or not able to run for any reason
        return 1
    fi

    echo "#include <stdlib.h>" > arm-gcc-test.c
    if ! gcc -c arm-gcc-test.c; then
        echo "*** WARNING: your gcc is unable to compile a simple program."
        rm arm-gcc-test.c
        return 1
    fi

    rm arm-gcc-test.c
    return 0
}

if [ $(uname) == "Darwin" ]; then
    echo "*** Installing dependencies for Mac..."
    echo
    # fixme: don't these require sudo?
    # can we check whether they are already installed, as on Ubuntu?
    if ! xcode-select -p &> /dev/null; then
        xcode-select --install
    fi
    # brew is "The missing package manager for macOS"
    # https://brew.sh
    if ! brew -v &> /dev/null; then
        ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    fi
   
    packages="python wget mercurial xz grep pkg-config glib automake libtool pixman mtools"
    for pkg in $packages; do
        brew list $pkg &> /dev/null || brew install $pkg
    done
   
    GREP=ggrep
fi

if apt-get -v &> /dev/null; then
    # apt-based system?
    # install these packages, if not already
    # only request sudo if any of them is missing
    # instead of GTK (libgtk2.0-dev), you may prefer SDL (libsdl1.2-dev)
    packages="
        build-essential mercurial pkg-config libtool
        git libglib2.0-dev libpixman-1-dev zlib1g-dev
        libgtk2.0-dev xz-utils mtools netcat-openbsd
        python python-pip python-docutils"

    # if a valid arm-none-eabi-gcc/gdb is already in PATH, try to use that
    # otherwise, we'll try to install something
    if ! valid_arm_gdb || ! valid_arm_gcc; then
        echo "*** You do not seem to have an usable arm-none-eabi-gcc and/or gdb installed."
        echo "*** 64-bit GDB is known not to work, so you'll have to install a 32-bit one for now."
        echo
        echo "*** You have a few options:"
        echo
        echo "1 - Install gdb-arm-none-eabi:i386 and gcc-arm-none-eabi from Ubuntu repo (recommended)"
        echo "    This will install 32-bit binaries - will not work under Windows Subsystem for Linux."
        echo
        echo "2 - Download a 32-bit gcc-arm-embedded and install it without the package manager."
        echo "    Will be installed in your home directory; to move it, you must edit the Makefiles."
        echo "    This will install 32-bit binaries - will not work under Windows Subsystem for Linux."
        echo
        echo "3 - Install gdb-arm-none-eabi from Ubuntu repository (64-bit)"
        echo "    WARNING: this will not be able to run all our GDB scripts."
        echo
        if dpkg -l binutils-arm-none-eabi 2>/dev/null | grep -q '^.i'; then
            echo "4 - Remove Ubuntu toolchain and install the one from gcc-arm-embedded PPA (gcc 6.x)"
            echo "    This will:"
            echo "    - sudo apt-get remove gcc-arm-none-eabi gdb-arm-none-eabi \\"
            echo "           binutils-arm-none-eabi libnewlib-arm-none-eabi"
        else
            echo "4 - Install the toolchain from gcc-arm-embedded PPA (gcc 6.x)"
            echo "    This will:"
        fi
        echo "    - sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa"
        echo "    - install the gcc-arm-embedded package."
        echo "    WARNING: this will not be able to run all our GDB scripts."
        echo
        echo "5 - Manually install arm-none-eabi-gdb from https://launchpad.net/gcc-arm-embedded"
        echo "    or any other source, make sure it is in PATH, then run this script again."

        if ! gdb -v &> /dev/null; then
            echo
        else
            echo
            echo "6 - Just use the current 64-bit toolchain."
            echo "    WARNING: this will not be able to run all our GDB scripts."
        fi

        echo
        echo -n "Your choice? "
        read answer
        echo
        case $answer in
            1)
                # Ubuntu's 32-bit arm-none-eabi-gdb works fine
                # gcc-arm-none-eabi:i386 does not include libnewlib - Ubuntu bug?
                packages="$packages gdb-arm-none-eabi:i386 "
                packages="$packages gcc-arm-none-eabi libnewlib-arm-none-eabi"
                ;;
            2)
                # 32-bit gdb will be downloaded after installing these packages
                packages="$packages libc6:i386 libncurses5:i386"
                ;;
            3)
                # Ubuntu's 64-bit arm-none-eabi-gdb works... sort of
                # it's unable to run 5D3 1.1.3 GUI and maybe others
                packages="$packages gdb-arm-none-eabi:amd64"
                packages="$packages gcc-arm-none-eabi:amd64 libnewlib-arm-none-eabi:amd64"
                ;;
            4)
                # gcc-arm-embedded conflicts with gcc-arm-none-eabi
                # but the dependencies are not configured properly
                # so we have to fix the conflict manually...
                if dpkg -l binutils-arm-none-eabi 2>/dev/null | grep -q '^.i'; then
                    echo
                    echo "*** Please double-check - the following might remove additional packages!"
                    echo
                    sudo apt-get remove gcc-arm-none-eabi gdb-arm-none-eabi \
                         binutils-arm-none-eabi libnewlib-arm-none-eabi
                fi
                packages="$packages gcc-arm-embedded"
                echo
                echo "*** Adding the team-gcc-arm-embedded PPA repository..."
                echo "    sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa"
                echo
                sudo add-apt-repository ppa:team-gcc-arm-embedded/ppa
                ;;
            5)
                # user will install arm-none-eabi-gdb and run the script again
                exit 0
                ;;
            6)
                # use the installed version, even though it's known not to work
                ;;
            *)
                # invalid choice
                exit 1
                ;;
        esac
    else
        echo "*** You have a valid ARM GCC/GDB already installed - using that one."
    fi

    echo "*** Checking dependencies for Ubuntu..."
    echo
    # https://wiki.debian.org/ListInstalledPackages
    # dpkg -l also returns packages that are not installed
    deps_installed=yes
    for package in $packages; do
        if ! dpkg -l $package 2>/dev/null | grep -q '^.i'; then
            echo Not installed: $package
            deps_installed=no
        fi
    done

    if [ "$deps_installed" == "no" ]; then
        echo
        echo "*** Installing dependencies for Ubuntu..."
        echo
        sudo apt-get update
        sudo apt-get install $packages
        echo
    fi

    deps_installed=yes
    for package in $packages; do
        if ! dpkg -l $package 2>/dev/null | grep -q '^.i'; then
            echo Not installed: $package
            deps_installed=no
        fi
    done

    if [ "$deps_installed" == "no" ]; then
        echo
        echo "*** Error: Ubuntu dependencies could not be installed."
        echo
        exit 1
    fi
fi

# all systems (including Mac, or Ubuntu if the installation from repositories failed)
# this one works on old systems as well, but it won't work under WSL
if ! valid_arm_gdb; then
    echo
    echo "*** WARNING: a valid arm-none-eabi-gdb could not be found."
    echo "*** Downloading a toolchain and installing it without the package manager."
    echo "*** Will be installed in your home directory (Makefile.user.default expects it there)."
    echo
    install_gdb
fi

# make sure we have a valid arm-none-eabi-gdb (regardless of operating system)
if ! valid_arm_gdb; then
    if ! arm-none-eabi-gdb -v &> /dev/null; then
        echo "*** Please set up a valid arm-none-eabi-gdb before continuing."
        exit 1
    else
        # valid_arm_gdb will print why the current one is not good
        echo -n "Continue anyway? [y/N] "
        read answer
        if test "$answer" != "Y" -a "$answer" != "y"; then exit 1; fi
        echo
    fi
fi

# same for arm-none-eabi-gcc
if ! valid_arm_gcc; then
    echo "*** Please set up a valid arm-none-eabi-gcc before continuing."
    exit 1
fi

echo
echo -n "*** Using GDB: "
command -v arm-none-eabi-gdb
arm-none-eabi-gdb -v | head -n1
echo
echo -n "*** Using GCC: "
command -v arm-none-eabi-gcc
arm-none-eabi-gcc -v 2>&1 | grep "gcc version"
echo

# install docutils (for compiling ML modules) and vncdotool (for test suite)
# only install if any of them is missing
pip2 list | grep docutils  || rst2html -h  > /dev/null || pip2 install docutils
pip2 list | grep vncdotool || vncdotool -h > /dev/null || pip2 install vncdotool

function die { echo "${1:-"Unknown Error"}" 1>&2 ; exit 1; }

pwd | grep $ML/contrib/qemu > /dev/null || die "error: we should be in $ML/contrib/qemu"

# go to the parent of magic-lantern folder
cd ../../..
ls | $GREP $ML > /dev/null || die "error: expecting to find $ML here"

mkdir -p qemu
cd qemu

echo
echo "*** Setting up QEMU in `pwd`..."
echo

# get qemu
wget -q --show-progress --progress=dot:giga -c http://wiki.qemu-project.org/download/$QEMU_NAME.tar.bz2
echo
tar jxf $QEMU_NAME.tar.bz2
echo

# initialize a git repo, to make it easy to track changes to QEMU source
cd $QEMU_NAME
if [ ! -d .git ]; then
  git init
  # git requires a valid email; if not setup, add one for this directory only
  git config user.email || git config user.email [email protected]
  git add .
  git commit -q -m "$QEMU_NAME vanilla"
fi
cd ..

echo "Copying files..."

# copy our helper scripts
cp -r ../$ML/contrib/qemu/scripts/* .
chmod +x *.sh

# copy our testing scripts
mkdir -p tests
cp -r ../$ML/contrib/qemu/tests/* tests/
chmod +x tests/*.sh

# apply our patch
cd ${QEMU_NAME}
mkdir -p hw/eos
cp -r ../../$ML/contrib/qemu/eos/* hw/eos/
cp -r ../../$ML/src/backtrace.[ch] hw/eos/dbi/
patch -N -p1 < ../../$ML/contrib/qemu/$QEMU_NAME.patch
cd ..

# setup the card image
if [ ! -f "sd.img" ]; then
    echo "Setting up SD card image..."
    cp -v ../$ML/contrib/qemu/sd.img.xz .
    unxz -v sd.img.xz
else
    echo "SD image already exists, skipping."
fi

if [ ! -f "cf.img" ]; then
    echo "Setting up CF card image..."
    cp -v sd.img cf.img
else
    echo "CF image already exists, skipping."
fi

echo ""
echo "Next steps:"
echo "==========="
echo
echo "1) Compile QEMU"
echo
echo "   cd `pwd`/${QEMU_NAME}"
echo "   ../configure_eos.sh"
echo "   make -j`$GREP -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"
echo
echo "2) Grab a copy of the Canon firmware from your own camera"
echo "   (don't request one and don't share it online - it's copyrighted)"
echo
echo "   Look on your SD card - you should find ML/LOGS/ROM0.BIN and ROM1.BIN"
echo "   Copy them under your camera model's subdirectory, for example:"
echo "   `pwd`/60D/"
echo
echo "   For models that use a serial flash, you may have to dump its contents"
echo "   using the sf_dump module, then copy SFDATA.BIN as well."
echo
echo "3) Mount the included SD (or CF) image (you may use mount.sh)"
echo "   and install ML on it, as usual. The card image must be bootable as well."
echo
echo "   The included card image is bootable and contains a small autoexec.bin"
echo "   that runs on all DIGIC 4/5 cameras and prints some basic info."
echo
echo "   To create your own SD/CF image, you need to copy the raw contents"
echo "   of the entire card, not just one partition. For example:"
echo "   dd if=/dev/mmcblk0 of=sd.img"
echo
echo "4) Start emulation with:"
echo
echo "   cd `pwd`/"
echo "   ./run_canon_fw.sh 60D"
echo
echo "   This will recompile QEMU, but not ML."
echo
echo "   Note: Canon GUI emulation (menu navigation, no LiveView) only works on"
echo -n "   "; $GREP --color=never -oP "(?<=GUI_CAMS=\( ).*(?=\))" tests/run_tests.sh;
echo
echo "5) Tips & tricks:"
echo "   - to enable or disable the boot flag in ROM, use something like:"
echo "     ./run_canon_fw.sh 60D,firmware=\"boot=1\""
echo "   - to use multiple firmware versions, place the ROMs under e.g. 5D3/113/ROM*.BIN and run:"
echo "     ./run_canon_fw.sh 5D3,firmware=\"113;boot=1\""
echo "   - to show MMIO activity (registers) and interrupts, use:"
echo "     ./run_canon_fw.sh 60D -d io,int"
echo "   - to show the executed ASM code, step by step, use:"
echo "     ./run_canon_fw.sh 60D -d exec,int -singlestep"
echo "   - to trace debug messages and various functions in the firmware, use:"
echo "     ./run_canon_fw.sh 60D -s -S & arm-none-eabi-gdb -x 60D/debugmsg.gdb"
echo "   - if the above is too slow, compile the dm-spy-experiments branch "
echo "     with CONFIG_QEMU=y and CONFIG_DEBUG_INTERCEPT_STARTUP=y and try:"
echo "     ./run_canon_fw.sh 60D,firmware=\"boot=1\" -d io,int"
echo "   - some camera models require GDB patches to bypass tricky code sequences:"
echo "     ./run_canon_fw.sh 700D -s -S & arm-none-eabi-gdb -x 700D/patches.gdb"
echo "   - to trace all function calls and export them to IDA:"
echo "     ./run_canon_fw.sh 60D -d calls -singlestep"
echo "   - you may enable additional debug code (such as printing to QEMU console)"
echo "     by compiling ML with CONFIG_QEMU=y in your Makefile.user (also run make clean)."
echo "   - caveat: you cannot run autoexec.bin compiled with CONFIG_QEMU on the camera."

echo
echo "Enjoy!"

[a1ex]

arm-none-eabi-gdb is correct - double-check the path. It should be:

Code Select Expand


PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH


[Danne]

You´re right. I was calling system specific gdb. However. The issue is that the checking command doesn´t return that it finds gdb. Will check more.

[Danne]

Ok, been through a few rounds now but it seems to boil down to the path not being properly set:

Code Select Expand

    echo "   PATH=~/$TOOLCHAIN/bin:\$PATH"

Only echoing the path won´t work. The backslash breaks it as well.

Usually I would do:

Code Select Expand

export PATH=~/$TOOLCHAIN/bin/:$PATH

This works:

Code Select Expand

    echo "   PATH=~/$TOOLCHAIN/bin/:$PATH"
    PATH=~/$TOOLCHAIN/bin/:$PATH

And this:

Code Select Expand

    echo "   PATH=~/$TOOLCHAIN/bin/:$PATH"
    export PATH=~/$TOOLCHAIN/bin/:$PATH

[Danne]

Here is dfort script. Should get you right into qemu after all else is installed:

Code Select Expand

cat <<'EOF' > mac_ml.sh
#!/bin/sh

#  mac_ml.sh
#
#  Script to almost automatically install
#  and configure a Magic Lantern development
#  environment on a Macintosh.
#
#  Reference:
#  http://magiclantern.fm/forum/index.php?topic=16012.0
#
#  Daniel Fort (dfort) - 2015-10-15
#  Modified 2016-03-17 - took out ugly gcc hack
#  Modified 2017-06-27 - added mingw-w64 for cross compiling Windows binaries
#  Modified 2017-06-28 - check if magic-lantern already exists
#  Modified 2017-07-12 - write a Makefile.user file
#  Modified 2017-09-23 - made mingw-w64 (Windows cross compiler) optional
#  Modified 2017-09-24 - added interactive options for mingw-w64 and QEMU
#  Modified 2017-09-26 - QEMU install.sh now installs dependencies so no need to do it here
#                                   now using gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
#
#  Thanks to Danne and kichetof and of course a1ex for their input

if ! test -d /usr/local; then
  echo "/usr/local directory not found."
  echo "Follow the steps at the beginning of"
  echo "tutorial to create /usr/local and"
  echo "then re-run this script."
  exit 1
fi

echo "/usr/local found -- "
echo "continuing with installation."

cd ~
if [ ! -d "/Library/Developer/CommandLineTools" ]; then xcode-select --install; fi
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install python wget mercurial

#
# Uncomment the following line if you want to cross compile Windows binaries.
# brew install mingw-w64
#

pip2 install docutils

brew tap homebrew/dupes; brew install homebrew/versions/gcc5

cd ~ && \
    [ ! -f gcc-arm-none-eabi-4_8-2013q4 ] &&
    wget -c https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2

if ! test -d ~/magic-lantern; then
    hg clone https://bitbucket.org/hudson/magic-lantern
    cd magic-lantern
    hg update unified
    cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT
    cd ~
fi

#OPTIONAL 1
clear
read -p $(tput bold)"Would you also like to install the windows cross compiler mingw-w64?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
clear
read -p $(tput bold)"Would you also like to compile QEMU?$(tput setaf 1)
Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
clear
echo "
installing mingw-w64"
brew install mingw-w64 || :
#compiling qemu
clear
echo "
compiling qemu"
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
export PATH=~/$TOOLCHAIN/bin/:$PATH
./install.sh
else
clear
echo "
installing mingw-w64"
brew install mingw-w64 || :
fi
else
#OPTIONAL 2
clear
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
hg update qemu -C
cd contrib/qemu/
TOOLCHAIN=gcc-arm-none-eabi-5_4-2016q3
export PATH=~/$TOOLCHAIN/bin/:$PATH
./install.sh
fi
fi

cd ~
exit 0
EOF
bash mac_ml.sh && rm mac_ml.sh

[a1ex]

Of course, echoing the path won't work, you need to copy/paste that command and run it outside the script (or place it into your .profile if you want). And I don't want to echo the entire path, that's the point of "\$".

The script itself is not able to set the path outside it (it won't change the environment of its parent shell). That's why I'm echoing the command.

The script itself has no need to run arm-none-eabi-gdb/gcc - it only makes sure *you* are going to be able to run them, as described in the QEMU guide.

See my screenshots above and this animation.