Compiling Magic Lantern on a Macintosh

[bouncyball]

Yay! Goodby ugly monster Cygwin...

[ilia3101]

I just wanna say thanks so so much dfort/Danne for the install script at the end
Couldn't get anything to work on my fresh macOS install until I just tried that!

Worked on 10.13 high sierra almost flawlessly, only had to make this small change for it to work:

Code: [Select]

pip install docutilschanged to this:

Code: [Select]

sudo pip install docutils
... and a suggestion: maybe don't automatically install mingw in the script, as most people wont be compiling this stuff for other platforms,
instead maybe add a note on the first post on how to for those who want to (please? ) Also space is precious on a 240gb SSD (real reason for my concern)

[Danne]

Great Ilia3101. This is thread is a lifesaver  . Went back in here yesterday after reinstalling python versions and whatnot. Suddenly lost docutils and then cr2hdr wouldn´t compile.
For docutils I did:

Code: [Select]

sudo pip2 install docutilsand when cr2hdr would complain I changed makefile as suggested in first post:

Now make a new file called Makefile.user and put in just these lines:

Code: [Select]

#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)Seems that only cr2hdr needs this change nowadays.

Worth mentioning, after installing docutils this wasn´t needed anymore:

Code: [Select]

for f in /usr/local/bin/*.py; do ln -s $f "${f/.py/}"; done

[dfort]

I made the mingw-w64 Windows cross compiler option in the quick install script and started looking into the pip issue. Yeah, looks like something has changed and my installation no longer works as it should. Glad you guys could figure it out on your systems but I'd like to get the quick install script installing quickly again. Need to find a virgin Mac to try it out.

[kichetof]

Need to find a virgin Mac to try it out.

Virtualization 
I did it for you!


Compiling Magic Lantern on macOS High Sierra _{BETA (10.13 17A362a)}


It's an adaptation of @dfort commands, thanks to you !

Compiling ML + Modules + tools (like cr2hdr, mlv_dump, ..)

macos-high-sierra:~ tof$ xcode-select --install
macos-high-sierra:~ tof$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
macos-high-sierra:~ tof$ brew install python wget mercurial
macos-high-sierra:~ tof$ pip2 install docutils
macos-high-sierra:~ tof$ brew tap homebrew/dupes; brew install homebrew/versions/gcc5
macos-high-sierra:~ tof$ cd ~ && wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2
macos-high-sierra:~ tof$ hg clone https://bitbucket.org/hudson/magic-lantern
macos-high-sierra:~ tof$ cd magic-lantern
macos-high-sierra:magic-lantern tof$ hg update unified
macos-high-sierra:magic-lantern tof$ cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT
# OPTIONAL cross-compile
macos-high-sierra:~ tof$ brew install mingw-w64

Compiling QEMU

macos-high-sierra:magic-lantern tof$ brew install xz grep pkg-config glib automake libtool pixman mtools
macos-high-sierra:magic-lantern tof$ hg update qemu
macos-high-sierra:magic-lantern tof$ cd contrib/qemu/
# "Fix" for QEMU on macOS (need custom install.sh : change grep -P to ggrep -P)
# need to PR a great solution not a workarround, like PYTHON=python2, GREP=ggrep
# I don't want to replace macOS grep with GNU grep (prevent eventual bug), if you want: @dfort solution and ignore last command
macos-high-sierra:qemu tof$ sed -i '' 's/grep/ggrep/g' install.sh
macos-high-sierra:qemu tof$ sed -i '' 's/grep/ggrep/g' scripts/run_canon_fw.sh
macos-high-sierra:qemu tof$ ./install
# Follow next steps from output

We're ready for tomorrow 

[dfort]

Wow, thanks @kichetof

So the whole "pip" issue was because Homebrew switched over from pip to pip2. No sudo needed and that line to make symbolic links on the docutils scripts is still needed.

In any case, I updated the install script and it is working again.

Need to do some more testing. Walter suggested a "virtual virgin" setup. (Sure that's not a virtuous virgin?  )

Nice work on the QEMU setup. I was stumbling around in the dark when I came across the Mac grep issue. It looks like kichetof has a better solution. Does it require some tweaks in the QEMU install script?

[EDIT] Just a thought, @Danne you're good at bash scripting--how about adding user interactive options at the end of the script to add Windows cross compiling and another for setting up a QEMU ready development environment?

[Danne]

EDIT] Just a thought, @Danne you're good at bash scripting--how about adding user interactive options at the end of the script to add Windows cross compiling and another for setting up a QEMU ready development environment?

Sure thing. Shouldn't be hard at all. Will take a look soon.
Great action @Kichetof.

[Danne]

This should work, check OPTIONAL 1 and OPTIONAL 2:

Modified following:
Added
./install.sh to the script (OPTION 2) and erased set -e at the top.
Took the liberty to erase:
sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
sudo mkdir -p /usr/local/ && sudo chflags norestricted /usr/local/ && sudo chown $(whoami):admin /usr/local/ && echo sudo chown -R $(whoami):admin /usr/local/
Also erased:
for f in /usr/local/bin/*.py; do ln -s $f "${f/.py/}"; done

Code: [Select]

cat <<'EOF' > mac_ml.sh
#!/bin/sh

#  mac_ml.sh
#
#  Script to almost automatically install
#  and configure a Magic Lantern development
#  environment on a Macintosh.
#
#  Reference:
#  http://magiclantern.fm/forum/index.php?topic=16012.0
#
#  Daniel Fort (dfort) - 2015-10-15
#  Modified 2016-03-17 - took out ugly gcc hack
#  Modified 2017-06-27 - added mingw-w64 for cross compiling Windows binaries
#  Modified 2017-06-28 - check if magic-lantern already exists
#  Modified 2017-07-12 - write a Makefile.user file
#  Modified 2017-09-23 - made mingw-w64 optional
#  Modified 2017-09-24 -
#
#  Thanks to Danne and kichetof for their input

if ! test -d /usr/local; then
  echo "/usr/local directory not found."
  echo "Follow the steps at the beginning of"
  echo "tutorial to create /usr/local and"
  echo "then re-run this script."
  exit 1
fi

echo "/usr/local found -- "
echo "continuing with installation."

cd ~
if [ ! -d "/Library/Developer/CommandLineTools" ]; then xcode-select --install; fi
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install python wget mercurial

#
# Uncomment the following line if you want to cross compile Windows binaries.
# brew install mingw-w64
#

pip2 install docutils

brew tap homebrew/dupes; brew install homebrew/versions/gcc5
cd ~ && wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2

if ! test -d ~/magic-lantern; then
    hg clone https://bitbucket.org/hudson/magic-lantern
    cd magic-lantern
    hg update unified
    cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT
fi

#OPTIONAL 1
clear
read -p $(tput bold)"Would you also like to install the windows cross compiler mingw-w64?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
brew install mingw-w64
fi

#OPTIONAL 2
clear
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
brew install xz grep pkg-config glib automake libtool pixman mtools
hg update qemu
cd contrib/qemu/
sed -i '' 's/gggrep/ggrep/g' install.sh
./install.sh
fi

cd ~
exit 0
EOF
bash mac_ml.sh && rm mac_ml.sh
A couple of things.

1 - The script really works best with a "virgin mary" machine. As soon as you are trying to reinstall things the script breaks telling you what is already installed. This means you won´t be able to rerun the script since it will break at the same place each time.Erase set -e

A solution: As the script breaks it could fall back into a bash menu again interacting with the user telling what is already installed and then ask user to continue anyway. Even better if files are already installed simply continue automatically.
*Update. Just checked and erase set -e to let the script run without breaks.

2 - compiling Qemu OPTION 2, last line:

Code: [Select]

sed -i '' 's/grep/ggrep/g' install.sh
replaces to ggrep in install.sh. Next step, run the install.sh script:
Simply finish with this?

Code: [Select]

./install.sh

The two interactive user options should then become:

Code: [Select]

#OPTIONAL 1
clear
read -p $(tput bold)"Would you also like to install the windows cross compiler mingw-w64?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
brew install mingw-w64
fi

#OPTIONAL 2
clear
read -p $(tput bold)"How about compiling QEMU?$(tput setaf 1)

Y/N?"$(tput sgr0) -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]
then
cd magic-lantern
brew install xz grep pkg-config glib automake libtool pixman mtools
hg update qemu
cd contrib/qemu/
sed -i '' 's/gggrep/ggrep/g' install.sh
./install.sh
fiAnd then the quemu install.sh script will tell you:

Code: [Select]

This will setup QEMU for emulating Magic Lantern.
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n]



Still using the for loop although running pip2?

Code: [Select]

pip2 install docutils
for f in /usr/local/bin/*.py; do ln -s $f "${f/.py/}"; done

[kichetof]

I forgot to add change for run_canon_fw.sh
edited my last post with changes

@Danne I didn't change permissions on /usr/local and didn't copy *.py

@a1ex what do you think about this change ? (for install.sh and run_canon_fw.sh)

Code: [Select]

tof-mbp:qemu tof$ diff run_canon_fw_tof.sh run_canon_fw.sh
3d2
< GREP=${GREP:=grep}
11c10
<     SD_DEV=`losetup -j $1 | $GREP -Po "(?<=/dev/)[^ ]*(?=:)"`
---
>     SD_DEV=`losetup -j $1 | grep -Po "(?<=/dev/)[^ ]*(?=:)"`
13c12
<         if cat /proc/mounts | $GREP /dev/mapper/$SD_DEV; then
---
>         if cat /proc/mounts | grep /dev/mapper/$SD_DEV; then
21c20
<     if [[ -n $(ls /Volumes | $GREP EOS_DIGITAL*) ]]; then
---
>     if [[ -n $(ls /Volumes | grep EOS_DIGITAL*) ]]; then
26,34d24
<     
<     if [[ -n $(which ggrep) ]]; then
<         GREP=ggrep
<     else
<         echo
<         echo "Error: you need GNU grep to run this script"
<         echo "brew install grep"
<         exit 1
<     fi
60c50
<     QEMU_EOS_DEBUGMSG=`cat $CAM/debugmsg.gdb | $GREP DebugMsg_log -B 1 | $GREP -Pom1 "(?<=b \*)0x.*"`
---
>     QEMU_EOS_DEBUGMSG=`cat $CAM/debugmsg.gdb | grep DebugMsg_log -B 1 | grep -Pom1 "(?<=b \*)0x.*"`

[Danne]

I didn't change permissions on /usr/local and didn't copy *.py

Yeah, me neither, could probably be deleted...

[kichetof]

could probably be deleted...

On macOS Sierra and High Sierra for me yes

[dfort]

Updated the quick install script.

A few notes:

• You are right Danne, the symbolic links are no longer required for docutils. It was fixed on this merge.
• All user input requires a carriage return but the options in the script will drop you out of the script if you hit the carriage return.
• So not only did you setup the system to install QEMU you are also running the QEMU installer script. Cool--but:

Code: [Select]

./install.sh: line 38: cd: .git: No such file or directory
Initialized empty Git repository in /Users/Fort/qemu/qemu-2.5.0/.git/

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: unable to auto-detect email address (got 'Fort@Rosies-Air.(none)')

Quite a few Macs are probably set up this way. Given enough time I could probably fix these issues but I'm using my wife's Mac to run these tests and she wants it back!

[DeafEyeJedi]

Sick progress guys. Now I'm anxious to try the script installation for QEMU. Probably should run the first script just to be sure my compiling environment is up to date. Thanks!

[dfort]

Looks like that git error is because the install.sh script is initializing a git repository in order to track changes. Maybe this could be a user option in install.sh or the Mac quick install script should verify that auto-detecting the email address works before getting into the QEMU installation?

Code: [Select]

# get qemu
wget --progress=dot:giga -c http://wiki.qemu-project.org/download/$QEMU_NAME.tar.bz2
tar jxf $QEMU_NAME.tar.bz2

# initialize a git repo, to make it easy to track changes to QEMU source
cd $QEMU_NAME
cd .git && cd .. || (git init && git add . && git commit -q -m "$QEMU_NAME vanilla")
cd ..


[a1ex]

Sorted it out - the same issue was present on a fresh Ubuntu.

Also updated the install script to set up the dependencies as well - WIP.

[dfort]

Excellent! Updated the Mac quick install script to take advantage of the latest changes in QEMU install.sh.

FYI for users wanting to install QEMU with the Mac quick install script--this is still a WIP and the QEMU installation currently ends with:

Code: [Select]

*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up arm-none-eabi-gdb before continuing.

You'll have to find your way from there. The journey I took to get QEMU running on the Mac started here:

http://www.magiclantern.fm/forum/index.php?topic=2864.msg184773#msg184773

[kichetof]

For those who have updated from macOS Sierra to macOS High Sierra, you need to reinstall gcc-5

Code: [Select]

brew reinstall gcc@5
take some beers and wait! 

[DeafEyeJedi]

Thanks for the heads up @kichetof as I'm about to update mine as well.

[DeafEyeJedi]

Alright guys here we go with me about to embarrass myself for beneficial reasons. 

My first attempt at installing QEMU after running the latest Quick Installation Script (Thanks @Danne!) which then shows what needed to be updated even tho most of the compiling tools were already installed previously awhile ago on this grumpy old Mac Mini running 10.12.6 (decided not to update this one yet to Sierra High 10.13 since after experiencing random issues when updating this on my MBP) and finally this is where I started w my attempt at installing QEMU...

What is it that I could be doing incorrectly and sorry for my stupidity. Since I am unable to empathize what 'please call configure before running make' means?

[a1ex]

Here's how it should work in Ubuntu; should be the same on Mac, except for the 64-bit GDB bug (so you may not be able to launch the 5D3 GUI - try 100D instead):

http://www.magiclantern.fm/forum/index.php?topic=2864.msg190596#msg190596

(I have a hard time reading your error messages, even when watching in full-screen...)

This animation might be useful (although I should probably add some captions to show the commands edit: DONE!).

[dfort]

You have a grep error message:



Then you ran the error message--that won't get you anywhere. What grep attempted to do was to give you configure instructions for your system. I ran into that same problem a while back:

http://www.magiclantern.fm/forum/index.php?topic=2864.msg184773#msg184773

After installing the dependencies using Homebrew and downloading the ARM compiler the install script should compile QEMU. I'm surprised you're getting as far as you got, I'm getting:

Code: [Select]

*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up arm-none-eabi-gdb before continuing.


[a1ex]

And copy-pasting the PATH command doesn't help?

(the script cannot change environment variables outside it, and I didn't dig too far to find a better and cross-platform way to solve this)

[dfort]

And copy-pasting the PATH command doesn't help?

Nope. Though arm-none-eabi-gdb is in the path:

Code: [Select]

which arm-none-eabi-gdb
/Users/rosiefort/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gdb

[a1ex]

Ah, it's because the script figures out the 64-bit GDB won't work...

Does the Mac run 32-bit binaries? If yes, can you try compiling a 32-bit GDB from source and place that one in PATH?

edit: found one here, does it help? https://acroname.com/software/arm-eabi-gcc-toolchain-mac-osx-macos-x-32bit

[dfort]

The Mac can run 32-bit binaries. Downloaded and placed in the right path but--

Code: [Select]

*** Installing dependencies for Mac...


*** WARNING: arm-none-eabi-gdb is not installed.
*** Downloading gcc-arm-embedded toolchain and installing it without the package manager.
*** Will be installed in your home directory (Makefile.user.default expects it there).

*** Will download the recommended 5.4-mac from gcc-arm-embedded.

*** Please add gcc binaries to your executable PATH:
    PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up arm-none-eabi-gdb before continuing.
RosieFoComputer:qemu rosiefort$ which arm-none-eabi-gdb
/Users/rosiefort/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gdb

It didn't download anything but it didn't recognize the installed arm-none-eabi-gdb even though as can see it is in the path.