Compiling Magic Lantern on a Macintosh

[Danne]

I see. .profile or manually copy paste. Won't be in here for a while. Maybe dfort takes a look before me.
Thanks.

[DeafEyeJedi]

Great progress guys on troubleshooting this QEMU bad boy down. Here's a quick summary after my 2nd attempt at installing QEMU...

Code: [Select]

==> Summary
🍺  /usr/local/Cellar/isl/0.18: 80 files, 3.8MB
==> Installing mingw-w64
==> Downloading https://homebrew.bintray.com/bottles/mingw-w64-5.0.2_3.sierra.bo
######################################################################## 100.0%
==> Pouring mingw-w64-5.0.2_3.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/mingw-w64/5.0.2_3: 7,834 files, 669MB


compiling qemu
0 files updated, 0 files merged, 0 files removed, 0 files unresolved

This will setup QEMU for emulating Magic Lantern.
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n] y

*** Setting up QEMU in /Users/SMJ/qemu...

--2017-09-29 08:42:03--  http://wiki.qemu-project.org/download/qemu-1.6.0.tar.bz2
Resolving wiki.qemu-project.org... 172.99.69.163
Connecting to wiki.qemu-project.org|172.99.69.163|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://wiki.qemu.org/download/qemu-1.6.0.tar.bz2 [following]
--2017-09-29 08:42:04--  http://wiki.qemu.org/download/qemu-1.6.0.tar.bz2
Resolving wiki.qemu.org... 172.99.69.163
Reusing existing connection to wiki.qemu-project.org:80.
HTTP request sent, awaiting response... 302 Found
Location: https://wiki.qemu.org/download/qemu-1.6.0.tar.bz2 [following]
--2017-09-29 08:42:06--  https://wiki.qemu.org/download/qemu-1.6.0.tar.bz2
Connecting to wiki.qemu.org|172.99.69.163|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://download.qemu.org/qemu-1.6.0.tar.bz2 [following]
--2017-09-29 08:42:07--  https://download.qemu.org/qemu-1.6.0.tar.bz2
Resolving download.qemu.org... 172.99.69.163
Connecting to download.qemu.org|172.99.69.163|:443... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

../magic-lantern/contrib/qemu/scripts/copy_back_to_contrib.sh -> ./copy_back_to_contrib.sh
../magic-lantern/contrib/qemu/scripts/gdbopts -> ./gdbopts
../magic-lantern/contrib/qemu/scripts/run_canon_fw.sh -> ./run_canon_fw.sh
../magic-lantern/contrib/qemu/scripts/run_ml.sh -> ./run_ml.sh
../magic-lantern/contrib/qemu/scripts/run_ml_100D.sh -> ./run_ml_100D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_500D.sh -> ./run_ml_500D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_50D.sh -> ./run_ml_50D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_550D.sh -> ./run_ml_550D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_5D2.sh -> ./run_ml_5D2.sh
../magic-lantern/contrib/qemu/scripts/run_ml_5D3.sh -> ./run_ml_5D3.sh
../magic-lantern/contrib/qemu/scripts/run_ml_5D3_123.sh -> ./run_ml_5D3_123.sh
../magic-lantern/contrib/qemu/scripts/run_ml_600D.sh -> ./run_ml_600D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_60D.sh -> ./run_ml_60D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_650D.sh -> ./run_ml_650D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_6D.sh -> ./run_ml_6D.sh
../magic-lantern/contrib/qemu/scripts/run_ml_7D.sh -> ./run_ml_7D.sh
../../magic-lantern/contrib/qemu/hw/eos.c -> hw/arm/eos.c
../../magic-lantern/contrib/qemu/hw/eos.h -> hw/arm/eos.h
patching file hw/arm/Makefile.objs
patching file memory.c
patching file target-arm/cpu.c

Next steps:
===========

1) Compile QEMU

   cd /Users/SMJ/qemu/qemu-1.6.0
   ./configure --target-list=arm-softmmu --disable-docs --enable-sdl
grep: /proc/cpuinfo: No such file or directory
   make -j

2) Grab a copy of the Canon firmware from your own camera
   (don't request one and don't share it online - it's copyrighted)

   Look on your SD card - you should find ML/LOGS/ROM0.BIN and ROM1.BIN
   Copy those in /Users/SMJ/qemu/ and then run (for 60D):

   cat ROM0.BIN ROM1.BIN > ROM-60D.BIN

3) Enable CONFIG_QEMU=y in your Makefile.user
   from magic-lantern directory, then run 'make clean' to make sure
   you will rebuild ML from scratch.

4) Create a subdirectory named 'sdcard' or 'cfcard' and copy ML files there.
   Make sure all ML files are UPPERCASE (just copy them from a FAT32 card).
   You should get something like:
   /Users/SMJ/qemu/sdcard/ML/AUTOEXEC.BIN
   /Users/SMJ/qemu/sdcard/ML/FONTS/ARGNOR32.RBF
   /Users/SMJ/qemu/sdcard/ML/MODULES/ARKANOID.MO
   and so on.

5) Start emulation with:

   cd /Users/SMJ/qemu/
   ./run_ml_60D.sh

   (this will recompile ML and QEMU - handy if you edit the sources often)

Enjoy!
Seans-Mac-mini-385:~ DeafEyeJedi$



Does this mean it's some kind of a success?

Uh-oh...

Code: [Select]

Seans-Mac-mini-385:~ DeafEyeJedi$ cd /Users/SMJ/qemu/qemu-1.6.0
Seans-Mac-mini-385:qemu-1.6.0 DeafEyeJedi$    ./configure --target-list=arm-softmmu --disable-docs --enable-sdl

ERROR: User requested feature sdl
       configure was not able to find it

Seans-Mac-mini-385:qemu-1.6.0 DeafEyeJedi$ grep: /proc/cpuinfo: No such file or directory
-bash: grep:: command not found
Seans-Mac-mini-385:qemu-1.6.0 DeafEyeJedi$    make -j
make: *** No rule to make target `/qemu-options.hx', needed by `qemu-options.def'.  Stop.
make: *** Waiting for unfinished jobs....
Please call configure before running make!
make: *** [config-host.mak] Error 1
Seans-Mac-mini-385:qemu-1.6.0 DeafEyeJedi$

Did I miss a step in here?

[a1ex]

When all else fails, read the instructions (and replies #132, #149, or this or latest commits or the latest tweet...)

[dfort]

@DeafEyeJedi -- You keep running the error message. Don't just blindly copy and paste. Read what it says first--grep is trying to tell you something is wrong!

Code: [Select]

1) Compile QEMU

   cd /Users/SMJ/qemu/qemu-1.6.0
   ./configure --target-list=arm-softmmu --disable-docs --enable-sdl
grep: /proc/cpuinfo: No such file or directory
   make -j

That said, the reason this isn't going well for you is because things have changed recently and you are trying to install QEMU without having the latest ML Mac development environment. Let's start over:

Code: [Select]

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)"
cd /usr/local
ls

[EDIT - removed the suggestion to use "rm -rf *"] Got anything left in there? It is ok to get rid of it if you're going to do a new Homebrew installation.

I'd do it a step at a time, you don't want to force remove everything if you're in the wrong directory!

Let's also delete the ML repository and the gcc toolchain. (That "cd" is to get you into your home directory, it is a shortcut for "cd ~/")

Code: [Select]

cd
rm -rf magic-lantern
rm -rf gcc-arm-none-eabi*
Now run the quick install script on the first post--I keep updating it with the latest suggestions.

If you answer "Y" to the QEMU install prompt it won't work the first time. The script installs a newer version of the gcc-arm-none-eabi toolchain and ends up with this message:

Code: [Select]

*** Please add gcc binaries to your executable PATH.
*** Run this command, or paste it into your .profile and reopen the terminal:
    export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** Please set up a valid arm-none-eabi-gdb before continuing.

So do exactly what that message says:

Code: [Select]

export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH
At this point my script has done all that it could and invoked the QEMU install.sh script so:

Code: [Select]

cd ~/magic-lantern
hg update qemu -C
cd contrib/qemu/
./install.sh

You will probably see something like this:

Code: [Select]

*** Will download gcc-arm-none-eabi-5_4-2016q3 from:
    https://developer.arm.com/open-source/gnu-toolchain/gnu-rm

*** Toolchain already installed in:
    ~/gcc-arm-none-eabi-5_4-2016q3

*** Please add gcc binaries to your executable PATH.
*** Run this command, or paste it into your .profile and reopen the terminal:
    export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH

*** WARNING: 64-bit GDB is known not to work.
Continue anyway? [y/N] y

Go for it, it works on the Mac. Well there maybe some things that don't work but this should be good enough to get you started so study those links a1ex gave you and head on over to the QEMU topic and join the fun. Here's a quick tip--since you have a 100D, find the folder for that camera in your ~/qemu directory and put your firmware dumps in there, ROM0.BIN, ROM1.BIN and SFDATA.BIN - yes, you need to dump your serial flash but you know how to do that, right? Then in your terminal:

Code: [Select]

cd ~/qemu
./run_canon_fw.sh 100D
Well? Did it work?

[Danne]

I wouldn't run rm -rf * on anything. What if the folder usr/local doesn't exist even?

[dfort]

usr/local

Make sure you're in "/usr/local" -- in any case I edited my post even though I did put in a warning about using "rm -rf *"

[DeafEyeJedi]

Thanks @Danne for the suggestions (especially on not using rm -rf * on anything) and @dfort for the comprehensive details. I will report my findings once this shitstorm of mine settles down this weekend.

This also reminds me that I should be looking for an old used MB Air to do this kind of work where there are no important files on the system for future references. 

[a1ex]

Latest qemu branch on a fresh Mac VM:

Virtualization 

Code: [Select]

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install mercurial

hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
hg update qemu -C
cd contrib/qemu
./install.sh



Thanks kichetof

[kichetof]

Thanks kichetof

Glad to help you    minor contribution for all your great works!

Happy to see Mac development on top level!

[dfort]

Looks like it is time to rebuild my QEMU environment. Thanks for making ML development on Mac so easy--well setting up the environment is easy, the rest is still pretty hard.

[henrus]

Thanks for all the effort setting up this tutorial. You guys are heros. I ran into an issue on my iMac running High Sierra (10.13.1). (Xcode 9.1 (9B55) installed but not used. From the terminal, after I proceeded with this step:

    brew tap homebrew/dupes/; brew install homebrew/versions/gcc5

I did receive the following error, though suspect it may not be a big deal:

    Error: Invalid tap name 'homebrew/dupes/'

The script went along fine, but then at this step:

    Hunk #1 succeeded at 228 with fuzz 2 (offset 12 lines).
    ==> ../configure --build=x86_64-apple-darwin17.2.0 --prefix=/usr/local/Cellar/gcc@5/5.5.0 --libdir=/usr/local/Cellar/gcc@5/5.5.0/lib/gcc/5 --enable-la
    ==> make bootstrap

everything seemed kind of stalled. The titlebar of my Terminal window was looping through multiple commands, but this kept going for like 10+ minutes. I wasn't sure if this was normal behavior or if it was stuck in some loop. I can post a screen video grab if needed. I ended up stopping all the processes and killing them. I read on a prior post here that someone said something took 37 minutes to complete, so perhaps it wasn't looping but I just need to wait for it to complete? If so, it would be useful to update the install instructions and let folks know that it may take a while to complete this step. Any confirmation would be helpful. Thanks.

[dfort]

Looks like there was another change at Homebrew:

Code: [Select]

Warning: Use gcc@5 instead of deprecated homebrew/versions/gcc5
I'll need to update the tutorial.

The stall you're experiencing is the compiler compiling a compiler from scratch. It takes a long time but if you let it keep running it should eventually finish and work. There should be a binary version available so that shouldn't be necessary. I'll update this topic when I get things working properly again.

[henrus]

Awesome. Thank you. By the way, while I was puttering around, I decided to try running the full script at the bottom of the article and that seemed to complete everything with no issues and without the long wait. Not sure why (perhaps most of the work was completed the first time through). All good now. Thanks for the quick response.

[dfort]

Glad you got it working. I reworked the script to accommodate that gcc@5 change in Homebrew and while I was at it improved the QEMU installation. The script can be run even if you have a development environment already setup, it won't clobber your current installation (I hope). 

[kichetof]

@dfort no need to symlink ggrep (https://bitbucket.org/hudson/magic-lantern/src/4895777de907c24ffd6332bcee23a7608450f6bd/contrib/qemu/install.sh?at=qemu&fileviewer=file-view-default#install.sh-103)

And if grep need -P, update script to use in same way 

Brew checks if Xcode commandline are already installed https://bitbucket.org/hudson/magic-lantern/commits/b145d5f6489d3d960d2a8d17a031748c5980684d?at=qemu

[dfort]

You're right -- removed the symlink. Thanks for the update.

Have you noticed that the script can now install QEMU from soup to nuts?

[DeafEyeJedi]

Ran the latest scripted installation from OP while on 10.13 w this MBP that already had previous compiling environments.

Love it that it can also install QEMU as an option towards the end. 

Seems all good to me even w the PIP update command that I ran (hence the in video above) afterwards w ease.

Thanks @dfort for the quick fix after @henrus's report. Good catch on that!

[dfort]

Great video @DeafEyeJedi - I like your choice of colors. 

[kichetof]

Have you noticed that the script can now install QEMU from soup to nuts?

Yeah! works well! 

[dariSSight]

HELP! I’m trying to sink my teeth into compiling but I’m hitting a snag, I’m at the part where you ask to modify Makefile.user.default. But I don’t see that file or all the different filed you ask us to edit as we go, so I’m kind of stuck after “brew install docutils”. I think I’m going to start over again but I just wanted some advice.

[Danne]

Are you running the quick installation script from the bottom of the first post?
The makefile you're referring to is in the root folder. Just copy it and rename it to Makefile.user and put in the suggested changes.

[dfort]

Actually, you don't even need to do that. The quick installation script does it for you:

Code: [Select]

    cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT

Here's a little secret -- you don't even need this for compiling ML. However, sooner or later you'll want to compile cr2hdr and you'll tear your hair out trying to figure out what is going on and it will always end up that you are missing this Makefile.user file.

[dfort]

Danne just published Compiler.app, a menu driven app that simplifies setting up and running an ML development environment.

He included the quick install script and started tweaking it to make it more user friendly. So instead of having the quick install script located at the bottom of the first post in this topic it is now on Bitbucket which means it is under full revision control and will be managed just like a "real" program. It can still be run as a stand alone script. Check the bottom of the first post to read how to download and run the script using the terminal.

[Levas]

Ok I screwed my compiling environment up    and got it working again 

But just wanted to check if this is going te be fine or if I can encounter new problems.

This happened
Yesterday I found this interesting program called 'butterflow' it can interpolate intermediate frames.
I wanted to test it how it works on 12.5 FPS UHD file, and if I can get it to 25fps with this program.
I'm not that savvy with this home-brew python stuff...
I'm on a Mac, so I typed this line in terminal:

Code: [Select]

brew install brewsci/science/butterflowA lot got installed on the way and in the end I could not get the programming to work.
Couldn't even get the help menu, this happens with 'butterflow -h' command:

Code: [Select]

Traceback (most recent call last):
  File "/usr/local/Cellar/butterflow/0.2.3_1/libexec/bin/butterflow", line 9, in <module>
    load_entry_point('butterflow==0.2.3', 'console_scripts', 'butterflow')()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 476, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2700, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2318, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/Cellar/butterflow/0.2.3_1/libexec/lib/python2.7/site-packages/butterflow/cli.py", line 11, in <module>
    from butterflow import ocl, avinfo, motion, ocl
ImportError: dlopen(/usr/local/Cellar/butterflow/0.2.3_1/libexec/lib/python2.7/site-packages/butterflow/avinfo.so, 2): Library not loaded: /usr/local/opt/ffmpeg/lib/libavcodec.57.dylib
  Referenced from: /usr/local/Cellar/butterflow/0.2.3_1/libexec/lib/python2.7/site-packages/butterflow/avinfo.so
  Reason: image not found

Too bad that I couldn't get it to work, BUT when I wanted to compile a new crop_rec module file from magic lantern, I got a new error, and it didn't compile the module.

Code: [Select]

/bin/sh: python2: command not found
After a lot of trial and error and reinstalling gcc and python (which gave messages that it was already there) I found out that there wasn't a file named 'python2' in my usr/bin/ directory.
and changed this line in the make.user.default file from:

Code: [Select]

PYTHON=python2changed to:

Code: [Select]

PYTHON=python
And now I can compile again, but the weird thing is, the butterflow installation couldn't mess with the make.user.file.
So apparently I was running on python 2 and the butterflow installation messed things up, maybe overwritten python files with newer or older versions ?
Is this fix I did a good solution, or is it a bit messy ?

[Danne]

Butterflow probably changed your python paths. What do you see when running this?

Code: [Select]

python -c "import sys; print('\n'.join(sys.path))"
Tried reinstalling python 2?

Code: [Select]

brew install python2Might ask to relink or so if somethig was messed up.