Running ARM-console on a Mac

[dfort]

I'm following these instructions:

http://magiclantern.wikia.com/wiki/Magic_Lantern_Development_on_Mac

Seems like I'm so close but stuck on this:

Code Select Expand

http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console
You might want to do this:
    D = load_dumps("autoexec")
------------------------------------------------------------
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    console.ipshell()
  File "/usr/local/lib/python2.7/site-packages/IPython/Shell.py", line 242, in __call__
    self.IP.embed_mainloop(banner,local_ns,global_ns,stack_depth=1)
  File "/usr/local/lib/python2.7/site-packages/IPython/iplib.py", line 1836, in embed_mainloop
    self.set_completer_frame()
  File "/usr/local/lib/python2.7/site-packages/IPython/iplib.py", line 1267, in set_completer_frame
    self.Completer.namespace = self.user_ns
AttributeError: 'InteractiveShell' object has no attribute 'Completer'


Any hints or should I just fire up the old Linux laptop for this?

[dfort]

More information. Completer is installed with the readline library but:

Code Select Expand

WARNING: Readline services not available on this platform.
WARNING: The auto-indent feature requires the readline library


[dfort]

Ok--figured it out. Here it is for other Mac users that find themselves in the same situation.

Download readline-6.2.4.1.tar.gz from here:

https://pypi.python.org/pypi/readline#downloads

Unpack it and run:

Code Select Expand

sudo python setup.py install

That's it. Yes, this is a depreciated package but then again the instructions for ARM_console specifies installing an old version of ipython-0.10.

[DeafEyeJedi]

Excellent news, Dan! Got me going on this as well and Thanks for starting yet another noteworthy thread!

Question though -- do we go ahead and download the latest GNU ARM (5_4-2016q3-20160926) from their site or use the one that's from 2013-14 (version 4_8-2013q4-20131218) or does it not matter?

*edit*

Actually decided to go with the latest (I know it's not a good idea to take risk but hey I can't help myself) and here's my progress so far... Stand by!

Code Select Expand

Last login: Thu Dec 15 22:48:35 on console
Seans-Mac-mini-385:~ DeafEyeJedi$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
==> The following existing directories will be made group writable:
/usr/local/sbin
==> The following existing directories will have their owner set to DeafEyeJedi:
/usr/local/sbin
==> The following existing directories will have their group set to admin:
/usr/local/sbin
==> The following new directories will be created:
/usr/local/Homebrew
/usr/local/var

Press RETURN to continue or any other key to abort
==> /usr/bin/sudo /bin/chmod u+rwx /usr/local/sbin
Password:
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/sbin
==> /usr/bin/sudo /usr/sbin/chown DeafEyeJedi /usr/local/sbin
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/sbin
==> /usr/bin/sudo /bin/mkdir -p /usr/local/Homebrew /usr/local/var
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/Homebrew /usr/local/var
==> /usr/bin/sudo /bin/chmod 755 /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /usr/sbin/chown DeafEyeJedi /usr/local/Homebrew /usr/local/var
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/Homebrew /usr/local/var
==> /usr/bin/sudo /bin/mkdir -p /Users/SMJ/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Users/SMJ/Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown DeafEyeJedi /Users/SMJ/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/mkdir -p /Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown DeafEyeJedi /Library/Caches/Homebrew
==> Searching online for the Command Line Tools
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> Installing Command Line Tools (macOS Sierra version 10.12) for Xcode-8.2
==> /usr/bin/sudo /usr/sbin/softwareupdate -i Command\ Line\ Tools\ (macOS\ Sierra\ version\ 10.12)\ for\ Xcode-8.2
Software Update Tool
Copyright 2002-2015 Apple Inc.


Downloading Command Line Tools (macOS Sierra version 10.12) for Xcode
Downloaded Command Line Tools (macOS Sierra version 10.12) for Xcode
Installing Command Line Tools (macOS Sierra version 10.12) for Xcode
Done with Command Line Tools (macOS Sierra version 10.12) for Xcode
Done.
==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
==> Downloading and installing Homebrew...
remote: Counting objects: 4235, done.
remote: Compressing objects: 100% (2795/2795), done.
remote: Total 4235 (delta 2179), reused 2729 (delta 1292), pack-reused 0
Receiving objects: 100% (4235/4235), 2.44 MiB | 4.64 MiB/s, done.
Resolving deltas: 100% (2179/2179), done.
From https://github.com/Homebrew/brew
* [new branch]      master     -> origin/master
* [new tag]         0.1        -> 0.1
* [new tag]         0.2        -> 0.2
* [new tag]         0.3        -> 0.3
* [new tag]         0.4        -> 0.4
* [new tag]         0.5        -> 0.5
* [new tag]         0.6        -> 0.6
* [new tag]         0.7        -> 0.7
* [new tag]         0.7.1      -> 0.7.1
* [new tag]         0.8        -> 0.8
* [new tag]         0.8.1      -> 0.8.1
* [new tag]         0.9        -> 0.9
* [new tag]         0.9.1      -> 0.9.1
* [new tag]         0.9.2      -> 0.9.2
* [new tag]         0.9.3      -> 0.9.3
* [new tag]         0.9.4      -> 0.9.4
* [new tag]         0.9.5      -> 0.9.5
* [new tag]         0.9.8      -> 0.9.8
* [new tag]         0.9.9      -> 0.9.9
* [new tag]         1.0.0      -> 1.0.0
* [new tag]         1.0.1      -> 1.0.1
* [new tag]         1.0.2      -> 1.0.2
* [new tag]         1.0.3      -> 1.0.3
* [new tag]         1.0.4      -> 1.0.4
* [new tag]         1.0.5      -> 1.0.5
* [new tag]         1.0.6      -> 1.0.6
* [new tag]         1.0.7      -> 1.0.7
* [new tag]         1.0.8      -> 1.0.8
* [new tag]         1.0.9      -> 1.0.9
* [new tag]         1.1.0      -> 1.1.0
* [new tag]         1.1.1      -> 1.1.1
* [new tag]         1.1.2      -> 1.1.2
* [new tag]         1.1.3      -> 1.1.3
* [new tag]         1.1.4      -> 1.1.4
HEAD is now at 9cd5a21 Merge pull request #1669 from vladshablinsky/fix_method_deprecated
==> Tapping homebrew/core
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core'...
remote: Counting objects: 3788, done.
remote: Compressing objects: 100% (3670/3670), done.
remote: Total 3788 (delta 19), reused 318 (delta 6), pack-reused 0
Receiving objects: 100% (3788/3788), 3.03 MiB | 4.63 MiB/s, done.
Resolving deltas: 100% (19/19), done.
Tapped 3664 formulae (3,817 files, 9.4M)
==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/SMJ/Library/Caches/Homebrew...
==> Deleting /Library/Caches/Homebrew...
Already up-to-date.
Error: Could not link:
/usr/local/share/doc/homebrew

Please delete these paths and run `brew update`.
Error: Could not link:
/usr/local/share/man/man1/brew.1

Please delete these paths and run `brew update`.
==> Installation successful!

==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
  https://git.io/brew-analytics

==> Next steps:
- Run `brew help` to get started
- Further documentation:
    https://git.io/brew-docs
Seans-Mac-mini-385:~ DeafEyeJedi$


Okay now I've ran into a wall in here (even tho it looks like an easy fix) but since I am using my Mac Mini which has been troublesome in the past when it comes to compiling for ML.

Code Select Expand

Seans-Mac-mini-385:~ DeafEyeJedi$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/Noesis.dylib

Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  gcc
  gdbm
  gmp
  isl
  libmpc
  mercurial
  mpfr
  python
  wget

Warning: Homebrew's sbin was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting the PATH for example like so
  echo 'export PATH="/usr/local/sbin:$PATH"' >> ~/.bash_profile

Warning: Broken symlinks were found. Remove them with `brew prune`:
  /usr/local/bin/truncate
Seans-Mac-mini-385:~ DeafEyeJedi$ brew prune
Pruned 1 symbolic links and 2 directories from /usr/local
Seans-Mac-mini-385:~ DeafEyeJedi$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!

Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

Unexpected dylibs:
  /usr/local/lib/Noesis.dylib

Warning: You have unlinked kegs in your Cellar
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  gcc
  gdbm
  gmp
  isl
  libmpc
  mercurial
  mpfr
  python
  wget

Warning: Homebrew's sbin was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting the PATH for example like so
  echo 'export PATH="/usr/local/sbin:$PATH"' >> ~/.bash_profile
Seans-Mac-mini-385:~ DeafEyeJedi$

So I think I should move on and do this tutorial again on the MBP since it has been kind to me when it comes to this environment. 

[dfort]

So you're going for it? I suggest you start here:

I like to use the HTML pages generated by ARM Console. Getting it setup is no easy task and it make take several days or more to run.

That's an experienced developer saying it is no easy task. Still going for it? Ok, it was very challenging for me. I'm still not sure if I got it set up properly, I couldn't load the stubs.S file but started rendering the HTML pages last night. This morning it is still running. How long will it take?

A full analysis of ML firmware takes 1-2 minutes. The same analyses for the 550D firmware takes around 1 day, or less if you help me optimize the algorithms

If you can leave the computer on for a week, just run this to analyze all your dumps:

Still going for it? Ok, on the Mac you need to follow these directions first:
http://magiclantern.wikia.com/wiki/Magic_Lantern_Development_on_Mac

Some of the python packages can be installed via pip and others need to be built manually including ipython-0.10. Note that once you get past the "test plot with matplotlib, just making sure things are working correctly" you aren't out of the woods. You need to get readline working, see my last post, and you need to put the ARM compiler in your path.

Question though -- do we go ahead and download the latest GNU ARM (5_4-2016q3-20160926) from their site or use the one that's from 2013-14 (version 4_8-2013q4-20131218) or does it not matter?

Yes, it matters. You need to use the same version you're using to compile ML and the objcopy and objdump executables need to be in your path. The way I did it was to put copies of ~/gcc-arm-none-eabi-4_8-2013q4/arm-none-eabi/arm-none-eabi-objcopy and arm-none-eabi-objdump in my /usr/local/bin directory.

Still going for it? Getting frustrated yet?

If you want to get a quick disassembly to get you started I'd suggest following the Tutorial: finding stubs and use disassemble.pl. The reason I'm going through the exercise of installing ARM-console is because on my disassembly file I seem to be missing a pointer dmilligan used to find the EOSM DEFAULT_RAW_BUFFER address. That's a lot of work to go just to try and find something that has already been solved but understanding the process might help resolve other issues that may come up.

Whatever reason you may have to go for it--Good luck!

[dfort]

Got the stubs.S file to load, just needed to get more familiar with ARM-console.

Note: for the 550D - renamed ROM1.BIN to 550d.109.0xff000000.BIN and stubs.S to stubs-550d.109.S

[EDIT] Ok, the 550D stubs load but not the EOSM stubs. Go figure.

[dfort]

Thought I'd post an update in case anyone else wants to attempt this. I started three parallel full disassemblies on a MacBook Pro, 2.3 GHz Intel Core i7, 5 days ago. Today I saw this message:

Code Select Expand

Function analysis... [27% done, ETA 12 days, 12:47:49]...
ASM function: sub_FF2B1C34 at 0xff2b1c34 in eosm.202.0xff000000.bin


A couple of days ago I saw a message on the 550D process that had an ETA of 4 days. Any bets on which one is the most accurate? Any bets on whether or not I do something stupid that will require a reboot before this is finished and I'll have to start over?

Just a guess at why the 550D stubs loaded and not the EOSM or 100D--the 550D doesn't have a RAM_OFFSET defined?

[dmilligan]

Trying to run 3 at once might not have been the best idea. I'm pretty sure even a single instance can use just about every bit of ram you have available.

[dfort]

Trying to run 3 at once might not have been the best idea.

Well it does turn the MacBook into a pretty effective space heater. Guess I've either got to kill a couple of processes or take a two week vacation.

[DeafEyeJedi]

Take a two week vacation!