Author Topic: How to run Magic Lantern into QEMU?!...  (Read 121819 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #250 on: December 24, 2017, 10:21:49 AM »
Okay, sorted out, it was a typo when trying to bring QEMU window to foreground. This is still tricky, for example it doesn't work when starting multiple instances, but I'm probably doing this wrong - how do you open a window in foreground from command line on Mac?!

Also found a better way to tell whether the SD/CF images are mounted (or otherwise in use). On Mac, checking them with lsof is enough - this handles both images mounted with "hdiutil attach" and being in use by QEMU itself. On Linux (or at least on my system), lsof only handles the latter case, so checking with losetup (or grepping the output of "mount") is still needed to make sure the images are not in use by some other process when running the emulation.

BTW - already old news, but you can already navigate Canon menus on 1300D and 40D in QEMU.

Current state: 20 EOS models able to run the GUI in the emulator! These are:
5D2 5D3 6D 40D 50D 60D 70D 450D 500D 550D 600D 650D 700D 100D 1000D 1100D 1200D 1300D EOSM EOSM2.

t3r4n

  • New to the forum
  • *
  • Posts: 48
Re: How to run Magic Lantern into QEMU?!...
« Reply #251 on: December 25, 2017, 12:09:33 PM »
@a1ex asked about this one:
Noticed a message that I don't remember seeing on previous versions:

Code: [Select]
49:53: execution error: The variable qemu is not defined. (-2753)
Doesn't seem to hurt anything, just wondering if others are seeing it.

Don't remember seeing it; when/where does it appear, and in what color?

it happens on the 750D around this time in the boot, I tried to recreate the colours as they were (normal colour is qemu and red is camera?):
(sorry the code tag and colours don't mix)

[SF] InstallSerialFlash 2 0xd20b0d8c 0x0 0x800000 1

[ROM-DMA0] Copy [0xFD13A000] -> [0x407FFFA0], length [0x00001363], flags [0x11100003]
[XDMAC0] OK
[ROM-DMA0] Copy [0xFDCC0000] -> [0x40D6C000], length [0x0015B6AF], flags [0x11100003]
[XDMAC0] OK
    0:     3.328 [STARTUP]
K393 ICU Firmware Version 1.0.0 ( 8.7.2 )
     5:     4.864 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x20000

49:53: execution error: Die Variable „qemu“ ist nicht definiert. (-2753)
[ROM-DMA0] Copy [0xFD200000] -> [0x408020C0], length [0x000205E3], flags [0x11100003]
[XDMAC0] OK
[ROM-DMA0] Copy [0xFD320000] -> [0x40842160], length [0x0001AEDB], flags [0x11100003]
[XDMAC0] OK
[ROM-DMA0] Copy [0xFD360000] -> [0x40862200], length [0x0001DCA7], flags [0x11100003]
[XDMAC0] OK
[ROM-DMA1] Copy [0xFE744B88] -> [0xDFF00000], length [0x00002E77], flags [0x11100003]
[XDMAC1] OK
[MPU] Received: 06 04 02 00 00 00  (Init - spell #1)
[MPU] Sending : 2c 2a 02 00 03 03 03 04 03 00 00 48 00 00 00 14 50 00 00 00 00 81 06 00 00 04 06 00 00 04 06 00 00 04 01 01 00 00 00 00 4d 4b 01 00  (Init)
    9:    64.512 [PROPAD] ERROR SearchPropertyPackage DataType (0) = 0x01000000(L:3294)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #252 on: December 25, 2017, 10:20:12 PM »
Don't remember seeing it; when/where does it appear, and in what color?

It was showing up here, in black:

Code: [Select]
./run_canon_fw.sh EOSM,firmware=boot=0 -s -S &
arm-none-eabi-gdb -x EOSM/patches.gdb
...
Setting BOOTDISK flag to 0
49:53: execution error: The variable qemu is not defined. (-2753)
0xffff0000 in ?? ()

However, this disappeared on the latest commits so -- nevermind!

By the way, the EOSM doesn't launch into the GUI so I'm not able to run the sf_dump you asked for--unless there's another way of doing this without using the GUI.
5D3.* 7D.206 700D.115 EOSM.203

t3r4n

  • New to the forum
  • *
  • Posts: 48
Re: How to run Magic Lantern into QEMU?!...
« Reply #253 on: December 27, 2017, 01:11:24 PM »
Hey,
I've done two minor changes to the qemu install script to make installation of an updated version a bit easier. That was the small step  :).

I hope I did everything right by forking the repository, creating a new branch, uploading the commit and creating the pull request on the emu branch (the old sticky thread here has lost all images so I was guessing from the remaining text).

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #254 on: December 30, 2017, 12:58:19 AM »
Applied, thanks.

More cool stuff:

- lots of MPU properties documented, cross-checking welcome
- mode dial emulation (not perfect, but...)
- you can change some MPU-based properties, as long as the change is initiated from the main CPU *)

These are:
- shutter, aperture, ISO
- exposure compensation, flash exposure compensation
- metering mode, drive mode, AF mode
- picture style, white balance
- ExpSim, ALO, HTP, MLU.

*) That means, from ML menus/scripts or from the Q dialog, but not directly from scrollwheels. Some properties (shutter, aperture, AF points) are changed from the MPU (that is, the MPU is expected to interpret the button presses on its own and decide to change these parameters, but this behavior is not emulated). This is a bit model-specific, e.g. you can change shutter speed on 6D or 100D (where the change is made from the main CPU), but not on most other models.

If you can identify other properties that cannot be changed, please report.

It now runs a large part of api_test.lua! (all except photo capture, autofocus, half-shutter and LiveView tests). It even handles the shooting mode switch and those random exposure loops (with animation!)

Code: [Select]
diff -r f37efb4d8d53 scripts/api_test.lua
--- a/scripts/api_test.lua
+++ b/scripts/api_test.lua
@@ -237,13 +237,6 @@
     assert(camera.gui.play == true)
     assert(camera.gui.mode == 1)
 
-    -- half-shutter should exit playback mode
-    key.press(KEY.HALFSHUTTER)
-    msleep(1000)
-    assert(camera.gui.play == false)
-    assert(camera.gui.mode == 0)
-    key.press(KEY.UNPRESS_HALFSHUTTER)
-
     -- randomly switch between PLAY, MENU and IDLE (with or without LiveView)
     for i = 1,100 do
         -- we can request MENU or PLAY mode from anywhere
@@ -281,32 +274,8 @@
                 assert(camera.gui.mode == 0)
             end
         end
-
-        -- also play around with LiveView
-        if camera.gui.menu == false and camera.gui.play == false then
-            if math.random(1,2) == 1 then
-                -- do something with LiveView, but not as often as switching MENU/PLAY
-                if not lv.enabled then
-                    printf("Start LiveView...\n");
-                    lv.start()
-                elseif lv.paused then
-                    printf("Resume LiveView...\n");
-                    lv.resume()
-                elseif math.random(1,10) < 9 then
-                    -- this gets taken less often than the next one, why?
-                    -- fixme: biased random?
-                    printf("Pause LiveView...\n");
-                    lv.pause()
-                else
-                    printf("Stop LiveView...\n");
-                    lv.stop()
-                end
-            end
-        end
     end
 
-    lv.stop()
-
     printf("Canon GUI tests completed.\n")
     printf("\n")
 end
@@ -1341,14 +1310,9 @@
     test_io()
     test_camera_gui()
     test_menu()
-    test_camera_take_pics()
     msleep(1000)
     test_multitasking()
-    test_keys()
-    test_lv()
-    test_lens_focus()
     test_camera_exposure()
-    test_movie()
     
     printf("Done!\n")



8)

t3r4n

  • New to the forum
  • *
  • Posts: 48
Re: How to run Magic Lantern into QEMU?!...
« Reply #255 on: January 02, 2018, 09:51:32 PM »
Hi,
question:
When running qemu with the -d calls option is it possible to suppress certain calls? If I start a run into FROM Utility the poll serial io call fill up a log file with hundreds of MB quite fast.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #256 on: January 02, 2018, 11:03:47 PM »
Not directly, but you may use grep (the -C option helps to show calls around some keyword, MMIO register, other function). Or, by hardcoding some custom filters in logging.c.

Most of the time, I find the call stack more helpful than the (huge) call trace. To use that one, run with -d callstack and print it for any function you wish, like this:

Code: [Select]
b *0x1234
commands
  silent
  print_current_location_with_callstack
  printf "whatever message\n"
  c
end

BTW, just committed a bunch of doc updates, mostly with DryOS internals, debugging tips and similar stuff.

esas

  • New to the forum
  • *
  • Posts: 22
  • 70D.112, 650D.104, EOSM.202
Re: How to run Magic Lantern into QEMU?!...
« Reply #257 on: January 04, 2018, 07:27:53 PM »
The README was linked a few times, including first post (also asked for some proof-reading).

Think I found a small error. Under headline "Running Canon firmware" shouldn't there be a "-x" in there like this:
Code: [Select]
./run_canon_fw.sh EOSM,firmware="boot=0" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb
At least I couldn't get it to work before I found that comment inside the patches.gdb.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #258 on: January 04, 2018, 08:00:09 PM »
Thanks - that was after I was pretty sure I've checked that section a couple of times.

Exactly that's why I've asked for proof-reading - to make sure it works for new users who were not familiar with QEMU before.

escho

  • Hero Member
  • *****
  • Posts: 539
Re: How to run Magic Lantern into QEMU?!...
« Reply #259 on: January 13, 2018, 11:14:32 PM »
I´m trying to get ML running into qemu...
... but I run into some little problems

My system is openSUSE Thumbleweed, my cams are 6D and 600D

Starting install.sh from contrib/qemu tells me:

i
Code: [Select]
nstall.sh: Zeile 338: pip2: Kommando nicht gefunden.
install.sh: Zeile 339: pip2: Kommando nicht gefunden.
install.sh: Zeile 339: vncdotool: Kommando nicht gefunden.
install.sh: Zeile 339: pip2: Kommando nicht gefunden.
These messages correspond to these lines in the install script:

Code: [Select]
# install docutils (for compiling ML modules) and vncdotool (for test suite)
# only install if any of them is missing
pip2 list | grep docutils  || rst2html -h  > /dev/null || pip2 install docutils
pip2 list | grep vncdotool || vncdotool -h > /dev/null || pip2 install vncdotool

pip2 doesn´t exist on my system. So I would have to use pip or pip3. Which one should I prefer?

Since I compile ML, I have docutils installed via paketmanager (Yast). But what about vncdotool? I don´t find any pakets for it (not in the installed repos, not in openSUSE build service).
I´m a bit afraid to install vncdotool outside the paketmanager (using the pip-stuff in the script) and not be able to revert it, if something goes wrong.

Would you recommand to run a VM to play with this stuff?

A last question for the moment:

Why do you tell the user to add gcc/gdb-bins to PATH? Why not a little line in the script  for automation?:
Code: [Select]
export PATH=$PATH:~/$TOOLCHAIN/bin

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #260 on: January 13, 2018, 11:27:23 PM »
That's interesting... I'm also running Tumbleweed right now and I have: pip pip2 pip2.7 pip3 pip3.6

Maybe you have to install python2? It should also work with python3/pip3.

You don't really need vncdotool, unless you want to run the test suite (which actually requires a patched vncdotool, I should submit a PR) or the examples that use it.

If you run the "export PATH" command in a bash script, it will only be valid within that script. To run the examples from the guide, one has to have arm-none-eabi-gdb in PATH (or modify the command lines).

escho

  • Hero Member
  • *****
  • Posts: 539
Re: How to run Magic Lantern into QEMU?!...
« Reply #261 on: January 14, 2018, 11:33:45 AM »
Yes, interesting. "cnf pip2" tells me, that it´s in python2-pip, but python2-pip isn´t installed per default here in Thumbleweed. Got the script running with the two "pip2-lines" commented out.

But now, I´m hanging here:
Code: [Select]
./run_canon_fw.sh 6D,firmware="boot=1"I will look later, whats going on.

Thanks for your help, Alex

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #262 on: January 17, 2018, 06:22:16 AM »
Got a suggestion for the documentation.

After hours of struggling to get QEMU working again on Windows Subsystem for Linux (WSL) after a bunch of Microsoft and Ubuntu updates -- what is really important but not obvious is having to start the X server. BTW, VcXsrv isn't working here after the updates but Xming is and it is just as easy to install and run.

[EDIT] Oh yeah, don't forget this:

Code: [Select]
export DISPLAY=:0
5D3.* 7D.206 700D.115 EOSM.203

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #263 on: January 22, 2018, 06:16:38 AM »
My main development system is a Mac PowerBook and since much of what I've been running in QEMU is not on the qemu branch I've had to unmount the disk image manually. Doing "make install" would trigger the "Error: please unmount the SD image." even though the disk didn't appear as a mounted volumes. Turns out that the fix was to simply "eject" instead of "unmount" -- this works great with both physical cards and disk images. According to the hdiutil man page:

Quote
NOTE: unmount does NOT detach any disk image associated with the volume.
          Images are attached and detached; volumes are mounted and unmounted.

Pull request submitted on the qemu branch.
5D3.* 7D.206 700D.115 EOSM.203

esas

  • New to the forum
  • *
  • Posts: 22
  • 70D.112, 650D.104, EOSM.202
Re: How to run Magic Lantern into QEMU?!...
« Reply #264 on: January 23, 2018, 06:47:42 PM »
[EDIT] Oh yeah, don't forget this:

Code: [Select]
export DISPLAY=:0

And maybe hint on putting this in the $HOME/.bashrc so you don't have to type it every time you launch WSL.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #265 on: February 02, 2018, 11:44:59 AM »
Will do, need to revive somehow the Windows VM...

Meanwhile, I did one bold change that I wanted before it gets into mainline: moved the installation directory to qemu-eos (less likely to conflict with your existing vanilla qemu repository, if you've got one).

Options for migrating:
- rename the existing qemu directory to qemu-eos, then run the install script (I prefer this one)
- install from scratch, then manually copy the ROM files and other stuff you may have
- "export QEMU_DIR=qemu" to install in the previous location (or, if you prefer another name...)

The first steps towards this come from an early 100D contributor who was very upset about the default installation path...

BTW, make install_qemu now works from modules as well, and the README also got some nice updates (split in two, since it got too big for bitbucket to render properly):

README.rst - user-level documentation (installation, how to run, how to use it for debugging ML)
HACKING.rst - development and reverse engineering guide (also covering some introductory concepts I could not grasp a few years ago)

Next step: update the videos, fix the remaining OS-specific issues, and if all goes well, it's ready for mainline.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #266 on: February 02, 2018, 04:49:43 PM »
Got a problem installing a new qemu environment on the Mac. Seems that Apple and/or Homebrew pushed some docutils changes on me.

Here's the fix:

Code: [Select]
brew upgrade python
Didn't need to update docutils via pip -- go figure. Anyway, if you get this error message try updating python.

Code: [Select]
./install.sh: line 298: rst2html: command not found
Collecting docutils
  Using cached docutils-0.14-py2-none-any.whl
Installing collected packages: docutils
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber
    ensure_dir(destdir)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/docutils'
5D3.* 7D.206 700D.115 EOSM.203

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3123
Re: How to run Magic Lantern into QEMU?!...
« Reply #267 on: February 03, 2018, 01:59:23 AM »
Will do, need to revive somehow the Windows VM...
i set up a new VM on magicroot with WSL installed and prepared
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #268 on: April 12, 2018, 05:37:33 PM »
Found some interesting things about the 50D worth sharing. This probably also applies to the 5D2.

The ROM dumps saved by ML are 0x800000 in size but it appears that the second half of it is a mirror of the first half. This is something that I noticed in the 5D2 disassembly but the 50D ROM dump I was using was only 0x400000 in size so I never noticed this on the 50D until I helped @Asiertxu with some questions about one of my Digic 4 experiments. Since I don't have a 50D and my wife will kill me if I bid on yet another old camera on ebay I thought I'd run the 50D in QEMU. The 0x400000 ROMs didn't work but the the fresh dumps from the camera worked great:

Hola!


I made a QEMU beginner's mistake and loaded ML on the sd.img but of course the 50D only has a CF slot so make sure ML is loaded on the cf.img and voilà!



So my question is why work with two 0x800000 sized firmware dumps when this camera is apparently using only one 0x400000 file? More of a theoretical question rather than a practical one because there's no need to fix something that is working.
5D3.* 7D.206 700D.115 EOSM.203

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #269 on: April 12, 2018, 06:47:55 PM »
You can use the small dump if you also declare the right size in model_list.c. The current code was written for the one-size-fits-all ROM backup code that runs at ML startup, so you can use the files auto-saved into ML/LOGS on the card. However, that code is no longer valid for 1300D and D6 models (these have ROMs larger than 16MB), so it's probably a good idea to declare the ROM size, address and possibly RAM size as well, somewhere in platform directory (consts.h? internals.h?)

Unsure where the 8MB dumps come from though, as both the portable ROM dumper and the autobackup code from boot-hack.c save 16MB files, and ROM1 size is declared as 16MB for all DIGIC 4 models (one size fits all, as some models have smaller ROMs).

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #270 on: April 13, 2018, 03:00:03 AM »
Unsure where the 8MB dumps come from...

I might have gotten them from Silk Road before they were shut down  :P

Seriously though, I understand why someone would split a 5D2 or 50D ROM1.BIN in half before disassembling it. After all, those ROMs are 8MB, right? In addition, only the ROM1.BIN was archived because from my understanding on these and several other cameras the ROM0.BIN is just noise. Now we've got Digic 6 cameras, some with a big fat single ROM. Didn't realize any of this until I started playing around with QEMU.

I suppose that declaring the ROM size will also change the firmware signature. Did you see this issue with the firmware signature on the 600D firmware update? QEMU to the rescue!
5D3.* 7D.206 700D.115 EOSM.203

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #271 on: April 13, 2018, 07:06:11 AM »
OK, but since the ROM size is declared as 16MB in QEMU, you'll get a warning at startup and the memory layout won't be identical to the one from a real camera (the ROM copies will no longer be "complete"). In particular, with an 8MB ROM, no matter what you put into it, QEMU won't cover the bootloader!

BTW, g3gg0 has a long answer why the ROM is mirrored like that.

On 5D2, ROM1 size (actually ROM0 in Canon strings; I just used g3gg0's notation) is 8MB (if you look at 16MB ROM, you see two mirrored copies, but if you trim it to 8MB, you get unique data in both halves). However, ROM0 is just 4MB on 5D2.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3145
Re: How to run Magic Lantern into QEMU?!...
« Reply #272 on: May 01, 2018, 05:05:07 AM »
Something I've been seeing on the Mac when compiling on some of the branches.

From the ~/qemu-eos directory I'm compiling this way:

Code: [Select]
make -C ../magic-lantern 6D_install_qemu
And this scrolls by:

Code: [Select]
[ SCRIPTS  ]   install_extra_data
for target in  CONFIG_QEMU_install  CONFIG_MODULES_install; do /Library/Developer/CommandLineTools/usr/bin/make $target; done
/Library/Developer/CommandLineTools/usr/bin/make -C qemu-eos
make: *** qemu-eos: No such file or directory.  Stop.
make[2]: *** [CONFIG_QEMU_compile] Error 2
/Library/Developer/CommandLineTools/usr/bin/make -C ../../modules

It doesn't seem to cause any problems but it does say Error 2 so thought I'd report it.
5D3.* 7D.206 700D.115 EOSM.203

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11717
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #273 on: May 01, 2018, 12:37:02 PM »
Appears to be related to this commit.

P.S. emulation ready for DIGIC 7 EOS models (most of the things were very similar to DIGIC 6).



Here's a proof of concept I'm experimenting with, written as a helper script to keep up with all these firmware updates. It finds a few common stubs from an initial test run, and creates an (updated) debugmsg.gdb.

Code: [Select]
#!/bin/bash
# Finds some common stubs from an initial test run of some EOS firmware in QEMU.
# Usage: ./find_stubs.sh 6D

CAM="$1"    # camera model
FW="$2"     # firmware version (optional)
DELAY=5     # time (in seconds) for the test run
GREP=${GREP:=grep}

[ "$FW" == "" ] && DEBUGMSG_GDB="$CAM/debugmsg.gdb" || DEBUGMSG_GDB="$CAM/$FW/debugmsg.gdb"

if [ $(uname) == "Darwin" ]; then
    if [[ -n $(which ggrep) ]]; then
        GREP=ggrep
    else
        echo
        echo "Error: you need GNU grep to run this script"
        echo "brew install grep"
        exit 1
    fi

    # Mac doesn't like piping the output of run_canon_fw.sh to other commands (why?!)
    echo "Test run..." 1>&2
    test_run=$( (sleep $DELAY; echo quit) | \
        ./run_canon_fw.sh $CAM,firmware="$FW;boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log \
        2>&1 )

    test_run=$( echo "$test_run" | ansi2txt )

else # not Mac

    # the above two-step piping would print a warning on Linux
    echo "Test run..." 1>&2
    test_run=$( (sleep $DELAY; echo quit) | \
        ./run_canon_fw.sh $CAM,firmware="$FW;boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log \
        2>&1 | ansi2txt )

fi

# print firmware version to stderr
( (cat uart.log | $GREP -a -m1 "Firmware Version") || (echo "$test_run" | $GREP -o -m1 '[^"]*Firmware Version [0-9][^"]*') ) 1>&2
echo 1>&2

function clear_line {
    echo -en "\r                                                                                \r"
}

# extract the called function from a line that looks like this:
# call 0x1234(...)
# call 0x1234 DebugMsg(...)
function extract_call {
    head -n1 | $GREP -oP '(?<=call ).*?(?=\()' | cut -d ' ' -f 1
}

# extract the address of a direct jump
# fixme: does not handle multiple direct jumps
# call 0x1234(...) at [init:4444:8888]
#  -> 0xFF1234     at [init:1234:4448]
function extract_jump {
    head -n2 | tail -n1 | $GREP -oP '(?<=-> ).*?(?= +at )' | cut -d ' ' -f 1
}

# looks for a function called with one of the arguments
# returns first match
function find_stub_from_strings {
    for str in "[email protected]"; do

        # status indicator
        clear_line 1>&2
        echo -en "\r$str ..." 1>&2

        # any direct jump? use that
        local stub=$(echo "$test_run" \
                        | $GREP -a -m1 -A1 "$str" \
                        | extract_jump)
        if [ "$stub" != "" ]; then
            echo $stub
            return
        fi

        # regular call?
        local stub=$(echo "$test_run" \
                        | $GREP -a -m1 "$str" \
                        | extract_call)
        if [ "$stub" != "" ]; then
            echo $stub
            return
        fi
    done
}

# fallback - if a stub cannot be found, get it from the existing GDB script
function get_stub_from_gdb_script {
    cat $DEBUGMSG_GDB | $GREP ${1}_log -B 1 | $GREP -Pom1 "(?<=b \*)0x.*"
}

# find some common stubs
DebugMsg=$(find_stub_from_strings startupEntry startupEventDispatch DisablePowerSave)
task_create=$(find_stub_from_strings TaskMain "[a-z]Task" systemtask CmdShell EvShel HotPlug PowerMgr PowerMan)
register_interrupt=$(find_stub_from_strings ICAPCHx OC4_14 SIO3_ISR)
CreateStateObject=$(find_stub_from_strings DMState EMState PropState SRMState)
create_semaphore=$(find_stub_from_strings PropSem mallocLock stdioLock dm_lock)
create_msg_queue=$(find_stub_from_strings MainMessQueue QueueForDeviceIn SystemTaskMSGQueue)

register_func=$(echo "$test_run" \
    | $GREP -a flashwrite \
    | $GREP -B1 -m1 NameService \
    | extract_call)

SIO3_ISR=$(echo "$test_run" | $GREP -a -i SIO3_ISR \
    | $GREP -o SIO3_ISR.* \
    | cut -d ',' -f 3 | tr -d ' ')

MREQ_ISR=$(echo "$test_run" | $GREP -a -i MREQ_ISR \
    | $GREP -o MREQ_ISR.* \
    | cut -d ',' -f 3 | tr -d ' ')

# GDB breakpoints must be without the Thumb bit, if any
function clr_thumb {
    param=$1
    if [[ "$param" != 0x* ]]; then
        param="0x$param"
    fi
    printf "0x%X\n" $((param & ~1))
}

# print a GDB logging stub
# fall back by taking it from the existing stub
function print_logging_stub {
    param=$1

    if [ "${!param}" != "" ]; then
        echo "b *$(clr_thumb ${!param})"
        echo "${param}_log"
    else
        local stub=$(get_stub_from_gdb_script $param)
        if [ "$stub" != "" ]; then
            echo "# from $DEBUGMSG_GDB"
            echo "b *$stub"
            echo "${param}_log"
        else
            echo "# not found"
            echo "# b *0x..."
            echo "# ${param}_log"
        fi
    fi
    echo
}

function print_commented_stub {
    param=$1
    if [ "${!param}" != "" ]; then
        echo "# $(clr_thumb ${!param}) ${param}"
    fi
}

# clear any previous status message
clear_line 1>&2

# output the GDB script
echo                        1>&2
echo "$DEBUGMSG_GDB"        1>&2
echo "====================" 1>&2
echo                        1>&2

echo "# ./run_canon_fw.sh $1 -d debugmsg"
echo "# ./run_canon_fw.sh $1 -d debugmsg -s -S & arm-none-eabi-gdb -x $DEBUGMSG_GDB"
echo
echo "source -v debug-logging.gdb"
echo
cat $DEBUGMSG_GDB | $GREP "To get debugging symbols"
cat $DEBUGMSG_GDB | $GREP "symbol-file"
echo
cat $DEBUGMSG_GDB | $GREP "macro define"
echo

if [ "$DebugMsg" != "" ]; then
    echo "# GDB hook is very slow; -d debugmsg is much faster"
    echo "# ./run_canon_fw.sh will use this address, don't delete it"
    echo "# b *$(clr_thumb $DebugMsg)"
    echo "# DebugMsg_log"
    echo
fi

print_logging_stub task_create
print_logging_stub assert
print_logging_stub register_interrupt
print_logging_stub register_func
print_logging_stub mpu_send
print_logging_stub mpu_recv
print_logging_stub create_semaphore
print_logging_stub create_msg_queue
print_logging_stub CreateStateObject

print_commented_stub SIO3_ISR
print_commented_stub MREQ_ISR


# finished
echo
echo cont

These stubs include DebugMsg, task_create, register_func, register_interrupt and CreateStateObject. Run the firmware in GDB using the generated script to get a few hundreds (possibly thousands) named functions.

Refer to Initial firmware analysis for a sample workflow (this script automates step 5).

t3r4n

  • New to the forum
  • *
  • Posts: 48
Re: How to run Magic Lantern into QEMU?!...
« Reply #274 on: May 01, 2018, 01:33:53 PM »
for us mac fanboys

replace
Code: [Select]
|& with
Code: [Select]
2>&1 | in the script

go to source forge, download ansi2txt
untar and change the directories in Makefile to /usr/local/....
Code: [Select]
make && make installl

Code: [Select]
brew install grep
and replace grep with ggrep in the script

this should get the script running....