How to run Magic Lantern into QEMU?!...

Started by jplxpto, September 23, 2012, 08:29:02 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.


Okay, sorted out, it was a typo when trying to bring QEMU window to foreground. This is still tricky, for example it doesn't work when starting multiple instances, but I'm probably doing this wrong - how do you open a window in foreground from command line on Mac?!

Also found a better way to tell whether the SD/CF images are mounted (or otherwise in use). On Mac, checking them with lsof is enough - this handles both images mounted with "hdiutil attach" and being in use by QEMU itself. On Linux (or at least on my system), lsof only handles the latter case, so checking with losetup (or grepping the output of "mount") is still needed to make sure the images are not in use by some other process when running the emulation.

BTW - already old news, but you can already navigate Canon menus on 1300D and 40D in QEMU.

Current state: 20 EOS models able to run the GUI in the emulator! These are:
5D2 5D3 6D 40D 50D 60D 70D 450D 500D 550D 600D 650D 700D 100D 1000D 1100D 1200D 1300D EOSM EOSM2.


@a1ex asked about this one:
Quote from: dfort on December 23, 2017, 08:22:48 AM
Noticed a message that I don't remember seeing on previous versions:

49:53: execution error: The variable qemu is not defined. (-2753)

Doesn't seem to hurt anything, just wondering if others are seeing it.

Quote from: a1ex on December 23, 2017, 08:25:16 AM
Don't remember seeing it; when/where does it appear, and in what color?

it happens on the 750D around this time in the boot, I tried to recreate the colours as they were (normal colour is qemu and red is camera?):
(sorry the code tag and colours don't mix)

[SF] InstallSerialFlash 2 0xd20b0d8c 0x0 0x800000 1

[ROM-DMA0] Copy [0xFD13A000] -> [0x407FFFA0], length [0x00001363], flags [0x11100003]
[ROM-DMA0] Copy [0xFDCC0000] -> [0x40D6C000], length [0x0015B6AF], flags [0x11100003]
    0:     3.328 [STARTUP]
K393 ICU Firmware Version 1.0.0 ( 8.7.2 )
     5:     4.864 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x20000

49:53: execution error: Die Variable ,,qemu" ist nicht definiert. (-2753)
[ROM-DMA0] Copy [0xFD200000] -> [0x408020C0], length [0x000205E3], flags [0x11100003]
[ROM-DMA0] Copy [0xFD320000] -> [0x40842160], length [0x0001AEDB], flags [0x11100003]
[ROM-DMA0] Copy [0xFD360000] -> [0x40862200], length [0x0001DCA7], flags [0x11100003]
[ROM-DMA1] Copy [0xFE744B88] -> [0xDFF00000], length [0x00002E77], flags [0x11100003]
[MPU] Received: 06 04 02 00 00 00  (Init - spell #1)
[MPU] Sending : 2c 2a 02 00 03 03 03 04 03 00 00 48 00 00 00 14 50 00 00 00 00 81 06 00 00 04 06 00 00 04 06 00 00 04 01 01 00 00 00 00 4d 4b 01 00  (Init)
    9:    64.512 [PROPAD] ERROR SearchPropertyPackage DataType (0) = 0x01000000(L:3294)


Quote from: a1ex on December 23, 2017, 08:25:16 AM
Don't remember seeing it; when/where does it appear, and in what color?

It was showing up here, in black:

./ EOSM,firmware=boot=0 -s -S &
arm-none-eabi-gdb -x EOSM/patches.gdb
Setting BOOTDISK flag to 0
49:53: execution error: The variable qemu is not defined. (-2753)
0xffff0000 in ?? ()

However, this disappeared on the latest commits so -- nevermind!

By the way, the EOSM doesn't launch into the GUI so I'm not able to run the sf_dump you asked for--unless there's another way of doing this without using the GUI.


I've done two minor changes to the qemu install script to make installation of an updated version a bit easier. That was the small step  :).

I hope I did everything right by forking the repository, creating a new branch, uploading the commit and creating the pull request on the emu branch (the old sticky thread here has lost all images so I was guessing from the remaining text).


Applied, thanks.

More cool stuff:

- lots of MPU properties documented, cross-checking welcome
- mode dial emulation (not perfect, but...)
- you can change some MPU-based properties, as long as the change is initiated from the main CPU *)

These are:
- shutter, aperture, ISO
- exposure compensation, flash exposure compensation
- metering mode, drive mode, AF mode
- picture style, white balance
- ExpSim, ALO, HTP, MLU.

*) That means, from ML menus/scripts or from the Q dialog, but not directly from scrollwheels. Some properties (shutter, aperture, AF points) are changed from the MPU (that is, the MPU is expected to interpret the button presses on its own and decide to change these parameters, but this behavior is not emulated). This is a bit model-specific, e.g. you can change shutter speed on 6D or 100D (where the change is made from the main CPU), but not on most other models.

If you can identify other properties that cannot be changed, please report.

It now runs a large part of api_test.lua! (all except photo capture, autofocus, half-shutter and LiveView tests). It even handles the shooting mode switch and those random exposure loops (with animation!)

diff -r f37efb4d8d53 scripts/api_test.lua
--- a/scripts/api_test.lua
+++ b/scripts/api_test.lua
@@ -237,13 +237,6 @@
     assert( == true)
     assert(camera.gui.mode == 1)

-    -- half-shutter should exit playback mode
-    msleep(1000)
-    assert( == false)
-    assert(camera.gui.mode == 0)
     -- randomly switch between PLAY, MENU and IDLE (with or without LiveView)
     for i = 1,100 do
         -- we can request MENU or PLAY mode from anywhere
@@ -281,32 +274,8 @@
                 assert(camera.gui.mode == 0)
-        -- also play around with LiveView
-        if == false and == false then
-            if math.random(1,2) == 1 then
-                -- do something with LiveView, but not as often as switching MENU/PLAY
-                if not lv.enabled then
-                    printf("Start LiveView...\n");
-                    lv.start()
-                elseif lv.paused then
-                    printf("Resume LiveView...\n");
-                    lv.resume()
-                elseif math.random(1,10) < 9 then
-                    -- this gets taken less often than the next one, why?
-                    -- fixme: biased random?
-                    printf("Pause LiveView...\n");
-                    lv.pause()
-                else
-                    printf("Stop LiveView...\n");
-                    lv.stop()
-                end
-            end
-        end

-    lv.stop()
     printf("Canon GUI tests completed.\n")
@@ -1341,14 +1310,9 @@
-    test_camera_take_pics()
-    test_keys()
-    test_lv()
-    test_lens_focus()
-    test_movie()



When running qemu with the -d calls option is it possible to suppress certain calls? If I start a run into FROM Utility the poll serial io call fill up a log file with hundreds of MB quite fast.


Not directly, but you may use grep (the -C option helps to show calls around some keyword, MMIO register, other function). Or, by hardcoding some custom filters in logging.c.

Most of the time, I find the call stack more helpful than the (huge) call trace. To use that one, run with -d callstack and print it for any function you wish, like this:

b *0x1234
  printf "whatever message\n"

BTW, just committed a bunch of doc updates, mostly with DryOS internals, debugging tips and similar stuff.


Quote from: a1ex on November 12, 2017, 08:46:39 PM
The README was linked a few times, including first post (also asked for some proof-reading).

Think I found a small error. Under headline "Running Canon firmware" shouldn't there be a "-x" in there like this:
./ EOSM,firmware="boot=0" -s -S & arm-none-eabi-gdb -x EOSM/patches.gdb

At least I couldn't get it to work before I found that comment inside the patches.gdb.


Thanks - that was after I was pretty sure I've checked that section a couple of times.

Exactly that's why I've asked for proof-reading - to make sure it works for new users who were not familiar with QEMU before.


I´m trying to get ML running into qemu...
... but I run into some little problems

My system is openSUSE Thumbleweed, my cams are 6D and 600D

Starting from contrib/qemu tells me: Zeile 338: pip2: Kommando nicht gefunden. Zeile 339: pip2: Kommando nicht gefunden. Zeile 339: vncdotool: Kommando nicht gefunden. Zeile 339: pip2: Kommando nicht gefunden.

These messages correspond to these lines in the install script:

# install docutils (for compiling ML modules) and vncdotool (for test suite)
# only install if any of them is missing
pip2 list | grep docutils  || rst2html -h  > /dev/null || pip2 install docutils
pip2 list | grep vncdotool || vncdotool -h > /dev/null || pip2 install vncdotool

pip2 doesn´t exist on my system. So I would have to use pip or pip3. Which one should I prefer?

Since I compile ML, I have docutils installed via paketmanager (Yast). But what about vncdotool? I don´t find any pakets for it (not in the installed repos, not in openSUSE build service).
I´m a bit afraid to install vncdotool outside the paketmanager (using the pip-stuff in the script) and not be able to revert it, if something goes wrong.

Would you recommand to run a VM to play with this stuff?

A last question for the moment:

Why do you tell the user to add gcc/gdb-bins to PATH? Why not a little line in the script  for automation?:
export PATH=$PATH:~/$TOOLCHAIN/bin
600D, 6D, openSUSE Tumbleweed


That's interesting... I'm also running Tumbleweed right now and I have: pip pip2 pip2.7 pip3 pip3.6

Maybe you have to install python2? It should also work with python3/pip3.

You don't really need vncdotool, unless you want to run the test suite (which actually requires a patched vncdotool, I should submit a PR) or the examples that use it.

If you run the "export PATH" command in a bash script, it will only be valid within that script. To run the examples from the guide, one has to have arm-none-eabi-gdb in PATH (or modify the command lines).


Yes, interesting. "cnf pip2" tells me, that it´s in python2-pip, but python2-pip isn´t installed per default here in Thumbleweed. Got the script running with the two "pip2-lines" commented out.

But now, I´m hanging here:
./ 6D,firmware="boot=1"
I will look later, whats going on.

Thanks for your help, Alex
600D, 6D, openSUSE Tumbleweed


Got a suggestion for the documentation.

After hours of struggling to get QEMU working again on Windows Subsystem for Linux (WSL) after a bunch of Microsoft and Ubuntu updates -- what is really important but not obvious is having to start the X server. BTW, VcXsrv isn't working here after the updates but Xming is and it is just as easy to install and run.

[EDIT] Oh yeah, don't forget this:

export DISPLAY=:0


My main development system is a Mac PowerBook and since much of what I've been running in QEMU is not on the qemu branch I've had to unmount the disk image manually. Doing "make install" would trigger the "Error: please unmount the SD image." even though the disk didn't appear as a mounted volumes. Turns out that the fix was to simply "eject" instead of "unmount" -- this works great with both physical cards and disk images. According to the hdiutil man page:

QuoteNOTE: unmount does NOT detach any disk image associated with the volume.
          Images are attached and detached; volumes are mounted and unmounted.

Pull request submitted on the qemu branch.


Quote from: dfort on January 17, 2018, 06:22:16 AM
[EDIT] Oh yeah, don't forget this:

export DISPLAY=:0

And maybe hint on putting this in the $HOME/.bashrc so you don't have to type it every time you launch WSL.


Will do, need to revive somehow the Windows VM...

Meanwhile, I did one bold change that I wanted before it gets into mainline: moved the installation directory to qemu-eos (less likely to conflict with your existing vanilla qemu repository, if you've got one).

Options for migrating:
- rename the existing qemu directory to qemu-eos, then run the install script (I prefer this one)
- install from scratch, then manually copy the ROM files and other stuff you may have
- "export QEMU_DIR=qemu" to install in the previous location (or, if you prefer another name...)

The first steps towards this come from an early 100D contributor who was very upset about the default installation path...

BTW, make install_qemu now works from modules as well, and the README also got some nice updates (split in two, since it got too big for bitbucket to render properly):

README.rst - user-level documentation (installation, how to run, how to use it for debugging ML)
HACKING.rst - development and reverse engineering guide (also covering some introductory concepts I could not grasp a few years ago)

Next step: update the videos, fix the remaining OS-specific issues, and if all goes well, it's ready for mainline.


Got a problem installing a new qemu environment on the Mac. Seems that Apple and/or Homebrew pushed some docutils changes on me.

Here's the fix:

brew upgrade python

Didn't need to update docutils via pip -- go figure. Anyway, if you get this error message try updating python.

./ line 298: rst2html: command not found
Collecting docutils
  Using cached docutils-0.14-py2-none-any.whl
Installing collected packages: docutils
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/", line 215, in main
    status =, args)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/", line 342, in run
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/", line 784, in install
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/", line 1064, in move_wheel_files
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/", line 316, in clobber
  File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/", line 83, in ensure_dir
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/docutils'


Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!


Found some interesting things about the 50D worth sharing. This probably also applies to the 5D2.

The ROM dumps saved by ML are 0x800000 in size but it appears that the second half of it is a mirror of the first half. This is something that I noticed in the 5D2 disassembly but the 50D ROM dump I was using was only 0x400000 in size so I never noticed this on the 50D until I helped @Asiertxu with some questions about one of my Digic 4 experiments. Since I don't have a 50D and my wife will kill me if I bid on yet another old camera on ebay I thought I'd run the 50D in QEMU. The 0x400000 ROMs didn't work but the the fresh dumps from the camera worked great:


I made a QEMU beginner's mistake and loaded ML on the sd.img but of course the 50D only has a CF slot so make sure ML is loaded on the cf.img and voilà!

So my question is why work with two 0x800000 sized firmware dumps when this camera is apparently using only one 0x400000 file? More of a theoretical question rather than a practical one because there's no need to fix something that is working.


You can use the small dump if you also declare the right size in model_list.c. The current code was written for the one-size-fits-all ROM backup code that runs at ML startup, so you can use the files auto-saved into ML/LOGS on the card. However, that code is no longer valid for 1300D and D6 models (these have ROMs larger than 16MB), so it's probably a good idea to declare the ROM size, address and possibly RAM size as well, somewhere in platform directory (consts.h? internals.h?)

Unsure where the 8MB dumps come from though, as both the portable ROM dumper and the autobackup code from boot-hack.c save 16MB files, and ROM1 size is declared as 16MB for all DIGIC 4 models (one size fits all, as some models have smaller ROMs).


Quote from: a1ex on April 12, 2018, 06:47:55 PM
Unsure where the 8MB dumps come from...

I might have gotten them from Silk Road before they were shut down  :P

Seriously though, I understand why someone would split a 5D2 or 50D ROM1.BIN in half before disassembling it. After all, those ROMs are 8MB, right? In addition, only the ROM1.BIN was archived because from my understanding on these and several other cameras the ROM0.BIN is just noise. Now we've got Digic 6 cameras, some with a big fat single ROM. Didn't realize any of this until I started playing around with QEMU.

I suppose that declaring the ROM size will also change the firmware signature. Did you see this issue with the firmware signature on the 600D firmware update? QEMU to the rescue!


OK, but since the ROM size is declared as 16MB in QEMU, you'll get a warning at startup and the memory layout won't be identical to the one from a real camera (the ROM copies will no longer be "complete"). In particular, with an 8MB ROM, no matter what you put into it, QEMU won't cover the bootloader!

BTW, g3gg0 has a long answer why the ROM is mirrored like that.

On 5D2, ROM1 size (actually ROM0 in Canon strings; I just used g3gg0's notation) is 8MB (if you look at 16MB ROM, you see two mirrored copies, but if you trim it to 8MB, you get unique data in both halves). However, ROM0 is just 4MB on 5D2.


Something I've been seeing on the Mac when compiling on some of the branches.

From the ~/qemu-eos directory I'm compiling this way:

make -C ../magic-lantern 6D_install_qemu

And this scrolls by:

[ SCRIPTS  ]   install_extra_data
for target in  CONFIG_QEMU_install  CONFIG_MODULES_install; do /Library/Developer/CommandLineTools/usr/bin/make $target; done
/Library/Developer/CommandLineTools/usr/bin/make -C qemu-eos
make: *** qemu-eos: No such file or directory.  Stop.
make[2]: *** [CONFIG_QEMU_compile] Error 2
/Library/Developer/CommandLineTools/usr/bin/make -C ../../modules

It doesn't seem to cause any problems but it does say Error 2 so thought I'd report it.


Appears to be related to this commit.

P.S. emulation ready for DIGIC 7 EOS models (most of the things were very similar to DIGIC 6).

Here's a proof of concept I'm experimenting with, written as a helper script to keep up with all these firmware updates. It finds a few common stubs from an initial test run, and creates an (updated) debugmsg.gdb.

# Finds some common stubs from an initial test run of some EOS firmware in QEMU.
# Usage: ./ 6D

CAM="$1"    # camera model
FW="$2"     # firmware version (optional)
DELAY=5     # time (in seconds) for the test run

[ "$FW" == "" ] && DEBUGMSG_GDB="$CAM/debugmsg.gdb" || DEBUGMSG_GDB="$CAM/$FW/debugmsg.gdb"

if [ $(uname) == "Darwin" ]; then
    if [[ -n $(which ggrep) ]]; then
        echo "Error: you need GNU grep to run this script"
        echo "brew install grep"
        exit 1

    # Mac doesn't like piping the output of to other commands (why?!)
    echo "Test run..." 1>&2
    test_run=$( (sleep $DELAY; echo quit) | \
        ./ $CAM,firmware="$FW;boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log \
        2>&1 )

    test_run=$( echo "$test_run" | ansi2txt )

else # not Mac

    # the above two-step piping would print a warning on Linux
    echo "Test run..." 1>&2
    test_run=$( (sleep $DELAY; echo quit) | \
        ./ $CAM,firmware="$FW;boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log \
        2>&1 | ansi2txt )


# print firmware version to stderr
( (cat uart.log | $GREP -a -m1 "Firmware Version") || (echo "$test_run" | $GREP -o -m1 '[^"]*Firmware Version [0-9][^"]*') ) 1>&2
echo 1>&2

function clear_line {
    echo -en "\r                                                                                \r"

# extract the called function from a line that looks like this:
# call 0x1234(...)
# call 0x1234 DebugMsg(...)
function extract_call {
    head -n1 | $GREP -oP '(?<=call ).*?(?=\()' | cut -d ' ' -f 1

# extract the address of a direct jump
# fixme: does not handle multiple direct jumps
# call 0x1234(...) at [init:4444:8888]
#  -> 0xFF1234     at [init:1234:4448]
function extract_jump {
    head -n2 | tail -n1 | $GREP -oP '(?<=-> ).*?(?= +at )' | cut -d ' ' -f 1

# looks for a function called with one of the arguments
# returns first match
function find_stub_from_strings {
    for str in "$@"; do

        # status indicator
        clear_line 1>&2
        echo -en "\r$str ..." 1>&2

        # any direct jump? use that
        local stub=$(echo "$test_run" \
                        | $GREP -a -m1 -A1 "$str" \
                        | extract_jump)
        if [ "$stub" != "" ]; then
            echo $stub

        # regular call?
        local stub=$(echo "$test_run" \
                        | $GREP -a -m1 "$str" \
                        | extract_call)
        if [ "$stub" != "" ]; then
            echo $stub

# fallback - if a stub cannot be found, get it from the existing GDB script
function get_stub_from_gdb_script {
    cat $DEBUGMSG_GDB | $GREP ${1}_log -B 1 | $GREP -Pom1 "(?<=b \*)0x.*"

# find some common stubs
DebugMsg=$(find_stub_from_strings startupEntry startupEventDispatch DisablePowerSave)
task_create=$(find_stub_from_strings TaskMain "[a-z]Task" systemtask CmdShell EvShel HotPlug PowerMgr PowerMan)
register_interrupt=$(find_stub_from_strings ICAPCHx OC4_14 SIO3_ISR)
CreateStateObject=$(find_stub_from_strings DMState EMState PropState SRMState)
create_semaphore=$(find_stub_from_strings PropSem mallocLock stdioLock dm_lock)
create_msg_queue=$(find_stub_from_strings MainMessQueue QueueForDeviceIn SystemTaskMSGQueue)

register_func=$(echo "$test_run" \
    | $GREP -a flashwrite \
    | $GREP -B1 -m1 NameService \
    | extract_call)

SIO3_ISR=$(echo "$test_run" | $GREP -a -i SIO3_ISR \
    | $GREP -o SIO3_ISR.* \
    | cut -d ',' -f 3 | tr -d ' ')

MREQ_ISR=$(echo "$test_run" | $GREP -a -i MREQ_ISR \
    | $GREP -o MREQ_ISR.* \
    | cut -d ',' -f 3 | tr -d ' ')

# GDB breakpoints must be without the Thumb bit, if any
function clr_thumb {
    if [[ "$param" != 0x* ]]; then
    printf "0x%X\n" $((param & ~1))

# print a GDB logging stub
# fall back by taking it from the existing stub
function print_logging_stub {

    if [ "${!param}" != "" ]; then
        echo "b *$(clr_thumb ${!param})"
        echo "${param}_log"
        local stub=$(get_stub_from_gdb_script $param)
        if [ "$stub" != "" ]; then
            echo "# from $DEBUGMSG_GDB"
            echo "b *$stub"
            echo "${param}_log"
            echo "# not found"
            echo "# b *0x..."
            echo "# ${param}_log"

function print_commented_stub {
    if [ "${!param}" != "" ]; then
        echo "# $(clr_thumb ${!param}) ${param}"

# clear any previous status message
clear_line 1>&2

# output the GDB script
echo                        1>&2
echo "$DEBUGMSG_GDB"        1>&2
echo "====================" 1>&2
echo                        1>&2

echo "# ./ $1 -d debugmsg"
echo "# ./ $1 -d debugmsg -s -S & arm-none-eabi-gdb -x $DEBUGMSG_GDB"
echo "source -v debug-logging.gdb"
cat $DEBUGMSG_GDB | $GREP "To get debugging symbols"
cat $DEBUGMSG_GDB | $GREP "symbol-file"
cat $DEBUGMSG_GDB | $GREP "macro define"

if [ "$DebugMsg" != "" ]; then
    echo "# GDB hook is very slow; -d debugmsg is much faster"
    echo "# ./ will use this address, don't delete it"
    echo "# b *$(clr_thumb $DebugMsg)"
    echo "# DebugMsg_log"

print_logging_stub task_create
print_logging_stub assert
print_logging_stub register_interrupt
print_logging_stub register_func
print_logging_stub mpu_send
print_logging_stub mpu_recv
print_logging_stub create_semaphore
print_logging_stub create_msg_queue
print_logging_stub CreateStateObject

print_commented_stub SIO3_ISR
print_commented_stub MREQ_ISR

# finished
echo cont

These stubs include DebugMsg, task_create, register_func, register_interrupt and CreateStateObject. Run the firmware in GDB using the generated script to get a few hundreds (possibly thousands) named functions.

Refer to Initial firmware analysis for a sample workflow (this script automates step 5).


for us mac fanboys

replace |& with 2>&1 | in the script

go to source forge, download ansi2txt
untar and change the directories in Makefile to /usr/local/....
make && make installl

brew install grep
and replace grep with ggrep in the script

this should get the script running....