How to run Magic Lantern into QEMU?!...

[kennetrunner]

@a1ex   I have qemu 2.9, and tried to install the patch contrib/qemu/qemu-2.5.0.patch   - which fails :-(
Do you have an updated patch file ?

.. ken

[a1ex]

Yes, on the qemu-2.9.0 branch.

However, some parts of the test suite fail in 2.9.0 (but are OK on 2.5.0), and 2.5.0 has a few more features on the EOS side (in particular, the memory tracing and related tools). Syncing them shouldn't be very difficult.

2.9.0 can be interesting for EOS M5, as it might emulate multi-core CPUs a little better (note: the self-test fails for this model) and for porting the code to other QEMU variants (e.g. Panda 2.0 for its binary instrumentation plugins, or Xilinx QEMU^*) if you want to look into UHS), but for the moment I've switched back to 2.5.0 (simply because it works and I'm used to it).

^*) Xilinx QEMU is currently based on 2.6.0, so the patch for it is probably something in-between.

[kennetrunner]

So, after a ton of hurdles I shelved my own attempts at getting qemu 2.9 running...
Instead I downloaded the vbox image from http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989 and compiled qemu 2.5 and it **seemed** to complete successfully...

However, now when I run

Code: [Select]

./run_canon_fw.sh 550D I get

Code: [Select]

qemu-system-arm: unsupported machine type
When I list the supported machine types, the only Canon one to show up is canon-a1100

Seems like I'm missing a step, somewhere ?... any pointers ?

[a1ex]

The vanilla QEMU does not include our modifications.

See the install log from the build server for reference.

[dfort]

I just started into trying to get the qemu branch working on macOS Sierra 10.12.4.

The ML repository needs to be in the home (user) directory or it won't work. The script in magic-lantern/contrib/qemu/install.sh seems to run fine up until this point:

Code: [Select]

Next steps:
===========

1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j

Running ../configure_eos.sh (without the cpuinfo) or "./configure --target-list=arm-softmmu --disable-docs --enable-sdl" results in:

Code: [Select]

ERROR: Cocoa and SDL UIs cannot both be enabled at once
QEMU can also be installed via Homebrew but of course it doesn't include the Magic Lantern modifications.

Maybe it is a matter of figuring out how to configure the "cpuinfo" for the Mac?

Code: [Select]

sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Though it is probably much more than just that. Maybe it is the clang compiler? Any hints?

[a1ex]

Comment out the affected line; plain make is fine.

Or, try something like this:

Code: [Select]

echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Compiling without SDL should be fine (it's used that way on the build server, where it runs without GUI).

[dfort]

Code: [Select]

echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"
grep: /proc/cpuinfo: No such file or directory
   make -j8
Ah ha, so the processor on this machine is:

Code: [Select]

sysctl -n hw.ncpu
8
So I tried:

Code: [Select]

make -j8
Please call configure before running make!
make: *** No rule to make target `trace/generated-events.h', needed by `Makefile'.  Stop.
make: *** Waiting for unfinished jobs....
make: *** [config-host.mak] Error 1

Changing configure_eos.sh so it uses gcc-5 instead of clang gives the warning noted in the script but it still comes up with:

Code: [Select]

C++ compiler c++ does not work with C compiler gcc-5 --std=gnu99
Disabling C++ specific optional code

ERROR: Cocoa and SDL UIs cannot both be enabled at once

It looks like the issue is setting this up so the configure script will select only the SDL UI. Need to dig into it a bit more.

[a1ex]

To specify the C++ compiler, try this:

Code: [Select]

CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

It almost compiles with clang, except for a tiny function which you can safely comment out, and some warnings. Will look into them.

For SDL, try --disable-sdl.

[dfort]

Almost there. Are you seeing these same errors?

Code: [Select]

/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:196:9: error:
      function definition is not allowed here
        {
        ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:200:16: error: use
      of undeclared identifier 'close_idc'; did you mean 'closedir'?
        atexit(close_idc);
               ^~~~~~~~~
               closedir
/usr/include/dirent.h:102:5: note: 'closedir' declared here
int closedir(DIR *) __DARWIN_ALIAS(closedir);
    ^
2 errors generated.
make[1]: *** [hw/arm/../eos/dbi/logging.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:91:20: warning:
      unused function 'set_uninitialized' [-Wunused-function]
static inline void set_uninitialized(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:99:20: warning:
      unused function 'set_freed' [-Wunused-function]
static inline void set_freed(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:107:20: warning:
      unused function 'clr_freed' [-Wunused-function]
static inline void clr_freed(uint32_t addr)
                   ^
3 warnings generated.
1 warning generated.
make: *** [subdir-arm-softmmu] Error 2

Hint on where to find the tiny function that can be commented out?

[a1ex]

Yes, exactly there.

[dfort]

Tried something different. Since cr2hdr doesn't compile on clang we need to install a different compiler on the Mac.

Code: [Select]

CC="gcc-5" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

Then ran "make -j8" like before and it compiled. There were some other errors but it finished compiling.

Got to run off to work now but hope to get back to this soon.

[dfort]

These are the errors I got running the gcc-5 compiler from the Homebrew distribution on the Mac:

Code: [Select]

In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
  CC    crypto/cipher.o
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));

It did finish compiling so did it work? Followed instructions with the rom dumps and mounted the sd card image and this is what happened:

Code: [Select]

./run_canon_fw.sh EOSM
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
CHK version_gen.h
  LINK  qemu-ga
  LINK  ivshmem-server
  CC    block/raw-posix.o
In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
block/raw-posix.c: In function 'hdev_open':
block/raw-posix.c:2129:23: warning: variable 'kernResult' set but not used [-Wunused-but-set-variable]
         kern_return_t kernResult;
                       ^
make: *** [block/raw-posix.o] Error 1
Looks like the same error messages.

Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?

[a1ex]

Searching this error message gives http://stackoverflow.com/questions/41143981/macos-sierra-corefoundation-error-while-compiling-wxwidgets-for-simspark

To me, it looks like qemu didn't finish compiling.

[dfort]

QEMU on Mac -- Yay!

This might be a real hack, here's what I did. First of all the CoreFoundation.framework issues affect the Homebrew gcc-5 compiler so I went back to the macos clang.

~/qemu/configure_eos.sh

Code: [Select]

CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

The compilation errors we had with clang were because of this section of code, so I commented it out.

~/qemu/qemu-2.5.0/hw/eos/dbi/logging.c line#194

Code: [Select]

        /* QEMU is usually closed with CTRL-C, so call this when finished */
//        void close_idc(void)
//        {
//            fprintf(idc, "}\n");
//            fclose(idc);
//        }
//        atexit(close_idc);

Loaded up the EOSM ROM0.BIN, ROM1.BIN and had to create an SFDATA.BIN using the sf_dump module and placed them in ~/qemu/EOSM. Mounted the sd.img and ran:

Code: [Select]

./run_canon_fw.sh EOSM
Now just because it launches doesn't mean that it is working properly but I wanted to share my progress.

[a1ex]

Nice. This is the cocoa interface, right? Do you have menus that allow you to switch between serial console, VGA and so on?

BTW, after this commit it should install cleanly on clang and/or Mac.

[dfort]

Speaking for all of us Mac users -- Thanks!

Here are a few things Mac users who want to get this working should watch out for.

install.sh prints some instructions at the end of the process. On osx it prints out the grep error instead of keeping silent.

Code: [Select]

1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j8
It is caused by this line:

install.sh

Code: [Select]

echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Switching the positions around eliminated the grep error on osx and created a sysctl command not found on Linux so I tried this to suppress the error messages and it worked:

Code: [Select]

echo "   make -j`grep -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"


This instruction doesn't work for osx:

Code: [Select]

3) Mount the included SD (or CF) image (you may use mount.sh)mount.sh calls "kpartx" which isn't available for osx. Mounting sd.img can be done by simply double clicking the sd.img icon but I'm not sure if that worked properly because when running qemu it shows this:

Code: [Select]

SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
SD LOAD OK.
Open file for read : AUTOEXEC.BIN
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
I looked for an alternate for kpartx and maybe hdiutil will work? I haven't figured out if there is some special way it needs to be invoked to mount the img files as loopback devices but either of these seem to work:

Code: [Select]

hdiutil mount sd.img
or
hdiutil attach sd.img

Back to install.sh, at the end it is supposed to list some camera models but on osx it displays this:

Code: [Select]

   Note: Canon GUI emulation (menu navigation, no LiveView) only works on
   usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
That problem is partially because the install script calls tests/run_tests.sh. Here is the output of running that script on osx:
https://pastebin.com/0T5GLRh5

Not sure if any of this would prevent qemu from running properly on osx. I'm just starting on this. Running the minimal autoexc.bin works on every ROM dump I've got but I haven't been able to go much beyond that.

[a1ex]

Code: [Select]

SD: CMD12 in a wrong state
[SDIO] Error

That's fine, I get those too. I'm not sure if this indicates an emulation bug / incomplete model, or it's just how the (simplified) SD driver used in bootloader is supposed to behave (note: CMD12 is STOP_TRANSMISSION).

In the main firmware, you'll get a similar error about CMD1; this one is OK, as it appears to be the way Canon code probes for MMC cards. Regular SD cards are probably not supposed to reply to CMD1 outside the SPI transfer mode (at least that's my understanding), so the SD emulation backend prints some messages. The full conversation can be watched with -d sdcf (or -d sdcf,io for more details) and cross-checked with the SD specification.

If the reader is familiar with SD protocol, I'd welcome any insights (in particular, for the UHS-I initialization sequence).

Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?

To avoid corrupting the data on the SD image, it's best to prevent starting the emulation if the image is mounted by the user (as there will be two processes wanting to write on the same card image, without knowing about each other). This is done on Linux by checking /proc/mounts (losetup is used with "loopback" devices - that is, when mounting an image as a filesystem). Don't know how this works on Mac, and don't know whether my method is portable across other Linux distributions either.


The other issues appear to be (more or less) just annoyances (as they don't print what the user expects to see) and probably easy to fix.

[a1ex]

Some recent additions (some of them were covered in other threads):

* Cleaned up the self-test log to include installation instructions (the self-test begins with reinstalling QEMU from scratch, so you can also use it to check the expected output of the commands etc).

* Call stack reconstruction (inspired from Panda's callstack_instr, but heavily customized for EOS firmware). This makes the call stack available to other analysis tools (e.g. when memcheck wants to print an error). Enable with -d callstack . Works on DIGIC 4 to 6 and Eeko; almost there on DIGIC 2 and 3; for best results, .current_task_addr should also be defined (because each task has its own stack).

* Call/return trace. Every function call and return (that could be identified automatically) is logged on the console, including arguments and return values (enable with -d calls ). Works best on DIGIC 4 to 6 and Eeko, almost there on DIGIC 2 and 3. Interrupts are also handled (though it still has some trouble with corner cases). Example for Eeko.

Tip: you can load these indented logs in your text editor, configure it for Python source code, and you've got collapse buttons

* Task switches. Make sure you have .current_task_addr defined in model_list.c and enable with -d tasks . Works on DIGIC 2 to 7 and Eeko. Example for EOS M2.

* Memory blocks copied from ROM to RAM can be identified automatically (-d romcpy). They are listed on the self-test log, and they should be a big time-saver when setting up the disassembler (usually you'll need to know what are the blocks worth disassembling, besides the ROM). Works on DIGIC 2 to 7. Example for EOS M5 (can be cross-checked with values found manually):

Code: [Select]

[ROMCPY] 0xE001AF2C -> 0xDF020000 size 0x3C0      at 0xE0005AA8
[ROMCPY] 0xE001B2E4 -> 0x4000     size 0xF1C      at 0xE000492C
[ROMCPY] 0xE115CF88 -> 0x8000     size 0x6054C    at 0xE002003C
[ROMCPY] 0xE11BD4D4 -> 0x1900000  size 0x1444     at 0xE0020060
[ROMCPY] 0xE11BE918 -> 0xDFFC4900 size 0x152A0    at 0xE0020084

* Symbol names from .elf files. QEMU has this feature built in, but had to customize it a bit (for example, it didn't recognize our stubs as function calls, because our build system doesn't mark them as such). Usage (requires bash):

Code: [Select]

. ./export_ml_syms.sh 550D.109
./run_canon_fw.sh 550D.109,firmware="boot=1" ...
...
Task switch to init:ff010470                                                     at [init:ff0164c4:ff076f18]
call 0x8B010 my_init_task(0, 8b010, 19980218, 19980218)                          at [init:ff0771dc:0]
 call 0xFF018D1C init_task(0, 8b010, 19980218, 19980218)                         at [init:8b014:ff0771e0] (my_init_task)
  call 0xFF0108FC(0, 8b010, 19980218, 19980218)                                  at [init:ff018d20:8b018]

In other words, if you want the log to use the function names you have annotated in the disassembly, just add them to stubs.S and compile ML; the above script will load them. If you don't want to run autoexec.bin, you can probably just load stubs.o (which is an elf file), or you can probably create a stubs file just for analysis (one that will not be included in ML, because it will usually have a lot more names than you actually want to call).

There's a lot of room for improvement on this one (for example, right now function names are only listed in the call/return trace). Adding this info in more places is generally an easy coding task - find what parts of the log you would like to have function names rather than just raw addresses, and change the output strings accordingly). Automating symbol name export from your favorite disassembler should also be easy (and in many cases, the code is available online from other open source tools; just needs to be integrated here).

* Memcheck warnings closer to valgrind's (e.g. show where the affected memory block was allocated or freed).

* Basic ABI checking (e.g. R4-R11 and SP should be restored after function call). This is done while creating the call stack (-d callstack).

* The auto-generated IDC file should be much more accurate (because we finally have a sane call/return trace).

A walkthrough for some of these features can be found in the EOS M2 topic, where, with dfort's help, I'd like to show in detail how to port Magic Lantern on a new camera from scratch.

Tip: to find all these analysis options, run QEMU with -d help. The ones interesting for us are:

Code: [Select]

int        show interrupts/exceptions in short format
exec       show trace before each executed TB (lots of logs)
nochain    do not chain compiled TBs so that "exec" and "cpu" show complete traces
io         EOS: log low-level I/O activity
mpu        EOS: log low-level MPU activity
sflash     EOS: log low-level serial flash activity
sdcf       EOS: log low-level SD/CF activity
uart       EOS: log low-level UART activity
ram        EOS: log all RAM reads and writes
rom        EOS: log all ROM reads and writes
ramr       EOS: log all RAM reads
romr       EOS: log all ROM reads
ramw       EOS: log all RAM writes
romw       EOS: log all ROM writes
ram_dbg    EOS: self-test for the RAM logging routines
callstack  EOS: reconstruct call stack (implies nochain,singlestep)
calls      EOS: log function calls (implies callstack,ramr,nochain,singlestep)
idc        EOS: export called functions to IDA (implies callstack,nochain,singlestep)
tasks      EOS: log task switches (implies callstack,nochain,singlestep)
romcpy     EOS: find memory blocks copied from ROM to RAM
memchk     EOS: check memory usage (malloc/free, uninitialized values)
v         
verbose    EOS: very detailed debug messages

Happy reversing!

[dfort]

Just sharing some of my recent QEMU adventures on Mac -- and Cygwin.

It seems to be working fine on Mac but there are several issues with run_tests.sh. It doesn't really affect the compiling but it does show a message that makes it look like something went terribly wrong. I haven't worked it all out but so far most of the issues have to so with missing programs on the Mac.

The bash version that comes with the Mac is 3.2.57 but the command "declare -A" requires bash 4. Installing bash via Homebrew gives you version 4.4.12 but the script uses "#!/bin/bash" which points to the old version. Changing the script's shebang fixes that and this should be portable:

Code: [Select]

#!/usr/bin/env bash
Next problem is that the Mac doesn't have "ansi2txt" and there doesn't seem to be one on Homebrew either. I installed it from the source. Because the Mac keeps you from installing programs in /bin I changed the Makefile like this:

Code: [Select]

BINPATH = /usr/local/bin
MANPATH = /usr/local/share/man/man1

Yet another missing program is "timeout" and this can be installed via "brew install coreutils" but then you get gtimeout so to get the default names I had to (just run these commands to update the bash preference, no need to put it in the Makefile):

Code: [Select]

PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Almost there, there is a grep error that I mentioned before:

Code: [Select]

   usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
 [-e pattern] [-f file] [--binary-files=value] [--color=when]
 [--context[=num]] [--directories=action] [--label] [--line-buffered]
 [--null] [pattern] [file ...]
At this point I ran out of time but will revisit this later.

I also tried running QEMU on Cygwin. I didn't compile though "install.sh" did finish successfully. If you set up a Cygwin environment using my tutorial you'll need to install a few extra packages to get through the actual compiling.

Code: [Select]

git
patch
gcc-g++

git needs to be set up:

Code: [Select]

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

The qemu-2.5.0 configure script supports Cygwin but it has a depreciated option:

Code: [Select]

CYGWIN*)
  mingw32="yes"
  QEMU_CFLAGS="-mno-cygwin $QEMU_CFLAGS"
  audio_possible_drivers="sdl"
  audio_drv_list="sdl"
;;

Simply delete this line:

Code: [Select]

  QEMU_CFLAGS="-mno-cygwin $QEMU_CFLAGS"

I still didn't get it to compile and am stuck with this error:

Code: [Select]

$ ../configure_eos.sh
Setting up QEMU on CYGWIN_NT-6.0...
Using gcc --std=gnu99 / g++ with -Wno-error=deprecated-declarations
Options:

ERROR: glib-2.22 gthread-2.0 is required to compile QEMU
Not sure which Cygwin packages are needed to satisfy those dependencies.

I opened a pull request that incorporates a few of these suggestions along with preliminary support for the EOSM2 in case any of this is useful.

[EDIT] That pull request had build system changes mixed with EOSM2 support. Best split them in separate pull requests. In the meantime I discovered that, "Builds with the normal Cygwin compiler are not supported." Even following those instructions I still haven't been able to build QEMU in Cygwin. In fact I've gotten just as far using the native (host) compiler. In both cases it ended like this:

Code: [Select]

/usr/include/w32api/winsock2.h:995:34: error: conflicting types for ‘select’
   WINSOCK_API_LINKAGE int WSAAPI select(int nfds,fd_set *readfds,fd_set *writefds,fd_set *exceptfds,const PTIMEVAL timeout);
                                  ^
In file included from /usr/include/sys/types.h:68:0,
                 from /usr/include/time.h:28,
                 from /usr/include/glib-2.0/glib/gtypes.h:35,
                 from /usr/include/glib-2.0/glib/galloca.h:32,
                 from /usr/include/glib-2.0/glib.h:30,
                 from qga/commands-win32.c:14:
/usr/include/sys/select.h:73:5: note: previous declaration of ‘select’ was here
 int select __P ((int __n, fd_set *__readfds, fd_set *__writefds,
     ^
qga/commands-win32.c: In function ‘qmp_guest_file_open’:

[EDIT] Put up pull requests for build system tweaks and EOSM2.

[dfort]

Found out why the Mac is having problems with "install.sh" -- it is a well known issue with BSD grep that is installed on OSX not being able to use the "-P" option. Replacing it with GNU grep "fixes" the issue.

Another solution would be to rewrite the script so it doesn't use the "grep -P" option which might not be a bad idea because that is a "highly experimental" option:

Code: [Select]

       -P, --perl-regexp
              Interpret  the  pattern  as  a  Perl-compatible  regular  expression  (PCRE).  This is highly
              experimental and grep -P may warn of unimplemented features.

To replace grep on the Mac with Homebrew:

Code: [Select]

brew tap homebrew/core; brew install grep --with-default-names

[dmilligan]

IIRC, if you can't use -P, you loose the vast majority of powerful parts of regex syntax (depending on what you're trying to do, it might not be possible to use grep at all, and so you would need to take on an extra dependency anyway). And I bet that statement about "highly experimental" was put in decades ago and never taken out. These more advanced parts of regular expression syntax are now quite standard.

There's also this: http://jlebar.com/2012/11/28/GNU_grep_is_10x_faster_than_Mac_grep.html

[a1ex]

Some more updates:

- 5D3 menu navigation works^*) (1.1.3 only)
- call/return trace: function arguments that look like strings are now identified
- experiment: include more debug info in ML elf files ⁺⁾
- a few other minor tricks

^*) To make this work, I had to port the GDB patch for date/time (easy to find in */debugmsg.gdb). On most (all?) other models, this patch is not really needed, because the date/time dialog is easy to bypass (just click OK or Cancel). Not so on 5D3; figure out why.

The 70D also seems to react to this trick, although menu navigation locks up very quickly on this model...



⁺⁾ This enables translation from binary address to source code line, for example:

Code: [Select]

# ML must be compiled from the qemu branch (for now)
hg up qemu -C
cd magic-lantern/platform/500D.111
make clean; make

# pick some address above RESTARTSTART
eu-addr2line -s -S -e magiclantern 0x4F000
meminfo_display+0x2bc
mem.c:1149

More to come.

[a1ex]

Some more:

- parameterized MPU spells ^*)
- description of known MPU messages
- log timestamps are now used to help deciding the relationships between MPU spells ⁺⁾
- build fixes for Mac (dfort)

^*) Many MPU messages represent properties (for example: PROP_ICU_UILOCK is 06 05 04 01 nn 00, where "nn" is the value of the property). So, rather than including a "spell" for every possible value, it makes sense to define a template: in this case, whenever our emulated MPU receives 06 05 04 01 something 00, it should reply the same message back. To define such behavior, you may use ARG0 to ARG3 to define parameters; for example:

Code: [Select]

    { { 0x06, 0x05, 0x04, 0x01, ARG0, 0x00 }, {
        { 0x06, 0x05, 0x04, 0x01, ARG0, 0x00 },
        { 0 } } },

That's a trivial example, where the reply simply echoes the same message. Not all messages are the same; for example, for changing the GUI state:

Code: [Select]

    { { 0x06, 0x05, 0x04, 0x00, ARG0, 0x00 }, {                 /* NotifyGUIEvent(ARG0) */
        { 0x06, 0x05, 0x04, 0x00, ARG0, 0x01 },                 /* PROP_GUI_STATE(ARG0) */
        { 0 } } },

In other words, whenever it receives 06 05 04 00 something 00 (which is a request for changing GUI state), it replies back with 06 05 04 00 that_something 01 (which probably means there's green light for changing GUI state, or something like that).

⁺⁾ Recent DebugMsg logs include a timestamp; this is now used to decide the relationship between MPU spells (previously, this was decided solely based on the sequence of the mpu_send/recv calls, with some exceptions for e.g. button messages). Only messages received shortly after a mpu_send call are now considered replies; messages that arrive later are assumed to be external inputs and commented out. Experimental; only 5D3, 5D2 and 50D have this enabled currently.

[dfort]

- build fixes for Mac (dfort)

Thanks! I'm still working on getting the EOSM2 setup by following your QEMU tutorial on that port -- you aren't going to let me get off easy are you? 

QEMU on Mac seems to be working. The most success I've been having is with the 550D (I don't have 500D ROMs):



I looked into mount.sh and got it working on OS X (kpartx isn't available on Mac):

Code: [Select]

#!/usr/bin/env bash

echo "This will mount sd.img and cf.img as a loopback device."
echo "Please enter your password (of course, after reviewing what this script does)."

if [ $(uname -s) == "Darwin" ]; then
    sudo hdiutil attach sd.img
    sudo hdiutil attach cf.img
    echo "Done."
    echo "To remove the device mappings, run:"
    echo '   sudo hdiutil detach "/Volumes/EOS_DIGITAL"'
    echo '   sudo hdiutil detach "/Volumes/EOS_DIGITAL 1"'
else
    sudo kpartx -av sd.img
    sudo kpartx -av cf.img
    echo "Done."
    echo "To remove the device mappings, run:"
    echo "   sudo kpartx -dv sd.img"
    echo "   sudo kpartx -dv cf.img"
fi

I would submit a pull request for this but it seems that on the Mac it isn't necessary to mount as root. I also realized that when running QEMU via ./run_canon_fw.sh the first thing that script does is to make sure sd.img and cd.img aren't mounted. That script uses losetup which is yet another one that isn't available on the Mac. It displays an error but that doesn't seem to cause any problems. I've run QEMU with and without sd.img mounted and it doesn't seem to make any difference so maybe we just leave this alone for now?

[a1ex]

I've run QEMU with and without sd.img mounted and it doesn't seem to make any difference

It will make a difference as soon as both you and the camera code will write something to the SD image during an emulation session. I did that by mistake a while ago and ended up with an unusable image (had to re-create it, copy ML again etc).

Additionally, at least under Linux, the data you write to some mounted directory is not flushed right away to disk. So, if you copy autoexec.bin or other files on the partition, you may find out they are not fully visible to the guest OS.