Author Topic: How to run Magic Lantern into QEMU?!...  (Read 72618 times)

dfort

  • Hero Member
  • *****
  • Posts: 1717
Re: How to run Magic Lantern into QEMU?!...
« Reply #175 on: June 08, 2017, 01:07:38 AM »
I see, so we should check if /Volumes/EOS_DIGITAL* exists on OS X.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #176 on: June 11, 2017, 12:34:21 AM »
More updates:

* Monitor console available by default as a UNIX socket; that means, during emulation you can interact with it with netcat (for quick commands or from a script), or with socat (for interactive console):
Code: [Select]
echo "log io" | nc -U qemu.monitor
socat - UNIX-CONNECT:qemu.monitor

* Log DebugMsg calls without GDB (very fast; credits go to @nkls - I've used a modified version of his initial DebugMsg hook).
Code: [Select]
./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg

To use it on plain Canon firmware (any shell):
env QEMU_EOS_DEBUGMSG=0x5b90 ./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg
or, with ML loaded (requires bash):
. ./export_ml_syms.sh 5D3.113
./run_canon_fw.sh 5D3,firmware="boot=1" -d debugmsg


* Verbose stack trace (to see where each message is coming from), for both -d debugmsg and GDB scripts (DIGIC 4-6). Example for the former:
Code: [Select]
./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg,callstack,v

Current stack: [14ff80-14ef80] sp=14fed8                                         at [FileMgr:5b90:ff0f9684]
0x17B60(51ec48 &"TaskClass", 17b60, 19980218, 19980218)                          at [FileMgr:de48:14ff78] (pc:sp)
 0xFF11B818(51ea28 &"FileMgr", 6, 0, 2)                                          at [FileMgr:17bbc:14ff50] (pc:sp)
  0x178B4(51ec1c &"StateObject", 51ea28 &"FileMgr", 6, 0)                        at [FileMgr:ff11b844:14ff38] (pc:sp)
   0x178EC(51ec1c &"StateObject", 51ea28 &"FileMgr", 6, 0)                       at [FileMgr:178e4:14ff28] (pc:sp)
    0xFF2C8F5C(51ea28 &"FileMgr", 0, 2, ff2c8f5c)                                at [FileMgr:1796c:14ff08] (pc:sp)
     0xFF0C5194(10, 0, 24, ff0c5194)                                             at [FileMgr:ff2c9050:14fef0] (pc:sp)
      0x5B90(0, 3, ff0f9784 "[SEQ] NotifyComplete (Cur = %d, %#x, Flag = %#x)", 4)
                                                                                 at [FileMgr:ff0f9680:14fed8] (pc:sp)
[     FileMgr:ff0f9680 ] (00:03) [SEQ] NotifyComplete (Cur = 4, 0x10, Flag = 0x10)

This tool is very powerful - rather than hunting for several minutes/hours to see where some error message might be coming from, you now get the answer in seconds (example).

* Thorough consistency check to make sure the stack trace gives the same information as if you would follow the call/return trace manually.

More to come (regarding 1300D, digic 6, memory checking, automatic testing of ML builds). Most of these were written some time ago, but it takes a while to integrate everything and make sure they pass the test suite. Though slow, this approach does catch a lot of bugs very early, and I hope to have soon the tools to use a similar development approach for the main ML codebase.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #177 on: July 14, 2017, 01:14:29 PM »
Some more:

- 650D GUI (come on guys, it was as simple as copying some 700D definitions and trying it...)
- EOSM2 GUI (see this walkthrough, thanks dfort for following it along)
- 1300D: minor updates
- 70D menu tests (they don't work all the time, figure out why)
- 5D4 emulation updated for latest firmware

- splitgdb.sh updated to use cgdb (installing it from source is recommended)

- run_ml_all_cams.sh can now be used to compile and run ML in QEMU with various options, very customizable

To get started: this compiles ML for 500D and 60D, copies each build to the virtual SD image, runs it for 10 seconds and takes a screenshot.
Code: [Select]
env ML_PLATFORMS="500D.111/ 60D.111/" TIMEOUT=10 SCREENSHOT=1 ./run_ml_all_cams.sh

Internally, this is how the emulator is invoked:
Code: [Select]
(
  sleep 10
  echo screendump 60D.111.ppm
  echo quit
) | (
  arm-none-eabi-gdb -x 60D/patches.gdb &
  ./run_canon_fw.sh 60D,firmware='boot=1' \
      -display none -monitor stdio  -s -S
) &> 60D.111.log

More examples: EOSM2 hello world, or running ML from the dm-spy-experiments branch in the emulator (QEMU-dm-spy).



- track direct jumps in call stacks (widely used on DIGIC 6)

Example for 80D:
Code: [Select]
# in 80D/debugmsg.gdb
macro define PRINT_CALLSTACK 1

./run_canon_fw.sh 80D,firmware="boot=0" -s -S -d callstack & arm-none-eabi-gdb -x 80D/debugmsg.gdb
...
0xFE0D3385(0, fe0d3385, 19980218, 19980218)                                      at [init:8000173d:2e9108] (pc:sp)
 0xFE533439(fe0d76f9, 0, fe0d76e5, fe0d76e7)                                     at [init:fe0d407d:2e90f0] (pc:sp)
  0xFE209159(fe0d76f9, 0, fe0d76e5, fe0d76e7)                                    at [init:fe53344f:2e90d8] (pc:sp)
   0xFE3FC0DC -> 0xFF1(fe209254 "PowerMgr", 20, 400, fe2090e1)                   at [init:fe209175:2e90c8] (pc:sp)
[      init:fe209175 ] task_create(PowerMgr, prio=20, stack=400, entry=fe2090e1, arg=0)

On the last line, 0xFE3FC0DC is an ARM wrapper (veneer) for 0xFF1 task_create. Wrappers like this are used in D6 code all over the place. Sometimes there are many different wrappers for the same functions, or even wrappers to wrappers to wrappers - these make the function calls very hard to track down.

Here's an example:
Code: [Select]
b *0xFE483D42
commands
  silent
  print_current_location_with_callstack
  printf "sei\n"
  c
end

Code: [Select]
Current stack: [2e9118-2e8118] sp=2e9000                                         at [init:fe483d42:fe483f2d]
0xFE0D3385(0, fe0d3385, 19980218, 19980218)                                      at [init:8000173d:2e9108] (pc:sp)
 0xFE3FC204 -> 0x800035E1(8b, 16, fe0d4790 "\n%s ICU Firmware Version %s ( %s )", fe0d39a8)
                                                                                 at [init:fe0d43e9:2e90f0] (pc:sp)
  0x8000579B(3cc000, 0, 2e9044, 2e9048 "[STARTUP] \nK350 ICU Firmware Version 1.0.1 ( 6.2.2 )")
                                                                                 at [init:8000367f:2e9040] (pc:sp)
   0x800061F5(3cc084, 0, ffffffff, 3cc0d4)                                       at [init:8000581b:2e9020] (pc:sp)
    0x800051DC -> 0xFE483F0D(3cc084, 0, ffffffff, 3cc0d4)                        at [init:80006201:2e9010] (pc:sp)
     0xFE3FC00C -> 0xCA9 -> 0x168B -> 0xFE483D43(0, 96, ffffffff, 3cc0d4)        at [init:fe483f29:2e9000] (pc:sp)
[      init:fe483f29 ] sei

Look at those functions from the last line - they do nothing but jumping to the next one:
Code: [Select]
FE3FC00C 04 F0 1F E5                 LDR             PC, =(sub_CA8+1)

00000CA8 00 F0 EF BC                 B.W             sub_168A

0000168A 43 F6 43 5C+                MOV             R12, #0xFE483D43
00001692 60 47                       BX              R12 ; sub_FE483D42

FE483D42 EF F3 00 81                 MRS.W           R1, CPSR
FE483D46 21 F0 80 01                 BIC.W           R1, R1, #0x80
...


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #178 on: July 15, 2017, 05:31:51 PM »
More:

- "-d debugmsg" now just works; you no longer need to setup additional stuff, just run something like:

Code: [Select]
./run_canon_fw.sh 60D,firmware="boot=0" -d debugmsg

- stack trace also available (in a more limited format) without instrumentation; the full stack trace ("-d callstack") requires instrumentation to detect function calls, so it's slower, but also captures function arguments and handles difficult cases better.

To show the stack trace (with any of the two methods):
* it's always enabled for assert in GDB scripts
* "macro define PRINT_CALLSTACK 1" in GDB scripts enables it for all logged functions
* "print_current_location_with_callstack" at GDB prompt
* "-d debugmsg,v" or "-d debugmsg,callstack,v" on the command line (very verbose)
* from custom code, just call eos_callstack_print_verbose

Internally, the two methods were cross-checked to make sure they give the same results or fail gracefully, on every single function call until GUI boot.

The non-instrumented stack trace is also usable in ML, on the camera (e.g. in crash logs).

dfort

  • Hero Member
  • *****
  • Posts: 1717
Re: How to run Magic Lantern into QEMU?!...
« Reply #179 on: July 16, 2017, 06:41:51 AM »
Got a really noob QEMU question.

Is it possible to bring up the ML menus using the Trash button? If so, where is the Trash button?

Maybe the Trash button is currently an unknown MPU spell? (If that's the right term.)

I've got the 550D and 1100D working with all sorts of builds including nightlies from the jenkins server but I haven't figured out how to bring up the ML menus.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3192
  • 60Da / 1100D / EOSM
Re: How to run Magic Lantern into QEMU?!...
« Reply #180 on: July 16, 2017, 07:39:34 PM »
You probably had trouble finding it b/c you are on a Mac right? Most Mac keyboards lack a dedicated forward delete key. Try: Fn + Delete

dfort

  • Hero Member
  • *****
  • Posts: 1717
Re: How to run Magic Lantern into QEMU?!...
« Reply #181 on: July 16, 2017, 08:51:54 PM »
Hi @dmilligan -- Yep, I'm on a Mac. No combination of delete with a modifier key is showing a key event in QEMU. Though hitting random keys brought up this interesting screen:



That's trying to run a full ML on the 1100D and bring up the ML menus. I've been successful running a minimal "Hello World" build but a full ML build doesn't get all the way to the Canon menu. Oh well, Have fun!
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #182 on: July 16, 2017, 09:44:19 PM »
1100D doesn't have a dedicated "delete" button - it uses Av, which is not emulated in QEMU, and it's not even a regular button - the event it sends is actually a refresh of the info display (Canon calls this OLC). So, we use a very weird heuristic to detect short presses (and I'm not even sure how reliable it works).

PoC: https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-ML-menu/

(still has a few bugs in it)

dfort

  • Hero Member
  • *****
  • Posts: 1717
Re: How to run Magic Lantern into QEMU?!...
« Reply #183 on: July 17, 2017, 06:02:06 PM »
That is so cool:

700D in QEMU


So is the next step adapting Canon EF lenses to the computer?
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #184 on: July 18, 2017, 07:36:59 PM »
So is the next step adapting Canon EF lenses to the computer?

Joking aside, you've probably noticed the Debug -> Lens Info menu is actually usable on some models. This info comes from MPU messages.

That means, you can use the startup-log builds while starting the camera with various lenses, and find out which parts of the log are lens-specific.

If we want to log things such as changing focus distance or focal length in LiveView (where these are reported back to the ARM CPU), we have a small issue: the log buffer fills up really quickly in LiveView. It may make sense to prepare a different set of builds, with reduced verbosity (or maybe skip logging unimportant messages once e.g. half of the buffer is full). Will look into that.

I should also make the mpu_send/recv stubs mandatory, as they are now known to work identically in all models, from digic 2 to digic 6 (not sure about 7 yet). And I'll need such logs from all cameras anyway.

BBA

  • Member
  • ***
  • Posts: 124
Re: How to run Magic Lantern into QEMU?!...
« Reply #185 on: July 18, 2017, 08:21:49 PM »
Sigma firmware updates can be downloaded for their lenses on Canon EF mount.... Don't know which processor they use...What do you think ?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10018
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #186 on: July 18, 2017, 08:29:40 PM »
Sorry, no experience with Sigma lenses.