Author Topic: How to run Magic Lantern into QEMU?!...  (Read 110157 times)

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #300 on: May 04, 2018, 06:41:27 PM »
hmm ...
Code: [Select]
9:1eb5]
   return 0 to 0x1EB5                                                                               at [init:80000d23:fe
0ce235]
  return 48000e to 0xFE0CE235                                                                                at [init:1e
81:80001735]
  call 0xFE3CDF44(fe0ce600 "TaskMain", 1d, 0, fe0cd4a9)
looks different

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #301 on: May 04, 2018, 06:51:20 PM »
Terminal window too small? Was grep -C5 used?

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #302 on: May 04, 2018, 07:16:38 PM »
Terminal full screen width.
And I've now even used ggrep and declared export TERM=ansi
Code: [Select]
      call 0xFE3CDE84(c0003, 60000053, 1, 5)
         at [SFRead:fe32bd4f:fe32a717]
       -> 0x186F                                                                           at [SFRead:fe3cde84:fe32bd53]

        call 0xFE3CDF94(90007, 0, 73, 0)
   at [TaskMain:fe1c1c7f:fe2eb9fb]
         -> 0x800020B3                                                                         at [TaskMain:fe3cdf94:fe1
c1c83]
         call 0x800056DC(2ee300, 0, 73,  0)
    at [TaskMain:800020bb:fe1c1c83]
          -> 0xFE172B85                                                                        at [TaskMain:800056dc:800

I thought I had left such problems back in 1998 ...  :(

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #303 on: May 04, 2018, 07:42:47 PM »
Tried smaller font? Or writing to a log file and copying the output from a text editor?

The second snippet can't be the first occurrence of TaskMain, btw.

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #304 on: May 04, 2018, 07:54:00 PM »
font :
nope ..
here is the output of the find  stub script
Code: [Select]
b *0xFE172BB2
task_create_log

# from 750D/debugmsg.gdb
b *0xFE52F980
assert_log

# from 750D/debugmsg.gdb
b *0x1774
register_interrupt_log

# from 750D/debugmsg.gdb
b *0xFE445CB8
register_func_log

# from 750D/debugmsg.gdb
b *0x...
mpu_send_log

# from 750D/debugmsg.gdb
b *0x...
mpu_recv_log

b *0xFE3CDFE4
create_semaphore_log

b *0x1C18
create_msg_queue_log

b *0x211A
CreateStateObject_log
for today I'm done ... I'll grab a beer and the camera and enjoy the nice weather and sunset out at the lake ;)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #305 on: May 05, 2018, 09:05:44 AM »
I was re-reading some of the posts and got excited when I realized that most of the lua API tests can run in QEMU as posted on Reply #254. So I applied the patch to api_test.lua and tried out the 1100D.106 lua_fix build.

It got through almost everything on the first try:

Code: [Select]
===============================================================================
ML/SCRIPTS/API_TEST.LUA - 2017-9-30 12:15:00
===============================================================================

Strict mode tests...
Strict mode tests passed.

Generic tests...
arg = table:
  [0] = "API_TEST.LUA"
camera = table:
  shutter = table:
    raw = 104
    apex = 6.
    ms = 16
    value = 0.015625
  aperture = table:
    raw = 75
    apex = 8.375
    value = 18.2
    min = table:
      raw = 45
      apex = 4.625
      value = 4.9
    max = table:
      raw = 88
      apex = 10.
      value = 32
  iso = table:
    raw = 72
    apex = 5.
    value = 100
  ec = table:
    raw = 0
    value = 0
  flash = true
  flash_ec = table:
    raw = 0
    value = 0
  kelvin = 5200
  mode = 3
  metering_mode = 3
  drive_mode = 4
  model = "Canon EOS 1100D"
  model_short = "1100D"
  firmware = "1.0.6"
  temperature = 146
  gui = table:
    menu = false
    play = false
    play_photo = false
    play_movie = false
    qr = false
    idle = true
  shoot = function: p
  bulb = function: p
  reboot = function: p
  wait = function: p
  burst = function: p
event = table:
  pre_shoot = nil
  post_shoot = nil
  shoot_task = nil
  seconds_clock = nil
  keypress = nil
  custom_picture_taking = nil
  intervalometer = nil
  config_save = nil
console = table:
  hide = function: p
  write = function: p
  show = function: p
  clear = function: p
lv = table:
  enabled = false
  paused = false
  running = false
  zoom = 1
  overlays = false
  pause = function: p
  resume = function: p
  start = function: p
  wait = function: p
  info = function: p
  stop = function: p
lens = table:
  name = "EF-S18-55mm f/3.5-5.6 IS"
  focal_length = 0
  focus_distance = 14080
  hyperfocal = 0
  dof_near = 0
  dof_far = 0
  af = false
  af_mode = 3
  focus = function: p
  autofocus = function: p
display = table:
  idle = nil
  height = 480
  width = 720
  off = function: p
  print = function: p
  notify_box = function: p
  pixel = function: p
  screenshot = function: p
  draw = function: p
  load = function: p
  rect = function: p
  circle = function: p
  clear = function: p
  on = function: p
  line = function: p
key = table:
  last = 10
  wait = function: p
  press = function: p
menu = table:
  visible = false
  close = function: p
  block = function: p
  open = function: p
  get = function: p
  new = function: p
  select = function: p
  set = function: p
movie = table:
  recording = false
  start = function: p
  stop = function: p
dryos = table:
  clock = 5
  ms_clock = 5081
  image_prefix = "IMG_"
  dcim_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "B:/DCIM/"
    path = "B:/DCIM/100CANON/"
  config_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "ML/"
    path = "ML/SETTINGS/"
  ml_card = table:
    cluster_size = 16384
    drive_letter = "B"
    file_number = 9321
    folder_number = 100
    free_space = 215520
    type = "SD"
    _card_ptr = userdata
    path = "B:/"
  shooting_card = table:
    cluster_size = 16384
    drive_letter = "B"
    file_number = 9321
    folder_number = 100
    free_space = 215520
    type = "SD"
    _card_ptr = userdata
    path = "B:/"
  date = table:
    hour = 12
    yday = 1
    month = 9
    isdst = false
    sec = 0
    day = 30
    min = 15
    year = 2017
    wday = 2
  rename = function: p
  remove = function: p
  call = function: p
  directory = function: p
interval = table:
  time = 10
  count = 0
  running = false
  stop = function: p
battery = table:
function not available on this camera
stack traceback:
[C]: in ?
[C]: in for iterator 'for iterator'
ML/SCRIPTS/LIB/logger.lua:125: in function 'logger.serialize'
ML/SCRIPTS/API_TEST.LUA:36: in function <ML/SCRIPTS/API_TEST.LUA:35>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:35: in function 'print_table'
ML/SCRIPTS/API_TEST.LUA:81: in function 'generic_tests'
ML/SCRIPTS/API_TEST.LUA:1307: in function 'api_tests'
ML/SCRIPTS/API_TEST.LUA:1328: in main chunktask = table:
  yield = function: p
  create = function: p
property = table:
Generic tests completed.

Module tests...
Testing file I/O...
Copy test: autoexec.bin -> tmp.bin
Copy test OK
Append test: tmp.txt
Append test OK
Rename test: apple.txt -> banana.txt
Rename test OK
Rename test: apple.txt -> ML/banana.txt
Rename test OK
File I/O tests completed.

Testing Canon GUI functions...
Enter MENU mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Exit PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Canon GUI tests completed.

Testing ML menu API...
Menu tests completed.

Testing multitasking...
Only one task allowed to interrupt...
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Multitasking tests completed.

Testing exposure settings...
Camera    : Canon EOS 1100D (1100D) 1.0.6
Lens      : EF-S18-55mm f/3.5-5.6 IS
Shoot mode: 3
Shutter   : Ç60 (raw 104, 0.015625s, 16ms, apex 6.)
Aperture  : Å18 (raw 75, f/18.2, apex 8.375)
Av range  : Å4.9..Å32 (raw 45..88, f/4.9..f/32, apex 4.625..10.)
ISO       : Ä1600 (raw 104, 1600, apex 9.)
EC        : 0.0 (raw 0, 0 EV)
Flash EC  : 0.0 (raw 0, 0 EV)
Setting shutter to random values...


It was pretty cool watching it go through the tests in QEMU but it ended up like this--stuck on the exposure test:



Hum--is that a memory issue? This camera has very little memory.

So I tried running just the exposure test and got a bit further:

Code: [Select]
===============================================================================
ML/SCRIPTS/API_TEST.LUA - 2017-9-30 12:15:00
===============================================================================

Module tests...
Testing exposure settings...
Camera    : Canon EOS 1100D (1100D) 1.0.6
Lens      : EF-S18-55mm f/3.5-5.6 IS
Shoot mode: 3
Shutter   : Ç60 (raw 104, 0.015625s, 16ms, apex 6.)
Aperture  : Å18 (raw 75, f/18.2, apex 8.375)
Av range  : Å4.9..Å32 (raw 45..88, f/4.9..f/32, apex 4.625..10.)
ISO       : Ä100 (raw 72, 100, apex 5.)
EC        : 0.0 (raw 0, 0 EV)
Flash EC  : 0.0 (raw 0, 0 EV)
Setting shutter to random values...
Setting ISO to random values...
Setting aperture to random values...
Please switch to Av mode.

Now the problem is--how do you switch this camera to Av mode? It is done by the mode dial on the camera so ???
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #306 on: May 05, 2018, 10:13:12 AM »
Hey a1ex,
I got it working.
- As written before export TERM=ansi
- font to 11 and fullscreen does only work up to a point and the output will wrap on the middle of screen.
- I had to eliminate the monitor and nodisplay options and run qemu into a file (this way I discovered it would crash, but it was enough to get the task and other calls except mpu).
- put something like :
Code: [Select]
test_run=$(cat test_run.txt) in the script.

The weekend is full now but maybe I can post some progress on the 750D now on sunday night.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #307 on: May 05, 2018, 10:48:20 AM »
Also got it working some minutes ago (ran the script in a Mac VM). Turns out:

- the output is broken if QEMU (run_canon_fw.sh) and ansi2txt are executed both in the same command => bad result
- the output is correctly formatted if the QEMU output is stored in a variable and then passed through ansi2txt => correct result

The same happens with bash 3.2 (that comes with Mac) and 4.4 (brew install bash).

The issue is present if piping to any other command (such as tr).

Now the problem is--how do you switch this camera to Av mode? It is done by the mode dial on the camera so ???

Press F1:

Code: [Select]
[MPU] Available keys:
...
- 0/9          : Mode dial (press only)
- V            : Movie mode (press only)
...

Wait a minute, does the Mac have numeric keys? They seem to work in the VM, but the same is true for buttons in Canon firmware present on other models :D

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #308 on: May 05, 2018, 04:05:01 PM »
Doh! Note to self--don't ask questions after midnight.

Wait a minute, does the Mac have numeric keys?

Yes, and the 0 and 9 keys do indeed go through the mode dial options. V switches to movie mode.

Been trying to follow the find_stubs.sh changes to get it working on Mac but still no luck over here.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #309 on: May 05, 2018, 05:52:02 PM »
Updated the script to address the above issue; this time it should work on Mac.

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #310 on: May 05, 2018, 07:55:25 PM »
Hi a1ex,
it's a bit confusing for newbies like me that you are updating the original posts. I was looking in hg for the script and thought man what did I miss ... but found it on page 12 ;)
The alias for Mac doesn't work ... may I suggest the following
Code: [Select]
...
GREP=grep
if [ $(uname) == "Darwin" ]; then
    if [[ -n $(which ggrep) ]]; then
        export GREP=ggrep
    else
        echo
        echo "Error: you need GNU grep to run this script"
        echo "brew install grep"
        exit 1
    fi
...

and then a s/grep/\$GREP/g

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #311 on: May 05, 2018, 08:34:10 PM »
Hum--The script on Reply #273 is not working out of the box yet.

I was trying to figure out what is up with the alias command on the Mac when t3r4n posted his suggestion. Got past that hump but it is still not working. Thought I'd take a look at what is in the test_run variable like this:

Code: [Select]
    test_run=$( echo "$test_run" | ansi2txt )

    echo $test_run
    exit 1

else # not Mac

This can't be right:

Code: [Select]
./find_stubs.sh 6D
Test run...
CHK version_gen.h ./run_canon_fw.sh 6D,firmware=;boot=0 -d calls,tail -display none -monitor stdio -serial file:uart.log DebugMsg=0x6824 (from GDB script) QEMU 2.5.0 monitor - type 'help' for more information (qemu) Lockdown read 0 Lockdown read 0 Lockdown read 1 Lockdown read 1 Lockdown read 2 Lockdown read 2 Lockdown read 3 Lockdown read 3 Lockdown read 4 Lockdown read 4 00000000 - 00000FFF: eos.tcm_code 40000000 - 40000FFF: eos.tcm_data 00001000 - 1FFFFFFF: eos.ram 40001000 - 5FFFFFFF: eos.ram_uncached F0000000 - F0FFFFFF: eos.rom0 F1000000 - F1FFFFFF: eos.rom0_mirror F2000000 - F2FFFFFF: eos.rom0_mirror F3000000 - F3FFFFFF: eos.rom0_mirror F4000000 - F4FFFFFF: eos.rom0_mirror F5000000 - F5FFFFFF: eos.rom0_mirror F6000000 - F6FFFFFF: eos.rom0_mirror F7000000 - F7FFFFFF: eos.rom0_mirror F8000000 - F8FFFFFF: eos.rom1 F9000000 - F9FFFFFF: eos.rom1_mirror FA000000 - FAFFFFFF: eos.rom1_mirror FB000000 - FBFFFFFF: eos.rom1_mirror FC000000 - FCFFFFFF: eos.rom1_mirror FD000000 - FDFFFFFF: eos.rom1_mirror FE000000 - FEFFFFFF: eos.rom1_mirror FF000000 - FFFFFFFF: eos.rom1_mirror C0000000 - DFFFFFFF: eos.mmio [EOS] enabling code execution logging. [EOS] enabling memory access logging (R). [EOS] enabling singlestep. [EOS] loading './6D/ROM0.BIN' to 0xF0000000-0xF0FFFFFF [EOS] mirrored data; unique 0x800000 bytes repeated 0x2 times [EOS] loading './6D/ROM1.BIN' to 0xF8000000-0xF8FFFFFF [EOS] loading './6D/SFDATA.BIN' as serial flash, size=0x800000 [MPU] warning: non-empty spell #2 (Complete WaitID = 0x80000001 Mode group) has duplicate(s): #6 [MPU] warning: non-empty spell #52 (PROP_VIDEO_MODE) has duplicate(s): #53 [MPU] Available keys: - Arrow keys : Navigation - PgUp, PgDn : Sub dial (rear scrollwheel) - [ and ] : Main dial (top scrollwheel) - SPACE : SET - DELETE : guess (press only) - M : MENU (press only) - P : PLAY (press only) - I : INFO/DISP (press only) - Q : guess (press only) - L : LiveView (press only) - Z : Zoom in - Shift : Half-shutter - 0/9 : Mode dial (press only) - V : Movie mode (press only) - B : Open battery door - C : Open card door - F10 : Power down switch - F1 : show this help Setting BOOTDISK flag to 0 quit [MPU] WARNING: forced shutdown. For clean shutdown, please use 'Machine -> Power Down' (or 'system_powerdown' in QEMU monitor.)

It is all on one line so how can grep with with it? Also note that it didn't get very far before it shutdown.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #312 on: May 05, 2018, 10:17:39 PM »
Heh, I must have installed ggrep to override the default grep, so that's why the script worked for me. Updated again.

That line looks OK, maybe you need to increase the delay. It's not just one line btw - the quotes are important.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #313 on: May 06, 2018, 02:49:57 AM »
Part of the problem I'm having is probably that I'm trying to get the codes for 6D.118 and I might not have the correct CURRENT_TASK and CURRENT_ISR (oh what I'd give for the addresses you're using.) However, on the 1100D.106 it didn't work with the default 5 seconds but check out what happens when I increase the DELAY to 20 seconds:

Code: [Select]
./find_stubs.sh 1100D
Test run...
K288 ICU Firmware Version 1.0.6 ( 3.7.4 )

                                                                               
1100D/debugmsg.gdb
====================

# ./run_canon_fw.sh 1100D -d debugmsg
# ./run_canon_fw.sh 1100D -d debugmsg -s -S & arm-none-eabi-gdb -x 1100D/debugmsg.gdb

source -v debug-logging.gdb

# To get debugging symbols from Magic Lantern, uncomment one of these:
#symbol-file ../magic-lantern/platform/1100D.106/magiclantern
#symbol-file ../magic-lantern/platform/1100D.106/autoexec
#symbol-file ../magic-lantern/platform/1100D.106/stubs.o

macro define CURRENT_TASK 0x1a2c
macro define CURRENT_ISR  (MEM(0x670) ? MEM(0x674) >> 2 : 0)

# GDB hook is very slow; -d debugmsg is much faster
# ./run_canon_fw.sh will use this address, don't delete it
# b *0xFF06C91C
# DebugMsg_log

b *0xFF06FAFC
task_create_log

# not found
# b *0x...
# assert_log

b *0xFF1E8638
register_interrupt_log

b *0xFF06D708
register_func_log

# not found
# b *0x...
# mpu_send_log

# not found
# b *0x...
# mpu_recv_log

b *0xFF06F414
create_semaphore_log

b *0xFF1E8754
create_msg_queue_log

b *0xFF1EE188
CreateStateObject_log

# 0xFF1CAD94 SIO3_ISR
# 0xFF1CAD04 MREQ_ISR

cont

Interesting that increasing DELAY to 60 sec and it doesn't find any of the stubs.

Now don't kill the messenger but ansi2txt isn't available on the Mac or in Homebrew so I had build it from source. Wouldn't "cat" work as well for striping out the ascii control codes?

Code: [Select]
    test_run=$( echo "$test_run" | cat )

@t3r4n - don't know how you're doing it because I'm getting nothing on the 750D.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #314 on: May 06, 2018, 08:25:08 AM »
@dfort:
well lets do a "watch check":
Code: [Select]
$ ansi2txt -v
ansi2txt - version 0.2.2, compiled on May  1 2018 at 13:15:18.

$ bash --version
bash --version
GNU bash, version 4.4.19(1)-release (x86_64-apple-darwin16.7.0)

$ ggrep -V
ggrep -V
ggrep (GNU grep) 3.1
Packaged by Homebrew

I noticed that I already have an alias grep=... in my bash_profile so that might be a reason.
If I run the script I need to have a delay greater 30 seconds to get the stubs.


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #315 on: May 06, 2018, 09:48:23 AM »
Part of the problem I'm having is probably that I'm trying to get the codes for 6D.118 and I might not have the correct CURRENT_TASK and CURRENT_ISR (oh what I'd give for the addresses you're using.)

I didn't try to find them, only noted the old ones won't work. The script does not use them; the logging backend does. I still have the ones for 1.1.6, so the context info (right column) in the test run is not correct, but the script doesn't look at it.

Code: [Select]
./find_stubs.sh 6D 118
Test run...
K302 ICU Firmware Version 1.1.8 ( 5.8.8 )
...
b *0x9798
task_create_log
...

These two were covered in the M2 topic, btw.

Quote
Interesting that increasing DELAY to 60 sec and it doesn't find any of the stubs.

Works here on the Mac VM, just very slow. Does it at least print the firmware version?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 11305
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #316 on: May 06, 2018, 01:18:47 PM »
Started to rework the script in Python.

After a painful struggle with pexpect (TLDR: broken output, 1998 problems), I've managed to get some clean output from QEMU with subprocess.

Does this work on Mac, or it's back to square one?

Code: [Select]
#!/usr/bin/env python2
from __future__ import print_function
import os, sys
import subprocess
import time
import re

string_stubs = {
    "DebugMsg"              : [ "startupEntry", "startupEventDispatch", "DisablePowerSave" ],
    "task_create"           : [ "TaskMain", 'Task"', "systemtask", "CmdShell", "EvShel", "HotPlug", "PowerMgr", "PowerMan" ],
    "register_interrupt"    : [ "ICAPCHx", "OC4_14", "SIO3_ISR" ],
    "CreateStateObject"     : [ "DMState", "EMState", "PropState", "SRMState" ],
    "create_semaphore"      : [ "PropSem", "mallocLock", "stdioLock", "dm_lock" ],
    "create_msg_queue"      : [ "MainMessQueue", "QueueForDeviceIn", "SystemTaskMSGQueue" ],
}

string_stubs_followed_by = {
    "register_func"         : ([ "flashwrite", "gpiowrite" ], ["NameService"])
}

def eprint(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

cam = sys.argv[1]
fw = sys.argv[2] if 2 in sys.argv else ""

eprint("Test run...")

cmd = ('./run_canon_fw.sh %s,firmware="%s;boot=0" -d calls,tail '
            '-display none -monitor stdio -serial file:uart.log' % (cam, fw))
eprint(cmd)
qemu = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# tried pexpect, but ran into lots of problems
# including broken terminal, missing newlines, broken pipe messages
# todo: find a minimal example and file a bug report?
qemu.output = ""

def qexpect(strings, timeout):
    t0 = time.time()
    while time.time() - t0 < timeout:
        output = qemu.stderr.readline()
        qemu.output += output
        if output == '' and qemu.poll() is not None:
            eprint("QEMU not running!")
            break
        if any([str in output for str in strings]):
            eprint(output)
            break
    rc = qemu.poll()
    return any([str in output for str in strings])

if qexpect(string_stubs["task_create"], 60):
    eprint("Task found")
else:
    eprint("Task not found")

# let it run for 5 seconds
qexpect([], 5)

try: print("quit", file=qemu.stdin)
except: pass

q_stdout, q_stderr = qemu.communicate()
qemu.wait()
qemu.output += q_stderr
qemu.lines = qemu.output.split("\n")

with open("find_stubs.log", "w") as log:
    print(qemu.output, file=log)

# extract the called function from a line that looks like this:
# call 0x1234(...)
# call 0x1234 DebugMsg(...)
#   -> 0x5678                  # optional (direct jump)
#    -> 0xFFABCD               # also optional
def extract_call(lines):
    assert len(lines) == 3
    if " -> " in lines[1]:
        jump_line = lines[2] if " -> " in lines[2] else lines[1]
        m = re.search('(?<=-> )(.*?)(?= +at )', jump_line)
        if m:
            return int(m.groups()[0], 16)
    else:
        m = re.search('(?<=call )(.*?)(?=\()', lines[0])
        if m:
            return int(m.groups()[0].split(" ")[0], 16)

# strings       : list of strings to be found
#                 first string has the highest priority
# next_strings  : one of these should be on the next line (optional)
def find_stub_from_strings(strings, next_strings):
    lines = qemu.lines
    for s in strings:
        for i,l in enumerate(lines[:-3]):
            if s in l and "call " in l:
                if next_strings is None or any([ns in lines[i+1] for ns in next_strings]):
                    return extract_call(lines[i:i+3])

eprint("")

stubs_found = {}
for name, strings in string_stubs.iteritems():
    stub = find_stub_from_strings(strings, None)
    if stub:
        stubs_found[name] = stub
        eprint("%8X %s" % (stub, name))
    else:
        eprint("     ???", name)

for name, (strings, next_strings) in string_stubs_followed_by.iteritems():
    stub = find_stub_from_strings(strings, next_strings)
    if stub:
        stubs_found[name] = stub
        eprint("%8X %s" % (stub, name))
    else:
        eprint("     ???", name)

Good luck making this work in Python 3...

t3r4n

  • New to the forum
  • *
  • Posts: 44
Re: How to run Magic Lantern into QEMU?!...
« Reply #317 on: May 06, 2018, 04:39:00 PM »
Quote
Does this work on Mac, or it's back to square one?
Well no error and if this output is expected yes
Code: [Select]
./findstub.py 750D
Test run...
./run_canon_fw.sh 750D,firmware=";boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log
    call 0xFE3CDF44(fe1f845c "PowerMgr", 20, 400, fe1f82e9)                      at [init:fe1f837d:fe506533]

Task found

    211B CreateStateObject
    1775 register_interrupt
80001FC5 create_semaphore
    1E45 task_create
     ??? DebugMsg
     ??? create_msg_queue
FE445CB9 register_func

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #318 on: May 06, 2018, 06:09:38 PM »
Here's what I'm getting:

Code: [Select]
$ python find_stubs.py 1100D
Test run...
./run_canon_fw.sh 1100D,firmware=";boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log
    call 0xFF06FAFC(ff1eaa10 "PowerMgr", 20, 400, ff1ea7dc)                      at [init:ff1ea910:ff072fe0]

Task found

FF1EE188 CreateStateObject
FF1E8638 register_interrupt
FF017630 create_semaphore
FF06FAFC task_create
FF06C91C DebugMsg
     ??? create_msg_queue
FF06D708 register_func

Noticed that create_semaphore is different (FF06F414 on the bash script) but the rest matches the values in Reply #313.

well lets do a "watch check":

Code: [Select]
$ ansi2txt -v
ansi2txt - version 0.2.2, compiled on May  6 2018 at 08:22:16.

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)

$ ggrep -V
ggrep (GNU grep) 3.1
Packaged by Homebrew

The difference might be that you're running bash 4. I had that running for a while but one of the software updates reset it to Apple's default which is version 3. The find_stubs.sh script uses "#!/bin/bash" so it should be using Apple's bash instead of the environmental preference used in the other QEMU scripts -- "#!/usr/bin/env bash"

To test options for ansi2txt I removed it so I had to re-compile it to do our "watch check" -- ansi2txt isn't available for Mac unless you build it from source, right? ansi2txt is used in other ML scripts but not having it installed doesn't seem to be a problem.

Turned out that the reason I wasn't having any success with the 750D was because I don't have a SFDATA.BIN for it. Substituted one from the 700D and got the same results you got. (bash script working too.)

Code: [Select]
$ python find_stubs.py 750D
Test run...
./run_canon_fw.sh 750D,firmware=";boot=0" -d calls,tail -display none -monitor stdio -serial file:uart.log
    call 0xFE3CDF44(fe1f845c "PowerMgr", 20, 400, fe1f82e9)                      at [init:fe1f837d:fe506533]

Task found

    211B CreateStateObject
    1775 register_interrupt
80001FC5 create_semaphore
    1E45 task_create
     ??? DebugMsg
     ??? create_msg_queue
FE445CB9 register_func

RE: DELAY=60

Does it at least print the firmware version?

Just tried it again and yes, it prints the firmware version. It is very slow and you're re-coding this in python but thought I'd re-check it anyway and the 60 sec. delay worked this time. Go figure.

RE: CURRENT_TASK and CURRENT_ISR

These two were covered in the M2 topic, btw.

Yes, I know. I've been going over that part over and over trying to understand it and as far as I can see the values haven't changed but I'm obviously not looking hard enough.

BTW--been going back to that EOSM2 topic many times for reference. Someday maybe I'll be able to get that port working properly.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 2938
Re: How to run Magic Lantern into QEMU?!...
« Reply #319 on: May 07, 2018, 09:30:35 PM »
Running the lua tests in QEMU is helping with the firmware updates I've been doing so I thought I'd try the lua tests on the EOSM2 but before going there I did a run with the EOSM. Besides some of the issues I posted about the lua test on a "real" EOSM, I've been getting this in QEMU:



But I am in M mode! Note that it doesn't matter if I try to switch to M while running the tests or switch to M before running the tests, I get the same message.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 500D.112