How to run Magic Lantern into QEMU?!...

[a1ex]

for us mac fanboys
[...]
this should get the script running....

Thanks; edited the script to add these changes.

[dfort]

Appears to be related to this commit.

Yup, that's it.

There seems to be more issues for Mac fanboys. The script uses ansi2txt which isn't available on the Mac. Ran into this problem before and it is easy enough to install from source. That's a rather long post, here's how to do it:

Mac doesn't have "ansi2txt" and there doesn't seem to be one on Homebrew either. I installed it from the source. Because the Mac keeps you from installing programs in /bin I changed the Makefile like this:

Code: [Select]

BINPATH = /usr/local/bin
MANPATH = /usr/local/share/man/man1


I can see what the script is supposed to do and wow, maybe something like this could be used to automate firmware updates and possibly even speed up new ports. I tried running the script on the 6D.118 firmware because that's the one that is really challenging me but it isn't working on the Mac.

Running just the QEMU command:

Code: [Select]

./run_canon_fw.sh 6D,firmware="boot=0" -d calls,tail -display none -monitor stdio 2>&1 | ansi2txt
will hang the terminal. Simplifying it to this:

Code: [Select]

./run_canon_fw.sh 6D,firmware="boot=0" -d calls,tail -display none -monitor stdio
shows the output on the terminal but it doesn't end well.

Code: [Select]

return from interrupt to fd60 (old=907c)                                                                                              at [:630:387f0]
                                -> 0xFD60                                                  at [:907c:387f0]
R0 changed
Assertion failed: (0), function eos_callstack_log_exec, file /Users/rosiefort/qemu-eos/qemu-2.5.0/hw/arm/../eos/dbi/logging.c, line 1469.
./run_canon_fw.sh: line 153: 20953 Abort trap: 6           env QEMU_EOS_DEBUGMSG="$QEMU_EOS_DEBUGMSG" $QEMU_PATH/arm-softmmu/qemu-system-arm -drive if=sd,format=raw,file=sd.img -drive if=ide,format=raw,file=cf.img -chardev socket,server,nowait,path=qemu.monitor$QEMU_JOB_ID,id=monsock -mon chardev=monsock,mode=readline -name $CAM -M $*


[t3r4n]

Hey dfort,
hmm strange I see your error when I run without the

Code: [Select]

2>&1 | ansi2txt portion,
but it seems to work ok when I add it.
I had to increase the time in the script to 60s to get most of the breakpoints right (still missing mpg_rec_log and mph_send_log) but otherwise it is working out.
Question:
When I now start qemu with -s -S and attach the debugger with -x debugmsg.gdb, the gdb will create a temporary breakpoint for each semaphore, msg_queue and so on. These breakpoints can not be stepped by with c only by disabling them? This is not happening while it is running over e.g. the function_register_log, how can I prevent this other than removing it from the .gdb file?

[dfort]

Question:
When I now start qemu with -s -S and attach the debugger with -x debugmsg.gdb, the gdb will create a temporary breakpoint for each semaphore, msg_queue and so on. These breakpoints can not be stepped by with c only by disabling them? This is not happening while it is running over e.g. the function_register_log, how can I prevent this other than removing it from the .gdb file?

What we have found is that on the Mac with the terminal window default size we get this when running the debugger:



Here's the explanation that a1ex gave on why this is happening:

https://www.magiclantern.fm/forum/index.php?topic=15895.msg196075#msg196075

I found that resizing the terminal window to something about twice as large it will skip that message and maybe it will resolve the issue you're having with the breakpoints.

[a1ex]

Question:
When I now start qemu with -s -S and attach the debugger with -x debugmsg.gdb, the gdb will create a temporary breakpoint for each semaphore, msg_queue and so on. These breakpoints can not be stepped by with c only by disabling them? This is not happening while it is running over e.g. the function_register_log, how can I prevent this other than removing it from the .gdb file?

These temporary breakpoints are to log return values. They work here out of the box (with 32-bit arm-none-eabi-gdb from gcc-arm-none-eabi-5_4-2016q3), but could reproduce your issue with a 64-bit gdb (tried a couple of versions, latest being 8.0.50.20171008-git).

Try compiling a 32-bit GDB for Mac, then let's include that in the installation script.

Assertion failed: (0), function eos_callstack_log_exec, file /Users/rosiefort/qemu-eos/qemu-2.5.0/hw/arm/../eos/dbi/logging.c, line 1469.

Repeatable? Can you find a value for -icount that would reproduce the issue? The -icount option ensures repeatable emulation (for example, timers and other interrupts will always fire at the same time, so you'll get the same execution trace every time). By default, emulation follows the real-time clock to some extent, but is not deterministic.

Here it boots just fine to Canon menu, with both 1.1.6 and 1.1.8, with or without -icount 5, but emulation is very slow.

[t3r4n]

oh you mean the warning with 64 bits and issues and stuff  never heard from that 
I'll look how to get a 32 bit gdb to work. From what I've read in the past it might be time consuming.

[dfort]

Repeatable?

Yes, as long as "-d calls" is included.

Can you find a value for -icount that would reproduce the issue?

Interesting--this works:

Code: [Select]

./run_canon_fw.sh 6D,firmware="boot=0" -icount 5 -d calls
So I tried different values and found that "-icount 7" will reproduce the issue. Other values I tried wouldn't reproduce that issue though "-icount 100" and "-icount 0" wouldn't bring up the Canon menu -- just testing the limits.

[reddeercity]

Going to make another attempt to get QEMU running on my AMD64 , I tried to run my VM Ubuntu development VDI from amd 64 in my i7 laptop with latest version (5.2.10) of Oracle Virtualbox.
But no go , all I have access to is i386 (32bit) versions of Linux on the Intel i7 cpu , so it will not load the Ubuntu 64 vdi   
For some reason I can't get a 64bit version of  Virtualbox keeps loading as i386 , Unless someone know a work around ?
I didn't what to load a fresh 32bit version of Ubuntu as my 64bit has all the Toolchains already enabled plus I has all my source code in there .

Not to sure if this is possible to run under Win7 (I just what to run the Emulator with my 5d2 rom) So I found the latest version ( 2.11.90) https://qemu.weilnetz.de/w64/2018/
it seem to be a experimental version for 64bit on windows platform ,
I see ARMS support , what would be the  differences between "arm.exe" & "armw.exe" 64bit ?

does emulator need to be in 32bit mode ? is the Arm on canon 32 or 64bit ?
It looks to be all self contain , dll's etc. ... has a folder for rom's & bin files


Do I need still need a tool chains etc. .... or other related software or is this not possible -- do I need to run this under Linux ? 

[reddeercity]

run C:\Program Files\qemu>qemu-system-arm -machine help and got this list
interesting that out of the box it can emulate "canon-a1100----------Canon PowerShot A1100 IS"
not that I know what the hell I'm doing , but I guess gotta start some where 
maybe helpful with some info for reverse engineering

Code: [Select]

Supported machines are:
akita                Sharp SL-C1000 (Akita) PDA (PXA270)
ast2500-evb          Aspeed AST2500 EVB (ARM1176)
bast                 Simtec Electronics BAST (S3C2410A, ARM920T)
borzoi               Sharp SL-C3100 (Borzoi) PDA (PXA270)
canon-a1100          Canon PowerShot A1100 IS
cheetah              Palm Tungsten|E aka. Cheetah PDA (OMAP310)
collie               Sharp SL-5500 (Collie) PDA (SA-1110)
connex               Gumstix Connex (PXA255)
cubieboard           cubietech cubieboard
emcraft-sf2          SmartFusion2 SOM kit from Emcraft (M2S010)
highbank             Calxeda Highbank (ECX-1000)
imx25-pdk            ARM i.MX25 PDK board (ARM926)
integratorcp         ARM Integrator/CP (ARM926EJ-S)
kzm                  ARM KZM Emulation Baseboard (ARM1136)
lm3s6965evb          Stellaris LM3S6965EVB
lm3s811evb           Stellaris LM3S811EVB
mainstone            Mainstone II (PXA27x)
mcimx7d-sabre        Freescale i.MX7 DUAL SABRE (Cortex A7)
midway               Calxeda Midway (ECX-2000)
mps2-an385           ARM MPS2 with AN385 FPGA image for Cortex-M3
mps2-an505           ARM MPS2 with AN505 FPGA image for Cortex-M33
mps2-an511           ARM MPS2 with AN511 DesignStart FPGA image for Cortex-M3
musicpal             Marvell 88w8618 / MusicPal (ARM926EJ-S)
n800                 Nokia N800 tablet aka. RX-34 (OMAP2420)
n810                 Nokia N810 tablet aka. RX-44 (OMAP2420)
netduino2            Netduino 2 Machine
none                 empty machine
nuri                 Samsung NURI board (Exynos4210)
palmetto-bmc         OpenPOWER Palmetto BMC (ARM926EJ-S)
raspi2               Raspberry Pi 2
realview-eb          ARM RealView Emulation Baseboard (ARM926EJ-S)
realview-eb-mpcore   ARM RealView Emulation Baseboard (ARM11MPCore)
realview-pb-a8       ARM RealView Platform Baseboard for Cortex-A8
realview-pbx-a9      ARM RealView Platform Baseboard Explore for Cortex-A9
romulus-bmc          OpenPOWER Romulus BMC (ARM1176)
sabrelite            Freescale i.MX6 Quad SABRE Lite Board (Cortex A9)
smdk2443             smdk2443 (ARM920-T)
smdkc210             Samsung SMDKC210 board (Exynos4210)
spitz                Sharp SL-C3000 (Spitz) PDA (PXA270)
sx1                  Siemens SX1 (OMAP310) V2
sx1-v1               Siemens SX1 (OMAP310) V1
terrier              Sharp SL-C3200 (Terrier) PDA (PXA270)
tosa                 Sharp SL-6000 (Tosa) PDA (PXA255)
tt                   OpenTom (ARM920-T)
tt666                OpenTom (ARM920-T)
verdex               Gumstix Verdex (PXA270)
versatileab          ARM Versatile/AB (ARM926EJ-S)
versatilepb          ARM Versatile/PB (ARM926EJ-S)
vexpress-a15         ARM Versatile Express for Cortex-A15
vexpress-a9          ARM Versatile Express for Cortex-A9
virt-2.10            QEMU 2.10 ARM Virtual Machine
virt-2.11            QEMU 2.11 ARM Virtual Machine
virt                 QEMU 2.12 ARM Virtual Machine (alias of virt-2.12)
virt-2.12            QEMU 2.12 ARM Virtual Machine
virt-2.6             QEMU 2.6 ARM Virtual Machine
virt-2.7             QEMU 2.7 ARM Virtual Machine
virt-2.8             QEMU 2.8 ARM Virtual Machine
virt-2.9             QEMU 2.9 ARM Virtual Machine
xilinx-zynq-a9       Xilinx Zynq Platform Baseboard for Cortex-A9
z2                   Zipit Z2 (PXA27x)So is the QEMU that is used here to run ML , different then what I have here ?

[a1ex]

- First post from this thread
- Main page -> Docs -> QEMU guide
- Main page -> big red QEMU link
- Latest commit (at the time of writing)
- Twitter -> sticky post -> video
- Getting started with development
- Mac guide with video
- Windows 10 guide
- docs linked all over the place

Should I go on?

[Ant123]

- Twitter -> sticky post -> video
Should I go on?

Please recommend the exact version of linux you are using(have used) for it.

[a1ex]

That was Ubuntu Xenial, but should work just as well in any recent distribution. Was your experience different?

(I'm currently using it on openSUSE Tumbleweed)

[dfort]

@a1ex - just wondering if you ran the find_stubs.sh script on the 6D.118 and if it came up the same changes I made on 6D/debugmsg.gdb.

I take it that in order to get the full QEMU messages working on this platform I'll need to run a startup log through the scripts to make a new qemu-eos/qemu-2.5.0/hw/eos/mpu_spells/6D.h -- right?

[a1ex]

I didn't do any changes to the MPU spells, but the script found some different stubs. It still requires updating CURRENT_TASK / CURRENT_ISR manually.

[t3r4n]

Hey a1ex,
its a bit rough for start but it works. Seems like gdb 8 resolves the issue with the temporary breakpoints.

Code: [Select]

#!/bin/bash

# Mirror of gnu.org to be used.
MIRROR=https://ftp.gnu.org/gnu

#create Directory
mkdir ~/crossgcc
cd ~/crossgcc

mkdir src
cd src

# get a bunch of stuff
wget -c $MIRROR/gdb/gdb-8.1.tar.xz

# let's unpack
tar jxf gdb-8.1.tar.xz

# now build
# read about CC='gcc -m32' but also that newer gdb should handle 32 bit fine ...
mkdir build-gdb
cd build-gdb
 ../gdb-8.1/configure --target=arm-none-eabi --prefix=$HOME/crossgcc/
make all install 2>&1 | tee make.log
echo "Done, please add: "
echo $HOME/crossgcc/bin
echo "to your PATH"
Can someone with a mac verify.

[dfort]

@t3r4n -- Tried out your script and it worked perfectly. At least I think it did.

It still requires updating CURRENT_TASK / CURRENT_ISR manually.

I'm pretty sure I got CURRENT_TASK correct for the 6D.118 but have no idea where to look for CURRENT_ISR:

6D/debugmsg.gdb

Code: [Select]

macro define CURRENT_TASK 0x74C80
macro define CURRENT_ISR  (MEM(0x648) ? MEM(0x64C) >> 2 : 0)


[reddeercity]

- First post from this thread
- Main page -> Docs -> QEMU guide
- Main page -> big red QEMU link
- Latest commit (at the time of writing)
- Twitter -> sticky post -> video
- Getting started with development
- Mac guide with video
- Windows 10 guide
- docs linked all over the place

Should I go on?

Yes I know ,
I don't have Win10 & I only use MAC for Video editing (I don't know MAC good enough to understand) (I'm windows sever2008 & win7 guy)
That's the reason I'm using a prebuilt VM(Ubuntu) from Quick Guide and Solutions for VirtualBox
I guess I should of mention that I tried to get it running mouths ago but run in to AMD cpu problem (I have a FX8350) and it needed some extra support files
that I could not get to work , I posted it in the thread Qemu problem I make back in Feb/'17 but couldn't resolve it so I moved on .
That's why I'm looking for other solutions , as I noted above I wanted to just run the VM (Ubuntu) on my i7 laptop but will not load the VDI as its 64bit
thou I laptop is only a few year old so it does support 64bit , I run MAC OS on it thought Multi Beast boot loader . (It's my Mac backup & portable video editing station)
VirtualBox keep loading the 32bit version of the software on my i7 laptop instead of the 64bit version that running on my AMD desktop .

I don't need it to compile ml just need to run qemu so I can get the final info for lossless

[a1ex]

From the first section:

What does not work (yet):
...
- Native Windows build (QEMU can be compiled on Windows => contribution welcome).
...

Given that, I've assumed you jumped into trying unsupported stuff without reading the guide. Actually I'm still pretty sure about that, as the steps you have tried are in the opposite direction.

https://wiki.qemu.org/Hosts/W32 (caveat: I did not try it)

I've ran into a similar problem with some older Ubuntu machines - their repositories were moved to archive.ubuntu.org:

https://superuser.com/questions/339537/where-can-i-get-the-repositories-for-old-ubuntu-versions

However, I don't remember testing this on a 32-bit VM, so there may be surprises. Will give it a try later.

Edit: there were a few build errors, fixed. After that, installation on Xenial 32-bit was as straightforward as in the Twitter clip.

[reddeercity]

 Ok , I'll keep at it -- thank

[t3r4n]

Okay thanks @dfort for testing.
Should I turn this script into a complete cross chain builder that in the end might even reside in the ML directory universe? That way we would have an reference compiler chain that should render the same results on all platforms (notice the should and not will ).

But back to the original topic on finding stubs. @a1ex I now get a lot of functions with weird binary names in the idc file. Is that normal or do I need to investigate?
 

[a1ex]

Also confirmed gdb 8.1 handles the temporary breakpoints well. However, latest precompiled toolchains are using older versions that do not have this fix.

Currently, I only use gdb with qemu, so it may be a good idea to have it in the install script. The same issue is on WSL; hopefully the steps for compiling gdb are the same.

Can you give an example of a function with weird binary names? On 750D, with the autogenerated GDB script I get 549 named functions. All of them appear valid to me at first sight. Some examples:

Code: [Select]

  MakeAutoNamedFunc(0xFE0FD5D1, "LoadScript");
  MakeAutoNamedFunc(0xFE27B4D7, "UTimerDriver_ISR");
  MakeAutoNamedFunc(0xFE1F82F1, "PowerMgr_task");
  MakeAutoNamedFunc(0xFE42ADE3, "SRMState_S00_I00");

The other idc, saved from QEMU with -d idc, does not have any named functions; that one only has functions called during execution (their start address).

[t3r4n]

Okay I've started completely clean and I now get a lot more functions but still a lot of these:

Code: [Select]

  MakeAutoNamedFunc(0x00002404, "�F�F��V��_task");
  MakeAutoNamedFunc(0x0000240E, "�F�F��V��_task");
  MakeAutoNamedFunc(0x00002418, "�F�F��V��_task");
  MakeAutoNamedFunc(0x00002422, "�F�F��V��_task");


[a1ex]

What stub did you get for task_create?

I've got:

Code: [Select]

b *0x1E44
task_create_log

then, in the console log:

Code: [Select]

[        init:fe1f8385 ] task_create(PowerMgr, prio=20, stack=400, entry=fe1f82f1, arg=0)
[        init:000022b7 ] task_create(DbgMgr, prio=1f, stack=0, entry=2233, arg=7fe800)
[        init:fe0d9fdb ] task_create(Startup, prio=19, stack=2800, entry=fe0d9ecd, arg=7feb3c)
[        init:fe0ce221 ] task_create(RomRead, prio=11, stack=400, entry=fe0cd53f, arg=0)
...

Do you get different things?

I get the same result with both the 64-bit GDB compiled with your script, and with my old 32-bit version.

Your addresses don't seem to be valid function start addresses btw. Where do you get the task_create call for these? Hopefully these are not tasks created by ML - run with boot=0 to be sure.

[t3r4n]

It found the task create at

Code: [Select]

b *0xFE172BB2
task_create_logAnd no command line is:

Code: [Select]

./run_canon_fw.sh 750D,firmware="boot=0" -d debugmsg -s -SWill try to reset to default task_create.

[a1ex]

May I see your call trace around the first occurence of TaskMain?

Code: [Select]

CAM=750D

test_run=$( (sleep 2; echo quit) | \
    ./run_canon_fw.sh $CAM,firmware="boot=0" -d calls,tail -display none -monitor stdio \
    2>&1 | ansi2txt )

echo "$test_run" | grep -m1 -C5 TaskMain

My result (relevant bits only):

Code: [Select]

  call 0xFE3CDF4C(fe0ce600 "TaskMain", 1d, 0, fe0cd4a9)                          at [init:fe0ce241:80001735]
   -> 0x1E45                                                                     at [init:fe3cdf4c:fe0ce245]

Address 0xFE172BB2/3 is not present in my call trace (?!)