Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: jplxpto on September 23, 2012, 08:29:02 PM

Title: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 23, 2012, 08:29:02 PM
How to run Magic Lantern into QEMU?!...

I would like to know if anyone has ever launched Magic Lantern on QEMU. I've used it a few times, and it seemed to me that however limited it may be useful to conduct some tests.

If anyone has knowledge about this subject, thank you give me some tips that may also be useful for other developers.

Thank you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nanomad on September 23, 2012, 09:10:22 PM
Last time I checked there was a patch for qemu in ML source code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 23, 2012, 09:48:30 PM
Ok ... thank you...

One of these days I'll test it ...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: miyake on September 24, 2012, 10:50:23 AM
I'm now watching here.
http://chdk.wikia.com/wiki/GPL_Qemu

I wish this for something help for you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 24, 2012, 02:21:25 PM
I'm now watching here.
http://chdk.wikia.com/wiki/GPL_Qemu

I wish this for something help for you.

Thank you
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 08, 2013, 09:31:25 PM
With a bit of tweaking, that qemu patch seems to do something :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu2.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: scrax on March 17, 2013, 10:03:16 PM
I'm interested in running ML in QEMU for testing scripts, is the tweaked patch in the source?
What do I need to do compile a patched ML version and load it with QEMU?
I'm reading the page linked by miyake now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 17, 2013, 10:29:30 PM
It's not yet ready for this. What's missing is task switching support; I've tried some hacks based on setjmp (this (http://homepage.cs.uiowa.edu/~jones/opsys/threads/) and this (https://www.cs.cmu.edu/~mihaib/articole/threads/)), but they weren't reliable at all, so I'm thinking to try something like FreeRTOS. A simple cooperative scheduler (to switch tasks when they call msleep) would be enough.

G3gg0 had some success in running the DryOS task switcher. My approach is a bit different, I'm trying to re-implement the stubs from scratch so only ML code is emulated - that is, menu, scripting engine, overlays etc.

The qemu patch is very rough and not yet published.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on March 23, 2013, 03:15:40 AM
I'm very interested in this subject.

I'll be glad to help you.

Maybe this can help us ... The eCos is another option (http://ecos.sourceware.org/about.html) and it already has support for GDB.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mark.farnell on March 29, 2013, 09:51:25 PM
I'm very interested in this subject.

I'll be glad to help you.

Maybe this can help us ... The eCos is another option (http://ecos.sourceware.org/about.html) and it already has support for GDB.

In this case, is eCos a replacement of qemu?  However it says it is an operating system.... so do you mean to replace DryOS with eCos altogether? 
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 29, 2013, 09:55:12 PM
Yes, only for emulation. I don't need to emulate the entire Canon firmware, just ML code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mark.farnell on March 29, 2013, 10:50:46 PM
Yes, only for emulation. I don't need to emulate the entire Canon firmware, just ML code.

So at this stage, is it possible to emulate the ML code with eCos?  If so, how?

Also will the ability of emulating the ML code make developing safer?  ( I mean detecting silly mistakes such as writing wrong values to NVRAM variables, that can potentially brick the camera)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 29, 2013, 11:10:23 PM
See my previous post. There was no progress since then.

Wrong NVRAM values depend only on how they are interpreted by Canon code, so emulating only ML code won't help. If you get permanent ERR70 after changing some setting, that's a clear sign that you have set an invalid value; there's no other way to tell this. Maybe full emulation could help here, but that's very difficult (of course, I won't be surprised if g3gg0 succeeds). Detecting memory leaks may be possible (maybe run ML under valgrind?)

I want to emulate ML so I can check how the menus look on each camera, without having to buy every single model. Also, it may be helpful when writing user scripts, or when working with graphics code, fonts etc.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on March 30, 2013, 12:26:45 AM
I got 600D booting in trix to the service/bootloader menu but didn't really know what to do with it afterwards.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 30, 2013, 12:53:32 AM
at the moment i am working on qemu to emulate the whole fimware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Indy on March 30, 2013, 12:58:10 AM
Updater code (at least 7d one) can run partially on qemu.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 30, 2013, 01:07:34 AM
here are the changes i made to qemu:
http://upload.g3gg0.de/pub_files/7dc0800617416fdbeb1490dcc4a2164d/qemu_eos.7z

they contain a lot of hardware emulation.

booting fails imho due to the MPU that is not emulated. init routines expect a property that is not available.
i think the MPU would send it to the main firmware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on April 04, 2013, 01:33:02 AM
here are the changes i made to qemu:
http://upload.g3gg0.de/pub_files/7dc0800617416fdbeb1490dcc4a2164d/qemu_eos.7z

they contain a lot of hardware emulation.

booting fails imho due to the MPU that is not emulated. init routines expect a property that is not available.
i think the MPU would send it to the main firmware.

Thank you! I will try this ..
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 12:12:32 AM
Just merged my version with g3gg0's and pushed it on the main repo. Works on 60D, 600D, 500D, 5D2 and 650D.

In theory, you only have to run the install script (https://bitbucket.org/hudson/magic-lantern/src/tip/contrib/qemu/install.sh) - it will download QEMU 1.4.0, apply our modifications, and it will tell you what to do next.

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

If successful, you should get the hello world picture, and a log like this:

Code: [Select]
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-650D.BIN' to 0xF7000000-0xF8FFFFFF
[EOS] loading 'qemu-helper.bin' to 0x30000000-0x300088E7
[EOS] loading 'autoexec.bin' to 0x00800000-0x00855F4F
...
[GPIO] [0xC022C188] <- 0x138800 at pc=0x855E90
[FlashIF] at [0x000D4020]: 'Write enable' enabled
[Basic] at [0x000D4020] [0x00000000] <- [0xC0400008]
[Basic] at [0x000D4020] [0x00430005] -> [0xC0400008]
create_init_task(7e1ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr07.650D101 (781e0140ec5a+ (unified) tip)
[DebugMsg] (50,3) Built on 2013-04-07 20:02:47 by [email protected]
...
Hello at QEMU console!

Let me know if it works for you (and what other dependencies you had to install).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: scrax on April 08, 2013, 05:26:24 AM
Just merged my version with g3gg0's and pushed it on the main repo. Works on 60D, 600D, 500D, 5D2 and 650D.

In theory, you only have to run the install script (https://bitbucket.org/hudson/magic-lantern/src/tip/contrib/qemu/install.sh) - it will download QEMU 1.4.0, apply our modifications, and it will tell you what to do next.

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

If successful, you should get the hello world picture, and a log like this:

Code: [Select]
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-650D.BIN' to 0xF7000000-0xF8FFFFFF
[EOS] loading 'qemu-helper.bin' to 0x30000000-0x300088E7
[EOS] loading 'autoexec.bin' to 0x00800000-0x00855F4F
...
[GPIO] [0xC022C188] <- 0x138800 at pc=0x855E90
[FlashIF] at [0x000D4020]: 'Write enable' enabled
[Basic] at [0x000D4020] [0x00000000] <- [0xC0400008]
[Basic] at [0x000D4020] [0x00430005] -> [0xC0400008]
create_init_task(7e1ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr07.650D101 (781e0140ec5a+ (unified) tip)
[DebugMsg] (50,3) Built on 2013-04-07 20:02:47 by [email protected]
...
Hello at QEMU console!

Let me know if it works for you (and what other dependencies you had to install).

Will try on osx the script and report back, but first I need to clean up some space from my main hd.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 09:37:44 AM
I'm trying to run this on Arch linux:
I had to build with these options because python 2 binary is called python2 and the docs building didn't work:
Code: [Select]
./configure --target-list=arm-softmmu --python=/usr/bin/python2 --disable-docs
when I try to launch:
Code: [Select]
➜  qemu  ./run_ml_5D3.sh
make: Entering directory `/home/tom/dev/software/qemu/qemu-1.4.0'
make: Leaving directory `/home/tom/dev/software/qemu/qemu-1.4.0'
make: Entering directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
[ VERSION  ]   ../../platform/5D3.113/version.c
[ CC       ]   version.o
[ MENU IDX ]   menuindexentries.h
No menuindex.txt not running "python2 menuindex.py"
[ CC       ]   menuindex.o
[ LD       ]   magiclantern
[ OBJCOPY  ]   magiclantern.bin
[ SYMBOLS  ]   magiclantern.sym
[ CC       ]   reboot.o
[ LD       ]   autoexec
autoexec.bin: 443984 bytes

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x06c550 0x000d5950 0x000d5950 0x00008 0x00008 R   0x4
  LOAD           0x000100 0x00069500 0x00069500 0x6c458 0x7bb6d RWE 0x100
[ OBJCOPY  ]   autoexec.bin
make: Leaving directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
make: Entering directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
make: `qemu-helper.bin' is up to date.
make: Leaving directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
rm: cannot remove ‘vram.txt’: No such file or directory
rm: cannot remove ‘vram.png’: No such file or directory
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
eos_load_image: file not found 'ROM-5D3.BIN'
run_ml.sh: line 15: 25600 Aborted                 (core dumped) $QEMU_PATH/arm-softmmu/qemu-system-arm -M ML-$1
convert: unable to open image `vram.txt': No such file or directory @ error/blob.c/OpenBlob/2641.
convert: no images defined `vram.png' @ error/convert.c/ConvertImageCommand/3103.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 09:46:22 AM
5D3 is not working yet, here it fails at create_init_task. You also have to get some Canon firmware dumps, see the install instructions.

You may have better luck if you add support for 50D (you need to add a ML_MACHINE definition and create a launch script).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 10:13:39 AM
Sorry, I didn't read that well! I thought I had the 5D3-ROM in the correct location but I guess not. I now get as far as create_init_task so I assume that qemu is at least working. I'll have a go at adding support for 50D.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 11:49:06 AM
5D3 works too, I was loading the old 1.1.2 dump by mistake.

Code: [Select]
create_init_task(695ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr08.5D3113 (087dd0afd6b8+ (unified) tip)
...

It fails at redraw, you will need to comment it out from ML code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 12:36:39 PM
(http://i.imgur.com/PXf5p5t.png)

50D worked without much effort - just copied 60D definitions, changed to 50D and added the appropriate run_ml script.

I tried to generate a patch for the changes but hg diff was producing a lot of spurious changes - maybe it doesn't like diffing a diff?!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 12:55:14 PM
I've used a new hg tree under qemu to generate the diff. It's a bit ugly now; I'm thinking to modify the install script to store the new files as plain files, and use the patch only for modification to QEMU sources.

Edit: did these changes and some small additions:

- to emulate Canon firmware, without ML:
    ./run_canon_fw.sh 600D

- to generate a diff or commit changes (say you have modified eos.c or added some script):
    ./copy_back_to_contrib.sh
    then normal hg commands or gui in contrib/qemu

- to run the firmware in gdb:
    qemu-1.4.0/arm-softmmu/qemu-system-arm -M 5D3 -s -S in one terminal
    arm-elf-gdb -x gdbopts in another
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 09, 2013, 12:24:45 AM
6D is running in Qemu.. ML seemed to run but I did not see hello world in VNC.
Ran the canon FW and debugger and it opened up stopping at ff0c0008.
I guess this can be connected as debugger to ida or another disassembler?

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 09, 2013, 07:26:00 AM
Yes, you can connect from IDA or GDB to localhost:1234.

The display device is not implemented - it just saves a screenshot when you call dispcheck.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 06:03:49 AM
I compiled on windows with mingw. It loads but get:

VNC server running on `::1:5900'

and can't connect to vnc or gdb.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on April 10, 2013, 12:31:35 PM
vnc isnt implemented for eos cameras, just gdb.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 04:05:38 PM
Works in linux. You can connect over vnc and shut down qemu.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Marsu42 on April 10, 2013, 08:01:01 PM
I compiled on windows with mingw. It loads but get

It would be nice to also get it running on Windows though - of course I have Linux/whatever vms, but for ml I'd like to get around them on my puny laptop just like there is no need to use Linux to compile ml.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 11:18:38 PM
I can post the binary... the armmmu folder is only something like 20-30mb.... but I can't connect to it in windows, like it needs another patch to use the networking. The linux binary will only connect on localhost so I can't push it out to gdb over the network.

Thats the prob I'm having with QEMU.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Marsu42 on April 11, 2013, 01:35:48 AM
The linux binary will only connect on localhost so I can't push it out to gdb over the network.

Use a ssh tunnel or network redirector?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 11, 2013, 02:52:02 AM
I turned UFW off and set some forwarding from lo to eth0... so far its working. Really would be nice to get it on one machine tho.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 05, 2013, 10:16:49 AM
Is there a way to run qemu for a 550D?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 12:32:41 AM
Yep, just edit the files and add 550D + values.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 06, 2013, 02:32:19 AM
@1% Thanks. 

I changed the content of "run_ml_550D.sh" to "sh run_ml.sh 550D 109" (550 instead of 500)
but it still won't work. The 550D is not listed under supported machines. I don't know if it's important and failure is caused by something else, or caused by my installation? Well I have no vram.txt as you see. Do you have any clue?
Code: [Select]
$ ./run_ml_550D.sh
make: Entering directory `/home/wolf/qemu/qemu-1.4.0'
make: Leaving directory `/home/wolf/qemu/qemu-1.4.0'
make: Entering directory `/home/wolf/magic-lantern/platform/550D.109'
[ VERSION  ]   ../../platform/550D.109/version.c
[ CC       ]   version.o
[ MENU IDX ]   menuindexentries.h
No menuindex.txt not running "python2 menuindex.py"
[ CC       ]   menuindex.o
[ LD       ]   magiclantern
[ OBJCOPY  ]   magiclantern.bin
[ STAT     ]   magiclantern.bin
magiclantern.bin: 449232 bytes
[ SYMBOLS  ]   magiclantern.sym
[ CC       ]   reboot.o
[ LD       ]   autoexec

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x06dbc8 0x00cedbc8 0x00cedbc8 0x00008 0x00008 R   0x4
  LOAD           0x000100 0x00c80100 0x00c80100 0x6dad0 0x7d6d9 RWE 0x100
[ OBJCOPY  ]   autoexec.bin
[ STAT     ]   autoexec.bin
autoexec.bin: 449728 bytes
make: Leaving directory `/home/wolf/magic-lantern/platform/550D.109'
make: Entering directory `/home/wolf/magic-lantern/platform/550D.109'
make: `qemu-helper.bin' is up to date.
make: Leaving directory `/home/wolf/magic-lantern/platform/550D.109'
rm: cannot remove 'vram.txt': No such file or directory
rm: cannot remove 'vram.png': No such file or directory
Supported machines are:
none                 empty machine
collie               Collie PDA (SA-1110)
ML-50D               Magic Lantern on Canon EOS 50D
ML-60D               Magic Lantern on Canon EOS 60D
ML-600D              Magic Lantern on Canon EOS 600D
ML-500D              Magic Lantern on Canon EOS 500D
ML-5D2               Magic Lantern on Canon EOS 5D2
ML-5D3               Magic Lantern on Canon EOS 5D3
ML-650D              Magic Lantern on Canon EOS 650D
50D                  Canon EOS 50D
60D                  Canon EOS 60D
600D                 Canon EOS 600D
500D                 Canon EOS 500D
5D2                  Canon EOS 5D2
5D3                  Canon EOS 5D3
650D                 Canon EOS 650D
nuri                 Samsung NURI board (Exynos4210)
smdkc210             Samsung SMDKC210 board (Exynos4210)
connex               Gumstix Connex (PXA255)
verdex               Gumstix Verdex (PXA270)
highbank             Calxeda Highbank (ECX-1000)
integratorcp         ARM Integrator/CP (ARM926EJ-S) (default)
kzm                  ARM KZM Emulation Baseboard (ARM1136)
mainstone            Mainstone II (PXA27x)
musicpal             Marvell 88w8618 / MusicPal (ARM926EJ-S)
n800                 Nokia N800 tablet aka. RX-34 (OMAP2420)
n810                 Nokia N810 tablet aka. RX-44 (OMAP2420)
sx1                  Siemens SX1 (OMAP310) V2
sx1-v1               Siemens SX1 (OMAP310) V1
cheetah              Palm Tungsten|E aka. Cheetah PDA (OMAP310)
realview-eb          ARM RealView Emulation Baseboard (ARM926EJ-S)
realview-eb-mpcore   ARM RealView Emulation Baseboard (ARM11MPCore)
realview-pb-a8       ARM RealView Platform Baseboard for Cortex-A8
realview-pbx-a9      ARM RealView Platform Baseboard Explore for Cortex-A9
akita                Akita PDA (PXA270)
spitz                Spitz PDA (PXA270)
borzoi               Borzoi PDA (PXA270)
terrier              Terrier PDA (PXA270)
lm3s811evb           Stellaris LM3S811EVB
lm3s6965evb          Stellaris LM3S6965EVB
tosa                 Tosa PDA (PXA255)
versatilepb          ARM Versatile/PB (ARM926EJ-S)
versatileab          ARM Versatile/AB (ARM926EJ-S)
vexpress-a9          ARM Versatile Express for Cortex-A9
vexpress-a15         ARM Versatile Express for Cortex-A15
xilinx-zynq-a9       Xilinx Zynq Platform Baseboard for Cortex-A9
z2                   Zipit Z2 (PXA27x)
convert: unable to open image `vram.txt': No such file or directory @ error/blob.c/OpenBlob/2641.
convert: no images defined `vram.png' @ error/convert.c/ConvertImageCommand/3106.
$

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 02:53:45 AM
You have to add add it to eos.c too.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 06, 2013, 09:06:40 PM
It's working. :-)
But is it possible to test a picoc script already?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 10:34:23 PM
I wish.. I couldn't get anything useful for debugging either.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 31, 2014, 08:42:55 PM
QEMU working again :)

- runs DryOS task scheduler, semaphores, message queues (massive credits to g3gg0 for the low-level emulation code and insights)
- loads files from a local directory (sdcard or cfcard)
- able to load config files, modules, cropmarks...
- menu navigation working
- file manager working
- arkanoid working (playable!)
- properties not working
- most canon code is not working :P

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-arkanoid.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu-menu.png)

https://bitbucket.org/hudson/magic-lantern/branch/qemu

Feel free to turn it into something useful... like script interpreter, testing server, source-level debugger, HDMI emulator, support for image buffers, add a nice GUI... or just port it for your camera. I've only tested it on 5D3 1.1.3, and it's been already used to debug the early 100D ML port in GDB.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2014, 05:39:25 PM
Code: [Select]
[DebugMsg] (139,22) Wait Master Wakeup
[GPIO] at [0xFF080090] [0x00000000] <- [0xC0220024]
....
[DebugMsg] (139,6) Wait Master Wakeup Timeout
[DebugMsg] (139,22) Master Wakeup

Other than that, emulating 7D ML in QEMU works quite well. Any hints about how to emulate the master processor and the communication between them?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on October 02, 2014, 11:41:45 AM
Works fine here for 6D. Needed to do some additional stuff like filename capitalization etc. I just ran into one last problem. I have somehow no write access - at least there's no "magic.cfg" saved neither "bench.ppm" nor are ROM dumps from debug menu written etc. Though qemu doesn't report an error on qemu monitor. It's just like qemu has write access to some nirvana place in memory....

Code: [Select]
[???] [0x00000010] -> [0xC020006C] PC: 0xFF0C7C14
*** FIO_CreateFile('B:/ML/SETTINGS/magic.cfg') => 17
*** FIO_WriteFile(11, 231)
{{{
# Magic Lantern Nightly.2014Oct02.6D113 (b59a1ac5fbfc+ (qemu) tip)
# Built on 2014-10-02 08:56:42 UTC by [email protected]
}}}
*** FIO_CloseFile(11)
[FIO wrapper] closefile() nothing open
Save configs...

Edit:
qemu-helper.c define R/W access. Do I have to change the values for 6D. Btw I compiled and tried via qemu branch
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 02, 2014, 12:19:44 PM
Correct, file writing is not implemented. The entire FIO implementation is weak (only single-task and very poor error checking) - that is, just barely enough to load ML :P.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 11, 2015, 12:26:12 PM
Just had success in emulating the display test (http://www.magiclantern.fm/forum/index.php?topic=14732) (on most cameras). On 6D, I had to emulate the bootloader as well (without it, the display init routine would get stuck).

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-boot-6D.png)

To run the display test, look for the following "if (0)" and enable them:
- "bootloader config, 4 bpp" -> required to run all boot display tests
- "6D bootloader experiment" -> required for 6D; launch with ./run_canon_fw.sh 6D

edit: after some small changes, Linux works as well :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-boot-linux.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mk11174 on April 24, 2015, 03:48:31 PM
Anyone know what this HDMI USB thing is, I don't see this on the 550D or 700D when I play with them, this loops so much and makes it super slow to work the menus?
(http://s3.postimg.org/asramqo6r/Capture.jpg)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 24, 2015, 10:48:27 PM
Major progress: I'm able to launch Canon GUI under QEMU :)

( side note: Nikon Hacker guys achieved this step a long time ago (http://www.magiclantern.fm/forum/index.php?topic=8823.0), so we are just playing catch-up :P )

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-canon-gui.png)

More details:
- it launches most Canon tasks
- unmodified 60D firmware (without autoexec.bin or ROM patches) runs as well (and starts the GUI too)
- SD card emulation also works (it loads autoexec.bin and even creates the DCIM directory on startup)
- MPU emulation kinda works (it replays messages from a log file)
- sample log: 60D-qemu-canon-gui-and-sd.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/60D-qemu-canon-gui-and-sd.log)

Next steps:
- emulate unmodified autoexec.bin
- remove all those CONFIG_QEMU hacks
- implement key events as MPU messages
- CF emulation
- enable the emulation for other cameras
- do something about those huge logs
- make the code more QEMU-ish and less hackish
- write a quick start guide
- do something useful with it :)

What's the use?
- much easier to understand Canon firmware (you can see exactly what some piece of code does with the hardware)
- very useful in diagnosing soft-bricked cameras
- a way to debug your code (or Canon's) in a GUI (gdb or IDA)
- test bench for Lua scripting or for module development
- automated tests for the nightly builds (see also this proposal (http://www.magiclantern.fm/forum/index.php?topic=12396.msg119592#msg119592))

Some tips, until a more complete guide will be available:
- to load a SD card image, use something like: ./run_canon_fw.sh 60D -sd sd.img
- to display a trace of the firmware code, with disassembly, use: ./run_canon_fw.sh 60D -sd sd.img -d exec,int -singlestep
- there is a monitor console as well: ./run_canon_fw.sh 60D -sd sd.img -monitor stdio
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 25, 2015, 03:02:40 AM
good work!
a milestone, indeed :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 25, 2015, 06:50:34 AM
Thanks.

Anyone know what this HDMI USB thing is [...] ?

That's Canon's HotPlug task reading those registers. I'll add an option to quiet them down ( -d ioport, as this one is standard in qemu).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on May 25, 2015, 10:23:20 AM
Well done and cool stuff. I'll have a closer look once solved some 70D issues. The detailed sd command states from your posted log re-activated my interest in researching sd card related stuff (http://magiclantern.fm/forum/index.php?topic=12862.0). I would also like to test if I can re-produce the ML menu timeout bug now in QEMU.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 05:05:39 PM
Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:
Code: [Select]
K287 ICU Firmware Version 1.1.1 ( 3.3.7 )
[DebugMsg] (139,5)
ICU Release DateTime 2012.05.14 13:18:25
[DebugMsg] (0,3) [SEQ] CreateSequencer (Startup, Num = 6)
[DebugMsg] (0,2) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[DebugMsg] (0,3) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[DebugMsg] (50,3) Magic Lantern Nightly.2015May30.60D111 (91e1ed550ec7 (qemu))
[DebugMsg] (50,3) Built on 2015-05-30 14:50:55 UTC by [email protected]
[*****] Starting task ff02b61c(3d2494) Startup
[DebugMsg] (0,5) [SEQ] seqEventDispatch (Startup, 0)
[GPIO]     at 0xFF012604     [0xC0222004] <- 0x12      : ???
[DebugMsg] (139,5) startupEntry
[*****] Starting task ff012584(0) Startup

Code: [Select]
[*****] Starting task 1fe07970(0) menu_task
[*****] Starting task 1fe09e24(0) menu_redraw_task
[*****] Starting task 1fe1ba88(0) bitrate_task
[*****] Starting task 1fe28180(0) focus_task
[*****] Starting task 1fe291a0(0) notifybox_task
[*****] Starting task 1fe2c124(0) fps_task
[*****] Starting task 1fe34244(0) shoot_task
[*****] Starting task 1fe2cd8c(0) clock_task
[*****] Starting task 1fe3b1bc(0) audio_common_task
[*****] Starting task 1fe43770(0) livev_hiprio_task
[*****] Starting task 1fe4245c(0) cls_task
[*****] Starting task 1fe452f0(0) beep_task
[*****] Starting task 1fe4d6d0(0) console_task
[*****] Starting task 1fe544dc(0) qemu_key_poll
[*****] Starting task 1fe0d7fc(0) debug_task
[*****] Starting task 1fe1d1e0(0) tweak_task

Code: [Select]
Load modules...
Linking..
Register modules...
Load configs...
Init modules...
Updating symbols...
  [i] 404: dual_iso_get_recovery_iso 1fe1fe00
  [i] 404: dual_iso_is_active 1fe1fe00
  [i] 404: auto_ettr_intervalometer_wait 1fe2cba0
  [i] 404: auto_ettr_intervalometer_warning 1fe2cba0
  [i] 404: auto_ettr_export_correction 1fe389d0
  [i] 404: mlv_snd_is_enabled 1fe3a8f0
  [i] 404: dual_iso_get_dr_improvement 1fe48da0
  [i] 404: dual_iso_get_recovery_iso 1fe48da0
Modules loaded

But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

Code: [Select]
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT

I can send the whole log if it will help

I haven't tried booting Canon firmware yet, I need to get a hold of an SD card small enough I can make an image of it...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 09:04:11 PM
@dmilligan
I tried to load the lua module, but nothing did appear in the menu.
Anyway, the QEMU progress is impressive.

(http://i.imgur.com/vaTqI4H.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 10:39:18 PM
Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 10:51:26 PM
Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.

Yes.
/qemu/sdcard/ML/SCRIPTS/{CALC.LUA,EDITOR.LUA,SOKOBAN.LUA}
It seems to me that QEMU doesn't "look" for the script folder.
I did run your lua module on my camera and it did work very well.

$grep SCRIPTS log.txt
[FIO wrapper] readdir() => SCRIPTS size=4096 mode=10 time=556420ee
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 11:01:19 PM
IDK, perhaps some FIO routines not emulated, or incomplete
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 11:27:32 PM
@ dmilligan
How did you compile QEMU version 2.3.0?
I've got 1.6.0.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 31, 2015, 01:08:24 AM
The latest in the qemu branch uses 2.3.0. Just run the install script.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on August 19, 2015, 04:44:38 PM
Hello,

I'm trying to emulate 100D with qemu 2.3.0 and I'm running into some issues. I've got it to run quite far but there seems to be multiple issues and I'm not sure which one to try to fix next...

I've attached the log file for the longest I've got it running, and the major changes from what's in the repo is:

- Added addresses for 100D
- No boot flag, just canon stuff
- Asserts aren't fatal, they just print to serial and return to caller (GDB hack)
- DebugMsg prints to log (with color output, GDB/QEMU magic)
- Sequencer events are delayed if being propagated too early, to avoid errors (again some GDB magic)
- Added some SPI EEPROM code
- Grepped out the USB/HDMI/MIC clobber


So some questions:

Is there an SPI EEPROM in 60D?
The 100D firmware tries to load some properties from there and isn't happy with getting zeros. Not sure whether real data is necessary for the emulation, but it seems to be the reason why some asserts are raised.

There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

It also complains about I2C and RTC at some points, but I don't think those are fatal errors.

Somehow it crashes to a $pc=4 state which I 'm trying to track down atm, but any thoughts on what to do to make it run further would be very appreciated. :)

This is the qemu log:
https://www.dropbox.com/s/y92sf7babtf3t2w/100D.log?dl=0
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on November 10, 2015, 01:52:00 PM
a1ex if you read this:
May we expect an update to the qemu branch anytime soon? Sort of having issues like dmilligan encountered since you last updated qemu to 2.3.

Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:
....
But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

For me it just looks like this:
(http://i.imgur.com/P6uQbL8.jpg?1)

Qemu 1.6 worked great for 70D but with 2.3 I just get the black graphics shown as posted above whether I try 70D or 100D. For my use case it does the job for porting ROMs to see if they boot up but ofc I am hesitated to see the canon gui emulation which you presented on forums, too. The actual qemu branch also gives me a compile error in file "eos.c" at "eos_tio_handle" where I have to bypass "msg_arg1". So I just hope an updated branch might fix my problems or I simply am going to setup a new virtual machine with an older ubuntu release for qemu only. No rush...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on December 28, 2015, 11:44:51 AM
Major progress: I'm able to launch Canon GUI under QEMU :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-canon-gui.png)

A1ex, how about an update to qemu? At it's actual state from bitbucket I always have to uncomment some lines to get qemu compiled. I just set-up a fresh 12.04 Ubuntu but had the exact same error on 14.04 Ubuntu. So quoting your post from May above I assume there's some local work on your build machine which needs to be merged:

Code: [Select]
/home/ml/qemu/qemu/qemu-2.3.0/hw/arm/eos.c:2716:9: error: variable ‘msg_arg1’ set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
make[1]: *** [hw/arm/eos.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [subdir-arm-softmmu] Error 2

And once you read this: I would welcome the dmspy-experiments branch being merged into unified,too.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 26, 2016, 08:46:18 PM
Updated to run the Lua module on 60D.

(http://a1ex.magiclantern.fm/bleeding-edge/lua/qemu-lua-bug-scriptsource.png)

@nikfreak: I'm not getting this error about msg_arg1 (not even a warning), probably different cflags (I'm always (re-)installing it from contrib/qemu/install.sh script, but on an older Ubuntu). I've fixed that one anyway (it was, indeed, a bug).

@nkls: unfortunately, the MPU spells are valid only for 60D for now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on February 29, 2016, 12:30:53 PM
Really strange. As I didn't have the graphical glitch shown below in early days of 70D QEMU porting
(http://i.imgur.com/P6uQbL8.jpg?1)

The only thing I can think of atm is a wrong conflicting merge of my 70D branch into qemu's as I already tried a new ubuntu installation and experience the same. Any clue when 70D might become part of unified and merged into qemu by you? Porting began Dec 2015 and I feel it's just about time to get it into unified...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 29, 2016, 01:09:54 PM
Yeah, the stuff in the QEMU branch expects Canon GUI to do all the display initializations (including color palette).

But that only works on 60D atm...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 28, 2016, 12:11:56 AM
Small progress: CF card emulation appears to work on 5D2 bootloader, enough to load autoexec.bin from a CF image file.

The 5D2 GUI isn't working yet though...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 11, 2016, 09:47:16 PM
I've pushed my changes to qemu to my bitbucket now, it's located at:

https://bitbucket.org/niklastisk/qemu (branches: eos-develop and 100D-testing)

It was a while since I touched the code but I hope parts of it can be useful to someone at least.

Things I recall I've done:
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 11, 2016, 10:13:25 PM
Very nice work! You seem pretty familiar with qemu internals (where I'm struggling).

I'll try to integrate it with the latest branch.

Regarding your questions (sorry, I somehow missed your post, so you probably got the answers meanwhile):

Is there an SPI EEPROM in 60D?

There is an EEPROM that you can dump with this module (https://bitbucket.org/hudson/magic-lantern/commits/8c8e3329).

I didn't use this file in my emulation though.

Quote
There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

They are taken from a startup log, from the dm-spy-experiments branch (CONFIG_DEBUG_INTERCEPT_STARTUP=y in Makefile.user), that contains calls to mpu_send/mpu_recv branch. The code is generated with this script (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/scripts/extract_init_spells.py).

They are quite incomplete - just enough to boot the main dialog. I know the buttons are also sent from MPU in the same way, and it's easy to get the pattern for them. I also figured out how to decode some properties, but but most other events are still a mystery to me.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 12, 2016, 09:58:12 PM
@nikls:
i am impressed and happy to see such rev.eng. skills :)
guessing right that you are using IDA?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 16, 2016, 04:50:26 PM
Thanks guys, and thanks for the info! I was trying to to get 100D working and I guess I learned a bit about the qemu internals on the way. I wouldn't call myself an expert on the system but I know my way around bigger C/linux projects like this. :)

I never realized the dm-spy-experiments dumped the mpu calls, I guess I have to make one of 100D when I get the time!

Yeah, I'm using IDA for the decompiling and code analysis.


About that serial flash though, I'm under the impression that it's only available in newer cameras (100D and 70D? (https://bitbucket.org/hudson/magic-lantern/issues/2535/70d-black-screen-menu#comment-27815246)), and that it is necessary to emulate them to get to the main dialog in qemu. But correct me if I'm wrong.

In 100D there are references to two flash chips PC28F128M29EWHF and K5C262GATMAF50 which are both 16MB iirc. There is a function "IsAddressSerialFlash" which is used by the property manager (or some flash access abstraction) to determine if the data is stored in flash rom (same as firmware) or serial flash rom (controlled over serial interface). I think I had it setup with DMA, interrupts and whatnot but I never made it to the main dialog. :( I'll give it another shot sometime but atm I'm too busy with other projects.

Here are some extracts from the factory menu, note the S and V options in FACTADJ and the SIO6 menu for accessing the serial rom.
Code: [Select]
FACTADJ

********** FACTORY ADJUSTMENT MENU VER 0.01 **********
0. Exit from Factory Adjustment
1. Leak Check
2. SDRAM Check
3. ROM Check
4. HDMI Check
5. Video Adjustment
6. ICU Version Check
7. Audio Check
8. Adjustment Data Display and Change
9. Check Flag Display and Initialization
A. ALL Check
B. ALLP Check
F. FROM ID Check
P. Power Domain Check
S. Serial Flash Check
V. Serial Flash Version Check
Z. Input_Device_Unique

S
Serial Flash Check.
346
Select OK or NG

V
SROM Version Check.
4.2.1
Select OK or NG


FROMUTILITY

************ FROMUTILITY MENU Ver 0.09 ************
[Type:346 Body:DC Revision:0.00]
0.Exit from FROM Menu
1.Erase Sector(0x20000)
2.Erase Chip
4.Write from card
5.Write from DRAM
6.Firm   flag 0xF8000000 0x00000000 ON
7.Boot   flag 0xF8000004 0x00000000 OFF
8.UpDate flag 0xF800000C 0xFFFFFFFF OFF
9.Create Boot Disk
A.Exec Program from card
B.ALL Block Unprotect
C.Connect card
G.Memory Dump
I.Write Data
J.Direct Jump
P.Power Domain
S.SROM Menu
U.Firm update
Z.RCbind.bin update

S

**** SROM(SIO6) Menu ****
0.Exit from SROM Menu
1.Erase Chip   0x01000000
2.Erase Block  0x0000F000
3.Erase Sector 0x00001000
4.Write Data
5.Write from Card
6.SROM Dump(SIO Read)
7.SROM Dump(QUAD Read)
8.Get Info
9.Get Version

8
8.Get Info 0x80000346

9
9.Get Version 4.2.1
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 16, 2016, 06:52:51 PM
Yes, experience with Linux definitely helps. I don't have that one either.

I tried to run your serial flash dumper on 5D3, but the serial flash appears unused here. There are no mentions of it in the startup log, and calling SF_CreateFlash locks up. Didn't try to debug further. The 5D3 firmware has these strings: PC28F128M29EWHF and MBM29LV640EBXEI70G.

There are plenty of serial flash messages on the 70D startup log, and from the errors, I think they are needed for emulation. Fortunately, the communication protocol doesn't seem too complicated.

70D startup log says:
Code: [Select]
[SF] InstallSerialFlash 4 0xc022002c 0x0 0x800000 1

and these strings are present: PC28F128M29EWHF and 64Mt8Kx8m64Kx126b8Kx8.



Quick question: when emulating the factory menu, how are you entering the input selection?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 16, 2016, 09:51:58 PM
The chip install code is run quite early in the init task, this is how it looks in IDA:
(http://i.imgur.com/0ENFsrm.png?1)

As you can see there is a flag (derived from FROM) which determines what chip to install. On a second thought, those routines must be for setting up writing to main flash.

The function call before "SerialFlash Initialize" is definitely for SROM though, I guess there would be some similar call in 70D but not in 5D3? Maybe the 5D3 has more space in main flash memory and thus no need for it?
In 100D the output is
Code: [Select]
[SF] InstallSerialFlash 6 0xc022c0d4 0xf0000000 0x1000000 1
so it's similar.

I think I've got most of the protocol working, take a look in this file in 100D-testing (https://bitbucket.org/niklastisk/qemu/src/ada7b6496bde7ebc6d7cc6b1557e9710156cc129/hw/eos/serial_flash.c?at=100D-testing&fileviewer=file-view-default). It gets messy once you try to emulate DMA since it is shared with the SD memory transfer.


Quick answer: (too much of a hack to push to bitbucket)
Code: [Select]
unsigned int eos_handle_tio ( unsigned int parm, EOSState *s, unsigned int address, unsigned char type, unsigned int value )
{
    unsigned int ret = 1;
    const char * msg = 0;
    int msg_arg1 = 0;

    switch(address & 0xFF)
    {
        case 0x00:
            if(type & MODE_WRITE)
            {
                if((value == 0x08 || value == 0x0A || value == 0x0D || (value >= 0x20 && value <= 0x7F)))
                {
                    printf("\x1B[31m%c\x1B[0m", value);
                    return 0;
                }
            }
            else
            {
                return 0;
            }
            break;

        case 0x04:
            msg = "Read byte: 0x%02X";
            msg_arg1 = s->tio_rxbyte & 0xFF;
            ret = s->tio_rxbyte & 0xFF;
            while (ret == 0 || ret == EOF) {
                ret = getchar();
            }
            return ret;
            break;
       
        case 0x08:
            /* quiet */
            return 0;

        case 0x14:
            if(type & MODE_WRITE)
            {
                if(value & 1)
                {
                    msg = "Reset RX indicator";
                    s->tio_rxbyte |= 0x100;
                    return 0; /* nkls: quiet */
                }
                else
                {
                    /* quiet */
                    return 0;
                }
            }
            else
            {
                if((s->tio_rxbyte & 0x100) == 0)
                {
                    msg = "Signalling RX indicator";
                    ret = 3;
                }
                else
                {
                    /* quiet */
                    return 3; /* nkls: signal something on the line */
                    //return 2;
                }
            }
            break;
    }

    io_log("TIO", s, address, type, value, ret, msg, msg_arg1, 0);
    return ret;
}
is there a hidden menu option (https://bitbucket.org/niklastisk/qemu/src/2d85ec1ee3a5feb3233183ee8180c8b10b04e5cd/hw/eos/eos_bufcon_100D.h?at=100D-testing&fileviewer=file-view-default) in other firmwares as well?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 17, 2016, 01:07:19 AM
one problem of qemu was, you couldnt emulate real flashes. (not SPI)
to write properties, the firmware is issuing flash state machine commands (writes to the flash base address)

QEMU cannot handle (passing writes to a driver) this. at least the version of 1.5 years ago wasn't able.
IIRC this has to do with the fact that the ARM instruction set isn't interpreted, but somewhat translated.

so it might be troublesome to get the whole camera hardware behave correctly using QEMU, if there is no workaround.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 19, 2016, 06:37:29 PM
@nkls: I think I've managed to integrate your code with my latest changes. It was a little difficult, and I'm not yet sure I did the merges right, so may I ask you to check whether it's still working fine on your side?

The DebugMsg hack is pretty cool - I just managed to boot the 60D GUI with full debug messages, without any additional guest code (no autoexec.bin loaded).

Didn't try the other stuff yet.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 21, 2016, 10:25:54 PM
Now that @nkls introduced me to the world of GDB scripting, here's a quick tip for tracing various functions around the firmware:

Code: [Select]
# task_create
br *0x98CC
commands
  silent
  printf "task_create(%s, prio=%x, stack=%x, entry=%x, arg=%x)\n", $r0, $r1, $r2, $r3, *(int*)$sp
  c
end

br *0xFF12CB14
commands
  silent
  printf "SearchFromProperty(%x,%x) from %x\n", $r0, $r1, $pc
  c
end

I also tried to implement the DebugMsg hook with pure gdb scripting, but I got stuck because gdb wants the exact number of arguments to the format string. For example, this works for a DebugMsg call with a single % in it, but fails otherwise.

Code: [Select]
  eval "printf \"[DebugMsg] (%%02x,%%02x) %s\n\", $r0, $r1, $r3", $r2

Any hints would be welcome.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 30, 2016, 10:26:16 PM
Nice to see things pushed to the main repo and that you've sorted out the open ends I left in there! It seems to work and looks correct, so I think it's all good.

I remember trying to get the debug messages showing through pure gdb for a while but then gave up and hacked it together in C instead. It might be possible to some extent but I bet it'll be ugly. Maybe count %-signs and branch to different cases?

The gdb-qemu link isn't ideal either, the current:
Code: [Select]
set *0xCF999001 = *0xCF999001triggers the IO-handle when it reads from *0xCF999001 but not when it writes to it and I can't figure out why. Not that it matters much, but it would be nice to be able to send values to qemu through a single address.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 12, 2016, 12:47:44 PM
Maybe count %-signs and branch to different cases?

Yes, that's exactly what I ended up with. I made a generic logging library for various functions around the firmware (tasks, semaphores, timers, interrupts, MPU communication), all in pure gdb:

https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/debug-logging.gdb

This file can be included in the model-specific GDB script, like this:
https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/5D3/debugmsg.gdb
https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/70D/debugmsg.gdb

and then get a log from both QEMU and GDB on the same terminal with a command like this:
Code: [Select]
./run_canon_fw.sh 5D3 -s -S & arm-none-eabi-gdb -x 5D3/debugmsg.gdb

The functionality is similar to the one in the dm-spy-experiments branch (http://www.magiclantern.fm/forum/index.php?topic=2388.msg113154#msg113154), but - as you pointed out - it works without changing the guest code or having to load autoexec.bin (for example, I could use it on 7D2).

Also, during the past two weeks (when I was without internet access), I made huge progress regarding emulation of 70D, 5D3, 7D2 and EOS M3. Canon GUI doesn't start yet, but it shouldn't be very far away.

I've also extended model_list to include other model-specific parameters, and made it easy to add new ones:

https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/eos/model_list.c

BTW, I suspect the serial flash contents are not fully correct. In your dumper, I changed the buffer to uncacheable (fio_malloc) and got slightly better results (property blocks were recognized), and I also had to do this change (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls) when reading the serial flash contents via DMA. The data appears offset by half-byte, and on regular (non-DMA) reads, Canon code fixes it in the same way (in ReadBlockSerialFlash, for blocks smaller than 0x200). I guess the DMA engine is expected to apply the same "fix".

However, after these two changes, I still suspect data corruption (couldn't parse the property data structures completely - the first few properties are fine, and after a while it's pure gibberish, at least on the 70D SF dump from nikfreak). Guess we'll have to dump the RAM address where the serial flash contents is loaded (or maybe attempt to read the entire serial flash with a single call in the dumper).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Walter Schulz on June 12, 2016, 03:03:05 PM
Also, during the past two weeks (when I was without internet access), I made huge progress regarding emulation of 70D, 5D3, 7D2 and EOS M3.

EOS M3? Didn't see that coming ...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 12, 2016, 04:46:49 PM
Well, it's the only other DIGIC 6 camera for which I have a firmware (or did any other camera get a firmware update meanwhile?)

Sure, it's a PowerShot, but the DryOS core is the same, so I actually used it in order to understand the 7D2 code better. I only tried to emulate Canon firmware, didn't try to load any sort of custom code on it yet.

I'll post more details about the M3 on the relevant thread, since this emulation can also be useful for those who are porting CHDK on this camera.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on June 13, 2016, 10:57:16 PM
Wow, I'm impressed! Nice job! :) I won't be able to try it out in the upcoming two weeks, but I'll give it a go after that.

You are right about the sf data being manipulated in some way, I wondered what strange format would produce data like:
Code: [Select]
0000fff0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00010000: 0fff f000 0dcf 70f0 0000 0000 1d0f 70f0  ......p.......p.
00010010: 0000 0000 0040 3000 0000 0000 10c0 0000  .....@0.........
00010020: 0fff ffff f010 0000 10c0 0000 0fea fdcb  ................
00010030: a080 0000 1180 0000 0000 0000 0000 0000  ................
00010040: 0000 0000 0000 0000 0020 0000 1180 0000  ......... ......
but now that you say it it looks very half-byte shifted at 0x10000.

Interestingly, the first block contains the version string "4.2.1" which is not half-byte shifted.
Code: [Select]
00000000: 4603 0080 342e 322e 3100 0000 0000 0000  F...4.2.1.......
00000010: 0000 0000 ffff ffff ffff ffff ffff ffff  ................
00000020: ffff ffff ffff ffff ffff ffff ffff ffff  ................

Is it just me or is the write address not incremented in your fix (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls)?

So wait, is the data still unshifted when you use uncacheable memory? What happens with the data block exactly -- is the first half-byte always set to zero, and the last half-byte discarded? Any indication of the higher-level calls requesting more data than asked for, or is the last byte of a read always assumed to be bogus? (note to self)

You could also try to force it to only use non-DMA/only-DMA with gdb and see if that works better, I recall getting different results when doing that.

Guess we'll have to dump the RAM address where the serial flash contents is loaded (or maybe attempt to read the entire serial flash with a single call in the dumper).
It'd be tricky to allocate 16MB(?) for the full flash wouldn't it? It would also be interesting to see how e.g. one 2kb read compares to two 1kb reads, maybe we just need to overlap the block reads by a few bytes and unshift them to make it work.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 14, 2016, 07:14:38 AM
Is it just me or is the write address not incremented in your fix (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls)?

Wow, great catch; it fixes a bunch of asserts in 70D log, regarding those properties located in serial flash.

Quote
So wait, is the data still unshifted when you use uncacheable memory?
Yes.

Quote
What happens with the data block exactly -- is the first half-byte always set to zero, and the last half-byte discarded?

Canon code does this:
Code: [Select]
if ( size < 0x200 )
{
      readSerialFlash(src, buf, 512);
      while ( offset < size )
      {
        *(_BYTE *)(out + offset) = 16 * *(_BYTE *)(buf + offset) | (*(_BYTE *)(buf + offset + 1) >> 4);
        ++offset;
      }
}
else
{
    v6 = readSerialFlashWithQuad(src, out, size);
}

Quote
It'd be tricky to allocate 16MB(?) for the full flash wouldn't it?

Nope, just try it.

(some background info here (http://www.magiclantern.fm/forum/index.php?topic=5071.msg166799#msg166799) and here (http://www.magiclantern.fm/forum/index.php?topic=8358.0))
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on June 19, 2016, 12:26:47 PM
I had another look at the flash memory yesterday, and I figured out a few things. Your code appears to load the serial flash correctly for the 100D without changes, and it might be that it's just the spells causing the property errors. (Still no canon gui.)

The chip in my camera is a Winbond W25Q128FV (datasheet (https://www.pjrc.com/teensy/W25Q128FV.pdf)), and not a Macronix as I thought before. The Manufacturer Code and device ID is {0xEF,0x40,0x18} which is what I got from SPI instruction 9Fh. This doesn't matter much since it works anyways, but it's good to have the actual data sheet to base the flash emulation of.

There are two high-level interfaces (with the same debug names) for the serial flash. One uses SPI-only transfer with "serial layout", while the other is "QUAD" read which reads data through DMA, or SPI if the read block is small enough. The DMA read has this shifted data layout, maybe due to physical wiring limitations or obfuscation or whatnot. The code you posted is used by the property manager, and is from the QUAD interface, which has to un-shift the SPI data to ensure proper data layout.

This explains why the version string in the first block is readable -- it is read and written by the SPI-only interface which don't expect DMA layout. I've used the address of the SPI-only interface in my dumper, so the data in the dump should be the same as in the actual flash.

I've also tried to change the block size, and there is no difference between the data received from a read-all-at-once dump file and a 1024-byte fio_malloc'ed dumper.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 19, 2016, 09:10:33 PM
it might be that it's just the spells causing the property errors.

These should be easy to fix. Did you take the spells from another camera, or did you log the 100D startup with dm-spy-experiments?

Spells from another camera will cause issues.



BTW, I've included a small SD/CF image here (https://bitbucket.org/hudson/magic-lantern/commits/84b3f86ec6921eb65783764fa341801d88841209), hopefully this makes it easier to install and get started. The card image is bootable and includes a small autoexec.bin that does the display test (http://www.magiclantern.fm/forum/index.php?topic=14732.0), as this runs on most (if not all) DIGIC 4/5 models out of the box.

That means: run the install script, compile QEMU, copy your ROM from the ML card and it should be ready to go.

Still, it won't display the GUI on cameras other than 60D...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 20, 2016, 11:28:05 AM
Okay, I know you won't believe this one.

After about 2 weeks of intensive work on QEMU, I couldn't manage to get 70D and 5D3 working.

Today, after about half an hour of tinkering... 1200D Canon GUI boots.... with the MPU SPELLS from 60D!

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-qemu.png)

Emulation log: 1200D-qemu.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-qemu.log)

These were the changes I did to QEMU for 1200D: [1] (https://bitbucket.org/hudson/magic-lantern/commits/7241ce6bec50947082bcec05ad5078cf963baf08) [2] (https://bitbucket.org/hudson/magic-lantern/commits/507c96ed43b51c80d7776d70ddd12d00e366b0a5) [3] (https://bitbucket.org/hudson/magic-lantern/commits/008aaa6fa261beb45a27bdd08de919dbd3724fcb). And a GDB script to help me see what's going on: [4] (https://bitbucket.org/hudson/magic-lantern/commits/60225f0e05c09b551aeaef001237723c73f96668).

I tried the 60D spells just to see how it goes, but I didn't hope it would go that far.

(side note: currently we can see the MPU spells only after ML is ported, as they come from a secondary CPU whose firmware is pretty much impossible to understand, at least for me)

At this point, I think it's worth trying most other DIGIC 4 cameras with SD card. They probably require a MPU spell log (easy to get, as described here (http://magiclantern.fm/forum/index.php?topic=2864.msg166938#msg166938)), and probably a few other minor tweaks. Happy hacking!

P.S. looks like it took me more time to write this message, than to actually get the 1200D GUI booting :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 20, 2016, 07:59:02 PM
Some more:

1100D also ran out of the box with 60D spells:
(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1100D-qemu.png)

550D is stubborn; I ended up with this after patching lvInit (not yet committed):
(http://a1ex.magiclantern.fm/bleeding-edge/qemu/550D-qemu.png)

600D needs its own MPU spell set (most likely because of the crop video mode settings, which are stored in the MPU), but I expect it to work without much trouble.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 24, 2016, 12:02:04 AM
alex, I am having a lot of errors like

/home/matias/qemu/qemu-2.5.0/hw/arm/../eos/eos_handle_serial_flash.c:38:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
/home/matias/qemu/qemu-2.5.0/hw/arm/../eos/eos_handle_serial_flash.c:38:5: note: use option -std=c99 or -std=gnu99 to compile your code

I've fixed some, but seems that some config is missing, or i am wrong?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2016, 03:38:14 PM
Interesting; I used to have C99 errors before, but they were no longer present after upgrading to QEMU 2.5.0 and Ubuntu 15.10. I thought QEMU devs enabled C99 in newer versions, so I started using C99 constructs in my code.

Looks like the answer is here:

https://gcc.gnu.org/onlinedocs/gcc-4.8.0/gcc/Standards.html
Quote
The default, if no C language dialect options are given, is -std=gnu90

https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Standards.html
Quote
The default, if no C language dialect options are given, is -std=gnu11

Try adding -std=gnu99 to CFLAGS in your QEMU Makefile, or when configuring it. For example:

Code: [Select]
/path/to/ml/qemu/qemu-2.5.0$ CFLAGS=-std=gnu99 ../configure_eos.sh

(this should be included in the installer, actually)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 24, 2016, 07:09:04 PM
Great,

I was able to run 1200D with Hello world in qemu 2.5.0
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2016, 11:48:28 PM
Interesting, after trying a couple of times (and getting different execution logs every time) I finally got your hello world running as well.

Looks like there is a race condition somewhere in the emulation.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 29, 2016, 05:50:04 AM
I don't know why but it's throwing an error if i remove hello world from define. the error is not clear (at least for my)

Error loading 'ML/MODULES/1200D_100.sym': File does not exist
while in the emulator I see some error like symbols not found (dissapears fast), if I hit del key screen goes black.

but tried in 1.6 version, no error and del key givesme ML menu.

I don't know how did you do to display GUI in qemu
(before testing port in my camera I always try to see it working in qemu)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 29, 2016, 07:31:07 AM
If the file does exist, ML cannot load it because its name is longer than 8 characters.

Were you able to get the GUI in QEMU 2.5.0 like this?

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-hello-qemu.png)

For me, it doesn't work every time; I have to start QEMU a couple of times to get this screen. This only happens when loading autoexec.bin; if I try to run plain Canon firmware (with bootflag disabled), it works every time.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on June 29, 2016, 07:47:20 AM
Error loading 'ML/MODULES/1200D_100.sym': File does not exist
while in the emulator I see some error like symbols not found (dissapears fast), if I hit del key screen goes black.

If 8.3 filenaming applies to qemu, too then you got your answer now: shorten to "x70_100.sym" or something like that.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 29, 2016, 12:23:03 PM
If 8.3 filenaming applies to qemu

QEMU actually emulates a SD card device, with all the low-level communication (including DMA transfers), so filesystem behavior should match Canon's. For example, if you start the emulation on a formatted SD image, you will see Canon firmware creating the DCIM and MISC folders.

I expect formatting from Canon menu should work as well, just didn't try it. This needs the MPU spells (button codes and whatever other GUI events there might be) for navigating Canon menu.

For CF cards, emulation currently works properly in the bootloader (e.g. loading autoexec.bin from card), but not in the main firmware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 30, 2016, 04:35:33 AM
Well tried to reinstall qemu (in case my fail) but same errors appears.
using 1200D branch, if I run it with HELLO_WORLD I get this: (no cannon GUI)
(https://s31.postimg.org/t6cdd17cn/Screenshot_from_2016_06_29_19_05_48.png) (https://postimg.org/image/t6cdd17cn/)

if i remove it i get file not found as I said before. (Double checked and the file exist, this model 1200D only has SD card so i am just mounting sd, notice the file in the screenshot)

(https://s31.postimg.org/rp8dhtvqf/Screenshot_from_2016_06_29_19_19_56.png) (https://postimg.org/image/rp8dhtvqf/)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 30, 2016, 06:45:11 AM
Copy all ML files (make install), not just autoexec ;)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 30, 2016, 06:17:01 PM
600D is just as stubborn as 550D, but easier to debug (more recent codebase).

With proper MPU spells, it gives the date/time screen, just like 550D.

With a small patch on the RTC init routine, it gives the sensor cleaning animation (real-time!)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/600D-qemu-datetime.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/600D-qemu-sensorcleaning.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 01, 2016, 05:35:51 PM
500D :
Code: [Select]
0xFF18A884 - mpu_send
0xFF05C1F0 - mpu_recv
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 08, 2016, 10:58:19 PM
Some progress understanding MPU messages:

Button codes

They are encoded like this:
Code: [Select]
0x06, 0x05, 0x06, 0x00, btn_code, btn_code_arg

The button codes can be found from bindReceiveSwitch - this translates the MPU button codes (btn_code, btn_code_arg) into GUI button codes as used by GuiMainTask (the BGMT constants from gui.h). When the GUI button codes are sent to GuiMainTask, they appear in the debug log as "GUI_Control:%d 0x%x". When they are actually processed, they appear as "GUI_CONTROL:%d".

Since finding these button codes for each camera would be incredibly boring, I wrote a Python script (https://bitbucket.org/hudson/magic-lantern/commits/d40c4d99e99351e3b6ee77933885dee9f85977cb#chg-contrib/qemu/eos/mpu_spells/extract_button_codes.py) to get them automatically from the ROM, by directly emulating bindReceiveSwitch in unicorn (http://www.unicorn-engine.org), trying all usual input values and checking debug messages.

btn_code_arg meaning can be: press/unpress (1,0), scrollwheel direction (1, -1) and number of steps for very fast turns (2, -3 etc), or some buttons can be grouped under a single btn_code (for example, the direction pad).

Many button codes are common across all cameras (DIGIC 4 and 5): MENU (0,1), INFO (1,1), PLAY (3,1), DELETE (4,1), SET (12,1/0), scrollwheels (13 and 14), others are not.

btn_code = 30 is ServiceMenu on all cameras. Interesting string on 1200D: "Enter Secret mode Electric Shutter!!!".

Some cameras also use a generic event, GUICMD_PRESS_BUTTON_SOMETHING, which I don't know how to interpret (other than some button was pressed).

Side note: this research uncovered a few subtle bugs regarding button codes on 600D, 100D and EOS M, and not-so-subtle on 1100D (see this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/743/misc-cleanups-some-gui-button-codes/diff)).

GUI modes

On 600D, menu navigation looks somewhat like this ("spell" being data sent from ICU to MPU, and "reply" being the response from MPU):
Code: [Select]
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #44 */
        { 0x06, 0x05, 0x03, 0x17, 0x9a, 0x00 },                 /* reply #44.1 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #44.2, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x00, 0x01, 0x00 },                 /* reply #44.3, BGMT_MENU, GUI_Control:6, bindReceiveSwitch(0, 1) */
        { 0x06, 0x05, 0x04, 0x0d, 0x00, 0x00 },                 /* reply #44.4 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #45 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #45.1, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x00, 0x01, 0x00 },                 /* reply #45.2, BGMT_MENU, GUI_Control:6, bindReceiveSwitch(0, 1) */
        { 0 } } }, {
    { 0x06, 0x05, 0x04, 0x00, 0x01, 0x00 }, {                   /* spell #46, NotifyGUIEvent(1) */
        { 0x06, 0x05, 0x06, 0x0a, 0x00, 0x00 },                 /* reply #46.1, BGMT_UNPRESS_ZOOMOUT_MAYBE, GUI_Control:17, bindReceiveSwitch(10, 0) */
        { 0x06, 0x05, 0x06, 0x09, 0x00, 0x00 },                 /* reply #46.2, BGMT_UNPRESS_ZOOMIN_MAYBE, GUI_Control:15, bindReceiveSwitch(9, 0) */
        { 0x06, 0x05, 0x04, 0x00, 0x01, 0x01 },                 /* reply #46.3 */
        { 0x0e, 0x0c, 0x0a, 0x08, 0x11, 0x00, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00 },/* reply #46.4 */
        { 0 } } }, {
    { 0x08, 0x06, 0x00, 0x00, 0x04, 0x00, 0x00 }, {             /* spell #47, Complete WaitID = 0x80020000 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x34, 0x00, 0x00 }, {                   /* spell #48 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #49 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #49.1, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x01, 0x00 },                 /* reply #49.2, BGMT_PRESS_RIGHT, GUI_Control:35, bindReceiveSwitch(26, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x00, 0x00 },                 /* reply #49.3, BGMT_UNPRESS_RIGHT, GUI_Control:36, bindReceiveSwitch(26, 0) */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #49.4, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x01, 0x00 },                 /* reply #49.5, BGMT_PRESS_RIGHT, GUI_Control:35, bindReceiveSwitch(26, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x00, 0x00 },                 /* reply #49.6, BGMT_UNPRESS_RIGHT, GUI_Control:36, bindReceiveSwitch(26, 0) */

NotifyGUIEvent (called by SetGUIRequestMode) sends a message like this:
Code: [Select]
0x06, 0x05, 0x04, 0x00, event_code, 0x00

The MPU is supposed to reply something, probably this:
Code: [Select]
0x06, 0x05, 0x04, 0x00, event_code, 0x01

Now the interesting part: if I enable the NotifyGUIEvent reply on 60D, and the other cameras that accept the same MPU spell set, they no longer boot the GUI: instead, they go to the date/time dialog. You can adjust the date/time in QEMU using the arrow keys, scrollwheels, and the spacebar (SET), but when pressing OK, the GUI freezes.

Obviously, with NotifyGUIEvent disabled, the GUI mode can't be changed (so you can't enter Canon menu, or playback mode, or whatever).

The question is: how to make the GUI mode switches work, so one can navigate the menu?



Note: with current implementation, you can already navigate ML menu without CONFIG_QEMU=y... if you define GUIMODE_ML_MENU = 0 (so it won't try to change the GUI mode, because that part doesn't work). One big step closer towards running unmodified ML in QEMU :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 18, 2016, 01:58:19 AM
500D :
(https://s31.postimg.org/aa32bvxff/500_D.png) (https://s32.postimg.org/teuuqtlvp/500d.gif) (https://s31.postimg.org/vxp1gewpn/500d1.png) (https://s31.postimg.org/5qnukgefv/500d2.png) (https://s31.postimg.org/ok9nhgcnv/500d3.png)
Code: [Select]
# ./run_canon_fw.sh 500D -s -S & arm-none-eabi-gdb -x 500D/debugmsg.gdb

source -v debug-logging.gdb

macro define CURRENT_TASK 0x1A74
macro define CURRENT_ISR  (*(int*)0x664 ? (*(int*)0x668) >> 2 : 0)

b *0xFF066A98
DebugMsg_log

b *0xFF069E2C
task_create_log

b *0xFF064520
load_default_date_time_log
macro define RTC_VALID_FLAG (*(int*)0x2BC4)

cont
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: budafilms on July 21, 2016, 12:09:46 PM
Hi everybody,
an optimistic question from someone without skills to use this: wich is the real utility/advantage of this?

(I mean, for example, more memory, more resolution, new language for coding, girls...)

Thanks!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Walter Schulz on July 21, 2016, 12:53:02 PM
Testing code without cam:
- Cam will not brick
- You have not to wear out gear (card, slot, cardreader, etc.) each time you have to replace binaries.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 21, 2016, 01:33:09 PM
Testing code without cam:
- Cam will not brick
- You have not to wear out gear (card, slot, cardreader, etc.) each time you have to replace binaries.

- debugging your code like a PC program, by running it step by step (not just with printf's)
- you can debug Canon code as well, in order to understand what it does
- possible to implement automated tests for the nightly builds (see also http://direct.magiclantern.fm/forum/index.php?topic=12396.0 )
- test Lua scripts in the emulator
- lower the entry barrier for new developers
- assist me in unbricking cameras (the main driving force behind this, sadly)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 21, 2016, 03:22:31 PM
500D, LV
Code: [Select]
mpu_send(06 04 09 00 00)
mpu_recv(3c 3a 09 00 3c 3c e0 00 3f 80 00 00 38 12 c0 00 b9 cb c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08 08 11 10 50 49 02 59 88 88 00 32 00 00 00 00 00 01 00 00 00)
PROP_LV_LENS


0x32 - focal length
0x10 - aperture

Now we can read lens_info in Photo mode.
Just call mpu_send(06 04 09 00 00). CPU receives data and automatically overwrite property lens_info.

(https://s32.postimg.org/3kk7nrbqd/500d.jpg) (https://s32.postimg.org/hfii6865h/500d2.jpg)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 22, 2016, 11:27:57 AM
Nice trick.

Let's continue the discussion regarding MPU communication here: http://www.magiclantern.fm/forum/index.php?topic=17596
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on July 29, 2016, 10:33:43 AM
I've managed to get the dm-spy-extra branch to run for 100D (fw 100A) with full logging, sources are at:
https://bitbucket.org/niklastisk/magic-lantern/branch/100D-mpu-spy

Still no success at emulating the 100D though, just including the correct mpu messages did not work out.


On a side note, me and a1ex discovered some interesting things about the memory management of the 100D's 256M RAM when using the CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP configuration:
Code: [Select]
      p1=&memtest; p3=UNCACHEABLE(p1);
      init:00b1c75c:00:00: POINTER   p1         p2         p3
      init:00b1c77c:00:00: ADDRESS   0x00B7F204 0x10B7F204 0x40B7F204
      init:00b1c7a8:00:00: (init)    0x00000123 0x00000123 0x00000123
      init:00b1c7e0:00:00: memtest++ 0x00000124 0x00000123 0x00000123
      init:00b1c818:00:00: (*p1)++   0x00000125 0x00000123 0x00000123
      init:00b1c850:00:00: (*p2)++   0x00000125 0x00000123 0x00000123
      init:00b1c888:00:00: (*p3)++   0x00000125 0x00000124 0x00000124
(source code: http://pastebin.com/vHWcH9ij)

My interpretation is that (0x10000000 | ptr) creates a pointer that wraps around the 256M ram, but bypasses the cache and is read-only such that write attempts fail silently. This obviously won't work for 512M cameras. :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on August 01, 2016, 11:04:47 AM
I've fixed the serial flash reads, turns out the QUAD mode discards the last 16 bytes in each 0x800-byte block. Now the property tables are read correctly.

https://bitbucket.org/niklastisk/qemu/branch/eos-develop-new

The spell sets for 100D are included as well, and I even got the GUI showing (once)! There still seems to be problems with the touch panel, but I don't know if they are critical or not.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 10, 2016, 11:54:09 PM
Just committed some updates, mostly written while I had limited network connectivity (which is why they weren't uploaded earlier):

- directory layout changed, you need to move the ROM files (see install script)
- 5D3 boots the GUI on 1.1.3 :)
- cache hack emulation (very incomplete, only tested on 60D dm-spy, which runs successfully and saves its log on the SD card image)
- SD emulation fix (card info is now read correctly and gets mounted on digic 5 models)
- experimental EDMAC emulation (didn't really play with it, but can be used to research the picture taking process, bit depth reduction tricks, eeko and lots of other low level stuff)
- initial support for 750D and 760D (nothing interesting, they crash in the same way as 80D)
- integrated nkls' changes about 100D, but not yet tested (they will certainly be helpful for 70D and most other recent cameras)

I'm also starting to get confident about emulating the TX19A in QEMU, thanks to leegong's findings, but I have nothing interesting to show yet.

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 26, 2016, 01:07:16 AM
Small updates:

- card LED emulation
- better SD emulation on DIGIC 6 (tested on EOS M3 - it loads DISKBOOT.BIN (https://chdk.setepontos.com/index.php?topic=12542.msg130078#msg130078) from the SD card image)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 26, 2016, 11:52:52 PM
We now have a test suite (https://bitbucket.org/hudson/magic-lantern/commits/99d05c437b26edd515342b03528c183f0f437e4c) for QEMU :)

Code: [Select]
./run_tests.sh
Compiling...

Testing bootloaders...
  5D2: K218 READY
  5D3: K285 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
 7D2S: K289S READY
  50D: FAILED!
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY

Testing display from bootloader...
Setting up a temporary SD card image...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’
  5D2: OK
  5D3: OK
   6D: OK
   7D: please check
 7D2M: OK
 7D2S: OK
  50D: please check
  60D: OK
  70D: OK
  80D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1100D: OK
1200D: OK
 EOSM: OK
Restoring your SD card image...

Testing Canon GUI...
  60D: OK
  5D3: OK
 600D: OK
1200D: OK
1100D: OK

Can you recommend some testing framework that could work in this case? I feel like I'm reinventing the wheel with all those scripts, but at least it works. For now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 30, 2016, 12:03:45 AM
Added support for a few more cameras (they all load autexec and run the display test):
- 5D, 40D, 400D, 450D, 1000D (oldies)
- 50D (minor fix)
- 5D4

This one just boots DryOS and starts a few tasks:
- A1100 (PowerShot)

Current test results:
Code: [Select]
Testing bootloaders...
   5D: ROM READY
  5D2: K218 READY
  5D3: K285 READY
  5D4: K349 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
 7D2S: K289S READY
  40D: K190 READY : Ver 4.0.1
  50D: K261 READY
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 400D: InitializeIntercom
 450D: K176 READY : Ver 4.0.3
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1000D: K254 READY : Ver 3.7.5
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY

Setting up temporary SD/CF card images...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’

Testing display from bootloader...
   5D: OK
  5D2: OK
  5D3: OK
  5D4: OK
   6D: OK
   7D: please check
 7D2M: OK
 7D2S: OK
  40D: OK
  50D: OK
  60D: OK
  70D: OK
  80D: OK
 400D: OK
 450D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1000D: OK
1100D: OK
1200D: OK
 EOSM: OK

Testing EOS M3...
SD boot: StartDiskboot
Display: TurnOnDisplay

Restoring your SD/CF card images...

Testing Canon GUI...
  60D: OK
  5D3: OK
 600D: OK
1200D: OK
1100D: OK

Hope all of this will be useful to other developers  8)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 21, 2016, 09:24:29 PM
Hello, I'm new to ML, I'm trying it on an EOSM. I wish to develop a module, but it would be really difficult using the "physical" camera. So I'm trying to setup an emulation environment... unluckily I miss something... I'm stuck on qemu... I need some help.

Code: [Select]
/opt/qemu/bin/qemu-system-arm -machine EOSM
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 3FFFFFFF: eos.ram
40001000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-EOSM.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading 'ROM-EOSM.BIN' to 0xF8000000-0xF8FFFFFF (offset 0x1000000)
FIXME: no MPU spells for EOSM.

Available keys:
- Arrow keys   : Navigation
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- M            : MENU
- P            : PLAY
- I            : INFO/DISP
- L            : LiveView
- F1           : show this help

VNC server running on '127.0.0.1;5900'

I know that without MPU spells I cannot display Canon menu. But the only think I can display in VNC is the garbled screen. How to load ML?
I compiled EOSM platform with CONFIG_QEMU=y and I made qemu-helper.bin.

(https://s4.postimg.org/qvvqp774d/desktop10.png)

This is my directory setup (all uppercase as required?):

Code: [Select]
├── ROM-EOSM.BIN
└── sdcard
     ├── AUTOEXEC.BIN
     ├── ML
     │   ├── CROPMKS
     │   │   ├── CINESCO2.BMP
     │   │   ├── CRSSMTR2.BMP
     │   │   ├── PASSPORT.BMP
     │   │   ├── PHIPHOTO.BMP
     │   │   └── PHIVIDEO.BMP
     │   ├── DATA
     │   │   ├── APSC8P.LUT
     │   │   ├── APSC8R.LUT
     │   │   ├── FF8P.LUT
     │   │   └── FF8R.LUT
     │   ├── DOC
     │   ├── DOCS
     │   ├── FONTS
     │   │   ├── ARGHLF22.RBF
     │   │   ├── ARGNOR23.RBF
     │   │   ├── ARGNOR28.RBF
     │   │   ├── ARGNOR32.RBF
     │   │   ├── TERM12.RBF
     │   │   └── TERM20.RBF
     │   ├── MODULES
     │   │   ├── ADV_INT.MO
     │   │   ├── ARKANOID.MO
     │   │   ├── AUTOEXPO.MO
     │   │   ├── BENCH.MO
     │   │   ├── DEFLICK.MO
     │   │   ├── DUAL_ISO.MO
     │   │   ├── EOSM_202.SYM
     │   │   ├── ETTR.MO
     │   │   ├── FILE_MAN.MO
     │   │   ├── LUA.MO
     │   │   ├── MLV_PLAY.MO
     │   │   ├── MLV_REC.MO
     │   │   ├── MLV_SND.MO
     │   │   ├── PANO.MO
     │   │   ├── PIC_VIEW.MO
     │   │   ├── RAW_REC.MO
     │   │   ├── SELFTEST.MO
     │   │   └── SILENT.MO
     │   ├── README
     │   └── SCRIPTS
     ├── MAGICLANTERN
     ├── ML-SETUP.FIR
     └── QEMU-HELPER.BIN

Qemu 2.5.50 from niklastisk repo, branch eos-develop-new (./configure --disable-docs --target-list=arm-softmmu).

...anyway, I'm asking myself: is there a way to display ML menu at startup in the emulator? Otherwise it would be a bit difficult to recall it on EOSM... on the real camera I use double touch on the screen...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 21, 2016, 09:51:07 PM
PS: I had to comment out a code fragment from eos_init_common(). Otherwise, after loading ROM-EOSM.BIN, qemu exits with a "CF init failed".

Code: [Select]
    /* init CF card */
    DriveInfo *dj;
    dj = drive_get_next(IF_IDE);
    if (!dj) {
        printf("CF init failed\n");
        exit(1);
    }

    ide_bus_new(&s->cf.bus, sizeof(s->cf.bus), NULL, 0, 2);
    ide_init2(&s->cf.bus, s->interrupt);
    ide_create_drive(&s->cf.bus, 0, dj);

EOSM doesn't have CF. It fails also if I create a cfcard directory. And now I have the dubt that sdcard and cfcard directory are totally ignored...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 01:37:07 PM
For now, I recommend working on a model that can display Canon menu (at the time of writing, 60D, 5D3, 1100D or 1200D).

There are two qemu branches, with different design goals:

- the one in unified, which only attempts to run some ML code:
   - it displays the ML menu for most models
   - requires several changes to ML code (lots of hacks - grep for CONFIG_QEMU)
   - runs only a tiny part of Canon code (some startup code, task scheduler and... that's pretty much it)
   - there is a filesystem emulation, but it's extremely hackish: ML file I/O calls are replaced with wrapped calls to the host system (in the sdcard/cfcard directories)
   - it also runs some LiveView and HDMI tests (also very hackish)
   - back then, emulating Canon GUI was a pipe dream
   - I don't remember touching it in the last 1-2 years, so it should probably go away.

- the one in the qemu branch, which attempts to run the complete Canon firmware and unmodified autoexec.bin:
   - ML menu only works on those models that can boot Canon GUI
   - to navigate it easily, define GUIMODE_ML_MENU as 0 in consts.h (that's because GUI mode switching doesn't work yet)
   - patching ML code is optional (it can load unmodified autoexec.bin files, but patching can still be useful)
   - storage emulation uses a SD or CF card image (a single big file)
   - SD/CF emulation works by emulating the low-level calls (Identify Drive, read/write sector, stuff like that)
   - bootloader emulation works on nearly all models (from DIGIC 2 to DIGIC 6), including loading autoexec.bin from the card image
   - file I/O calls from both Canon and ML code are expected to work well on SD (e.g. loading ML modules, saving settings, creating the DCIM and MISC directories from Canon code)
   - CF emulation is not very solid (there are nondeterministic errors)
   - changing most Canon settings (properties) doesn't work (it requires talking to MPU)
   - LiveView and HDMI tests are very hard to run (it requires hacking some very hackish code)
   - you can debug it and trace various calls in Canon firmware using gdb (very useful for reverse engineering)

Basically, that's the current state of the emulation. Not exactly straightforward for new developers, but slowly getting there.

It also serves as "executable documentation" for a very large part of our knowledge about Canon code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 22, 2016, 02:17:48 PM
Thank for the clarifications, A1ex!

Ok, I could works indifferently on an emulated 60D, 5D3 or so. But the only ROM I have is the one dumped from my camera, so firstly I need to obtain another ROM (how?).

About ML branches, you recommend to use qemu branch for new developers? If it would be more reliable I'll use it. Actually, I wanted to use unified because of SD emulation wrapping on host fs... it would be simpler and more agile to work in ML directory without mounting and unmounting an image file.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 03:28:18 PM
I think the newer branch is a lot more robust and the emulation is much closer to the real thing.

For mounting/unmounting, it should be similar to a real SD card: once you have set up your SD card image with kpartx (see mount.sh), you click on EOS_DIGITAL in your file browser, then you run "make install" from your camera or module directory, then you start QEMU. You can automate this as well, if you like.

FYI, the install script sets up an initial SD card image, cloned from a 256MB card, which is already bootable and contains the portable display test autoexec.bin (handy to test your initial installation).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 22, 2016, 10:07:04 PM
Yeah! :D After several peregrinations between repositories and patches, and thanks to install.sh and run_canon_fw.sh scripts and the suggestions of a1ex, now I have a semi-working qemu environment. After seconds, I got the Canon menu...

(https://s17.postimg.org/4e4xu2dxr/desktop11.png)

But this is the maximum I can do. When I press a key, I can see only these warnings in qemu console...

Code: [Select]
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112

Did I miss something about keycodes?

PS: I tried to run 60D and EOS ROMs... I seen the Magic Lantern Rescue screen for both. But the sd card image I prepared with ML installation is 60D.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 10:19:37 PM
Looks fine (I get the same screen here).

The scancode error is internal to qemu; maybe it doesn't like some keyboard layouts?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 23, 2016, 12:47:28 AM
BTW, some small updates:

- initial support for 5D4 AECU aka K349AE (a secondary core, also ARM with DryOS)
- a test for GDB scripts (in the test suite)
- interactive UART support (you can navigate those bootloader menus)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/fromutil-550D.png)

Here's an easy coding task for you: some models don't have a debugmsg.gdb script. Creating one requires just finding some stubs. Right now, those models are: 5D2, 6D, 7D, 50D, 500D, 650D, 100D, 1100D and EOS M.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on October 25, 2016, 10:26:31 PM
I don't see menu on Lubuntu 16.04:
(http://thumbnails116.imagebam.com/51158/249e29511571455.jpg) (http://www.imagebam.com/image/249e29511571455)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on November 01, 2016, 08:38:20 PM
Some more updates:

- emulate the picture taking process (60D and 1200D):
   - can "capture" a full-res silent picture; that includes:
     - EDMAC transfers and interrupts
     - image processing modules (http://magiclantern.wikia.com/wiki/Register_Map#Image_PreProcessing) (dummy emulation of ADKIZ and HIV for now)
     - HEAD timers (dummy emulation for now)
   - you will need a reference image at qemu/<camera>/VRAM/PH-QR/RAW-000.DNG (a full-res silent picture)
   - it cannot emulate the unmodified silent.mo because of some GUI calls
   - you can either patch silent.c & raw.c, or use a minimal implementation
- 550D shows the date/time screen (but refuses to show the main screen)
- heartbeat timer now works on VxWorks models (can switch tasks)
- 450D emulates file I/O from main firmware
- a few more self-tests:
   - portable ROM dumper test (SD I/O from bootloader; no CF yet)
   - file I/O test from main firmware (check whether the DCIM directory is created at startup; SD models only)
   - test for picture taking emulation (test code under minimal/qemu-frsp)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-60D-frsp.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-1200D-frsp.png)

The first feature also reveals the register configurations required for taking a picture by directly driving the sensor with our own code :)

Full-res photo capture log: qemu-60D-FA_CaptureTestImage.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-60D-FA_CaptureTestImage.log)

Current self-test log:
Code: [Select]
Testing bootloaders...
   5D: ROM READY
  5D2: K218 READY
  5D3: K285 READY
  5D4: K349 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
  40D: K190 READY : Ver 4.0.1
  50D: K261 READY
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 400D: InitializeIntercom
 450D: K176 READY : Ver 4.0.3
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1000D: K254 READY : Ver 3.7.5
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY
5D4AE: K349AE AECU Firm Ver. 5.8.1(5.8.1)
 7D2S: K289S READY

Testing Canon GUI...
  60D: OK
  5D3: OK
 550D: OK
 600D: OK
1200D: OK
1100D: OK

Testing GDB scripts...
   5D: [  tExcTask:ffb223cc ] task_create(CmdShell, prio=a, stack=0, entry=ffb22420, arg=0)
  5D2: 5D2/debugmsg.gdb not present
  5D3: [      init:ff0c32d4 ] task_create(Startup, prio=11, stack=400, entry=ff0c2928, arg=0)
  5D4: [      init:fe0e14c9 ] task_create(SFRead, prio=11, stack=400, entry=fe0e0ef7, arg=620008)
   6D: 6D/debugmsg.gdb not present
   7D: 7D/debugmsg.gdb not present
 7D2M: [      init:00002207 ] task_create(OmarInit, prio=f, stack=400, entry=2183, arg=4ae1ec)
  40D: [ tTaskMain:ff812a60 ] task_create(HotPlug, prio=1e, stack=0, entry=ff812cb8, arg=0)
  50D: 50D/debugmsg.gdb not present
  60D: [   Startup:ff1dcc18 ] task_create(PropMgr, prio=14, stack=0, entry=ff1dcb24, arg=807b1c)
  70D: [      init:ff0c3360 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28ac, arg=0)
  80D: [      init:fe0d449f ] task_create(TaskMain, prio=1d, stack=0, entry=fe0d3619, arg=0)
 400D: [  tStartup:ffb1d1bc ] task_create(ImagePlayDriverTask, prio=19, stack=0, entry=ffa0ccd0, arg=0)
 450D: [ tTaskMain:ffd0a664 ] task_create(PropMgr, prio=15, stack=0, entry=ffd0a9e4, arg=384e5c)
 500D: 500D/debugmsg.gdb not present
 550D: [   Startup:ff1d8b30 ] task_create(PropMgr, prio=14, stack=0, entry=ff1d8a3c, arg=726d20)
 600D: [   Startup:ff1fbba8 ] task_create(PropMgr, prio=14, stack=0, entry=ff1fbab4, arg=757140)
 650D: 650D/debugmsg.gdb not present
 700D: [   Startup:ff0c38fc ] task_create(Startup2, prio=11, stack=400, entry=ff0c35b0, arg=0)
 750D: [      init:fe0ce241 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0cd4a9, arg=0)
 760D: [      init:fe0ce445 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0cd6ad, arg=0)
 100D: FAILED!
1000D: [          :ff812eec ] task_create(HotPlug, prio=1e, stack=0, entry=ff813050, arg=0)
1100D: 1100D/debugmsg.gdb not present
1200D: [      K327:ff2b9bd8 ] task_create(PropMgr, prio=14, stack=0, entry=ff2b9ae4, arg=71a11c)
 EOSM: EOSM/debugmsg.gdb not present
5D4AE: [      init:fe0a2aa1 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0a2159, arg=0)
 7D2S: 7D2S/debugmsg.gdb not present
EOSM3: [   Startup:010e17a9 ] task_create(SD1stInit, prio=18, stack=0, entry=10e1739, arg=0)
A1100: [   Startup:ffc3f6b0 ] task_create(SD1stInit, prio=18, stack=0, entry=ffc3f604, arg=0)

Setting up temporary SD/CF card images...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’

Testing FA_CaptureTestImage...
  60D: OK
1200D: OK

Testing file I/O (DCIM directory)...
  60D: OK
  5D3: OK
 550D: OK
 600D: OK
1200D: OK
1100D: OK
 100D: OK
 450D: OK

Testing display from bootloader...
   5D: OK
  5D2: OK
  5D3: OK
  5D4: OK
   6D: OK
   7D: please check
 7D2M: OK
  40D: OK
  50D: OK
  60D: OK
  70D: OK
  80D: OK
 400D: OK
 450D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1000D: OK
1100D: OK
1200D: OK
 EOSM: OK

Testing EOS M3...
SD boot: StartDiskboot
Display: TurnOnDisplay

Preparing portable ROM dumper...
Testing portable ROM dumper...
   5D: skipping
  5D2: skipping
  5D3: ROM0.BIN: OK ROM1.BIN: OK
  5D4: ROM1.BIN: OK
   6D: ROM0.BIN: OK ROM1.BIN: OK
   7D: ROMs not saved
 7D2M: ROM1.BIN: OK
  40D: skipping
  50D: skipping
  60D: ROM0.BIN: OK ROM1.BIN: OK
  70D: ROM0.BIN: OK ROM1.BIN: OK
  80D: ROM1.BIN: OK
 400D: skipping
 450D: skipping
 500D: skipping
 550D: ROM0.BIN: OK ROM1.BIN: OK
 600D: ROM0.BIN: OK ROM1.BIN: OK
 650D: ROM0.BIN: OK ROM1.BIN: OK
 700D: ROM0.BIN: OK ROM1.BIN: OK
 750D: ROM1.BIN: OK
 760D: ROM1.BIN: OK
 100D: ROM0.BIN: OK ROM1.BIN: OK
1000D: skipping
1100D: ROM0.BIN: OK ROM1.BIN: OK
1200D: ROM0.BIN: OK ROM1.BIN: OK
 EOSM: ROM0.BIN: OK ROM1.BIN: OK

Restoring your SD/CF card images...

It's getting close to being usable for testing the nightly builds :)



You may wonder why I'm looking at those old VxWorks models - they do a lot of things in a slightly different way, yet still compatible with the new models. This gives extra information for understanding hardware internals (e.g. different debug strings, or just exercising different code paths in QEMU).

For example, the photo taking code on 60D uses an old-style interface for the interrupt controller, but most other code uses the new one. There is a single interrupt controller, with two interfaces (see eos_handle_intengine and eos_handle_intengine_vx in eos.c). The VxWorks models use both interfaces (very old models probably use only the old interface).

Before looking at VxWorks models, I had no idea what the old-style registers might even be doing.

Will this extra knowledge translate into new features or new models ported? I hope so, but - as usual - can't promise anything. Feel free to jump in and get your hands dirty with the code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 08, 2016, 11:59:15 PM
i still want 1k FPS :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on November 09, 2016, 07:13:35 PM
i still want 1k FPS :)
it's better to want a replacement for cache hacks on Digic 6  :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 09, 2016, 10:24:54 PM
what do you mean?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on November 10, 2016, 11:56:13 AM
I mean that cache debug operations are not found in Cortex-R4 Technical Reference Manual (http://infocenter.arm.com/help/topic/com.arm.doc.ddi0363g/CIHBJBAI.html)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 12, 2016, 03:57:50 PM
oh. didnt know. where was the discussion?
this one? http://www.magiclantern.fm/forum/index.php?topic=13746.msg170772#msg170772
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on January 18, 2017, 01:18:50 AM
More updates:

- 100D boots Canon GUI
- 70D shows date/time screen (but refuses to boot Canon GUI if date/time is patched)
- EOSM and 450D run most of the initialization and pass the DCIM dir test
- very basic (incomplete) support for Eeko and JPCORE
- photo capture emulation works on 5D3 as well (test doesn't fully pass yet though)
- memory protection registers are printed at startup
- full list here (https://bitbucket.org/hudson/magic-lantern/commits/branch/qemu).

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/100D-qemu.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/70D-qemu.png)

The test suite now runs on ML build server as well, here (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 29, 2017, 09:58:38 PM
500D with spells from 550D has a new menu item :

(https://s23.postimg.org/l9ynkv557/qemux1.png) (https://s27.postimg.org/y44mzqjpv/qemux2.png) (https://s28.postimg.org/4i58gvjn1/qemux3.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on January 29, 2017, 11:03:34 PM
Very cool. I had this setting enabled on 550D, so it must be saved on the MPU side (probably EEPROM).

Old notes about it: https://groups.google.com/d/topic/ml-devel/ti8GyVqEZmo/discussion
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 30, 2017, 02:21:39 AM
Yes it is stored in the MPU.
Code: [Select]
Studio mode on  { 0x06, 0x05, 0x01, 0x42, 0x01, 0x00 },
Studio mode off { 0x06, 0x05, 0x01, 0x42, 0x00, 0x00 },
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 31, 2017, 01:01:49 AM
500D 30MB/s  :P
(https://s30.postimg.org/72oj0zpw1/qemubench.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 12, 2017, 05:59:45 PM
GUI switch progress :

(https://s12.postimg.org/72qetgkz1/qemu.png) (https://s17.postimg.org/5jxrwq0cv/qemu1.png) (https://s31.postimg.org/4qttjvtej/qemu2.png) (https://s31.postimg.org/vpxof1fvf/qemu3.png) (https://s10.postimg.org/4k248c1op/qemu4.png)

Live View :

(https://s12.postimg.org/4gfhq9j59/qemulv.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 14, 2017, 09:03:32 PM
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on February 16, 2017, 10:18:39 PM
i am still stunned how good the emulation works.. :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 17, 2017, 04:48:58 PM
Live View VRAM patch :

(https://s24.postimg.org/fry3mwyyt/qemu_lv.png)

Code: [Select]
[EDMAC#18] Starting transfer to 0x1B07800 from conn #4, 1440x424, flags=0x20000080
Loading photo raw data from ./500D/VRAM/PH-LV/LV-000.422...
[EDMAC#18] 610560 bytes written to 1B07800-1B9C900.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 18, 2017, 11:28:54 PM
500D menu navigation works here too, thanks Greg :D

Currently, this old camera is the only one that lets you navigate Canon menu. All other models show either a static GUI or the date/time screen.

Also committed:
- initial support for EOS M10 and M5 (for CHDK)
- an option to export function calls to IDA
- an experiment to group related MPU messages from timestamps (in the dm-spy-experiments branch)
- some auto comments regarding MPU messages
- minor fixes here and there.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 21, 2017, 01:46:02 AM
Formatting the virtual card works too, both from Canon and ML (of course, on 500D) :)

This is the first test in the suite that runs an unmodified ML binary. It actually downloads the current nightly build (at the time of writing) and checks both the GUI (expected screens) and the card contents (whether ML still boots after being restored).

Test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/console)

This log should also contain useful info (what commands to run), should you want to reproduce these experiments on your PC. I should probably write a guide, other than the tips from the install script.

Menu screens currently covered by the test suite:

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/menu.png)

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/format.png)

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/fmtrestore.png)

There are still some nondeterministic bugs (that's why some tests are retried a few times, until sucess); those will need fixing before using QEMU as a test platform for ML builds. Still, it already starts to be useful (for example, for getting menu screenshots).

At this stage, I think the old implementation (http://www.magiclantern.fm/forum/index.php?topic=2864.msg173682#msg173682) is no longer useful, so we may start thinking about merging the QEMU branch into unified. This will remove most of those CONFIG_QEMU hacks from the source code.

BTW, if you have experience with some testing framework, and you know a nicer way to implement the current tests (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/tests/run_tests.sh), I'd be interested in hearing from you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 26, 2017, 06:17:25 PM
Firmware update
ROM modified with hexeditor "DisableMainFirm" - http://magiclantern.wikia.com/wiki/Bootflags
500D 1.1.1 -> 1.1.2

(https://s27.postimg.org/9l5keyg83/qemu1.png) (https://s32.postimg.org/d6bg9eb91/qemu2.png) (https://s10.postimg.org/ao5qy413d/qemu3.png) (https://s27.postimg.org/537g4rl43/qemu4.png)

Code: [Select]
200:  5337.856 [UPD] Welcome to Update program
201:  5337.856 [UPD]   Program Ver.Slave 0.2.0
208:  5338.112 [UPD] ------------ Initialized
277:  5343.232 [UPD] CurrentVersion=1.1.1
278:  5343.232 [UPD] DS_MODELID =0x80000252
295:  5350.144 [MS] LOCK (1)
535:  8316.672 [UPD] StartFirmupProgress
569:  8319.232 [UPD] ERROR Do not read
571:  8358.912 [UPD] 0=UPD_VerifyFirmware
572:  8830.464 [UPD] 0=UPD_DecryptoFirmware
574:  8850.944 [UPD] CheckSum file=0xd960afec buffer=0xd960afec
575:  8947.968 [UPD] SAFEMODE
620: 35370.240 [UPD] ERR 1=updSpecificPartner

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 07, 2017, 04:44:59 PM
First ML change I've tested in QEMU on all unified models:

https://bitbucket.org/hudson/magic-lantern/pull-requests/796/new-method-for-getting-current-task-names/diff

Latest update adds partial 7D support (slave CPU only, without IPC).

Test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/48/console).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 24, 2017, 01:08:42 AM
A small change that unlocked Canon menu navigation on many models:

https://bitbucket.org/hudson/magic-lantern/commits/c881ba2

After some refactoring and porting the 500D MPU messages required for GUI, Canon menu navigation is now also working on...

60D, 550D, 600D, 700D, 100D, 1100D and 1200D!

Screenshots (guess the cam):

(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/100D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/1100D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/600D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/60D-format.png)

Test log (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/54/console).

All screenshots here (https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/) (click on Expand all).

This is a big breakthrough, as it effectively lets me review ML ports or code changes on cameras I don't own :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: DeafEyeJedi on March 24, 2017, 05:34:39 AM
Holy cow, @a1ex!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 25, 2017, 05:40:31 PM
please don't let jenkins test builds format the card. i am scared it will wipe our build server  :o

kidding :D
cool work!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on March 27, 2017, 10:12:32 PM
Nice work! Finally some running cameras to toy around with.  :D

I've managed to merge the latest changes into qemu v2.9.0-rc1 (https://bitbucket.org/niklastisk/qemu/branch/eos-2.9.0-rc1-merge). It's probably not a perfect merge, the camera gui hangs up more often than on v2.5.0 for me.

The patches to qemu code merged quite well, but someone with more knowledge of them should probably review whether they are ok/redundant in 2.9.0.

A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 27, 2017, 10:22:20 PM
A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.

Yay! That's what I wanted to do next, hoping it would solve the GUI lock-ups. I/O lock was another issue that I didn't know how to solve.

Quote
I've managed to merge the latest changes into qemu v2.9.0-rc1 (https://bitbucket.org/niklastisk/qemu/branch/eos-2.9.0-rc1-merge). It's probably not a perfect merge, the camera gui hangs up more often than on v2.5.0 for me.

My attempt to merge with 2.8.0 was not very successful (got stuck at making the serial port work, so ended up disabling it), but otherwise it seemed to run fairly well. Will try to integrate your changes and see how it goes.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 17, 2017, 07:05:01 PM
A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.

This solved the intermittent I/O lock-ups that I was unable to track down for a long time!!!

5D3 SD card test successfully ran 10 times in a row. Previously, I had to run it about 5 times to get one successful run...

All the menu navigation tests from test suite passed with flying colors, without any retries required!

Thanks nkls!!!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: DeafEyeJedi on April 17, 2017, 07:17:26 PM
Hell yeah and way to go @nkls!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 24, 2017, 10:18:01 PM
Added a couple of jobs on the build server:

- QEMU-dm-spy (http://builds.magiclantern.fm/jenkins/job/QEMU-dm-spy/): compiles the dm-spy-experiments branch and runs the binary in the emulator. These logs contain all Canon's debug messages (http://www.magiclantern.fm/forum/index.php?topic=2388.0), and optionally all MMIO activity. Should be useful for anyone who wants to understand the startup process.
- QEMU-boot-check (http://builds.magiclantern.fm/jenkins/job/QEMU-boot-check/): compiles ML from every camera model (from nightly) with CONFIG_QEMU=y and runs it for a few seconds in the emulator; this compilation flag enables additional debug info at startup, useful for checking the boot process (where autoexec.bin is loaded, how much memory it takes, what it does to reserve it and so on).
- QEMU-FA_CaptureTestImage (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-FA_CaptureTestImage/): compiles a minimal autoexec.bin that calls FA_CaptureTestImage (therefore taking a full-res silent picture). All the debug messages from Canon and all the MMIO activity are logged. Might be useful for understanding the still photo capture process (http://www.magiclantern.fm/forum/index.php?topic=1915.0).
- QEMU-tests (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/): that's the test suite for QEMU (presented earlier in this thread)

All these tests have HTML logs (actually just plain text with colors) and screenshots (where it's the case).

I'm also thinking to run some basic tests on the nightly, on those models with functional GUI (tests such as menu screenshots, load each module, check memory usage, run some simple Lua scripts). The emulation is not there yet for more complex tests (for example, we cannot take a CR2 picture or go to LiveView).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 26, 2017, 09:25:05 PM
Some updates:
- upgraded to QEMU 2.9.0, thanks nkls (still experimental, as I had quite a bit of trouble with it, so it's in a different branch for now)
- fixed another (or maybe the same?) nondeterministic lock-up (see a few posts above)
- initial support for 1300D (http://www.magiclantern.fm/forum/index.php?topic=17969) (WIP)
- options to log memory accesses (aka memory tracing); run with "-d help" to get the list

The lock-up bug was showing up very rarely on 2.5.0 after the timer refactoring from nkls (let's say about 1 out of 100 runs was bad), but after upgrading to 2.9.0 it showed up in more than half of the test runs (or about 1/5 of the test runs if the log was redirected to file). Narrowed down to interrupt controller (from a change made many months ago to support 1000D and other VxWorks models).

I'm also experimenting with logging all memory accesses made by the guest firmware, on 2.5.0. Examples for 1300D:

Code: [Select]
./run_canon_fw.sh 1300D -d romw
...
Firm Jump RAM to ROM 0xFE0C0000
K404 READY
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0x6       : 8-bit
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0x6       : 8-bit
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0xE9      : 8-bit
[DMA1] Copy [0xF8E60000] -> [0x402D4000], length [0x0026BBF8], flags [0x00030001]
[DMA1] OK
     0:    20.480 [STARTUP]

Code: [Select]
./run_canon_fw.sh 1300D -d ramw,romr
...
[rom1]     at 0xFE0C000C:001000EC [0xFEA7A270] -> 0xE92D4010
[ram]      at 0xFE0C000C:001000EC [0x00001900] <- 0xE92D4010
[rom1]     at 0xFE0C009C:001000EC [0xFEA7A274] -> 0xEB000BAB
[ram]      at 0xFE0C009C:001000EC [0x00001904] <- 0xEB000BAB
...

I know I'm (https://github.com/aclements/mtrace) almost (https://github.com/moyix/panda) certainly (https://github.com/panda-re/panda) reinventing (https://projects.gso.ac.upc.edu/projects/qemu-dbi/wiki) the (https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-qemu.html) wheel (http://web.eece.maine.edu/~vweaver/projects/qemu-trace/), but I had only limited success with these modified versions:
- mtrace uses a very very old QEMU
- panda 1.0 uses QEMU 1.0.1, examples work, lots of nice tools, but appears deprecated (shouldn't be hard to roll back our patches to the older version)
- panda 2.0 uses a very recent QEMU, but could not run any ARM examples (segmentation fault). Also, most of the cool tools from panda 1.0 are not ported yet.
- QEMU-DBI is "being upstreamed into QEMU", and a large part of it is already in 2.9.0 (the main reason I've upgraded). TODO: figure out how to use it...
- QEMU-CHERI is a mod for MIPS that also traces memory and instructions (nice to see how it works)
- the last one, QEMU-trace, is a very simple patch that showed me where to place the hooks in the QEMU codebase (also with this message (https://lists.nongnu.org/archive/html/qemu-devel/2014-04/msg02875.html) and this thread (https://lists.linaro.org/pipermail/linaro-dev/2012-March/010887.html) from mailing lists).

So, yeah, I still want to use the state-of-art method for logging memory accesses, just need to figure out how. Until then, my monkey-patched method appears to work pretty well (can rebuild the memory contents from the trace) and has very little overhead as long as I'm not printing each access to the console.