Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: jplxpto on September 23, 2012, 08:29:02 PM

Title: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 23, 2012, 08:29:02 PM
How to run Magic Lantern into QEMU?!...

I would like to know if anyone has ever launched Magic Lantern on QEMU. I've used it a few times, and it seemed to me that however limited it may be useful to conduct some tests.

If anyone has knowledge about this subject, thank you give me some tips that may also be useful for other developers.

Thank you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nanomad on September 23, 2012, 09:10:22 PM
Last time I checked there was a patch for qemu in ML source code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 23, 2012, 09:48:30 PM
Ok ... thank you...

One of these days I'll test it ...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: miyake on September 24, 2012, 10:50:23 AM
I'm now watching here.
http://chdk.wikia.com/wiki/GPL_Qemu

I wish this for something help for you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on September 24, 2012, 02:21:25 PM
I'm now watching here.
http://chdk.wikia.com/wiki/GPL_Qemu

I wish this for something help for you.

Thank you
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 08, 2013, 09:31:25 PM
With a bit of tweaking, that qemu patch seems to do something :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu2.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: scrax on March 17, 2013, 10:03:16 PM
I'm interested in running ML in QEMU for testing scripts, is the tweaked patch in the source?
What do I need to do compile a patched ML version and load it with QEMU?
I'm reading the page linked by miyake now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 17, 2013, 10:29:30 PM
It's not yet ready for this. What's missing is task switching support; I've tried some hacks based on setjmp (this (http://homepage.cs.uiowa.edu/~jones/opsys/threads/) and this (https://www.cs.cmu.edu/~mihaib/articole/threads/)), but they weren't reliable at all, so I'm thinking to try something like FreeRTOS. A simple cooperative scheduler (to switch tasks when they call msleep) would be enough.

G3gg0 had some success in running the DryOS task switcher. My approach is a bit different, I'm trying to re-implement the stubs from scratch so only ML code is emulated - that is, menu, scripting engine, overlays etc.

The qemu patch is very rough and not yet published.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on March 23, 2013, 03:15:40 AM
I'm very interested in this subject.

I'll be glad to help you.

Maybe this can help us ... The eCos is another option (http://ecos.sourceware.org/about.html) and it already has support for GDB.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mark.farnell on March 29, 2013, 09:51:25 PM
I'm very interested in this subject.

I'll be glad to help you.

Maybe this can help us ... The eCos is another option (http://ecos.sourceware.org/about.html) and it already has support for GDB.

In this case, is eCos a replacement of qemu?  However it says it is an operating system.... so do you mean to replace DryOS with eCos altogether? 
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 29, 2013, 09:55:12 PM
Yes, only for emulation. I don't need to emulate the entire Canon firmware, just ML code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mark.farnell on March 29, 2013, 10:50:46 PM
Yes, only for emulation. I don't need to emulate the entire Canon firmware, just ML code.

So at this stage, is it possible to emulate the ML code with eCos?  If so, how?

Also will the ability of emulating the ML code make developing safer?  ( I mean detecting silly mistakes such as writing wrong values to NVRAM variables, that can potentially brick the camera)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 29, 2013, 11:10:23 PM
See my previous post. There was no progress since then.

Wrong NVRAM values depend only on how they are interpreted by Canon code, so emulating only ML code won't help. If you get permanent ERR70 after changing some setting, that's a clear sign that you have set an invalid value; there's no other way to tell this. Maybe full emulation could help here, but that's very difficult (of course, I won't be surprised if g3gg0 succeeds). Detecting memory leaks may be possible (maybe run ML under valgrind?)

I want to emulate ML so I can check how the menus look on each camera, without having to buy every single model. Also, it may be helpful when writing user scripts, or when working with graphics code, fonts etc.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on March 30, 2013, 12:26:45 AM
I got 600D booting in trix to the service/bootloader menu but didn't really know what to do with it afterwards.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 30, 2013, 12:53:32 AM
at the moment i am working on qemu to emulate the whole fimware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Indy on March 30, 2013, 12:58:10 AM
Updater code (at least 7d one) can run partially on qemu.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 30, 2013, 01:07:34 AM
here are the changes i made to qemu:
http://upload.g3gg0.de/pub_files/7dc0800617416fdbeb1490dcc4a2164d/qemu_eos.7z

they contain a lot of hardware emulation.

booting fails imho due to the MPU that is not emulated. init routines expect a property that is not available.
i think the MPU would send it to the main firmware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: jplxpto on April 04, 2013, 01:33:02 AM
here are the changes i made to qemu:
http://upload.g3gg0.de/pub_files/7dc0800617416fdbeb1490dcc4a2164d/qemu_eos.7z

they contain a lot of hardware emulation.

booting fails imho due to the MPU that is not emulated. init routines expect a property that is not available.
i think the MPU would send it to the main firmware.

Thank you! I will try this ..
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 12:12:32 AM
Just merged my version with g3gg0's and pushed it on the main repo. Works on 60D, 600D, 500D, 5D2 and 650D.

In theory, you only have to run the install script (https://bitbucket.org/hudson/magic-lantern/src/tip/contrib/qemu/install.sh) - it will download QEMU 1.4.0, apply our modifications, and it will tell you what to do next.

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

If successful, you should get the hello world picture, and a log like this:

Code: [Select]
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-650D.BIN' to 0xF7000000-0xF8FFFFFF
[EOS] loading 'qemu-helper.bin' to 0x30000000-0x300088E7
[EOS] loading 'autoexec.bin' to 0x00800000-0x00855F4F
...
[GPIO] [0xC022C188] <- 0x138800 at pc=0x855E90
[FlashIF] at [0x000D4020]: 'Write enable' enabled
[Basic] at [0x000D4020] [0x00000000] <- [0xC0400008]
[Basic] at [0x000D4020] [0x00430005] -> [0xC0400008]
create_init_task(7e1ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr07.650D101 (781e0140ec5a+ (unified) tip)
[DebugMsg] (50,3) Built on 2013-04-07 20:02:47 by [email protected]
...
Hello at QEMU console!

Let me know if it works for you (and what other dependencies you had to install).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: scrax on April 08, 2013, 05:26:24 AM
Just merged my version with g3gg0's and pushed it on the main repo. Works on 60D, 600D, 500D, 5D2 and 650D.

In theory, you only have to run the install script (https://bitbucket.org/hudson/magic-lantern/src/tip/contrib/qemu/install.sh) - it will download QEMU 1.4.0, apply our modifications, and it will tell you what to do next.

(http://a1ex.magiclantern.fm/bleeding-edge/qemu.png)

If successful, you should get the hello world picture, and a log like this:

Code: [Select]
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-650D.BIN' to 0xF7000000-0xF8FFFFFF
[EOS] loading 'qemu-helper.bin' to 0x30000000-0x300088E7
[EOS] loading 'autoexec.bin' to 0x00800000-0x00855F4F
...
[GPIO] [0xC022C188] <- 0x138800 at pc=0x855E90
[FlashIF] at [0x000D4020]: 'Write enable' enabled
[Basic] at [0x000D4020] [0x00000000] <- [0xC0400008]
[Basic] at [0x000D4020] [0x00430005] -> [0xC0400008]
create_init_task(7e1ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr07.650D101 (781e0140ec5a+ (unified) tip)
[DebugMsg] (50,3) Built on 2013-04-07 20:02:47 by [email protected]
...
Hello at QEMU console!

Let me know if it works for you (and what other dependencies you had to install).

Will try on osx the script and report back, but first I need to clean up some space from my main hd.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 09:37:44 AM
I'm trying to run this on Arch linux:
I had to build with these options because python 2 binary is called python2 and the docs building didn't work:
Code: [Select]
./configure --target-list=arm-softmmu --python=/usr/bin/python2 --disable-docs
when I try to launch:
Code: [Select]
➜  qemu  ./run_ml_5D3.sh
make: Entering directory `/home/tom/dev/software/qemu/qemu-1.4.0'
make: Leaving directory `/home/tom/dev/software/qemu/qemu-1.4.0'
make: Entering directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
[ VERSION  ]   ../../platform/5D3.113/version.c
[ CC       ]   version.o
[ MENU IDX ]   menuindexentries.h
No menuindex.txt not running "python2 menuindex.py"
[ CC       ]   menuindex.o
[ LD       ]   magiclantern
[ OBJCOPY  ]   magiclantern.bin
[ SYMBOLS  ]   magiclantern.sym
[ CC       ]   reboot.o
[ LD       ]   autoexec
autoexec.bin: 443984 bytes

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x06c550 0x000d5950 0x000d5950 0x00008 0x00008 R   0x4
  LOAD           0x000100 0x00069500 0x00069500 0x6c458 0x7bb6d RWE 0x100
[ OBJCOPY  ]   autoexec.bin
make: Leaving directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
make: Entering directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
make: `qemu-helper.bin' is up to date.
make: Leaving directory `/home/tom/dev/software/magic-lantern/platform/5D3.113'
rm: cannot remove ‘vram.txt’: No such file or directory
rm: cannot remove ‘vram.png’: No such file or directory
00000000 - 3FFFFFFF: eos.ram
40000000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
eos_load_image: file not found 'ROM-5D3.BIN'
run_ml.sh: line 15: 25600 Aborted                 (core dumped) $QEMU_PATH/arm-softmmu/qemu-system-arm -M ML-$1
convert: unable to open image `vram.txt': No such file or directory @ error/blob.c/OpenBlob/2641.
convert: no images defined `vram.png' @ error/convert.c/ConvertImageCommand/3103.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 09:46:22 AM
5D3 is not working yet, here it fails at create_init_task. You also have to get some Canon firmware dumps, see the install instructions.

You may have better luck if you add support for 50D (you need to add a ML_MACHINE definition and create a launch script).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 10:13:39 AM
Sorry, I didn't read that well! I thought I had the 5D3-ROM in the correct location but I guess not. I now get as far as create_init_task so I assume that qemu is at least working. I'll have a go at adding support for 50D.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 11:49:06 AM
5D3 works too, I was loading the old 1.1.2 dump by mistake.

Code: [Select]
create_init_task(695ac)
*** init_task
[DebugMsg] (50,3) Magic Lantern v2.3.NEXT.2013Apr08.5D3113 (087dd0afd6b8+ (unified) tip)
...

It fails at redraw, you will need to comment it out from ML code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: trsaunders on April 08, 2013, 12:36:39 PM
(http://i.imgur.com/PXf5p5t.png)

50D worked without much effort - just copied 60D definitions, changed to 50D and added the appropriate run_ml script.

I tried to generate a patch for the changes but hg diff was producing a lot of spurious changes - maybe it doesn't like diffing a diff?!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 08, 2013, 12:55:14 PM
I've used a new hg tree under qemu to generate the diff. It's a bit ugly now; I'm thinking to modify the install script to store the new files as plain files, and use the patch only for modification to QEMU sources.

Edit: did these changes and some small additions:

- to emulate Canon firmware, without ML:
    ./run_canon_fw.sh 600D

- to generate a diff or commit changes (say you have modified eos.c or added some script):
    ./copy_back_to_contrib.sh
    then normal hg commands or gui in contrib/qemu

- to run the firmware in gdb:
    qemu-1.4.0/arm-softmmu/qemu-system-arm -M 5D3 -s -S in one terminal
    arm-elf-gdb -x gdbopts in another
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 09, 2013, 12:24:45 AM
6D is running in Qemu.. ML seemed to run but I did not see hello world in VNC.
Ran the canon FW and debugger and it opened up stopping at ff0c0008.
I guess this can be connected as debugger to ida or another disassembler?

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 09, 2013, 07:26:00 AM
Yes, you can connect from IDA or GDB to localhost:1234.

The display device is not implemented - it just saves a screenshot when you call dispcheck.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 06:03:49 AM
I compiled on windows with mingw. It loads but get:

VNC server running on `::1:5900'

and can't connect to vnc or gdb.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on April 10, 2013, 12:31:35 PM
vnc isnt implemented for eos cameras, just gdb.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 04:05:38 PM
Works in linux. You can connect over vnc and shut down qemu.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Marsu42 on April 10, 2013, 08:01:01 PM
I compiled on windows with mingw. It loads but get

It would be nice to also get it running on Windows though - of course I have Linux/whatever vms, but for ml I'd like to get around them on my puny laptop just like there is no need to use Linux to compile ml.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 10, 2013, 11:18:38 PM
I can post the binary... the armmmu folder is only something like 20-30mb.... but I can't connect to it in windows, like it needs another patch to use the networking. The linux binary will only connect on localhost so I can't push it out to gdb over the network.

Thats the prob I'm having with QEMU.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Marsu42 on April 11, 2013, 01:35:48 AM
The linux binary will only connect on localhost so I can't push it out to gdb over the network.

Use a ssh tunnel or network redirector?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on April 11, 2013, 02:52:02 AM
I turned UFW off and set some forwarding from lo to eth0... so far its working. Really would be nice to get it on one machine tho.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 05, 2013, 10:16:49 AM
Is there a way to run qemu for a 550D?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 12:32:41 AM
Yep, just edit the files and add 550D + values.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 06, 2013, 02:32:19 AM
@1% Thanks. 

I changed the content of "run_ml_550D.sh" to "sh run_ml.sh 550D 109" (550 instead of 500)
but it still won't work. The 550D is not listed under supported machines. I don't know if it's important and failure is caused by something else, or caused by my installation? Well I have no vram.txt as you see. Do you have any clue?
Code: [Select]
$ ./run_ml_550D.sh
make: Entering directory `/home/wolf/qemu/qemu-1.4.0'
make: Leaving directory `/home/wolf/qemu/qemu-1.4.0'
make: Entering directory `/home/wolf/magic-lantern/platform/550D.109'
[ VERSION  ]   ../../platform/550D.109/version.c
[ CC       ]   version.o
[ MENU IDX ]   menuindexentries.h
No menuindex.txt not running "python2 menuindex.py"
[ CC       ]   menuindex.o
[ LD       ]   magiclantern
[ OBJCOPY  ]   magiclantern.bin
[ STAT     ]   magiclantern.bin
magiclantern.bin: 449232 bytes
[ SYMBOLS  ]   magiclantern.sym
[ CC       ]   reboot.o
[ LD       ]   autoexec

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x06dbc8 0x00cedbc8 0x00cedbc8 0x00008 0x00008 R   0x4
  LOAD           0x000100 0x00c80100 0x00c80100 0x6dad0 0x7d6d9 RWE 0x100
[ OBJCOPY  ]   autoexec.bin
[ STAT     ]   autoexec.bin
autoexec.bin: 449728 bytes
make: Leaving directory `/home/wolf/magic-lantern/platform/550D.109'
make: Entering directory `/home/wolf/magic-lantern/platform/550D.109'
make: `qemu-helper.bin' is up to date.
make: Leaving directory `/home/wolf/magic-lantern/platform/550D.109'
rm: cannot remove 'vram.txt': No such file or directory
rm: cannot remove 'vram.png': No such file or directory
Supported machines are:
none                 empty machine
collie               Collie PDA (SA-1110)
ML-50D               Magic Lantern on Canon EOS 50D
ML-60D               Magic Lantern on Canon EOS 60D
ML-600D              Magic Lantern on Canon EOS 600D
ML-500D              Magic Lantern on Canon EOS 500D
ML-5D2               Magic Lantern on Canon EOS 5D2
ML-5D3               Magic Lantern on Canon EOS 5D3
ML-650D              Magic Lantern on Canon EOS 650D
50D                  Canon EOS 50D
60D                  Canon EOS 60D
600D                 Canon EOS 600D
500D                 Canon EOS 500D
5D2                  Canon EOS 5D2
5D3                  Canon EOS 5D3
650D                 Canon EOS 650D
nuri                 Samsung NURI board (Exynos4210)
smdkc210             Samsung SMDKC210 board (Exynos4210)
connex               Gumstix Connex (PXA255)
verdex               Gumstix Verdex (PXA270)
highbank             Calxeda Highbank (ECX-1000)
integratorcp         ARM Integrator/CP (ARM926EJ-S) (default)
kzm                  ARM KZM Emulation Baseboard (ARM1136)
mainstone            Mainstone II (PXA27x)
musicpal             Marvell 88w8618 / MusicPal (ARM926EJ-S)
n800                 Nokia N800 tablet aka. RX-34 (OMAP2420)
n810                 Nokia N810 tablet aka. RX-44 (OMAP2420)
sx1                  Siemens SX1 (OMAP310) V2
sx1-v1               Siemens SX1 (OMAP310) V1
cheetah              Palm Tungsten|E aka. Cheetah PDA (OMAP310)
realview-eb          ARM RealView Emulation Baseboard (ARM926EJ-S)
realview-eb-mpcore   ARM RealView Emulation Baseboard (ARM11MPCore)
realview-pb-a8       ARM RealView Platform Baseboard for Cortex-A8
realview-pbx-a9      ARM RealView Platform Baseboard Explore for Cortex-A9
akita                Akita PDA (PXA270)
spitz                Spitz PDA (PXA270)
borzoi               Borzoi PDA (PXA270)
terrier              Terrier PDA (PXA270)
lm3s811evb           Stellaris LM3S811EVB
lm3s6965evb          Stellaris LM3S6965EVB
tosa                 Tosa PDA (PXA255)
versatilepb          ARM Versatile/PB (ARM926EJ-S)
versatileab          ARM Versatile/AB (ARM926EJ-S)
vexpress-a9          ARM Versatile Express for Cortex-A9
vexpress-a15         ARM Versatile Express for Cortex-A15
xilinx-zynq-a9       Xilinx Zynq Platform Baseboard for Cortex-A9
z2                   Zipit Z2 (PXA27x)
convert: unable to open image `vram.txt': No such file or directory @ error/blob.c/OpenBlob/2641.
convert: no images defined `vram.png' @ error/convert.c/ConvertImageCommand/3106.
$

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 02:53:45 AM
You have to add add it to eos.c too.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 06, 2013, 09:06:40 PM
It's working. :-)
But is it possible to test a picoc script already?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: 1% on May 06, 2013, 10:34:23 PM
I wish.. I couldn't get anything useful for debugging either.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 31, 2014, 08:42:55 PM
QEMU working again :)

- runs DryOS task scheduler, semaphores, message queues (massive credits to g3gg0 for the low-level emulation code and insights)
- loads files from a local directory (sdcard or cfcard)
- able to load config files, modules, cropmarks...
- menu navigation working
- file manager working
- arkanoid working (playable!)
- properties not working
- most canon code is not working :P

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-arkanoid.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu-menu.png)

https://bitbucket.org/hudson/magic-lantern/branch/qemu

Feel free to turn it into something useful... like script interpreter, testing server, source-level debugger, HDMI emulator, support for image buffers, add a nice GUI... or just port it for your camera. I've only tested it on 5D3 1.1.3, and it's been already used to debug the early 100D ML port in GDB.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2014, 05:39:25 PM
Code: [Select]
[DebugMsg] (139,22) Wait Master Wakeup
[GPIO] at [0xFF080090] [0x00000000] <- [0xC0220024]
....
[DebugMsg] (139,6) Wait Master Wakeup Timeout
[DebugMsg] (139,22) Master Wakeup

Other than that, emulating 7D ML in QEMU works quite well. Any hints about how to emulate the master processor and the communication between them?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on October 02, 2014, 11:41:45 AM
Works fine here for 6D. Needed to do some additional stuff like filename capitalization etc. I just ran into one last problem. I have somehow no write access - at least there's no "magic.cfg" saved neither "bench.ppm" nor are ROM dumps from debug menu written etc. Though qemu doesn't report an error on qemu monitor. It's just like qemu has write access to some nirvana place in memory....

Code: [Select]
[???] [0x00000010] -> [0xC020006C] PC: 0xFF0C7C14
*** FIO_CreateFile('B:/ML/SETTINGS/magic.cfg') => 17
*** FIO_WriteFile(11, 231)
{{{
# Magic Lantern Nightly.2014Oct02.6D113 (b59a1ac5fbfc+ (qemu) tip)
# Built on 2014-10-02 08:56:42 UTC by [email protected]
}}}
*** FIO_CloseFile(11)
[FIO wrapper] closefile() nothing open
Save configs...

Edit:
qemu-helper.c define R/W access. Do I have to change the values for 6D. Btw I compiled and tried via qemu branch
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 02, 2014, 12:19:44 PM
Correct, file writing is not implemented. The entire FIO implementation is weak (only single-task and very poor error checking) - that is, just barely enough to load ML :P.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 11, 2015, 12:26:12 PM
Just had success in emulating the display test (http://www.magiclantern.fm/forum/index.php?topic=14732) (on most cameras). On 6D, I had to emulate the bootloader as well (without it, the display init routine would get stuck).

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-boot-6D.png)

To run the display test, look for the following "if (0)" and enable them:
- "bootloader config, 4 bpp" -> required to run all boot display tests
- "6D bootloader experiment" -> required for 6D; launch with ./run_canon_fw.sh 6D

edit: after some small changes, Linux works as well :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu-boot-linux.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mk11174 on April 24, 2015, 03:48:31 PM
Anyone know what this HDMI USB thing is, I don't see this on the 550D or 700D when I play with them, this loops so much and makes it super slow to work the menus?
(http://s3.postimg.org/asramqo6r/Capture.jpg)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 24, 2015, 10:48:27 PM
Major progress: I'm able to launch Canon GUI under QEMU :)

( side note: Nikon Hacker guys achieved this step a long time ago (http://www.magiclantern.fm/forum/index.php?topic=8823.0), so we are just playing catch-up :P )

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-canon-gui.png)

More details:
- it launches most Canon tasks
- unmodified 60D firmware (without autoexec.bin or ROM patches) runs as well (and starts the GUI too)
- SD card emulation also works (it loads autoexec.bin and even creates the DCIM directory on startup)
- MPU emulation kinda works (it replays messages from a log file)
- sample log: 60D-qemu-canon-gui-and-sd.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/60D-qemu-canon-gui-and-sd.log)

Next steps:
- emulate unmodified autoexec.bin
- remove all those CONFIG_QEMU hacks
- implement key events as MPU messages
- CF emulation
- enable the emulation for other cameras
- do something about those huge logs
- make the code more QEMU-ish and less hackish
- write a quick start guide
- do something useful with it :)

What's the use?
- much easier to understand Canon firmware (you can see exactly what some piece of code does with the hardware)
- very useful in diagnosing soft-bricked cameras
- a way to debug your code (or Canon's) in a GUI (gdb or IDA)
- test bench for Lua scripting or for module development
- automated tests for the nightly builds (see also this proposal (http://www.magiclantern.fm/forum/index.php?topic=12396.msg119592#msg119592))

Some tips, until a more complete guide will be available:
- to load a SD card image, use something like: ./run_canon_fw.sh 60D -sd sd.img
- to display a trace of the firmware code, with disassembly, use: ./run_canon_fw.sh 60D -sd sd.img -d exec,int -singlestep
- there is a monitor console as well: ./run_canon_fw.sh 60D -sd sd.img -monitor stdio
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 25, 2015, 03:02:40 AM
good work!
a milestone, indeed :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 25, 2015, 06:50:34 AM
Thanks.

Anyone know what this HDMI USB thing is [...] ?

That's Canon's HotPlug task reading those registers. I'll add an option to quiet them down ( -d ioport, as this one is standard in qemu).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on May 25, 2015, 10:23:20 AM
Well done and cool stuff. I'll have a closer look once solved some 70D issues. The detailed sd command states from your posted log re-activated my interest in researching sd card related stuff (http://magiclantern.fm/forum/index.php?topic=12862.0). I would also like to test if I can re-produce the ML menu timeout bug now in QEMU.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 05:05:39 PM
Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:
Code: [Select]
K287 ICU Firmware Version 1.1.1 ( 3.3.7 )
[DebugMsg] (139,5)
ICU Release DateTime 2012.05.14 13:18:25
[DebugMsg] (0,3) [SEQ] CreateSequencer (Startup, Num = 6)
[DebugMsg] (0,2) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[DebugMsg] (0,3) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[DebugMsg] (50,3) Magic Lantern Nightly.2015May30.60D111 (91e1ed550ec7 (qemu))
[DebugMsg] (50,3) Built on 2015-05-30 14:50:55 UTC by [email protected]
[*****] Starting task ff02b61c(3d2494) Startup
[DebugMsg] (0,5) [SEQ] seqEventDispatch (Startup, 0)
[GPIO]     at 0xFF012604     [0xC0222004] <- 0x12      : ???
[DebugMsg] (139,5) startupEntry
[*****] Starting task ff012584(0) Startup

Code: [Select]
[*****] Starting task 1fe07970(0) menu_task
[*****] Starting task 1fe09e24(0) menu_redraw_task
[*****] Starting task 1fe1ba88(0) bitrate_task
[*****] Starting task 1fe28180(0) focus_task
[*****] Starting task 1fe291a0(0) notifybox_task
[*****] Starting task 1fe2c124(0) fps_task
[*****] Starting task 1fe34244(0) shoot_task
[*****] Starting task 1fe2cd8c(0) clock_task
[*****] Starting task 1fe3b1bc(0) audio_common_task
[*****] Starting task 1fe43770(0) livev_hiprio_task
[*****] Starting task 1fe4245c(0) cls_task
[*****] Starting task 1fe452f0(0) beep_task
[*****] Starting task 1fe4d6d0(0) console_task
[*****] Starting task 1fe544dc(0) qemu_key_poll
[*****] Starting task 1fe0d7fc(0) debug_task
[*****] Starting task 1fe1d1e0(0) tweak_task

Code: [Select]
Load modules...
Linking..
Register modules...
Load configs...
Init modules...
Updating symbols...
  [i] 404: dual_iso_get_recovery_iso 1fe1fe00
  [i] 404: dual_iso_is_active 1fe1fe00
  [i] 404: auto_ettr_intervalometer_wait 1fe2cba0
  [i] 404: auto_ettr_intervalometer_warning 1fe2cba0
  [i] 404: auto_ettr_export_correction 1fe389d0
  [i] 404: mlv_snd_is_enabled 1fe3a8f0
  [i] 404: dual_iso_get_dr_improvement 1fe48da0
  [i] 404: dual_iso_get_recovery_iso 1fe48da0
Modules loaded

But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

Code: [Select]
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT
[GPIO]     at 0x1FE3B230     [0xC0220070] -> 0x1       : VIDEO CONNECT

I can send the whole log if it will help

I haven't tried booting Canon firmware yet, I need to get a hold of an SD card small enough I can make an image of it...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 09:04:11 PM
@dmilligan
I tried to load the lua module, but nothing did appear in the menu.
Anyway, the QEMU progress is impressive.

(http://i.imgur.com/vaTqI4H.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 10:39:18 PM
Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 10:51:26 PM
Did you have any scripts? There is no menu for lua itself, scripts can create or add their own menu items wherever they see fit. You can check the console for script loading/status/errors.

Yes.
/qemu/sdcard/ML/SCRIPTS/{CALC.LUA,EDITOR.LUA,SOKOBAN.LUA}
It seems to me that QEMU doesn't "look" for the script folder.
I did run your lua module on my camera and it did work very well.

$grep SCRIPTS log.txt
[FIO wrapper] readdir() => SCRIPTS size=4096 mode=10 time=556420ee
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 30, 2015, 11:01:19 PM
IDK, perhaps some FIO routines not emulated, or incomplete
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wolf on May 30, 2015, 11:27:32 PM
@ dmilligan
How did you compile QEMU version 2.3.0?
I've got 1.6.0.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on May 31, 2015, 01:08:24 AM
The latest in the qemu branch uses 2.3.0. Just run the install script.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on August 19, 2015, 04:44:38 PM
Hello,

I'm trying to emulate 100D with qemu 2.3.0 and I'm running into some issues. I've got it to run quite far but there seems to be multiple issues and I'm not sure which one to try to fix next...

I've attached the log file for the longest I've got it running, and the major changes from what's in the repo is:

- Added addresses for 100D
- No boot flag, just canon stuff
- Asserts aren't fatal, they just print to serial and return to caller (GDB hack)
- DebugMsg prints to log (with color output, GDB/QEMU magic)
- Sequencer events are delayed if being propagated too early, to avoid errors (again some GDB magic)
- Added some SPI EEPROM code
- Grepped out the USB/HDMI/MIC clobber


So some questions:

Is there an SPI EEPROM in 60D?
The 100D firmware tries to load some properties from there and isn't happy with getting zeros. Not sure whether real data is necessary for the emulation, but it seems to be the reason why some asserts are raised.

There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

It also complains about I2C and RTC at some points, but I don't think those are fatal errors.

Somehow it crashes to a $pc=4 state which I 'm trying to track down atm, but any thoughts on what to do to make it run further would be very appreciated. :)

This is the qemu log:
https://www.dropbox.com/s/y92sf7babtf3t2w/100D.log?dl=0
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on November 10, 2015, 01:52:00 PM
a1ex if you read this:
May we expect an update to the qemu branch anytime soon? Sort of having issues like dmilligan encountered since you last updated qemu to 2.3.

Thanks for updating to qemu 2.3.0, I can get it to run now on OSX 10.10 (older qemu versions won't compile), well sort of. Things seem to boot:
....
But I never get a screen image (the qemu window comes up, but it's just black), just this over and over again:

For me it just looks like this:
(http://i.imgur.com/P6uQbL8.jpg?1)

Qemu 1.6 worked great for 70D but with 2.3 I just get the black graphics shown as posted above whether I try 70D or 100D. For my use case it does the job for porting ROMs to see if they boot up but ofc I am hesitated to see the canon gui emulation which you presented on forums, too. The actual qemu branch also gives me a compile error in file "eos.c" at "eos_tio_handle" where I have to bypass "msg_arg1". So I just hope an updated branch might fix my problems or I simply am going to setup a new virtual machine with an older ubuntu release for qemu only. No rush...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on December 28, 2015, 11:44:51 AM
Major progress: I'm able to launch Canon GUI under QEMU :)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-canon-gui.png)

A1ex, how about an update to qemu? At it's actual state from bitbucket I always have to uncomment some lines to get qemu compiled. I just set-up a fresh 12.04 Ubuntu but had the exact same error on 14.04 Ubuntu. So quoting your post from May above I assume there's some local work on your build machine which needs to be merged:

Code: [Select]
/home/ml/qemu/qemu/qemu-2.3.0/hw/arm/eos.c:2716:9: error: variable ‘msg_arg1’ set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors
make[1]: *** [hw/arm/eos.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [subdir-arm-softmmu] Error 2

And once you read this: I would welcome the dmspy-experiments branch being merged into unified,too.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 26, 2016, 08:46:18 PM
Updated to run the Lua module on 60D.

(http://a1ex.magiclantern.fm/bleeding-edge/lua/qemu-lua-bug-scriptsource.png)

@nikfreak: I'm not getting this error about msg_arg1 (not even a warning), probably different cflags (I'm always (re-)installing it from contrib/qemu/install.sh script, but on an older Ubuntu). I've fixed that one anyway (it was, indeed, a bug).

@nkls: unfortunately, the MPU spells are valid only for 60D for now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on February 29, 2016, 12:30:53 PM
Really strange. As I didn't have the graphical glitch shown below in early days of 70D QEMU porting
(http://i.imgur.com/P6uQbL8.jpg?1)

The only thing I can think of atm is a wrong conflicting merge of my 70D branch into qemu's as I already tried a new ubuntu installation and experience the same. Any clue when 70D might become part of unified and merged into qemu by you? Porting began Dec 2015 and I feel it's just about time to get it into unified...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 29, 2016, 01:09:54 PM
Yeah, the stuff in the QEMU branch expects Canon GUI to do all the display initializations (including color palette).

But that only works on 60D atm...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 28, 2016, 12:11:56 AM
Small progress: CF card emulation appears to work on 5D2 bootloader, enough to load autoexec.bin from a CF image file.

The 5D2 GUI isn't working yet though...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 11, 2016, 09:47:16 PM
I've pushed my changes to qemu to my bitbucket now, it's located at:

https://bitbucket.org/niklastisk/qemu (branches: eos-develop and 100D-testing)

It was a while since I touched the code but I hope parts of it can be useful to someone at least.

Things I recall I've done:
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 11, 2016, 10:13:25 PM
Very nice work! You seem pretty familiar with qemu internals (where I'm struggling).

I'll try to integrate it with the latest branch.

Regarding your questions (sorry, I somehow missed your post, so you probably got the answers meanwhile):

Is there an SPI EEPROM in 60D?

There is an EEPROM that you can dump with this module (https://bitbucket.org/hudson/magic-lantern/commits/8c8e3329).

I didn't use this file in my emulation though.

Quote
There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

They are taken from a startup log, from the dm-spy-experiments branch (CONFIG_DEBUG_INTERCEPT_STARTUP=y in Makefile.user), that contains calls to mpu_send/mpu_recv branch. The code is generated with this script (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/scripts/extract_init_spells.py).

They are quite incomplete - just enough to boot the main dialog. I know the buttons are also sent from MPU in the same way, and it's easy to get the pattern for them. I also figured out how to decode some properties, but but most other events are still a mystery to me.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 12, 2016, 09:58:12 PM
@nikls:
i am impressed and happy to see such rev.eng. skills :)
guessing right that you are using IDA?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 16, 2016, 04:50:26 PM
Thanks guys, and thanks for the info! I was trying to to get 100D working and I guess I learned a bit about the qemu internals on the way. I wouldn't call myself an expert on the system but I know my way around bigger C/linux projects like this. :)

I never realized the dm-spy-experiments dumped the mpu calls, I guess I have to make one of 100D when I get the time!

Yeah, I'm using IDA for the decompiling and code analysis.


About that serial flash though, I'm under the impression that it's only available in newer cameras (100D and 70D? (https://bitbucket.org/hudson/magic-lantern/issues/2535/70d-black-screen-menu#comment-27815246)), and that it is necessary to emulate them to get to the main dialog in qemu. But correct me if I'm wrong.

In 100D there are references to two flash chips PC28F128M29EWHF and K5C262GATMAF50 which are both 16MB iirc. There is a function "IsAddressSerialFlash" which is used by the property manager (or some flash access abstraction) to determine if the data is stored in flash rom (same as firmware) or serial flash rom (controlled over serial interface). I think I had it setup with DMA, interrupts and whatnot but I never made it to the main dialog. :( I'll give it another shot sometime but atm I'm too busy with other projects.

Here are some extracts from the factory menu, note the S and V options in FACTADJ and the SIO6 menu for accessing the serial rom.
Code: [Select]
FACTADJ

********** FACTORY ADJUSTMENT MENU VER 0.01 **********
0. Exit from Factory Adjustment
1. Leak Check
2. SDRAM Check
3. ROM Check
4. HDMI Check
5. Video Adjustment
6. ICU Version Check
7. Audio Check
8. Adjustment Data Display and Change
9. Check Flag Display and Initialization
A. ALL Check
B. ALLP Check
F. FROM ID Check
P. Power Domain Check
S. Serial Flash Check
V. Serial Flash Version Check
Z. Input_Device_Unique

S
Serial Flash Check.
346
Select OK or NG

V
SROM Version Check.
4.2.1
Select OK or NG


FROMUTILITY

************ FROMUTILITY MENU Ver 0.09 ************
[Type:346 Body:DC Revision:0.00]
0.Exit from FROM Menu
1.Erase Sector(0x20000)
2.Erase Chip
4.Write from card
5.Write from DRAM
6.Firm   flag 0xF8000000 0x00000000 ON
7.Boot   flag 0xF8000004 0x00000000 OFF
8.UpDate flag 0xF800000C 0xFFFFFFFF OFF
9.Create Boot Disk
A.Exec Program from card
B.ALL Block Unprotect
C.Connect card
G.Memory Dump
I.Write Data
J.Direct Jump
P.Power Domain
S.SROM Menu
U.Firm update
Z.RCbind.bin update

S

**** SROM(SIO6) Menu ****
0.Exit from SROM Menu
1.Erase Chip   0x01000000
2.Erase Block  0x0000F000
3.Erase Sector 0x00001000
4.Write Data
5.Write from Card
6.SROM Dump(SIO Read)
7.SROM Dump(QUAD Read)
8.Get Info
9.Get Version

8
8.Get Info 0x80000346

9
9.Get Version 4.2.1
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 16, 2016, 06:52:51 PM
Yes, experience with Linux definitely helps. I don't have that one either.

I tried to run your serial flash dumper on 5D3, but the serial flash appears unused here. There are no mentions of it in the startup log, and calling SF_CreateFlash locks up. Didn't try to debug further. The 5D3 firmware has these strings: PC28F128M29EWHF and MBM29LV640EBXEI70G.

There are plenty of serial flash messages on the 70D startup log, and from the errors, I think they are needed for emulation. Fortunately, the communication protocol doesn't seem too complicated.

70D startup log says:
Code: [Select]
[SF] InstallSerialFlash 4 0xc022002c 0x0 0x800000 1

and these strings are present: PC28F128M29EWHF and 64Mt8Kx8m64Kx126b8Kx8.



Quick question: when emulating the factory menu, how are you entering the input selection?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 16, 2016, 09:51:58 PM
The chip install code is run quite early in the init task, this is how it looks in IDA:
(http://i.imgur.com/0ENFsrm.png?1)

As you can see there is a flag (derived from FROM) which determines what chip to install. On a second thought, those routines must be for setting up writing to main flash.

The function call before "SerialFlash Initialize" is definitely for SROM though, I guess there would be some similar call in 70D but not in 5D3? Maybe the 5D3 has more space in main flash memory and thus no need for it?
In 100D the output is
Code: [Select]
[SF] InstallSerialFlash 6 0xc022c0d4 0xf0000000 0x1000000 1
so it's similar.

I think I've got most of the protocol working, take a look in this file in 100D-testing (https://bitbucket.org/niklastisk/qemu/src/ada7b6496bde7ebc6d7cc6b1557e9710156cc129/hw/eos/serial_flash.c?at=100D-testing&fileviewer=file-view-default). It gets messy once you try to emulate DMA since it is shared with the SD memory transfer.


Quick answer: (too much of a hack to push to bitbucket)
Code: [Select]
unsigned int eos_handle_tio ( unsigned int parm, EOSState *s, unsigned int address, unsigned char type, unsigned int value )
{
    unsigned int ret = 1;
    const char * msg = 0;
    int msg_arg1 = 0;

    switch(address & 0xFF)
    {
        case 0x00:
            if(type & MODE_WRITE)
            {
                if((value == 0x08 || value == 0x0A || value == 0x0D || (value >= 0x20 && value <= 0x7F)))
                {
                    printf("\x1B[31m%c\x1B[0m", value);
                    return 0;
                }
            }
            else
            {
                return 0;
            }
            break;

        case 0x04:
            msg = "Read byte: 0x%02X";
            msg_arg1 = s->tio_rxbyte & 0xFF;
            ret = s->tio_rxbyte & 0xFF;
            while (ret == 0 || ret == EOF) {
                ret = getchar();
            }
            return ret;
            break;
       
        case 0x08:
            /* quiet */
            return 0;

        case 0x14:
            if(type & MODE_WRITE)
            {
                if(value & 1)
                {
                    msg = "Reset RX indicator";
                    s->tio_rxbyte |= 0x100;
                    return 0; /* nkls: quiet */
                }
                else
                {
                    /* quiet */
                    return 0;
                }
            }
            else
            {
                if((s->tio_rxbyte & 0x100) == 0)
                {
                    msg = "Signalling RX indicator";
                    ret = 3;
                }
                else
                {
                    /* quiet */
                    return 3; /* nkls: signal something on the line */
                    //return 2;
                }
            }
            break;
    }

    io_log("TIO", s, address, type, value, ret, msg, msg_arg1, 0);
    return ret;
}
is there a hidden menu option (https://bitbucket.org/niklastisk/qemu/src/2d85ec1ee3a5feb3233183ee8180c8b10b04e5cd/hw/eos/eos_bufcon_100D.h?at=100D-testing&fileviewer=file-view-default) in other firmwares as well?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on May 17, 2016, 01:07:19 AM
one problem of qemu was, you couldnt emulate real flashes. (not SPI)
to write properties, the firmware is issuing flash state machine commands (writes to the flash base address)

QEMU cannot handle (passing writes to a driver) this. at least the version of 1.5 years ago wasn't able.
IIRC this has to do with the fact that the ARM instruction set isn't interpreted, but somewhat translated.

so it might be troublesome to get the whole camera hardware behave correctly using QEMU, if there is no workaround.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 19, 2016, 06:37:29 PM
@nkls: I think I've managed to integrate your code with my latest changes. It was a little difficult, and I'm not yet sure I did the merges right, so may I ask you to check whether it's still working fine on your side?

The DebugMsg hack is pretty cool - I just managed to boot the 60D GUI with full debug messages, without any additional guest code (no autoexec.bin loaded).

Didn't try the other stuff yet.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 21, 2016, 10:25:54 PM
Now that @nkls introduced me to the world of GDB scripting, here's a quick tip for tracing various functions around the firmware:

Code: [Select]
# task_create
br *0x98CC
commands
  silent
  printf "task_create(%s, prio=%x, stack=%x, entry=%x, arg=%x)\n", $r0, $r1, $r2, $r3, *(int*)$sp
  c
end

br *0xFF12CB14
commands
  silent
  printf "SearchFromProperty(%x,%x) from %x\n", $r0, $r1, $pc
  c
end

I also tried to implement the DebugMsg hook with pure gdb scripting, but I got stuck because gdb wants the exact number of arguments to the format string. For example, this works for a DebugMsg call with a single % in it, but fails otherwise.

Code: [Select]
  eval "printf \"[DebugMsg] (%%02x,%%02x) %s\n\", $r0, $r1, $r3", $r2

Any hints would be welcome.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on May 30, 2016, 10:26:16 PM
Nice to see things pushed to the main repo and that you've sorted out the open ends I left in there! It seems to work and looks correct, so I think it's all good.

I remember trying to get the debug messages showing through pure gdb for a while but then gave up and hacked it together in C instead. It might be possible to some extent but I bet it'll be ugly. Maybe count %-signs and branch to different cases?

The gdb-qemu link isn't ideal either, the current:
Code: [Select]
set *0xCF999001 = *0xCF999001triggers the IO-handle when it reads from *0xCF999001 but not when it writes to it and I can't figure out why. Not that it matters much, but it would be nice to be able to send values to qemu through a single address.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 12, 2016, 12:47:44 PM
Maybe count %-signs and branch to different cases?

Yes, that's exactly what I ended up with. I made a generic logging library for various functions around the firmware (tasks, semaphores, timers, interrupts, MPU communication), all in pure gdb:

https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/debug-logging.gdb

This file can be included in the model-specific GDB script, like this:
https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/5D3/debugmsg.gdb
https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/scripts/70D/debugmsg.gdb

and then get a log from both QEMU and GDB on the same terminal with a command like this:
Code: [Select]
./run_canon_fw.sh 5D3 -s -S & arm-none-eabi-gdb -x 5D3/debugmsg.gdb

The functionality is similar to the one in the dm-spy-experiments branch (http://www.magiclantern.fm/forum/index.php?topic=2388.msg113154#msg113154), but - as you pointed out - it works without changing the guest code or having to load autoexec.bin (for example, I could use it on 7D2).

Also, during the past two weeks (when I was without internet access), I made huge progress regarding emulation of 70D, 5D3, 7D2 and EOS M3. Canon GUI doesn't start yet, but it shouldn't be very far away.

I've also extended model_list to include other model-specific parameters, and made it easy to add new ones:

https://bitbucket.org/hudson/magic-lantern/src/qemu-nkls/contrib/qemu/eos/model_list.c

BTW, I suspect the serial flash contents are not fully correct. In your dumper, I changed the buffer to uncacheable (fio_malloc) and got slightly better results (property blocks were recognized), and I also had to do this change (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls) when reading the serial flash contents via DMA. The data appears offset by half-byte, and on regular (non-DMA) reads, Canon code fixes it in the same way (in ReadBlockSerialFlash, for blocks smaller than 0x200). I guess the DMA engine is expected to apply the same "fix".

However, after these two changes, I still suspect data corruption (couldn't parse the property data structures completely - the first few properties are fine, and after a while it's pure gibberish, at least on the 70D SF dump from nikfreak). Guess we'll have to dump the RAM address where the serial flash contents is loaded (or maybe attempt to read the entire serial flash with a single call in the dumper).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Walter Schulz on June 12, 2016, 03:03:05 PM
Also, during the past two weeks (when I was without internet access), I made huge progress regarding emulation of 70D, 5D3, 7D2 and EOS M3.

EOS M3? Didn't see that coming ...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 12, 2016, 04:46:49 PM
Well, it's the only other DIGIC 6 camera for which I have a firmware (or did any other camera get a firmware update meanwhile?)

Sure, it's a PowerShot, but the DryOS core is the same, so I actually used it in order to understand the 7D2 code better. I only tried to emulate Canon firmware, didn't try to load any sort of custom code on it yet.

I'll post more details about the M3 on the relevant thread, since this emulation can also be useful for those who are porting CHDK on this camera.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on June 13, 2016, 10:57:16 PM
Wow, I'm impressed! Nice job! :) I won't be able to try it out in the upcoming two weeks, but I'll give it a go after that.

You are right about the sf data being manipulated in some way, I wondered what strange format would produce data like:
Code: [Select]
0000fff0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00010000: 0fff f000 0dcf 70f0 0000 0000 1d0f 70f0  ......p.......p.
00010010: 0000 0000 0040 3000 0000 0000 10c0 0000  .....@0.........
00010020: 0fff ffff f010 0000 10c0 0000 0fea fdcb  ................
00010030: a080 0000 1180 0000 0000 0000 0000 0000  ................
00010040: 0000 0000 0000 0000 0020 0000 1180 0000  ......... ......
but now that you say it it looks very half-byte shifted at 0x10000.

Interestingly, the first block contains the version string "4.2.1" which is not half-byte shifted.
Code: [Select]
00000000: 4603 0080 342e 322e 3100 0000 0000 0000  F...4.2.1.......
00000010: 0000 0000 ffff ffff ffff ffff ffff ffff  ................
00000020: ffff ffff ffff ffff ffff ffff ffff ffff  ................

Is it just me or is the write address not incremented in your fix (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls)?

So wait, is the data still unshifted when you use uncacheable memory? What happens with the data block exactly -- is the first half-byte always set to zero, and the last half-byte discarded? Any indication of the higher-level calls requesting more data than asked for, or is the last byte of a read always assumed to be bogus? (note to self)

You could also try to force it to only use non-DMA/only-DMA with gdb and see if that works better, I recall getting different results when doing that.

Guess we'll have to dump the RAM address where the serial flash contents is loaded (or maybe attempt to read the entire serial flash with a single call in the dumper).
It'd be tricky to allocate 16MB(?) for the full flash wouldn't it? It would also be interesting to see how e.g. one 2kb read compares to two 1kb reads, maybe we just need to overlap the block reads by a few bytes and unshift them to make it work.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 14, 2016, 07:14:38 AM
Is it just me or is the write address not incremented in your fix (https://bitbucket.org/hudson/magic-lantern/commits/097d66e6ba8da80736096feccb2184112a1ef603?at=qemu-nkls)?

Wow, great catch; it fixes a bunch of asserts in 70D log, regarding those properties located in serial flash.

Quote
So wait, is the data still unshifted when you use uncacheable memory?
Yes.

Quote
What happens with the data block exactly -- is the first half-byte always set to zero, and the last half-byte discarded?

Canon code does this:
Code: [Select]
if ( size < 0x200 )
{
      readSerialFlash(src, buf, 512);
      while ( offset < size )
      {
        *(_BYTE *)(out + offset) = 16 * *(_BYTE *)(buf + offset) | (*(_BYTE *)(buf + offset + 1) >> 4);
        ++offset;
      }
}
else
{
    v6 = readSerialFlashWithQuad(src, out, size);
}

Quote
It'd be tricky to allocate 16MB(?) for the full flash wouldn't it?

Nope, just try it.

(some background info here (http://www.magiclantern.fm/forum/index.php?topic=5071.msg166799#msg166799) and here (http://www.magiclantern.fm/forum/index.php?topic=8358.0))
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on June 19, 2016, 12:26:47 PM
I had another look at the flash memory yesterday, and I figured out a few things. Your code appears to load the serial flash correctly for the 100D without changes, and it might be that it's just the spells causing the property errors. (Still no canon gui.)

The chip in my camera is a Winbond W25Q128FV (datasheet (https://www.pjrc.com/teensy/W25Q128FV.pdf)), and not a Macronix as I thought before. The Manufacturer Code and device ID is {0xEF,0x40,0x18} which is what I got from SPI instruction 9Fh. This doesn't matter much since it works anyways, but it's good to have the actual data sheet to base the flash emulation of.

There are two high-level interfaces (with the same debug names) for the serial flash. One uses SPI-only transfer with "serial layout", while the other is "QUAD" read which reads data through DMA, or SPI if the read block is small enough. The DMA read has this shifted data layout, maybe due to physical wiring limitations or obfuscation or whatnot. The code you posted is used by the property manager, and is from the QUAD interface, which has to un-shift the SPI data to ensure proper data layout.

This explains why the version string in the first block is readable -- it is read and written by the SPI-only interface which don't expect DMA layout. I've used the address of the SPI-only interface in my dumper, so the data in the dump should be the same as in the actual flash.

I've also tried to change the block size, and there is no difference between the data received from a read-all-at-once dump file and a 1024-byte fio_malloc'ed dumper.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 19, 2016, 09:10:33 PM
it might be that it's just the spells causing the property errors.

These should be easy to fix. Did you take the spells from another camera, or did you log the 100D startup with dm-spy-experiments?

Spells from another camera will cause issues.



BTW, I've included a small SD/CF image here (https://bitbucket.org/hudson/magic-lantern/commits/84b3f86ec6921eb65783764fa341801d88841209), hopefully this makes it easier to install and get started. The card image is bootable and includes a small autoexec.bin that does the display test (http://www.magiclantern.fm/forum/index.php?topic=14732.0), as this runs on most (if not all) DIGIC 4/5 models out of the box.

That means: run the install script, compile QEMU, copy your ROM from the ML card and it should be ready to go.

Still, it won't display the GUI on cameras other than 60D...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 20, 2016, 11:28:05 AM
Okay, I know you won't believe this one.

After about 2 weeks of intensive work on QEMU, I couldn't manage to get 70D and 5D3 working.

Today, after about half an hour of tinkering... 1200D Canon GUI boots.... with the MPU SPELLS from 60D!

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-qemu.png)

Emulation log: 1200D-qemu.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-qemu.log)

These were the changes I did to QEMU for 1200D: [1] (https://bitbucket.org/hudson/magic-lantern/commits/7241ce6bec50947082bcec05ad5078cf963baf08) [2] (https://bitbucket.org/hudson/magic-lantern/commits/507c96ed43b51c80d7776d70ddd12d00e366b0a5) [3] (https://bitbucket.org/hudson/magic-lantern/commits/008aaa6fa261beb45a27bdd08de919dbd3724fcb). And a GDB script to help me see what's going on: [4] (https://bitbucket.org/hudson/magic-lantern/commits/60225f0e05c09b551aeaef001237723c73f96668).

I tried the 60D spells just to see how it goes, but I didn't hope it would go that far.

(side note: currently we can see the MPU spells only after ML is ported, as they come from a secondary CPU whose firmware is pretty much impossible to understand, at least for me)

At this point, I think it's worth trying most other DIGIC 4 cameras with SD card. They probably require a MPU spell log (easy to get, as described here (http://magiclantern.fm/forum/index.php?topic=2864.msg166938#msg166938)), and probably a few other minor tweaks. Happy hacking!

P.S. looks like it took me more time to write this message, than to actually get the 1200D GUI booting :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 20, 2016, 07:59:02 PM
Some more:

1100D also ran out of the box with 60D spells:
(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1100D-qemu.png)

550D is stubborn; I ended up with this after patching lvInit (not yet committed):
(http://a1ex.magiclantern.fm/bleeding-edge/qemu/550D-qemu.png)

600D needs its own MPU spell set (most likely because of the crop video mode settings, which are stored in the MPU), but I expect it to work without much trouble.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 24, 2016, 12:02:04 AM
alex, I am having a lot of errors like

/home/matias/qemu/qemu-2.5.0/hw/arm/../eos/eos_handle_serial_flash.c:38:5: error: ‘for’ loop initial declarations are only allowed in C99 mode
/home/matias/qemu/qemu-2.5.0/hw/arm/../eos/eos_handle_serial_flash.c:38:5: note: use option -std=c99 or -std=gnu99 to compile your code

I've fixed some, but seems that some config is missing, or i am wrong?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2016, 03:38:14 PM
Interesting; I used to have C99 errors before, but they were no longer present after upgrading to QEMU 2.5.0 and Ubuntu 15.10. I thought QEMU devs enabled C99 in newer versions, so I started using C99 constructs in my code.

Looks like the answer is here:

https://gcc.gnu.org/onlinedocs/gcc-4.8.0/gcc/Standards.html
Quote
The default, if no C language dialect options are given, is -std=gnu90

https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Standards.html
Quote
The default, if no C language dialect options are given, is -std=gnu11

Try adding -std=gnu99 to CFLAGS in your QEMU Makefile, or when configuring it. For example:

Code: [Select]
/path/to/ml/qemu/qemu-2.5.0$ CFLAGS=-std=gnu99 ../configure_eos.sh

(this should be included in the installer, actually)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 24, 2016, 07:09:04 PM
Great,

I was able to run 1200D with Hello world in qemu 2.5.0
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 24, 2016, 11:48:28 PM
Interesting, after trying a couple of times (and getting different execution logs every time) I finally got your hello world running as well.

Looks like there is a race condition somewhere in the emulation.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 29, 2016, 05:50:04 AM
I don't know why but it's throwing an error if i remove hello world from define. the error is not clear (at least for my)

Error loading 'ML/MODULES/1200D_100.sym': File does not exist
while in the emulator I see some error like symbols not found (dissapears fast), if I hit del key screen goes black.

but tried in 1.6 version, no error and del key givesme ML menu.

I don't know how did you do to display GUI in qemu
(before testing port in my camera I always try to see it working in qemu)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 29, 2016, 07:31:07 AM
If the file does exist, ML cannot load it because its name is longer than 8 characters.

Were you able to get the GUI in QEMU 2.5.0 like this?

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/1200D-hello-qemu.png)

For me, it doesn't work every time; I have to start QEMU a couple of times to get this screen. This only happens when loading autoexec.bin; if I try to run plain Canon firmware (with bootflag disabled), it works every time.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nikfreak on June 29, 2016, 07:47:20 AM
Error loading 'ML/MODULES/1200D_100.sym': File does not exist
while in the emulator I see some error like symbols not found (dissapears fast), if I hit del key screen goes black.

If 8.3 filenaming applies to qemu, too then you got your answer now: shorten to "x70_100.sym" or something like that.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 29, 2016, 12:23:03 PM
If 8.3 filenaming applies to qemu

QEMU actually emulates a SD card device, with all the low-level communication (including DMA transfers), so filesystem behavior should match Canon's. For example, if you start the emulation on a formatted SD image, you will see Canon firmware creating the DCIM and MISC folders.

I expect formatting from Canon menu should work as well, just didn't try it. This needs the MPU spells (button codes and whatever other GUI events there might be) for navigating Canon menu.

For CF cards, emulation currently works properly in the bootloader (e.g. loading autoexec.bin from card), but not in the main firmware.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: mathias on June 30, 2016, 04:35:33 AM
Well tried to reinstall qemu (in case my fail) but same errors appears.
using 1200D branch, if I run it with HELLO_WORLD I get this: (no cannon GUI)
(https://s31.postimg.org/t6cdd17cn/Screenshot_from_2016_06_29_19_05_48.png) (https://postimg.org/image/t6cdd17cn/)

if i remove it i get file not found as I said before. (Double checked and the file exist, this model 1200D only has SD card so i am just mounting sd, notice the file in the screenshot)

(https://s31.postimg.org/rp8dhtvqf/Screenshot_from_2016_06_29_19_19_56.png) (https://postimg.org/image/rp8dhtvqf/)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 30, 2016, 06:45:11 AM
Copy all ML files (make install), not just autoexec ;)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 30, 2016, 06:17:01 PM
600D is just as stubborn as 550D, but easier to debug (more recent codebase).

With proper MPU spells, it gives the date/time screen, just like 550D.

With a small patch on the RTC init routine, it gives the sensor cleaning animation (real-time!)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/600D-qemu-datetime.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/600D-qemu-sensorcleaning.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 01, 2016, 05:35:51 PM
500D :
Code: [Select]
0xFF18A884 - mpu_send
0xFF05C1F0 - mpu_recv
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 08, 2016, 10:58:19 PM
Some progress understanding MPU messages:

Button codes

They are encoded like this:
Code: [Select]
0x06, 0x05, 0x06, 0x00, btn_code, btn_code_arg

The button codes can be found from bindReceiveSwitch - this translates the MPU button codes (btn_code, btn_code_arg) into GUI button codes as used by GuiMainTask (the BGMT constants from gui.h). When the GUI button codes are sent to GuiMainTask, they appear in the debug log as "GUI_Control:%d 0x%x". When they are actually processed, they appear as "GUI_CONTROL:%d".

Since finding these button codes for each camera would be incredibly boring, I wrote a Python script (https://bitbucket.org/hudson/magic-lantern/commits/d40c4d99e99351e3b6ee77933885dee9f85977cb#chg-contrib/qemu/eos/mpu_spells/extract_button_codes.py) to get them automatically from the ROM, by directly emulating bindReceiveSwitch in unicorn (http://www.unicorn-engine.org), trying all usual input values and checking debug messages.

btn_code_arg meaning can be: press/unpress (1,0), scrollwheel direction (1, -1) and number of steps for very fast turns (2, -3 etc), or some buttons can be grouped under a single btn_code (for example, the direction pad).

Many button codes are common across all cameras (DIGIC 4 and 5): MENU (0,1), INFO (1,1), PLAY (3,1), DELETE (4,1), SET (12,1/0), scrollwheels (13 and 14), others are not.

btn_code = 30 is ServiceMenu on all cameras. Interesting string on 1200D: "Enter Secret mode Electric Shutter!!!".

Some cameras also use a generic event, GUICMD_PRESS_BUTTON_SOMETHING, which I don't know how to interpret (other than some button was pressed).

Side note: this research uncovered a few subtle bugs regarding button codes on 600D, 100D and EOS M, and not-so-subtle on 1100D (see this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/743/misc-cleanups-some-gui-button-codes/diff)).

GUI modes

On 600D, menu navigation looks somewhat like this ("spell" being data sent from ICU to MPU, and "reply" being the response from MPU):
Code: [Select]
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #44 */
        { 0x06, 0x05, 0x03, 0x17, 0x9a, 0x00 },                 /* reply #44.1 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #44.2, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x00, 0x01, 0x00 },                 /* reply #44.3, BGMT_MENU, GUI_Control:6, bindReceiveSwitch(0, 1) */
        { 0x06, 0x05, 0x04, 0x0d, 0x00, 0x00 },                 /* reply #44.4 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #45 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #45.1, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x00, 0x01, 0x00 },                 /* reply #45.2, BGMT_MENU, GUI_Control:6, bindReceiveSwitch(0, 1) */
        { 0 } } }, {
    { 0x06, 0x05, 0x04, 0x00, 0x01, 0x00 }, {                   /* spell #46, NotifyGUIEvent(1) */
        { 0x06, 0x05, 0x06, 0x0a, 0x00, 0x00 },                 /* reply #46.1, BGMT_UNPRESS_ZOOMOUT_MAYBE, GUI_Control:17, bindReceiveSwitch(10, 0) */
        { 0x06, 0x05, 0x06, 0x09, 0x00, 0x00 },                 /* reply #46.2, BGMT_UNPRESS_ZOOMIN_MAYBE, GUI_Control:15, bindReceiveSwitch(9, 0) */
        { 0x06, 0x05, 0x04, 0x00, 0x01, 0x01 },                 /* reply #46.3 */
        { 0x0e, 0x0c, 0x0a, 0x08, 0x11, 0x00, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00 },/* reply #46.4 */
        { 0 } } }, {
    { 0x08, 0x06, 0x00, 0x00, 0x04, 0x00, 0x00 }, {             /* spell #47, Complete WaitID = 0x80020000 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x34, 0x00, 0x00 }, {                   /* spell #48 */
        { 0 } } }, {
    { 0x06, 0x05, 0x03, 0x19, 0x00, 0x00 }, {                   /* spell #49 */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #49.1, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x01, 0x00 },                 /* reply #49.2, BGMT_PRESS_RIGHT, GUI_Control:35, bindReceiveSwitch(26, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x00, 0x00 },                 /* reply #49.3, BGMT_UNPRESS_RIGHT, GUI_Control:36, bindReceiveSwitch(26, 0) */
        { 0x06, 0x05, 0x06, 0x26, 0x01, 0x00 },                 /* reply #49.4, GUI_Control:76, bindReceiveSwitch(38, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x01, 0x00 },                 /* reply #49.5, BGMT_PRESS_RIGHT, GUI_Control:35, bindReceiveSwitch(26, 1) */
        { 0x06, 0x05, 0x06, 0x1a, 0x00, 0x00 },                 /* reply #49.6, BGMT_UNPRESS_RIGHT, GUI_Control:36, bindReceiveSwitch(26, 0) */

NotifyGUIEvent (called by SetGUIRequestMode) sends a message like this:
Code: [Select]
0x06, 0x05, 0x04, 0x00, event_code, 0x00

The MPU is supposed to reply something, probably this:
Code: [Select]
0x06, 0x05, 0x04, 0x00, event_code, 0x01

Now the interesting part: if I enable the NotifyGUIEvent reply on 60D, and the other cameras that accept the same MPU spell set, they no longer boot the GUI: instead, they go to the date/time dialog. You can adjust the date/time in QEMU using the arrow keys, scrollwheels, and the spacebar (SET), but when pressing OK, the GUI freezes.

Obviously, with NotifyGUIEvent disabled, the GUI mode can't be changed (so you can't enter Canon menu, or playback mode, or whatever).

The question is: how to make the GUI mode switches work, so one can navigate the menu?



Note: with current implementation, you can already navigate ML menu without CONFIG_QEMU=y... if you define GUIMODE_ML_MENU = 0 (so it won't try to change the GUI mode, because that part doesn't work). One big step closer towards running unmodified ML in QEMU :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 18, 2016, 01:58:19 AM
500D :
(https://s31.postimg.org/aa32bvxff/500_D.png) (https://s32.postimg.org/teuuqtlvp/500d.gif) (https://s31.postimg.org/vxp1gewpn/500d1.png) (https://s31.postimg.org/5qnukgefv/500d2.png) (https://s31.postimg.org/ok9nhgcnv/500d3.png)
Code: [Select]
# ./run_canon_fw.sh 500D -s -S & arm-none-eabi-gdb -x 500D/debugmsg.gdb

source -v debug-logging.gdb

macro define CURRENT_TASK 0x1A74
macro define CURRENT_ISR  (*(int*)0x664 ? (*(int*)0x668) >> 2 : 0)

b *0xFF066A98
DebugMsg_log

b *0xFF069E2C
task_create_log

b *0xFF064520
load_default_date_time_log
macro define RTC_VALID_FLAG (*(int*)0x2BC4)

cont
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: budafilms on July 21, 2016, 12:09:46 PM
Hi everybody,
an optimistic question from someone without skills to use this: wich is the real utility/advantage of this?

(I mean, for example, more memory, more resolution, new language for coding, girls...)

Thanks!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Walter Schulz on July 21, 2016, 12:53:02 PM
Testing code without cam:
- Cam will not brick
- You have not to wear out gear (card, slot, cardreader, etc.) each time you have to replace binaries.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 21, 2016, 01:33:09 PM
Testing code without cam:
- Cam will not brick
- You have not to wear out gear (card, slot, cardreader, etc.) each time you have to replace binaries.

- debugging your code like a PC program, by running it step by step (not just with printf's)
- you can debug Canon code as well, in order to understand what it does
- possible to implement automated tests for the nightly builds (see also http://direct.magiclantern.fm/forum/index.php?topic=12396.0 )
- test Lua scripts in the emulator
- lower the entry barrier for new developers
- assist me in unbricking cameras (the main driving force behind this, sadly)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on July 21, 2016, 03:22:31 PM
500D, LV
Code: [Select]
mpu_send(06 04 09 00 00)
mpu_recv(3c 3a 09 00 3c 3c e0 00 3f 80 00 00 38 12 c0 00 b9 cb c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08 08 11 10 50 49 02 59 88 88 00 32 00 00 00 00 00 01 00 00 00)
PROP_LV_LENS


0x32 - focal length
0x10 - aperture

Now we can read lens_info in Photo mode.
Just call mpu_send(06 04 09 00 00). CPU receives data and automatically overwrite property lens_info.

(https://s32.postimg.org/3kk7nrbqd/500d.jpg) (https://s32.postimg.org/hfii6865h/500d2.jpg)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on July 22, 2016, 11:27:57 AM
Nice trick.

Let's continue the discussion regarding MPU communication here: http://www.magiclantern.fm/forum/index.php?topic=17596
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on July 29, 2016, 10:33:43 AM
I've managed to get the dm-spy-extra branch to run for 100D (fw 100A) with full logging, sources are at:
https://bitbucket.org/niklastisk/magic-lantern/branch/100D-mpu-spy

Still no success at emulating the 100D though, just including the correct mpu messages did not work out.


On a side note, me and a1ex discovered some interesting things about the memory management of the 100D's 256M RAM when using the CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP configuration:
Code: [Select]
      p1=&memtest; p3=UNCACHEABLE(p1);
      init:00b1c75c:00:00: POINTER   p1         p2         p3
      init:00b1c77c:00:00: ADDRESS   0x00B7F204 0x10B7F204 0x40B7F204
      init:00b1c7a8:00:00: (init)    0x00000123 0x00000123 0x00000123
      init:00b1c7e0:00:00: memtest++ 0x00000124 0x00000123 0x00000123
      init:00b1c818:00:00: (*p1)++   0x00000125 0x00000123 0x00000123
      init:00b1c850:00:00: (*p2)++   0x00000125 0x00000123 0x00000123
      init:00b1c888:00:00: (*p3)++   0x00000125 0x00000124 0x00000124
(source code: http://pastebin.com/vHWcH9ij)

My interpretation is that (0x10000000 | ptr) creates a pointer that wraps around the 256M ram, but bypasses the cache and is read-only such that write attempts fail silently. This obviously won't work for 512M cameras. :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on August 01, 2016, 11:04:47 AM
I've fixed the serial flash reads, turns out the QUAD mode discards the last 16 bytes in each 0x800-byte block. Now the property tables are read correctly.

https://bitbucket.org/niklastisk/qemu/branch/eos-develop-new

The spell sets for 100D are included as well, and I even got the GUI showing (once)! There still seems to be problems with the touch panel, but I don't know if they are critical or not.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 10, 2016, 11:54:09 PM
Just committed some updates, mostly written while I had limited network connectivity (which is why they weren't uploaded earlier):

- directory layout changed, you need to move the ROM files (see install script)
- 5D3 boots the GUI on 1.1.3 :)
- cache hack emulation (very incomplete, only tested on 60D dm-spy, which runs successfully and saves its log on the SD card image)
- SD emulation fix (card info is now read correctly and gets mounted on digic 5 models)
- experimental EDMAC emulation (didn't really play with it, but can be used to research the picture taking process, bit depth reduction tricks, eeko and lots of other low level stuff)
- initial support for 750D and 760D (nothing interesting, they crash in the same way as 80D)
- integrated nkls' changes about 100D, but not yet tested (they will certainly be helpful for 70D and most other recent cameras)

I'm also starting to get confident about emulating the TX19A in QEMU, thanks to leegong's findings, but I have nothing interesting to show yet.

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 26, 2016, 01:07:16 AM
Small updates:

- card LED emulation
- better SD emulation on DIGIC 6 (tested on EOS M3 - it loads DISKBOOT.BIN (https://chdk.setepontos.com/index.php?topic=12542.msg130078#msg130078) from the SD card image)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 26, 2016, 11:52:52 PM
We now have a test suite (https://bitbucket.org/hudson/magic-lantern/commits/99d05c437b26edd515342b03528c183f0f437e4c) for QEMU :)

Code: [Select]
./run_tests.sh
Compiling...

Testing bootloaders...
  5D2: K218 READY
  5D3: K285 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
 7D2S: K289S READY
  50D: FAILED!
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY

Testing display from bootloader...
Setting up a temporary SD card image...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’
  5D2: OK
  5D3: OK
   6D: OK
   7D: please check
 7D2M: OK
 7D2S: OK
  50D: please check
  60D: OK
  70D: OK
  80D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1100D: OK
1200D: OK
 EOSM: OK
Restoring your SD card image...

Testing Canon GUI...
  60D: OK
  5D3: OK
 600D: OK
1200D: OK
1100D: OK

Can you recommend some testing framework that could work in this case? I feel like I'm reinventing the wheel with all those scripts, but at least it works. For now.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on September 30, 2016, 12:03:45 AM
Added support for a few more cameras (they all load autexec and run the display test):
- 5D, 40D, 400D, 450D, 1000D (oldies)
- 50D (minor fix)
- 5D4

This one just boots DryOS and starts a few tasks:
- A1100 (PowerShot)

Current test results:
Code: [Select]
Testing bootloaders...
   5D: ROM READY
  5D2: K218 READY
  5D3: K285 READY
  5D4: K349 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
 7D2S: K289S READY
  40D: K190 READY : Ver 4.0.1
  50D: K261 READY
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 400D: InitializeIntercom
 450D: K176 READY : Ver 4.0.3
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1000D: K254 READY : Ver 3.7.5
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY

Setting up temporary SD/CF card images...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’

Testing display from bootloader...
   5D: OK
  5D2: OK
  5D3: OK
  5D4: OK
   6D: OK
   7D: please check
 7D2M: OK
 7D2S: OK
  40D: OK
  50D: OK
  60D: OK
  70D: OK
  80D: OK
 400D: OK
 450D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1000D: OK
1100D: OK
1200D: OK
 EOSM: OK

Testing EOS M3...
SD boot: StartDiskboot
Display: TurnOnDisplay

Restoring your SD/CF card images...

Testing Canon GUI...
  60D: OK
  5D3: OK
 600D: OK
1200D: OK
1100D: OK

Hope all of this will be useful to other developers  8)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 21, 2016, 09:24:29 PM
Hello, I'm new to ML, I'm trying it on an EOSM. I wish to develop a module, but it would be really difficult using the "physical" camera. So I'm trying to setup an emulation environment... unluckily I miss something... I'm stuck on qemu... I need some help.

Code: [Select]
/opt/qemu/bin/qemu-system-arm -machine EOSM
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 3FFFFFFF: eos.ram
40001000 - 7FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror_F1
F2000000 - F2FFFFFF: eos.rom0_mirror_F2
F3000000 - F3FFFFFF: eos.rom0_mirror_F3
F4000000 - F4FFFFFF: eos.rom0_mirror_F4
F5000000 - F5FFFFFF: eos.rom0_mirror_F5
F6000000 - F6FFFFFF: eos.rom0_mirror_F6
F7000000 - F7FFFFFF: eos.rom0_mirror_F7
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror_F9
FA000000 - FAFFFFFF: eos.rom1_mirror_FA
FB000000 - FBFFFFFF: eos.rom1_mirror_FB
FC000000 - FCFFFFFF: eos.rom1_mirror_FC
FD000000 - FDFFFFFF: eos.rom1_mirror_FD
FE000000 - FEFFFFFF: eos.rom1_mirror_FE
FF000000 - FFFFFFFF: eos.rom1_mirror_FF
C0000000 - CFFFFFFF: eos.iomem
[EOS] loading 'ROM-EOSM.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading 'ROM-EOSM.BIN' to 0xF8000000-0xF8FFFFFF (offset 0x1000000)
FIXME: no MPU spells for EOSM.

Available keys:
- Arrow keys   : Navigation
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- M            : MENU
- P            : PLAY
- I            : INFO/DISP
- L            : LiveView
- F1           : show this help

VNC server running on '127.0.0.1;5900'

I know that without MPU spells I cannot display Canon menu. But the only think I can display in VNC is the garbled screen. How to load ML?
I compiled EOSM platform with CONFIG_QEMU=y and I made qemu-helper.bin.

(https://s4.postimg.org/qvvqp774d/desktop10.png)

This is my directory setup (all uppercase as required?):

Code: [Select]
├── ROM-EOSM.BIN
└── sdcard
     ├── AUTOEXEC.BIN
     ├── ML
     │   ├── CROPMKS
     │   │   ├── CINESCO2.BMP
     │   │   ├── CRSSMTR2.BMP
     │   │   ├── PASSPORT.BMP
     │   │   ├── PHIPHOTO.BMP
     │   │   └── PHIVIDEO.BMP
     │   ├── DATA
     │   │   ├── APSC8P.LUT
     │   │   ├── APSC8R.LUT
     │   │   ├── FF8P.LUT
     │   │   └── FF8R.LUT
     │   ├── DOC
     │   ├── DOCS
     │   ├── FONTS
     │   │   ├── ARGHLF22.RBF
     │   │   ├── ARGNOR23.RBF
     │   │   ├── ARGNOR28.RBF
     │   │   ├── ARGNOR32.RBF
     │   │   ├── TERM12.RBF
     │   │   └── TERM20.RBF
     │   ├── MODULES
     │   │   ├── ADV_INT.MO
     │   │   ├── ARKANOID.MO
     │   │   ├── AUTOEXPO.MO
     │   │   ├── BENCH.MO
     │   │   ├── DEFLICK.MO
     │   │   ├── DUAL_ISO.MO
     │   │   ├── EOSM_202.SYM
     │   │   ├── ETTR.MO
     │   │   ├── FILE_MAN.MO
     │   │   ├── LUA.MO
     │   │   ├── MLV_PLAY.MO
     │   │   ├── MLV_REC.MO
     │   │   ├── MLV_SND.MO
     │   │   ├── PANO.MO
     │   │   ├── PIC_VIEW.MO
     │   │   ├── RAW_REC.MO
     │   │   ├── SELFTEST.MO
     │   │   └── SILENT.MO
     │   ├── README
     │   └── SCRIPTS
     ├── MAGICLANTERN
     ├── ML-SETUP.FIR
     └── QEMU-HELPER.BIN

Qemu 2.5.50 from niklastisk repo, branch eos-develop-new (./configure --disable-docs --target-list=arm-softmmu).

...anyway, I'm asking myself: is there a way to display ML menu at startup in the emulator? Otherwise it would be a bit difficult to recall it on EOSM... on the real camera I use double touch on the screen...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 21, 2016, 09:51:07 PM
PS: I had to comment out a code fragment from eos_init_common(). Otherwise, after loading ROM-EOSM.BIN, qemu exits with a "CF init failed".

Code: [Select]
    /* init CF card */
    DriveInfo *dj;
    dj = drive_get_next(IF_IDE);
    if (!dj) {
        printf("CF init failed\n");
        exit(1);
    }

    ide_bus_new(&s->cf.bus, sizeof(s->cf.bus), NULL, 0, 2);
    ide_init2(&s->cf.bus, s->interrupt);
    ide_create_drive(&s->cf.bus, 0, dj);

EOSM doesn't have CF. It fails also if I create a cfcard directory. And now I have the dubt that sdcard and cfcard directory are totally ignored...
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 01:37:07 PM
For now, I recommend working on a model that can display Canon menu (at the time of writing, 60D, 5D3, 1100D or 1200D).

There are two qemu branches, with different design goals:

- the one in unified, which only attempts to run some ML code:
   - it displays the ML menu for most models
   - requires several changes to ML code (lots of hacks - grep for CONFIG_QEMU)
   - runs only a tiny part of Canon code (some startup code, task scheduler and... that's pretty much it)
   - there is a filesystem emulation, but it's extremely hackish: ML file I/O calls are replaced with wrapped calls to the host system (in the sdcard/cfcard directories)
   - it also runs some LiveView and HDMI tests (also very hackish)
   - back then, emulating Canon GUI was a pipe dream
   - I don't remember touching it in the last 1-2 years, so it should probably go away.

- the one in the qemu branch, which attempts to run the complete Canon firmware and unmodified autoexec.bin:
   - ML menu only works on those models that can boot Canon GUI
   - to navigate it easily, define GUIMODE_ML_MENU as 0 in consts.h (that's because GUI mode switching doesn't work yet)
   - patching ML code is optional (it can load unmodified autoexec.bin files, but patching can still be useful)
   - storage emulation uses a SD or CF card image (a single big file)
   - SD/CF emulation works by emulating the low-level calls (Identify Drive, read/write sector, stuff like that)
   - bootloader emulation works on nearly all models (from DIGIC 2 to DIGIC 6), including loading autoexec.bin from the card image
   - file I/O calls from both Canon and ML code are expected to work well on SD (e.g. loading ML modules, saving settings, creating the DCIM and MISC directories from Canon code)
   - CF emulation is not very solid (there are nondeterministic errors)
   - changing most Canon settings (properties) doesn't work (it requires talking to MPU)
   - LiveView and HDMI tests are very hard to run (it requires hacking some very hackish code)
   - you can debug it and trace various calls in Canon firmware using gdb (very useful for reverse engineering)

Basically, that's the current state of the emulation. Not exactly straightforward for new developers, but slowly getting there.

It also serves as "executable documentation" for a very large part of our knowledge about Canon code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 22, 2016, 02:17:48 PM
Thank for the clarifications, A1ex!

Ok, I could works indifferently on an emulated 60D, 5D3 or so. But the only ROM I have is the one dumped from my camera, so firstly I need to obtain another ROM (how?).

About ML branches, you recommend to use qemu branch for new developers? If it would be more reliable I'll use it. Actually, I wanted to use unified because of SD emulation wrapping on host fs... it would be simpler and more agile to work in ML directory without mounting and unmounting an image file.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 03:28:18 PM
I think the newer branch is a lot more robust and the emulation is much closer to the real thing.

For mounting/unmounting, it should be similar to a real SD card: once you have set up your SD card image with kpartx (see mount.sh), you click on EOS_DIGITAL in your file browser, then you run "make install" from your camera or module directory, then you start QEMU. You can automate this as well, if you like.

FYI, the install script sets up an initial SD card image, cloned from a 256MB card, which is already bootable and contains the portable display test autoexec.bin (handy to test your initial installation).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: wildstray on October 22, 2016, 10:07:04 PM
Yeah! :D After several peregrinations between repositories and patches, and thanks to install.sh and run_canon_fw.sh scripts and the suggestions of a1ex, now I have a semi-working qemu environment. After seconds, I got the Canon menu...

(https://s17.postimg.org/4e4xu2dxr/desktop11.png)

But this is the maximum I can do. When I press a key, I can see only these warnings in qemu console...

Code: [Select]
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112
Warning: no scancode found for keysym 112

Did I miss something about keycodes?

PS: I tried to run 60D and EOS ROMs... I seen the Magic Lantern Rescue screen for both. But the sd card image I prepared with ML installation is 60D.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 22, 2016, 10:19:37 PM
Looks fine (I get the same screen here).

The scancode error is internal to qemu; maybe it doesn't like some keyboard layouts?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on October 23, 2016, 12:47:28 AM
BTW, some small updates:

- initial support for 5D4 AECU aka K349AE (a secondary core, also ARM with DryOS)
- a test for GDB scripts (in the test suite)
- interactive UART support (you can navigate those bootloader menus)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/fromutil-550D.png)

Here's an easy coding task for you: some models don't have a debugmsg.gdb script. Creating one requires just finding some stubs. Right now, those models are: 5D2, 6D, 7D, 50D, 500D, 650D, 100D, 1100D and EOS M.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on October 25, 2016, 10:26:31 PM
I don't see menu on Lubuntu 16.04:
(http://thumbnails116.imagebam.com/51158/249e29511571455.jpg) (http://www.imagebam.com/image/249e29511571455)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on November 01, 2016, 08:38:20 PM
Some more updates:

- emulate the picture taking process (60D and 1200D):
   - can "capture" a full-res silent picture; that includes:
     - EDMAC transfers and interrupts
     - image processing modules (http://magiclantern.wikia.com/wiki/Register_Map#Image_PreProcessing) (dummy emulation of ADKIZ and HIV for now)
     - HEAD timers (dummy emulation for now)
   - you will need a reference image at qemu/<camera>/VRAM/PH-QR/RAW-000.DNG (a full-res silent picture)
   - it cannot emulate the unmodified silent.mo because of some GUI calls
   - you can either patch silent.c & raw.c, or use a minimal implementation
- 550D shows the date/time screen (but refuses to show the main screen)
- heartbeat timer now works on VxWorks models (can switch tasks)
- 450D emulates file I/O from main firmware
- a few more self-tests:
   - portable ROM dumper test (SD I/O from bootloader; no CF yet)
   - file I/O test from main firmware (check whether the DCIM directory is created at startup; SD models only)
   - test for picture taking emulation (test code under minimal/qemu-frsp)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-60D-frsp.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-1200D-frsp.png)

The first feature also reveals the register configurations required for taking a picture by directly driving the sensor with our own code :)

Full-res photo capture log: qemu-60D-FA_CaptureTestImage.log (http://a1ex.magiclantern.fm/bleeding-edge/qemu/qemu-60D-FA_CaptureTestImage.log)

Current self-test log:
Code: [Select]
Testing bootloaders...
   5D: ROM READY
  5D2: K218 READY
  5D3: K285 READY
  5D4: K349 READY
   6D: K302 READY
   7D: FAILED!
 7D2M: K289M READY
  40D: K190 READY : Ver 4.0.1
  50D: K261 READY
  60D: K287 READY
  70D: K325 READY
  80D: K350 READY
 400D: InitializeIntercom
 450D: K176 READY : Ver 4.0.3
 500D: K252 READY
 550D: K270 READY
 600D: K286 READY
 650D: K301 READY
 700D: K326 READY
 750D: K393 READY
 760D: K347 READY
 100D: K346 READY
1000D: K254 READY : Ver 3.7.5
1100D: K288 READY
1200D: K327 READY
 EOSM: K331 READY
5D4AE: K349AE AECU Firm Ver. 5.8.1(5.8.1)
 7D2S: K289S READY

Testing Canon GUI...
  60D: OK
  5D3: OK
 550D: OK
 600D: OK
1200D: OK
1100D: OK

Testing GDB scripts...
   5D: [  tExcTask:ffb223cc ] task_create(CmdShell, prio=a, stack=0, entry=ffb22420, arg=0)
  5D2: 5D2/debugmsg.gdb not present
  5D3: [      init:ff0c32d4 ] task_create(Startup, prio=11, stack=400, entry=ff0c2928, arg=0)
  5D4: [      init:fe0e14c9 ] task_create(SFRead, prio=11, stack=400, entry=fe0e0ef7, arg=620008)
   6D: 6D/debugmsg.gdb not present
   7D: 7D/debugmsg.gdb not present
 7D2M: [      init:00002207 ] task_create(OmarInit, prio=f, stack=400, entry=2183, arg=4ae1ec)
  40D: [ tTaskMain:ff812a60 ] task_create(HotPlug, prio=1e, stack=0, entry=ff812cb8, arg=0)
  50D: 50D/debugmsg.gdb not present
  60D: [   Startup:ff1dcc18 ] task_create(PropMgr, prio=14, stack=0, entry=ff1dcb24, arg=807b1c)
  70D: [      init:ff0c3360 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28ac, arg=0)
  80D: [      init:fe0d449f ] task_create(TaskMain, prio=1d, stack=0, entry=fe0d3619, arg=0)
 400D: [  tStartup:ffb1d1bc ] task_create(ImagePlayDriverTask, prio=19, stack=0, entry=ffa0ccd0, arg=0)
 450D: [ tTaskMain:ffd0a664 ] task_create(PropMgr, prio=15, stack=0, entry=ffd0a9e4, arg=384e5c)
 500D: 500D/debugmsg.gdb not present
 550D: [   Startup:ff1d8b30 ] task_create(PropMgr, prio=14, stack=0, entry=ff1d8a3c, arg=726d20)
 600D: [   Startup:ff1fbba8 ] task_create(PropMgr, prio=14, stack=0, entry=ff1fbab4, arg=757140)
 650D: 650D/debugmsg.gdb not present
 700D: [   Startup:ff0c38fc ] task_create(Startup2, prio=11, stack=400, entry=ff0c35b0, arg=0)
 750D: [      init:fe0ce241 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0cd4a9, arg=0)
 760D: [      init:fe0ce445 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0cd6ad, arg=0)
 100D: FAILED!
1000D: [          :ff812eec ] task_create(HotPlug, prio=1e, stack=0, entry=ff813050, arg=0)
1100D: 1100D/debugmsg.gdb not present
1200D: [      K327:ff2b9bd8 ] task_create(PropMgr, prio=14, stack=0, entry=ff2b9ae4, arg=71a11c)
 EOSM: EOSM/debugmsg.gdb not present
5D4AE: [      init:fe0a2aa1 ] task_create(TaskMain, prio=1d, stack=0, entry=fe0a2159, arg=0)
 7D2S: 7D2S/debugmsg.gdb not present
EOSM3: [   Startup:010e17a9 ] task_create(SD1stInit, prio=18, stack=0, entry=10e1739, arg=0)
A1100: [   Startup:ffc3f6b0 ] task_create(SD1stInit, prio=18, stack=0, entry=ffc3f604, arg=0)

Setting up temporary SD/CF card images...
‘../magic-lantern/contrib/qemu/sd.img.xz’ -> ‘./sd.img.xz’

Testing FA_CaptureTestImage...
  60D: OK
1200D: OK

Testing file I/O (DCIM directory)...
  60D: OK
  5D3: OK
 550D: OK
 600D: OK
1200D: OK
1100D: OK
 100D: OK
 450D: OK

Testing display from bootloader...
   5D: OK
  5D2: OK
  5D3: OK
  5D4: OK
   6D: OK
   7D: please check
 7D2M: OK
  40D: OK
  50D: OK
  60D: OK
  70D: OK
  80D: OK
 400D: OK
 450D: OK
 500D: OK
 550D: OK
 600D: OK
 650D: OK
 700D: OK
 750D: OK
 760D: OK
 100D: OK
1000D: OK
1100D: OK
1200D: OK
 EOSM: OK

Testing EOS M3...
SD boot: StartDiskboot
Display: TurnOnDisplay

Preparing portable ROM dumper...
Testing portable ROM dumper...
   5D: skipping
  5D2: skipping
  5D3: ROM0.BIN: OK ROM1.BIN: OK
  5D4: ROM1.BIN: OK
   6D: ROM0.BIN: OK ROM1.BIN: OK
   7D: ROMs not saved
 7D2M: ROM1.BIN: OK
  40D: skipping
  50D: skipping
  60D: ROM0.BIN: OK ROM1.BIN: OK
  70D: ROM0.BIN: OK ROM1.BIN: OK
  80D: ROM1.BIN: OK
 400D: skipping
 450D: skipping
 500D: skipping
 550D: ROM0.BIN: OK ROM1.BIN: OK
 600D: ROM0.BIN: OK ROM1.BIN: OK
 650D: ROM0.BIN: OK ROM1.BIN: OK
 700D: ROM0.BIN: OK ROM1.BIN: OK
 750D: ROM1.BIN: OK
 760D: ROM1.BIN: OK
 100D: ROM0.BIN: OK ROM1.BIN: OK
1000D: skipping
1100D: ROM0.BIN: OK ROM1.BIN: OK
1200D: ROM0.BIN: OK ROM1.BIN: OK
 EOSM: ROM0.BIN: OK ROM1.BIN: OK

Restoring your SD/CF card images...

It's getting close to being usable for testing the nightly builds :)



You may wonder why I'm looking at those old VxWorks models - they do a lot of things in a slightly different way, yet still compatible with the new models. This gives extra information for understanding hardware internals (e.g. different debug strings, or just exercising different code paths in QEMU).

For example, the photo taking code on 60D uses an old-style interface for the interrupt controller, but most other code uses the new one. There is a single interrupt controller, with two interfaces (see eos_handle_intengine and eos_handle_intengine_vx in eos.c). The VxWorks models use both interfaces (very old models probably use only the old interface).

Before looking at VxWorks models, I had no idea what the old-style registers might even be doing.

Will this extra knowledge translate into new features or new models ported? I hope so, but - as usual - can't promise anything. Feel free to jump in and get your hands dirty with the code.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 08, 2016, 11:59:15 PM
i still want 1k FPS :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on November 09, 2016, 07:13:35 PM
i still want 1k FPS :)
it's better to want a replacement for cache hacks on Digic 6  :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 09, 2016, 10:24:54 PM
what do you mean?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Ant123 on November 10, 2016, 11:56:13 AM
I mean that cache debug operations are not found in Cortex-R4 Technical Reference Manual (http://infocenter.arm.com/help/topic/com.arm.doc.ddi0363g/CIHBJBAI.html)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on November 12, 2016, 03:57:50 PM
oh. didnt know. where was the discussion?
this one? http://www.magiclantern.fm/forum/index.php?topic=13746.msg170772#msg170772
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on January 18, 2017, 01:18:50 AM
More updates:

- 100D boots Canon GUI
- 70D shows date/time screen (but refuses to boot Canon GUI if date/time is patched)
- EOSM and 450D run most of the initialization and pass the DCIM dir test
- very basic (incomplete) support for Eeko and JPCORE
- photo capture emulation works on 5D3 as well (test doesn't fully pass yet though)
- memory protection registers are printed at startup
- full list here (https://bitbucket.org/hudson/magic-lantern/commits/branch/qemu).

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/100D-qemu.png) (http://a1ex.magiclantern.fm/bleeding-edge/qemu/70D-qemu.png)

The test suite now runs on ML build server as well, here (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 29, 2017, 09:58:38 PM
500D with spells from 550D has a new menu item :

(https://s23.postimg.org/l9ynkv557/qemux1.png) (https://s27.postimg.org/y44mzqjpv/qemux2.png) (https://s28.postimg.org/4i58gvjn1/qemux3.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on January 29, 2017, 11:03:34 PM
Very cool. I had this setting enabled on 550D, so it must be saved on the MPU side (probably EEPROM).

Old notes about it: https://groups.google.com/d/topic/ml-devel/ti8GyVqEZmo/discussion
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 30, 2017, 02:21:39 AM
Yes it is stored in the MPU.
Code: [Select]
Studio mode on  { 0x06, 0x05, 0x01, 0x42, 0x01, 0x00 },
Studio mode off { 0x06, 0x05, 0x01, 0x42, 0x00, 0x00 },
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on January 31, 2017, 01:01:49 AM
500D 30MB/s  :P
(https://s30.postimg.org/72oj0zpw1/qemubench.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 12, 2017, 05:59:45 PM
GUI switch progress :

(https://s12.postimg.org/72qetgkz1/qemu.png) (https://s17.postimg.org/5jxrwq0cv/qemu1.png) (https://s31.postimg.org/4qttjvtej/qemu2.png) (https://s31.postimg.org/vpxof1fvf/qemu3.png) (https://s10.postimg.org/4k248c1op/qemu4.png)

Live View :

(https://s12.postimg.org/4gfhq9j59/qemulv.png)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 14, 2017, 09:03:32 PM
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on February 16, 2017, 10:18:39 PM
i am still stunned how good the emulation works.. :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 17, 2017, 04:48:58 PM
Live View VRAM patch :

(https://s24.postimg.org/fry3mwyyt/qemu_lv.png)

Code: [Select]
[EDMAC#18] Starting transfer to 0x1B07800 from conn #4, 1440x424, flags=0x20000080
Loading photo raw data from ./500D/VRAM/PH-LV/LV-000.422...
[EDMAC#18] 610560 bytes written to 1B07800-1B9C900.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 18, 2017, 11:28:54 PM
500D menu navigation works here too, thanks Greg :D

Currently, this old camera is the only one that lets you navigate Canon menu. All other models show either a static GUI or the date/time screen.

Also committed:
- initial support for EOS M10 and M5 (for CHDK)
- an option to export function calls to IDA
- an experiment to group related MPU messages from timestamps (in the dm-spy-experiments branch)
- some auto comments regarding MPU messages
- minor fixes here and there.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on February 21, 2017, 01:46:02 AM
Formatting the virtual card works too, both from Canon and ML (of course, on 500D) :)

This is the first test in the suite that runs an unmodified ML binary. It actually downloads the current nightly build (at the time of writing) and checks both the GUI (expected screens) and the card contents (whether ML still boots after being restored).

Test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/console)

This log should also contain useful info (what commands to run), should you want to reproduce these experiments on your PC. I should probably write a guide, other than the tips from the install script.

Menu screens currently covered by the test suite:

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/menu.png)

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/format.png)

(http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/47/artifact/qemu/tests/fmtrestore.png)

There are still some nondeterministic bugs (that's why some tests are retried a few times, until sucess); those will need fixing before using QEMU as a test platform for ML builds. Still, it already starts to be useful (for example, for getting menu screenshots).

At this stage, I think the old implementation (http://www.magiclantern.fm/forum/index.php?topic=2864.msg173682#msg173682) is no longer useful, so we may start thinking about merging the QEMU branch into unified. This will remove most of those CONFIG_QEMU hacks from the source code.

BTW, if you have experience with some testing framework, and you know a nicer way to implement the current tests (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/tests/run_tests.sh), I'd be interested in hearing from you.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: Greg on February 26, 2017, 06:17:25 PM
Firmware update
ROM modified with hexeditor "DisableMainFirm" - http://magiclantern.wikia.com/wiki/Bootflags
500D 1.1.1 -> 1.1.2

(https://s27.postimg.org/9l5keyg83/qemu1.png) (https://s32.postimg.org/d6bg9eb91/qemu2.png) (https://s10.postimg.org/ao5qy413d/qemu3.png) (https://s27.postimg.org/537g4rl43/qemu4.png)

Code: [Select]
200:  5337.856 [UPD] Welcome to Update program
201:  5337.856 [UPD]   Program Ver.Slave 0.2.0
208:  5338.112 [UPD] ------------ Initialized
277:  5343.232 [UPD] CurrentVersion=1.1.1
278:  5343.232 [UPD] DS_MODELID =0x80000252
295:  5350.144 [MS] LOCK (1)
535:  8316.672 [UPD] StartFirmupProgress
569:  8319.232 [UPD] ERROR Do not read
571:  8358.912 [UPD] 0=UPD_VerifyFirmware
572:  8830.464 [UPD] 0=UPD_DecryptoFirmware
574:  8850.944 [UPD] CheckSum file=0xd960afec buffer=0xd960afec
575:  8947.968 [UPD] SAFEMODE
620: 35370.240 [UPD] ERR 1=updSpecificPartner

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 07, 2017, 04:44:59 PM
First ML change I've tested in QEMU on all unified models:

https://bitbucket.org/hudson/magic-lantern/pull-requests/796/new-method-for-getting-current-task-names/diff

Latest update adds partial 7D support (slave CPU only, without IPC).

Test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/48/console).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 24, 2017, 01:08:42 AM
A small change that unlocked Canon menu navigation on many models:

https://bitbucket.org/hudson/magic-lantern/commits/c881ba2

After some refactoring and porting the 500D MPU messages required for GUI, Canon menu navigation is now also working on...

60D, 550D, 600D, 700D, 100D, 1100D and 1200D!

Screenshots (guess the cam):

(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/100D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/1100D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/600D-menu.png)
(https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/artifact/qemu/tests/60D-format.png)

Test log (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/54/console).

All screenshots here (https://builds.magiclantern.fm/jenkins/job/QEMU-tests/54/) (click on Expand all).

This is a big breakthrough, as it effectively lets me review ML ports or code changes on cameras I don't own :)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: DeafEyeJedi on March 24, 2017, 05:34:39 AM
Holy cow, @a1ex!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: g3gg0 on March 25, 2017, 05:40:31 PM
please don't let jenkins test builds format the card. i am scared it will wipe our build server  :o

kidding :D
cool work!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: nkls on March 27, 2017, 10:12:32 PM
Nice work! Finally some running cameras to toy around with.  :D

I've managed to merge the latest changes into qemu v2.9.0-rc1 (https://bitbucket.org/niklastisk/qemu/branch/eos-2.9.0-rc1-merge). It's probably not a perfect merge, the camera gui hangs up more often than on v2.5.0 for me.

The patches to qemu code merged quite well, but someone with more knowledge of them should probably review whether they are ok/redundant in 2.9.0.

A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on March 27, 2017, 10:22:20 PM
A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.

Yay! That's what I wanted to do next, hoping it would solve the GUI lock-ups. I/O lock was another issue that I didn't know how to solve.

Quote
I've managed to merge the latest changes into qemu v2.9.0-rc1 (https://bitbucket.org/niklastisk/qemu/branch/eos-2.9.0-rc1-merge). It's probably not a perfect merge, the camera gui hangs up more often than on v2.5.0 for me.

My attempt to merge with 2.8.0 was not very successful (got stuck at making the serial port work, so ended up disabling it), but otherwise it seemed to run fairly well. Will try to integrate your changes and see how it goes.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 17, 2017, 07:05:01 PM
A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.

This solved the intermittent I/O lock-ups that I was unable to track down for a long time!!!

5D3 SD card test successfully ran 10 times in a row. Previously, I had to run it about 5 times to get one successful run...

All the menu navigation tests from test suite passed with flying colors, without any retries required!

Thanks nkls!!!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: DeafEyeJedi on April 17, 2017, 07:17:26 PM
Hell yeah and way to go @nkls!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 24, 2017, 10:18:01 PM
Added a couple of jobs on the build server:

- QEMU-dm-spy (http://builds.magiclantern.fm/jenkins/job/QEMU-dm-spy/): compiles the dm-spy-experiments branch and runs the binary in the emulator. These logs contain all Canon's debug messages (http://www.magiclantern.fm/forum/index.php?topic=2388.0), and optionally all MMIO activity. Should be useful for anyone who wants to understand the startup process.
- QEMU-boot-check (http://builds.magiclantern.fm/jenkins/job/QEMU-boot-check/): compiles ML from every camera model (from nightly) with CONFIG_QEMU=y and runs it for a few seconds in the emulator; this compilation flag enables additional debug info at startup, useful for checking the boot process (where autoexec.bin is loaded, how much memory it takes, what it does to reserve it and so on).
- QEMU-FA_CaptureTestImage (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-FA_CaptureTestImage/): compiles a minimal autoexec.bin that calls FA_CaptureTestImage (therefore taking a full-res silent picture). All the debug messages from Canon and all the MMIO activity are logged. Might be useful for understanding the still photo capture process (http://www.magiclantern.fm/forum/index.php?topic=1915.0).
- QEMU-tests (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/): that's the test suite for QEMU (presented earlier in this thread)

All these tests have HTML logs (actually just plain text with colors) and screenshots (where it's the case).

I'm also thinking to run some basic tests on the nightly, on those models with functional GUI (tests such as menu screenshots, load each module, check memory usage, run some simple Lua scripts). The emulation is not there yet for more complex tests (for example, we cannot take a CR2 picture or go to LiveView).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on April 26, 2017, 09:25:05 PM
Some updates:
- upgraded to QEMU 2.9.0, thanks nkls (still experimental, as I had quite a bit of trouble with it, so it's in a different branch for now)
- fixed another (or maybe the same?) nondeterministic lock-up (see a few posts above)
- initial support for 1300D (http://www.magiclantern.fm/forum/index.php?topic=17969) (WIP)
- options to log memory accesses (aka memory tracing); run with "-d help" to get the list

The lock-up bug was showing up very rarely on 2.5.0 after the timer refactoring from nkls (let's say about 1 out of 100 runs was bad), but after upgrading to 2.9.0 it showed up in more than half of the test runs (or about 1/5 of the test runs if the log was redirected to file). Narrowed down to interrupt controller (from a change made many months ago to support 1000D and other VxWorks models).

I'm also experimenting with logging all memory accesses made by the guest firmware, on 2.5.0. Examples for 1300D:

Code: [Select]
./run_canon_fw.sh 1300D -d romw
...
Firm Jump RAM to ROM 0xFE0C0000
K404 READY
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0x6       : 8-bit
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0x6       : 8-bit
[rom1]     at 0x0001D54C:0001D54C [0xF8000000] <- 0xE9      : 8-bit
[DMA1] Copy [0xF8E60000] -> [0x402D4000], length [0x0026BBF8], flags [0x00030001]
[DMA1] OK
     0:    20.480 [STARTUP]

Code: [Select]
./run_canon_fw.sh 1300D -d ramw,romr
...
[rom1]     at 0xFE0C000C:001000EC [0xFEA7A270] -> 0xE92D4010
[ram]      at 0xFE0C000C:001000EC [0x00001900] <- 0xE92D4010
[rom1]     at 0xFE0C009C:001000EC [0xFEA7A274] -> 0xEB000BAB
[ram]      at 0xFE0C009C:001000EC [0x00001904] <- 0xEB000BAB
...

I know I'm (https://github.com/aclements/mtrace) almost (https://github.com/moyix/panda) certainly (https://github.com/panda-re/panda) reinventing (https://projects.gso.ac.upc.edu/projects/qemu-dbi/wiki) the (https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-qemu.html) wheel (http://web.eece.maine.edu/~vweaver/projects/qemu-trace/), but I had only limited success with these modified versions:
- mtrace uses a very very old QEMU
- panda 1.0 uses QEMU 1.0.1, examples work, lots of nice tools, but appears deprecated (shouldn't be hard to roll back our patches to the older version)
- panda 2.0 uses a very recent QEMU, but could not run any ARM examples (segmentation fault). Also, most of the cool tools from panda 1.0 are not ported yet.
- QEMU-DBI is "being upstreamed into QEMU", and a large part of it is already in 2.9.0 (the main reason I've upgraded). TODO: figure out how to use it...
- QEMU-CHERI is a mod for MIPS that also traces memory and instructions (nice to see how it works)
- the last one, QEMU-trace, is a very simple patch that showed me where to place the hooks in the QEMU codebase (also with this message (https://lists.nongnu.org/archive/html/qemu-devel/2014-04/msg02875.html) and this thread (https://lists.linaro.org/pipermail/linaro-dev/2012-March/010887.html) from mailing lists).

So, yeah, I still want to use the state-of-art method for logging memory accesses, just need to figure out how. Until then, my monkey-patched method appears to work pretty well (can rebuild the memory contents from the trace) and has very little overhead as long as I'm not printing each access to the console.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 01, 2017, 01:49:29 AM
Currently experimenting with a binary instrumentation tool similar to valgrind's memcheck (though a lot more primitive, as I'm reinventing the wheel again). It's written on top of the memory tracing (which is already committed) and a similar hook calling every time a new code block (TranslationBlock) is executed.

Quick example (don't click me):
Code: [Select]
    uint32_t * p = malloc(1234);
    qprintf("p=%x\n", p);
    p[100] = p[200] + 1;            /* use of uninitialized value (read) */
    free(p);
    qprintf("p freed\n");
    p[20] = p[30] + 1;              /* use after free (both read and write) */
    qprintf("test complete\n");

From emulation log:
Code: [Select]
p=fb440
[run_test:589c8:589c8] fb760 uninitialized
p freed
[run_test:589e4:589e4] fb4b8 read after free (0)
[run_test:589f0:589e4] fb490 written after free (1)
test complete

The current state is just a very rough proof of concept, but it already found a bunch of null pointer, uninitialized memory (https://bitbucket.org/hudson/magic-lantern/commits/branch/memcheck-fixes) and  thread safety (https://bitbucket.org/hudson/magic-lantern/commits/branch/thread-safety) bugs :D
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: eduperez on May 01, 2017, 10:21:10 PM
The current state is just a very rough proof of concept, but it already found a bunch of null pointer, uninitialized memory (https://bitbucket.org/hudson/magic-lantern/commits/branch/memcheck-fixes) and  thread safety (https://bitbucket.org/hudson/magic-lantern/commits/branch/thread-safety) bugs :D

For a moment I though you where talking about bugs in Canon's code...  :o
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 01, 2017, 10:29:41 PM
For your viewing pleasure:

http://builds.magiclantern.fm/jenkins/job/QEMU-memcheck/QEMU_memcheck_logs/500D.111-memchk.log.html

The analysis only has 500D stubs for now (though it's easy to add for other models).

The first bunch of TCM warnings can be ignored (these are the initialization sequence). The remaining TCM accesses from Canon tasks are probably bugs in Canon firmware, or in my emulation.

Here's an obvious one, if you look it up in the disassembly:
Code: [Select]
[FileMgr:ff3b5d38:ff3b5d38] address 0 written to TCM (12)
[FileMgr:ff3b5d48:ff3b5d48] address 0 written to TCM (2000)
[FileMgr:ff3b5d58:ff3b5d58] address 0 written to TCM (100)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 02, 2017, 09:08:07 PM
More updates:

- 50D boots the GUI! (figured it out from this log (http://www.magiclantern.fm/forum/index.php?topic=9852.msg184191#msg184191))
- 5D2 is very close (http://www.magiclantern.fm/forum/index.php?topic=11205.msg184202#msg184202)
- faster emulation (test suite about twice as fast)
- code coverage report (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-coverage/QEMU_code_coverage_(lcov)/)

(https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/98/artifact/qemu/tests/50D-menu.png)

Self-testing log (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/98/console)
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: kennetrunner on May 11, 2017, 05:07:41 PM
@a1ex   I have qemu 2.9, and tried to install the patch contrib/qemu/qemu-2.5.0.patch   - which fails :-(
Do you have an updated patch file ?

.. ken
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 11, 2017, 09:58:20 PM
Yes, on the qemu-2.9.0 branch.

However, some parts of the test suite fail in 2.9.0 (but are OK on 2.5.0), and 2.5.0 has a few more features on the EOS side (in particular, the memory tracing and related tools). Syncing them shouldn't be very difficult.

2.9.0 can be interesting for EOS M5, as it might emulate multi-core CPUs a little better (note: the self-test fails for this model) and for porting the code to other QEMU variants (e.g. Panda 2.0 for its binary instrumentation plugins, or Xilinx QEMU*) if you want to look into UHS), but for the moment I've switched back to 2.5.0 (simply because it works and I'm used to it).

*) Xilinx QEMU is currently based on 2.6.0, so the patch for it is probably something in-between.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: kennetrunner on May 12, 2017, 02:48:58 PM
So, after a ton of hurdles I shelved my own attempts at getting qemu 2.9 running...
Instead I downloaded the vbox image from http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989 (http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989) and compiled qemu 2.5 and it **seemed** to complete successfully...

However, now when I run
Code: [Select]
./run_canon_fw.sh 550D I get
Code: [Select]
qemu-system-arm: unsupported machine type
When I list the supported machine types, the only Canon one to show up is canon-a1100

Seems like I'm missing a step, somewhere ?... any pointers ?

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 12, 2017, 03:50:03 PM
The vanilla QEMU does not include our modifications.

See the install log (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/consoleFull) from the build server for reference.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 17, 2017, 03:08:44 PM
I just started into trying to get the qemu branch working on macOS Sierra 10.12.4.

The ML repository needs to be in the home (user) directory or it won't work. The script in magic-lantern/contrib/qemu/install.sh seems to run fine up until this point:

Code: [Select]
Next steps:
===========

1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j

Running ../configure_eos.sh (without the cpuinfo) or "./configure --target-list=arm-softmmu --disable-docs --enable-sdl" results in:

Code: [Select]
ERROR: Cocoa and SDL UIs cannot both be enabled at once
QEMU can also be installed via Homebrew but of course it doesn't include the Magic Lantern modifications.

Maybe it is a matter of figuring out how to configure the "cpuinfo" for the Mac?
Code: [Select]
sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz

Though it is probably much more than just that. Maybe it is the clang compiler? Any hints?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 17, 2017, 03:11:35 PM
Comment out the affected line; plain make is fine.

Or, try something like this:

Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Compiling without SDL should be fine (it's used that way on the build server, where it runs without GUI).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 17, 2017, 03:52:07 PM
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"
grep: /proc/cpuinfo: No such file or directory
   make -j8

Ah ha, so the processor on this machine is:
Code: [Select]
sysctl -n hw.ncpu
8

So I tried:
Code: [Select]
make -j8
Please call configure before running make!
make: *** No rule to make target `trace/generated-events.h', needed by `Makefile'.  Stop.
make: *** Waiting for unfinished jobs....
make: *** [config-host.mak] Error 1


Changing configure_eos.sh so it uses gcc-5 instead of clang gives the warning noted in the script but it still comes up with:
Code: [Select]
C++ compiler c++ does not work with C compiler gcc-5 --std=gnu99
Disabling C++ specific optional code

ERROR: Cocoa and SDL UIs cannot both be enabled at once

It looks like the issue is setting this up so the configure script will select only the SDL UI. Need to dig into it a bit more.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 17, 2017, 04:00:25 PM
To specify the C++ compiler, try this:

Code: [Select]
CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

It almost compiles with clang, except for a tiny function which you can safely comment out, and some warnings. Will look into them.

For SDL, try --disable-sdl.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 17, 2017, 04:48:09 PM
Almost there. Are you seeing these same errors?
Code: [Select]
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:196:9: error:
      function definition is not allowed here
        {
        ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:200:16: error: use
      of undeclared identifier 'close_idc'; did you mean 'closedir'?
        atexit(close_idc);
               ^~~~~~~~~
               closedir
/usr/include/dirent.h:102:5: note: 'closedir' declared here
int closedir(DIR *) __DARWIN_ALIAS(closedir);
    ^
2 errors generated.
make[1]: *** [hw/arm/../eos/dbi/logging.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:91:20: warning:
      unused function 'set_uninitialized' [-Wunused-function]
static inline void set_uninitialized(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:99:20: warning:
      unused function 'set_freed' [-Wunused-function]
static inline void set_freed(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:107:20: warning:
      unused function 'clr_freed' [-Wunused-function]
static inline void clr_freed(uint32_t addr)
                   ^
3 warnings generated.
1 warning generated.
make: *** [subdir-arm-softmmu] Error 2

Hint on where to find the tiny function that can be commented out?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 17, 2017, 05:01:06 PM
Yes, exactly there.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 17, 2017, 05:34:25 PM
Tried something different. Since cr2hdr doesn't compile on clang we need to install a different compiler on the Mac.
Code: [Select]
CC="gcc-5" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

Then ran "make -j8" like before and it compiled. There were some other errors but it finished compiling.

Got to run off to work now but hope to get back to this soon.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 18, 2017, 12:51:49 AM
These are the errors I got running the gcc-5 compiler from the Homebrew distribution on the Mac:
Code: [Select]
In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
  CC    crypto/cipher.o
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));

It did finish compiling so did it work? Followed instructions with the rom dumps and mounted the sd card image and this is what happened:
Code: [Select]
./run_canon_fw.sh EOSM
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
CHK version_gen.h
  LINK  qemu-ga
  LINK  ivshmem-server
  CC    block/raw-posix.o
In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
block/raw-posix.c: In function 'hdev_open':
block/raw-posix.c:2129:23: warning: variable 'kernResult' set but not used [-Wunused-but-set-variable]
         kern_return_t kernResult;
                       ^
make: *** [block/raw-posix.o] Error 1

Looks like the same error messages.

Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 18, 2017, 12:57:19 AM
Searching this error message gives http://stackoverflow.com/questions/41143981/macos-sierra-corefoundation-error-while-compiling-wxwidgets-for-simspark

To me, it looks like qemu didn't finish compiling.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 22, 2017, 12:11:20 AM
QEMU on Mac -- Yay!

(https://c1.staticflickr.com/5/4169/34767851946_d027c59a4f_z.jpg)

This might be a real hack, here's what I did. First of all the CoreFoundation.framework issues affect the Homebrew gcc-5 compiler so I went back to the macos clang.

~/qemu/configure_eos.sh
Code: [Select]
CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

The compilation errors we had with clang were because of this section of code, so I commented it out.

~/qemu/qemu-2.5.0/hw/eos/dbi/logging.c line#194
Code: [Select]
        /* QEMU is usually closed with CTRL-C, so call this when finished */
//        void close_idc(void)
//        {
//            fprintf(idc, "}\n");
//            fclose(idc);
//        }
//        atexit(close_idc);

Loaded up the EOSM ROM0.BIN, ROM1.BIN and had to create an SFDATA.BIN using the sf_dump module and placed them in ~/qemu/EOSM. Mounted the sd.img and ran:

Code: [Select]
./run_canon_fw.sh EOSM
Now just because it launches doesn't mean that it is working properly but I wanted to share my progress.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 22, 2017, 11:45:27 AM
Nice. This is the cocoa interface, right? Do you have menus that allow you to switch between serial console, VGA and so on?

BTW, after this commit (https://bitbucket.org/hudson/magic-lantern/commits/970d60759f1ea6251925c04874b854208885cc4a) it should install cleanly on clang and/or Mac.

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on May 23, 2017, 02:51:37 PM
Speaking for all of us Mac users -- Thanks!

Here are a few things Mac users who want to get this working should watch out for.

install.sh prints some instructions at the end of the process. On osx it prints out the grep error instead of keeping silent.
Code: [Select]
1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j8

It is caused by this line:

install.sh
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Switching the positions around eliminated the grep error on osx and created a sysctl command not found on Linux so I tried this to suppress the error messages and it worked:
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"


This instruction doesn't work for osx:
Code: [Select]
3) Mount the included SD (or CF) image (you may use mount.sh)mount.sh calls "kpartx" which isn't available for osx. Mounting sd.img can be done by simply double clicking the sd.img icon but I'm not sure if that worked properly because when running qemu it shows this:
Code: [Select]
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
SD LOAD OK.
Open file for read : AUTOEXEC.BIN
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error

I looked for an alternate for kpartx and maybe hdiutil will work? I haven't figured out if there is some special way it needs to be invoked to mount the img files as loopback devices but either of these seem to work:
Code: [Select]
hdiutil mount sd.img
or
hdiutil attach sd.img


Back to install.sh, at the end it is supposed to list some camera models but on osx it displays this:

Code: [Select]
   Note: Canon GUI emulation (menu navigation, no LiveView) only works on
   usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]

That problem is partially because the install script calls tests/run_tests.sh. Here is the output of running that script on osx:
https://pastebin.com/0T5GLRh5

Not sure if any of this would prevent qemu from running properly on osx. I'm just starting on this. Running the minimal autoexc.bin works on every ROM dump I've got but I haven't been able to go much beyond that.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 24, 2017, 10:05:48 AM
Code: [Select]
SD: CMD12 in a wrong state
[SDIO] Error

That's fine, I get those too. I'm not sure if this indicates an emulation bug / incomplete model, or it's just how the (simplified) SD driver used in bootloader is supposed to behave (note: CMD12 is STOP_TRANSMISSION).

In the main firmware, you'll get a similar error about CMD1; this one is OK, as it appears to be the way Canon code probes for MMC cards. Regular SD cards are probably not supposed to reply to CMD1 outside the SPI transfer mode (at least that's my understanding), so the SD emulation backend prints some messages. The full conversation can be watched with -d sdcf (or -d sdcf,io for more details) and cross-checked with the SD specification (https://www.sdcard.org/downloads/pls/).

If the reader is familiar with SD protocol, I'd welcome any insights (in particular, for the UHS-I initialization sequence (http://www.magiclantern.fm/forum/index.php?topic=12862.0)).


Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?

To avoid corrupting the data on the SD image, it's best to prevent starting the emulation if the image is mounted by the user (as there will be two processes wanting to write on the same card image, without knowing about each other). This is done on Linux by checking /proc/mounts (losetup is used with "loopback" devices - that is, when mounting an image as a filesystem). Don't know how this works on Mac, and don't know whether my method is portable across other Linux distributions either.



The other issues appear to be (more or less) just annoyances (as they don't print what the user expects to see) and probably easy to fix.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on May 28, 2017, 03:29:02 PM
Some recent additions (some of them were covered in other threads):

* Cleaned up the self-test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console) to include installation instructions (the self-test begins with reinstalling QEMU from scratch, so you can also use it to check the expected output of the commands etc).

* Call stack reconstruction (inspired from Panda's callstack_instr (https://github.com/panda-re/panda/blob/master/panda/plugins/callstack_instr/callstack_instr.cpp), but heavily customized for EOS firmware). This makes the call stack available to other analysis tools (e.g. when memcheck wants to print an error). Enable with -d callstack . Works on DIGIC 4 to 6 and Eeko; almost there on DIGIC 2 and 3; for best results, .current_task_addr should also be defined (because each task has its own stack).

* Call/return trace. Every function call and return (that could be identified automatically) is logged on the console, including arguments and return values (enable with -d calls ). Works best on DIGIC 4 to 6 and Eeko, almost there on DIGIC 2 and 3. Interrupts are also handled (though it still has some trouble with corner cases). Example for Eeko (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/QEMU_self-testing_results/tests/5D3eeko/calls-fint-raw.log.html).

Tip: you can load these indented logs in your text editor, configure it for Python source code, and you've got collapse buttons :)

* Task switches. Make sure you have .current_task_addr defined in model_list.c and enable with -d tasks . Works on DIGIC 2 to 7 and Eeko. Example for EOS M2 (http://www.magiclantern.fm/forum/index.php?topic=15895.msg185280#msg185280).

* Memory blocks copied from ROM to RAM can be identified automatically (-d romcpy). They are listed on the self-test log (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console), and they should be a big time-saver when setting up the disassembler (usually you'll need to know what are the blocks worth disassembling, besides the ROM). Works on DIGIC 2 to 7. Example for EOS M5 (can be cross-checked with values found manually (https://chdk.setepontos.com/index.php?topic=13014.msg131205#msg131205)):
Code: [Select]
[ROMCPY] 0xE001AF2C -> 0xDF020000 size 0x3C0      at 0xE0005AA8
[ROMCPY] 0xE001B2E4 -> 0x4000     size 0xF1C      at 0xE000492C
[ROMCPY] 0xE115CF88 -> 0x8000     size 0x6054C    at 0xE002003C
[ROMCPY] 0xE11BD4D4 -> 0x1900000  size 0x1444     at 0xE0020060
[ROMCPY] 0xE11BE918 -> 0xDFFC4900 size 0x152A0    at 0xE0020084

* Symbol names from .elf files. QEMU has this feature built in, but had to customize it a bit (for example, it didn't recognize our stubs as function calls, because our build system doesn't mark them as such). Usage (requires bash):
Code: [Select]
. ./export_ml_syms.sh 550D.109
./run_canon_fw.sh 550D.109,firmware="boot=1" ...
...
Task switch to init:ff010470                                                     at [init:ff0164c4:ff076f18]
call 0x8B010 my_init_task(0, 8b010, 19980218, 19980218)                          at [init:ff0771dc:0]
 call 0xFF018D1C init_task(0, 8b010, 19980218, 19980218)                         at [init:8b014:ff0771e0] (my_init_task)
  call 0xFF0108FC(0, 8b010, 19980218, 19980218)                                  at [init:ff018d20:8b018]

In other words, if you want the log to use the function names you have annotated in the disassembly, just add them to stubs.S and compile ML; the above script will load them. If you don't want to run autoexec.bin, you can probably just load stubs.o (which is an elf file), or you can probably create a stubs file just for analysis (one that will not be included in ML, because it will usually have a lot more names than you actually want to call).

There's a lot of room for improvement on this one (for example, right now function names are only listed in the call/return trace). Adding this info in more places is generally an easy coding task - find what parts of the log you would like to have function names rather than just raw addresses, and change the output strings accordingly). Automating symbol name export from your favorite disassembler should also be easy (and in many cases, the code is available online from other open source tools; just needs to be integrated here).

* Memcheck warnings closer to valgrind's (e.g. show where the affected memory block was allocated or freed).

* Basic ABI checking (e.g. R4-R11 and SP should be restored after function call). This is done while creating the call stack (-d callstack).

* The auto-generated IDC file should be much more accurate (because we finally have a sane call/return trace).

A walkthrough for some of these features can be found in the EOS M2 (http://www.magiclantern.fm/forum/index.php?topic=15895.msg185103#msg185103) topic, where, with dfort's help, I'd like to show in detail how to port Magic Lantern on a new camera from scratch.

Tip: to find all these analysis options, run QEMU with -d help. The ones interesting for us are:
Code: [Select]
int        show interrupts/exceptions in short format
exec       show trace before each executed TB (lots of logs)
nochain    do not chain compiled TBs so that "exec" and "cpu" show complete traces
io         EOS: log low-level I/O activity
mpu        EOS: log low-level MPU activity
sflash     EOS: log low-level serial flash activity
sdcf       EOS: log low-level SD/CF activity
uart       EOS: log low-level UART activity
ram        EOS: log all RAM reads and writes
rom        EOS: log all ROM reads and writes
ramr       EOS: log all RAM reads
romr       EOS: log all ROM reads
ramw       EOS: log all RAM writes
romw       EOS: log all ROM writes
ram_dbg    EOS: self-test for the RAM logging routines
callstack  EOS: reconstruct call stack (implies nochain,singlestep)
calls      EOS: log function calls (implies callstack,ramr,nochain,singlestep)
idc        EOS: export called functions to IDA (implies callstack,nochain,singlestep)
tasks      EOS: log task switches (implies callstack,nochain,singlestep)
romcpy     EOS: find memory blocks copied from ROM to RAM
memchk     EOS: check memory usage (malloc/free, uninitialized values)
v         
verbose    EOS: very detailed debug messages

Happy reversing!
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on June 03, 2017, 06:26:55 PM
Just sharing some of my recent QEMU adventures on Mac -- and Cygwin.

It seems to be working fine on Mac but there are several issues with run_tests.sh. It doesn't really affect the compiling but it does show a message that makes it look like something went terribly wrong. I haven't worked it all out but so far most of the issues have to so with missing programs on the Mac.

The bash version that comes with the Mac is 3.2.57 but the command "declare -A" requires bash 4. Installing bash via Homebrew gives you version 4.4.12 but the script uses "#!/bin/bash" which points to the old version. Changing the script's shebang fixes that and this should be portable:
Code: [Select]
#!/usr/bin/env bash
Next problem is that the Mac doesn't have "ansi2txt" and there doesn't seem to be one on Homebrew either. I installed it from the source (https://sourceforge.net/projects/ansi2txt/). Because the Mac keeps you from installing programs in /bin I changed the Makefile like this:
Code: [Select]
BINPATH = /usr/local/bin
MANPATH = /usr/local/share/man/man1

Yet another missing program is "timeout" and this can be installed via "brew install coreutils" but then you get gtimeout so to get the default names I had to (just run these commands to update the bash preference, no need to put it in the Makefile):
Code: [Select]
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
MANPATH="/usr/local/opt/coreutils/libexec/gnuman:$MANPATH"

Almost there, there is a grep error that I mentioned before:
Code: [Select]
   usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
 [-e pattern] [-f file] [--binary-files=value] [--color=when]
 [--context[=num]] [--directories=action] [--label] [--line-buffered]
 [--null] [pattern] [file ...]

At this point I ran out of time but will revisit this later.

I also tried running QEMU on Cygwin. I didn't compile though "install.sh" did finish successfully. If you set up a Cygwin environment using my tutorial (http://www.magiclantern.fm/forum/index.php?topic=15894.msg#new) you'll need to install a few extra packages to get through the actual compiling.
Code: [Select]
git
patch
gcc-g++

git needs to be set up:
Code: [Select]
  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"

The qemu-2.5.0 configure script supports Cygwin but it has a depreciated option:
Code: [Select]
CYGWIN*)
  mingw32="yes"
  QEMU_CFLAGS="-mno-cygwin $QEMU_CFLAGS"
  audio_possible_drivers="sdl"
  audio_drv_list="sdl"
;;

Simply delete this line:
Code: [Select]
  QEMU_CFLAGS="-mno-cygwin $QEMU_CFLAGS"

I still didn't get it to compile and am stuck with this error:
Code: [Select]
$ ../configure_eos.sh
Setting up QEMU on CYGWIN_NT-6.0...
Using gcc --std=gnu99 / g++ with -Wno-error=deprecated-declarations
Options:

ERROR: glib-2.22 gthread-2.0 is required to compile QEMU

Not sure which Cygwin packages are needed to satisfy those dependencies.

I opened a pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/833/preliminary-support-for-eosm2-and-ml-qemu/diff) that incorporates a few of these suggestions along with preliminary support for the EOSM2 in case any of this is useful.

[EDIT] That pull request had build system changes mixed with EOSM2 support. Best split them in separate pull requests. In the meantime I discovered that, "Builds with the normal Cygwin compiler are not supported. (http://wiki.qemu.org/Hosts/W32#Native_builds_with_Cygwin)" Even following those instructions I still haven't been able to build QEMU in Cygwin. In fact I've gotten just as far using the native (host) compiler. In both cases it ended like this:

Code: [Select]
/usr/include/w32api/winsock2.h:995:34: error: conflicting types for ‘select’
   WINSOCK_API_LINKAGE int WSAAPI select(int nfds,fd_set *readfds,fd_set *writefds,fd_set *exceptfds,const PTIMEVAL timeout);
                                  ^
In file included from /usr/include/sys/types.h:68:0,
                 from /usr/include/time.h:28,
                 from /usr/include/glib-2.0/glib/gtypes.h:35,
                 from /usr/include/glib-2.0/glib/galloca.h:32,
                 from /usr/include/glib-2.0/glib.h:30,
                 from qga/commands-win32.c:14:
/usr/include/sys/select.h:73:5: note: previous declaration of ‘select’ was here
 int select __P ((int __n, fd_set *__readfds, fd_set *__writefds,
     ^
qga/commands-win32.c: In function ‘qmp_guest_file_open’:

[EDIT] Put up pull requests for build system tweaks (https://bitbucket.org/hudson/magic-lantern/pull-requests/834/qemu-build-tweaks-1/diff) and EOSM2 (https://bitbucket.org/hudson/magic-lantern/pull-requests/835/eosm2-preliminary-setup/diff).
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on June 04, 2017, 06:19:53 PM
Found out why the Mac is having problems with "install.sh" -- it is a well known issue with BSD grep that is installed on OSX not being able to use the "-P" option. Replacing it with GNU grep "fixes" the issue.

Another solution would be to rewrite the script so it doesn't use the "grep -P" option which might not be a bad idea because that is a "highly experimental" option:
Code: [Select]
       -P, --perl-regexp
              Interpret  the  pattern  as  a  Perl-compatible  regular  expression  (PCRE).  This is highly
              experimental and grep -P may warn of unimplemented features.

To replace grep on the Mac with Homebrew:
Code: [Select]
brew tap homebrew/core; brew install grep --with-default-names

Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dmilligan on June 04, 2017, 08:57:34 PM
IIRC, if you can't use -P, you loose the vast majority of powerful parts of regex syntax (depending on what you're trying to do, it might not be possible to use grep at all, and so you would need to take on an extra dependency anyway). And I bet that statement about "highly experimental" was put in decades ago and never taken out. These more advanced parts of regular expression syntax are now quite standard.

There's also this: http://jlebar.com/2012/11/28/GNU_grep_is_10x_faster_than_Mac_grep.html
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 06, 2017, 11:56:47 PM
Some more updates:

- 5D3 menu navigation works*) (1.1.3 only)
- call/return trace: function arguments that look like strings are now identified
- experiment: include more debug info in ML elf files +)
- a few other minor tricks

*) To make this work, I had to port the GDB patch for date/time (easy to find in */debugmsg.gdb). On most (all?) other models, this patch is not really needed, because the date/time dialog is easy to bypass (just click OK or Cancel). Not so on 5D3; figure out why.

The 70D also seems to react to this trick, although menu navigation locks up very quickly on this model...

(http://builds.magiclantern.fm/jenkins/view/Experiments/job/QEMU-tests/ws/qemu/tests/5D3/menu10.png)

+) This enables translation from binary address to source code line, for example:
Code: [Select]
# ML must be compiled from the qemu branch (for now)
hg up qemu -C
cd magic-lantern/platform/500D.111
make clean; make

# pick some address above RESTARTSTART
eu-addr2line -s -S -e magiclantern 0x4F000
meminfo_display+0x2bc
mem.c:1149

More to come.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 07, 2017, 01:38:22 AM
Some more:

- parameterized MPU spells *)
- description of known MPU messages
- log timestamps are now used to help deciding the relationships between MPU spells +)
- build fixes for Mac (dfort)

*) Many MPU messages represent properties (for example: PROP_ICU_UILOCK is 06 05 04 01 nn 00, where "nn" is the value of the property). So, rather than including a "spell" for every possible value, it makes sense to define a template: in this case, whenever our emulated MPU receives 06 05 04 01 something 00, it should reply the same message back. To define such behavior, you may use ARG0 to ARG3 to define parameters; for example:
Code: [Select]
    { { 0x06, 0x05, 0x04, 0x01, ARG0, 0x00 }, {
        { 0x06, 0x05, 0x04, 0x01, ARG0, 0x00 },
        { 0 } } },

That's a trivial example, where the reply simply echoes the same message. Not all messages are the same; for example, for changing the GUI state:
Code: [Select]
    { { 0x06, 0x05, 0x04, 0x00, ARG0, 0x00 }, {                 /* NotifyGUIEvent(ARG0) */
        { 0x06, 0x05, 0x04, 0x00, ARG0, 0x01 },                 /* PROP_GUI_STATE(ARG0) */
        { 0 } } },

In other words, whenever it receives 06 05 04 00 something 00 (which is a request for changing GUI state), it replies back with 06 05 04 00 that_something 01 (which probably means there's green light for changing GUI state, or something like that).

+) Recent DebugMsg logs include a timestamp; this is now used to decide the relationship between MPU spells (previously, this was decided solely based on the sequence of the mpu_send/recv calls, with some exceptions for e.g. button messages). Only messages received shortly after a mpu_send call are now considered replies; messages that arrive later are assumed to be external inputs and commented out. Experimental; only 5D3, 5D2 and 50D have this enabled currently.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on June 07, 2017, 09:17:29 PM
- build fixes for Mac (dfort)

Thanks! I'm still working on getting the EOSM2 setup by following your QEMU tutorial (http://www.magiclantern.fm/forum/index.php?topic=15895.msg185089#msg185089) on that port -- you aren't going to let me get off easy are you?  :D

QEMU on Mac seems to be working. The most success I've been having is with the 550D (I don't have 500D ROMs):

(https://c1.staticflickr.com/5/4216/34352621253_0af2cae91d_z.jpg)

I looked into mount.sh and got it working on OS X (kpartx isn't available on Mac):

Code: [Select]
#!/usr/bin/env bash

echo "This will mount sd.img and cf.img as a loopback device."
echo "Please enter your password (of course, after reviewing what this script does)."

if [ $(uname -s) == "Darwin" ]; then
    sudo hdiutil attach sd.img
    sudo hdiutil attach cf.img
    echo "Done."
    echo "To remove the device mappings, run:"
    echo '   sudo hdiutil detach "/Volumes/EOS_DIGITAL"'
    echo '   sudo hdiutil detach "/Volumes/EOS_DIGITAL 1"'
else
    sudo kpartx -av sd.img
    sudo kpartx -av cf.img
    echo "Done."
    echo "To remove the device mappings, run:"
    echo "   sudo kpartx -dv sd.img"
    echo "   sudo kpartx -dv cf.img"
fi

I would submit a pull request for this but it seems that on the Mac it isn't necessary to mount as root. I also realized that when running QEMU via ./run_canon_fw.sh the first thing that script does is to make sure sd.img and cd.img aren't mounted. That script uses losetup which is yet another one that isn't available on the Mac. It displays an error but that doesn't seem to cause any problems. I've run QEMU with and without sd.img mounted and it doesn't seem to make any difference so maybe we just leave this alone for now?
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 07, 2017, 09:22:47 PM
I've run QEMU with and without sd.img mounted and it doesn't seem to make any difference

It will make a difference as soon as both you and the camera code will write something to the SD image during an emulation session. I did that by mistake a while ago and ended up with an unusable image (had to re-create it, copy ML again etc).

Additionally, at least under Linux, the data you write to some mounted directory is not flushed right away to disk. So, if you copy autoexec.bin or other files on the partition, you may find out they are not fully visible to the guest OS.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: dfort on June 08, 2017, 01:07:38 AM
I see, so we should check if /Volumes/EOS_DIGITAL* exists on OS X.
Title: Re: How to run Magic Lantern into QEMU?!...
Post by: a1ex on June 11, 2017, 12:34:21 AM
More updates:

* Monitor console available by default as a UNIX socket; that means, during emulation you can interact with it with netcat (for quick commands or from a script), or with socat (for interactive console):
Code: [Select]
echo "log io" | nc -U qemu.monitor
socat - UNIX-CONNECT:qemu.monitor

* Log DebugMsg calls without GDB (very fast; credits go to @nkls - I've used a modified version of his initial DebugMsg hook). To use it on plain Canon firmware (any shell):
Code: [Select]
env QEMU_EOS_DEBUGMSG=0x5b90 ./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg
or, with ML loaded (requires bash):
Code: [Select]
. ./export_ml_syms.sh 5D3.113
./run_canon_fw.sh 5D3,firmware="boot=1" -d debugmsg

* Verbose stack trace (to see where each message is coming from), for both -d debugmsg and GDB scripts (DIGIC 4-6). Example for the former:
Code: [Select]
env QEMU_EOS_DEBUGMSG=0x5b90 ./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg,callstack,v

Current stack: [14ff80-14ef80] sp=14fed8                                         at [FileMgr:5b90:ff0f9684]
0x17B60(51ec48 &"TaskClass", 17b60, 19980218, 19980218)                          at [FileMgr:de48:14ff78] (pc:sp)
 0xFF11B818(51ea28 &"FileMgr", 6, 0, 2)                                          at [FileMgr:17bbc:14ff50] (pc:sp)
  0x178B4(51ec1c &"StateObject", 51ea28 &"FileMgr", 6, 0)                        at [FileMgr:ff11b844:14ff38] (pc:sp)
   0x178EC(51ec1c &"StateObject", 51ea28 &"FileMgr", 6, 0)                       at [FileMgr:178e4:14ff28] (pc:sp)
    0xFF2C8F5C(51ea28 &"FileMgr", 0, 2, ff2c8f5c)                                at [FileMgr:1796c:14ff08] (pc:sp)
     0xFF0C5194(10, 0, 24, ff0c5194)                                             at [FileMgr:ff2c9050:14fef0] (pc:sp)
      0x5B90(0, 3, ff0f9784 "[SEQ] NotifyComplete (Cur = %d, %#x, Flag = %#x)", 4)
                                                                                 at [FileMgr:ff0f9680:14fed8] (pc:sp)
[     FileMgr:ff0f9680 ] (00:03) [SEQ] NotifyComplete (Cur = 4, 0x10, Flag = 0x10)

This tool is very powerful - rather than hunting for several minutes/hours to see where some error message might be coming from, you now get the answer in seconds. TODO: example on finding and fixing some error.

* Thorough consistency check to make sure the stack trace gives the same information as if you would follow the call/return trace manually.

More to come (regarding 1300D, digic 6, memory checking, automatic testing of ML builds). Most of these were written some time ago, but it takes a while to integrate everything and make sure they pass the test suite. Though slow, this approach does catch a lot of bugs very early, and I hope to have soon the tools to use a similar development approach for the main ML codebase.