How to run Magic Lantern into QEMU?!...

[a1ex]

Playing with different toolchains on Ubuntu (Xenial 64-bit):

1) gdb-arm-none-eabi:i386 and gcc-arm-none-eabi from Ubuntu repo (gcc 4.9 64-bit, 32-bit gdb): animation (3MB)
2) 32-bit gcc-arm-embedded (gcc-arm-none-eabi-5_4-2016q3): animation (3MB)
3) gdb-arm-none-eabi and gcc-arm-none-eabi from Ubuntu repo (gcc 4.9, 64-bit): animation (3MB) - using 60D without GDB
4) gcc-arm-embedded from ppa:team-gcc-arm-embedded/ppa (gcc 6.x, 64-bit): TODO (need to use a different camera - 5D3 requires 32-bit GDB)

Scripts used (should you want to re-create the above scenario, maybe on another OS):
qemu-demo-xenial1.sh
qemu-demo-xenial2.sh
qemu-demo-xenial3.sh
anim.py (to render the animation)

[a1ex]

Some good news for those affected by the 64-bit GDB bug:

- Found out why 5D3 GUI wasn't coming up without patching the date/time: it was waiting for... PROP_MPU_GPS (06 04 03 54 00 00 from MPU). After this change, 5D3 GUI booted without GDB (and the date/time dialog could be bypassed by clicking OK)!
- Even better - g3gg0 figured out how to emulate the real-time clock! This change superseded a bunch of GDB scripts - no more need to patch the date/time dialog!
- Exception: 5D2 and 50D appear to use a different RTC chip edit: g3gg0 just solved it!
- EOS M boots Canon GUI (with the same limitations as EOS M2)
- EOS M and M2 have the date/time dialog left enabled on purpose (to prevent the camera from entering LiveView, which is not emulated)

That means:
- The good news: to boot the GUI, you no longer need GDB for most models (exceptions: M and M2)
- The date/time dialog at startup is gone! (exceptions: 5D2, 50D, M and M2)
- More Canon menus navigable without locking up on DIGIC 5 models (because of that GPS property...)
- The bad news: if you want to do actual debugging in GDB, you will need a 32-bit arm-none-eabi-gdb.

Current status:
- Models able to run the GUI and navigate Canon menu: 16 (most DIGIC 4 and 5, some DIGIC 3).
- Models with major GUI issues: 70D, 5D3 1.2.3.
- Models unable to run the GUI: 6D (help needed), 7D (hard), all DIGIC 6 models, most VxWorks models.

Final note: if you have QEMU already set up, I recommend installing the new one from scratch - the install script will not delete old patches.gdb files. Or just delete these files manually.

[kichetof]

Guys, you're awesome!! 
Canon 5D3 113 on MacOS High Sierra! Happy!



@a1ex no module menu, need to enable Debug -> Modules debug -> Load modules after crash; reboot and module menu appear

[a1ex]

@kichetof: "Common issues and workarounds" in the README (or watch my animations)

[DeafEyeJedi]

Looking great out there @kichetof! Could use some of your beers.

Some good news for those affected by the 64-bit GDB bug:

Great progress @a1ex!

Final note: if you have QEMU already set up, I recommend installing the new one from scratch - the install script will not delete old patches.gdb files. Or just delete these files manually.

Could you pleaase shed some light on this? It seems I am pretty close to getting the QEMU to emulate on OS X 10.13 (I can see the black window popping up briefly before it disappears) with this message below:

Code: [Select]

./run_canon_fw.sh 100D
DebugMsg=0x4A74 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
./run_canon_fw.sh 100D,firmware=boot=1
DebugMsg=0x4A74 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
./run_canon_fw.sh 100D,firmware=boot=1
DebugMsg=0x4A74 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] loading './100D/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './100D/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
Could not open ./100D/SFDATA.BIN
Seans-Mac-mini-385:qemu DeafEyeJedi$
Especially the 'Could not open ./100D/SFDATA.BIN' part? BTW I did make this virtual SD bootable w Macboot. So close I can feel it!

[dfort]

Code: [Select]

Could not open ./100D/SFDATA.BIN
Seans-Mac-mini-385:qemu DeafEyeJedi$
Especially the 'Could not open ./100D/SFDATA.BIN' part? BTW I did make this virtual SD bootable w Macboot. So close I can feel it!

Do you have your serial flash dump (SFDATA.BIN) next to your ROM0.BIN and ROM1.BIN files?

You don't need to make the virtual SD bootable with Macboot--where did you read that? Maybe it is corrupted now? No big deal, if you get into trouble or want to try out the latest QEMU changes simply delete your qemu directory (or rename it if you want to save it) and re-run the install.sh script in magic-lantern/contrib/qemu.

[DeafEyeJedi]

Do you have your serial flash dump (SFDATA.BIN) next to your ROM0.BIN and ROM1.BIN files?

Actually I do not. This is where I hit a wall. Not sure where I can actually get the serial flash dump -- care to refresh my memory in here, please?

You don't need to make the virtual SD bootable with Macboot--where did you read that? Maybe it is corrupted now? No big deal, if you get into trouble or want to try out the latest QEMU changes simply delete your qemu directory (or rename it if you want to save it) and re-run the install.sh script in magic-lantern/contrib/qemu.

Thought I read it somewhere that we had to make sure the virtual SD mount was bootable before running QEMU on it or no?

[dfort]

You need to run the sf_dump module to get a SFDATA.BIN dump and the virtual sd card already has the boot flag set. At least I never needed to run anything special to make it bootable.

[DeafEyeJedi]

You need to run the sf_dump module to get a SFDATA.BIN dump...

I read that. Clearly. Guess what I should have asked was how to get this sf_dump module? Can't seem to find it anywhere. Or maybe looking at the wrong directories?

...and the virtual sd card already has the boot flag set. At least I never needed to run anything special to make it bootable.

Gotcha. Thanks for the clarification.

[nikfreak]

You can find sf_dump in modules directory: Check:
https://bitbucket.org/hudson/magic-lantern/src/08720b28c58db49ae4de925c17240cf49050c74f/modules/sf_dump/?at=unified

[DeafEyeJedi]

Thanks for pointing me to that @nikfreak! Actually ended up getting a copy from @dfort that I shared w him from last year or so -- whew good save on that one, right?   

Anyhow, here's my 2nd attempt (more like 3rd or 4th, ha) at emulating QEMU on OS X 10.12.6 below...

...notice I ended up with a 'Camera was not shut down cleanly - Skipping module loading' message -- perhaps I didn't shut down the Emulator properly earlier and is there a way to 'turn off the camera' without having to force quit QEMU-system-arm manually?

and lastly what would be the 'trash' button on this keyboard? 

Because I have placed the required ML files in the virtual SD mount or at least seem to think so

[a1ex]

...notice I ended up with a 'Camera was not shut down cleanly - Skipping module loading' message -- perhaps I didn't shut down the Emulator properly earlier and is there a way to 'turn off the camera' without having to force quit QEMU-system-arm manually?

Maybe this needs reworded in a different way? (sorry, not native English speaker)

@kichetof: "Common issues and workarounds" in the README (or watch my animations)

[kichetof]

Maybe this needs reworded in a different way?

It's perfect, but, is it possible to keep the warning on the screen when you run it with QEMU ?
In my case, I didn't see the warning (too stuff in parallels )

[a1ex]

Good point, will see if that can be done without changing QEMU core code (as the shutdown behavior is hardcoded in every single GUI backend...)

edit: got it working atexit to the rescue...

[dfort]

...notice I ended up with a 'Camera was not shut down cleanly - Skipping module loading' message -- perhaps I didn't shut down the Emulator properly earlier and is there a way to 'turn off the camera' without having to force quit QEMU-system-arm manually?

If you get that message remove remove the LOADING.LCK from the ML/modules directory and the modules will load next time you start QEMU. I had to do that every time on the EOSM2 because I couldn't find a way to do a proper camera shutdown. Development on the qemu is moving quite rapidly so make sure you update your local repository and re-install QEMU if you encounter an issue--it might already be fixed!

[kichetof]

edit: got it working atexit to the rescue...

You're the best! 
If anymore ever complains about it, make a psychedelic flash of the warning 

[a1ex]

A little Easter egg (tested on 100D and 700D, but likely on all other D5 models):

Start the camera, go to PLAY mode (without any image present) and press DELETE. Watch the console output:

Code: [Select]

ON_ERASE
open B:/AUTOEXEC.SC
Not Found B:/AUTOEXEC.SC
ffffffff

Have fun discovering the language! (hint)

[nikfreak]

tried to follow http://chdk.wikia.com/wiki/Canon_Basic and used EOScard to set SCRIPT flag but somehow failing for the moment. Did you get it to work outside QEMU?

[a1ex]

Didn't try, but I bet this is not PowerShot Basic (so there's no point in following their guide).

[g3gg0]

- Even better - g3gg0 figured out how to emulate the real-time clock! This change superseded a bunch of GDB scripts - no more need to patch the date/time dialog!

for the record, got information from ricoh that the 5D3 chip is a R2262K
(thanks, guys!)

[dfort]

This is a really crazy idea that will probably never work but I'll ask anyway.

Would it be possible to update the firmware in QEMU?



Here is what happens on the 500D when selecting the "OK" button with the space bar:

Code: [Select]

Key event: 39 -> 0c01
[MPU] Sending : 06 05 06 0c 01 00  (GUI_SWITCH)
[MPU] Received: 06 05 04 01 7f 00  (PROP_ICU_UILOCK - spell #40)
[MPU] Sending : 06 05 04 01 7f 00  (PROP_ICU_UILOCK)
FF012F50: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
FF012F50: MCR p15, ...          : CACHEMAINT x5773 (omitted)
FF012F50: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0051079
FF012F84: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0051079
FF012F70: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF012F84: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0050079
FF012ED0: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0050079
FF012EBC: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF012ED0: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005007D
FF012EFC: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005007D
FF012EE8: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF012EFC: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[MPU] Received: 06 05 03 11 02 00  (unknown - unnamed)
[MPU] Received: 06 04 04 07 00 00  (unnamed - spell #45)
[MPU] Sending : 06 05 02 0b 00 00  (unnamed)
   938: 11598.080 [TERM] SHUTDOWN_REQUEST
   951: 11598.080 [LVCFG] PROP_LV_LOCK PROHIBIT
   952: 11598.336 [LV] JudgeStartLV 0x1 0x0 0xFFFF 2 0 0 5145
Key event: b9 -> 0c00
[MPU] Sending : 06 05 06 0c 00 00  (GUI_SWITCH)
[MPU] Received: 06 05 03 19 01 00  (PROP_TFT_STATUS - spell #47)
[MPU] Received: 06 05 02 0b 02 00  (unnamed - spell #46)
[MPU] Shutdown requested.

Looks like it sets up some registers then reboots the camera but in QEMU it simply shuts down.

Why would we want to do this? Getting a ROM dump on a Canon firmware update without actually updating the firmware on the camera would be an obvious reason though that might get complicated. Testing out a new .FIR would be another use though only a few developers can create .FIR files so they probably already have a way of testing them out. So the only good reason I could come up with is like that old mountaineering saying, “Because It's There.”

[EDIT] .FIR files need to be signed so to get this to work you need access to a key that will most likely open Pandora's box. So let's just call this post a muse rather than a feature request.

[t3r4n]

Hi,
a few days back a1ex asked me in the 750D thread to proofread his new README for the qemu branch.
Well as time is a premium I was able to do some testing only the other day. So here are some remarks, in what is hopefully the right thread to post them.

1. I was first testing with my 750D ROM files, which are not booting fully at the moment as we need to find some more places to patch away the endless waiting loops but with the instructions given in the readme and a looooot of hitting step on gdb that will be a nice task for cold winter evenings .

2. I wanted to understand better what is going on and managed to get my hands on an EOS 700D, compiled and installed ML and copied the ROM0 and ROM1 as described to the qemu/700D115 directory. Tried to start but it wasn't booting  as it was missing a SFDATA.BIN file.
So there is a missing section in the  README telling you to also compile the sf_dump Module and putting it on the ML Card and activate it and then Reboot the camera and use the module from the Debug Menu of ML. (only found it through full text search on the whole ML directory)

3. In the README under DEBUGGING you also write to use "make CONFIG_qemu=y" and the "make install_qemu" which wouldn't compile in the unified branch for the 700D. I found out that the qemu (or no dm_spy_experiments) branch is needed to use these options, maybe stress that out a bit more in the section.

These are my comments so far. Hope it helps.

[dfort]

I've got several firmware update projects that seem to be working fine except I don't have access to a ML-SETUP.FIR in order to set the camera bootflag on and off  so I need to downgrade the firmware, change the bootflag then upgrade firmware again to check and see if ML is affecting the camera. 

Of course we can do this when launching QEMU:

Bootflag off, starts Canon menus.

Code: [Select]

500D,firmware="boot=0"


Bootflag on, starts Magic Lantern.

Code: [Select]

500D,firmware="boot=1"


QEMU actually patches the ROM1.BIN with the "boot=" command, I learned that when working on the ML on EOS-M2 project.

I had some ROM dumps that were done without the bootflag set and when ML saves the ROM files the bootflag is set so I thought I'd look into where the bootflag was being set. I searched the forum and wiki for this and couldn't figure it out but once I looked at the ROM1.BIN files in a hex editor it didn't take long to find it.

Here is a ROM1.BIN that doesn't have the camera bootflag set. Note that only the ROM1.BIN needs to be changed.



And here it is with the camera bootflag set:



Pretty simple, huh? I could have figured it out by looking at the QEMU code but didn't think about that at the time. Looks like I'm not the only one reinventing the wheel.

eos.c

Code: [Select]

        /* fixme: reinventing the wheel */
        if (strstr(options, "boot=1") || strstr(options, "boot=0"))
        {
            /* change the boot flag */
            uint32_t flag = strstr(options, "boot=1") ? 0xFFFFFFFF : 0;
            fprintf(stderr, "Setting BOOTDISK flag to %X\n", flag);
            MEM_WRITE_ROM(s->model->bootflags_addr + 4, (uint8_t*) &flag, 4);
        }

The address to look for is "bootflags_addr + 4" so what is the bootflags_addr?

model_list.c

Code: [Select]

        /* defaults for DIGIC 4 cameras */
        .bootflags_addr         = 0xF8000000,
...
        /* defaults for DIGIC 5 cameras */
        .bootflags_addr         = 0xF8000000,
...
        /* defaults for DIGIC 6 cameras */
        .bootflags_addr         = 0xFC040000,

Just thought I'd share this for those of us who like to copy and paste the commands from debugmsg.gdb or patches.gdb without having to backspace to add the "boot=1" or "boot=0" depending on what we're checking out.

[dfort]

This might be basic but I've been having problems navigating around the 5D3 GUI. The up/down arrow keys work fine when the Canon menu first comes up but as soon as I press the left/right arrow keys the GUI freezes up. This doesn't happen on other cameras I tried but it does happen all the firmware versions of the 5D3 that I tried.

Oh yeah, I'm using a MacBook Pro so using the "keypad" to simulate the joystick doesn't seem to be an option.

Is there some sort of a workaround for this?

[a1ex]

Reproduced with your 1.1.3 ROM.

Can you get a dm-spy log from photo mode, with LOG_INTERRUPTS enabled?

edit: it gets stuck into some routines related to LightMeasure, so it must be related to auto LCD brightness. Workarounds:
- set LCD brightness to Manual before dumping the ROM
- change property 0x204000D to 1

Confirmed the second workaround on your ROM.