Magic Lantern Forum

Magic Lantern Releases => Camera-specific discussion => Topic started by: Palpatine on September 22, 2015, 02:48:23 PM

Title: ML on EOS-M2
Post by: Palpatine on September 22, 2015, 02:48:23 PM
It's my first post so... hello guys! :)

Have anyone tried to install ML on EOS-M2?

I'm looking for a low weight camera because I want to use it with a hexacopter. EOS-M2 is surprisingly cheap on ebay and I think it has a proper quality for tasks that I'm going to do. It would be also nice to run ML on it. I know it was released only for Asian market so it's not popular, but maybe someone can assure me that it's possible to run ML without problems.

I'll be thankful for all opinions and advices :)
Title: Re: ML on EOS-M2
Post by: Walter Schulz on September 22, 2015, 05:31:23 PM
Each ML port is pretty much hardcoded for a cam and a specific firmware version. Porting ML takes a skilled programmer fluently speaking ARM processor (="maintainer") and several hundred hours of work. And at time of writing there is a lack of maintainers resulting in orphaned/blindly maintained ports.
A spare developer would come handy ...
Title: Re: ML on EOS-M2
Post by: neoplanta on September 24, 2015, 12:54:40 AM
I was asking the same on the other topic. It would be nice if someone is considering to do this to tell us :) I did not even look when I was buying eos m 2 about this, I asumed that it was the same ML for both 1 and 2.
Title: Re: ML on EOS-M2
Post by: dfort on September 24, 2015, 06:02:57 AM
I just checked up on prices and yes the M2 is priced very low but the M1 is even lower and there is a Magic Lantern port for it. According to a former M2 owner, now an M3 owner:

Dumb question, what's necessary to customize the video bitrate on the firmware? Not a programmer, but I can script.

The EOS M2 had 50mb which produced beautiful 1080P video and no audio compression which resulted in great fidelity; vs the low bit H264 and AAC that Canon throws on the EOS M3 to neuter it so it doesn't compete with pro offerings since the autofocus and software IS is good on the M3/new APS-C's. I almost want to buy another M2 (sold it once I bought the M3) since it did such a good job with video.

So if you have an M2 you might consider holding on to it even without a ML port.

The Canon EOS M3 (http://www.magiclantern.fm/forum/index.php?topic=14990.0) discussion is hinting on some serious interest in porting ML but there is no indication whether or not it has gotten off the ground yet. Though it does look like the CHDK (http://chdk.wikia.com/wiki/CHDK) project has something working with the M3 (http://chdk.wikia.com/wiki/M3). Sorry, nothing that I could find for the M2.
Title: Re: ML on EOS-M2
Post by: godashram on October 14, 2015, 01:44:25 AM
doesn't help that not a single update has been released for the M2...

My M2 would like some love as well...but coding is not my specialty :( much less figuring out if it'll run ML like the Eos M or CHDK like the M3
Title: Re: ML on EOS-M2
Post by: a1ex on October 14, 2015, 12:40:42 PM
On ExifTool website there is a hint that EOS M2 is still based on DSLR firmware (unlike M3, which is based on PowerShot firmware).

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html

So, if anyone is tempted to port ML on it, I can prepare a dumper.
Title: Re: ML on EOS-M2
Post by: neoplanta on October 14, 2015, 05:16:14 PM
It would be great if somone would do this. I know this is a dumb question, but I am pretty much idiot when it comes to coding, is it posible to install eos m 1 firmware to eos m 2, or just make focus peaking option somehow on eos m 2?
Title: Re: ML on EOS-M2
Post by: godashram on October 14, 2015, 05:57:41 PM
A1ex, you prep a dumper, i'll dump mine as soon as it's up (well, after driving home to grab the M2)

granted, the most I could do after that is try out builds, my code-fu is non-existent :(
Title: Re: ML on EOS-M2
Post by: a1ex on October 14, 2015, 06:31:15 PM
Alright, here you go:

- the portable display test: hello-M2.fir (http://a1ex.magiclantern.fm/debug/portable-hello-world/EOSM2/hello-M2.fir)

- the ROM dumper: dumperM2.fir (http://a1ex.magiclantern.fm/bleeding-edge/blind-dumper/dumperM2.fir)

It dumps 32 MB from 0xF7000000, which covers both ROM0 and ROM1 - that's what you need to run ML in QEMU. For analyzing the code, you can load ROM1 (the second half of that file) at 0xFF000000.

Usage: run firmware update, then go to play mode (make sure you have an image there), then look on your card. Keep the dump for yourself, do not publish it online.

edit: dumper confirmed working by @godashram :)
Title: Re: ML on EOS-M2
Post by: Licaon_Kter on October 15, 2015, 10:44:10 AM
Interesting.
Title: Re: ML on EOS-M2
Post by: vyskocil on October 15, 2015, 01:26:51 PM
Great, new gears ! Is it possible to have the same for the 7D mkII to start looking at a port ? Perhaps the dumper is not needed as we already have a fir upgrade available but then could you point out the steps to lookup the keys and decrypt the firmware ?
Thanks in advance!
Title: Re: ML on EOS-M2
Post by: godashram on October 15, 2015, 07:19:50 PM
wouldn't it make more sense to start a new thread for the 7D MKII instead of posting the question here??
Title: Re: ML on EOS-M2
Post by: glassescreditsroll on October 19, 2015, 09:15:54 AM
does this mean we could start  porting ml to eos m2?
Title: Re: ML on EOS-M2
Post by: godashram on October 20, 2015, 12:57:29 AM
I hope it does.... I've heard nothing since giving A1ex the M2 dump. Guess the biggest issue would be finding someone to actively work on an M2 build?
Title: Re: ML on EOS-M2
Post by: a1ex on November 01, 2015, 07:08:36 PM
does this mean we could start porting ml to eos m2?

Correct. The M2 looks very similar to the original M, so I don't expect any real difficulties here.

Guess the biggest issue would be finding someone to actively work on an M2 build?

Correct.
Title: Re: ML on EOS-M2
Post by: glassescreditsroll on November 06, 2015, 04:50:30 AM
So what needs to be done now? I have an eos m2 and would love to be able to use ML on it I'll do anything I can to help
Title: Re: ML on EOS-M2
Post by: Audionut on November 06, 2015, 06:08:50 AM
Guess the biggest issue would be finding someone to actively work on an M2 build?

Correct.

Current developers are happy to help.  But there needs to be someone else to take on the bulk of the development work.
Title: Re: ML on EOS-M2
Post by: godashram on November 10, 2015, 05:42:11 PM
Current developers are happy to help.  But there needs to be someone else to take on the bulk of the development work.

and that is where the issue is.... I don't have the patience for development, and well, not many M2's seem to be in the hands of people that would care (my opinion).

to be honest, I'd be happy with just the continuous recording feature :)
Title: Re: ML on EOS-M2
Post by: neoplanta on November 10, 2015, 06:39:44 PM
And I would be happy only with focus peaking option  :D It is too bad that m2 is not wide spread as other cameras, and it is waste that there are people who are willing to help, but unfortunetly there are not many people who know coding to take the bulk of the development work.
Title: Re: ML on EOS-M2
Post by: godashram on November 10, 2015, 10:20:08 PM
well, it's lack of availability outside of Japan and China and the eos m fire sales didn't help the m2 in any way at all.
Title: Re: ML on EOS-M2
Post by: GenMeiHikaru on December 16, 2015, 04:26:22 PM
Just bought a EOS M2 kit (+ a EF-M 22mm and a 18-55mm) with a kinda low price in Japan, just about $285. Seems like the only thing missing is ML firmware, coding is also a no-go for me but I'll be glad to be of any help. Thanks...!
Title: Re: ML on EOS-M2
Post by: neoplanta on December 16, 2015, 06:57:43 PM
Wow, thats prety cheap. I bought mine for about 280 dolars, without lens.
Title: Re: ML on EOS-M2
Post by: godashram on December 16, 2015, 07:32:08 PM
WOW, for $285 with 2 lenses is NUTS! it's a great camera... but hunting down someone to work on it isn't easy :(
Title: Re: ML on EOS-M2
Post by: GenMeiHikaru on December 17, 2015, 03:25:31 PM
Surprisingly it's not just the lens, the EF-EOS M -> EF-S adapter and a 90EX flash are also included in the package.
If u guys still want the link, it's here
http://www.amazon.co.jp/gp/product/B00H2ABRKS/ref=gbps_img_s-3_0229_0fa1bf1e?smid=AN1VRQENFRJN5&pf_rd_p=264150229&pf_rd_s=slot-3&pf_rd_t=701&pf_rd_i=gb_main&pf_rd_m=AN1VRQENFRJN5&pf_rd_r=1F660AF0S908GYXQ8A9F
It's Amazon for Japan so i'm not sure if it's available for shipment outside of Japan though... I bought it with a Lightning Deal (¥34800~$285), but without the deal it's ¥40498(~$330), still very affordable (I think)...
Edit: on sale again...
Title: Re: ML on EOS-M2
Post by: godashram on December 18, 2015, 02:22:18 AM
fulfilled by amazon.... so maybe...
Title: Re: ML on EOS-M2
Post by: jonkjon on December 18, 2015, 04:49:45 PM
Is this something that needs to be built from scratch or is there something to build from? I have some c experience (mostly windoze) and i'd be willing to take a look. I have perused the getting started stuff but it pretty much just shows how to setup a compiler etc. I haven't looked at the source code at all.

--Jon
Title: Re: ML on EOS-M2
Post by: godashram on December 18, 2015, 05:55:28 PM
the m2 was dumped (a1ex has the dump I made)

from there, i guess it would be a port of the existing build, beyond that, no clue.
Title: Re: ML on EOS-M2
Post by: Walter Schulz on December 18, 2015, 06:37:09 PM
What you are looking for is a maintainer. Someone knowing C, assembler programming embedded devices (preferable ARM architecture) and willing and able to spend several hundred hours of his/her time in porting ML to M2. And don't expect any of the devs asking for that additional workload. They will support this maintainer-to-be, of course.
Title: Re: ML on EOS-M2
Post by: bender on March 06, 2016, 10:16:17 AM
any updates for the M2?  :)
Title: Re: ML on EOS-M2
Post by: brocolimayo on March 15, 2016, 08:10:19 PM
same question :) my programming skills are almost zero, but I'd love to help, I own a m2 too, and would be wonderful to have ml on it :)
Title: Re: ML on EOS-M2
Post by: godashram on March 16, 2016, 06:54:44 AM
nope, as far as I know, there isn't anyone willing to work on it.
Title: Re: ML on EOS-M2
Post by: Skylin3 on March 16, 2017, 11:31:14 PM
Bump. Is there any progress with m2? I can help with testing ML, I have m2.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 05:50:16 AM
Got one.

(https://c1.staticflickr.com/5/4227/34014053584_75ea69bf64_z.jpg)

Not sure if I'm ready for this but I was going to quietly work on Magic Lantern for the EOSM2. Seems like this one should be one of the "easier" ports.

Then I thought, that's no way to work on an open source project, let's get this out there in the open. My first stumbling block is setting the boot flag on the camera. How to do this without bricking the camera? Tried searching for the answer and it looks like there is already a working .FIR file for this camera -- so how can I get my hands on it?
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 09:43:20 AM
Seems like this one should be one of the "easier" ports.

Definitely, as it's plain DIGIC 5 (unlike 1300D, which looks like a mix between D4, 5 and 6).

The boot flag is "traditionally" enabled after getting the Hello World working. There are two flavors: the minimal target (which requires a tiny set of stubs and only compiles a few small files) and the regular one (CONFIG_HELLO_WORLD in the full source code, which also requires file I/O for loading fonts, and a bunch of other initializations). Once the second version is working, CONFIG_DUMPER_BOOTFLAG will be straightforward to compile and run. The "user-friendly" installer comes a bit later, as it requires some more stubs.

The boot flag can be also enabled earlier, from the bootloader context; we did that on the DIGIC 2 and 3 models, blindly (without having the display available for debugging or user feedback). Now that we have display access from bootloader, that's a valid option as well (though I wasn't comfortable doing this step on 7D Mark II).

Until enabling the boot flag, you'll need my assistance to run a binary on the camera. You can run them in QEMU though, by setting "boot=1" though.

Will explain how to get the Hello World on models that don't boot to Canon GUI on the 1300D thread, as it was covered on IRC a few days ago.
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 12:17:11 PM
I'll start with a walkthrough using latest QEMU and IDA. First step is to assume it's very similar to EOSM, so after installing QEMU (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console), head over to qemu/qemu-2.5.0/hw/eos/model_list.c and copy the basic entries from the section for EOSM:

Code: [Select]
    {
        .name                   = "EOSM2",
        .digic_version          = 5,
     }

The others are likely model-specific, so let's not assume too much from the beginning. Let's try:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0"
...
FFFF3F50: MCR p15,0,Rd,cr6,cr0,0:  946_PRBS0 <- 0x3F       (00000000 - FFFFFFFF, 0x100000000)
FFFF3F58: MCR p15,0,Rd,cr6,cr1,0:  946_PRBS1 <- 0x3D       (00000000 - 7FFFFFFF, 0x80000000)
FFFF3F60: MCR p15,0,Rd,cr6,cr2,0:  946_PRBS2 <- 0xE0000039 (E0000000 - FFFFFFFF, 0x20000000)
FFFF3F68: MCR p15,0,Rd,cr6,cr3,0:  946_PRBS3 <- 0xC0000039 (C0000000 - DFFFFFFF, 0x20000000)
FFFF3F70: MCR p15,0,Rd,cr6,cr4,0:  946_PRBS4 <- 0xFF00002F (FF000000 - FFFFFFFF, 0x1000000)
FFFF3F78: MCR p15,0,Rd,cr6,cr5,0:  946_PRBS5 <- 0x37       (00000000 - 0FFFFFFF, 0x10000000)
FFFF3F80: MCR p15,0,Rd,cr6,cr6,0:  946_PRBS6 <- 0xF700002F (F7000000 - F7FFFFFF, 0x1000000)
...
K355 READY
128K Sector FROM From BL 0xffff
[SF] InstallSerialFlash 6 0xc022c0d4 0x0 0x1000000 1
...
K355 ICU Firmware Version 1.0.2 ( 6.0.5 )

OK, so we've got a rough idea about its memory map (same as other DIGIC 4 and 5 models). We also noticed it has a serial flash of size 0x1000000. This is similar to 100D (lookup serial_flash_size in model_list.c), so it may be a good idea to reuse the serial flash data from this model:
Code: [Select]
        .serial_flash_size      = 0x1000000,

Unfortunately, this step doesn't change anything in the emulation. Let's look at the next messages:
Code: [Select]
...
    10:    46.336 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x10000
...
    14:    56.576 [STARTUP] InitializeIntercom
...
    42:   722.688 ERROR [DL] ########## DL ERROR!!!!! ###########

- Many errors regarding missing properties. These are either stored in ROM, or in the serial flash, or they come from the MPU. The ones from ROM should not pose any problems, since we have the complete ROM. For the others, we did not see any sign of activity, so we have to dig in to find out whether anything is different (usually these communication channels may use different registers, depending on model, but overall the communication protocol is the same).

- DL ERROR: if you look in EOSM/debugmsg.gdb, you'll find a way to get past this message (by skipping the initialization of "DL", whatever that is).

- The debug messages are not very informative. To get more insights, find out a few essential functions (such as task_create and DebugMsg) and place them in EOSM2/debugmsg.gdb (look at other models for how the declarations should look like).

OK, so at this point we have to disassemble the firmware. If you use IDA, the initial steps would be:
- load ROM1.BIN
- select ARM processor, architecture ARMv5TEJ (we have ARMv5TE on all models from DIGIC 2 to DIGIC 5)
- uncheck 'Delete instructions with no xrefs' (we have plenty of those)
- uncheck 'Perform no-return analysis' (IDA usually gets it wrong)
- RAM section: 0, size 0x100000 (we can always add more later)
- ROM start address: 0xFF000000
- ROM size: 0xFFFFFC (off-by-one bug in IDA; I have an older version, so YMMV)
- File load address / size: same as ROM.

Now that our binary is loaded, go to 0xFFFF0000 and press "c" to mark this as code. This is the bootloader start address (aka HIVECS in ARM documentation). Next step is to find where the main firmware starts (where the code path leaves the bootloader). It's easy to assume similarity with other models, but let's try to find it from scratch (just to show off the latest QEMU tool):

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d calls
...
call 0xFFFF0FCC(0, ffffdfff, 2000, ffff0b20)                                     at [ffff09c8:0]
 call 0x100000(0, 11836c, 11836c, 100000)                                        at [ffff1010:ffff09cc]
...

OK, so the bootloader runs from RAM.

Where does the bootloader end? When it stops executing from its address range (near 0xFFFF0000 or 0x100000):

Code: [Select]
  return 0 to 0x10013C                                                           at [1022f4:ffff1014]
 return 1 to 0xFFFF1014                                                          at [1000bc:ffff09cc]
return 0 to 0xFFFF09CC                                                           at [ffff101c:0]
Warning: R10 not restored (0xa -> 0x1)                                           at [ffff101c:ffff09cc]
PC jump? 0xF80C0000 lr=ffff09cc                                                  at [ffff0a04:ffff09cc]
0xffff0a04:  e1a0f000      mov pc, r0
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
0xf80c0000:  e59ff0c4      ldr pc, [pc, #196] ; 0xfffffffff80c00cc

Ta-da! Same as other DIGIC 5 models: ROM1 startup code is at 0xF80C0000, but code from main firmware expects to run from a mirror copy (details) (https://www.magiclantern.fm/forum/index.php?topic=6785.msg58899#msg58899). So, our ROMBASEADDR is 0xFF0C0000.

Let's go there in IDA and mark this section as code. At this point, IDA already recognized a few functions, but let's get some more from the QEMU execution trace:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d idc
...
EOSM2.idc saved.

Now load this IDC script into IDA, or convert it for your favorite disassembler, and start looking at the functions called during the execution in QEMU:
Code: [Select]
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
...
call 0xFF0C1BD4(1000, 698, eeeeeeee, 1000)                                       at [ff0c0dbc:0]
 call 0x866B4(f88, 74, eeeeeeee, 1000)                                           at [ff0c1be4:ff0c0dc0]
 return ffc to 0xFF0C1BE8                                                        at [866f4:ff0c0dc0]
 call 0x3168(f88, ff0c57a4, 0, 0)                                                at [ff0c1c80:ff0c0dc0]
...

0xFF0C1BD4 is cstart, 0x866B4 is bzero32, 0x3168 must be create_init_task and 0xFF0C57A4 must be init_task. Some of these functions are called from RAM, so we'll have to identify where its contents come from (what is copied where). TODO.

Next step, for the emulation, would be to find the MPU registers (near InitializeIntercom), mpu_send/mpu_recv stubs may also help, and we also need to check the serial flash communication.

For executing user code on the camera, next step would be to reserve memory for our application. This is usually done by shrinking the "malloc" buffer. Take a look at boot-hack.c from the "qemu" branch (as it's a bit more verbose and better explained). These debug messages can be seen on the QEMU-boot-check (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-boot-check/) page.

To be continued.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 06:19:03 PM
Wow.

Let's take half a step back before embarking on this journey. When I posted my intent to attempt this port I skipped one important step before asking for a firmware dumper--check to see if there is already one available. The list of available ROM dumpers are listed at the bottom of the Magic Lantern Nightly Builds (https://builds.magiclantern.fm/) page. The one for the EOSM2 point to a post on this topic (http://www.magiclantern.fm/forum/index.php?topic=15895.msg155455#msg155455) with the links to download the dumpers.

It appears that the EOSM2 ROM was first dumped about 1.5 years ago. I'm assuming that it was the original firmware version. The camera I got has 1.0.2 installed and 1.0.3 is available. My question is, should I try to make a dump of 1.0.2 before updating the firmware or does it matter? Maybe the dumpers won't work with 1.0.3? Maybe we should archive a 1.0.2 dump before starting? The 1.0.3 firmware update is a minor fix that Canon released for most of their cameras to deal with this issue:
Quote
Corrects a phenomenon in which when using the camera with the EF-S 18-135mm f/3.5-5.6 IS USM or EF 70-300mm f/4-5.6 IS II USM lens, even if lens aberration correction is set to "Enable", correction will not be applied.

[EDIT]

hello-M2.fir (http://a1ex.magiclantern.fm/debug/portable-hello-world/EOSM2/hello-M2.fir) running on EOSM.102
(https://c1.staticflickr.com/5/4200/34481675560_77a17b04d7_z.jpg)

However, dumperM2.fir (http://a1ex.magiclantern.fm/bleeding-edge/blind-dumper/dumperM2.fir) doesn't dump the ROM. I tried various cards including an old 2GB card that has worked on all the other ROM dumps I made.
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 09:23:14 PM
The dumper linked above runs on the main firmware (in contrast with the portable ROM dumper, which runs from bootloader), so it has no issues with card sizes, but it's limited to DIGIC 4 and 5 models. Does it help if you follow the instructions in the linked post? (it's different from the portable dumper you are used to)

You can start from the latest firmware.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 10:40:55 PM
Does it help if you follow the instructions in the linked post?

Doh!

The single file dump is named "(NULL)" on the sd card and it is 33.6 MB. I also updated the firmware to 1.0.3 though I also dumped 1.0.2 just to have it.

Did the change to model_list.c and tried running qemu but it stopped on:
Code: [Select]
eos_load_image: file not found './EOSM2/ROM0.BIN'
./run_canon_fw.sh: line 35: 16058 Abort trap: 6           $QEMU_PATH/arm-softmmu/qemu-system-arm -drive if=sd,format=raw,file=sd.img -drive if=ide,format=raw,file=cf.img -M $*

So I renamed (NULL) to ROM0.BIN then it stopped on "file not found './EOSM2/ROM1.BIN'" but since the dumper only creates the one file I'm afraid I hit another wall.

It dumps 32 MB from 0xF7000000, which covers both ROM0 and ROM1 - that's what you need to run ML in QEMU. For analyzing the code, you can load ROM1 (the second half of that file) at 0xFF000000.

I'm searching how to run ML in QEMU with a single file dump but so far I can't find an answer.

Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 11:15:20 PM
Split it in two halves: first ROM0, second ROM1.

Back then, it was much easier to patch Canon code to dump everything in one file, so it covers both ROMs. For this reason, older QEMU versions used to require the ROMs to be joined like this (but it was changed to allow using the backup ROM copies already saved by ML on the cards under ML/LOGS, and to allow more unusual ROM configurations).
Title: Re: ML on EOS-M2
Post by: dfort on May 25, 2017, 01:23:00 AM
Making some progress. Mac uses the BSD version of split which doesn't have the -n option but I was able to split the dump in two like this:
Code: [Select]
split -b 16800000 "(NULL)" ROM
That gave me two files, ROMaa and ROMab which I renamed ROM0.BIN and ROM1.BIN and placed in ~/qemu/EOSM2 then ran:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0"
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
CHK version_gen.h
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFA6FF
[MPU] FIXME: no MPU spells for EOSM2.
[MPU] FIXME: no MPU button codes for EOSM2.
Setting BOOTDISK flag to 0

It looks like this:

(https://c1.staticflickr.com/5/4221/34833502446_a6c5cf5e35_z.jpg)

QEMU didn't crash but it also isn't displaying the output from the minimal qemu autoexec.bin.
Title: Re: ML on EOS-M2
Post by: dfort on May 25, 2017, 05:02:25 AM
Took some more baby steps. First a fall.
OK, so we've got a rough idea about its memory map (same as other DIGIC 4 models). We also noticed it has a serial flash of size 0x1000000. This is similar to 100D (lookup serial_flash_size in model_list.c), so it may be a good idea to reuse the serial flash data from this model:
Code: [Select]
        .serial_flash_size      = 0x1000000,

All that did was to consistently give me this error when running QEMU:

Code: [Select]
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFA6FF
Could not open ./EOSM2/SFDATA.BIN

I don't have IDA but I do have ARM-console (http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console) and the good old reliable disassemble.pl from the finding stubs tutorial (http://www.magiclantern.fm/forum/index.php?topic=12177.msg117735#msg117735). So I took another baby step and tried to disassembly the ROM1.BIN that was split off the single file dumperM2.fir dump. I ran the usual "perl disassemble.pl 0xFF000000 ROM1.BIN" but something looked wrong.

Code: [Select]
ff0ba700: e59ff0c4 ldr pc, [pc, #196] ; ff0ba7cc: (ff0c000c)
"gaonisoy":

All indications are that it should be more like the EOSM and 700D and start on 0xFF0C0000 so I did a second disassemble, "perl disassemble.pl 0xFF005900 ROM1.BIN"

Code: [Select]
ff0c0000: e59ff0c4 ldr pc, [pc, #196] ; ff0c00cc: (ff0c000c)
"gaonisoy":

Now it "looks" right. Well, "looks" is mostly what I'm going on right now. I don't really know what I'm doing. That weird offset might be a result of splitting the dump in two? The EOSM2 "looks" like a combination of the EOSM, 100D with wifi features added on. This means that I should be able to find some stubs using the pattern matching method I used for the minor firmware updates. Of course porting a new camera is certainly not a minor undertaking!

I want to continue getting QEMU working properly on the Mac and figure things out the "right" way but now that I've got a disassembly at least I'm on familiar territory so do you think I'm at the stage where I can start hunting down some stubs?
Title: Re: ML on EOS-M2
Post by: a1ex on May 25, 2017, 08:40:40 AM
You need two *equal* halves, without any approximation.

(edit: updated QEMU to give better warnings in this case)

SFDUMP.BIN can probably be used from 700D or EOSM as well, but these cameras have a smaller size. Padding it with zeros may or may not work. How to get it? It's covered in QEMU's install script.
Title: Re: ML on EOS-M2
Post by: dfort on May 26, 2017, 08:36:33 AM
You need two *equal* halves, without any approximation.

(edit: updated QEMU to give better warnings in this case)

Yeah, that seems to be a problem. "split -b 16800000 "(NULL)" ROM" produced two files that aren't exactly equal. I'll keep up with the QEMU branch commits on this project.

Code: [Select]
-rw-r--r--   1 rosiefort  staff  16800000 May 24 15:15 ROM0.BIN
-rw-r--r--   1 rosiefort  staff  16754432 May 24 15:15 ROM1.BIN

[EDIT] Looks like the lesson learned here is not to trust the finder because it rounds KB to MB.

(https://c1.staticflickr.com/5/4248/34063971594_70bf4993c9.jpg)

I might be able to get SFDUMP.BIN from a 100D.
Title: Re: ML on EOS-M2
Post by: dfort on May 26, 2017, 07:09:10 PM
The equal split did the trick. Now I'm seeing the messages a1ex posted back on Reply #34 (http://www.magiclantern.fm/forum/index.php?topic=15895.msg185103#msg185103). Also made a new disassembly and it looks good so I'll start hunting for stubs.

To be continued.

Looking forward to it!
Title: Re: ML on EOS-M2
Post by: a1ex on May 26, 2017, 11:09:39 PM
Alright, so let's start from where we've left off.

0xFF0C1BD4 is cstart, 0x866B4 is bzero32, 0x3168 must be create_init_task and 0xFF0C57A4 must be init_task. Some of these functions are called from RAM, so we'll have to identify where its contents come from (what is copied where). TODO.

Since the ROMs are placed at some high address (after 0xF0000000), and we've noticed some functions called outside this range, a good guess would be that some of the functions may be copied from ROM to RAM during startup. Let's examine the memory access patterns. We are interested in some code that reads from ROM and writes to RAM. If you have no trouble following the assembly code for the startup sequence, you'll spot this piece of code right away. Otherwise, let's ask for some help from QEMU:

Running QEMU with -d help gives, among many other options:
Code: [Select]
ram        EOS: log all RAM reads and writes
rom        EOS: log all ROM reads and writes
ramr       EOS: log all RAM reads
romr       EOS: log all ROM reads
ramw       EOS: log all RAM writes
romw       EOS: log all ROM writes

We could try romr,ramw (or rom,ram if you don't mind a little more verbosity):
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d romr,ramw

The log is huge, but if you scroll around, there are some large blocks that look similar. Some of them are clearly a copying operation (read some value from ROM, write it to RAM). One of these covers the functions we are interested in:
Code: [Select]
[rom1]     at 0xFF0C000C:FFFF09CC [0xFFD1F02C] -> 0xE92D4010
[ram]      at 0xFF0C000C:FFFF09CC [0x00001900] <- 0xE92D4010: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F030] -> 0xE1A04001
[ram]      at 0xFF0C00A4:FFFF09CC [0x00001904] <- 0xE1A04001: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F034] -> 0xE59F1040
...
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFDD60C8] -> 0xFFD1EFFC
[ram]      at 0xFF0C00A4:FFFF09CC [0x000B899C] <- 0xFFD1EFFC: was 0x0;

In other words, a memory block from 0xFFD1F02C to 0xFFDD60C8 (actually 0xFFDD60CB, since we are looking at 32-bit operations) is copied to 0x1900 - 0xB899F. The size of the copied block is 0xb70a0 bytes. To extract this block, grab the terminal and run this under a Bash prompt:

Code: [Select]
dd if=ROM1.BIN of=1900.BIN bs=1 skip=$((0xD1F02C)) count=$((0xB70A0))

(please note the above numbers are only valid for firmware 1.0.2)

Now you can disassemble this file starting from 0x1900. In IDA, load this file as Additional binary file. If you use ARM-console, this should auto-detect the above copying operation (so you can simply start browsing the functions copied into RAM).

There are other similar blocks copied into RAM (newer DIGIC 6 and 7 models have plenty of those), and finding them manually can become tedious, so it's a good candidate for automation.

One of these blocks is the interrupt handler routine. On ARM, hardware interrupts are executed from address 0x18 (see e.g. ARM ARM - A2.6 - Exceptions (https://www.scss.tcd.ie/~waldroj/3d1/arm_arm.pdf#G6.1052026) or these slides (http://osnet.cs.nchu.edu.tw/powpoint/Embedded94_1/Chapter%207%20ARM%20Exceptions.pdf)). That means, we should expect to find some code that writes something to address 0x18 (that can be anywhere in the boot process before enabling interrupts), and on EOS firmwares, at this address you'll find a jump to 0x4B0 on many models, or something close on others. Let's examine it in GDB:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & arm-none-eabi-gdb -x debug-logging.gdb
(note: we will need to create a GDB file for EOS M2, named EOSM2/debugmsg.gdb to match other models, but since we don't have one yet, we can just use the generic version - which we are going to include in the platform-specific file later)

Let it run for a while (continue), then hit CTRL-C and examine the exception vector (located at address 0):

Code: [Select]
(gdb) disas 0,0x40
Dump of assembler code from 0x0 to 0x40:
   0x00000000: nop ; (mov r0, r0)
   0x00000004: ldr pc, [pc, #20] ; 0x20
   0x00000008: ldr pc, [pc, #20] ; 0x24
   0x0000000c: ldr pc, [pc, #20] ; 0x28
   0x00000010: ldr pc, [pc, #20] ; 0x2c
   0x00000014: nop ; (mov r0, r0)
   0x00000018: ldr pc, [pc, #16] ; 0x30
   0x0000001c: ldr pc, [pc, #16] ; 0x34
   0x00000020: ; <UNDEFINED> instruction: 0xff0c0fec
   0x00000024: ; <UNDEFINED> instruction: 0xff0c105c
   0x00000028: ; <UNDEFINED> instruction: 0xff0c1000
   0x0000002c: ; <UNDEFINED> instruction: 0xff0c1018
   0x00000030: ; <UNDEFINED> instruction: 0x000004b0
   0x00000034: ; <UNDEFINED> instruction: 0xff0c1060
   0x00000038: ; <UNDEFINED> instruction: 0xffff0b08
   0x0000003c: ; <UNDEFINED> instruction: 0xffff0b20

That means, once a hardware interrupt happens (in other words, some device wants to tell the main CPU that it needs attention), the program counter will jump to 0x18, and there it will find a LDR PC, =0x4B0. That's where Canon's interrupt handler is located. Let's find its contents:

- Method 1: scan the memory access log for a copy operation that covers 0x4B0 and above. Easy to find.

- Method 2: disassemble it directly from GDB. How much? 0x1900 - 0x4B0 would be the upper limit (since we already know what's at 0x1900). The interrupt handler is much smaller than that - 512 bytes are more than enough:
Code: [Select]
disas 0x4B0,0x6B0

Method 3: dump the memory from gdb:
Code: [Select]
(gdb) dump memory 4B0.BIN 0x4B0 0x6B0

Method 4: dump the memory from QEMU monitor (https://qemu.weilnetz.de/doc/qemu-doc.html#pcsys_005fmonitor):
Code: [Select]
(qemu) memsave 0x4B0 0x200 4B0.BIN

For the debugmsg.gdb file, we'll need some basic stubs: DebugMsg and task_create (easy to find - one is used for most debug messages, the other can be easily identified from things that look like task names) and two addresses: CURRENT_TASK and CURRENT_ISR (interrupt service routine). We can find the latter in the interrupt handler - quote from debugmsg.gdb:
Quote
# CURRENT_ISR:
#   From interrupt handler (PC=0x18), find an expression that evaluates to
#   the current interrupt ID if any is running, or 0 if a normal task is running.
#   - on DIGIC 4/5, interrupt ID is MEM(0xC0201004) >> 2
#   - on DIGIC 6,   interrupt ID is MEM(0xD4011000)

If you can find that by examining disassembled code, great. If not, QEMU to the rescue:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,nochain -singlestep
...
Taking exception 5 [IRQ]
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah

(note: -d nochain -singlestep gives more precise locations in I/O logs - otherwise, QEMU groups the ARM instructions in blocks for faster execution, so the reported location will be approximate)

OK, so the register we are looking for is read at 0x510. Let's disassemble with GDB:
Code: [Select]
(gdb) disas 0x4B0,0x6B0
...
   0x0000050c: ldr r0, [pc, #368] ; 0x684
   0x00000510: ldr r4, [r0]
   0x00000514: str r4, [pc, #304] ; 0x64c
...
(gdb) x 0x684
0x684: 0xc0201004
(gdb) x 0x64c
0x64c: 0x00000028
(gdb) print 0x28>>2
$1 = 0xa

The register we are looking for is loaded in R0, then the interrupt handler reads from this register (result in R4) and stores its value at 0x64C. If you check the memory contents at this address, you'll find 0x28. The interrupt ID would be 0x28 >> 2 = 0x0A, which is DryOS timer interrupt (which fires every 10ms and is used for the task switch - see e.g. FreeRTOS Tick (http://www.learnitmakeit.com/freertos-tick/) for some background info).

Note: ARM has a generic hardware interrupt (PC jumping at 0x18), but in an embedded system we usually have more devices that can trigger an interrupt. To tell which device needs attention, it's helpful to identify these devices somehow - and an interrupt ID is used for that. On EOS, there is a custom interrupt controller, and on DIGIC 4 and 5 models, the register used for identifying the interrupt ID is 0xC0201004. The interrupt ID is hardwired to the device that triggers it (for example, SIO3 and MREQ, which are used for MPU communication, always use interrupts 0x36 and 0x50). This mapping can change across models, but usually it's consistent (at least within the same generation of models). Probably g3gg0 can explain these concepts a bit better, as he identified all this stuff a few years ago (http://www.magiclantern.fm/forum/index.php?topic=2882.0), when I was mostly clueless about how these things work.

OK, so the memory contents at 0x64C can tell us about the interrupt currently handled. There's a problem - this address is not cleared when the interrupt handler is done, so, just by looking at it, we can't tell whether the code is servicing an interrupt or running a regular task. We'll need to look at some other address - and in many cases the previous address is a good candidate:
Code: [Select]
   0x000004d4: ldr r1, [pc, #364] ; 0x648
   0x000004d8: cmp r1, #0
   0x000004dc: add r1, r1, #1
   0x000004e0: str r1, [pc, #352] ; 0x648
   ...
   0x00000598: ldr r1, [pc, #168] ; 0x648
   0x0000059c: sub r1, r1, #1
   0x000005a0: str r1, [pc, #160] ; 0x648

This 0x648 looks like a counter that tells how many nested interrupts we are handling (yes, they can be nested - unfortunately, as this makes it a lot harder to understand, debug, emulate and so on). Let's confirm its functionality with RAM tracing:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,ram,nochain -singlestep
...
[tcm_code] at 0x000004D4:19980218 [0x00000648] -> 0x0       
[tcm_code] at 0x000004E0:19980218 [0x00000648] <- 0x1       : was 0x0;
...
[tcm_code] at 0x0000050C:0000050C [0x00000684] -> 0xC0201004
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[tcm_code] at 0x00000514:0000050C [0x0000064C] <- 0x28      : was 0x0;
...
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah
[tcm_code] at 0x00000598:00000568 [0x00000648] -> 0x1       
[tcm_code] at 0x000005A0:00000568 [0x00000648] <- 0x0       : was 0x1;

Looks right!

Note: "Enabled interrupt 0Ah" means the interrupt routine finished handling it, so the same interrupt can be triggered again from now on (you can't have the same interrupt nested with itself). The same register configuration happens when an interrupt is enabled for the first time, and that's why the message reads "enabled interrupt". Might be a bit confusing at first sight.

Now that we know what 0x648 does, let's use it as boolean (0 = regular task, nonzero = handling some interrupt):
Code: [Select]
macro define CURRENT_ISR  (*(int*)0x648 ? (*(int*)0x64C) >> 2 : 0)

Next is CURRENT_TASK. One way to find that is by pattern matching in task_create and its subroutines.

I'll try an alternate method, based on one of the recent addition to QEMU: the ability to track function calls and returns. From the ARM calling convention (AAPCS (http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042f/IHI0042F_aapcs.pdf), or the summary from wikipedia (https://en.wikipedia.org/wiki/Calling_convention#ARM_.28A32.29)), you'll notice that registers R4-R11 are supposed to be unchanged after a function call. When tracking function calls, the logging code checks this assumption as well, and will print lots of warnings if a task switch happens without the logger knowing about it. Let's use these warnings to narrow down the location where DryOS switches between tasks:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw

The log is huge, but we are going to look for some warnings about registers (run with only -d callstack to see them without the clutter from RAM writes). These warnings are likely caused by some memory writes performed just before the warnings. We are looking for some invariant address (one that doesn't change with the warnings):
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw |& grep -B 20 "not restored"
...
[ram]      at 0x000019D8:0000E24C [0x0018F8FC] <- 0xE24C    : was 0xE24C;
[ram]      at 0x00001A04:0000E24C [0x0008FBCC] <- 0x17EED4  : was 0x17EF28;
[ram]      at 0xFF0C1064:0000E24C [0x0018F8FC] <- 0xE24C    : was 0xE24C;
...
[ram]      at 0xFF0C1074:0000E24C [0x0018F8C0] <- 0x60000093: was 0x19980218;
[ram]      at 0xFF0C1078:0000E24C [0x0017EF78] <- 0x18F8C0  : was 0x18F8D0;
Warning: R4 not restored (0x17ef28 -> 0x17eed4)                                  at [ff0c1088:e24c]
...
[ram]      at 0x000019D8:0000E24C [0x0018E9A4] <- 0xE24C    : was 0xE24C;
[ram]      at 0x00001A04:0000E24C [0x0008FBCC] <- 0x17EF28  : was 0x17EED4;
[ram]      at 0xFF0C1064:0000E24C [0x0018E9A4] <- 0xE24C    : was 0xE24C;
...
[ram]      at 0xFF0C1074:0000E24C [0x0018E968] <- 0x60000093: was 0x18E9FC;
[ram]      at 0xFF0C1078:0000E24C [0x0017EF24] <- 0x18E968  : was 0x18E970;
Warning: R4 not restored (0x17eed4 -> 0x17ef28)                                  at [ff0c1088:e24c]

Most of these addresses are different for each warning message, except one. Let's try it. Find task_create and DebugMsg (easy) and add everything to EOSM2/debugmsg.gdb (look at other cameras for a template):
Code: [Select]
# ./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

source -v debug-logging.gdb

macro define CURRENT_TASK 0x8FBCC
macro define CURRENT_ISR  (*(int*)0x648 ? (*(int*)0x64C) >> 2 : 0)

b *0x4398
DebugMsg_log

b *0x7360
task_create_log

Code: [Select]
./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
[      init:ff352264 ] task_create(PowerMgr, prio=20, stack=400, entry=ff352088, arg=0)
[      init:ff1470d4 ] (00:01) [PM] DisablePowerSave (Counter = 1)
[      init:0003671c ] task_create(DbgMgr, prio=1f, stack=0, entry=36628, arg=46e584)
[      init:ff0c3334 ] (8b:16)
K355 ICU Firmware Version 1.0.2 ( 6.0.5 )
[      init:ff0c3348 ] (8b:05)
ICU Release DateTime 2013.12.02 09:28:54
[      init:ff0f5ea0 ] (00:03) [SEQ] CreateSequencer (Startup, Num = 6)
[      init:ff0f5f28 ] task_create(Startup, prio=19, stack=2800, entry=ff0f5d6c, arg=46e880)
[      init:ff0f60f4 ] (00:02) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[      init:ff0f6158 ] (00:03) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[      init:ff0c33d4 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28d8, arg=0)
[   Startup:ff0f5db4 ] (00:05) [SEQ] seqEventDispatch (Startup, 0)
[   Startup:ff0c373c ] (8b:05) startupEntry
[   Startup:ff0c375c ] task_create(Startup2, prio=11, stack=400, entry=ff0c3604, arg=0)
[  Startup2:ff1310c4 ] (02:16) PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
[  Startup2:ff14a204 ] (00:01) [SF] IsAddressSerialFlash 0x2d0000
...

We now have task information and debug messages!

To make this information available in QEMU (not just in GDB), add this to model_list.c under the EOSM2 section:
Code: [Select]
        .current_task_addr      = 0x8FBCC,

That's it for today.
Title: Re: ML on EOS-M2
Post by: a1ex on May 28, 2017, 03:13:05 PM
There are other similar blocks copied into RAM (newer DIGIC 6 and 7 models have plenty of those), and finding them manually can become tedious, so it's a good candidate for automation.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d romcpy |& grep ROMCPY
[ROMCPY] 0xFFFF0000 -> 0x0        size 0x40       at 0xFFFF0980
[ROMCPY] 0xFFFE0000 -> 0x100000   size 0xFF2C     at 0xFFFF0FCC
[ROMCPY] 0xFFD1F02C -> 0x1900     size 0xB70A0    at 0xFF0C000C
[ROMCPY] 0xFF0C0E04 -> 0x4B0      size 0x1E8      at 0xFF0C0D70
[ROMCPY] 0xFFA12904 -> 0x4E0E98   size 0xC7C      at 0x8645C   

This should work on all models from DIGIC 2 to DIGIC 7 (both EOS and PowerShot).

We now have task information and debug messages!

... and also an option to show all context switches:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d tasks
...
Task switch to idle:ff0c108c                                                     at [idle:1db8:ff0c1c84]
Task switch to init:ff0c1064                                                     at [init:1a18:c73c]
K355 READY
128K Sector FROM From BL 0xffff
...
SerialFlash Initialize
Task switch to Startup:ff0c108c                                                  at [Startup:1db8:c9e0]
Task switch to Startup2:ff0c1064                                                 at [Startup2:1a18:228c]
...
Task switch to DbgMgr:ff0c1064                                                   at [DbgMgr:1a18:e24c]
     4:    54.784 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
Task switch to Startup:ff0c1064                                                  at [Startup:1a18:2e40]
[RTC] !! RTC_TIME_CORRECT_CHANGE!  0x0 ---> 0x9a
[RTC] !! RTC UNDER MIN_YEAR !!
Task switch to PropMgr:ff0c1064                                                  at [PropMgr:1a18:da2c]
...

8)
Title: Re: ML on EOS-M2
Post by: DeafEyeJedi on May 28, 2017, 05:58:31 PM
All of this is smoking hot guys and have been on the outlook for a used M2 body just because I would like to join the foam party!
Title: Re: ML on EOS-M2
Post by: a1ex on May 29, 2017, 11:47:06 AM
The nice part is that all other models can follow (more or less) the same procedure. Credits go to dfort for getting the tutorials ball rolling (https://www.magiclantern.fm/forum/index.php?topic=19417.0) - that's the reason I've picked this model for a walkthrough.

In other words, owners of other models will no longer have excuses for not knowing how to port ML to their camera :D

Anyway, let's continue figuring out the emulation.

Code: [Select]
        .serial_flash_size      = 0x1000000,

Unfortunately, this step doesn't change anything in the emulation.

Oops, I was wrong here. To quote Ange Albertini - https://twitter.com/angealbertini/status/773650987839926272
Quote
Reverse engineering tip: it's perfectly fine to
- have no idea what to do next
- have made wrong assumptions
- take 'too long'

Here's the detail I've missed the first time:
Code: [Select]
SerialFlash Initialize
[EEPROM-DMA]! [0x2D0000] -> [0x416D5B00] (0x1DC400 bytes)

Also, error messages like this will disappear after declaring .serial_flash_size:
Code: [Select]
     6:    16.640 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x2D0000

That's good - it probably means we have nothing else to fix regarding serial flash. Running with -d sflash shows plenty of serial flash activity (but I swear I did ran this command back then and noticed none of it...)

Let's move on to the next item - MPU messages. You can find the registers used for these in InitializeIntercom (surrounded in the disassembly by "InitializeIntercom" ... "InitializeIntercom End(%#x)"). The function call between these two strings is:
Code: [Select]
ROM:FF0C3A9C                 LDR     R2, =0xC020302C
ROM:FF0C3AA0                 LDR     R0, =0xC022006C
ROM:FF0C3AA4                 MOV     R1, #0x50 ; 'P'
ROM:FF0C3AA8                 BL      sub_3AD0

Looking up these registers, you'll find the former on the section with DIGIC 5 defaults (so nothing to change about it), and the latter on the EOSM, 100D and some other models. Let's copy it:
Code: [Select]
        .mpu_request_register   = 0xC022006C,
        .mpu_status_register    = 0xC022006C,

Side note: we were very lucky, as EOS M2 is very similar to 100D and other models from the same generation. 1300D was a lot more different (used registers different from all other models, different registers for request and status - unlike all other models - and also different values for driving these GPIOs). Refer to this commit (https://bitbucket.org/hudson/magic-lantern/commits/f6951853578016789becca598345dbb6ed29c833) for the gory details - generally you'll find this kind of changes with cameras from different generations. I expect this kind of changes on DIGIC 6 models - didn't look into this part yet because 1) mixed ARM/Thumb code is a lot harder to reverse engineer (at least with the tools I know) and 2) I don't have any MPU communication logs from a DIGIC 6 models (it might work with replaying one from a recent DIGIC 5, no idea).

Still, after enabling the MPU registers, there's very little MPU activity:
Code: [Select]
[MPU] Received: 06 04 02 00 00 00  (unknown spell)
[MPU] Received: 0a 08 03 57 10 2f 00 01 00 00  (unknown spell)
[MPU] Received: 08 07 03 6a 01 08 00 00  (unknown spell)

And these warnings:
Code: [Select]
[MPU] FIXME: no MPU spells for EOSM2.
[MPU] FIXME: no MPU button codes for EOSM2.

Let's try our luck by assuming the MPU communication is similar to 100D. That model already boots Canon GUI, so it looks like a perfect match. Lookup this message in the source code to see what file you should edit next:
Code: [Select]
grep -nr "no MPU spells" qemu-2.5.0/
qemu-2.5.0/hw/eos/mpu.c:742:        MPU_EPRINTF("FIXME: no MPU spells for %s.\n", s->model->name);

So we have to edit mpu.c (near line 742). Open it and notice this similarity trick worked on a few other models (such as 1200D or 1100D). Add an entry for EOS M2, using the 100D as model (look for MPU_SPELL_SET_OTHER_CAM).

Looks like it's working! There's a lot of MPU activity now.

Still, the screen is filled with DL messages. Let's look it up in other models - it's already patched on EOS M:
Code: [Select]
EOSM/debugmsg.gdb:17:# patch DL to avoid DL ERROR messages
EOSM/debugmsg.gdb-18-set *(int*)0xFF1BE4AC = 0xe3a00015

Looks like it might be exactly what we need.

Wait a minute - what's up with these black magic values? The first one is a ROM address (obvious if you managed to follow the walkthrough until here), but what about the second one?

The code near that address, on EOS M, looks like this:
Code: [Select]
ROM:FF1BE4AC                 BL      sub_FF1BDFD4
ROM:FF1BE4B0                 CMP     R0, #0x15

In other words, it calls a function and checks whether it returns 0x15. Therefore, 0xe3a00015 must be some ARM instruction. You could throw it at a disassembler (tip: there's even an online disassembler on Twitter (http://www.capstone-engine.org/bot.html) :) ) or you could find a similar pattern in arm-mcr.h. Whatever method you choose, you'll find out it's MOV R0, #0x15.

Use your pattern matching skills to find the same piece of code on EOS M2 (nearby string: "IsPossibleActiveSweep OVERRUN") and add the patch to EOSM2/debugmsg.gdb.

Run the EOSM2 firmware under GDB and notice it now gets stuck at localI2C_Write. Look it up in the GDB scripts and notice the same thing was (again) patched on EOSM.

Code: [Select]
EOSM/debugmsg.gdb:20:# patch localI2C_Write to always return 1 (success)
EOSM/debugmsg.gdb-21-set *(int*)0xFF3460C4 = 0xe3a00001
EOSM/debugmsg.gdb-22-set *(int*)0xFF3460C8 = 0xe12fff1e

You may ask: why there are no comments for these black magic values?
1) the ROM addresses have a comment (the name of the function they are referring to).
2) these ARM instructions are obvious to me (how else you could write "return 1" in ARM assembler?)
3) the comment does describe what the patch is doing, what function is patched, what's the meaning of the returned value... what else should I include?

OK, find this one for EOSM2 (easy) and run it again under gdb.

Emulation goes a bit further, and you can spot this message on the last few lines:
Code: [Select]
    61:   203.008 [STARTUP] ERROR SerialFlash Version!! Firm : 6.0.3 SF : 4.2.1

Look up the string in the disassembly, and also the code that references it:
Code: [Select]
ROM:FF0C4270                 MOV     R2, #5
ROM:FF0C4274                 ADD     R1, SP, #0x30
ROM:FF0C4278                 ADR     R0, a6_0_3      ; "6.0.3"
ROM:FF0C427C                 BL      sub_FF0C8F40
ROM:FF0C4280                 CMP     R0, #0
ROM:FF0C4284                 BEQ     loc_FF0C42A4

ROM:FF0C4290                 ADR     R3, a6_0_3      ; "6.0.3"
ROM:FF0C4294                 ADR     R2, aErrorSerialfla ; "ERROR SerialFlash Version!! Firm : %s S"...
ROM:FF0C42A0                 BL      DebugMsg
ROM:FF0C42A4
ROM:FF0C42A4 loc_FF0C42A4               

So, to avoid this message, we need sub_FF0C8F40 to return 0. Looking at its arguments, it compares "6.0.3" to something else (something from the stack, likely a local variable), and there's a third argument set to 5. Looking at its code, it appears to do something similar to strncpy. Let's try to bypass this function call and let the caller code behave as if sub_FF0C8F40 would return 0:
Code: [Select]
# skip SerialFlash version check
set *(int*)0xFF0C427C = 0xe3a00000

We have just replaced the BL sub_FF0C8F40 call with MOV R0, #0.

The emulation goes a lot further now, it initializes RscMgr, FileMgr, PropMgr, starts a bunch of tasks and... gets stuck in an infinite loop at:
Code: [Select]
[   Startup:ff0c5138 ] (8b:03) Wait LeoLens Complete

Look up the string in the disassembly, and notice the decision for exiting this loop is made here:
Code: [Select]
ROM:FF0C512C loc_FF0C512C                            ; CODE XREF: sub_FF0C4E80+2CCj
ROM:FF0C512C                 ADR     R2, aWaitLeolensCom ; "Wait LeoLens Complete"
...
ROM:FF0C5144                 LDR     R0, [R4,#0x28]
ROM:FF0C5148                 CMP     R0, #0
ROM:FF0C514C                 BEQ     loc_FF0C512C
...

So, we need R0 to be nonzero in order to avoid the branch that prints "Wait LeoLens Complete". The memory address where R0 is loaded from is R4 + 0x28. What the contents of R4 is supposed to be, is not clear from the disassembly. Hit CTRL-C in gdb and set a breakpoint at FF0C5148 to find out:
Code: [Select]
(gdb) b *0xFF0C5148
Breakpoint 3 at 0xff0c5148
(gdb) c
Continuing.

Shortly, gdb prompts us again, this time with the program stopped exactly where the loop condition is checked. Examine the registers:
Code: [Select]
Breakpoint 3, 0xff0c5148 in ?? ()
(gdb) info registers
r0             0x0 0x0
r4             0x8fad4 0x8fad4

So, the memory address the program expects to become nonzero is 0x8fad4+0x28 = 0x8fafc. Look up what other code uses this address (references to either 0x8fad4 - the main data structure - or 0x8fafc - the address of the structure field expected to be updated) and you'll find out this tiny function:
Code: [Select]
ROM:FF0C35F4                 LDR     R1, =dword_8FAD4
ROM:FF0C35F8                 MOV     R0, #1
ROM:FF0C35FC                 STR     R0, [R1,#(dword_8FAFC - 0x8FAD4)]
ROM:FF0C3600                 BX      LR

OK, so the value at this memory address is expected to become 1 at some point.

Who calls this function? Look it up (references to FF0C35F4) and you'll find this snippet:
Code: [Select]
ROM:FF0C3B30                 LDR     R0, =loc_FF0C35F4
ROM:FF0C3B34                 MOV     R1, #0
ROM:FF0C3B38                 BL      sub_FF0F7B18
...
ROM:FF0C3B44                 ADRNE   R2, aInitializeprop ; "InitializePropertySatellite (%#x)"

It must be a tiny function passed as pointer to another function (which is why it wasn't recognized as function by IDA - it's not called directly). When it's supposed to be called? No idea, probably from a missing property that's not present in the serial flash dump we took from 100D. Just a guess.

The property code is usually quite complex - in this case it's a lot easier to just patch the affected loop and hope it will work. If the value would have been clearly retrieved from some MMIO register, or set by some interrupt handler, it would probably have been better to emulate that register or that interrupt (because, on the way to that function expected to be called, the code may do a bunch of other initializations, side effects and whatnot).

In this case, let's just patch the value in gdb and see how it goes:
Code: [Select]
(gdb) set *(int*)($r4 + 0x28) = 1
(gdb) c
Continuing.
[   Startup:ff0c5138 ] (8b:03) Wait LeoLens Complete

Breakpoint 3, 0xff0c5148 in ?? ()
(gdb) c
Continuing.
[   Startup:ff0e29a8 ] (30:03) MOVW_Initialize
[   Startup:0003671c ] task_create(MovWriter, prio=15, stack=2000, entry=36628, arg=67262c)
[   Startup:ff0e6f34 ] (2f:03) MVR_Initialize
[   Startup:0003671c ] task_create(MovieRecorder, prio=11, stack=2000, entry=36628, arg=672d4c)

Looks like the emulation went past that point. Let's add our trick to the gdb script:
Code: [Select]
# break infinite loop at Wait LeoLens Complete
b *0xFF0C5148
commands
  printf "Patching LeoLens (infinite loop)\n"
  set *(int*)($r4 + 0x28) = 1
  c
end

The emulation goes further, mounts the SD card (CSMgrTask), starts even more tasks, and appears to be stuck at:
Code: [Select]
   167:   851.456 [TCH]Touch IC Ver : 0x0000

At this point, seeing there's no progress after 10 seconds or so, you'll be tempted to hit CTRL-C. Or you can forget the emulation window open (e.g. for posting gibberish like this on the forum), and when you hit ALT+TAB you notice this:
Code: [Select]
  1383: 57907.968 WARN [I2C] localI2C_Read : 378 (Task : CEC)
  1384: 57927.680 WARN [I2C] localI2C_Read : 378 (Task : CEC)
  1385: 57947.392 WARN [I2C] localI2C_Read : 378 (Task : CEC)
  1386: 57966.592 WARN [I2C] localI2C_Read : 378 (Task : CEC)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/M2-gui.png)

Wait, what?

Trying to navigate this menu gives no reaction. We forgot to address this warning (at the beginning of the emulation log):
Code: [Select]
[MPU] FIXME: no MPU button codes for EOSM2.

Use the button codes from 100D and - to quote kichetof - take a beer and enjoy it!

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/M2-qemu.gif)

Tip: comment out DebugMsg calls from the GDB script for a much faster boot.

Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop;
- (easy) print Hello World with the minimal ML target;
- (moderate) print Hello World with the regular ML source code;
- (moderate) implement touch screen emulation;
- (moderate) bring ML menu in QEMU;
- (harder) full ML port, with most features working;
- (extreme) LiveView emulation.
Title: Re: ML on EOS-M2
Post by: yazcui on May 29, 2017, 07:12:06 PM
M2 owner here. All this jargon is way over my head, but I'm thrilled to learn that ML may be available for the M2 soon. Thanks for your efforts!
Title: Re: ML on EOS-M2
Post by: dfort on June 03, 2017, 07:02:50 AM
I'm trying to follow using the tools I've got available to me. QEMU on a Mac (seems to be working), ARM-console on Mac (sort of half-way works), good ol' disassemble.pl and a fine arts degree (?).

I'm on firmware version 1.0.3 so my numbers don't quite match up to the 1.0.2 dump a1ex is using.

...One of these covers the functions we are interested in:
Code: [Select]
[rom1]     at 0xFF0C000C:FFFF09CC [0xFFD1F02C] -> 0xE92D4010
[ram]      at 0xFF0C000C:FFFF09CC [0x00001900] <- 0xE92D4010: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F030] -> 0xE1A04001
[ram]      at 0xFF0C00A4:FFFF09CC [0x00001904] <- 0xE1A04001: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F034] -> 0xE59F1040
...
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFDD60C8] -> 0xFFD1EFFC
[ram]      at 0xFF0C00A4:FFFF09CC [0x000B899C] <- 0xFFD1EFFC: was 0x0;

In other words, a memory block from 0xFFD1F02C to 0xFFDD60C8 (actually 0xFFDD60CB, since we are looking at 32-bit operations) is copied to 0x1900 - 0xB899F. The size of the copied block is 0xb70a0 bytes. To extract this block, grab the terminal and run this under a Bash prompt:

Code: [Select]
dd if=ROM1.BIN of=1900.BIN bs=1 skip=$((0xD1F02C)) count=$((0xB70A0))

(please note the above numbers are only valid for firmware 1.0.2)

Now you can disassemble this file starting from 0x1900. In IDA, load this file as Additional binary file. If you use ARM-console, this should auto-detect the above copying operation (so you can simply start browsing the functions copied into RAM).

This is the same bit in 1.0.3:

Code: [Select]
[rom1]     at 0xFF0C000C:FFFF09CC [0xFFD1F0E4] -> 0xE92D4010
[ram]      at 0xFF0C000C:FFFF09CC [0x00001900] <- 0xE92D4010: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F0E8] -> 0xE1A04001
[ram]      at 0xFF0C00A4:FFFF09CC [0x00001904] <- 0xE1A04001: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F0EC] -> 0xE59F1040
...
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFDD6180] -> 0xFFD1F0B4
[ram]      at 0xFF0C00A4:FFFF09CC [0x000B899C] <- 0xFFD1F0B4: was 0x0;

I see a memory block from 0xFFD1F0E4 to 0xFFDD6180. I'm not sure how you came up with 0xFFDD60CB from 0xFFDD60C8 to make it conform to a 32-bit operation. I tried looking it up but don't quite get it.

Moving on, 0xFFDD6180 - 0xFFD1F0E4 = 0xB709C. It starts at 0x1900 and we need to calculate the end (0x1900 + 0xB709C = 0xB899C) so it is copied to 0x1900 - 0xB899C. The size of the copied block is 0xB709F but it needs to be rounded up to be evenly divisible by 32 (0x20) right? So if I followed the lesson plan properly it is also 0xB70A0 just like the 1.0.2 firmware.

Code: [Select]
dd if=ROM1.BIN of=EOSM2.103.0x1900.BIN bs=1 skip=$((0xD1F0E4)) count=$((0xB70A0))
That's the name I used for ARM-console to auto-detect. I also did a "perl disassemble.pl 0x1900 EOSM2.103.0x1900.BIN" in order to get a "second opinion" and both methods seem to be in agreement.

[EDIT] Set up a branch to start the port (https://bitbucket.org/daniel_fort/magic-lantern/branch/EOSM2.103_wip). First step was just to create an EOSM.202 build but call it EOSM2.103.
Title: Re: ML on EOS-M2
Post by: a1ex on June 03, 2017, 08:44:44 AM
I'm not sure how you came up with 0xFFDD60CB from 0xFFDD60C8 to make it conform to a 32-bit operation. I tried looking it up but don't quite get it.

A 32-bit read from 0xFFDD60C8 actually means 4 bytes from that address: 0xFFDD60C8, 0xFFDD60C9, 0xFFDD60CA, 0xFFDD60CB.

Quote
The size of the copied block is 0xB709F but it needs to be rounded up to be evenly divisible by 32 (0x20) right? So if I followed the lesson plan properly it is also 0xB70A0 just like the 1.0.2 firmware.

Nothing is rounded up; this is the exact size. Copying 4 bytes at a time cannot give you an odd number of bytes ;)
Title: Re: ML on EOS-M2
Post by: dfort on June 05, 2017, 03:45:14 PM
Alright, so let's start from where we've left off.

I'm going through this slowly and trying to make sure my results are matching up. Some things aren't quite the same which is to be expected because I'm using a different firmware version (1.0.3) and another reason may be because I'm QEMU on a Mac which might not be set up quite right just yet.

For the debugmsg.gdb file, we'll need some basic stubs: DebugMsg and task_create (easy to find - one is used for most debug messages, the other can be easily identified from things that look like task names) and two addresses: CURRENT_TASK and CURRENT_ISR (interrupt service routine). We can find the latter in the interrupt handler - quote from debugmsg.gdb:
If you can find that by examining disassembled code, great. If not, QEMU to the rescue:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,nochain -singlestep
...
Taking exception 5 [IRQ]
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah

(note: -d nochain -singlestep gives more precise locations in I/O logs - otherwise, QEMU groups the ARM instructions in blocks for faster execution, so the reported location will be approximate)

First of all I went through the disassembly and as best as I can determine the stubs that we're using haven't changed between the firmware versions. Great! However, I am seeing different results.
Code: [Select]
(gdb) disas 0x4B0,0x6B0[EOS] trigger int 0x0A (delayed)
Taking exception 5 [IRQ]
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[TIMER]    at 0xFF350400:000072A4 [0xC0242014] -> 0x6E900   : DIGIC clock
[INT]      at 0x00000594:FF0C1578 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah

The clock is running continuously so I suppose that can be disregarded but the difference between 0x00000568 and 0xFF0C1578 isn't exactly "approximate."

OK, so the register we are looking for is read at 0x510. Let's disassemble with GDB:
Code: [Select]
(gdb) disas 0x4B0,0x6B0
...
   0x0000050c: ldr r0, [pc, #368] ; 0x684
   0x00000510: ldr r4, [r0]
   0x00000514: str r4, [pc, #304] ; 0x64c
...
(gdb) x 0x684
0x684: 0xc0201004
(gdb) x 0x64c
0x64c: 0x00000028
(gdb) print 0x28>>2
$1 = 0xa

Hit a wall here because I'm not seeing anything at all like that.

I've been updating the EOSM2 preliminary setup (https://bitbucket.org/hudson/magic-lantern/pull-requests/835/eosm2-preliminary-setup/diff) pull request as I work my way through this.
Title: Re: ML on EOS-M2
Post by: a1ex on June 05, 2017, 05:06:05 PM
You need to type the gdb commands at the gdb prompt. You'll need to start qemu like this:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

Of course, you can mix gdb with other options freely. In the above invocation, gdb will be the foreground process (the one you interact with on the console); if you swap their order, QEMU will be the foreground process (which doesn't do anything interesting by default, unless you redirect its monitor to console, with -monitor stdio).

The DIGIC clock line appeared after me fine-tuning the verbosity (so it's OK). Interrupts are not deterministic (since they use the host's wall clock as a reference, so the results depend a lot on the emulation speed, and you will not get the same code sequence on two different runs. The same happens on real hardware (run the startup-log twice and compare the logs).

Besides, the execution trace within a timer interrupt also depend on whatever DryOS decides to do in that interrupt (such as switching tasks or running some timer event). To see this particular case more clearly, try:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,nochain -singlestep |& grep "Enabled interrupt 0Ah"
Title: Re: ML on EOS-M2
Post by: dfort on June 07, 2017, 08:54:50 PM
Hope I'm not wearing out my QEMU welcome -- still stuck here:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & arm-none-eabi-gdb -x debug-logging.gdb
(note: we will need to create a GDB file for EOS M2, named EOSM2/debugmsg.gdb to match other models, but since we don't have one yet, we can just use the generic version - which we are going to include in the platform-specific file later)

Let it run for a while (continue), then hit CTRL-C and examine the exception vector (located at address 0):

Code: [Select]
(gdb) disas 0,0x40
Dump of assembler code from 0x0 to 0x40:
   0x00000000: nop ; (mov r0, r0)
   0x00000004: ldr pc, [pc, #20] ; 0x20
   0x00000008: ldr pc, [pc, #20] ; 0x24
...

The line in red is what is tripping me up, specifically the (continue). How am I supposed to do that?

Code: [Select]
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[EOS] loading './EOSM2/SFDATA.BIN' as serial flash, size=0x1000000
[MPU] FIXME: no MPU spells for EOSM2.
[MPU] FIXME: no MPU button codes for EOSM2.
Setting BOOTDISK flag to 0
continue
-bash: continue: only meaningful in a `for', `while', or `until' loop

So that's not right. QEMU on Mac has a "Machine" menu item where you can "Pause" and "Resume" and that shows some more action in the terminal but CTRL-C always gets me back to the bash prompt--shouldn't I be getting the (gdb) prompt? Launching gdb and continuing your instructions always results in:

Code: [Select]
(gdb) disas 0,0x40
Dump of assembler code from 0x0 to 0x40:
   0x00000000: Cannot access memory at address 0x0

This might be a Macintosh issue but more likely a problem with a clueless user.
Title: Re: ML on EOS-M2
Post by: a1ex on June 07, 2017, 09:05:23 PM
After typing that long command, I get the (gdb) prompt. Type "continue" there.

Selecting "Pause" from QEMU menu also gets me to the (gdb) prompt.
Title: Re: ML on EOS-M2
Post by: dfort on June 08, 2017, 12:57:32 AM
I don't get that at all, that long command launches QEMU. I do see an error message flash by but terminal isn't show the full history. I redirected the error messages like this
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & arm-none-eabi-gdb -x debug-logging.gdb 2> error.txt
and found this:
Code: [Select]
-bash: arm-none-eabi-gdb: command not found
Seems like I've got a configuration issue. It appears to be in the right place.
Code: [Select]
/Users/rosiefort/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gdb
Title: Re: ML on EOS-M2
Post by: dfort on June 08, 2017, 08:29:47 AM
If I make that long command even longer, it works:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & ~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gdb -x debug-logging.gdb
Is there supposed to be a symlink in ~/qemu for arm-none-eabi-gdb?
Title: Re: ML on EOS-M2
Post by: a1ex on June 08, 2017, 10:30:49 AM
This may help: https://unix.stackexchange.com/questions/183295/adding-programs-to-path
Title: Re: ML on EOS-M2
Post by: vagaboundedu on June 14, 2017, 02:18:37 AM
Just wanted you all to know that I created an account on this site specifically to say I'm really looking forward to being able to install ML on my M2! Can't wait! Thank you so much for all your work.
Title: Re: ML on EOS-M2
Post by: dfort on June 18, 2017, 11:49:29 PM
Thought I'd make a little noise on this topic just to let people know that I'm still working my way through this albeit at a very slow pace.

QEMU is under active development so before posting anything I pull in the latest commits and merge my EOSM2 and Mac specific build tweaks.

Ok, so here's where I left off:

Alright, so let's start from where we've left off.
...
This 0x648 looks like a counter that tells how many nested interrupts we are handling (yes, they can be nested - unfortunately, as this makes it a lot harder to understand, debug, emulate and so on). Let's confirm its functionality with RAM tracing:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,ram,nochain -singlestep
...
[tcm_code] at 0x000004D4:19980218 [0x00000648] -> 0x0       
[tcm_code] at 0x000004E0:19980218 [0x00000648] <- 0x1       : was 0x0;
...
[tcm_code] at 0x0000050C:0000050C [0x00000684] -> 0xC0201004
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[tcm_code] at 0x00000514:0000050C [0x0000064C] <- 0x28      : was 0x0;
...
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah
[tcm_code] at 0x00000598:00000568 [0x00000648] -> 0x1       
[tcm_code] at 0x000005A0:00000568 [0x00000648] <- 0x0       : was 0x1;

Looks right!

I'm seeing pretty much the same only with some additional information probably as a result of some of the enhancements a1ex has been adding to QEMU.

Code: [Select]
...
[tcm_code] at TaskM:000004D4:19980218 [0x00000648] -> 0x0       
[tcm_code] at TaskM:000004E0:19980218 [0x00000648] <- 0x1       : was 0x0;
...
[tcm_code]  at init:0000050C:0000050C [0x00000684] -> 0xC0201004
[INT]       at init:00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[tcm_code]  at init:00000514:0000050C [0x0000064C] <- 0x28      : was 0x0;
...
[INT]       at init:00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah
[tcm_code]  at init:00000598:00000568 [0x00000648] -> 0x1       
[tcm_code]  at init:000005A0:00000568 [0x00000648] <- 0x0       : was 0x1;

In some cases I'm getting errors that shouldn't be happening. For example:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw |& grep -B 20 "not restored"
-bash: syntax error near unexpected token `&'

The '|&' shorthand was added to bash 4 as an abbreviation for '2>&1 |' and I'm running this version of bash:

Code: [Select]
GNU bash, version 4.4.12(1)-release (x86_64-apple-darwin16.3.0)
Go figure -- must be a Mac thing. Changing '|&' to '2>&1 |' still didn't give me the same results a1ex posted.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw 2>&1 | grep -B 20 "not restored"
[tcm_data] at 0x0010DA10:00000040 [0x40000F60] <- 0x4       : was 0x4C;
[tcm_data] at 0x0010DA10:00000040 [0x40000F64] <- 0x20      : was 0x4C;
[tcm_data] at 0x0010DA10:00000040 [0x40000F68] <- 0x8       : was 0x4C;
[tcm_data] at 0x0010DA10:00000040 [0x40000F6C] <- 0x40      : was 0x1F;
[tcm_data] at 0x0010DA08:00000100 [0x40000F70] <- 0x10      : was 0x4C;
[tcm_data] at 0x0010DA08:00000100 [0x40000F74] <- 0x80      : was 0x4C;
[tcm_data] at 0x0010DA08:00000100 [0x40000F78] <- 0x20      : was 0x40;
[tcm_data] at 0x0010DA08:00000100 [0x40000F7C] <- 0x100     : was 0x40;
[tcm_data] at 0x0010DA10:00000C00 [0x40000F80] <- 0x40      : was 0x1F;
[tcm_data] at 0x0010DA10:00000C00 [0x40000F84] <- 0x200     : was 0x1;
[tcm_data] at 0x0010DA10:00000C00 [0x40000F88] <- 0x80      : was 0x0;
[tcm_data] at 0x0010DA10:00000C00 [0x40000F8C] <- 0xC00     : was 0x0;
[tcm_data] at 0x0010DA2C:00000C00 [0x40000F90] <- 0xFF      : was 0x7;
[tcm_data] at 0x0010DA2C:00000C00 [0x40000F94] <- 0xFFF     : was 0x1;
[tcm_data] at 0x0010329C:001000C8 [0x40000F9C] <- 0x1       : was 0x0;
[tcm_data] at 0x0010329C:001000C8 [0x40000FA0] <- 0x1000C8  : was 0x10007C;
[tcm_data] at 0x0010329C:001000F8 [0x40000F9C] <- 0x1       : was 0x1;
[tcm_data] at 0x0010329C:001000F8 [0x40000FA0] <- 0x1000F8  : was 0x1000C8;
[tcm_data] at 0x001022C0:0010013C [0x40000F9C] <- 0x1       : was 0x1;
[tcm_data] at 0x001022C0:0010013C [0x40000FA0] <- 0x10013C  : was 0x1000F8;
Warning: R10 not restored (0xa -> 0x1)                                           at [ffff101c:ffff09cc]

Moving on or I'll never get through this. Here we can finally see some indication of the different firmware versions that we're looking at, a1ex on EOSM2 1.0.2 while I'm on 1.0.3.


Code: [Select]
./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
[      init:ff352264 ] task_create(PowerMgr, prio=20, stack=400, entry=ff352088, arg=0)
[      init:ff1470d4 ] (00:01) [PM] DisablePowerSave (Counter = 1)
[      init:0003671c ] task_create(DbgMgr, prio=1f, stack=0, entry=36628, arg=46e584)
[      init:ff0c3334 ] (8b:16)
K355 ICU Firmware Version 1.0.2 ( 6.0.5 )
[      init:ff0c3348 ] (8b:05)
ICU Release DateTime 2013.12.02 09:28:54
[      init:ff0f5ea0 ] (00:03) [SEQ] CreateSequencer (Startup, Num = 6)
[      init:ff0f5f28 ] task_create(Startup, prio=19, stack=2800, entry=ff0f5d6c, arg=46e880)
[      init:ff0f60f4 ] (00:02) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[      init:ff0f6158 ] (00:03) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[      init:ff0c33d4 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28d8, arg=0)
[   Startup:ff0f5db4 ] (00:05) [SEQ] seqEventDispatch (Startup, 0)
[   Startup:ff0c373c ] (8b:05) startupEntry
[   Startup:ff0c375c ] task_create(Startup2, prio=11, stack=400, entry=ff0c3604, arg=0)
[  Startup2:ff1310c4 ] (02:16) PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
[  Startup2:ff14a204 ] (00:01) [SF] IsAddressSerialFlash 0x2d0000
...

And my results running that same command.

Code: [Select]
./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
[      init:ff352260 ] task_create(PowerMgr, prio=20, stack=400, entry=ff352084, arg=0)
[      init:ff1470d0 ] (00:01) [PM] DisablePowerSave (Counter = 1)
[      init:0003671c ] task_create(DbgMgr, prio=1f, stack=0, entry=36628, arg=46e584)
[      init:ff0c3334 ] (8b:16)
K355 ICU Firmware Version 1.0.3 ( 6.0.6 )
[      init:ff0c3348 ] (8b:05)
ICU Release DateTime 2016.08.09 13:28:42
[      init:ff0f5e9c ] (00:03) [SEQ] CreateSequencer (Startup, Num = 6)
[      init:ff0f5f24 ] task_create(Startup, prio=19, stack=2800, entry=ff0f5d68, arg=46e880)
[      init:ff0f60f0 ] (00:02) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[      init:ff0f6154 ] (00:03) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[      init:ff0c33d4 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28d8, arg=0)
[   Startup:ff0f5db0 ] (00:05) [SEQ] seqEventDispatch (Startup, 0)
[   Startup:ff0c373c ] (8b:05) startupEntry
[   Startup:ff0c375c ] task_create(Startup2, prio=11, stack=400, entry=ff0c3604, arg=0)
[  Startup2:ff1310c0 ] (02:16) PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
[  Startup2:ff14a200 ] (00:01) [SF] IsAddressSerialFlash 0x2d0000
...

Crawling along to a1ex's next post I ran into the '|&' issue again.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d romcpy |& grep ROMCPY
[ROMCPY] 0xFFFF0000 -> 0x0        size 0x40       at 0xFFFF0980
[ROMCPY] 0xFFFE0000 -> 0x100000   size 0xFF2C     at 0xFFFF0FCC
[ROMCPY] 0xFFD1F02C -> 0x1900     size 0xB70A0    at 0xFF0C000C
[ROMCPY] 0xFF0C0E04 -> 0x4B0      size 0x1E8      at 0xFF0C0D70
[ROMCPY] 0xFFA12904 -> 0x4E0E98   size 0xC7C      at 0x8645C   

Only this time substituting '2>&1 |' worked, sort of. Seems it ended a little earlier.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d romcpy 2>&1 | grep ROMCPY
[ROMCPY] 0xFFFF0000 -> 0x0        size 0x40       at 0xFFFF0980
[ROMCPY] 0xFFFE0000 -> 0x100000   size 0xFF2C     at 0xFFFF0FCC
[ROMCPY] 0xFFD1F0E4 -> 0x1900     size 0xB70A0    at 0xFF0C000C
[ROMCPY] 0xFF0C0E04 -> 0x4B0      size 0x1E8      at 0xFF0C0D70
Binary file (standard input) matches
Title: Re: ML on EOS-M2
Post by: a1ex on June 19, 2017, 12:10:34 AM
Quote
Changing '|&' to '2>&1 |' still didn't give me the same results a1ex posted.

These warnings should only appear if .current_task_addr is not defined. There are lots of them, on all models.

Quote
Binary file (standard input) matches

Maybe grep gets confused by ANSI color codes. Adding ansi2txt to the pipeline, or using grep --text, may help.
Title: Re: ML on EOS-M2
Post by: dfort on June 19, 2017, 02:05:14 AM
As I suspected the bash issue was a Mac thing.

Code: [Select]
echo $BASH_VERSION
3.2.57(1)-release

Hold on there, what happened to bash 4 that we installed via Homebrew a while back? That works with the shell scrips but the default shell was still the old BSD version. To make GNU bash the default needs an adjustment of the terminal preferences.

(https://c1.staticflickr.com/5/4228/35392882555_3285a9895b_z.jpg)

The default login was still going to /bin/bash but the new shell is in /usr/local/bin. I've got "Shell opens with: Default login shell" picked and it is now going to the new bash.

Code: [Select]
echo $BASH_VERSION
4.4.12(1)-release

And of course the '|&' shortcut is now working.

I also got the full gcc-arm-none-eabi toolset working by doing a:

Code: [Select]
ln -s ~/gcc-arm-none-eabi-4_8-2013q4/bin/* /usr/local/bin/
Lots of gotchas running QEMU on the Mac. Someday I should put them all together into a tutorial.


Title: Re: ML on EOS-M2
Post by: vagaboundedu on June 19, 2017, 03:00:19 AM
Been checking back everyday and was excited to see the updates today. Thanks so much for your continued work on this!
Title: Re: ML on EOS-M2
Post by: dfort on June 19, 2017, 06:52:38 AM
I was getting excited because I was on the post where QEMU was showing the camera's UI but I got nothing yet -- probably because I neglected to understand this part:

Anyway, let's continue figuring out the emulation.
Code: [Select]
        .serial_flash_size      = 0x1000000,Oops, I was wrong here.

I wasn't really sure what to do here. We are using the SFDATA.BIN from a 100D and added ".serial_flash_size      = 0x1000000," to the new EOSM2 section in "qemu/qemu-2.5.0/hw/eos/model_list.c" -- right? So maybe this isn't right and I need to change the size to something else? Do I also need to somehow "fix" the SFDATA.BIN?

Taking a hint I did a QEMU run with "-d sflash" like this:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d sflash
...
K355 READY
128K Sector FROM From BL 0xffff
[SF] InstallSerialFlash 6 0xc022c0d4 0x0 0x1000000 1

[SF] Bufcon Base 0xc022c0d4
[EEPROM] CS = 1
SerialFlash Initialize
[EEPROM] CS = 0
[EEPROM] Verbose: Got READ (03h)
[EEPROM] Verbose: address is now: 0x2D0000
...
     0:     3.072 [STARTUP]
K355 ICU Firmware Version 1.0.3 ( 6.0.6 )
     4:     3.328 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
     5:     3.840 [PROPAD] SerialFlash Packages!! 0x2d0000
[SFIO] sfio_trigger_int_DMA
[SFIO] eos_handle_sfio (copying now)
[EEPROM-DMA]! [0x2D0000] -> [0x416D5B00] (0x1DC400 bytes)
...
     6:     9.216 [PROPAD] SerialFlash Packages!! 0x7
     7:     9.216 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x41435b00
     8:     9.728 [PROPAD] SerialFlash Packages!! 0x10000
[SFIO] sfio_trigger_int_DMA
[SFIO] eos_handle_sfio (copying now)

At first I skipped over this and continued to set up QEMU for the EOSM2 but when it didn't work out as I hoped I returned to the part I didn't quite understand. I also committed the changes I made to my QEMU EOSM2 pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/835/qemu-eosm2-preliminary-setup/diff) even though it isn't working yet. Maybe someone could check if I'm on the right track?  ;D

I also committed some more changes to the build tweaks (https://bitbucket.org/hudson/magic-lantern/pull-requests/836/qemu-build-tweaks-2/diff) I've been doing to get QEMU on Mac running a little smoother.
Title: Re: ML on EOS-M2
Post by: a1ex on June 19, 2017, 09:43:34 AM
The serial flash activity looks alright. The issue is at MPU messages.

Simply adding files to the repository will not change anything - they must be included somehow in the build process. For example, if it's a .c file, generally you'll need to include the corresponding object file into the executable (example (https://bitbucket.org/hudson/magic-lantern/commits/a23f3fe2b470a2b5a155f87d18354bafd29263f6#chg-contrib/qemu/eos/Makefile.objs)). If it's a .h file, you'll need to #include it somewhere. And once you have done that, you usually have to reference the new stuff somehow (call some functions or use some data structures from the new files), unless these files are used to replace or override some older definitions. That's generally how it works in a C program.

QEMU does something interesting here - any function or variable must be used somehow (unless it's a global API - in this case it requires a prototype - or an inline function), otherwise you get a compile error. So, if you define button_codes_EOSM2[], but you don't reference it from anywhere, you should get this:
Code: [Select]
error: ‘button_codes_EOSM2’ defined but not used [-Werror=unused-variable]
 static int button_codes_EOSM2[] = {
            ^

In this case, you don't even have to create new files or new definitions. Simply look in mpu.c to see how it's done on other models.
Title: Re: ML on EOS-M2
Post by: dfort on June 19, 2017, 04:17:45 PM
Simply look in mpu.c to see how it's done on other models.

I actually did that but forgot to commit mpu.c. Fixed that on the pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/835/qemu-eosm2-preliminary-setup/diff#chg-contrib/qemu/eos/mpu.c). I'm juggling three branches, the Mac specific build tweaks, the EOSM2 setup and a working branch where these two branches are merged. I'm also merging in changes from the main qemu branch to keep everything up to date. When I want to save a change I made in the working branch I commit the change in either build tweaks or EOSM2 so the pull requests get updated then merge back into the working branch, rinse and repeat.

Whatever problem I'm experiencing probably isn't because I'm on a Mac because QEMU seems to be running properly. I saved the terminal outputs for a fresh QEMU build and run in PASTEBIN in order to take a closer look at the warnings and error messages.

install.sh (https://pastebin.com/C9TMEPJU)
configure_eos.sh (https://pastebin.com/sAfzVrQU)
make -j8 (https://pastebin.com/3ybMCSzY)
run_canon_fw.sh EOSM2 (https://pastebin.com/rMGV4mVK) (The terminal buffer resets so some of the history is lost. Don't know if I'm using the right terminology--no pun intended.)

I would think that running "run_canon_fw.sh EOSM2" would show the minimal autoexc.bin output like on the EOSM but maybe not? I've been able to follow most of the walk through except for the part where QEMU displays the camera's UI. If I can get past that hump I would think that the next step would be to compile a "hello world" autoexec.bin to run in QEMU then figure out how to turn on the camera boot flag to run "hello world" in the camera--or not, I may be jumping an important step here.

Title: Re: ML on EOS-M2
Post by: a1ex on June 19, 2017, 05:11:16 PM
All OK so far - just keep following the walkthrough.

The terminal reset is intended, as it allows you to scroll to the beginning of the log. Would be nice to print the command line after clearing the screen.
Title: Re: ML on EOS-M2
Post by: dfort on June 20, 2017, 01:52:04 AM
Seems oh so close.

EOSM2/debugmsg.gdb
Code: [Select]
# patch localI2C_Write to always return 1 (success)
set *(int*)0xFF356DE0 = 0xe3a00001
set *(int*)0xFF356DE8 = 0xe12fff1e

Not 0xe12fff1e, right? Having a tough time with these black magic numbers. I can sort of see how 0xe3a00001 returns 1 but nothing jumps out at me on that second value.
Title: Re: ML on EOS-M2
Post by: dfort on June 20, 2017, 07:43:49 AM
Thanks for the hints a1ex. Fortunately I saved a dump of the old 1.0.2 firmware so I could check my addresses against yours. I was close which would have counted for something in horseshoes but not here. Haven't managed to find my way around the menus but I did change the time date and place to document this historic event.  8)

(https://c1.staticflickr.com/5/4199/34577687884_5333a32e27_z.jpg)

Title: Re: ML on EOS-M2
Post by: dfort on June 20, 2017, 03:49:55 PM
Ok, so I got menus:

(https://c1.staticflickr.com/5/4205/35295962291_bc58d17aec_z.jpg)

Should the next goal be to boot into ML like the EOSM?

(https://c1.staticflickr.com/5/4270/35038642510_b6d8767934_z.jpg)

Quote
Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop;
- (easy) print Hello World with the minimal ML target;
- (moderate) print Hello World with the regular ML source code;
- (moderate) implement touch screen emulation;
- (moderate) bring ML menu in QEMU;
- (harder) full ML port, with most features working;
- (extreme) LiveView emulation.
Title: Re: ML on EOS-M2
Post by: a1ex on June 20, 2017, 04:25:13 PM
The second screen should be as easy as specifying "boot=1".

Reason: the ROM dumped from EOS M2 does not have the boot flag enabled, while the ROM dumped from EOS M probably has it.

At this point, we have everything we need to port ML and debug it in the emulator. At least, things like Hello World or ML menu navigation will work without requiring additional changes to QEMU.
Title: Re: ML on EOS-M2
Post by: dfort on June 21, 2017, 06:49:39 AM
The second screen should be as easy as specifying "boot=1".

Yes indeed. Strange thing is that this is an EOSM2 running firmware version 1.0.3 but it is showing unknown with 1.0.2.

(https://c1.staticflickr.com/5/4256/35052267590_bee58b9b8b_z.jpg)

Working on a virtual camera. This will be interesting.
Title: Re: ML on EOS-M2
Post by: a1ex on June 21, 2017, 10:06:33 AM
That comes from the way properties are stored in memory: there are many setting blocks, and only one of them is active. The property parser (https://bitbucket.org/hudson/magic-lantern/src/recovery/src/prop_diag.c) from the recovery branch is was not very smart (it doesn't look at whether a block is active or not - it just parses everything). So, it takes the values from the last block in memory that has them (not necessarily the active one).

It probably takes a couple of restarts after the firmware upgrade to replace all the old blocks with 1.0.3.

Updated prop_diag to only parse the active blocks (recovery branch). You can build the display test from that branch, from platform/portable.000 (comment out the ROM dumping routine and cache disabling tricks and that's the plain display test).

Tip: compiling the same branch from some camera directory will give additional capabilities (such as LED blinking), but the binary will no longer be portable.
Title: Re: ML on EOS-M2
Post by: dfort on June 21, 2017, 04:05:07 PM
Updated prop_diag to only parse the active blocks (recovery branch). You can build the display test from that branch, from platform/portable.000...

Looks good with the latest recovery autoexec.bin.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=1"(https://c1.staticflickr.com/5/4232/35278925112_5a8d253997.jpg)

It also created a fresh ROM dump in QEMU:
(https://c1.staticflickr.com/5/4245/35058581330_6599c62ee0.jpg)

However, the MD5 values from the "original" ROM1 and the new QEMU dump don't match. Is this expected?

Code: [Select]
MD5 (/Users/rosiefort/qemu/EOSM2/ROM0.BIN) = f1712be00d98f8cea522f32da216cf96
MD5        (/Volumes/EOS_DIGITAL/ROM0.BIN) = f1712be00d98f8cea522f32da216cf96

MD5 (/Users/rosiefort/qemu/EOSM2/ROM1.BIN) = 3788c332c4a8b2a6b3f521cc92c508ce
MD5        (/Volumes/EOS_DIGITAL/ROM1.BIN) = 5a78ec64480d11275dd064da2cedd136
Title: Re: ML on EOS-M2
Post by: vagaboundedu on June 21, 2017, 07:47:08 PM
Wow--big progress! Did you already try it on your M2?
Title: Re: ML on EOS-M2
Post by: a1ex on June 21, 2017, 09:35:37 PM
However, the MD5 values from the "original" ROM1 and the new QEMU dump don't match. Is this expected?

"boot=1" works by patching the ROM. If you do the same change in a hex editor, you'll get the second MD5.
Title: Re: ML on EOS-M2
Post by: dfort on June 24, 2017, 03:43:22 AM
Wow--big progress! Did you already try it on your M2?

Not yet. Still in QEMU. Trying to run Hello World but my MacBook doesn't look like an EOSM2:

(https://c1.staticflickr.com/5/4217/35455229096_54c5570334.jpg)

I found everything in stubs.S but then I started going through consts.h and yikes! Looks like finding stubs was the easy part. I did read the part about firmware signature failing in QEMU so I made a new branch, merged QEMU and EOSM2 but found out that if there is no value for "#define SIG_EOSM2_103" then I get a blank QEMU screen. Is this a Catch-22?

Oh well, at least I've got a good stubs.S file -- uh, I think.
Title: Re: ML on EOS-M2
Post by: a1ex on June 24, 2017, 10:17:13 AM
Time to learn some debugging tricks :)

Look at recent QEMU updates - one of them is compiling ML with debug information (http://www.magiclantern.fm/forum/index.php?topic=2864.msg185649#msg185649). So far I have only used it for translating code addresses from ML code into source code lines (addr2line). Let's see if we can enable source code debugging in GDB, so we can run the code step by step, as we would with a PC program.

Notes:
- This is the first time I'm trying this trick. I have used source code debugging in IDA + QEMU before, but that was based on decompiled code (and the free version can't do debugging if I'm not mistaken). Here I want to run ML step by step, on the original source code.
- I want the QEMU updates in the mainline, as there are some APIs for debugging ML as well (there's a PR open), which means you can merge "qemu" in your EOSM2 branch. Or, simply experiment there, and once the debugging session is over, move the changes to the "clean" branch (this is what I usually do if I need the dm-spy-experiments branch).

Enough chit-chat, let's try it:
Code: [Select]
# from ML directory
hg update qemu -C
cd platform/EOSM2.103
make clean; make
# mount the SD image
make install

Next, we have to set up a breakpoint in our code (autoexec.bin); we can do that from debugmsg.gdb.

Without debugging symbols, we would do it like this and debug the assembly code (which is not very easy):
Code: [Select]
# breakpoint on autoexec.bin
b *0x800000

To use debugging symbols in gdb, we'll have to use symbol-file (to load the symbols from an elf file). The .o files that appear when compiling ML are elf files. Also, the files without extension, "autoexec" and "magiclantern", are elf files. The first one refers to reboot.c (look it up in Makefile.src) and gets loaded where Canon's bootloader loads autoexec.bin (at AUTOEXEC_BASE = 0x800000), while the second refers to the regular ML code (starting in boot-hack.c), that gets loaded into its final location (RESTARTSTART).

Why all this trouble? Because our binary can't stay at 0x800000. How comes? Well, autoexec.bin (which is a file loaded in Canon code, probably for debugging or development purposes) wasn't meant (read: designed by Canon) to be a program to run alongside Canon's main firmware; therefore DryOS is free to use the addresses near 0x800000 (where we are loaded). So, we have to move somewhere else. More on that later.

Back to our debugging session: the issue is in reboot.c (that's where this message is printed from), so we are going to run "autoexec.bin" as a plain binary (machine code), and use "autoexec" (elf) for the debugging symbols. Add this to debugmsg.gdb:

Code: [Select]
# breakpoint on autoexec.bin (cstart)
symbol-file ../magic-lantern/platform/EOSM2.103/autoexec
b cstart

You can keep both breakpoints, as there are some other things that execute before cstart (such as checking the integrity of the binary file, to avoid executing random code when loading from a corrupted filesystem). Without further ado, let's dive in:

Code: [Select]
# from QEMU directory (assuming default paths and bash)
. ./export_ml_syms.sh EOSM2.103
./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

We'll hit the first breakpoint first:
Code: [Select]
Now jump to AUTOEXEC.BIN!!
...
Breakpoint 3, 0x00800000 in _start ()
=> 0x00800000 <_start+0>: 0c 40 8f e2 add r4, pc, #12

Type "continue" ("c") to reach cstart:
Code: [Select]
Breakpoint 4, cstart () at ../../src/reboot.c:215
215     int s = compute_signature((int*)SIG_START, SIG_LEN);

Type "layout src". Now you are debugging on the original source code.

Tip: you can set breakpoints like this:
Code: [Select]
b reboot.c:216
Refer to GDB manual for the stepping commands (https://sourceware.org/gdb/onlinedocs/gdb/Continuing-and-Stepping.html), or use a GDB GUI.

Let's try nemiver:
Code: [Select]
. ./export_ml_syms.sh EOSM2.103
./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & nemiver --gdb-binary=$(which arm-none-eabi-gdb) --remote="localhost:1234" ../magic-lantern/platform/EOSM2.103/autoexec

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/gdb/nemiver.png)

For some reason, I'm unable to step. Too bad, as it seems to be working for these (https://www.jann.cc/2012/04/14/using_nemiver_for_remote_debugging_on_arm_microcontrollers.html) guys (https://github.com/ajhc/demo-cortex-m3/blob/master/stm32f0-discovery/extra/debug_nemiver.sh). Maybe someone else can figure it out?

Let's try gdbgui (https://github.com/cs01/gdbgui):
Code: [Select]
. ./export_ml_syms.sh EOSM2.103
./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & gdbgui -g "arm-none-eabi-gdb"

Make sure you view its GUI with Google Chrome and enter this at the gdb prompt:
Code: [Select]
source EOSM2/debugmsg.gdb

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/gdb/gdbgui.png)

This one appears to work a little better, you can run the program step by step using the GUI, but still gives a bunch of errors. Go figure.

cgdb is a terminal-based GUI, but doesn't like QEMU printing debug messages on the same window. Fortunately, nkls provided a script to split the terminal window in two - splitgdb.sh. Replace its contents with something like this:
Code: [Select]
#!/usr/bin/env bash

. ./export_ml_syms.sh $1.$2
tmux new-session -d "./run_canon_fw.sh $1,firmware='boot=1' -s -S"
tmux split-window -h "cgdb -d arm-none-eabi-gdb -x $1/debugmsg.gdb"
tmux attach-session -d

Code: [Select]
./splitgdb.sh EOSM2 103

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/gdb/cgdb.png)


The user interface seems to be very rough though, and it doesn't like our colored debug messages either...

Let's try the "classic" ddd:
Code: [Select]
. ./export_ml_syms.sh EOSM2.103
./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S & ddd --debugger "arm-none-eabi-gdb -x EOSM2/debugmsg.gdb"

This works. The interface is not pretty, but at least it works.

(I've ran EOSM ML on EOSM2 ROM, in case you are wondering)

(http://a1ex.magiclantern.fm/bleeding-edge/qemu/gdb/ddd.png)

Other suggestions?
Title: Re: ML on EOS-M2
Post by: dfort on June 25, 2017, 08:43:42 AM
Code: [Select]
# from ML directory
hg update qemu -C
cd platform/EOSM2.103

Oh I wish. For now I'm having to merge EOSM2.103_wip, qemu-EOSM2-wip_1, qemu-build-tweaks-2 and of course qemu into a qemu-wip branch in order to get all of the pieces working with EOSM2 on a Mac. I'm keeping these separate branches to keep a couple of my pull requests in order. The build tweaks for the Mac (https://bitbucket.org/hudson/magic-lantern/pull-requests/836/qemu-build-tweaks-2/diff) are working nicely as is the EOSM2 QEMU setup (https://bitbucket.org/hudson/magic-lantern/pull-requests/835/qemu-eosm2-preliminary-setup/diff). It will be a happy day for me when these get merged into the qemu branch.  :D

Code: [Select]
Now jump to AUTOEXEC.BIN!!
...
Breakpoint 3, 0x00800000 in _start ()
=> 0x00800000 <_start+0>: 0c 40 8f e2 add r4, pc, #12

Type "continue" ("c") to reach cstart:
Code: [Select]
Breakpoint 4, cstart () at ../../src/reboot.c:215
215     int s = compute_signature((int*)SIG_START, SIG_LEN);

I'm seeing something slightly different probably because I'm working on a different autoexec.bin.

Code: [Select]
Breakpoint 1 at 0x4398
Breakpoint 2 at 0x7360
Breakpoint 3 at 0xff0c5144
Breakpoint 4 at 0x800000
Breakpoint 5 at 0x8646ec: file ../../src/fw-signature.h, line 47.
(gdb) c
Continuing.
...
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D

Breakpoint 4, 0x00800000 in _start ()
(gdb) c
Continuing.

Breakpoint 5, cstart () at ../../src/reboot.c:215
215     int s = compute_signature((int*)SIG_START, SIG_LEN);
(gdb)


Type "layout src". Now you are debugging on the original source code.

Nice! Got the breakpoints tip working too.

(https://c1.staticflickr.com/5/4261/35478510606_404f5f365b_z.jpg)

Let's try nemiver:

Spent too many hours trying to get nemiver working on Mac (building from the source code) but never succeeded.

Moving on to gdbgui. It this one is coded in python and is easy to install on Mac:

Code: [Select]
pip install gdbgui --upgrade
Yeah, easy but it took me a while because the instructions on the project page didn't work for me. Maybe because I'm using the Homebrew python that doesn't need root access to install packages? So if you're on a Mac and followed my ML tutorial (http://www.magiclantern.fm/forum/index.php?topic=16012.0), don't do this:

Code: [Select]
sudo pip install gdbgui --upgrade --user
Now we're on the same page--sort of.

(https://c1.staticflickr.com/5/4254/35387409601_e32971dffb_z.jpg) (https://flic.kr/p/VV4HZK)

It is getting late so I'll skip to ddd:

Code: [Select]
brew tap homebrew/core
brew install ddd

And...
Code: [Select]
Setting BOOTDISK flag to FFFFFFFF
Error: Unresolved inheritance operation

Xt error (Unresolved inheritance operation).

Oops!  You have found a bug in DDD.

If you can reproduce this bug, please send a bug report
to <[email protected]>, giving a subject like

    DDD 3.3.12 (i386-apple-darwin16.3.0) gets Xt error

To enable us to fix the bug, you should include the following information:
* What you were doing to get this message.  Report all the facts.
* The contents of the `~/.ddd/log' file as generated by this session.
Please read also the section "Reporting Bugs" in the DDD manual.

We thank you for your support.

ddd: Cannot save options
[1]   Done                    ./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S

Ugh!

I'll try cgdb tomorrow.
Title: Re: ML on EOS-M2
Post by: dfort on June 25, 2017, 04:46:14 PM
Ok--trying out cgdb. The splitgdb.sh script uses tmux which I already had on my system, don't when I installed it but it is also available from brew as is cgdb:
 
Code: [Select]
brew install cgdb
Wow, this is one is impressive when run in full screen mode.

(https://c1.staticflickr.com/5/4278/35394128311_e41eeddb26_z.jpg) (https://flic.kr/p/VVEaez)

A couple of questions:

Code: [Select]
│Breakpoint 2 at 0x7360     
│Breakpoint 3 at 0xff0c5144 
│Breakpoint 4 at 0x800000   
│Breakpoint 5 at 0x8646ec: file ../../src/fw-signature.h, line 47. 
│(gdb) c                     
│Continuing.                 

│Breakpoint 4, 0x00800000 in _start ()                   
│(gdb) c                     
│Continuing.                 

│Breakpoint 5, cstart () at ../../src/reboot.c:215       
│(gdb) print s               
│$1 = 0xceeeeeec             
│(gdb) print _signature     
│$2 = 0x2d7c6dcf             
│(gdb)   

[EDIT] Never mind, it is showing the EOSM_202 signature I'm using as a temp. Still find myself in a Catch 22 and can't get "hello world" working much less find the firmware signature. That has to be done in camera, right?

Code: [Select]
│Breakpoint 5, cstart () at ../../src/reboot.c:211         
│(gdb) print s               
│No symbol "s" in current context.                         
│(gdb) print _signature       
│No symbol "_signature" in current context.                 
│(gdb)

Continuing from there:

Code: [Select]
DRYOS PANIC: Module Code = 1, Panic Code = 2
Catch 22 -- Need to run "hello world" to get the firmware signature but QEMU doesn't run "hello world" because it can't get the firmware signature. If we could get the firmware signature maybe "hello world" will run on QEMU but then, what's the point?

Is it too early to turn on the camera bootflag? (Not that I have a clue how to do that.)
Title: Re: ML on EOS-M2
Post by: a1ex on June 25, 2017, 05:16:06 PM
Why isn't this version of splitgdb.sh by nkls in the qemu branch?

The version from nkls is there (https://bitbucket.org/hudson/magic-lantern/src/9912f99c674c81c89ee77e81704439c304d753e1/contrib/qemu/scripts/splitgdb.sh?at=qemu&fileviewer=file-view-default). What I wrote above was something I came up with at the time of writing; I didn't use this approach before.

Would be nice to improve it to accept any number of command-line options (and pass all of them to QEMU).

Quote
Is the firmware signature it displayed to be trusted?

Yes, but only after compute_signature() returns. Not before.

The value from my screenshots should be valid, although I have not set up an EOSM2 platform directory yet (so it's untested).

Quote
Oops!  You have found a bug in DDD.

That's too bad - a very powerful piece of software apparently unmaintained for about 8 years...

Quote
DRYOS PANIC: Module Code = 1, Panic Code = 2

That's a good sign - this message can only appear from the main firmware, so we are no longer in bootloader context. Still, probably something went wrong when patching the startup process.

Time to debug what happens between loading autoexec.bin and this message. Some useful tools:
Code: [Select]
-d calls or callstack
-d ramw (RAM writes)
-d autoexec (to avoid verbose messages from the bootloader)

For example, -d autoexec,ramw,calls,io,v should give some very good insights into how the startup code is patched, and why it doesn't work. The log will be huge though, and it will miss the call to copy_and_restart from boot-hack.c (because it's not actually a call, but a direct jump), but will catch the next call (from copy_and_restart to reloc_entry), which looks like this (500D, since that's the binary I happened to have on the virtual card):

Code: [Select]
       (big bunch of zeroing out some RAM - zero_bss)
       (big bunch of copying from ROM to RAM - blob_memcpy)
       [ram] at 0x0004D4D4:000B8818 [0x000BB300] <- 0xE12FFF1E: was 0xEBFFF769;
       [ram] at 0x0004D4DC:000B8818 [0x000B9154] <- 0xCB400   : was 0x4C5C4;
       [ram] at 0x0004D4E4:000B8818 [0x000B8704] <- 0x7E400   : was 0x0;
       [ram] at 0x0004D4FC:000B8818 [0x000B90BC] <- 0xEBCCCAFD: was 0xEB0F6D03;
       [ram] at 0x0004D514:000B8818 [0x000B9144] <- 0xEBBD78E5: was 0xEB001AEB;
       [ram] at 0x0004D51C:000B8818 [0x000B9160] <- 0x4D010   : was 0xFF011DBC;
       call 0xB8824(0, c00003e0, 400, b8824)                                     at [4d560:86ba30] (copy_and_restart)
        [FlashIF] at 0x000B882C:0004D564 [0xC0000010] <- 0xD9C5D9C5: 'Write enable' enabled
        [*unk*] at 0x000B8838:0004D564 [0xC020010C] <- 0x1       : ???
        ...
        (more startup activity follows)

Right before this call, you'll see the patching process (those "hijack" macros - where they go). I can use this log to check if that went well.

Tip: in the qemu branch, in boot-hack.c (but not in minimal.c), there are some startup messages (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-boot-check/QEMU_boot_check_logs/) (qprint*). These are useful - to see them on the QEMU console, compile with CONFIG_QEMU=y from the regular target (not minimal). Binaries compiled in this way will not run on the camera. More about this on the 1300D thread (http://www.magiclantern.fm/forum/index.php?topic=17969.msg185641#msg185641).
Title: Re: ML on EOS-M2
Post by: dfort on June 25, 2017, 05:30:03 PM
The version from nkls is there (https://bitbucket.org/hudson/magic-lantern/src/9912f99c674c81c89ee77e81704439c304d753e1/contrib/qemu/scripts/splitgdb.sh?at=qemu&fileviewer=file-view-default). What I wrote above was something I came up with at the time of writing; I didn't use this approach before.

Would be nice to improve it to accept any number of command-line options (and pass all of them to QEMU).

Yes, but your latest version of splitgdb.sh seems to be working more better.  :D

...I have not set up an EOSM2 platform directory yet (so it's untested).

I have. It obviously isn't finished but it compiles.

https://bitbucket.org/daniel_fort/magic-lantern/branch/EOSM2.103_wip
Title: Re: ML on EOS-M2
Post by: dfort on June 26, 2017, 07:55:05 AM
I'm still inching my way through consts.h and hit another speed bump. Got to YUV422_LV_BUFFER_1,2,3 and looked up the link to the wiki that's in the source code:

Quote
// http://magiclantern.wikia.com/wiki/VRAM_ADDR_from_code
// stateobj_disp[1]

Not sure what stateobj_disp[1] referrers to but determined that the wiki page wants me to run this code on the EOSM2.103:

Code: [Select]
int i;
unsigned int *bmp_ram_addr = bmp_vram_info;
for (i=0; i<2; i++)
  DebugMsg( DM_MAGIC, 3, "bmp_vram[]=0x%08x, 0x%08x, 0x%08x",
    bmp_ram_addr[3*i+0],  bmp_ram_addr[3*i+1], bmp_ram_addr[3*i+2] );
unsigned int *vram_info_addr = vram_info;
for (i=0; i<3; i++)
  DebugMsg( DM_MAGIC, 3, "vram_info[]=0x%08x, w=%4d, h=%4d, p=%4d, n=%4d",
      vram_info_addr[5*i+0],  vram_info_addr[5*i+1],
      vram_info_addr[5*i+2], vram_info_addr[5*i+3], vram_info_addr[5*i+4] );
unsigned int *stateobj_disp = 0x90494+0x128; // see ff137acc SetBitmapVramAddress
DebugMsg( DM_MAGIC, 3, "stateobj_disp+0xb0[]=0x%08x,0x%08x,0x%08x,",
  stateobj_disp[0], stateobj_disp[1], stateobj_disp[2]);

I was able to figure out the changes to the code but not sure where to run it--probably in "Don't click me?" (run_test in src/debug.c) However, that seems to be something that needs to be run in the camera, or can it be done in QEMU? Also not having much luck getting the firmware signature in QEMU, seems like I'm at the point of being able to run "Hello world" on the camera but can't seem to be able to get it working in QEMU. (I also tend to use "seem" way too much.)

The unexpected  benefactor of this port is the original EOSM as I keep finding blips in the code (http://www.magiclantern.fm/forum/index.php?topic=9741.msg186289#msg186289) for that platform.
Title: Re: ML on EOS-M2
Post by: a1ex on June 26, 2017, 11:00:55 AM
60D in QEMU (from don't click me):

Code: [Select]
. ./export_ml_syms.sh 60D.111 # needed for DebugMsg
./run_canon_fw.sh 60D,firmware="boot=1" -d debugmsg
...
[    run_test:1fe0bc9c ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000000, 0x03f87100
[    run_test:1fe0bcc0 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000000, 0x03f87100
[    run_test:1fe0bce8 ] (32:03) stateobj_disp+0xbc[]=0x43f80008,0x5c307800,0x40a8e470,

From real camera (the easiest way was to use printf):
Code: [Select]
# 60D 1.1.1, 0x2458+0xBC
bmp_vram[]=0xc0f140d0, 0x00000000, 0x03f87100
bmp_vram[]=0xc0f140d4, 0x00000000, 0x03f87100
stateobj_disp+0xbc[]=0x43f80008,0x41b07800,0x48a902c4

In both cases, stateobj_disp[1] shows a valid address for the image buffer. It's not always the same; on this model there are 3 possible addresses; they are not fixed (as we thought in early days) but allocated dynamically. In many cases (but not always), this dynamic allocation gives the same results every time, so hardcoding the image buffer addresses usually works. This area really needs some cleanup (wip here (https://bitbucket.org/hudson/magic-lantern/branch/yuv_buffers)).

However, D5 models don't seem to work that well in QEMU:

Code: [Select]
# 5D3 1.1.3, 0x246A4+0x118, regular photo mode
[    run_test:000753ec ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000000, 0x00dc3100
[    run_test:00075410 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000000, 0x00dc3100
[    run_test:00075438 ] (32:03) stateobj_disp+0x118[]=0x40dbc008,0x00000000,0x40881520,

# 5D3 1.1.3, 0x246A4+0x118, playback mode (not exactly working well in QEMU)
[    run_test:000753ec ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000000, 0x00d43100
[    run_test:00075410 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000000, 0x00d43100
[    run_test:00075438 ] (32:03) stateobj_disp+0x118[]=0x00000000,0x40881414,0x00000000,

# 700D 1.1.4, 0x23C20+0x118 (in QEMU it starts in movie mode and asks for a lens)
[     ml_init:0007f1d0 ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000000, 0x01307100
[     ml_init:0007f1f4 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000000, 0x01307100
[     ml_init:0007f21c ] (32:03) stateobj_disp+0x118[]=0x41300008,0x00000000,0x4092e400,

# 700D 1.1.4, 0x23C20+0x118, playback mode
[     ml_init:0007f1ec ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000000, 0x01387100
[     ml_init:0007f210 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000000, 0x01387100
[     ml_init:0007f238 ] (32:03) stateobj_disp+0x118[]=0x41380008,0x00000000,0x4092e3a8

# EOSM 2.0.2, 0x3E650+0x114 (no GUI in QEMU)
[     ml_init:0009e70c ] (32:03) bmp_vram[]=0xc0f140d0, 0x00000001, 0x01387100
[     ml_init:0009e730 ] (32:03) bmp_vram[]=0xc0f140d4, 0x00000001, 0x01387100
[     ml_init:0009e758 ] (32:03) stateobj_disp+0x114[]=0x41380008,0x00000000,0x00000000,

In 5D3 1.1.3 playback, there is some nonzero value, but I don't recognize it as a display buffer address (it might be correct), while on 700D and EOSM I've only got 0 in QEMU. Probably that's the current state of the emulation.

In any case, this test does not reveal any new addresses, so you don't really need to run it; matching the stub with other models should be enough.
Title: Re: ML on EOS-M2
Post by: dfort on June 27, 2017, 11:31:31 PM
In any case, this test does not reveal any new addresses, so you don't really need to run it; matching the stub with other models should be enough.

This is proving to be a devilishly difficult stub to ferret out of the disassembly. I found only one match in one of my EOSM.202 disassemblies:

Code: [Select]
"[LVEVF] evVdInterrupt LastRamClear Start!![4f1d7800]":
That matches the first of the three buffers I'm searching for:

Code: [Select]
#define YUV422_LV_BUFFER_1 0x4F1D7800
#define YUV422_LV_BUFFER_2 0x4F5E7800
#define YUV422_LV_BUFFER_3 0x4F9F7800

Though it is sort of a dead end because all the other models, including the EOSM2 shows this:

Code: [Select]
"evVdInterrupt LastRamClear Start!![%lx]":
There's a lot of "0x40000000" all over the disassemblies and maybe that's a clue?

I'm probably on the wrong path but I searched through the forum and wiki and came up empty handed. Also don't know if I should be pressing forward finding stubs or if I should try to get "Hello world" working in QEMU in order to find the firmware signature.

In any case, I'll keep searching.
Title: Re: ML on EOS-M2
Post by: a1ex on June 28, 2017, 12:16:01 AM
Couldn't find it from LastRamClear Start, but this (http://www.magiclantern.fm/forum/index.php?topic=9741.msg186291#msg186291) does apply to both M and M2.
Title: Re: ML on EOS-M2
Post by: dfort on June 28, 2017, 05:23:37 AM
I can see the hint near ImageEffect that shows the values for YUV422_LV_BUFFER_DISPLAY_ADDR but I'm still clueless where to find YUV422_LV_BUFFER_1,2,3.

I'll keep going and get back to this later, maybe it will become obvious the second time around.
Title: Re: ML on EOS-M2
Post by: a1ex on June 28, 2017, 08:26:05 AM
Ah, right. I've found one in QEMU - 0x4f3d7800. You could use that value for all of them, as a workaround.

I've found it by pressing P to go to play mode, then printing the contents of 0x90494+0x12C in gdb. I was lucky, as this didn't work on the other D5 models I've tried before (double-checked today, still no luck there).

I've deleted the YUV422_(LV|HD)* constants locally, but there are some bits of code that had to be updated (on the yuv_buffers branch mentioned before, merged with new-lv-buffer-detection - which attempts to autodetect these addresses at runtime, rather than using hardcoded values). Unfortunately, the current state is "compiles, but doesn't work" (after more than half a day of fiddling with it), and I doubt I'll be able to fix it today.
Title: Re: ML on EOS-M2
Post by: dfort on June 30, 2017, 04:50:13 PM
Progressing slowly.

Some of the stubs and constants have hints that include which two numbers to add together to get the value we're looking for. This helps when stub hunting by pattern matching.

For example, in consts.h I think I may have found these for the EOSM2.103:

Code: [Select]
#define CURRENT_GUI_MODE (*(int*)0x928BC) // in SetGUIRequestMode
#define ISO_ADJUSTMENT_ACTIVE ((*(int*)0x96388) == 0xF) // dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)

But it is easier to check against the disassembly if we write them out like this:

Code: [Select]
#define CURRENT_GUI_MODE (*(int*)(0x92860+0x5C)) // in SetGUIRequestMode
#define ISO_ADJUSTMENT_ACTIVE ((*(int*)(0x96338+0x50)) == 0xF) // dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)

I'm not sure if ISO_ADJUSTMENT_ACTIVE is valid on the EOSM2 because it is missing on the EOSM. From what I can figure out the EOSM should be this:

Code: [Select]
#define ISO_ADJUSTMENT_ACTIVE ((*(int*)(0x4BE28+0x44)) == 0xF) // dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)

On the 100D it is commented out:

Code: [Select]
// #define ISO_ADJUSTMENT_ACTIVE ((*(int*)(0x6B930)) == 0xF)       // dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)
but maybe the problem is because it is the wrong value? Shouldn't it be 0x6B860 instead of 0x6B930? Easier to check the disassembly if it is written like this:

Code: [Select]
#define ISO_ADJUSTMENT_ACTIVE ((*(int*)(0x6B810+0x50)) == 0xF)       // dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)
I'm not sure how to "look for: if arg1 == 1: MEM(0x79B8) = *(arg2)" or if I'm even on the right track here but this does fit into a pattern using the values from the 5D3.113 and 700D.114.
Title: Re: ML on EOS-M2
Post by: nikfreak on June 30, 2017, 05:12:04 PM
For firmware 1.01 on 100D it's like you found out:

Code: [Select]
#define ISO_ADJUSTMENT_ACTIVE ((*(int*)(0x6B810+0x50)) == 0xF)
I am unsure myself why it's commented out but the "wrong" value was correct one of the older A/B/C revision on 1.00 firmware. I was mainly doing copy&paste for the revisions as well while upgrading support for fw 1.01. Anyways thx for pointing it out as it has tbd for 70D too (942C0+50).

Quote
// dec ptpNotifyOlcInfoChanged and look for: if arg1 == 1: MEM(0x79B8) = *(arg2)

that's also copy&paste from one of the older supported cams.
Title: Re: ML on EOS-M2
Post by: dfort on July 01, 2017, 07:25:35 PM
I don't want to come off like a little kid asking "why" all the time but as I'm working my way through consts.h some things pique my curiosity.

MOV_RES_AND_FPS_COMBINATIONS

Found this constant used in bitrate.c:
Code: [Select]
    for (i = 0; i < MOV_RES_AND_FPS_COMBINATIONS; i++) // 7 combinations of resolution / fps

Makes sense because these are all the possible settings I could come up with:

1-192024NTSC/PAL
2-192025PAL
3-192030NTSC
4-128050PAL
5-128060NTSC
5-64025PAL
7-64030NTSC

However, this is what is in consts.h:

5D3.113
Code: [Select]
#define MOV_RES_AND_FPS_COMBINATIONS 5 // 3 fullhd, 2 hd, not changing the two VGA modes; worth trying with 9

100D.101
Code: [Select]
#define MOV_RES_AND_FPS_COMBINATIONS 4 // 2FHD, 1HD, 1VGA

700D.114, EOSM.202
Code: [Select]
#define MOV_RES_AND_FPS_COMBINATIONS 9
Ok--so on the 5D3 we aren't messing around with the VGA modes but the 100D should have the same combinations as the other cameras and how do you come up with 9 combinations? 1920/24 is available in both NTSC and PAL so that might give you 8 but where's the final combination?

HALFSHUTTER_PRESSED

The hint is to look for string "[MC] permit LV instant", it's the struct referenced in this function. I can find "[MC] permit LV instant" in the disassemblies of all the cameras except the EOSM2. There is a function named "LvPermit(DelivLvLock)" that might be what replaced it but I can't be sure. How to figure this out? Is it possible to do a half-shutter press in QEMU?

[EDIT] Off topic but on the 700D:

Code: [Select]
#define AE_STATE (*(int8_t*)(0x366B8 + 0x1C))
#define AE_VALUE (*(int8_t*)(0x366B8 + 0x1D))

Shouldn't it be?

Code: [Select]
#define AE_STATE (*(int8_t*)(0x366B4 + 0x1C))
#define AE_VALUE (*(int8_t*)(0x366B4 + 0x1D))

I've got a 700D but not sure how to check this.
Title: Re: ML on EOS-M2
Post by: nikfreak on July 01, 2017, 08:16:47 PM
100D is crippled by Canon. It really only has 4 movie modes to select in Canon menu and this camera even profits from the bitrate modifications as it has no ALL-I option by default  :P 700D should have same amount of movie modes as 650D.

Take this for 700D.114:
Code: [Select]
#define AE_STATE (*(int8_t*)(0x367B4 + 0x1C))
#define AE_VALUE (*(int8_t*)(0x367B4 + 0x1D))
...


#define HALFSHUTTER_PRESSED (*(int*)0x24884) is ok [0x2486C+0x18].
Title: Re: ML on EOS-M2
Post by: a1ex on July 01, 2017, 08:34:21 PM
MVR: see platform/include/mvr.h - some of them are commented. Basically, look up these functions in the firmware and map (or verify) the offsets in the mvr_config structure.

On recent models, there are some gaps (2 positions) between 720p and VGA modes. Not sure what's up with them and whether they are used or not (I'll find out in QEMU, if I'll ever manage to emulate LiveView and video recording - light years from current state). I tend to say we should update all of them to 5, to be on the safe side (so bitrate adjustment won't overwrite memory locations we are unsure about), even if that will leave the VGA modes uncovered.

That 4 on 100D is probably a mistake (thanks for pointing out, as I didn't notice it when looking at the PR). I bet bitrate control doesn't work in 50p mode, but works on 60 (or maybe the other way).

[MC] permit LV instant - all D4 and D5 models have it.

Half-shutter support in QEMU is very incomplete (https://bitbucket.org/hudson/magic-lantern/commits/a55cd409ee1e5374c116ca9305dd1402fc530625).

The AE_VALUEs are for the autoexpo module. Not sure how to use it, couldn't figure it out...
Title: Re: ML on EOS-M2
Post by: JohanJ on July 01, 2017, 08:38:32 PM
100D is crippled by Canon. It really only has 4 movie modes to select in Canon menu

In fact all 7 modes are available also from canon menu, just that only the ones fitting to the chosen video mode (PAL/NTSC) are shown.

[EDIT] On 60D there are 2 additional VGA modes Crop 640x480 NTSC and Crop 640x480 PAL. So here you get in  total 9. Just an idea (even though it is another cam generation)
Title: Re: ML on EOS-M2
Post by: dfort on July 01, 2017, 11:55:24 PM
On 60D there are 2 additional VGA modes Crop 640x480 NTSC and Crop 640x480 PAL. So here you get in  total 9. Just an idea (even though it is another cam generation)

Right, and with FEATURE_CROP_MODE_HACK it adds crop modes to 1920/24 and 25 so that does get 9 possible MOV_RES_AND_FPS_COMBINATIONS. However, it looks like maybe it should be changed to 5 for the reasons that a1ex noted. I'll leave it at 9 on the EOSM2 for now and follow along with any changes to the other cameras.

...
Take this for 700D.114:
...
#define HALFSHUTTER_PRESSED (*(int*)0x24884) is ok [0x2486C+0x18].

That's a clue that got me closer.
5D3[0x251C4+0x10]
100D[0x6674C+0x18]
700D[0x2486C+0x18]
EOSM[0x3F204+0x20]
EOSM2[0x910A8+0x??]

[MC] permit LV instant - all D4 and D5 models have it.

Pretty sure that on the EOSM2 disassembly it shows up as - judge_permit_lv.

Take this for 700D.114:
Code: [Select]
#define AE_STATE (*(int8_t*)(0x367B4 + 0x1C))
#define AE_VALUE (*(int8_t*)(0x367B4 + 0x1D))
...


That's right--my bad.

The AE_VALUEs are for the autoexpo module. Not sure how to use it, couldn't figure it out...

Hokey Smoke!

autoexpo.mo on 700D
(https://farm5.staticflickr.com/4211/35488279622_e6fc748a5a.jpg)

Don't want to get into this right now but maybe this is something that should be looked into:
Issue #2643 Cannot get module autoexpo to work on EOS 700d (https://bitbucket.org/hudson/magic-lantern/issues/2643/cannot-get-module-autoexpo-to-work-on-eos)

[EDIT] Issue with autoexpo.mo on 700D mentioned here:

Hello,

I have download this module for the 700D and it doesn't work.

Then I will instal cygwin and check out the code source and after read this post, see by debug that the AE_STATE is never set (always 0).
So I change the consts.h file for the 700D platform in replacement of the constants AE_STATE ans VALUE as written somewhere in a topic page after having check in hexa code. It will be then the following :

#define EXPO_COMP (*(int16_t*)(0x367B4+0x1C))
#define AE_VALUE (EXPO_COMP-1) * 8 / 2048
#define AE_STATE (*(int8_t*)(0x367B4+0x1C))

It works (I check by printing the values of ae, iso, and so on).

I can check in if you want.

Just a question, what it is the use of AE_STATE ?

Xav
Title: Re: ML on EOS-M2
Post by: Licaon_Kter on July 02, 2017, 12:22:36 AM
But you can't use autoexpo.mo anyway on M/M2 because you don't have (one) more hardware buttons, right?
Title: Re: ML on EOS-M2
Post by: dfort on July 02, 2017, 01:49:25 AM
Which buttons are missing?

autoexpo.mo appears to be working on the EOSM though I have no idea how to work it.

There are lots of reports of it not working on the 700D and it looks like there is a solution to that.
Title: Re: ML on EOS-M2
Post by: a1ex on July 02, 2017, 08:39:12 AM
Quote
Pretty sure that on the EOSM2 disassembly it shows up as - judge_permit_lv.

Right - it does the same thing.

Pressing half-shutter in QEMU gives plenty of info regarding this event (enough to identify this address, but not enough to change things on the user interface or pass the API tests):
Code: [Select]
Key event: 2a -> 0e0e002d
[MPU] Sending : 06 05 06 26 01 00
[    PowerMgr:ff1470d4 ] (00:01) <0 0 0 0 0 0 7> [PM] DisablePowerSave (Counter = 2)
[    PowerMgr:ff147144 ] (00:01) <0 0 0 0 0 0 7> [PM] EnablePowerSave (Counter = 1)
[    MainCtrl:ff0d01c0 ] (9c:01) <0 0 0 0 0 0 7> ID:26(30)
[    MainCtrl:ff0cd8ac ] (89:03) <0 0 0 0 0 0 7> bindReceiveSwitch (38, 1)
[    MainCtrl:ff0cdd88 ] (89:03) <0 0 0 0 0 0 7> COM_SW_SOMETHING
[    MainCtrl:ff0d8ba0 ] (85:03) <0 0 0 0 0 0 7> GUI_Control:84 0x0
[ GuiMainTask:ff0d8f54 ] (84:01) <0 0 0 0 0 0 7> GUI_CONTROL:84
[MPU] Sending : 06 04 05 00 00 00
[ GuiMainTask:ff1470d4 ] (00:01) <0 0 0 0 0 0 7> [PM] DisablePowerSave (Counter = 2)
[ GuiMainTask:ff147144 ] (00:01) <0 0 0 0 0 0 7> [PM] EnablePowerSave (Counter = 1)
[ GuiMainTask:ff1c0728 ] (84:03) <0 0 0 0 0 0 7> GUICMD_PRESS_BUTTON_SOMETHING(0x0)
[    EventMgr:ff224a04 ] (8d:01) <0 0 0 0 0 0 7> emDeliverMulticastEvent : SW1ON
[    MainCtrl:ff0d01c0 ] (9c:01) <0 0 0 0 0 0 7> ID:0(31)
[    MainCtrl:ff154660 ] (9c:03) <0 0 0 0 0 0 7> MeteringStart
[    MainCtrl:ff3da3b0 ] (c0:03) <1 0 0 0 0 0 7> MainCtrl GuiCancelMoviePlay(4151)
[    MainCtrl:ff3e3814 ] (c0:02) <1 0 0 0 0 0 7> [G_P]PC:0xff3e3804 LR:0xff154718 SP:0x197f38
[    MainCtrl:ff0dffe4 ] (2c:03) <1 0 0 0 0 0 7> MVP_CancelMoviePlay
[    MainCtrl:ff0d8ba0 ] (85:03) <1 0 0 0 0 0 7> GUI_Control:80 0x0
[    MainCtrl:ff1549a4 ] (89:03) <1 0 0 0 0 0 7> EVENTID_METERING_START
[      RscMgr:ff0e87dc ] (80:02) <1 0 0 0 0 0 7> srmEventDispatch: Current = 0, dwEventID = 12, dwParam = -442503148 S
[      RscMgr:ff0e89e4 ] (80:02) <1 0 0 0 0 0 7> srmEventDispatch: Current = 0, dwEventID = 12, dwParam = -442503148 E
[ GuiMainTask:ff1c1880 ] (84:01) <1 0 0 0 0 0 7> gui control end
[ GuiMainTask:ff1c18a0 ] (84:01) <1 0 0 0 0 0 7> 0msec = 47810 - 47810
[ GuiMainTask:ff1c18bc ] (84:01) <1 0 0 0 0 0 7> 3584msec = 776448 - 780032
[ GuiMainTask:ff0d8f54 ] (84:01) <1 0 0 0 0 0 7> GUI_CONTROL:80
[ GuiMainTask:ff1c1880 ] (84:01) <1 0 0 0 0 0 7> gui control end
[ GuiMainTask:ff1c18a0 ] (84:01) <1 0 0 0 0 0 7> 0msec = 47810 - 47810
[ GuiMainTask:ff1c18bc ] (84:01) <1 0 0 0 0 0 7> 0msec = 780544 - 780544
[     CtrlSrv:ff3ab264 ] (83:03) <1 0 0 0 0 0 7> IDLEHandler PRESS_SW1_BUTTON
[    Fstorage:ff1c8fac ] (9e:03) <1 0 0 0 0 0 7> fssSW1On

[         CEC:ff347750 ] (c2:01) <1 0 0 0 0 0 7> cecCompleteStart Start
[         CEC:ff357118 ] (00:26) <1 0 0 0 0 0 7> [I2C] localI2C_Read : 378 (Task : CEC)
   980: 47851.520 WARN [I2C] localI2C_Read : 378 (Task : CEC)
[         CEC:ff347750 ] (c2:01) <1 0 0 0 0 0 7> cecCompleteStart Start
[         CEC:ff357118 ] (00:26) <1 0 0 0 0 0 7> [I2C] localI2C_Read : 378 (Task : CEC)
   981: 47868.416 WARN [I2C] localI2C_Read : 378 (Task : CEC)
[         CEC:ff347750 ] (c2:01) <1 0 0 0 0 0 7> cecCompleteStart Start
[         CEC:ff357118 ] (00:26) <1 0 0 0 0 0 7> [I2C] localI2C_Read : 378 (Task : CEC)
   982: 47888.640 WARN [I2C] localI2C_Read : 378 (Task : CEC)

Key event: aa -> 0e0e002e
[MPU] Sending : 06 04 05 0b 00 00
[    PowerMgr:ff1470d4 ] (00:01) <1 0 0 0 0 0 7> [PM] DisablePowerSave (Counter = 2)
[    PowerMgr:ff147144 ] (00:01) <1 0 0 0 0 0 7> [PM] EnablePowerSave (Counter = 1)
[    EventMgr:ff224b7c ] (8d:01) <1 0 0 0 0 0 7> emDeliverMulticastEvent : SW1OFF
[    MainCtrl:ff0d01c0 ] (9c:01) <1 0 0 0 0 0 7> ID:B(32)
[    MainCtrl:ff154748 ] (9c:03) <1 0 0 0 0 0 7> MeteringTimerStart
[    MainCtrl:ff0d8ba0 ] (85:03) <0 0 0 0 0 0 7> GUI_Control:81 0x0
[    MainCtrl:ff1549a4 ] (89:03) <0 0 0 0 0 0 7> EVENTID_METERING_TIMER_START
[      RscMgr:ff0e87dc ] (80:02) <0 0 0 0 0 0 7> srmEventDispatch: Current = 0, dwEventID = 13, dwParam = -442503148 S
[      RscMgr:ff0e89e4 ] (80:02) <0 0 0 0 0 0 7> srmEventDispatch: Current = 0, dwEventID = 13, dwParam = -442503148 E
[ GuiMainTask:ff0d8f54 ] (84:01) <0 0 0 0 0 0 7> GUI_CONTROL:81
[ GuiMainTask:ff1c1880 ] (84:01) <0 0 0 0 0 0 7> gui control end
[ GuiMainTask:ff1c18a0 ] (84:01) <0 0 0 0 0 0 7> 0msec = 47880 - 47880
[ GuiMainTask:ff1c18bc ] (84:01) <0 0 0 0 0 0 7> 0msec = 844032 - 844032
[     CtrlSrv:ff3ab2f4 ] (83:03) <0 0 0 0 0 0 7> IDLEHandler UNPRESS_SW1_BUTTON
[         CEC:ff347750 ] (c2:01) <0 0 0 0 0 0 7> cecCompleteStart Start
[         CEC:ff357118 ] (00:26) <0 0 0 0 0 0 7> [I2C] localI2C_Read : 378 (Task : CEC)

The additional values are the ones found in that function:
Code: [Select]
    APPEND("(%02x:%02x) <%x %x %x %x %x %x %x> ", r0, r1, MEM(0x910C8), MEM(0x910CC), MEM(0x910D4), MEM(0x910F8), MEM(0x910FC), MEM(0x91100), MEM(0x91178));
Code: [Select]
static uint32_t MEM(uint32_t addr)
{
    uint32_t buf;
    cpu_physical_memory_read(addr, &buf, sizeof(buf));
    return buf;
}

Therefore, the first one is what we are looking for (the one compared to 1 in the code).

This gives another hint to find half-shutter state: MeteringStart and MeteringTimerStart. I knew it was related to EVENTID_METERING_START / EVENTID_METERING_TIMER_START, but I didn't know where exactly it was set - this one was found with the mem_spy tool (now a module), that monitors memory addresses to find "interesting" changes.

To quote g3gg0:

... would have saved a lot of work if we had that earlier :D

P.S. Here's an easy coding task - print the name of the buttons in the logs.
Title: Re: ML on EOS-M2
Post by: Licaon_Kter on July 02, 2017, 06:40:16 PM
Which buttons are missing?

autoexpo.mo appears to be working on the EOSM though I have no idea how to work it.
Looks ok now though, you can change lower limits with left-right and upper values with rotating wheel, strange that (IIRC) this was not possible in the past, albeit the sourcode was not modified since forever.  :-\ ::) :o
Title: Re: ML on EOS-M2
Post by: dfort on July 03, 2017, 06:49:25 PM
Oh where oh where has my little fonts gone?

Been here before (http://www.magiclantern.fm/forum/index.php?topic=18957.0) and thought I had my head wrapped around the solution but probably not so let's start from scratch.

There is a block of font addresses in consts.h. Let's look at what you would think would be the closest cousin to the EOSM2 -- the EOSM:

Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xffcb9c04
#define BFNT_BITMAP_OFFSET 0xffcbcb88
#define BFNT_BITMAP_DATA   0xffcbfb0c

The base address of DIGIC 5 cameras like the EOSM2 (and EOSM, 700D, 100D, etc.) is 0xFF0C0000 and addresses greater than the base address should show up in the ROM1.BIN disassembly. In fact it is quite easy to find where the fonts are in the disassembly because of strings like "HCanonGothic" and "CanonMonospace".

However, here is where the EOSM2 differs from its closest cousins. The fonts are located in ROM0.BIN like the 5D3.

If you follow the wikia link it will eventually lead you to a python script named find_fnt.py. A copy of that script is also located in magic-lantern/contrib/indy/. What it can do is display the font information buried in ROM0.BIN. The script was written back in the day when DIGIC 4 cameras were the norm so it assumes the base address is 0xFF010000 but it can accept a different address via a command line argument. Let's try it out on the EOSM2.

python find_fnt.py ROM0.BIN ff0c0000
Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xff1e4838: FNT
0xff1e483c: (+0x04) 0xffd8
0xff1e483e: (+0x06) font_width = 40
0xff1e4840: (+0x08) charmap_offset = 0x24
0xff1e4844: (+0x0c) charmap_size = 0x31b4
0xff1e4848: (+0x10) bitmap_size = 0x8ab3e
0xff1e484c: (+0x14) font name = 'HCanonGothic///'
0xff1e485c: (+0x24) char_codes[]. 3181 chars
0xff1e7a10: (+0x31d8) offsets[]. Last offset value = 0x8ab08
0xff1eabc4: (+0x638c) bitmaps[]
  0xff2756cc: (+0x90e94) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xff275704: FNT
0xff275708: (+0x04) 0xffd8
0xff27570a: (+0x06) font_width = 40
0xff27570c: (+0x08) charmap_offset = 0x24
0xff275710: (+0x0c) charmap_size = 0x188
0xff275714: (+0x10) bitmap_size = 0x31c4
0xff275718: (+0x14) font name = 'CanonMonospace'
0xff275728: (+0x24) char_codes[]. 98 chars
0xff2758b0: (+0x1ac) offsets[]. Last offset value = 0x3142
0xff275a38: (+0x334) bitmaps[]
  0xff278b7a: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

Now the EOSM2 is showing that it is also different from the 5D3 because the 5D3 shows each block twice at different addresses which allows you to easily figure out the ROM0 size.

Last time I searched for fonts was on the 5D3.134 firmware update (http://www.magiclantern.fm/forum/index.php?topic=18966.0) and I didn't have QEMU working but now that I do I can check out the ROM layout (http://www.magiclantern.fm/forum/index.php?topic=6785.msg58899#msg58899):

EOSM2
Code: [Select]
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror

So that means instead of using an address that starts at 0xFF we need to use 0xF[0-7] for ROM0 addresses. The 5D3 uses 0xF7 so let's use that for the EOSM2:

EOSM2
Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xf71e485c
#define BFNT_BITMAP_OFFSET 0xf71e7a10
#define BFNT_BITMAP_DATA   0xf71eabc4

Oops--there is something wrong here. How do I know? Because on the 5D3 find_fnt.py shows this:

python find_fnt.py ROM0.BIN ff0c0000
Code: [Select]
0xff423a50: (+0x24) char_codes[]. 3165 chars
And the addressed used in consts.h is:
Code: [Select]
#define BFNT_CHAR_CODES    0xF7363A50
Revisiting the last time I ventured into the font rabbit hole:

Therefore, 0xF[0-7][3B]63A50 should all be valid choices for BFNT_CHAR_CODES.

[3B]?

So confused -- why doesn't find_fnt.py just spit out the right numbers?
Title: Re: ML on EOS-M2
Post by: a1ex on July 03, 2017, 07:24:38 PM
So confused -- why doesn't find_fnt.py just spit out the right numbers?

It does - try this:
Code: [Select]
python find_fnt.py ROM0.BIN 0xF0000000
Title: Re: ML on EOS-M2
Post by: dfort on July 03, 2017, 09:12:39 PM
python find_fnt.py ROM0.BIN 0xF0000000
Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xf0124838: FNT
0xf012483c: (+0x04) 0xffd8
0xf012483e: (+0x06) font_width = 40
0xf0124840: (+0x08) charmap_offset = 0x24
0xf0124844: (+0x0c) charmap_size = 0x31b4
0xf0124848: (+0x10) bitmap_size = 0x8ab3e
0xf012484c: (+0x14) font name = 'HCanonGothic///'
0xf012485c: (+0x24) char_codes[]. 3181 chars
0xf0127a10: (+0x31d8) offsets[]. Last offset value = 0x8ab08
0xf012abc4: (+0x638c) bitmaps[]
  0xf01b56cc: (+0x90e94) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xf01b5704: FNT
0xf01b5708: (+0x04) 0xffd8
0xf01b570a: (+0x06) font_width = 40
0xf01b570c: (+0x08) charmap_offset = 0x24
0xf01b5710: (+0x0c) charmap_size = 0x188
0xf01b5714: (+0x10) bitmap_size = 0x31c4
0xf01b5718: (+0x14) font name = 'CanonMonospace'
0xf01b5728: (+0x24) char_codes[]. 98 chars
0xf01b58b0: (+0x1ac) offsets[]. Last offset value = 0x3142
0xf01b5a38: (+0x334) bitmaps[]
  0xf01b8b7a: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

So this should work?

Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xf012485c
#define BFNT_BITMAP_OFFSET 0xf0127a10
#define BFNT_BITMAP_DATA   0xf012abc4

In addition 0xF[0-7] can be used so 0xf712485c, etc. will make it look more like the 5D3 and 5D2 addresses used in consts.h.

Wow, talk about overcomplicating things. So I guess my mistake was using the wrong base address--but isn't the base address for the EOSM2 0xFF0C0000?

Just for fun (yeah, this is fun) I tried a DIGIC 4 camera that has the fonts in ROM0 -- the 5D2. We want to get to this:

Code: [Select]
#define BFNT_CHAR_CODES    0xf7c5E9C0
Running "python find_fnt.py ROM0.BIN" returns 0xff06e9c0 which wouldn't work but "python find_fnt.py ROM0.BIN 0xF0c00000" returns 0xf0c5e9c0 which is equivalent to 0xf7c5E9C0. (There's mixed upper/lower case in the 5D2 code but that shouldn't matter.)

So what's up with this?

Code: [Select]
if (len(sys.argv)>2):
  base = int(sys.argv[2], 16)
else:
  base = 0xff010000

I thought that was supposed to work with DIGIC 4 cameras.

P.S. Here's an easy coding task - print the name of the buttons in the logs.

You mean in the mem_spy module interpret the button codes to print out the names of the buttons?
Title: Re: ML on EOS-M2
Post by: a1ex on July 03, 2017, 09:50:39 PM
... that script assumes the ROM dump loads at 0xff010000 (this was the main firmware start address for DIGIC 4 models, so in the early days, ROM files were saved from that address)

Meaning: the script expects the address where the ROM file was dumped from; it doesn't care where the main firmware starts inside that ROM.

Quote
You mean in the mem_spy module interpret the button codes to print out the names of the buttons?

No, that's complicated. What I meant was about QEMU logs (the user interface side) - instead of "Key event: 2a -> 0e0e002d", print something human-readable.

Mapping button codes to names is doable as well; it's done in ARM-console to some extent (bgmt command), but should probably be rewritten with unicorn (http://www.unicorn-engine.org) instead (similar to extract_button_codes.py). That requires some low-level knowledge though.
Title: Re: ML on EOS-M2
Post by: dfort on July 03, 2017, 10:36:36 PM
Got it...I think.

boot-hack.c
Code: [Select]
static void backup_rom_task()
{
    backup_region("ML/LOGS/ROM1.BIN", 0xF8000000, 0x01000000);
    backup_region("ML/LOGS/ROM0.BIN", 0xF0000000, 0x01000000);
}

Shouldn't all of the new ROM dumps be saved at these addresses? If that's the case find_fnt.py should probably be updated to:

Code: [Select]
if (len(sys.argv)>2):
  base = int(sys.argv[2], 16)
else:
  base = 0xf0000000

I suppose that the ROM dumpers also use these addresses?

In any case...moving on.

...QEMU logs...instead of "Key event: 2a -> 0e0e002d", print something human-readable.

You mean like "Key event: 2a -> 0e0e002d (half-shutter press)", "Key event: aa -> 0e0e002e (half-shutter release)" or unpress -- if I got that right.
Title: Re: ML on EOS-M2
Post by: dfort on July 04, 2017, 08:20:31 PM
Finished all I could on stubs.S and consts.h. If anyone wants to follow along I'm working on a work-in-progress branch (https://bitbucket.org/daniel_fort/magic-lantern/branch/EOSM2.103_wip) which is sort of a scratch pad to take notes as I go along. Before making the pull request (provided I can get that far) I'll make a new branch so the pull request will be as clean as possible.

Following some of the hints and references in the code led me to some helpful scripts like find_fnt.py that could use some updating to make it easier on the next guy that uses it.

https://bitbucket.org/hudson/magic-lantern/pull-requests/844/find_fntpy-update/diff

Another interesting script is in magic-lantern/modules/adtg_log and is called parse_bin.py. I'm not sure how to use it (no documentation I could find)

[EDIT] Well I did find this:

the bin can be parsed with the python script in the module directory ;)

In any case, running this:

Code: [Select]
python parse_bin.py -f ROM1.BIN
created a large amount of information. I was looking for FRAME_SHUTTER_BLANKING_ZOOM and FRAME_SHUTTER_BLANKING_NOZOOM which required looking up some ADTG registers. I used the adtg_gui before but never figured out adtg_log. In any case, there was this discussion (http://www.magiclantern.fm/forum/index.php?topic=6751.msg69043#msg69043) on the forum which looked promising but I never could find what I was looking for using parse_bin.py.

[EDIT] To be more specific, I was trying to find this:

Code: [Select]
#define FRAME_SHUTTER_BLANKING_ZOOM   (*(uint16_t*)0x40481B20) // ADTG register 805f
#define FRAME_SHUTTER_BLANKING_NOZOOM (*(uint16_t*)0x40481B24) // ADTG register 8061

There are a few more addresses to look up then time to get back to QEMU to try and get "Hello world" working and hopefully find the firmware signature.
Title: Re: ML on EOS-M2
Post by: dfort on July 06, 2017, 02:29:32 AM
Thought I'd try something easy:

Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop;
- (easy) print Hello World with the minimal ML target;
- (moderate) print Hello World with the regular ML source code;
- (moderate) implement touch screen emulation;
- (moderate) bring ML menu in QEMU;
- (harder) full ML port, with most features working;
- (extreme) LiveView emulation.

So I went into magic-lantern/minimal/qemu-frsp and tried this:

Code: [Select]
MODEL=EOSM2 make
...
[ LD       ]   magiclantern
stdio.o: In function `my_fprintf':
/Users/rosiefort/magic-lantern/minimal/qemu-frsp/../../src/stdio.c:32: undefined reference to `FIO_WriteFile'
make: *** [magiclantern] Error 1

Am I missing something in EOSM2? Maybe but I can't build any model. Next I went into magic-lantern/minimal/qemu-hptimer and:

Code: [Select]
MODEL=EOSM2 make
...
magiclantern.bin: 3052 bytes
[ CC       ]   reboot.o
[ CC       ]   disp_direct.o
[ CC       ]   footer.o
[ LD       ]   autoexec
[ OBJCOPY  ]   autoexec.bin
[ XOR_CHK  ]   autoexec.bin

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000060 0x0009e1e0 0x0009e1e0 0x00bec 0x10c20 RWE 0x10

Yay!

Running autoexec.bin in QEMU:

(https://farm5.staticflickr.com/4290/35615450431_8154d034a4.jpg)

Boo!

Is this because I haven't gotten the firmware signature yet? Keep thinking that maybe that is the next challenge.

[EDIT] Doh! Figured out why that message is coming up--I was using the EOSM.202 firmware signature as a filler for the still undiscovered EOSM2.103 firmware signature. Commenting out the firmware signature and the minimal autoexec.bin will just hang. So I'm back in that Catch 22 situation. I did try some of the debugging tricks with a regular ML Hello World build but that's a moderately difficult exercise to complete.

Here is how that went:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=1" -s -S -d autoexec,ramw,calls,io,v & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] enabling code execution logging.
[EOS] enabling memory access logging.
[EOS] enabling singlestep.
[EOS] disabling verbose logging until autoexec.bin starts...
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[EOS] loading './EOSM2/SFDATA.BIN' as serial flash, size=0x1000000
[MPU] warning: non-empty spell #31 (08 06 03 1f) has duplicate(s): #40
[MPU] warning: non-empty spell #41 (0a 08 01 3b) has duplicate(s): #37 #38
[MPU] warning: non-empty spell #45 (26 24 01 4e) has duplicate(s): #46

[MPU] Available keys:
- Arrow keys   : Navigation
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- DELETE       : guess
- M            : MENU
- P            : PLAY
- I            : INFO/DISP
- Q            : guess
- L            : LiveView
- Shift        : Half-shutter
- B            : Open battery cover
- F1           : show this help

Setting BOOTDISK flag to FFFFFFFF
0xffff0000 in ?? ()
+macro define CURRENT_TASK   ((int)0xFFFFFFFF)
+macro define CURRENT_ISR    ((int)0xFFFFFFFF)
+macro define RTC_VALID_FLAG ((int)0xFFFFFFFF)
+macro define NUM_CORES      1
+macro define PRINT_CALLSTACK 0
+set pagination off
+set output-radix 16
+define hook-quit
+define KRED
+define KCYN
+define KBLU
+define KGRN
+define KYLW
+define KRESET
+macro define CURRENT_TASK_NAME (((int*)CURRENT_TASK)[0] ? ((char***)CURRENT_TASK)[0][9] : CURRENT_TASK)
+define print_callstack
+define print_current_location
+define print_current_location_with_callstack
+define print_formatted_string
+define DebugMsg_log
+define DebugMsg1_log
+define printf_log
+define task_create_log
+define task_switch_log
+define msleep_log
+define assert_log
+define assert0_log
+define create_semaphore_log
+define delete_semaphore_log
+define print_sem_name
+define take_semaphore_log
+define give_semaphore_log
+define create_msg_queue_log
+define print_mq_name
+define post_msg_queue_log
+define try_receive_msg_queue_log
+define receive_msg_queue_log
+define register_interrupt_log
+define register_func_log
+define mpu_decode
+define mpu_send_log
+define mpu_recv_log
+define try_expand_ram_struct
+define try_post_event_log
+define delayed_call_print_name
+define delayed_call_log
+define SetTimerAfter_log
+define SetHPTimerAfterNow_log
+define SetHPTimerNextTick_log
+define CancelTimer_log
+define engine_resource_description
+define engine_resources_list
+define CreateResLockEntry_log
+define LockEngineResources_log
+define AsyncLockEngineResources_log
+define UnLockEngineResources_log
+define StartEDmac_log
+define SetEDmac_log
+define print_date_time
+define set_date_time
+define load_default_date_time_log
+define log_result
Breakpoint 1 at 0x4398
Breakpoint 2 at 0x7360
Breakpoint 3 at 0xff0c5144
FFFF096C: MCR p15,0,Rd,cr9,cr1,0: XSCALE_LOCK_ICACHE_LINE <- 0x40000006 (40000000 - 40000FFF, 0x1000)
FFFF0970: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0x2078
FFFF0978: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0x12078   
FFFF3F54: MCR p15,0,Rd,cr6,cr0,0:  946_PRBS0 <- 0x3F       (00000000 - FFFFFFFF, 0x100000000)
FFFF3F5C: MCR p15,0,Rd,cr6,cr1,0:  946_PRBS1 <- 0x3D       (00000000 - 7FFFFFFF, 0x80000000)
FFFF3F64: MCR p15,0,Rd,cr6,cr2,0:  946_PRBS2 <- 0xE0000039 (E0000000 - FFFFFFFF, 0x20000000)
FFFF3F6C: MCR p15,0,Rd,cr6,cr3,0:  946_PRBS3 <- 0xC0000039 (C0000000 - DFFFFFFF, 0x20000000)
FFFF3F74: MCR p15,0,Rd,cr6,cr4,0:  946_PRBS4 <- 0xFF00002F (FF000000 - FFFFFFFF, 0x1000000)
FFFF3F7C: MCR p15,0,Rd,cr6,cr5,0:  946_PRBS5 <- 0x37       (00000000 - 0FFFFFFF, 0x10000000)
FFFF3F84: MCR p15,0,Rd,cr6,cr6,0:  946_PRBS6 <- 0xF700002F (F7000000 - F7FFFFFF, 0x1000000)
FFFF3F8C: MCR p15,0,Rd,cr2,cr0,0: DCACHE_CFG <- 0x70       
FFFF3F90: MCR p15,0,Rd,cr3,cr0,0:       DACR <- 0x70       
FFFF3F94: MCR p15,0,Rd,cr2,cr0,1: ICACHE_CFG <- 0x70       
FFFF3F9C: MCR p15,0,Rd,cr5,cr0,0:    DATA_AP <- 0x3FFF     
FFFF3FA0: MCR p15,0,Rd,cr5,cr0,1:    INSN_AP <- 0x3FFF     
FFFF3FA4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0x12078
FFFF3FC8: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC001307D
FFFF0984: MCR p15,0,Rd,cr9,cr1,1: XSCALE_UNLOCK_ICACHE <- 0x6        (00000000 - 00000FFF, 0x1000)
FFFF0988: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC001307D
FFFF0990: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005307D
FFFF09B8: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005307D
FFFF09C0: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
SD LOAD OK.
Open file for read : AUTOEXEC.BIN
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
File size : 0x671C0
Now jump to AUTOEXEC.BIN!!
0010E5F8: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E600: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[EOS] enabling verbose logging for autoexec.bin...
      PC jump? 0x800000 lr=10e65c                                                at [10e674:10e65c]
      0x0010e674:  e1a0f001      mov pc, r1
[EOS] enabling verbose logging for autoexec.bin...
[BOOT] changing user_mem_start from 0x17ec74 (1567860) to 0x117940 (1145152)
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x9e1f0 (647664)
DRYOS PANIC: Module Code = 1, Panic Code = 2
^C
Program received signal SIGINT, Interrupt.
0xff0c19f4 in ?? ()
(gdb)
Title: Re: ML on EOS-M2
Post by: a1ex on July 06, 2017, 05:24:47 PM
Took a closer look at the firmware signature - it's affected by our QEMU patches. So, to find it, either debug it without enabling other ROM patches (those with SET), or find it out outside GDB (e.g. print it on the screen), or just keep it commented out.

Next:
Code: [Select]
[BOOT] changing user_mem_start from 0x17ec74 (1567860) to 0x117940 (1145152)

This one looks fishy to me. Why? Because we "push" this address to the right in order to make room from our application. Let's take a look at the memory map (printed by 0xFF0C6E70).

Start the firmware under GDB, then, once it's running, hit CTRL-C and call this function. The simplest (and most hackish) way to do this, without loading guest code (autoexec.bin), is:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
(wait until it boots)
CTRL-C
set $pc = 0xFF0C6E70
continue

This will break the execution flow (after returning from that function, there will be dragons), but at least we'll get what we are looking for. The function probably has no arguments (R0 and R1 are first written, and then read), therefore simply jumping to it should be fine.

Result:
Code: [Select]
...
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max        515828
...

Here, user_mem is malloc, sys_mem is AllocateMemory.

That means, user_mem_start is not 0x17EC74, but 0x100d80. You'll find that in cstart as well. Set RESTARTSTART to that, or a little bit higher - e.g. 0x100E00). Now, the boot process will "push" user_mem_start to start right where our code ends.

BTW - most models have "RESTARTSTART is selected to be just above the end of the bss". Coincidentally or not, Canon's BSS (the end of what they zero out before cstart) is also 0x100D80; however, it is my understanding that we are not modifying Canon's BSS, but user_mem_start (therefore shrinking their malloc memory pool).

Hence, the 7D2 has an updated comment:
Code: [Select]
# DryOS memory map
# RESTARTSTART is selected to be at user_mem_start
# (aka heap start / DRY_HEAP_START / malloc memory pool)
#
RESTARTSTART  = 0x001CC400

There, Canon's BSS ends at 0x745E8, so the previous comments about BSS were incorrect. Time to fix them.

Having fixed RESTARTSTART and the (misnamed) HIJACK_INSTR_BSS_END, let's check the memory map again, this time loading some minimal ML:

Code: [Select]
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00104580
user_mem_max        501492

Notice the user_mem (malloc) pool is now smaller and shifted to the right. The size needed by our binary (MemSiz when compiling) is 0x03780 (ymmv), so it appears to match.

How much free space do we have? The 7D2 source contains 3 important hints: malloc_info, sysmem_info and smemShowFix (all easily found in the firmware). Let's call the first one (from GDB, to avoid recompiling):

Without ML:
Code: [Select]
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x0017ea60
  Total Size          = 0x0007dcd8 (   515288)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00068228 (   426536)
  Free Block Max Size = 0x000681f0 (   426480)
  Free Block Count    = 0x00000002 (        2)

With minimal ML:
Code: [Select]
Malloc Information (onetime type)
  Start Address       = 0x00104588
  End Address         = 0x0017ea60
  Total Size          = 0x0007a4d8 (   500952)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00064a28 (   412200)
  Free Block Max Size = 0x000649f0 (   412144)
  Free Block Count    = 0x00000002 (        2)


OK, so we don't have enough space to load the entire ML (maybe only if you compile it without any features). We'll have move somewhere else.

This was the "classic" boot process, originally used by 5D2. Not all models have enough free space here to accommodate the full autoexec.bin.

One option is to try the AllocateMemory (like 550D or 1100D), and another is to hunt for possibly unused memory areas (smemShowFix - FF222B68) - see http://www.magiclantern.fm/forum/index.php?topic=5071.0.

Note: I wouldn't call random stuff on a real camera without triple-checking the stub, the number of arguments, any prerequisites that might be and so on. Here we are on a virtual machine, so there's nothing to break - feel free to experiment away.
Title: Re: ML on EOS-M2
Post by: dfort on July 07, 2017, 05:18:22 PM
(after returning from that function, there will be dragons)

I've been meticulously going through all your notes and when I don't understand something I look it up. If this is a reference to the John Ringo book:

Quote
The world is a paradise-and then, in a moment, it ends.

Yep, that's pretty much what happened.

All that debugging you did was due to a very basic mistake on my part. I started out by copying over all the EOSM.202 settings and then went through every line and changed the values to what I could find in the EOSM2.103 disassembly. Well, almost every line. Somehow I skipped over Makefile.platform.default. Oops. The hints were there and I might have been able to figure it out, except for the "a little bit higher" part. From my understanding of your last post:

magic-lantern/platform/EOSM2.103/Makefile.platform.default
Code: [Select]
#Makefile.setup.platform for EOS M2

ifeq ($(FW_VERSION),103)
CANON_NAME_FIR=EOSM2103.FIR

ROMBASEADDR = 0xFF0C0000

# DryOS memory map
# RESTARTSTART is selected to be at user_mem_start
# (aka heap start / DRY_HEAP_START / malloc memory pool)
# (actually 0x00100d80 but make it a little bit higher)
#
RESTARTSTART = 0x100E00
endif

Having fixed RESTARTSTART and the (misnamed) HIJACK_INSTR_BSS_END...

Not sure about the HIJACK_INSTR_BSS_END fix. There is a comment that should be changed:

magic-lantern/platform/EOSM2.103/consts.h
Code: [Select]
#define HIJACK_INSTR_BSS_END 0xFF0C1C94  //BSS_END is 0x17EC74
in order to match the disassembly:
Code: [Select]
; ff0c1c90: (00100d80)
; ff0c1c94: (0017ec74)

...let's check the memory map again, this time loading some minimal ML:

The hackish way to do this doesn't work when loading the QEMU minimal autoexec.bin so this probably requires doing it via debugmsg.gbd, right? Uh, how?

Speaking of debugmsg.gbd - looking at other similar cameras (EOSM and 700D), shouldn't this work on the EOSM2?

Code: [Select]
b *0x1900
assert_log

That might be useful.

The 7D2 source contains 3 important hints: malloc_info, sysmem_info and smemShowFix (all easily found in the firmware).

The closest I could find in the EOSM2 source is: "Malloc Information (%s type)", "ShowWinsysMemory" and "..smemShowFix"

Let's call the first one (from GDB, to avoid recompiling):

I'm lost. How did you do that?

Ok--getting back to the "easy" stuff:

Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop;
- (easy) print Hello World with the minimal ML target;

The loop that times out is annoying.

Code: [Select]
...
[  TouchMgr:ff346008 ] (c1:16) Touch IC Ver : 0x0000
   154:   993.280 [TCH]Touch IC Ver : 0x0000
[  TouchMgr:ff345160 ] (c1:03) ExecuteSIO32 err
...
[  TouchMgr:ff344f58 ] (c1:03) ExecuteSIO8 err
...
[  TouchMgr:ff34602c ] (c1:06)  TouchPanelIC Ack Error
[   Startup:0003671c ] task_create(DisplayMgr, prio=12, stack=0, entry=36628, arg=8ce0e8)

Am I on the right track here?

Code: [Select]
# patch TouchMgr to avoid err messages and loop timeout
# ExecuteSIO8
set *(int*)0xff344f?? = 0xe???????
# ExecuteSIO32
set *(int*)0xff3451?? = 0xe???????

I'm still at a loss at how to print Hello World with the minimal ML target. For now I've got the firmware signature commented out. However, it seems that you keep hinting that it can be found with QEMU.

Took a closer look at the firmware signature - it's affected by our QEMU patches. So, to find it, either debug it without enabling other ROM patches (those with SET), or find it out outside GDB (e.g. print it on the screen), or just keep it commented out.

The code to find it doesn't seem too complex:

Code: [Select]
static int compute_signature(int* start, int num)
{
    int c = 0;
    int* p;
    for (p = start; p < start + num; p++)
    {
        c += *p;
    }
    return c;
}

start = SIG_START which on Digic V cameras like the EOSM2 is 0xFF0C0000
num = 0x10000

So shouldn't it be possible to write a rather simple script to find the firmware signature from ROM1.BIN or would that be short circuiting the porting process?
Title: Re: ML on EOS-M2
Post by: a1ex on July 07, 2017, 06:36:17 PM
Quote
The world is a paradise-and then, in a moment, it ends.

What happens here is because of how function calls are done on ARM processors. When a BL executes, the next address (where the code should return after the function ends) is placed into LR, then PC is updated to the first address of the called function.

What we did in GDB was to interrupt the firmware somewhere (we don't know where) and change the PC to some function (that takes no arguments). That function will run just fine (we can use its results), but it won't know where to return - and that's where the dragon appears.

In QEMU it's fine - it will just return somewhere. It may lock up, it may go back to the caller function without finishing the function we interrupted... or it may somehow jump to EraseSectorOfRom or other similar address. That's why you must never try this on the real firmware.

Quote
...let's check the memory map again, this time loading some minimal ML:
...
The hackish way to do this doesn't work when loading the QEMU minimal autoexec.bin so this probably requires doing it via debugmsg.gbd, right? Uh, how?

First you need to compile ML from some minimal target (you already did). Then place it on the card image, then start it with boot=1.

To call the function, you may now use either GDB (ctrl-c + jumping to the function), or insert a call to that function somewhere in the minimal ML. The second method will not break the execution flow, as the function will know where to return.

Quote
The closest I could find in the EOSM2 source is: "Malloc Information (%s type)", "ShowWinsysMemory" and "..smemShowFix"

Yes, these are. For example, malloc_info is FF0C750C (the function that references the first string). It takes no arguments, so you can just jump to it.

Quote
shouldn't this work on the EOSM2?
Code: [Select]
b *0x1900
assert_log

It does. Not all models have assert at 0x1900 (1300D doesn't).

Quote
So shouldn't it be possible to write a rather simple script to find the firmware signature from ROM1.BIN or would that be short circuiting the porting process?

It is. But once you patch something via GDB (such as "set *(int*)0xFF356E28 = 0xe3a00001"), the value will change.

Quote
I'm still at a loss at how to print Hello World with the minimal ML target.

Look in src/minimal.c.
Title: Re: ML on EOS-M2
Post by: dfort on July 08, 2017, 03:04:49 AM
Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop;

One small step for a1ex, one giant leap for dfort:

Code: [Select]
# patch TouchMgr to avoid loop timeout
# ExecuteSIO8
set *(int*)0xFF344F40 = 0xe1a0000b
# ExecuteSIO32
set *(int*)0xFF345148 = 0xe1a0000b

At least it doesn't slow down to a crawl when it hits the TouchMgr section.

Not sure what the issue is with the localI2C_Read loop. I'm not noticing any slowdown or delays though this is printed in red to the console:

Code: [Select]
   154:   824.832 [TCH]ERROR  TouchPanelIC Ack Error
   155:   760.576 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   156:   760.832 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   157:   761.088 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   158:   761.344 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   159:   761.600 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   160:   761.856 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   161:   761.856 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   162:   762.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   163:   762.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   164:   833.536 [IMPP] H264E InitializeH264EncodeFor1080pDZoom
   165:   833.536 [IMPP] H264E InitializeH264EncodeFor1080p25fpsDZoom
   172:   836.608 [STARTUP] startupInitializeComplete
   173:   841.216 [MR] mvrChangeAckCBR : Video - Mode=0, Type=2, Rate=30, GOP=15
   174:   850.176 [MR] mvrChangeAckCBR : Video - Mode=0, Type=2, Rate=30, GOP=15
   180:   863.232 [CEC]CEC OFF
   182:   882.176 [RTC] !! RTC CHECK ERROR !!

Also had a breakthrough on making a minimal ML. I simply copied what was done in magic-lantern/minimal/600D.102 (and other cameras). Continuing from where I got lost last time:

Code: [Select]
^C
Program received signal SIGINT, Interrupt.
0xff352318 in ?? ()
(gdb) set $pc = 0xFF0C6E70
(gdb) continue
Continuing.
...
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max         14336
...

With that same minimal ML:

Code: [Select]
^C
Program received signal SIGINT, Interrupt.
0xff352318 in ?? ()
(gdb) set $pc = 0xFF0C750C
(gdb) continue
Continuing.
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x00104370
  Total Size          = 0x000035e8 (    13800)
  Allocated Size      = 0x000025c0 (     9664)
  Allocated Peak      = 0x000025c0 (     9664)
  Allocated Count     = 0x00000011 (       17)
  Free Size           = 0x00001028 (     4136)
  Free Block Max Size = 0x00001028 (     4136)
  Free Block Count    = 0x00000001 (        1)

To be continued (still no Hello World) but first - to quote kichetof - take a beer and enjoy it!
Title: Re: ML on EOS-M2
Post by: a1ex on July 08, 2017, 07:32:58 AM
At least it doesn't slow down to a crawl when it hits the TouchMgr section.

... it no longer boots the GUI ...
Title: Re: ML on EOS-M2
Post by: dfort on July 08, 2017, 09:03:31 AM
... it no longer boots the GUI ...

That's strange -- I can see the GUI here. You are talking about the Canon menus, right? I still don't have a minimal ML that displays "Hello World" much less any ML GUI with or without that TouchMgr "fix."

This is the entire debugmsg.gdb:

Code: [Select]
# ./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

source -v debug-logging.gdb

macro define CURRENT_TASK 0x8FBCC
macro define CURRENT_ISR  (*(int*)0x648 ? (*(int*)0x64C) >> 2 : 0)

b *0x4398
DebugMsg_log

b *0x1900
assert_log

b *0x7360
task_create_log

# patch DL to avoid DL ERROR messages
set *(int*)0xFF156348 = 0xe3a00015

# patch localI2C_Write to always return 1 (success)
set *(int*)0xFF356E24 = 0xe3a00001
set *(int*)0xFF356E28 = 0xe12fff1e

# patch TouchMgr to avoid loop timeout
# ExecuteSIO8
set *(int*)0xFF344F40 = 0xe1a0000b
# ExecuteSIO32
set *(int*)0xFF345148 = 0xe1a0000b

# skip SerialFlash version check
set *(int*)0xFF0C4278 = 0xe3a00000

# break infinite loop at Wait LeoLens Complete
b *0xFF0C5144
commands
  printf "Patching LeoLens (infinite loop)\n"
  set *(int*)($r4 + 0x28) = 1
  c
end

continue

Think I found ShowWinsysMemory

No ML:
Code: [Select]
   185:  2214.144 [WINSYS] allocated=44140 max_region=412876

Minimal ML:
Code: [Select]
[  PowerMgr:ff13432c ] (04:16) allocated=0 max_region=1066808

Couldn't find smemShowFix ("..smemShowFix") but this looks interesting:

No ML:
Code: [Select]
(gdb) set $pc = 0xff133b80
(gdb) c
Continuing.
[SVG] Free/Total:3229904/8118272
[SVG] SVG Alloc:0
[SVG] Max Region:2763608
[SVG] number of allocated block:4863
[SVG] dslr_malloc ResetAllocatePointer!!!!!!
[SVG] Memory Allocation Error at :429392408
[SVG] svg_malloc(size:0, pFileName, lineNo)
MaxRegion = 2763608
[SVG] Free/Total:3229904/8118272
[SVG] Max Region:2763608
[SVG] number of allocated block:4863

Minimal ML:
Code: [Select]
[SVG] Free/Total:6431420/8118272
[SVG] SVG Alloc:0
[SVG] Max Region:6427636
[SVG] number of allocated block:1537
[SVG] dslr_malloc ResetAllocatePointer!!!!!!
[SVG] Memory Allocation Error at :429392408
[SVG] svg_malloc(size:0, pFileName, lineNo)
MaxRegion = 6427636
[SVG] Free/Total:6431420/8118272
[SVG] Max Region:6427636
[SVG] number of allocated block:1537
Title: Re: ML on EOS-M2
Post by: a1ex on July 08, 2017, 09:38:43 AM
Hm, it's working now. Previously it got stuck in a loop and stayed like that for some 15 minutes...

Tip: comment out DebugMsg_log and replace it with -d debugmsg (much faster (http://www.magiclantern.fm/forum/index.php?topic=2864.msg185769#msg185769))
Title: Re: ML on EOS-M2
Post by: dfort on July 08, 2017, 06:02:29 PM
Yes, much faster.

Am I doing it right?

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
Following your link (http://www.magiclantern.fm/forum/index.php?topic=2864.msg185769#msg185769) shows this neat trick:

Code: [Select]
env QEMU_EOS_DEBUGMSG=0x5b90 ./run_canon_fw.sh 5D3,firmware="boot=0" -d debugmsg,callstack,v
but that 0x5b90 address is specific to the 5D3, right?

Booting into the Canon menus "boot=0" seems pretty solid though there are some error messages logged. Making the minimal autoexec.bin to print "Hello World" is another story. I thought I'd try it with the 5D3.113 but--

Code: [Select]
cd minimal/5D3.113/
RosieFoComputer:5D3.113 rosiefort$ make
../../platform/Makefile.platform.base:46: *** ROMBASEADDR is not defined.  Stop.

What the?
Title: Re: ML on EOS-M2
Post by: a1ex on July 08, 2017, 09:10:41 PM
Yep, looks fine. Of course, each model has a different DebugMsg address; will try to automate it.

5D3 has 2 firmware versions, so the build system doesn't know which one to pick. Adding FW_VERSION=113 fixes it.

This was probably broken when merging 5D3 1.2.3 (and since the minimal target is neither widely used, nor covered by self-tests, it wasn't noticed).

Back to the topic - I've got minimal hello world working on M2 after fixing RESTARTSTART and related consts as discussed above, then compiling your source. That was before the timeout patches, but it's still working afterwards.
Title: Re: ML on EOS-M2
Post by: dfort on July 09, 2017, 06:35:35 AM
I've got minimal hello world working on M2...

Nice!

I've got a minimal "Hello World" working on a real 5D3.113 but not on QEMU. Wanna trade?   :D

The 5D3.113 minimal ML was built in the minimal/5D3.113 directory and the firmware signature was commented out so it should work like an early port. This is what it looks like on the camera. (Thanks for the FW_VERSION=113 tip):

(https://farm5.staticflickr.com/4242/34997476353_d772eee7b8.jpg)

That same minimal ML on QEMU:

(https://farm5.staticflickr.com/4241/35675527851_d6cd354971_z.jpg) (https://flic.kr/p/WmwpvM)

The point of this exercise was to see if the problem is with the minimal ML or the QEMU setup or maybe both. Looks like that at least for the 5D3 it is QEMU. That is if my theory holds up which is that the same minimal ML autoexec.bin should be able to run on both the camera and QEMU. This is also assuming that ML on 5D3.113 in QEMU is working.

Note that I also tried the qemu-hptimer version with this result: (Note the "Hello from task init" that wasn't on the previous test.)

Code: [Select]
File size : 0x1940
Now jump to AUTOEXEC.BIN!!
0010DCCC: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010DCCC: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
Hello from task init
K285 READY
[      init:ff3014ac ] create_semaphore('EventProcedure', 1)
Temporary breakpoint 10 at 0xff3014b0

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff3014b0 in ?? ()
(gdb)

qemu-frsp isn't compiling over here.
Title: Re: ML on EOS-M2
Post by: a1ex on July 09, 2017, 09:10:21 AM
For 5D3, try running it with "patches.gdb" instead - that's the only one where I've tried the GUI.

The "Hello world" works without GDB script at all, although that way I get blank screen and no menus (not sure why).
Title: Re: ML on EOS-M2
Post by: dfort on July 09, 2017, 06:38:58 PM
What finally worked with the 5D3.113 for me was simply:

Code: [Select]
./run_canon_fw.sh 5D3,firmware="boot=1"
(https://farm5.staticflickr.com/4265/35008319273_51ebe033b3_z.jpg) (https://flic.kr/p/VkyMDB)

That is working with and without the "#define SIG_5D3_113  0x2e2f65f5" in fw-signature.h so in theory this should also work with the EOSM2 minimal ML, right?

Here is the output (https://pastebin.com/AdwsiYM6) from ./run_canon_fw.sh EOSM2,firmware="boot=1"
Title: Re: ML on EOS-M2
Post by: a1ex on July 09, 2017, 07:54:37 PM
Some models are able to boot the GUI without any GDB patches (e.g. 60D, 500D). They will show the date/time dialog, but you can close it cleanly - then you get the usual Canon GUI. If you patch date/time, it boots directly to the GUI. The patches.gdb are there, but they are no longer necessary (they used to be before figuring out menu navigation).

Other models (700D, EOSM2) require some sort of patching. They won't boot the GUI without these patches.

5D3 is in-between. Without any patches, it shows a blank screen (no menus), but other than that, it appears to boot almost completely. For example, it initializes the bitmap VRAM - that's why you were able to print Hello World. Patching date/time is enough to enable the GUI, menu navigation and info screens (which is a mystery to me - why is the behavior different from e.g. 60D?)
Title: Re: ML on EOS-M2
Post by: dfort on July 10, 2017, 03:16:27 AM
So I created and ran minimal ML versions for the EOSM and 700D. No GUI so I was messing around trying to find patches but then realized--if it doesn't initializes the bitmap VRAM then "Hello World" will never print, will it?

Maybe I already have a working EOSM2 minimal ML and don't know it?

That long printout I posted on pastebin (https://pastebin.com/AdwsiYM6) might be the print to screen code or maybe calculating the firmware signature? It ends with:

Code: [Select]
[DM] FROM Write Complete!!!
That doesn't look like an error message and the navigation keys seem to be working--so is that it?

I built in the minimal directory and from the platform/EOSM.103 directory with the firmware signature commented out and CONFIG_HELLO_WORLD defined -- both worked. I keep thinking that the next step is to get the firmware signature. Looks like we pretty much there on June 25 but then:

Took a closer look at the firmware signature - it's affected by our QEMU patches. So, to find it, either debug it without enabling other ROM patches (those with SET), or find it out outside GDB (e.g. print it on the screen), or just keep it commented out.

I kept it commented out but it looks like it is time to figure it out in order to get to the next step.

Exercises:
- (easy) patch the loop that times out (probably when initializing the touch IC) and the localI2C_Read loop; Check?
- (easy) print Hello World with the minimal ML target; Maybe check this off too?
- (moderate) print Hello World with the regular ML source code; Next exercise?
- (moderate) implement touch screen emulation;
- (moderate) bring ML menu in QEMU;
- (harder) full ML port, with most features working;
- (extreme) LiveView emulation.

Then there is this:

OK, so we don't have enough space to load the entire ML (maybe only if you compile it without any features). We'll have move somewhere else.
...
Note: I wouldn't call random stuff on a real camera without triple-checking the stub, the number of arguments, any prerequisites that might be and so on. Here we are on a virtual machine, so there's nothing to break - feel free to experiment away.

So at what point can we run a minimal ML on the camera?

(https://farm5.staticflickr.com/4239/35825884435_1a61ce1843_o.jpg)
Title: Re: ML on EOS-M2
Post by: a1ex on July 10, 2017, 09:26:03 AM
Quote
[DM] FROM Write Complete!!!

This message usually follows some sort of error (assert). The log has many DL messages, which means it was not run under GDB.

Quote
Maybe I already have a working EOSM2 minimal ML and don't know it?

Very likely :D

Quote
CONFIG_HELLO_WORLD

That won't work, since it will compile with the full feature set (thus reserving as much RAM as a full binary would). However, the minimal hello world requires very little RAM, so it should work (under GDB, with firmware signature commented out).
Title: Re: ML on EOS-M2
Post by: dfort on July 10, 2017, 05:08:03 PM
...the minimal hello world requires very little RAM, so it should work (under GDB, with firmware signature commented out).

Ok then, it's working. At least QEMU buttons seem to be working--no GUI.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb > log_file.txt 2> err_file.txt

Empty error file. That's a good thing, right? However, the log file shows that maybe things ended badly:

Code: [Select]

This GDB was configured as "--host=x86_64-apple-darwin10 --target=arm-none-eabi".

Breakpoint 1 at 0x408660d4: file ../../src/reboot.c, line 211.
Breakpoint 2 at 0x1900
Breakpoint 3 at 0x7360
Breakpoint 4 at 0xff0c5144
[      init:ff352260 ] task_create(PowerMgr, prio=20, stack=400, entry=ff352084, arg=0)
[      init:0003671c ] task_create(DbgMgr, prio=1f, stack=0, entry=36628, arg=46e584)
[      init:ff0f5f24 ] task_create(Startup, prio=19, stack=2800, entry=ff0f5d68, arg=46e880)
[      init:ff0c33d4 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28d8, arg=0)
[   Startup:ff0c375c ] task_create(Startup2, prio=11, stack=400, entry=ff0c3604, arg=0)
[   Startup:0003671c ] task_create(PropMgr, prio=14, stack=0, entry=36628, arg=4d7ca8)
[   Startup:ff0d261c ] task_create(DL, prio=f, stack=400, entry=ff0d23fc, arg=0)
[   Startup:0003671c ] task_create(EventMgr, prio=f, stack=0, entry=36628, arg=4e3898)
[   Startup:0003671c ] task_create(FileCache, prio=14, stack=0, entry=36628, arg=603d08)
[   Startup:0003671c ] task_create(RscMgr, prio=12, stack=0, entry=36628, arg=604110)
[   Startup:ff129dbc ] [ASSERT] this at ./FileMgr/FileMgr.c:2279, ff129dc0

Looks like it stopped on "FM_RegisterSpaceNotifyCallback" -- more patching to do?

Title: Re: ML on EOS-M2
Post by: dfort on July 10, 2017, 09:17:47 PM
Looks to me like something is going on in the GUI code. Does [DM] stand for Display Manager?

Tried to figure out what was going on here:

Code: [Select]
FF0C11BC: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
FF0C11BC: MCR p15, ...          : CACHEMAINT x2456 (omitted)
FF0C11BC: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0051079
FF0C11F0: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0051079
FF0C11DC: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C11F0: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0050079
FF0C113C: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0050079
...
FF0C1194: MCR p15, ...          : CACHEMAINT x257 (omitted)
FF0C1194: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0051079
FF0C11F0: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0051079
FF0C1180: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C11F0: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0050079
FF0C113C: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0050079
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C113C: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005007D
FF0C1168: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005007D
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C1168: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D


[DM] FROM Write Complete!!!

Checked stubs.S and it seems to me that these might GUI calls so I doubled checked all the GUI stubs and they look right.

This message usually follows some sort of error (assert). The log has many DL messages, which means it was not run under GDB.

Looks like this might be the assert:

Code: [Select]
[   Startup:ff129dbc ] [ASSERT] this at ./FileMgr/FileMgr.c:2279, ff129dc0
Here is a log with GDB. (https://pastebin.com/YnvhMATs)

I was trying to find a patch to get around this and came up with something rather interesting.

Code: [Select]
# FM_RegisterSpaceNotifyCallback
set *(int*)0xff129d94 = 0xe3a00001

What this does is to create the same issue when running EOSM2,firmware="boot=0" as it does with "boot=1" without this "patch". In other words, I was able to break it in the same way as the minimal ML. Running "boot=0" without the "patch" will boot all the way to the Canon GUI.

Feel like a kid sticking a fork into an electrical outlet to see what happens. As much as I'd like to see something running on the camera I'm glad we've got QEMU to work out these issues.
Title: Re: ML on EOS-M2
Post by: dfort on July 11, 2017, 06:45:41 AM
Did a little more QEMU'ing with the 550D. Why? Because it seems to be the best supported in QEMU and this:

OK, so we don't have enough space to load the entire ML (maybe only if you compile it without any features). We'll have move somewhere else.
...
One option is to try the AllocateMemory (like 550D or 1100D)...

So the minimal ML looks like this: [Edit] Check out that little dot in the lower right corner--I think it represents the camera's LED because it flashes on and off just like on the camera.

(https://farm5.staticflickr.com/4206/35681451342_8d6163ba37.jpg)

That "Hello World" in the retro graphical font shows up everywhere reminding you that ML is running. Kind of cool and annoying at the same time. Ok, we aren't going to do anything with a minimal ML so how about the full ML only without a firmware signature defined and "Hello World" configured?

(https://farm5.staticflickr.com/4264/35462590070_b8737a2014.jpg)

What is cool about this is that the 550D dump that I'm using doesn't have the boot flag enabled so it is pretty much like the EOSM2 ROMs that I'm using. Is is displaying the correct firmware signature--nice! That's what I thought we were going for with these QEMU exercises.

By the way, these were run with a simple:

Code: [Select]
./run_canon_fw.sh 550D,firmware="boot=1"
Ok so how about setting up AllocateMemory on the EOSM2?

I'm pretty sure that these are correct:

~/magic-lantern/platform/EOSM2.103/consts.h
Code: [Select]
// Used in boot-hack.c with CONFIG_ALLOCATE_MEMORY_POOL
#define ROM_ITASK_START 0xff0c57a0
#define ROM_ITASK_END  0xffc45984 // kinda iffy
#define ROM_CREATETASK_MAIN_START 0xff0c3064
#define ROM_CREATETASK_MAIN_END 0xff0c33dc
#define ROM_ALLOCMEM_END 0xff0c3074 // kinda iffy
#define ROM_ALLOCMEM_INIT 0xff0c307c // kinda iffy
#define ROM_B_CREATETASK_MAIN 0xff0c5814

Then this needs to be turned on:

~/magic-lantern/platform/EOSM2.103/internals.h
Code: [Select]
/** This camera loads ML into the AllocateMemory pool **/
#define CONFIG_ALLOCATE_MEMORY_POOL

Now there's some other adjustments that need to be made like maybe:

now you have to patch the DATA for the LDR. this is the address where the LDR     R3, =0x1F300000 gets the 1F3000000 from.
should be right below the function.
> #define RSCMGR_MEMORY_PATCH_END          0xF80.....   (yes, 0xF800!!! also on your cameras)

interesting that the 650D has 0x1F2C4000 bytes only.... hmm

after you defined that struct, make sure you set RESTARTSTART to 0x1FE00100  (original value minus 2MiB)

Ok--gotta ask a noob question, what is 2MiB? I looked it up but am not sure I'm getting it. Is it 0x800?

Anyway, I don't quite have all the pieces yet because it doesn't build:

Code: [Select]
...
[ LD       ]   magiclantern
boot-hack.o: In function `init_task_patched':
/Users/rosiefort/magic-lantern/platform/EOSM2.103/../../src/boot-hack.c:642: undefined reference to `reloc'
/Users/rosiefort/magic-lantern/platform/EOSM2.103/../../src/boot-hack.c:650: undefined reference to `reloc'
make: *** [magiclantern] Error 1

So reading further along that topic I ran into this:

EOS M2, QEMU:

(https://a1ex.magiclantern.fm/bleeding-edge/RscMgr/EOSM2.png)

Code: [Select]
  1062: 48619.520 [RSC] --- Common Top ----
  1063: 48619.520 [RSC] [TOP1]                  0x40C2A000
  1064: 48619.776 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
  1065: 48623.104 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
  1066: 48626.432 [RSC] SDS_DEVELOP_WORK        0x40C94000 0x00038000 229376
  1067: 48629.760 [RSC] DARKCUR_COMP_WORK       0x40CCC000 0x00020000 131072
  1069: 48637.440 [RSC] FENCING_WORK            0x40CEC000 0x00010000 65536
  1070: 48640.768 [RSC] DCFNO                   0x40CFC000 0x00004000 16384
  1071: 48644.096 [RSC] LVMARGE_P_DEF_DATA_1    0x40D00000 0x0000A000 40960
  1072: 48647.424 [RSC] LVMARGE_P_DEF_DATA_2    0x40D0A000 0x0000A000 40960
  1074: 48654.848 [RSC] LVMARGE_P_DEF_DATA_3    0x41FF0000 0x0000A000 40960
  1075: 48657.920 [RSC] LVMARGE_P_DEF_DATA_ZOOM 0x40D14000 0x0000C000 49152
  1076: 48661.504 [RSC] FILE_HEADER             0x40D20000 0x00240000 2359296
  1077: 48664.320 [RSC] SAF WORK                0x40FA0000 0x00200000 2097152
  1078: 48667.136 [RSC] BMPVRAM1                0x411A0000 0x00080000 524288
  1080: 48676.864 [RSC] BMPVRAM2                0x41220000 0x00080000 524288
  1081: 48680.704 [RSC] ENGINE_MIRROR           0x412A0000 0x00044000 278528
  1082: 48688.640 [RSC] VSHADING_COMP_WORK      0x412E4000 0x000DC000 901120
  1084: 48692.992 [RSC] STILL SCAR              0x413C0000 0x00075B00 482048
  1085: 48692.992 [RSC] TUNEDATA                0x41435B00 0x00140000 1310720
  1086: 48692.992 [RSC] TUNEDATA2               0x41575B00 0x00160000 1441792
  1087: 48693.504 [RSC] FIXDATA                 0x416D5B00 0x0021E500 2221312
  1088: 48693.504 [RSC] LVMARGE_P_DEF_DATA_CROP 0x418F4000 0x0000C000 49152
  1089: 48693.760 [RSC] WIRELESS_WORK2          0x41900000 0x00300000 3145728
  1090: 48699.648 [RSC] WIRELESS_WORK1   *      0x41C00000 0x00200000 2097152
  1091: 48702.464 [RSC] ADAPTER_TRANSFER *      0x0 0x00000000 0
  1092: 48702.464 [RSC] EEKO                    0x41E00000 0x001E0000 1966080
  1093: 48702.720 [RSC] SHOOTING_CREATIVEFILTER 0x41FE0000 0x00010000 65536
  1094: 48704.512 [RSC] EXMEM3_AREA_4           0x0 0x00000000 0
  1095: 48706.304 [RSC] --- Usually Mode ----
  1096: 48708.352 [RSC] MEMORY_MGR              0x42000000 0x04000000 67108864
  1098: 48715.520 [RSC] ONLY MEM1 1             0x46000000 0x02000000 33554432
  1099: 48717.568 [RSC] ONLY MEM1 2             0x48000000 0x02000000 33554432
  1100: 48721.152 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
  1101: 48724.736 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
  1102: 48727.808 [RSC] SS_DEVELOP1             0x4A000000 0x00E00000 14680064
  1104: 48734.976 [RSC] EXMEM3_AREA_2           0x4AE00000 0x000EA000 958464
  1105: 48738.816 [RSC] AVERAGE_WORK_TOP        0x4AEEA000 0x01116000 17915904
  1106: 48742.144 [RSC] AVERAGE_WORK_BOTTOM     0x4C000000 0x01116000 17915904
  1107: 48745.472 [RSC] SS_DEVELOP_OTHER_WORK   0x4D116000 0x00400000 4194304
  1109: 48752.384 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
  1110: 48753.920 [RSC] CAPTURE_WORK1           0x4D600000 0x00240000 2359296
  1111: 48760.576 [RSC] EXMEM3_AREA_1           0x4DD16000 0x016BA000 23830528
  1112: 48764.416 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
  1113: 48764.928 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
  1114: 48765.184 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
  1116: 48771.584 [RSC] ---   GIS Mode   ----
  1117: 48772.352 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
  1118: 48775.680 [RSC] WORK                    0x4A000000 0x08600000 140509184
  1119: 48775.680 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
  1120: 48775.680 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
  1121: 48779.520 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
  1122: 48779.520 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
  1123: 48782.848 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
  1124: 48787.200 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
  1126: 48790.528 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
  1127: 48796.160 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
  1128: 48796.160 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
  1129: 48797.696 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
  1130: 48800.256 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
  1131: 48802.304 [RSC] ---   HDR Mode   ----
  1133: 48806.912 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
  1134: 48808.192 [RSC] WORK                    0x4800000 0x0A600000 174063616
  1135: 48813.312 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
  1136: 48815.360 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
  1137: 48815.360 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
  1138: 48815.360 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
  1139: 48815.872 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
  1140: 48816.896 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
  1141: 48817.920 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
  1142: 48817.920 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
  1143: 48817.920 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
  1144: 48820.224 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
  1145: 48823.808 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
  1146: 48824.320 [RSC] ---    NR Mode   ----
  1148: 48829.952 [RSC] NR_MEMORY_MGR           0x42000000 0x08000000 134217728
  1149: 48833.024 [RSC] COMPOSITION_WORK_TOP    0x0 0x00000000 0
  1150: 48836.096 [RSC] COMPOSITION_WORK_BOTTOM 0x0 0x00000000 0
  1151: 48839.424 [RSC] ---    DP Mode   ----
  1152: 48841.216 [RSC] DP_SINGLE               0x42000000 0x05E00000 98566144
  1153: 48841.216 [RSC] DP_MULTI                0x47E00000 0x04D94000 81346560
  1154: 48843.520 [RSC] DP_CAPTURE_WORK1        0x4CB94000 0x00040000 262144
  1155: 48844.288 [RSC] DP_AVERAGE_TOP          0x4DBD4000 0x01116000 17915904
  1157: 48848.128 [RSC] DP_AVERAGE_BOTTOM       0x4ECEA000 0x01116000 17915904
  1158: 48849.408 [RSC] --- Indev Mode ----
  1159: 48851.200 [RSC] [INDVMGR]               0x0
  1160: 48852.480 [RSC] YUV                     0x4AEEA000 0x0222C000 35831808
  1161: 48852.480 [RSC] YUV_OUT                 0x42000000 0x0222C000 35831808
  1162: 48853.760 [RSC] INDV_WORK               0x0 0x00000000 0

256MB. Note the WORK region has invalid size (it would overflow), so I've patched it to get a proper graph.

No unused areas for us.

No unused areas for us?

Oh please say it ain't so!
Title: Re: ML on EOS-M2
Post by: dmilligan on July 12, 2017, 01:49:44 AM
https://en.wikipedia.org/wiki/Mebibyte
Title: Re: ML on EOS-M2
Post by: dfort on July 13, 2017, 03:06:04 AM
Thanks!

So -- 1 MiB = 220 bytes = 1024 kibibytes = 1048576bytes = 0x100000

~/magic-lantern/platform/EOSM2.103/Makefile.platform.default
Code: [Select]
# DryOS memory map
# RESTARTSTART is selected to be at user_mem_start
# (aka heap start / DRY_HEAP_START / malloc memory pool)
# (actually 0x00100d80 but make it a little bit higher)
#
RESTARTSTART = 0x100E00

RESTARTSTART   = (0x100E00 - 0x200000) = -0xF200

Uh oh.

I may have found the stubs needed for CONFIG_ALLOCATE_MEMORY_POOL. Got it to compile. Might be useful if we ever find some free memory.

Still don't think I've got minimal ML working. Shouldn't it boot all the way to the Canon GUI? Maybe this camera doesn't have enough memory even for a minimal ML?

Thought I'd take a baby step and try to get the Model ID working in the recovery branch. We know there's enough memory for that:

(https://farm5.staticflickr.com/4288/35756843091_b634e90675.jpg)

Yay!

The Camera model isn't working but it isn't working for the EOSM, 100D or 700D either. Haven't been able to figure out why.
Title: Re: ML on EOS-M2
Post by: a1ex on July 13, 2017, 12:26:27 PM
https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/ - compiled from dfort's repo.

(https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/13/artifact/qemu/EOSM2.png)

Build log (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/13/consoleFull)

Will explain CONFIG_ALLOCATE_MEMORY_POOL later.
Title: Re: ML on EOS-M2
Post by: dfort on July 13, 2017, 02:08:59 PM
From your console output (https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/13/consoleFull):

Code: [Select]
sed -i 's/0xFF0C1C94/0xFF0C1C90/' platform/EOSM2.103/consts.h

I was that close?

[EDIT] Minimal ML working perfectly over here now. Well, it needs the debugmsg.gdb file while the 550D doesn't but that's a QEMU mystery.

Title: Re: ML on EOS-M2
Post by: dfort on July 16, 2017, 01:32:15 AM
Will explain CONFIG_ALLOCATE_MEMORY_POOL later.

Turned it on:

internals.h
Code: [Select]
/** This camera loads ML into the AllocateMemory pool **/
#define CONFIG_ALLOCATE_MEMORY_POOL

and set up a special case for the EOSM2

boot-hack.c
Code: [Select]
    #if defined(CONFIG_550D)
    // change end limit to 0xc60000 => reserve 640K for ML
    *addr_AllocMem_end = MOV_R1_0xC60000_INSTR;
    ml_reserved_mem = 640 * 1024;
   
    #elif defined(CONFIG_EOSM2) // experimenting - Hello World autoexec.bin is 425K
    *addr_AllocMem_end = MOV_R1_0xC80000_INSTR;
    ml_reserved_mem = 512 * 1024;
   
    #else
    // change end limit to 0xc80000 => reserve 512K for ML
    *addr_AllocMem_end = MOV_R1_0xC80000_INSTR;
    ml_reserved_mem = 512 * 1024;
    #endif

Of course like everything I try this doesn't work right away.

However, according to this post (http://www.magiclantern.fm/forum/index.php?topic=5071.msg186874#msg186874) from the "placing ML into shoot memory" topic the EOSM2 only has 256MB, same with the 1300D, so does this mean trying to get ML running on these cameras is a lost cause or should we keep trying to shoehorn it in somehow?
Title: Re: ML on EOS-M2
Post by: dfort on July 16, 2017, 04:42:15 AM
Made a test pull request against unified from my repository (kept in sync with the master unified branch) just to see what has changed and make it easy to find what is still left to do.

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/4/eosm2103-wip/diff
Title: Re: ML on EOS-M2
Post by: a1ex on July 16, 2017, 10:06:22 AM
However, according to this post (http://www.magiclantern.fm/forum/index.php?topic=5071.msg186874#msg186874) from the "placing ML into shoot memory" topic the EOSM2 only has 256MB, same with the 1300D, so does this mean trying to get ML running on these cameras is a lost cause [...] ?

How much memory do your other cameras have?

How much memory does the 1100D have?
Title: Re: ML on EOS-M2
Post by: dfort on July 16, 2017, 08:02:45 PM
How much memory does the 1100D have?

According to the "placing ML into shoot memory" topic it has only 128MB (http://www.magiclantern.fm/forum/index.php?topic=5071.msg166993#msg166993) and of course it can run ML. So I started thinking, how did you do that trick? That is when I realized that I was in the Twilight Zone--in other words I was lost in another dimension, obsessing over shot memory and ignoring other options.

One option is to try the AllocateMemory (like 550D or 1100D), and another is to hunt for possibly unused memory areas (smemShowFix - FF222B68) - see http://www.magiclantern.fm/forum/index.php?topic=5071.0.

That lead me down a path where I was able to track down some more stubs that might be useful to experiment with the AllocateMemoryResource method used on the 60D and 7D:

Code: [Select]
/* here we are patching RscMgr/SRM initialisation to use less memory */
// used on 7D and 60D - found values for EOSM2 but haven't figured out how to implement it yet
// #define HIJACK_CACHE_HACK
// #define RSCMGR_MEMORY_PATCH_END          0xff0c4a8c
// #define HIJACK_CACHE_HACK_INITTASK_ADDR  0xff0c1c98

Of course this isn't working (not enough shot memory?) and am back to pondering over these options:

in canon's firmware we have three options how to allocate memory and where to place data.

1) malloc
2) AllocateMemory (MemoryManager)
3) AllocateMemoryResource (RscMgr, Srm)

By the way, I like looking at graphs better than lists of hexadecimal numbers and the graphs on the placing ML into shoot memory (http://www.magiclantern.fm/forum/index.php?topic=5071.0) topic have disappeared when using the Chrome browser. Looking into the issue it seems that Google has been implementing stricter rules that is affecting the link to your graph: https://a1ex.magiclantern.fm/bleeding-edge/RscMgr/EOSM2.png

(https://farm5.staticflickr.com/4298/35920873166_b2b57e31eb.jpg)

That graph is helpful so if you don't mind I uploaded it to Flickr in case that is happening with others who are following this topic.

(https://farm5.staticflickr.com/4323/35122119534_84ba616817_z.jpg) (https://flic.kr/p/VvC3v5)EOSM2 (https://flic.kr/p/VvC3v5) by Daniel Fort (https://www.flickr.com/photos/dan_fort/), on Flickr

Ok--so what are we looking at here? Is that two ways of looking at the same thing? Before and after memory allocation? Something there must have caused your to come up with this:

...I'm thinking to move autoexec.bin to AllocateMemory, and use the 7MB block as general-purpose memory for ML, using some custom allocator (maybe Canon's).

It hasn't sunk into my thick skull yet and I found myself in another Catch 22 situation. In order to create the data needed for these graphs the instructions (http://www.magiclantern.fm/forum/index.php?topic=5071.msg166799#msg166799) are to:

- compile the dm-spy-experiments branch, with CONFIG_DEBUG_INTERCEPT=y in Makefile.user
- put this in don't click me...

(For the EOSM2)
Code: [Select]
    void debug_intercept();
    debug_intercept();
    void (*smemShowFix)(void) = 0xff222b6c;
    smemShowFix();
    debug_intercept();

There are also some stubs to look for so I followed the excellent example by @josepvm and translated it to the EOSM2

Code: [Select]
/** Low-level allocators */

NSTUB(0x79fc, init_memory_pool) 
NSTUB(0x8080, allocate_memory_from_pool)
NSTUB(0x83ec, free_memory_to_pool)
NSTUB(0x7ea0, get_max_region_of_pool)

Now how did you run that? I've got a feeling you're 10 steps ahead of me and already have a full ML for the EOSM2 working in QEMU--or you have an EOSM2 running ML and are keeping it a secret.
Title: Re: ML on EOS-M2
Post by: a1ex on July 17, 2017, 12:39:54 AM
Now how did you run that?

As simple as... (http://www.magiclantern.fm/forum/index.php?topic=15895.msg186872#msg186872)

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
CTRL-C
(gdb) set $pc = 0xff222b64
(gdb) continue

Quote
Is that two ways of looking at the same thing?
Yes.

Quote
Before and after memory allocation?
No.

Quote
Ok--so what are we looking at here?

Besides standard C malloc (aka user_mem in DryOS), and a similar allocator named AllocateMemory (sys_mem - nope, that's a third one), which both live in a tiny space of about 12MB (the gray bar at the left - 13MB on most models), for the memory-hungry stuff, Canon code uses a bunch of hardcoded regions (which may change according to operating modes). The graph shows these regions (top: scaled to show their real size, bottom: scaled so you can read the text).

Some of these regions are managed by allocators (shoot_malloc, srm_malloc), but it's not very clear which and when. It's one thing in photo mode and another in LiveView. Consider modes like video recording, raw developing, direct print (anyone uses it?) or wifi operation, and it starts to get really complicated. The safest way is to rely on these allocators when a large amount of memory is needed, and free it as soon as you no longer need it (otherwise you may not be able to change between some of the above modes). That's why the memory backend avoids shoot/srm allocators until it has no other choice. That's also the explanation for the second way to trigger ERR70 described in the stack trace PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/849/stack-trace-aka-backtrace-in-crash-logs/diff).

In some cases, there are memory areas that are never used by Canon code (they are marked as full-height gray bars). We can double-check whether these are really unused (by making sure nobody writes to them while operating the camera in all possible modes we can think of), and if they are really free, we can take them. Caveat: they might be using stuff that's not listed in this log (as parts of that log are also hardcoded, so there's nothing to make sure what's in the log actually matches reality - they could have printed any numbers). On 60D, a region identified as free in this graph was later found to be actually used by Canon code (see the RscMgr PR, where we ran into trouble because of that). So, take this graph with a grain of salt.

For ML, we only need about 512K to load autoexec.bin (compare that to 128MB or even 64 for the 1000D). The other stuff (modules, work memory, whatever) are allocated dynamically, which is less troublesome, as the memory backend figures out which allocators to use, and lets you configure additional ones. But the initial code (autoexec.bin) has to be loaded statically, at some predefined address (unless you relocate it).

Of course, we can move everything to modules and end up with a very small static binary. It's a little more difficult than copy/paste though. Or you can try to compile as Thumb (good luck figuring it out).

Or, we can disable some features. You can even comment out features.h, CONFIG_RAW_* and fix up whatever gives compilation errors - that way, you'll get just the menu with a few debug items inside. It's best to do that anyway, and enable features one by one - otherwise you may end up with lots of possibly broken features (as it's really hard to test every single thing - I used to do that manually years ago), and if it's not something really widely used, you get a bug report after some months or years (even if the feature is not working at all).

Exercise: draw a similar memory map for DryOS (the first ~13 MB), based on the info from this post (http://www.magiclantern.fm/forum/index.php?topic=15895.msg186872#msg186872). Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

After finishing this, you will have both the Hello World working and a very good understanding of how the memory is organized.
Title: Re: ML on EOS-M2
Post by: dfort on July 17, 2017, 05:35:44 PM
Excellent! Here are some highlights of the output--totaling over 3625 lines!

Code: [Select]
^C
Program received signal SIGINT, Interrupt.
0xff352318 in ?? ()
(gdb) set $pc = 0xff222b6c
(gdb) continue
Continuing.
[    PowerMgr:ff222b78 ] (80:16)
   185:  3179.520 [RSC]
[    PowerMgr:ff222b94 ] (80:16) --- Common Top ----
   186:  3179.520 [RSC] --- Common Top ----
[    PowerMgr:ff222bb0 ] (80:16) [TOP1]                  0x40C2A000
   187:  3179.776 [RSC] [TOP1]                  0x40C2A000
[    PowerMgr:ff222bd4 ] (80:16) REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
   188:  3179.776 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
[    PowerMgr:ff222bfc ] (80:16) SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
   189:  3180.032 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
...
[    PowerMgr:ff223f78 ] (80:16) INDV_WORK               0x0 0x00000000 0
   274:  3187.712 [RSC] INDV_WORK               0x0 0x00000000 0
[    PowerMgr:ff0c2994 ] (8b:16) ###exceptionhandlercbr 0xda2c 0
   275:  3187.712 [STARTUP] ###exceptionhandlercbr 0xda2c 0
< Error Exception>
TYPE        : 4
ISR         : 0
TASK IDSR   : 1835011
TASK Name   : PowerMgr
R 0         : 0
R 1         : 19980218
R 2         : 0
R 3         : 0
R 4         : 19980218
R 5         : 19980218
R 6         : 19980218
R 7         : 19980218
R 8         : 19980218
R 9         : ca18
R10         : 19980218
R11         : 19980218
R12         : 1008
R13         : b89a0
R14         : da2c
PC          : da2c
CPSR        : 600000d3
[    PowerMgr:ff0c2ac4 ] (8b:16) #####exceptionhandlercbr 0xda2c
   276:  3188.480 [STARTUP] #####exceptionhandlercbr 0xda2c
[    PowerMgr:ff0c2ad4 ] (8b:03) < Error Exception>         
[    PowerMgr:ff0c2ae8 ] (8b:03) TYPE        : 4         
[    PowerMgr:ff0c2afc ] (8b:03) ISR         : 0         
[    PowerMgr:ff0c2b10 ] (8b:03) TASK IDSR   : 1835011       
[    PowerMgr:ff0c2b24 ] (8b:03) TASK Name   : PowerMgr     
[    PowerMgr:ff0c2b38 ] (8b:03) R 0         : 0       
[    PowerMgr:ff0c2b4c ] (8b:03) R 1         : 19980218   
[    PowerMgr:ff0c2b60 ] (8b:03) R 2         : 0     
[    PowerMgr:ff0c2b74 ] (8b:03) R 3         : 0       
[    PowerMgr:ff0c2b88 ] (8b:03) R 4         : 19980218       
[    PowerMgr:ff0c2b9c ] (8b:03) R 5         : 19980218
[    PowerMgr:ff0c2bb0 ] (8b:03) R 6         : 19980218       
[    PowerMgr:ff0c2bc4 ] (8b:03) R 7         : 19980218         
[    PowerMgr:ff0c2bd8 ] (8b:03) R 8         : 19980218         
[    PowerMgr:ff0c2bec ] (8b:03) R 9         : ca18         
[    PowerMgr:ff0c2c00 ] (8b:03) R10         : 19980218
[    PowerMgr:ff0c2c14 ] (8b:03) R11         : 19980218       
[    PowerMgr:ff0c2c28 ] (8b:03) R12         : 1008         
[    PowerMgr:ff0c2c3c ] (8b:03) R13         : b89a0         
[    PowerMgr:ff0c2e7c ] (8b:03) R14         : da2c           
[    PowerMgr:ff0c2e90 ] (8b:03) PC          : da2c             
[    PowerMgr:ff0c2ea4 ] (8b:03) CPSR        : 600000d3               
[    PowerMgr:ff0c2ee8 ] (8b:16) Exception : Time 2000/1/1 0:0:3
   277:  3188.736 [STARTUP] Exception : Time 2000/1/1 0:0:3
[      DbgMgr:ff1470d0 ] (00:01) [PM] DisablePowerSave (Counter = 2)
FF0C11BC: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
FF0C11BC: MCR p15, ...          : CACHEMAINT x11722 (omitted)
FF0C11BC: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0051079
FF0C11F0: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0051079
...
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C113C: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005007D
FF0C1168: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005007D
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C1168: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D


[DM] FROM Write Complete!!!
[      DbgMgr:ff147140 ] (00:01) [PM] EnablePowerSave (Counter = 1)



And trimming it down to just what we're interested in:

Code: [Select]
   186:  3179.520 [RSC] --- Common Top ----
   187:  3179.776 [RSC] [TOP1]                  0x40C2A000
   188:  3179.776 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
   189:  3180.032 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
   190:  3180.032 [RSC] SDS_DEVELOP_WORK        0x40C94000 0x00038000 229376
   191:  3180.032 [RSC] DARKCUR_COMP_WORK       0x40CCC000 0x00020000 131072
   192:  3180.288 [RSC] FENCING_WORK            0x40CEC000 0x00010000 65536
   193:  3180.288 [RSC] DCFNO                   0x40CFC000 0x00004000 16384
   194:  3180.544 [RSC] LVMARGE_P_DEF_DATA_1    0x40D00000 0x0000A000 40960
   195:  3180.544 [RSC] LVMARGE_P_DEF_DATA_2    0x40D0A000 0x0000A000 40960
   196:  3180.800 [RSC] LVMARGE_P_DEF_DATA_3    0x41FF0000 0x0000A000 40960
   197:  3180.800 [RSC] LVMARGE_P_DEF_DATA_ZOOM 0x40D14000 0x0000C000 49152
   198:  3180.800 [RSC] FILE_HEADER             0x40D20000 0x00240000 2359296
   199:  3181.056 [RSC] SAF WORK                0x40FA0000 0x00200000 2097152
   200:  3181.056 [RSC] BMPVRAM1                0x411A0000 0x00080000 524288
   201:  3181.056 [RSC] BMPVRAM2                0x41220000 0x00080000 524288
   202:  3181.312 [RSC] ENGINE_MIRROR           0x412A0000 0x00044000 278528
   203:  3181.568 [RSC] VSHADING_COMP_WORK      0x412E4000 0x000DC000 901120
   204:  3181.568 [RSC] STILL SCAR              0x413C0000 0x00075B00 482048
   205:  3181.568 [RSC] TUNEDATA                0x41435B00 0x00140000 1310720
   206:  3181.824 [RSC] TUNEDATA2               0x41575B00 0x00160000 1441792
   207:  3181.824 [RSC] FIXDATA                 0x416D5B00 0x0021E500 2221312
   208:  3181.824 [RSC] LVMARGE_P_DEF_DATA_CROP 0x418F4000 0x0000C000 49152
   209:  3182.080 [RSC] WIRELESS_WORK2          0x41900000 0x00300000 3145728
   210:  3182.080 [RSC] WIRELESS_WORK1   *      0x41C00000 0x00200000 2097152
   211:  3182.080 [RSC] ADAPTER_TRANSFER *      0x0 0x00000000 0
   212:  3182.336 [RSC] EEKO                    0x41E00000 0x001E0000 1966080
   213:  3182.336 [RSC] SHOOTING_CREATIVEFILTER 0x41FE0000 0x00010000 65536
   214:  3182.336 [RSC] EXMEM3_AREA_4           0x0 0x00000000 0
   215:  3182.592 [RSC] --- Usually Mode ----
   216:  3182.592 [RSC] MEMORY_MGR              0x42000000 0x04000000 67108864
   217:  3182.592 [RSC] ONLY MEM1 1             0x46000000 0x02000000 33554432
   218:  3182.848 [RSC] ONLY MEM1 2             0x48000000 0x02000000 33554432
   219:  3182.848 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   220:  3182.848 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   221:  3183.104 [RSC] SS_DEVELOP1             0x4A000000 0x00E00000 14680064
   222:  3183.104 [RSC] EXMEM3_AREA_2           0x4AE00000 0x000EA000 958464
   223:  3183.104 [RSC] AVERAGE_WORK_TOP        0x4AEEA000 0x01116000 17915904
   224:  3183.360 [RSC] AVERAGE_WORK_BOTTOM     0x4C000000 0x01116000 17915904
   225:  3183.360 [RSC] SS_DEVELOP_OTHER_WORK   0x4D116000 0x00400000 4194304
   226:  3183.360 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   227:  3183.616 [RSC] CAPTURE_WORK1           0x4D600000 0x00240000 2359296
   228:  3183.616 [RSC] EXMEM3_AREA_1           0x4DD16000 0x016BA000 23830528
   229:  3183.872 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   230:  3183.872 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   231:  3183.872 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   232:  3183.872 [RSC] ---   GIS Mode   ----
   233:  3184.128 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
   234:  3184.128 [RSC] WORK                    0x4A000000 0x08600000 140509184
   235:  3184.128 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   236:  3184.384 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   237:  3184.384 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
   238:  3184.384 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
   239:  3184.640 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   240:  3184.640 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
   241:  3184.640 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   242:  3184.896 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   243:  3184.896 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   244:  3185.152 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
   245:  3185.152 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
   246:  3185.152 [RSC] ---   HDR Mode   ----
   247:  3185.152 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
   248:  3185.408 [RSC] WORK                    0x4800000 0x0A600000 174063616
   249:  3185.408 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   250:  3185.408 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   251:  3185.664 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
   252:  3185.664 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
   253:  3185.664 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   254:  3185.920 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
   255:  3185.920 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   256:  3185.920 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   257:  3186.176 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   258:  3186.176 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
   259:  3186.176 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
   260:  3186.432 [RSC] ---    NR Mode   ----
   261:  3186.688 [RSC] NR_MEMORY_MGR           0x42000000 0x08000000 134217728
   262:  3186.688 [RSC] COMPOSITION_WORK_TOP    0x0 0x00000000 0
   263:  3186.688 [RSC] COMPOSITION_WORK_BOTTOM 0x0 0x00000000 0
   264:  3186.688 [RSC] ---    DP Mode   ----
   265:  3186.944 [RSC] DP_SINGLE               0x42000000 0x05E00000 98566144
   266:  3186.944 [RSC] DP_MULTI                0x47E00000 0x04D94000 81346560
   267:  3186.944 [RSC] DP_CAPTURE_WORK1        0x4CB94000 0x00040000 262144
   268:  3187.200 [RSC] DP_AVERAGE_TOP          0x4DBD4000 0x01116000 17915904
   269:  3187.200 [RSC] DP_AVERAGE_BOTTOM       0x4ECEA000 0x01116000 17915904
   270:  3187.200 [RSC] --- Indev Mode ----
   271:  3187.456 [RSC] [INDVMGR]               0x0
   272:  3187.456 [RSC] YUV                     0x4AEEA000 0x0222C000 35831808
   273:  3187.456 [RSC] YUV_OUT                 0x42000000 0x0222C000 35831808
   274:  3187.712 [RSC] INDV_WORK               0x0 0x00000000 0

It is also interesting to look at an alternate version and follow along in the disassembly:

Code: [Select]
[    PowerMgr:ff223a54 ] (80:16) ---   HDR Mode   ----
[    PowerMgr:ff223a74 ] (80:16) TEMPMEM1                0x48000000 0x02000000 33554432
[    PowerMgr:ff223a94 ] (80:16) WORK                    0x4800000 0x0A600000 174063616
[    PowerMgr:ff223ab4 ] (80:16) IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
[    PowerMgr:ff223ad4 ] (80:16) IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
[    PowerMgr:ff223af4 ] (80:16) MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
[    PowerMgr:ff223b14 ] (80:16) MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
[    PowerMgr:ff223b34 ] (80:16) SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
[    PowerMgr:ff223b54 ] (80:16) MOVIE_STREAM            0x4DD16000 0x01400000 20971520
[    PowerMgr:ff223b74 ] (80:16) IMGVRAM1                0x4F3D0000 0x00410000 4259840
[    PowerMgr:ff223d90 ] (80:16) IMGVRAM2                0x4F7E0000 0x00410000 4259840
[    PowerMgr:ff223db0 ] (80:16) IMGVRAM3                0x4FBF0000 0x00410000 4259840
[    PowerMgr:ff223dd0 ] (80:16) EXMEM3_1_AREA           0x42000000 0x06000000 100663296
[    PowerMgr:ff223dec ] (80:16) EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848

BTW--interesting side trip into the Thumb (https://en.wikipedia.org/wiki/ARM_architecture#Thumb) instruction set. Looks like it is possible to mix it with standard ARM instructions (http://www.simplemachines.it/doc/ARM_COMBO_ap01.html).

Ok--back to work:

Exercise: draw a similar memory map for DryOS (the first ~13 MB), based on the info from this post (http://www.magiclantern.fm/forum/index.php?topic=15895.msg186872#msg186872). Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.
Title: Re: ML on EOS-M2
Post by: a1ex on July 17, 2017, 06:15:22 PM
You may wonder - why are all these messages doubled?
Code: [Select]
[    PowerMgr:ff222b94 ] (80:16) --- Common Top ----
   186:  3179.520 [RSC] --- Common Top ----

The first format is written by "-d debugmsg" (we are printing the arguments to DebugMsg and some context - task and address where it was called from).

The second format is written to UART (possibly one of these pins (https://www.magiclantern.fm/forum/index.php?topic=7531)) by Canon code - they do that for important messages (whatever that means - we don't know much about it (http://magiclantern.wikia.com/wiki/Debugging_Magic_Lantern#DebugMsg_and_DEBUG.28.29_macro)).

You can redirect the UART messages to a file (e.g. "-serial file:uart.log"). The bootloader menu (printed e.g. if you start from a bootable card without autoexec.bin present) also goes to the serial port - you can interact with it from the console if you use "-serial stdio". These are standard QEMU options - you can use them on any other guest OS that uses a serial port.

TODO: the serial console is also usable while running the firmware (and you can call these info functions from there - example (https://nada-labs.net/2014/finding-jtag-on-a-canon-elph100hs-ixus115/)). The emulation is probably missing an interrupt or some status bits on the UART in order to accept incoming characters.
Title: Re: ML on EOS-M2
Post by: dfort on July 22, 2017, 12:59:44 AM
I've amassed a boat load of data but have no idea how to draw a graph that makes any sense of it.

Feels like this is very close. Here are some highlights of where things are at (using HIJACK_CACHE_HACK):

Code: [Select]
SD LOAD OK.
Open file for read : AUTOEXEC.BIN
...
File size : 0x67A00
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] patching init_task from 0xff0c57a0 (-15968352)
[BOOT] autoexec.bin loaded at 100E00 - 178140.

Great -- autoexec.bin is loaded, next surprise is that the Canon GUI comes up in the QEMU window.

Code: [Select]
[     FileMgr:ff304f9c ] (24:02) PROP_TFT_STATUS = 0
[     FileMgr:ff304fb4 ] (24:03) LED Permit
[MPU] Received: 06 05 03 19 00 00  (spell #52)
[    PowerMgr:ff13826c ] (82:01) VdInterruptHandler bmp=ff136384 img=0 localWaitBmpCBR=ff136384
[    PowerMgr:ff136398 ] (82:01) WaitBmpCBR(local) pParam=0
[  DisplayMgr:ff13a118 ] (82:03) MuteOffDisplay finish Type=0
[  DisplayMgr:ff1790cc ] (9a:02) fTunOnDisp = On

Then it gets ugly:

Code: [Select]
[      init:0010137c ] task_create(ml_init, prio=1e, stack=4000, entry=101510, arg=0)
[     ml_init:ff696760 ] (23:01) sdReadBlk: st=73344, num=32, buf=0x4035e000
[     ml_init:ff6951bc ] (23:01) sdDMAReadBlk: st=73344, num=32
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

R00=ffffffff R01=00987c0a R02=001f3912 R03=00000000
R04=19980030 R05=00000000 R06=596f0ae0 R07=19980218
R08=1998002e R09=19980218 R10=19980218 R11=19980218
R12=00000007 R13=001f3918 R14=00101cc0 R15=20000000
PSR=60000013 -ZC- A svc32
FPSCR: 00000000
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

Here's the full console output: https://pastebin.com/5P45Ujtz

Homework assignment --> F-incomplete (graph missing, Hello World not working)

(gdb) set $pc = 0xFF0C6E70
Code: [Select]
# No ML
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max        515828

# Minimal ML
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00104580
user_mem_max        501492

# Current almost working full ML Hello World
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max        515828

(gdb) set $pc = 0xff0c750c
Code: [Select]
# No ML
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x0017ea60
  Total Size          = 0x0007dcd8 (   515288)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00068228 (   426536)
  Free Block Max Size = 0x000681f0 (   426480)
  Free Block Count    = 0x00000002 (        2)

# Minimal ML
Malloc Information (onetime type)
  Start Address       = 0x00104588
  End Address         = 0x0017ea60
  Total Size          = 0x0007a4d8 (   500952)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00064a28 (   412200)
  Free Block Max Size = 0x000649f0 (   412144)
  Free Block Count    = 0x00000002 (        2)

# Current almost working full ML Hello World
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x0017ea60
  Total Size          = 0x0007dcd8 (   515288)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00068228 (   426536)
  Free Block Max Size = 0x000681f0 (   426480)
  Free Block Count    = 0x00000002 (        2)

(gdb) set $pc = 0xff134304
Code: [Select]
# No ML
   189:  1284.352 [WINSYS] allocated=44140 max_region=412876

# Minimal ML
   185:  1840.896 [WINSYS] allocated=44140 max_region=412876

# Current almost working full ML Hello World
   185:  1042.176 [WINSYS] allocated=44140 max_region=412876

# No change in the above but when experimenting with CONFIG_ALLOCATE_MEMORY_POOL this came up:
[    PowerMgr:ff13432c ] (04:16) allocated=0 max_region=1568300

(gdb) set $pc = 0xff1343f4
Code: [Select]
# No ML
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  1667.328 [MEM] NG AllocateMemory 0
   186:  1667.584 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a320c
   187:  1667.840 [MEM] Num Alloc = 4864, Num Free = 460

# Minimal ML
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  2236.928 [MEM] NG AllocateMemory 0
   186:  2236.928 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a31a0
   187:  2237.184 [MEM] Num Alloc = 4864, Num Free = 460

# Current almost working full ML Hello World
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  1226.752 [MEM] NG AllocateMemory 0
   186:  1227.008 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a31f0
   187:  1227.264 [MEM] Num Alloc = 4864, Num Free = 460
Title: Re: ML on EOS-M2
Post by: dfort on July 22, 2017, 04:19:51 AM
My previous post was about what is almost working over here. This post is about what should be working but is a little further away than "almost."

#define CONFIG_ALLOCATE_MEMORY_POOL

Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

Is that QEMU source hint in memcheck.c? There are stubs in there but only for the 500D. So other cameras don't need this or is the 500D supported better in QEMU than any other camera? Ok, lots of other interesting stuff in there including something that looks like it is triggering the error message on my HIJACK_CACHE_HACK attempt:

Code: [Select]
assert(addr < 0x20000000);
[EDIT] This also looks like it can be useful once I learn how to use it:

Code: [Select]
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf ~/magic-lantern/platform/EOSM2.103/magiclantern -a | grep " memcpy$"
  5694: 00148c94   360 FUNC    GLOBAL DEFAULT    1 memcpy

I'm still trying to understand how AllocateMemory works and believe me I'm trying. Here I created a huge table and found that while most addresses didn't change between No ML, Minimal ML and the almost working HIJACK_CACHE_HACK build, there were some very significant changes in the AllocateMemory attempt.

Code: [Select]
   186:  1221.888 [RSC] --- Common Top ----     No ML                               Minimal ML                          Almost working full ML Hello World  AllocateMemory (not working)                             
   187:  1221.888 [RSC] [TOP1]                  0x40C2A000                          0x40C2A000                          0x40C2A000                          0x0
   188:  1222.144 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800        0x40C2A000 0x00032000 204800        0x40C2A000 0x00032000 204800        0x0 0x00032000 204800
   189:  1222.144 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376        0x40C5C000 0x00038000 229376        0x40C5C000 0x00038000 229376        0x32000 0x00038000 229376
   190:  1222.400 [RSC] SDS_DEVELOP_WORK        0x40C94000 0x00038000 229376        0x40C94000 0x00038000 229376        0x40C94000 0x00038000 229376        0x6A000 0x00038000 229376
   191:  1222.400 [RSC] DARKCUR_COMP_WORK       0x40CCC000 0x00020000 131072        0x40CCC000 0x00020000 131072        0x40CCC000 0x00020000 131072        0xA2000 0x00020000 131072
   192:  1222.656 [RSC] FENCING_WORK            0x40CEC000 0x00010000 65536         0x40CEC000 0x00010000 65536         0x40CEC000 0x00010000 65536         0xC2000 0x00010000 65536
   193:  1222.656 [RSC] DCFNO                   0x40CFC000 0x00004000 16384         0x40CFC000 0x00004000 16384         0x40CFC000 0x00004000 16384         0xD2000 0x00004000 16384
   194:  1222.912 [RSC] LVMARGE_P_DEF_DATA_1    0x40D00000 0x0000A000 40960         0x40D00000 0x0000A000 40960         0x40D00000 0x0000A000 40960         0xD6000 0x0000A000 40960
   195:  1223.168 [RSC] LVMARGE_P_DEF_DATA_2    0x40D0A000 0x0000A000 40960         0x40D0A000 0x0000A000 40960         0x40D0A000 0x0000A000 40960         0xE0000 0x0000A000 40960
   196:  1223.168 [RSC] LVMARGE_P_DEF_DATA_3    0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960
   197:  1223.424 [RSC] LVMARGE_P_DEF_DATA_ZOOM 0x40D14000 0x0000C000 49152         0x40D14000 0x0000C000 49152         0x40D14000 0x0000C000 49152         0xEA000 0x0000C000 49152
   198:  1223.424 [RSC] FILE_HEADER             0x40D20000 0x00240000 2359296       0x40D20000 0x00240000 2359296       0x40D20000 0x00240000 2359296       0xF6000 0x00240000 2359296
   199:  1223.680 [RSC] SAF WORK                0x40FA0000 0x00200000 2097152       0x40FA0000 0x00200000 2097152       0x40FA0000 0x00200000 2097152       0x376000 0x00200000 2097152
   200:  1223.936 [RSC] BMPVRAM1                0x411A0000 0x00080000 524288        0x411A0000 0x00080000 524288        0x411A0000 0x00080000 524288        0x576000 0x00080000 524288
   201:  1223.936 [RSC] BMPVRAM2                0x41220000 0x00080000 524288        0x41220000 0x00080000 524288        0x41220000 0x00080000 524288        0x5F6000 0x00080000 524288
   202:  1224.192 [RSC] ENGINE_MIRROR           0x412A0000 0x00044000 278528        0x412A0000 0x00044000 278528        0x412A0000 0x00044000 278528        0x676000 0x00044000 278528
   203:  1224.192 [RSC] VSHADING_COMP_WORK      0x412E4000 0x000DC000 901120        0x412E4000 0x000DC000 901120        0x412E4000 0x000DC000 901120        0x6BA000 0x000DC000 901120
   204:  1224.192 [RSC] STILL SCAR              0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048
   205:  1224.448 [RSC] TUNEDATA                0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720
   206:  1224.704 [RSC] TUNEDATA2               0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792
   207:  1224.704 [RSC] FIXDATA                 0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312
   208:  1224.704 [RSC] LVMARGE_P_DEF_DATA_CROP 0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152
   209:  1224.960 [RSC] WIRELESS_WORK2          0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728
   210:  1224.960 [RSC] WIRELESS_WORK1   *      0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152
   211:  1224.960 [RSC] ADAPTER_TRANSFER *      0x0 0x00000000                      00x0 0x00000000                     00x0 0x0000000                      00x0 0x00000000 0
   212:  1225.216 [RSC] EEKO                    0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080
   213:  1225.216 [RSC] SHOOTING_CREATIVEFILTER 0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536
   214:  1225.216 [RSC] EXMEM3_AREA_4           0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   215:  1225.472 [RSC] --- Usually Mode ----
   216:  1225.472 [RSC] MEMORY_MGR              0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864
   217:  1225.472 [RSC] ONLY MEM1 1             0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432
   218:  1225.728 [RSC] ONLY MEM1 2             0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432
   219:  1225.728 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   220:  1225.728 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   221:  1225.984 [RSC] SS_DEVELOP1             0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064
   222:  1225.984 [RSC] EXMEM3_AREA_2           0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464
   223:  1225.984 [RSC] AVERAGE_WORK_TOP        0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904
   224:  1226.240 [RSC] AVERAGE_WORK_BOTTOM     0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904
   225:  1226.240 [RSC] SS_DEVELOP_OTHER_WORK   0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304
   226:  1226.496 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   227:  1226.496 [RSC] CAPTURE_WORK1           0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296
   228:  1226.496 [RSC] EXMEM3_AREA_1           0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528
   229:  1226.752 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   230:  1226.752 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   231:  1226.752 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   232:  1227.008 [RSC] ---   GIS Mode   ----
   233:  1227.008 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432
   234:  1227.008 [RSC] WORK                    0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184
   235:  1227.264 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   236:  1227.264 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   237:  1227.264 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000
   238:  1227.520 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064
   239:  1227.520 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   240:  1227.776 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520
   241:  1227.776 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   242:  1228.032 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   243:  1228.032 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   244:  1228.032 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296
   245:  1228.288 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848
   246:  1228.288 [RSC] ---   HDR Mode   ----
   247:  1228.288 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 3355443       20x48000000 0x02000000 33554432
   248:  1228.544 [RSC] WORK                    0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616
   249:  1228.544 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   250:  1228.544 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   251:  1228.800 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000
   252:  1228.800 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064
   253:  1228.800 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   254:  1229.056 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520
   255:  1229.312 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   256:  1229.312 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   257:  1229.568 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   258:  1229.568 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296
   259:  1229.824 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848
   260:  1229.824 [RSC] ---    NR Mode   ----
   261:  1229.824 [RSC] NR_MEMORY_MGR           0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728
   262:  1230.080 [RSC] COMPOSITION_WORK_TOP    0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   263:  1230.080 [RSC] COMPOSITION_WORK_BOTTOM 0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   264:  1230.080 [RSC] ---    DP Mode   ----
   265:  1230.336 [RSC] DP_SINGLE               0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144
   266:  1230.336 [RSC] DP_MULTI                0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560
   267:  1230.336 [RSC] DP_CAPTURE_WORK1        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144
   268:  1230.592 [RSC] DP_AVERAGE_TOP          0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904
   269:  1230.592 [RSC] DP_AVERAGE_BOTTOM       0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904
   270:  1230.848 [RSC] --- Indev Mode ----
   271:  1230.848 [RSC] [INDVMGR]               0x00x00x00x0
   272:  1230.848 [RSC] YUV                     0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808
   273:  1230.848 [RSC] YUV_OUT                 0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808
   274:  1231.104 [RSC] INDV_WORK               0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0

I've also been pouring over the code and zeroed in on this comment:

boot-hack.c
Code: [Select]
    // We shrink the AllocateMemory (system memory) pool in order to make space for ML binary
    // Example for the 1100D firmware
    // ff0197d8: init_task:
    // ff01984c: b CreateTaskMain
    //
    // ff0123c4 CreateTaskMain:
    // ff0123e4: mov r1, #13631488  ; 0xd00000  <-- end address
    // ff0123e8: mov r0, #3997696   ; 0x3d0000  <-- start address
    // ff0123ec: bl  allocatememory_init_pool

    // So... we need to patch CreateTaskMain, which is called by init_task.
    //
    // First we use Trammell's reloc.c code to relocate init_task and CreateTaskMain...

I looked over my disassemble.pl version of the 1100D (and 550D) and without giving away any more than what was already published--came up with this for the EOSM2:

Code: [Select]
EOSM2 stub:
NSTUB(0xFF0C57A0,  init_task)

ff0c57a0: push {r4, lr}
ff0c5814: b loc_ff0c3064 <-- call CreateTaskMain
...
ff0c3064: push {r1, r2, r3, r4, r5, lr}
...
ff0c30a4: ldr r1, [pc, #344] ; ff0c3204: (00c2a000)   <-- end address??
ff0c30a8: ldr r0, [pc, #344] ; ff0c3208: (0046c000)   <-- start address??
...
ff0c30c4: bl loc_ff146d3c <-- allocatememory_init_pool?

I'm pretty sure about the allocatememory_init_pool but not sure at all that I found the start and end addresses. I've been trying different addresses for ROM_ALLOCMEM_END but keep getting the same disappointing results.

consts.h
Code: [Select]
// Used in boot-hack.c with CONFIG_ALLOCATE_MEMORY_POOL
#define ROM_ITASK_START 0xff0c57a0
#define ROM_ITASK_END  0xff0cb994
#define ROM_CREATETASK_MAIN_START 0xff0c3064
#define ROM_CREATETASK_MAIN_END 0xff0c33dc
#define ROM_ALLOCMEM_END 0xff0c4a8c // 0xff0c30a8 0xff0c30a4 // kinda iffy
#define ROM_ALLOCMEM_INIT 0xff0c30c4
#define ROM_B_CREATETASK_MAIN 0xff0c5814

Had this camera been able to use the "classic" boot process (a.k.a. malloc?) we would probably be running code on the camera by now, right?
Title: Re: ML on EOS-M2
Post by: a1ex on July 22, 2017, 05:51:57 PM
Quote
So other cameras don't need this or is the 500D supported better in QEMU than any other camera?

500D was the first to get menu navigation in QEMU, and right now there are still some things that only work there (such as changing display color, or the memcheck tool). Also, in some places such as photo capture, Canon code is simpler (easier to understand).

Quote
ff0c30a4:    ldr r1, [pc, #344] ; ff0c3204: (00c2a000)   <-- end address??
ff0c30a8:    ldr r0, [pc, #344] ; ff0c3208: (0046c000)   <-- start address??
...
ff0c30c4:    bl loc_ff146d3c <-- allocatememory_init_pool?

Almost there - start and end address are correct, but allocatememory_init_pool is not.

During a function call, R0-R3 are not preserved; in particular, R0 is also used for return value. Therefore, whatever was loaded in R0 and R1 are parameters for the function call that follows immediately, but not for the subsequent functions.

Therefore, allocatememory_init_pool is the one called right after loading the parameters into registers:
Code: [Select]
FF0C30AC                 BL      0x7ABC <-- this one

Note that both 0x81F0 (AllocateMemory) and 0x7ABC (allocatememory_init_pool) reference the same memory address (0x90CC8). The latter (0x7ABC) writes something to this address (the result of 0x79FC), and this call contains a solid hint about the meaning of R0 and R1: start and end. Why? Because 0x79FC receives R0 = start and R1 = end-start, so the latter must be size:
Code: [Select]
00007ABC
...                      ; nothing else touching R0=start and R1=end
00007AD0                 SUB     R1, R1, R0
00007AD4                 BL      sub_79FC

In other words, the AllocateMemory pool is from 0x46c000 to 0xc2a000. Note that, unlike my previous comments (je me suis trompé), this is not the system memory area:

Code: [Select]
(gdb) set $pc=0xFF0C77EC
System Memory Information
  Start Address       = 0x0018ac08
  End Address         = 0x002aac00
  Total Size          = 0x0011fff8 (  1179640)
  Allocated Size      = 0x00062328 (   402216)
  Allocated Peak      = 0x00064b30 (   412464)
  Allocated Count     = 0x00000051 (       81)
  Free Size           = 0x000bdcd0 (   777424)
  Free Block Max Size = 0x000bb240 (   766528)
  Free Block Count    = 0x00000003 (        3)

I have a feeling this might be where DryOS allocates stack space for its tasks. Let's check:
Code: [Select]
arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
...
Current stack: [18af38-18acb8]
...
Current stack: [1ed9b0-1ec9b0]

Confirmed. Finding the allocator for the system memory would be good for cameras with very little RAM (such as 1100D).

Back to our case - AllocateMemory uses a different block of memory (not sys_mem), so I'd better update the wrong comments in the source code.

Let's try to find out how much free space we have (GetMemoryInformation). From mem.c, we know this about it:
Code: [Select]
extern int GetMemoryInformation(int* total, int* free);

So, it takes 2 parameters, pointers to 32-bit integers. It will write the total amount into the first integer, and the amount of free space into the second integer.

Let's call it from gdb. Since we don't care about code execution after this function returns, we can just pick two arbitrary addresses to store the two integers (never do this on real hardware!)
Code: [Select]
(gdb) set $r0 = 4
(gdb) set $r1 = 8
(gdb) set $pc = 0xFFD25774
(gdb) b *$lr
Breakpoint 2 at 0xff352220
(gdb) c
Continuing.

Breakpoint 2, 0xff352220 in ?? ()
(gdb) x/2 4
0x4: 0x7be000 0x3148d0

Cross-checking: 0xc2a000 - 0x46c000 = 0x7be000.

So, that's what we have to patch in order to load ML.

TODO: I'd like to update 100D to use the "classic" boot process as well.
Title: Re: ML on EOS-M2
Post by: nikfreak on July 22, 2017, 07:23:33 PM

TODO: I'd like to update 100D to use the "classic" boot process as well.

as discussed here (https://bitbucket.org/hudson/magic-lantern/pull-requests/672/dryos-task-hooks-for-newer-cameras-6d-70d/diff) I failed to use AllocateMemory pool but:

Quote from: nikfreak
...I am able to load 100d into malloc but the interesting part I can only bring up ML menu with the task related changes from this PR in addition with a fixed task_dispatch_hook and undefining hijack_task_addr as instructed earlier. Otherwise ML menu wouldn't come up.

I'm ready If you want to give it another go.
Title: Re: ML on EOS-M2
Post by: dfort on July 23, 2017, 02:12:25 AM
Almost there - start and end address are correct, but allocatememory_init_pool is not.
...
...allocatememory_init_pool is the one called right after loading the parameters into registers:
Code: [Select]
FF0C30AC                 BL      0x7ABC <-- this one

Yikes! That's the first one I tried but later dismissed it.

In order to follow along on my disassembly I'm doing this:

Code: [Select]
0x7ABC + RAM_OFFSET
Code: [Select]
00007ABC
...                      ; nothing else touching R0=start and R1=end
00007AD0                 SUB     R1, R1, R0
00007AD4                 BL      sub_79FC

In other words, the AllocateMemory pool is from 0x46c000 to 0xc2a000. Note that, unlike my previous comments (je me suis trompé)...

Je suis clueless -- Hey, Google did the translation.

Let's check:
Code: [Select]
arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
...
Current stack: [18af38-18acb8]
...
Current stack: [1ed9b0-1ec9b0]

Here is what I got:

Code: [Select]
arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
[8] 28220

Getting rid of all the tricky stuff there really isn't much to search.

Code: [Select]
./run_canon_fw.sh EOSM2,firmware=boot=0 -s -S -d debugmsg,v
DebugMsg=0x4398 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] enabling code execution logging.
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[EOS] loading './EOSM2/SFDATA.BIN' as serial flash, size=0x1000000
[MPU] warning: non-empty spell #31 (08 06 03 1f) has duplicate(s): #40
[MPU] warning: non-empty spell #41 (0a 08 01 3b) has duplicate(s): #37 #38
[MPU] warning: non-empty spell #45 (26 24 01 4e) has duplicate(s): #46

[MPU] Available keys:
- Arrow keys   : Navigation
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- DELETE       : guess
- M            : MENU
- P            : PLAY
- I            : INFO/DISP
- Q            : guess
- L            : LiveView
- Shift        : Half-shutter
- B            : Open battery cover
- F1           : show this help

Setting BOOTDISK flag to 0

Maybe a problem with this Mac QEMU? Seemed to be working fine up until now. Ok--moving on.

Finding the allocator for the system memory would be good for cameras with very little RAM (such as 1100D).

I was thinking of trying an alternate booting method on the original EOSM to see if that takes care of the shutter-bug. That is if I can ever get through this.

Code: [Select]
(gdb) set $r0 = 4
(gdb) set $r1 = 8
(gdb) set $pc = 0xFFD25774
(gdb) b *$lr
Breakpoint 2 at 0xff352220
(gdb) c
Continuing.

Breakpoint 2, 0xff352220 in ?? ()
(gdb) x/2 4
0x4: 0x7be000 0x3148d0

Ok--this works but how did you know to give r0 = 4 and r1 = 8? Then there's that tricky x/2 4. I tried other values and of course came up with different results. In any case, the important part is that:

Code: [Select]
end address - start address = free space
Great psudo code heh? Now is that really free space? Free as in beer? Or--is that total space? The latest autoexec.bin file I made was:

Code: [Select]
File size : 0x686A0

So--

Code: [Select]
0x7be000 - 0x686A0 = 0x755960
Plenty of space but is it shared with the Canon firmware? Yes--but there is still enough room for everybody?

So, that's what we have to patch in order to load ML.

Great, let's do this. First I saw what was done for the 550D and 1100D (the only other cameras that uses AllocateMemory pool) and added a special case for the EOSM2:

boot-hack.c
Code: [Select]
    #elif defined(CONFIG_EOSM2)
    // change end limit to 0xc2a000 => reserve 512K for ML
    *addr_AllocMem_end = MOV_R1_0xC2A000_INSTR;
    ml_reserved_mem = 512 * 1024;   

Next I added MOV_R1_0xC2A000_INSTR for the start address of the EOSM2:

arm-mcr.h
Code: [Select]
#define MOV_R0_0_INSTR 0xe3a00000
#define MOV_R1_0xC2A000_INSTR 0xe3a0???? // mov r1, 0xc2a000

I'm kind of stuck figuring out what to plug in and if there is anything else that needs to be changed. For example RESTARTSTART is set to 0x100E00 so I'm getting this:

Code: [Select]
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x1015f8 (1054200)
[BOOT] autoexec.bin loaded at 100E00 - 181840.

I'm also getting these message that makes me think this isn't working:

Code: [Select]
ff0c8314: !!!! can not fixup jump from 0016dc9c to fd77ab1c (offset -00a7cc62)
...
ff0c8948: !!!! can not fixup jump from 0016e2d0 to fd77b150 (offset -00a7cc62)

Finally, seeing how the 500D work, it ends up like this:

Code: [Select]
Fixups=cd4730 entry=cd4740 free_space=10
[BOOT] changing sys_mem_end:
0x00cd4760:  e3a0160d      mov r1, #13631488 ; 0xd00000
0x00cd4760:  e3a018c6      mov r1, #12976128 ; 0xc60000
K270 READY
[      init:ff1d3b58 ] create_semaphore('EventProcedure', 1)
Temporary breakpoint 15 at 0xff1d3b5c

While the EOSM2 winds up like this:

Code: [Select]
Fixups=171390 entry=17135c free_space=ffffffcc
Title: Re: ML on EOS-M2
Post by: a1ex on July 23, 2017, 08:36:46 AM
In order to follow along on my disassembly I'm doing this:

Code: [Select]
0x7ABC + RAM_OFFSET

That's exactly the reason I've listed the RAM stubs in the ROM_ADDR - RAM_OFFSET format in stubs.S - so one can just copy the ROM address and look it up in a ROM disassembly, without requiring the RAM segments to be set up. I should probably include the RAM address in comments as well - a job for the stubs formatting scripts.

Quote
Ok--this works but how did you know to give r0 = 4 and r1 = 8?

Just picked two RAM addresses that I thought they won't be used in this routine. Since in some cases, null pointers are used for optional output arguments, I've chosen non-zero pointers. Addresses 4 and 8 are actually exception vectors (http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0211h/Babfeega.html) - Undefined instruction and Software interrupt. Our routine is not going to trigger these, and they are easy to type.

x/2 4 means 2 items from address 4, and by default, an item is a 32-bit integer. Therefore, it's printing the memory contents at 4 and 8 (our pointers).

Alternatives:
- allocate on the stack (below $sp) - probably the safest option if you debug a real target
- use some high address that's unlikely to be used (anything above 0x10000000 on a 256M camera will do, since QEMU uses 512M for all models, to keep it simple)
- find some other address unlikely to be used (e.g. in the middle of some RAM string, or some other statically allocated variable)
- pick something random and repeat until it succeeds (emulation only!)

Quote
Code: [Select]
#define MOV_R1_0xC2A000_INSTR 0xe3a0???? // mov r1, 0xc2a000

You don't need to hardcode a MOV here - it uses PC-relative addressing. Look at ff0c3204.

Will take a closer look later, also for 100D, since now I have the tools to do so :D
Title: Re: ML on EOS-M2
Post by: dfort on July 23, 2017, 01:36:10 PM
You don't need to hardcode a MOV here - it uses PC-relative addressing. Look at ff0c3204.

That's why my patching isn't working? It always comes up like this:

Code: [Select]
File size : 0x67D40
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x1015e0 (1054176)
[BOOT] autoexec.bin loaded at 100E00 - 181840.
Fixing from ff0c57a0 to ff0cb994

Those [BOOT] messages are handy but only available in the QEMU branch. If merging QEMU into unified (https://bitbucket.org/hudson/magic-lantern/pull-requests/818/qemu-full-firmware-emulation/diff) doesn't break anything it would be lovely. :D

In any case, it seems to be doing the right thing.

This is still a problem--noticed I didn't give enough information in my previous post:

Code: [Select]
ff0c8314: 3b9aca00 BL ff9aca00 => fd77ab1c
ff0c8314: !!!! can not fixup jump from 0016dc9c to fd77ab1c (offset -00a7cc62)
...
ff0c8948: 3b9aca00 BL ff9aca00 => fd77b150
ff0c8948: !!!! can not fixup jump from 0016e2d0 to fd77b150 (offset -00a7cc62)

Another problem is that we're not getting to this (using the 550D example):

Code: [Select]
[BOOT] changing sys_mem_end: