Magic Lantern Forum

Magic Lantern Releases => Camera-specific discussion => Topic started by: Palpatine on September 22, 2015, 02:48:23 PM

Title: ML on EOS-M2
Post by: Palpatine on September 22, 2015, 02:48:23 PM
It's my first post so... hello guys! :)

Have anyone tried to install ML on EOS-M2?

I'm looking for a low weight camera because I want to use it with a hexacopter. EOS-M2 is surprisingly cheap on ebay and I think it has a proper quality for tasks that I'm going to do. It would be also nice to run ML on it. I know it was released only for Asian market so it's not popular, but maybe someone can assure me that it's possible to run ML without problems.

I'll be thankful for all opinions and advices :)
Title: Re: ML on EOS-M2
Post by: Walter Schulz on September 22, 2015, 05:31:23 PM
Each ML port is pretty much hardcoded for a cam and a specific firmware version. Porting ML takes a skilled programmer fluently speaking ARM processor (="maintainer") and several hundred hours of work. And at time of writing there is a lack of maintainers resulting in orphaned/blindly maintained ports.
A spare developer would come handy ...
Title: Re: ML on EOS-M2
Post by: neoplanta on September 24, 2015, 12:54:40 AM
I was asking the same on the other topic. It would be nice if someone is considering to do this to tell us :) I did not even look when I was buying eos m 2 about this, I asumed that it was the same ML for both 1 and 2.
Title: Re: ML on EOS-M2
Post by: dfort on September 24, 2015, 06:02:57 AM
I just checked up on prices and yes the M2 is priced very low but the M1 is even lower and there is a Magic Lantern port for it. According to a former M2 owner, now an M3 owner:

Dumb question, what's necessary to customize the video bitrate on the firmware? Not a programmer, but I can script.

The EOS M2 had 50mb which produced beautiful 1080P video and no audio compression which resulted in great fidelity; vs the low bit H264 and AAC that Canon throws on the EOS M3 to neuter it so it doesn't compete with pro offerings since the autofocus and software IS is good on the M3/new APS-C's. I almost want to buy another M2 (sold it once I bought the M3) since it did such a good job with video.

So if you have an M2 you might consider holding on to it even without a ML port.

The Canon EOS M3 (http://www.magiclantern.fm/forum/index.php?topic=14990.0) discussion is hinting on some serious interest in porting ML but there is no indication whether or not it has gotten off the ground yet. Though it does look like the CHDK (http://chdk.wikia.com/wiki/CHDK) project has something working with the M3 (http://chdk.wikia.com/wiki/M3). Sorry, nothing that I could find for the M2.
Title: Re: ML on EOS-M2
Post by: godashram on October 14, 2015, 01:44:25 AM
doesn't help that not a single update has been released for the M2...

My M2 would like some love as well...but coding is not my specialty :( much less figuring out if it'll run ML like the Eos M or CHDK like the M3
Title: Re: ML on EOS-M2
Post by: a1ex on October 14, 2015, 12:40:42 PM
On ExifTool website there is a hint that EOS M2 is still based on DSLR firmware (unlike M3, which is based on PowerShot firmware).

http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html

So, if anyone is tempted to port ML on it, I can prepare a dumper.
Title: Re: ML on EOS-M2
Post by: neoplanta on October 14, 2015, 05:16:14 PM
It would be great if somone would do this. I know this is a dumb question, but I am pretty much idiot when it comes to coding, is it posible to install eos m 1 firmware to eos m 2, or just make focus peaking option somehow on eos m 2?
Title: Re: ML on EOS-M2
Post by: godashram on October 14, 2015, 05:57:41 PM
A1ex, you prep a dumper, i'll dump mine as soon as it's up (well, after driving home to grab the M2)

granted, the most I could do after that is try out builds, my code-fu is non-existent :(
Title: Re: ML on EOS-M2
Post by: a1ex on October 14, 2015, 06:31:15 PM
Alright, here you go:

- the portable display test: hello-M2.fir (http://a1ex.magiclantern.fm/debug/portable-hello-world/EOSM2/hello-M2.fir)

- the ROM dumper: dumperM2.fir (http://a1ex.magiclantern.fm/bleeding-edge/blind-dumper/dumperM2.fir)

It dumps 32 MB from 0xF7000000, which covers both ROM0 and ROM1 - that's what you need to run ML in QEMU. For analyzing the code, you can load ROM1 (the second half of that file) at 0xFF000000.

Usage: run firmware update, then go to play mode (make sure you have an image there), then look on your card. Keep the dump for yourself, do not publish it online.

edit: dumper confirmed working by @godashram :)
Title: Re: ML on EOS-M2
Post by: Licaon_Kter on October 15, 2015, 10:44:10 AM
Interesting.
Title: Re: ML on EOS-M2
Post by: vyskocil on October 15, 2015, 01:26:51 PM
Great, new gears ! Is it possible to have the same for the 7D mkII to start looking at a port ? Perhaps the dumper is not needed as we already have a fir upgrade available but then could you point out the steps to lookup the keys and decrypt the firmware ?
Thanks in advance!
Title: Re: ML on EOS-M2
Post by: godashram on October 15, 2015, 07:19:50 PM
wouldn't it make more sense to start a new thread for the 7D MKII instead of posting the question here??
Title: Re: ML on EOS-M2
Post by: glassescreditsroll on October 19, 2015, 09:15:54 AM
does this mean we could start  porting ml to eos m2?
Title: Re: ML on EOS-M2
Post by: godashram on October 20, 2015, 12:57:29 AM
I hope it does.... I've heard nothing since giving A1ex the M2 dump. Guess the biggest issue would be finding someone to actively work on an M2 build?
Title: Re: ML on EOS-M2
Post by: a1ex on November 01, 2015, 07:08:36 PM
does this mean we could start porting ml to eos m2?

Correct. The M2 looks very similar to the original M, so I don't expect any real difficulties here.

Guess the biggest issue would be finding someone to actively work on an M2 build?

Correct.
Title: Re: ML on EOS-M2
Post by: glassescreditsroll on November 06, 2015, 04:50:30 AM
So what needs to be done now? I have an eos m2 and would love to be able to use ML on it I'll do anything I can to help
Title: Re: ML on EOS-M2
Post by: Audionut on November 06, 2015, 06:08:50 AM
Guess the biggest issue would be finding someone to actively work on an M2 build?

Correct.

Current developers are happy to help.  But there needs to be someone else to take on the bulk of the development work.
Title: Re: ML on EOS-M2
Post by: godashram on November 10, 2015, 05:42:11 PM
Current developers are happy to help.  But there needs to be someone else to take on the bulk of the development work.

and that is where the issue is.... I don't have the patience for development, and well, not many M2's seem to be in the hands of people that would care (my opinion).

to be honest, I'd be happy with just the continuous recording feature :)
Title: Re: ML on EOS-M2
Post by: neoplanta on November 10, 2015, 06:39:44 PM
And I would be happy only with focus peaking option  :D It is too bad that m2 is not wide spread as other cameras, and it is waste that there are people who are willing to help, but unfortunetly there are not many people who know coding to take the bulk of the development work.
Title: Re: ML on EOS-M2
Post by: godashram on November 10, 2015, 10:20:08 PM
well, it's lack of availability outside of Japan and China and the eos m fire sales didn't help the m2 in any way at all.
Title: Re: ML on EOS-M2
Post by: GenMeiHikaru on December 16, 2015, 04:26:22 PM
Just bought a EOS M2 kit (+ a EF-M 22mm and a 18-55mm) with a kinda low price in Japan, just about $285. Seems like the only thing missing is ML firmware, coding is also a no-go for me but I'll be glad to be of any help. Thanks...!
Title: Re: ML on EOS-M2
Post by: neoplanta on December 16, 2015, 06:57:43 PM
Wow, thats prety cheap. I bought mine for about 280 dolars, without lens.
Title: Re: ML on EOS-M2
Post by: godashram on December 16, 2015, 07:32:08 PM
WOW, for $285 with 2 lenses is NUTS! it's a great camera... but hunting down someone to work on it isn't easy :(
Title: Re: ML on EOS-M2
Post by: GenMeiHikaru on December 17, 2015, 03:25:31 PM
Surprisingly it's not just the lens, the EF-EOS M -> EF-S adapter and a 90EX flash are also included in the package.
If u guys still want the link, it's here
http://www.amazon.co.jp/gp/product/B00H2ABRKS/ref=gbps_img_s-3_0229_0fa1bf1e?smid=AN1VRQENFRJN5&pf_rd_p=264150229&pf_rd_s=slot-3&pf_rd_t=701&pf_rd_i=gb_main&pf_rd_m=AN1VRQENFRJN5&pf_rd_r=1F660AF0S908GYXQ8A9F
It's Amazon for Japan so i'm not sure if it's available for shipment outside of Japan though... I bought it with a Lightning Deal (¥34800~$285), but without the deal it's ¥40498(~$330), still very affordable (I think)...
Edit: on sale again...
Title: Re: ML on EOS-M2
Post by: godashram on December 18, 2015, 02:22:18 AM
fulfilled by amazon.... so maybe...
Title: Re: ML on EOS-M2
Post by: jonkjon on December 18, 2015, 04:49:45 PM
Is this something that needs to be built from scratch or is there something to build from? I have some c experience (mostly windoze) and i'd be willing to take a look. I have perused the getting started stuff but it pretty much just shows how to setup a compiler etc. I haven't looked at the source code at all.

--Jon
Title: Re: ML on EOS-M2
Post by: godashram on December 18, 2015, 05:55:28 PM
the m2 was dumped (a1ex has the dump I made)

from there, i guess it would be a port of the existing build, beyond that, no clue.
Title: Re: ML on EOS-M2
Post by: Walter Schulz on December 18, 2015, 06:37:09 PM
What you are looking for is a maintainer. Someone knowing C, assembler programming embedded devices (preferable ARM architecture) and willing and able to spend several hundred hours of his/her time in porting ML to M2. And don't expect any of the devs asking for that additional workload. They will support this maintainer-to-be, of course.
Title: Re: ML on EOS-M2
Post by: bender on March 06, 2016, 10:16:17 AM
any updates for the M2?  :)
Title: Re: ML on EOS-M2
Post by: brocolimayo on March 15, 2016, 08:10:19 PM
same question :) my programming skills are almost zero, but I'd love to help, I own a m2 too, and would be wonderful to have ml on it :)
Title: Re: ML on EOS-M2
Post by: godashram on March 16, 2016, 06:54:44 AM
nope, as far as I know, there isn't anyone willing to work on it.
Title: Re: ML on EOS-M2
Post by: Skylin3 on March 16, 2017, 11:31:14 PM
Bump. Is there any progress with m2? I can help with testing ML, I have m2.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 05:50:16 AM
Got one.

(https://c1.staticflickr.com/5/4227/34014053584_75ea69bf64_z.jpg)

Not sure if I'm ready for this but I was going to quietly work on Magic Lantern for the EOSM2. Seems like this one should be one of the "easier" ports.

Then I thought, that's no way to work on an open source project, let's get this out there in the open. My first stumbling block is setting the boot flag on the camera. How to do this without bricking the camera? Tried searching for the answer and it looks like there is already a working .FIR file for this camera -- so how can I get my hands on it?
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 09:43:20 AM
Seems like this one should be one of the "easier" ports.

Definitely, as it's plain old DIGIC 4 (unlike 1300D, which looks like a mix between D4, 5 and 6).

The boot flag is "traditionally" enabled after getting the Hello World working. There are two flavors: the minimal target (which requires a tiny set of stubs and only compiles a few small files) and the regular one (CONFIG_HELLO_WORLD in the full source code, which also requires file I/O for loading fonts, and a bunch of other initializations). Once the second version is working, CONFIG_DUMPER_BOOTFLAG will be straightforward to compile and run. The "user-friendly" installer comes a bit later, as it requires some more stubs.

The boot flag can be also enabled earlier, from the bootloader context; we did that on the DIGIC 2 and 3 models, blindly (without having the display available for debugging or user feedback). Now that we have display access from bootloader, that's a valid option as well (though I wasn't comfortable doing this step on 7D Mark II).

Until enabling the boot flag, you'll need my assistance to run a binary on the camera. You can run them in QEMU though, by setting "boot=1" though.

Will explain how to get the Hello World on models that don't boot to Canon GUI on the 1300D thread, as it was covered on IRC a few days ago.
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 12:17:11 PM
I'll start with a walkthrough using latest QEMU and IDA. First step is to assume it's very similar to EOSM, so after installing QEMU (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-tests/lastSuccessfulBuild/console), head over to qemu/qemu-2.5.0/hw/eos/model_list.c and copy the basic entries from the section for EOSM:

Code: [Select]
    {
        .name                   = "EOSM2",
        .digic_version          = 5,
     }

The others are likely model-specific, so let's not assume too much from the beginning. Let's try:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0"
...
FFFF3F50: MCR p15,0,Rd,cr6,cr0,0:  946_PRBS0 <- 0x3F       (00000000 - FFFFFFFF, 0x100000000)
FFFF3F58: MCR p15,0,Rd,cr6,cr1,0:  946_PRBS1 <- 0x3D       (00000000 - 7FFFFFFF, 0x80000000)
FFFF3F60: MCR p15,0,Rd,cr6,cr2,0:  946_PRBS2 <- 0xE0000039 (E0000000 - FFFFFFFF, 0x20000000)
FFFF3F68: MCR p15,0,Rd,cr6,cr3,0:  946_PRBS3 <- 0xC0000039 (C0000000 - DFFFFFFF, 0x20000000)
FFFF3F70: MCR p15,0,Rd,cr6,cr4,0:  946_PRBS4 <- 0xFF00002F (FF000000 - FFFFFFFF, 0x1000000)
FFFF3F78: MCR p15,0,Rd,cr6,cr5,0:  946_PRBS5 <- 0x37       (00000000 - 0FFFFFFF, 0x10000000)
FFFF3F80: MCR p15,0,Rd,cr6,cr6,0:  946_PRBS6 <- 0xF700002F (F7000000 - F7FFFFFF, 0x1000000)
...
K355 READY
128K Sector FROM From BL 0xffff
[SF] InstallSerialFlash 6 0xc022c0d4 0x0 0x1000000 1
...
K355 ICU Firmware Version 1.0.2 ( 6.0.5 )

OK, so we've got a rough idea about its memory map (same as other DIGIC 4 models). We also noticed it has a serial flash of size 0x1000000. This is similar to 100D (lookup serial_flash_size in model_list.c), so it may be a good idea to reuse the serial flash data from this model:
Code: [Select]
        .serial_flash_size      = 0x1000000,

Unfortunately, this step doesn't change anything in the emulation. Let's look at the next messages:
Code: [Select]
...
    10:    46.336 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x10000
...
    14:    56.576 [STARTUP] InitializeIntercom
...
    42:   722.688 ERROR [DL] ########## DL ERROR!!!!! ###########

- Many errors regarding missing properties. These are either stored in ROM, or in the serial flash, or they come from the MPU. The ones from ROM should not pose any problems, since we have the complete ROM. For the others, we did not see any sign of activity, so we have to dig in to find out whether anything is different (usually these communication channels may use different registers, depending on model, but overall the communication protocol is the same).

- DL ERROR: if you look in EOSM/debugmsg.gdb, you'll find a way to get past this message (by skipping the initialization of "DL", whatever that is).

- The debug messages are not very informative. To get more insights, find out a few essential functions (such as task_create and DebugMsg) and place them in EOSM2/debugmsg.gdb (look at other models for how the declarations should look like).

OK, so at this point we have to disassemble the firmware. If you use IDA, the initial steps would be:
- load ROM1.BIN
- select ARM processor, architecture ARMv5TEJ (we have ARMv5TE on all models from DIGIC 2 to DIGIC 5)
- uncheck 'Delete instructions with no xrefs' (we have plenty of those)
- uncheck 'Perform no-return analysis' (IDA usually gets it wrong)
- RAM section: 0, size 0x100000 (we can always add more later)
- ROM start address: 0xFF000000
- ROM size: 0xFFFFFC (off-by-one bug in IDA; I have an older version, so YMMV)
- File load address / size: same as ROM.

Now that our binary is loaded, go to 0xFFFF0000 and press "c" to mark this as code. This is the bootloader start address (aka HIVECS in ARM documentation). Next step is to find where the main firmware starts (where the code path leaves the bootloader). It's easy to assume similarity with other models, but let's try to find it from scratch (just to show off the latest QEMU tool):

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d calls
...
call 0xFFFF0FCC(0, ffffdfff, 2000, ffff0b20)                                     at [ffff09c8:0]
 call 0x100000(0, 11836c, 11836c, 100000)                                        at [ffff1010:ffff09cc]
...

OK, so the bootloader runs from RAM.

Where does the bootloader end? When it stops executing from its address range (near 0xFFFF0000 or 0x100000):

Code: [Select]
  return 0 to 0x10013C                                                           at [1022f4:ffff1014]
 return 1 to 0xFFFF1014                                                          at [1000bc:ffff09cc]
return 0 to 0xFFFF09CC                                                           at [ffff101c:0]
Warning: R10 not restored (0xa -> 0x1)                                           at [ffff101c:ffff09cc]
PC jump? 0xF80C0000 lr=ffff09cc                                                  at [ffff0a04:ffff09cc]
0xffff0a04:  e1a0f000      mov pc, r0
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
0xf80c0000:  e59ff0c4      ldr pc, [pc, #196] ; 0xfffffffff80c00cc

Ta-da! Same as other DIGIC 5 models: ROM1 startup code is at 0xF80C0000, but code from main firmware expects to run from a mirror copy (details) (https://www.magiclantern.fm/forum/index.php?topic=6785.msg58899#msg58899). So, our ROMBASEADDR is 0xFF0C0000.

Let's go there in IDA and mark this section as code. At this point, IDA already recognized a few functions, but let's get some more from the QEMU execution trace:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d idc
...
EOSM2.idc saved.

Now load this IDC script into IDA, or convert it for your favorite disassembler, and start looking at the functions called during the execution in QEMU:
Code: [Select]
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
...
call 0xFF0C1BD4(1000, 698, eeeeeeee, 1000)                                       at [ff0c0dbc:0]
 call 0x866B4(f88, 74, eeeeeeee, 1000)                                           at [ff0c1be4:ff0c0dc0]
 return ffc to 0xFF0C1BE8                                                        at [866f4:ff0c0dc0]
 call 0x3168(f88, ff0c57a4, 0, 0)                                                at [ff0c1c80:ff0c0dc0]
...

0xFF0C1BD4 is cstart, 0x866B4 is bzero32, 0x3168 must be create_init_task and 0xFF0C57A4 must be init_task. Some of these functions are called from RAM, so we'll have to identify where its contents come from (what is copied where). TODO.

Next step, for the emulation, would be to find the MPU registers (near InitializeIntercom), mpu_send/mpu_recv stubs may also help, and we also need to check the serial flash communication.

For executing user code on the camera, next step would be to reserve memory for our application. This is usually done by shrinking the "malloc" buffer. Take a look at boot-hack.c from the "qemu" branch (as it's a bit more verbose and better explained). These debug messages can be seen on the QEMU-boot-check (http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-boot-check/) page.

To be continued.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 06:19:03 PM
Wow.

Let's take half a step back before embarking on this journey. When I posted my intent to attempt this port I skipped one important step before asking for a firmware dumper--check to see if there is already one available. The list of available ROM dumpers are listed at the bottom of the Magic Lantern Nightly Builds (https://builds.magiclantern.fm/) page. The one for the EOSM2 point to a post on this topic (http://www.magiclantern.fm/forum/index.php?topic=15895.msg155455#msg155455) with the links to download the dumpers.

It appears that the EOSM2 ROM was first dumped about 1.5 years ago. I'm assuming that it was the original firmware version. The camera I got has 1.0.2 installed and 1.0.3 is available. My question is, should I try to make a dump of 1.0.2 before updating the firmware or does it matter? Maybe the dumpers won't work with 1.0.3? Maybe we should archive a 1.0.2 dump before starting? The 1.0.3 firmware update is a minor fix that Canon released for most of their cameras to deal with this issue:
Quote
Corrects a phenomenon in which when using the camera with the EF-S 18-135mm f/3.5-5.6 IS USM or EF 70-300mm f/4-5.6 IS II USM lens, even if lens aberration correction is set to "Enable", correction will not be applied.

[EDIT]

hello-M2.fir (http://a1ex.magiclantern.fm/debug/portable-hello-world/EOSM2/hello-M2.fir) running on EOSM.102
(https://c1.staticflickr.com/5/4200/34481675560_77a17b04d7_z.jpg)

However, dumperM2.fir (http://a1ex.magiclantern.fm/bleeding-edge/blind-dumper/dumperM2.fir) doesn't dump the ROM. I tried various cards including an old 2GB card that has worked on all the other ROM dumps I made.
Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 09:23:14 PM
The dumper linked above runs on the main firmware (in contrast with the portable ROM dumper, which runs from bootloader), so it has no issues with card sizes, but it's limited to DIGIC 4 and 5 models. Does it help if you follow the instructions in the linked post? (it's different from the portable dumper you are used to)

You can start from the latest firmware.
Title: Re: ML on EOS-M2
Post by: dfort on May 24, 2017, 10:40:55 PM
Does it help if you follow the instructions in the linked post?

Doh!

The single file dump is named "(NULL)" on the sd card and it is 33.6 MB. I also updated the firmware to 1.0.3 though I also dumped 1.0.2 just to have it.

Did the change to model_list.c and tried running qemu but it stopped on:
Code: [Select]
eos_load_image: file not found './EOSM2/ROM0.BIN'
./run_canon_fw.sh: line 35: 16058 Abort trap: 6           $QEMU_PATH/arm-softmmu/qemu-system-arm -drive if=sd,format=raw,file=sd.img -drive if=ide,format=raw,file=cf.img -M $*

So I renamed (NULL) to ROM0.BIN then it stopped on "file not found './EOSM2/ROM1.BIN'" but since the dumper only creates the one file I'm afraid I hit another wall.

It dumps 32 MB from 0xF7000000, which covers both ROM0 and ROM1 - that's what you need to run ML in QEMU. For analyzing the code, you can load ROM1 (the second half of that file) at 0xFF000000.

I'm searching how to run ML in QEMU with a single file dump but so far I can't find an answer.

Title: Re: ML on EOS-M2
Post by: a1ex on May 24, 2017, 11:15:20 PM
Split it in two halves: first ROM0, second ROM1.

Back then, it was much easier to patch Canon code to dump everything in one file, so it covers both ROMs. For this reason,
older QEMU versions used to require the ROMs to be joined like this (but it was changed to allow using the backup ROM copies already saved by ML on the cards under ML/LOGS, and to allow more unusual ROM configurations).
Title: Re: ML on EOS-M2
Post by: dfort on May 25, 2017, 01:23:00 AM
Making some progress. Mac uses the BSD version of split which doesn't have the -n option but I was able to split the dump in two like this:
Code: [Select]
split -b 16800000 "(NULL)" ROM
That gave me two files, ROMaa and ROMab which I renamed ROM0.BIN and ROM1.BIN and placed in ~/qemu/EOSM2 then ran:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0"
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
CHK version_gen.h
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFA6FF
[MPU] FIXME: no MPU spells for EOSM2.
[MPU] FIXME: no MPU button codes for EOSM2.
Setting BOOTDISK flag to 0

It looks like this:

(https://c1.staticflickr.com/5/4221/34833502446_a6c5cf5e35_z.jpg)

QEMU didn't crash but it also isn't displaying the output from the minimal qemu autoexec.bin.
Title: Re: ML on EOS-M2
Post by: dfort on May 25, 2017, 05:02:25 AM
Took some more baby steps. First a fall.
OK, so we've got a rough idea about its memory map (same as other DIGIC 4 models). We also noticed it has a serial flash of size 0x1000000. This is similar to 100D (lookup serial_flash_size in model_list.c), so it may be a good idea to reuse the serial flash data from this model:
Code: [Select]
        .serial_flash_size      = 0x1000000,

All that did was to consistently give me this error when running QEMU:

Code: [Select]
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFA6FF
Could not open ./EOSM2/SFDATA.BIN

I don't have IDA but I do have ARM-console (http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console) and the good old reliable disassemble.pl from the finding stubs tutorial (http://www.magiclantern.fm/forum/index.php?topic=12177.msg117735#msg117735). So I took another baby step and tried to disassembly the ROM1.BIN that was split off the single file dumperM2.fir dump. I ran the usual "perl disassemble.pl 0xFF000000 ROM1.BIN" but something looked wrong.

Code: [Select]
ff0ba700: e59ff0c4 ldr pc, [pc, #196] ; ff0ba7cc: (ff0c000c)
"gaonisoy":

All indications are that it should be more like the EOSM and 700D and start on 0xFF0C0000 so I did a second disassemble, "perl disassemble.pl 0xFF005900 ROM1.BIN"

Code: [Select]
ff0c0000: e59ff0c4 ldr pc, [pc, #196] ; ff0c00cc: (ff0c000c)
"gaonisoy":

Now it "looks" right. Well, "looks" is mostly what I'm going on right now. I don't really know what I'm doing. That weird offset might be a result of splitting the dump in two? The EOSM2 "looks" like a combination of the EOSM, 100D with wifi features added on. This means that I should be able to find some stubs using the pattern matching method I used for the minor firmware updates. Of course porting a new camera is certainly not a minor undertaking!

I want to continue getting QEMU working properly on the Mac and figure things out the "right" way but now that I've got a disassembly at least I'm on familiar territory so do you think I'm at the stage where I can start hunting down some stubs?
Title: Re: ML on EOS-M2
Post by: a1ex on May 25, 2017, 08:40:40 AM
You need two *equal* halves, without any approximation.

(edit: updated QEMU to give better warnings in this case)

SFDUMP.BIN can probably be used from 700D or EOSM as well, but these cameras have a smaller size. Padding it with zeros may or may not work. How to get it? It's covered in QEMU's install script.
Title: Re: ML on EOS-M2
Post by: dfort on May 26, 2017, 08:36:33 AM
You need two *equal* halves, without any approximation.

(edit: updated QEMU to give better warnings in this case)

Yeah, that seems to be a problem. "split -b 16800000 "(NULL)" ROM" produced two files that aren't exactly equal. I'll keep up with the QEMU branch commits on this project.

Code: [Select]
-rw-r--r--   1 rosiefort  staff  16800000 May 24 15:15 ROM0.BIN
-rw-r--r--   1 rosiefort  staff  16754432 May 24 15:15 ROM1.BIN

[EDIT] Looks like the lesson learned here is not to trust the finder because it rounds KB to MB.

(https://c1.staticflickr.com/5/4248/34063971594_70bf4993c9.jpg)

I might be able to get SFDUMP.BIN from a 100D.
Title: Re: ML on EOS-M2
Post by: dfort on May 26, 2017, 07:09:10 PM
The equal split did the trick. Now I'm seeing the messages a1ex posted back on Reply #34 (http://www.magiclantern.fm/forum/index.php?topic=15895.msg185103#msg185103). Also made a new disassembly and it looks good so I'll start hunting for stubs.

To be continued.

Looking forward to it!
Title: Re: ML on EOS-M2
Post by: a1ex on May 26, 2017, 11:09:39 PM
Alright, so let's start from where we've left off.

0xFF0C1BD4 is cstart, 0x866B4 is bzero32, 0x3168 must be create_init_task and 0xFF0C57A4 must be init_task. Some of these functions are called from RAM, so we'll have to identify where its contents come from (what is copied where). TODO.

Since the ROMs are placed at some high address (after 0xF0000000), and we've noticed some functions called outside this range, a good guess would be that some of the functions may be copied from ROM to RAM during startup. Let's examine the memory access patterns. We are interested in some code that reads from ROM and writes to RAM. If you have no trouble following the assembly code for the startup sequence, you'll spot this piece of code right away. Otherwise, let's ask for some help from QEMU:

Running QEMU with -d help gives, among many other options:
Code: [Select]
ram        EOS: log all RAM reads and writes
rom        EOS: log all ROM reads and writes
ramr       EOS: log all RAM reads
romr       EOS: log all ROM reads
ramw       EOS: log all RAM writes
romw       EOS: log all ROM writes

We could try romr,ramw (or rom,ram if you don't mind a little more verbosity):
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d romr,ramw

The log is huge, but if you scroll around, there are some large blocks that look similar. Some of them are clearly a copying operation (read some value from ROM, write it to RAM). One of these covers the functions we are interested in:
Code: [Select]
[rom1]     at 0xFF0C000C:FFFF09CC [0xFFD1F02C] -> 0xE92D4010
[ram]      at 0xFF0C000C:FFFF09CC [0x00001900] <- 0xE92D4010: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F030] -> 0xE1A04001
[ram]      at 0xFF0C00A4:FFFF09CC [0x00001904] <- 0xE1A04001: was 0x0;
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFD1F034] -> 0xE59F1040
...
[rom1]     at 0xFF0C00A4:FFFF09CC [0xFFDD60C8] -> 0xFFD1EFFC
[ram]      at 0xFF0C00A4:FFFF09CC [0x000B899C] <- 0xFFD1EFFC: was 0x0;

In other words, a memory block from 0xFFD1F02C to 0xFFDD60C8 (actually 0xFFDD60CB, since we are looking at 32-bit operations) is copied to 0x1900 - 0xB899F. The size of the copied block is 0xb70a0 bytes. To extract this block, grab the terminal and run this under a Bash prompt:

Code: [Select]
dd if=ROM1.BIN of=1900.BIN bs=1 skip=$((0xD1F02C)) count=$((0xB70A0))

(please note the above numbers are only valid for firmware 1.0.2)

Now you can disassemble this file starting from 0x1900. In IDA, load this file as Additional binary file. If you use ARM-console, this should auto-detect the above copying operation (so you can simply start browsing the functions copied into RAM).

There are other similar blocks copied into RAM (newer DIGIC 6 and 7 models have plenty of those), and finding them manually can become tedious, so it's a good candidate for automation.

One of these blocks is the interrupt handler routine. On ARM, hardware interrupts are executed from address 0x18 (see e.g. ARM ARM - A2.6 - Exceptions (https://www.scss.tcd.ie/~waldroj/3d1/arm_arm.pdf#G6.1052026) or these slides (http://osnet.cs.nchu.edu.tw/powpoint/Embedded94_1/Chapter%207%20ARM%20Exceptions.pdf)). That means, we should expect to find some code that writes something to address 0x18 (that can be anywhere in the boot process before enabling interrupts), and on EOS firmwares, at this address you'll find a jump to 0x4B0 on many models, or something close on others. Let's examine it in GDB:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S & arm-none-eabi-gdb -x debug-logging.gdb
(note: we will need to create a GDB file for EOS M2, named EOSM2/debugmsg.gdb to match other models, but since we don't have one yet, we can just use the generic version - which we are going to include in the platform-specific file later)

Let it run for a while (continue), then hit CTRL-C and examine the exception vector (located at address 0):

Code: [Select]
(gdb) disas 0,0x40
Dump of assembler code from 0x0 to 0x40:
   0x00000000: nop ; (mov r0, r0)
   0x00000004: ldr pc, [pc, #20] ; 0x20
   0x00000008: ldr pc, [pc, #20] ; 0x24
   0x0000000c: ldr pc, [pc, #20] ; 0x28
   0x00000010: ldr pc, [pc, #20] ; 0x2c
   0x00000014: nop ; (mov r0, r0)
   0x00000018: ldr pc, [pc, #16] ; 0x30
   0x0000001c: ldr pc, [pc, #16] ; 0x34
   0x00000020: ; <UNDEFINED> instruction: 0xff0c0fec
   0x00000024: ; <UNDEFINED> instruction: 0xff0c105c
   0x00000028: ; <UNDEFINED> instruction: 0xff0c1000
   0x0000002c: ; <UNDEFINED> instruction: 0xff0c1018
   0x00000030: ; <UNDEFINED> instruction: 0x000004b0
   0x00000034: ; <UNDEFINED> instruction: 0xff0c1060
   0x00000038: ; <UNDEFINED> instruction: 0xffff0b08
   0x0000003c: ; <UNDEFINED> instruction: 0xffff0b20

That means, once a hardware interrupt happens (in other words, some device wants to tell the main CPU that it needs attention), the program counter will jump to 0x18, and there it will find a LDR PC, =0x4B0. That's where Canon's interrupt handler is located. Let's find its contents:

- Method 1: scan the memory access log for a copy operation that covers 0x4B0 and above. Easy to find.

- Method 2: disassemble it directly from GDB. How much? 0x1900 - 0x4B0 would be the upper limit (since we already know what's at 0x1900). The interrupt handler is much smaller than that - 512 bytes are more than enough:
Code: [Select]
disas 0x4B0,0x6B0

Method 3: dump the memory from gdb:
Code: [Select]
(gdb) dump memory 4B0.BIN 0x4B0 0x6B0

Method 4: dump the memory from QEMU monitor (https://qemu.weilnetz.de/doc/qemu-doc.html#pcsys_005fmonitor):
Code: [Select]
(qemu) memsave 0x4B0 0x200 4B0.BIN

For the debugmsg.gdb file, we'll need some basic stubs: DebugMsg and task_create (easy to find - one is used for most debug messages, the other can be easily identified from things that look like task names) and two addresses: CURRENT_TASK and CURRENT_ISR (interrupt service routine). We can find the latter in the interrupt handler - quote from debugmsg.gdb:
Quote
# CURRENT_ISR:
#   From interrupt handler (PC=0x18), find an expression that evaluates to
#   the current interrupt ID if any is running, or 0 if a normal task is running.
#   - on DIGIC 4/5, interrupt ID is MEM(0xC0201004) >> 2
#   - on DIGIC 6,   interrupt ID is MEM(0xD4011000)

If you can find that by examining disassembled code, great. If not, QEMU to the rescue:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,nochain -singlestep
...
Taking exception 5 [IRQ]
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah

(note: -d nochain -singlestep gives more precise locations in I/O logs - otherwise, QEMU groups the ARM instructions in blocks for faster execution, so the reported location will be approximate)

OK, so the register we are looking for is read at 0x510. Let's disassemble with GDB:
Code: [Select]
(gdb) disas 0x4B0,0x6B0
...
   0x0000050c: ldr r0, [pc, #368] ; 0x684
   0x00000510: ldr r4, [r0]
   0x00000514: str r4, [pc, #304] ; 0x64c
...
(gdb) x 0x684
0x684: 0xc0201004
(gdb) x 0x64c
0x64c: 0x00000028
(gdb) print 0x28>>2
$1 = 0xa

The register we are looking for is loaded in R0, then the interrupt handler reads from this register (result in R4) and stores its value at 0x64C. If you check the memory contents at this address, you'll find 0x28. The interrupt ID would be 0x28 >> 2 = 0x0A, which is DryOS timer interrupt (which fires every 10ms and is used for the task switch - see e.g. FreeRTOS Tick (http://www.learnitmakeit.com/freertos-tick/) for some background info).

Note: ARM has a generic hardware interrupt (PC jumping at 0x18), but in an embedded system we usually have more devices that can trigger an interrupt. To distinguish between them, it's helpful to identify them somehow - on EOS, there is a custom interrupt controller, and on DIGIC 4 and 5 models, the register used for identifying the interrupt ID is 0xC0201004. The interrupt ID is hardwired to the device that triggers it (for example, SIO3 and MREQ, which are used for MPU communication, always use interrupts 0x36 and 0x50). This mapping can change across models, but usually it's consistent (at least within the same generation of models). Probably g3gg0 can explain these concepts a bit better, as he identified all this stuff a few years ago (http://www.magiclantern.fm/forum/index.php?topic=2882.0), when I was mostly clueless about how these things work.

OK, so the memory contents at 0x64C can tell us about the interrupt currently handled. There's a problem - this address is not cleared when the interrupt handler is done, so, just by looking at it, we can't tell whether the code is servicing an interrupt or running a regular task. We'll need to look at some other address - and in many cases the previous address is a good candidate:
Code: [Select]
   0x000004d4: ldr r1, [pc, #364] ; 0x648
   0x000004d8: cmp r1, #0
   0x000004dc: add r1, r1, #1
   0x000004e0: str r1, [pc, #352] ; 0x648
   ...
   0x00000598: ldr r1, [pc, #168] ; 0x648
   0x0000059c: sub r1, r1, #1
   0x000005a0: str r1, [pc, #160] ; 0x648

This 0x648 looks like a counter that tells how many nested interrupts we are handling (yes, they can be nested - unfortunately, as this makes it a lot harder to understand, debug, emulate and so on). Let's confirm its functionality with RAM tracing:
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d io,int,ram,nochain -singlestep
...
[tcm_code] at 0x000004D4:19980218 [0x00000648] -> 0x0       
[tcm_code] at 0x000004E0:19980218 [0x00000648] <- 0x1       : was 0x0;
...
[tcm_code] at 0x0000050C:0000050C [0x00000684] -> 0xC0201004
[INT]      at 0x00000510:0000050C [0xC0201004] -> 0x28      : Requested int reason 28 (INT 0Ah)
[tcm_code] at 0x00000514:0000050C [0x0000064C] <- 0x28      : was 0x0;
...
[INT]      at 0x00000594:00000568 [0xC0201010] <- 0xA       : Enabled interrupt 0Ah
[tcm_code] at 0x00000598:00000568 [0x00000648] -> 0x1       
[tcm_code] at 0x000005A0:00000568 [0x00000648] <- 0x0       : was 0x1;

Looks right!

Note: "Enabled interrupt 0Ah" means the interrupt routine finished handling it, so the same interrupt can be triggered again from now on (you can't have the same interrupt nested with itself). The same register configuration happens when an interrupt is enabled for the first time, and that's why the message reads "enabled interrupt". Might be a bit confusing at first sight.

Now that we know what 0x648 does, let's use it as boolean (0 = regular task, nonzero = handling some interrupt):
Code: [Select]
macro define CURRENT_ISR  (*(int*)0x648 ? (*(int*)0x64C) >> 2 : 0)

Next is CURRENT_TASK. One way to find that is by pattern matching in task_create and its subroutines.

I'll try an alternate method, based on one of the recent addition to QEMU: the ability to track function calls and returns. From the ARM calling convention (AAPCS (http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042f/IHI0042F_aapcs.pdf), or the summary from wikipedia (https://en.wikipedia.org/wiki/Calling_convention#ARM_.28A32.29)), you'll notice that registers R4-R11 are supposed to be unchanged after a function call. When tracking function calls, the logging code checks this assumption as well, and will print lots of warnings if a task switch happens without the logger knowing about it. Let's use these warnings to narrow down the location where DryOS switches between tasks:

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw

The log is huge, but we are going to look for some warnings about registers (run with only -d callstack to see them without the clutter from RAM writes). These warnings are likely caused by some memory writes performed just before the warnings. We are looking for some invariant address (one that doesn't change with the warnings):
Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d callstack,ramw |& grep -B 20 "not restored"
...
[ram]      at 0x000019D8:0000E24C [0x0018F8FC] <- 0xE24C    : was 0xE24C;
[ram]      at 0x00001A04:0000E24C [0x0008FBCC] <- 0x17EED4  : was 0x17EF28;
[ram]      at 0xFF0C1064:0000E24C [0x0018F8FC] <- 0xE24C    : was 0xE24C;
...
[ram]      at 0xFF0C1074:0000E24C [0x0018F8C0] <- 0x60000093: was 0x19980218;
[ram]      at 0xFF0C1078:0000E24C [0x0017EF78] <- 0x18F8C0  : was 0x18F8D0;
Warning: R4 not restored (0x17ef28 -> 0x17eed4)                                  at [ff0c1088:e24c]
...
[ram]      at 0x000019D8:0000E24C [0x0018E9A4] <- 0xE24C    : was 0xE24C;
[ram]      at 0x00001A04:0000E24C [0x0008FBCC] <- 0x17EF28  : was 0x17EED4;
[ram]      at 0xFF0C1064:0000E24C [0x0018E9A4] <- 0xE24C    : was 0xE24C;
...
[ram]      at 0xFF0C1074:0000E24C [0x0018E968] <- 0x60000093: was 0x18E9FC;
[ram]      at 0xFF0C1078:0000E24C [0x0017EF24] <- 0x18E968  : was 0x18E970;
Warning: R4 not restored (0x17eed4 -> 0x17ef28)                                  at [ff0c1088:e24c]

Most of these addresses are different for each warning message, except one. Let's try it. Find task_create and DebugMsg (easy) and add everything to EOSM2/debugmsg.gdb (look at other cameras for a template):
Code: [Select]
# ./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

source -v debug-logging.gdb

macro define CURRENT_TASK 0x8FBCC
macro define CURRENT_ISR  (*(int*)0x648 ? (*(int*)0x64C) >> 2 : 0)

b *0x4398
DebugMsg_log

b *0x7360
task_create_log

Code: [Select]
./run_canon_fw.sh EOSM2 -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
[      init:ff352264 ] task_create(PowerMgr, prio=20, stack=400, entry=ff352088, arg=0)
[      init:ff1470d4 ] (00:01) [PM] DisablePowerSave (Counter = 1)
[      init:0003671c ] task_create(DbgMgr, prio=1f, stack=0, entry=36628, arg=46e584)
[      init:ff0c3334 ] (8b:16)
K355 ICU Firmware Version 1.0.2 ( 6.0.5 )
[      init:ff0c3348 ] (8b:05)
ICU Release DateTime 2013.12.02 09:28:54
[      init:ff0f5ea0 ] (00:03) [SEQ] CreateSequencer (Startup, Num = 6)
[      init:ff0f5f28 ] task_create(Startup, prio=19, stack=2800, entry=ff0f5d6c, arg=46e880)
[      init:ff0f60f4 ] (00:02) [SEQ] NotifyComplete (Startup, Flag = 0x10000)
[      init:ff0f6158 ] (00:03) [SEQ] NotifyComplete (Cur = 0, 0x10000, Flag = 0x10000)
[      init:ff0c33d4 ] task_create(TaskMain, prio=1d, stack=0, entry=ff0c28d8, arg=0)
[   Startup:ff0f5db4 ] (00:05) [SEQ] seqEventDispatch (Startup, 0)
[   Startup:ff0c373c ] (8b:05) startupEntry
[   Startup:ff0c375c ] task_create(Startup2, prio=11, stack=400, entry=ff0c3604, arg=0)
[  Startup2:ff1310c4 ] (02:16) PROPAD_CreateFROMPropertyHandle DRAMAddr 0x416d5b00
[  Startup2:ff14a204 ] (00:01) [SF] IsAddressSerialFlash 0x2d0000
...

We now have task information and debug messages!

To make this information available in QEMU (not just in GDB), add this to model_list.c under the EOSM2 section:
Code: [Select]
        .current_task_addr      = 0x8FBCC,

That's it for today.