ML on EOS-M2

[platerytter]

Just registered to say there's more people cheering for you guys! The m2 is a fantastic camera and deserves a little magic

[dfort]

Yes, it would be great to end the year with at least "Hello World" running on the M2 but that requires an ML-SETUP.FIR file and a1ex decides who's been naughty or nice enough to get that present.

[vagaboundedu]

 

[dfort]

Enabling the boot flag on the camera might be one small step but it feels like a giant leap.



First issue that came up was that the firmware signature that I got out of QEMU didn't match what I got running the code on the camera:

src/fw-signature

Code: [Select]

-#define SIG_EOSM2_103 0x17D200D6
+#define SIG_EOSM2_103 0x1f321405

Still to do--figure out the Trash button code so we can get to the ML menu. There's also some other obvious issues but hey, it's alive!

[Lars Steenhoff]

Thats a great step!  congrats 

[Kharak]

Yes, it would be great to end the year with at least "Hello World" running on the M2 but that requires an ML-SETUP.FIR file and a1ex decides who's been naughty or nice enough to get that present.

You've beev a good boy it seems Congrats!

[dfort]

Got a basic in-camera startup log. Still looking for that magic Trash button code.

[chrissul13]

This makes me SO happy.  I just got my EOS M2 which i bought without realizing ML did not work with it...and now it may!!! can't wait.  I'm mainly just wanting focus peaking as it is HARD to focus this camera manually

[platerytter]

Terrifc update dfort! Happy new years to you!

[vagaboundedu]

This is awesome! It made my day to see the pictures you posted!

I agree--the focus peaking will be very helpful, as will clean video output for recording productions.

[dfort]

@a1ex committed a change for the EOSM2 on the qemu branch so of course I tried it out. Wish I could say everything was rainbows and unicorns but--

Code: [Select]

./run_canon_fw.sh EOSM2 -s -S &
arm-none-eabi-gdb -x EOSM2/patches.gdb -ex quit

...
Breakpoint 1, 0xff0c5144 in ?? ()
Patching LeoLens (infinite loop)
[MPU] Received: 06 05 09 11 01 00  (PROP_LV_DISPSIZE - spell #22)
[MPU] Received: 12 11 09 15 00 00 00 00 00 00 00 00 00 00 00 00 00 00  (PROP 80050020 - spell #23)
[MPU] Received: 14 13 09 25 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  (unnamed - spell #24)
[MPU] Received: 06 05 01 5a 00 00  (PROP_CONTINUOUS_AF_VALID - spell #25)
[MPU] Sending : 06 05 01 59 01 00  (PROP_MOVIE_SERVO_AF)
[MPU] Received: 06 05 09 2f 01 00  (unnamed - spell #26)
[MPU] Received: 06 05 0e 22 1e 00  (unknown - unnamed)
    70:   121.344 [MR] mvrChangeAckCBR : Video - Mode=0, Type=2, Rate=30, GOP=15
    88:   125.440 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050038
    89:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050041
    90:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050042
    91:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050043
    92:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050044
    93:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105004F
    94:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050050
    95:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050051
    96:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050052
    97:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010E
    98:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010F
    99:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050110
   100:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050111
   101:    60.160 [PROPAD] ERROR GetPropertyData ID (1) = 0x0104000B
   102:    60.160 [LVCOM] InitializeLiveViewDefectDetection
   103:    60.416 [LVCOM] ExecuteDefectMarge Start
   104:    60.416 [LVCOM] ExecuteDefectMarge End
   106:    61.184 [LV]  PROP_LV_BLOCK  PROP_LV_UNBLOCKING 0
   108:    62.208 [AUDIO] RegisterCBRSDIOCableConnect
   109:    64.256 [WFT] InitializeAdapterControl END
SD: Unknown CMD1
[SDIO] Error
SD: Unknown CMD1
[SDIO] Error
SD: Unknown CMD1
[SDIO] Error
   124:   151.808 [SD] ERROR SDINTREP=0x00000000
   125:   151.808 [SD] ERROR UNEXPECTED ERROR
   131:   113.152 [RSC] WARN ILLEGAL SUBFREECLUSTERCOUNT ShootStorage.c 686
   157:   184.832 [DISP] ERROR PROP_TFT_SETTING_PARAM DON'T FIND
   158:   184.832 [DISP] ERROR Factory Adjust For TftCom
   159:   185.088 [MR_MOV] (Empty Func) MVW_RegisterXmpDataCallback
   160:   142.336 [TCH]Touch IC Ver : 0x0000
   161:   569.600 [TCH]ERROR  TouchPanelIC Ack Error
   162:   569.856 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   163:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   164:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   165:   570.112 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   166:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   167:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   168:   570.368 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   169:   570.624 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   170:   570.624 WARN [I2C] localI2C_Read : 378 (Task : Startup)
   171:   572.672 [IMPP] H264E InitializeH264EncodeFor1080pDZoom
   172:   572.672 [IMPP] H264E InitializeH264EncodeFor1080p25fpsDZoom
   179:   578.816 [STARTUP] startupInitializeComplete
   185:   580.864 [CEC]CEC OFF

I usually use debugmsg.gdb instead of patches.gdb so I gave that a go but couldn't get Hello World to print, much less the firmware signature. Going back to my previous build it worked--sort of. As expected, it printed the wrong firmware signature.

There is some positive news to report. Although I still can't get to the ML menu on the camera or QEMU I hacked the ML/SETTINGS/MAGIC.CFG file to see if any of the features are working--say like focus peaking?

It works! Yay!

So what's the next step of this walk through? Seems like we need to find the code for the Trash button, which on this camera is also the down key or bottom of the thumb wheel. Maybe do a serial flash dump? I could probably get everything set up but how to trigger a module without access to the ML menu?

[a1ex]

Hello World is working out of the box here, with latest sources from EOSM2.103_wip (both minimal and CONFIG_HELLO_WORLD, both debugmsg.gdb and patches.gdb), same firmware signature as on real hardware (does this make any more sense now?)

Didn't test on Mac though; I'll try later if it still doesn't work. Without loading ML, is the Canon menu still working? (GUI emulation is covered in the test suite, which runs for every few commits, or every non-trivial commit).

Next steps were covered before: new-dryos-task-hooks (done for 1300D, some stubs apparently wrong on M2), a tiny hack in menu.c to bring menu in QEMU (not needed for camera), and if there are any buttons not recognized, just print their codes.

For working with modules, this tutorial should help: http://www.magiclantern.fm/forum/index.php?topic=19232.0

[dacampora]

Hello.  First post but long time reader. Just here to vouch my support for the M2. Mostly interested in time lapse.

[dfort]

Got it working on Windows Subsystem for Linux (WSL) -- (Scream like a little girl!)

(does this make any more sense now?)

Yes it does. Anytime the ROM is patched the firmware signature changes. Is that the lesson here?

Didn't test on Mac though; I'll try later if it still doesn't work. Without loading ML, is the Canon menu still working?

Yeah, looks like there is a Mac issue, but only with patches.gdb. Running ML with debugmsg.gdb does show the firmware signature and it matches what I'm getting in camera. It was just that the commit you pointed out was on patches.gdb so I thought it was necessary to run it with patches.gdb.

One other observation - on Mac it pauses for user input while on WSL it doesn't:

Code: [Select]

---Type <return> to continue, or q <return> to quit---
I merged in the new-dryos-task-hooks a while back but maybe I screwed it up. I'll try it again and see if I can track down the bogus stubs.

..a tiny hack in menu.c to bring menu in QEMU (not needed for camera), and if there are any buttons not recognized, just print their codes.

A tiny hint on how to print out the button codes?

[a1ex]

Yeah, looks like there is a Mac issue, but only with patches.gdb.

Reproduced on Mac. With just patches.gdb, it didn't work before that commit either...

However, running with patches.gdb and with -d debugmsg.gdb brings the GUI. Must be a timing issue, and the additional debug messages are slowing it down; in the end, that slowdown ends up helping. These timing issues are why many of the tests are still failing (nondeterministically; in many cases just giving slightly different screenshots, not what the test suite expects).

Reproduced on Linux with -icount 1 or 2 (this gives deterministic execution). Any other number between 3 and 10 is working.

BTW, debugmsg.gdb always includes patches.gdb, and adds additional debug messages.

One other observation - on Mac it pauses for user input while on WSL it doesn't:

I only get the prompt when the terminal window is too small. It doesn't interfere with the "-ex quit" added recently; that's good.

A tiny hint on how to print out the button codes?

Debug -> Show GUI events.

[chrissul13]

I'm not saying i'm squealing like a little girl, but....well, focus peaking would be awesome.  I'm getting to know the camera and that's like my only weakness with it. 

Goodluck, can't wait for more

[dfort]

Next steps were covered before: new-dryos-task-hooks (done for 1300D, some stubs apparently wrong on M2)

I tried merging new-dryos-task-hooks again and this time it looks like it worked. At least "Hello World" is running--tested on the camera too.

You can now see DryOS tasks switching if you compile with CONFIG_QEMU=y and you enable DEBUG_TASK_HOOK in boot-hack.c. Without the latter, only new tasks will be displayed.

Did that too. Don't know what exactly to look for but there are a lot more messages flying through the terminal when run in QEMU.

So--one small step.

[a1ex]

You should see DryOS tasks switching back and forth (try on some other camera model, in particular, 6D, 100D, 70D or 1300D). Caveat: you should undefine CONFIG_HELLO_WORLD, as that option disables the task hook (to prevent running unwanted GUI code or other tasks).

Similar info is printed when running with "-d tasks" (useful for cross-checking the messages).

I don't see any of these messages here -> my_task_dispatch_hook does not get called. That's why I believe some stubs might be wrong - on 1300D I had to fix task_dispatch_hook (see also the commit for 6D)

[dfort]

Fixed task_dispatch_hook so it matches the changes on the 1300D and 6D. I couldn't follow the comments on the task_dispatch_hook for those cameras but I used my best pattern matching skills and also checked against the 100D and 70D so I'm pretty confident that I've got this right. By the way, the EOSM doesn't fall into the same pattern as the other cameras. Is new-dryos-task-hooks working properly on the EOSM?

In the process I also found a typo on task_trampoline and fixed it.

Code: [Select]

/** Tasks **/
NSTUB(   0x8FBE0,  task_dispatch_hook)                      // changed from 0x8FAB4 - now it doesn't work on the camera.
NSTUB(0xFFD24B44 - RAM_OFFSET,  task_create)
NSTUB(0xFFD2A1D8 - RAM_OFFSET,  task_trampoline)

However, as you can see from my comment it no longer works on the camera. LED turns on and camera won't boot up. So there must be something else that needs fixing. I know there are other things to fix. The lens focal length is displayed as 0 thought the f/stop and shutter speed is fine.

So it is one step forward and two steps back.

Still to do:

...a tiny hack in menu.c to bring menu in QEMU (not needed for camera), and if there are any buttons not recognized, just print their codes.

Big discovery for today is that adding this line to MAGIC.CFG the camera boots up with the console open.

Code: [Select]

module.console = 1



Now how to get the console to show button events? I'll bet the answer is to get the task hooks working properly.

[dfort]

Took another step forward -- got a SFDATA.BIN dump from the camera. I still can't get to the ML menu so here's how I did it.

In order to load modules on startup simply add an empty file with the name of the module in the ML/SETTINGS directory. For the sf_dump module it has to be named like this:

Code: [Select]

SF_DUMP.EN
Now in order to get the module to auto run without having to start it from the Debug menu I did this hack to sf_dump_init():

Code: [Select]

//    menu_add("Debug", sf_dump_menu, COUNT(sf_dump_menu));
    sf_dump_task();
    return 0;

It also worked in QEMU but since we've been using a 100D SFDATA.BIN all that happened was that it copied the 100D file to the card image. Now I've got a "real" EOSM2 SFDATA.BIN to work with.

[a1ex]

Fixed task_dispatch_hook so it matches the changes on the 1300D and 6D. I couldn't follow the comments on the task_dispatch_hook for those cameras but I used my best pattern matching skills and also checked against the 100D and 70D so I'm pretty confident that I've got this right.

Looks right to me.

The comments are still valid; for example the 1300D one:

// task_trampoline -> last call -> last non-empty BL -> one indirect call here
task_trampoline (0xFFD2A1D8 - RAM_OFFSET) -> 0xC9F4.
Last call: B sub_C8FC
Last non-empty BL: sub_C8FC ends at C9F0, the BL at C9E8 calls an empty function, the one before it calls sub_1D7C.
One indirect call here:

Code: [Select]

1D88                 LDR     R0, =0x8FBE0
1D90                 LDR     R3, [R0]
1D94                 CMP     R3, #0
1DA4                 BLXNE   R3

Answer: 0x8FBE0.

The comments from 6D are easier to follow with a decompiler, but they follow the same logic.

By the way, the EOSM doesn't fall into the same pattern as the other cameras. Is new-dryos-task-hooks working properly on the EOSM?

From what I could tell in QEMU, it works fine. If ML is still working on the camera with new-dryos-task-hooks (or lua_fix, which includes the former), then it works fine.

The EOSM uses old-style task hooks (that's a property of Canon code, we can't change it); our debug messages are still printed with the new-dryos-task-hooks branch (on the old-style code path; they look slightly different). The point of that PR is to have a unified codebase that covers both old-style and new-style cameras.

However, as you can see from my comment it no longer works on the camera. LED turns on and camera won't boot up.

That's odd, as with your vanilla code, it boots fine in QEMU. Just in case: you haven't run the binary compiled with CONFIG_QEMU=y on the camera by mistake, right? (that would result in lock-up)

You still have to enable the new-style DryOS hooks; noticed that was a bit non-intuitive, so I've just refactored it to use CONFIG_NEW_DRYOS_TASK_HOOKS in internals.h. Previously, you had to comment out HIJACK_TASK_ADDR to enable the new hooks. Not enabling the new-style hooks should not result in camera lock-up.

Side note: confirmed the new DryOS hooks, in their current shape, are also compatible with DIGIC 6 (tested on 80D) and from what I could tell without trying, also with DIGIC 7 (not tested, but the task hook call looks very similar).

Now how to get the console to show button events? I'll bet the answer is to get the task hooks working properly.

Right.

[dfort]

It's Alive!

I've just refactored it to use CONFIG_NEW_DRYOS_TASK_HOOKS in internals.h.

That's all it needed. The Trash button brings up the ML menu on the camera, no need to hunt down the button code.

Of course this isn't the end of the adventure, there's still lots to do. The ML menus look a little rough, the screenshot feature doesn't work and [EDIT - Didn't work on the first boot but now it works] there are other obvious things that need fixing but hey--it is finally running on the camera.

BTW--Yesterday was my birthday and this is the best present!

[JohanJ]

Congratulations to this breakthrough - and happy birthday dfort!!


Sent from my SM-G930F using Tapatalk

[dfort]

@a1ex -- A question about porting a new camera.

I would think that the goal is to eventually make a pull request in order to add a new camera to the unified branch so I created a "dummy" pull request in my ML fork:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/4/eosm2103-wip/diff

Looks like I've got lots of things in there that aren't a part of the unified branch yet. Should I continue merging in branches to my work in progress branch or start over with a clean branch from unified and add in my changes? My gut feeling is that I should continue on my wip branch. The reason I'm asking is because I believe that the next step I should take is to merge in the latest dm-spy-experiments and make an in camera startup log. This might resolve issues we're having with QEMU like not recognizing some of the buttons and losing the GUI after going into Live View.

Then again maybe the goal is to make a pull request to the "bleeding edge" crop_rec_4k branch and bring in all the good stuff from the various branches as long as it doesn't break the other platforms?

I know we're far from a pull request, much less getting it approved and merged but thought I'd ask anyway. After all it is working--sort of. Let's just say it is crawling for now.

[EDIT] Speaking of QEMU issues, when running "./run_canon_fw.sh EOSM2 -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb" it works fine but running "./run_canon_fw.sh EOSM2 -d debugmsg" I'm getting an infinite loop like this on the Mac:

Code: [Select]

   252:  6418.176 WARN [I2C] localI2C_Write : 317 (Task : Startup)
[     Startup:ff349fd8 ] (14:02) I2C > [01] 02
[     Startup:ff356f04 ] (00:26) [I2C] localI2C_Write : 317 (Task : Startup)
[     Startup:ff34a020 ] (14:06) I2C abort 0(1020000)
   253:  6466.816 WARN [I2C] localI2C_Write : 317 (Task : Startup)
   254:  6466.816 [AUDIO] ERROR I2C abort 0(1020000)
[     Startup:ff349fd8 ] (14:02) I2C > [01] 02
[     Startup:ff356f04 ] (00:26) [I2C] localI2C_Write : 317 (Task : Startup)
[     Startup:ff34a0a4 ] (14:03) I2C_Write retry 1 1 c3

No issues with pausing if the terminal window is "too small" though.

[a1ex]

For 1300D, I've merged new-dryos-task-hooks (which you need anyway), qemu, lua_fix (which should reach the mainline soon) and 1200D (for similarity). I think it's best to start a clean branch starting from unified, with the first 3 branches merged.

You could also consider including 100D, which is very similar to EOSM2 and also waits for the first 3 backends to reach the mainline (otherwise it's pretty much ready). 70D is a bit more bleeding edge (it uses patchmgr, which I can still crash with ERR70, although not exactly reproducible), so let's leave this one aside for now.

You can include dm-spy-experiments in your current wip branch and use it for reverse engineering (understanding how things work). This branch breaks things on some cameras (for example, 600D did not even boot), so let's not include this one in the PR yet.