ML on EOS-M2

[a1ex]

In this case, the contents of boot-hack.c are identical on dm-spy-experiments-EOSM2.103 (de9f09df06e7) and EOSM2.103_wip (495f8c25235c):

Code: [Select]

hg diff -r 495f8c25235c -r de9f09df06e7 src/boot-hack.c
(no output)

In other words, it does not contain the dm-spy changes.

In my merge tool (Meld), if they are conflicts, I should first solve them, and then select Merge all. Sometimes I forget to select "Merge all" and run into similar errors - that's probably what happened on my side with 2214781fcb95. In this case I redo the merge and/or compare the merged changeset with both ancestors. For example, between dm-spy-experiments and dm-spy-experiments-EOSM2.103 I expect to see only the M2 changes, for example:

Code: [Select]

diff -r 63d4b3396a5f -r 501eb2ccd140 src/boot-hack.c
--- a/src/boot-hack.c
+++ b/src/boot-hack.c
@@ -660,31 +660,32 @@
 
     uint32_t* addr_AllocMem_end     = (void*)(CreateTaskMain_reloc_buf + ROM_ALLOCMEM_END + CreateTaskMain_offset);
     uint32_t* addr_BL_AllocMem_init = (void*)(CreateTaskMain_reloc_buf + ROM_ALLOCMEM_INIT + CreateTaskMain_offset);
+    uint32_t* addr_B_CreateTaskMain = (void*)(init_task_reloc_buf + ROM_B_CREATETASK_MAIN + init_task_offset);
 
-    #if defined(CONFIG_550D) || defined(CONFIG_60D) || defined(CONFIG_7D)
-    // change end limit to 0xc60000 => reserve 640K for ML
+    #if defined(CONFIG_EOSM2)
+    /* R0: 0x44C000 (start address, easier to patch, change to 0x4E0000 => reserve 592K for ML) */
+    /* R1: 0xD3C000 [6D, 700D] / 0xC3C000 [100D] / 0xD6C000 [EOSM] / 0xC2A000 [EOSM2] (end address, unchanged) */
+    addr_AllocMem_end[1] = MOV_R0_0x4E0000_INSTR;
+    ml_reserved_mem = 0x4E0000 - RESTARTSTART;
+    #elif defined(CONFIG_550D) || defined(CONFIG_60D) || defined(CONFIG_7D)
+    // change end limit from 0xd00000 to 0xc60000 => reserve 640K for ML
     *addr_AllocMem_end = MOV_R1_0xC60000_INSTR;
-    ml_reserved_mem = 640 * 1024;
+    ml_reserved_mem = 0xD00000 - RESTARTSTART;
     #else
-    // change end limit to 0xc80000 => reserve 512K for ML
+    // change end limit from 0xd00000 to 0xc80000 => reserve 512K for ML
     *addr_AllocMem_end = MOV_R1_0xC80000_INSTR;
-    ml_reserved_mem = 512 * 1024;
+    ml_reserved_mem = 0xD00000 - RESTARTSTART;
     #endif
 
     // relocating CreateTaskMain does some nasty things, so, right after patching,
     // we jump back to ROM version; at least, what's before patching seems to be relocated properly
     *addr_BL_AllocMem_init = B_INSTR(addr_BL_AllocMem_init, ROM_ALLOCMEM_INIT);
     
-    uint32_t* addr_B_CreateTaskMain = (void*)init_task_reloc_buf + ROM_B_CREATETASK_MAIN + init_task_offset;
+    // replace call to CreateMainTask (last sub in init_task)
     *addr_B_CreateTaskMain = B_INSTR(addr_B_CreateTaskMain, new_CreateTaskMain);
     
-   
-    /* FIO_RemoveFile("B:/dump.hex");
-    FILE* f = FIO_CreateFile("B:/dump.hex");
-    FIO_WriteFile(f, UNCACHEABLE(new_CreateTaskMain), CreateTaskMain_len);
-    FIO_CloseFile(f);
-   
-    NotifyBox(10000, "%x ", new_CreateTaskMain); */
+    /* before we execute code, make sure a) data caches are drained and b) instruction caches are clean */
+    sync_caches();
     
     // Well... let's cross the fingers and call the relocated stuff
     return new_init_task;

If you run this, you'll spot the mistake right away:

Code: [Select]

hg diff -r 63d4b3396a5f -r de9f09df06e7

I've also got differences in edmac-memcpy.c (where hg considered it was a conflict, but in Meld, I only had to select Merge all without changing anything) and in qemu-util.c (which should be deleted).

That said, I've ran 495f8c25235c+2214781fcb95 in QEMU and only seems to work with CONFIG_QEMU=y, but not without. Looking into it.

BTW - two tricks to speed up the compilation:

Code: [Select]

# only recompiles and installs autoexec.bin, without modules
cd platform/EOSM2.103; make installq ML_MODULES_DYNAMIC=


Code: [Select]

# requires a tiny change from latest unified (g3gg0 found out yesterday)
make -j4


[dfort]

If the dm-spy-experiments branch saves a valid log in QEMU with CONFIG_DEBUG_INTERCEPT_STARTUP=y and CONFIG_QEMU=n, that means we are already there and I'll enable the boot flag.

Cleaned up and updated the EOSM2.103_wip branch so that a pull request on the latest unified branch is showing only the M2 changes. Then I used that to do a new dm-spy-experiments merge and was able to save a DM-0000.LOG. Yay! There are a few error messages in it so as a sanity check I tried the 700D from the latest dm-spy-experiments branch (without the EOSM2 merge) and there are more error messages in the 700D log than in the EOSM2 log!

We'll also need the sf_dump module - that should re-create the SFDATA.BIN file, although I've never tested it that way (todo: include this in the test suite).

Ok--so I included the sf_dump module and on the 700D I was able to get into the ML menu, turn it on and on the next QEMU run (after removing the LOADING.LCK file because of the unclean shutdown) I was able to get into the Debug menu and select "Dump serial flash" but it didn't save a SFDATA.BIN file.

Basically, the EOSM2 is behaving pretty much like the 700D except that I can't get into the ML menus because of the unresponsive LiveView behavior of the EOSM2. Haven't figured out how to workaround it yet.

[dfort]

I can't get to the Magic Lantern menu to check it out.

That's probably because the M2 does not show the "idle" Canon screen (the one with shooting settings); as soon as you close the date/time dialog, it will go to LiveView (which doesn't work in QEMU).

You should be able to work around it by allowing the menu to come up in any GUI state, not just when "idle".

Still struggling with this. No matter what I tried, closing the Canon menu sends it into a black abyss. The QEMU console shows activity when pressing the buttons but nothing comes up on the screen.

How does the dm-spy-experiments startup log look?

[EDIT] Superfast compilation:

Code: [Select]

make -j4 installq ML_MODULES_DYNAMIC=
Note that the card needs to have at least the ML/modules directory in order to save the .sym file or compilation will fail and running QEMU requires the other directories along with the files that belong in them or else:

[dfort]

I might be bumping up against imperfect emulation. Are we ready to run some tests on the real hardware?

Those errors are likely just imperfect emulation (we are using properties from 100D, not from a real camera).

However, we should be close to getting these things from real hardware. If the dm-spy-experiments branch saves a valid log in QEMU with CONFIG_DEBUG_INTERCEPT_STARTUP=y and CONFIG_QEMU=n, that means we are already there and I'll enable the boot flag.

We'll also need the sf_dump module - that should re-create the SFDATA.BIN file, although I've never tested it that way (todo: include this in the test suite).

[vagaboundedu]

Do you already have an M2 to test on? How do you know when it's time?

[dfort]

Yes and when a1ex makes the decision that it is time.

Like I implied in prior posts, I hit a wall. I can't get the ML menus to come up in QEMU. Maybe I need to re-read a1ex's posts about MPU communication or maybe it is because we're still using a 100D SFDATA.BIN and need to figure out how to get the "real" EOSM2 SFDATA.BIN.

Then again a1ex hinted that he might go ahead and enable the boot flag, something that only a developer with the right tools can do. If so maybe we'll be able to see ML working on the M2--or maybe we'll brick it.

[vagaboundedu]

Okay! Well, I definitely hope it's the former!

[dfort]

@a1ex

Don't want to be a pest but the EOSM2 port is feeling a little jealous lately. Any hint on what the next step should be? Maybe turn on the boot flag?

[a1ex]

You are not running into emulation issues yet (except for the lack of LiveView, not needed at this stage) - this camera requires new-style DryOS task hooks (that's why it doesn't handle the buttons). There are two ways to deal with it:

- the old 6D method from unified (which I don't like, as it increases the "ifdef" complexity and has some missing functionality)
- the new one, which was recently confirmed on 100D and - less than 24 hours ago - on 70D.

If you are not rushing, let me merge the new DryOS hooks into unified first. I've already started to clean up QEMU for merging, since that's a pre-requisite. Maybe also 100D and 70D (the former looks good to me, but I wanted to use the new backends for it, the latter has a few of rough edges, but hopefully not show-stoppers).

[dfort]

I'm not rushing--just feeling that I lost the momentum.

Last night I built a new QEMU testing environment with all the latest changes and that's when I discovered that silent.mo wasn't building in my EOSM2 branch. Digging deeper I found the same problem existed on all the other cameras. Now that I'm a little more familiar with the ML code thanks to this porting project, I was able to submit a fix instead of just submit a bug report.

Of course I'd like to get the camera boot flag turned on and start playing around with ML on this camera but that's the little kid in me wanting to play with his toy.

[bookemdano]

As someone who just bought an M2 I'm really pulling for you dfort. I specifically chose the M2 over the M3 and newer cameras because like the M it runs DryOS and is therefore capable of running ML. Plus it uses the uber-cheap LP-E12 batteries and there's no stupid hassle with using third party batteries/chargers. I chose it over the M because of the faster focusing speed & more focusing points, the lighter/smaller body and the built-in WiFi (since none of the Ms can do tethered shooting over USB, WiFi control is better than nothing at all and is a possible avenue for future innovation).

I am not well-versed enough in coding to assist with the actual porting, but I have a fairly high risk tolerance and a mind for diagnostics. So when it comes time to put an alpha through its paces, sign me up. The M2 is not my main camera, and they can be had somewhat cheaply on ebay so if I brick it, it's not the end of the world.

I'm heartened to hear A1ex's comments that this should be a fairly straightforward port. And dfort I've seen enough of your other contributions here to know that you've got what it takes to do this. So I guess I just want to give you simultaneous pat on the back/shove to keep going. The M2 is the best Canon mirrorless model capable of running ML, so it deserves to run ML

Seriously though, thank you guys for all the time you've spent so far on this endeavor. I'll enjoy my new M2 even without ML, but man I hope I get to run ML on it.

[dfort]

@bookemdano -- are you able to use Mercurial (command line or through a GUI like SourceTree) and compile ML? That would be the best way to test, once the camera boot flag is enabled of course. So while we're waiting for that, take the time to set up a testing/development environment. It isn't that hard. I've got a few tutorials on how to get something up quickly on Mac or Windows. If you're on Linux you don't need a tutorial.

[odell_matt]

Im sorry but I have no idea whats going on. I have a Canon Rebel T6 and i want to know if it possible to run it. Someone recommended this forum but i don't know like anything about coding or that stuff. Can anyone help me with what i need to do?

[bookemdano]

@bookemdano -- are you able to use Mercurial (command line or through a GUI like SourceTree) and compile ML? That would be the best way to test, once the camera boot flag is enabled of course. So while we're waiting for that, take the time to set up a testing/development environment. It isn't that hard. I've got a few tutorials on how to get something up quickly on Mac or Windows. If you're on Linux you don't need a tutorial.

First of all, sorry for the delay in response. I posted my earlier message and then promptly left on a two week vacation. I'm back now and followed your very thorough OS X tutorial. I just successfully compiled the Oct 1 nightly for the 6D, although I haven't tried actually running it on my 6D yet (guess I should do that before I deem it successful). But I think I'm ready to go for when you've got something ready to test.

I used the M2 on my trip and I'm really impressed with how much camera Canon packed into such a tiny body. I was prepared for the slow-ish focus--it's not a big deal to me. My only real gripe is that it chewed through the wimpy LP-E12 batteries like candy. I think I'm going to pick up a generic ACK-E12 AC adapter for when I test ML on it. I'm still glad to not have to hassle with the E17 and the "feature" Canon added that makes using generic batteries as annoying as possible.

So... any progress lately? I know each body's ML port is like a crying puppy badly needing love and there aren't enough devs here to tend to them all. And the M2 would definitely be the runt of the litter given how few of them were sold.  So no worries if you're prioritizing something else that will benefit more people. But by all means, if there is anything I can do to assist, let me know. I don't have much knowledge about ML development but I'm pretty decent at (eventually) figuring stuff out with some google-fu, asking dumb questions and good old trial and error.

[dfort]

...I know each body's ML port is like a crying puppy badly needing love and there aren't enough devs here to tend to them all. And the M2 would definitely be the runt of the litter given how few of them were sold...

Great analogy!

I went as far as I could go for now. I either need to wait get the camera boot flag enabled to test it on the camera or wait until full emulation is available on QEMU. Not an unlikely scenario, the original EOSM has just recently had the menus working in QEMU. In any case, a1ex has a long list and being the runt of the litter we're near the end of the queue.

[a1ex]

Full emulation (aka LiveView) is unlikely to happen anytime soon; the EOS M was just catching up and it's currently in the same state as M2, emulation-wise.

What the M2 requires is new DryOS task hooks. That's not a QEMU limitation; the DryOS internals changed on recent models - if I enable the boot flag now, you'll have the same issue on the real camera. If you haven't been able to solve it in the emulator (where you have debugging tools available), I doubt you'll be able to solve it on the physical camera.

I'm trying to get new DryOS hooks into unified, but we've got into trouble on 100D (some cameras don't boot with this method, or they do so randomly, apparently dependent on some Canon settings).

Of course, you can try the method meanwhile (just merge new-dryos-task-hooks), or try the old 6D workarounds, or try this exercise on the original M (which has old-style DryOS).

Or, find some configuration (Canon settings) on both the original M and on the M2 where the camera doesn't go directly in LiveView when powering on, and ML menu can be started from that screen (not sure if there is such screen on these cameras), dump the ROM in that state, maybe get a startup-log (dm-spy-experiments) and try it.

And new-dryos-task-hooks depends on QEMU, so I should get that one in unified first (that was the reason for making sure it compiles and installs on most common operating systems, btw).

So, there are still a couple of things to try until the new backends will be ready.

[dfort]

Thanks for the tips. I've also been trying to work out the kinks on my other cameras, EOSM and 700D, and have actually gotten out in the field with the 5D3. Looks like I should spend some more time with this little runt. When I first started this port I thought it would be easy--ha!

[bookemdano]

In hopes of trying to contribute something I spent the last hour seeing if there was some way of getting the M2 to power on to any other screen other than LV, and unfortunately I wasn't able to find one. I know with my PowerShot S95 I can press the Play button when the camera is off and it powers on and goes directly to playback mode. But with the M2, none of the buttons appear functional until the Power button is pressed, and that directly goes to LV mode, no matter what settings I tried in the menus. Does the original M have the same limitation?

[dfort]

The EOSM and EOSM2 work the same when it comes to starting the camera with the play button. They start up in playback mode which is not LiveView.

I've been busy on other projects so I haven't been able to look into it but the problems with the new DryOS task hooks that are causing problems on the 100D are likely a problem on the EOSM2 because these cameras are very similar. I'll follow the "fix" on the 100D when it is ready.

In the meantime you might look into doing a firmware dump and running it in QEMU. Yeah, I know it is a challenge. Just start here, take it a step at a time and follow what was already posted. Note that compiling ML and getting QEMU up and running is much easier these days.

[bookemdano]

The EOSM and EOSM2 work the same when it comes to starting the camera with the play button. They start up in playback mode which is not LiveView.

I've been busy on other projects so I haven't been able to look into it but the problems with the new DryOS task hooks that are causing problems on the 100D are likely a problem on the EOSM2 because these cameras are very similar. I'll follow the "fix" on the 100D when it is ready.

In the meantime you might look into doing a firmware dump and running it in QEMU. Yeah, I know it is a challenge. Just start here, take it a step at a time and follow what was already posted. Note that compiling ML and getting QEMU up and running is much easier these days.

Oh, derp. I see now you have to hold down the play button for 1+ seconds to get it to start in playback mode--a momentary press won't cut it. That's probably even in the manual for the camera which I probably ought to read.

Was that what a1ex was talking about then--a way to start up the camera in a screen other than LV and launch ML from that screen? Or can ML not run from Playback mode either? Maybe he was just referring to running it within QEMU. Sorry if those are dumb questions.

I'll take your last paragraph as a stretch goal and see how far I can get.

[dfort]

What a1ex was talking about was to save the firmware dump in a setting other than LiveView. His exact words:

...find some configuration (Canon settings) on both the original M and on the M2 where the camera doesn't go directly in LiveView when powering on, and ML menu can be started from that screen (not sure if there is such screen on these cameras), dump the ROM in that state...

[dfort]

What the M2 requires is new DryOS task hooks. That's not a QEMU limitation; the DryOS internals changed on recent models - if I enable the boot flag now, you'll have the same issue on the real camera. If you haven't been able to solve it in the emulator (where you have debugging tools available), I doubt you'll be able to solve it on the physical camera.
...
Of course, you can try the method meanwhile (just merge new-dryos-task-hooks)...

Finally got around to trying out the new DryOS task hooks but still no luck getting to the ML menus. Haven't tried the other suggestions yet.

On the Mac "fn delete" brings up the ML menu on the 700D but nothing on the EOSM2. In fact the EOSM2 still goes into the black abyss when leaving the Canon menu and the "M" key doesn't bring it back to life.

I've been trying out the EOSM2 on various branches and here is what I'm doing in QEMU:

Code: [Select]

make -C ../magic-lantern/platform/EOSM2.103 install_qemu
./run_canon_fw.sh EOSM2,firmware="boot=1" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

Don't have to bother mounting and unmounting the sd disk image. Nice!

[vagaboundedu]

Thank you for your work on this. Here's hoping there will be a way to get through this issue!

[orielsy]

Finally got around to trying out the new DryOS task hooks but still no luck getting to the ML menus. Haven't tried the other suggestions yet.

On the Mac "fn delete" brings up the ML menu on the 700D but nothing on the EOSM2. In fact the EOSM2 still goes into the black abyss when leaving the Canon menu and the "M" key doesn't bring it back to life.

I've been trying out the EOSM2 on various branches and here is what I'm doing in QEMU:

Code: [Select]

make -C ../magic-lantern/platform/EOSM2.103 install_qemu
./run_canon_fw.sh EOSM2,firmware="boot=1" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

Don't have to bother mounting and unmounting the sd disk image. Nice!

Hello all,

I'm a computer science graduate and would like to join in on tackling this as I just purchased the M2. These days I mostly deal with Frontend Development so it's been a while since I've touched C++, Assembly or anything like that.  I'm hoping I can chip in though.

Any info to help me get up to speed helps. In the mean time I'll be reading all the comments here.

[Audionut]

Sticky topics in this section will get you started.