Author Topic: DIGIC 8 'PowerShot' development (M50, SX70, SX740)  (Read 6073 times)

srsa

  • New to the forum
  • *
  • Posts: 11
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #25 on: December 18, 2018, 06:13:04 PM »
sorry, but I didnt have success on my M50, I prepared the Card with EOSCard after true formating in the M50
(but with my M50 I also did not have success with the dumper scripts)
Thanks for trying. I guess we'll know more when emulation of the M50 becomes possible.
EOScard?
Is there a bootflag enabler for the real (not emulated) M50 body?
Wondering if I missed something ...
EOSCard can also set PowerShot-related flags, in this case the SCRIPT signature.

Walter Schulz

  • Hero Member
  • *****
  • Posts: 6569
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #26 on: December 18, 2018, 06:28:18 PM »
EOSCard can also set PowerShot-related flags, in this case the SCRIPT signature.

Thanks for the explanation! I should get accustomed to those "hybrid cams" (for lack of a better word) terms...
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12037
  • 5D Mark Free
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #27 on: December 24, 2018, 10:39:36 PM »
Please find a ROM dumper for the EOS/PowerShot M50. It took an entire week of trial and error to figure it out - thanks @71m363nd3r for testing!

D_M50101.FIR (only for M50 firmware 1.0.1)

Source code: commit 568e05a.

This is not using the portable codebase, but runs directly from the main firmware, using the regular ML boot process. For this reason, it is only compatible with M50 firmware 1.0.1. On other firmwares, it will just lock up.

There were two mysterious bugs that were preventing ML from booting on the M50 on real hardware, although it was working fine in the emulator, and its early startup code was apparently identical to 200D's:

1) The FIR is running on both CPU cores (unlike DIGIC 7, but I need to double-check). The two cores do not start at the same time; the main one (CPU0) wakes up the secondary one (CPU1) somewhere in cstart. It was easy to fix, but figuring it out with LED blinks was very puzzling.

2) The startup code, although nearly identical to 200D's, expected one of the cache maintenance functions to preserve R3 across the call (something not compatible with the C ABI). In our relocated startup code (that makes room for loading ML), I had to use a veneer to make a long call to that function. From all of the available registers, GCC picked exactly R3 to make that long call...

... and the side effect was disabling a loop that was zeroing out some memory. The trick is that, at camera startup, the RAM is not zeroed; you still have some electrical noise or maybe even some data from previous run. In the emulator, the RAM is always zeroed at startup. The result was that my test code was booting just fine in the emulator, but was locking up on real hardware. Good luck figuring out with binary feedback from the camera (i.e. works or locks up). I've solved it comparing the execution traces between the files reported to work (i.e. jumping to unmodified Canon firmware), and the ones reported to lock up (jumping to relocated Canon firmware); if anyone is curious, . I highly doubt I could have figured it out without an emulator. This change fixed it.

Once the two quirks were fixed, the startup code previously written for 200D worked out of the box.

When dumping the ROM, there was yet another quirk: initial attempt (calling dump_file with a ROM address) saved a file with correct size, that contained garbage. Oddly enough, it had fragments from the diagnostic log. Best guess: the DMA channel used for file I/O might be simply unable to read from the ROM. Instead, it probably ignores the higher address bits and just saves something from the RAM (matching the lower address bits). I'm not sure whether this step was working on 200D - need to double-check. Anyway, fixing this was straightforward: copying the ROM contents to RAM before saving did the trick.

Emulation is not ready yet; still working on it. I am able to emulate the bootloader (including file I/O), but main firmware doesn't initialize the SD card, so I'm unable to test Canon Basic scripts yet.

The same dumper source code works on 200D (I've actually used it to test the M50 dumper in QEMU, as file I/O isn't fully working), and - after finding a few stubs - on all the other DIGIC 7 models. If anyone is looking for some low-hanging fruit to get started with development, the new stubs are relatively easy to find and they can be tested in QEMU. Once you do that, I'll enable the boot flag, so you'll be able to run custom code on real hardware.

Dear Santa, please give us the time and energy to port ML on these new cameras!

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #28 on: December 27, 2018, 08:52:31 PM »
so, apologies for being that guy, but where exactly do I start here?

I'm a skilled developer, but new to photography.  I have an M50 with the 1.0.1 firmware, and a decent amount of experience reverse engineering software.  I don't need any excessive hand-holding, just links, docs, and maybe a little setup troubleshooting.

If anyone is looking for some low-hanging fruit to get started with development, the new stubs are relatively easy to find and they can be tested in QEMU

Given the above, I can definitely give this a shot.  Sounds fun.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12037
  • 5D Mark Free
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #29 on: December 28, 2018, 09:02:48 AM »
Docs:
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst

However, that easy coding task mostly applies to models other than M50 and 200D. The current experiments are working out of the box on these two cameras; it's the other 3 models that could use some help from the community in order to reach the same stage.

I have yet to commit the emulator bits for M50, but I've got most of the stuff working locally (including file I/O after manually calling the initialization routine from the serial console). For now, you may run the ROM dumper and follow the steps from the "Initial firmware analysis" section in the docs. ROM layout and startup code on M50 are pretty much identical to DIGIC 7 (to the point that some stubs are identical with 200D).

Once you are ready to run this code on your camera, I can prepare a tool to enable the boot flag (for any model from this thread).

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #30 on: December 28, 2018, 04:23:36 PM »
Docs:
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst

However, that easy coding task mostly applies to models other than M50 and 200D. The current experiments are working out of the box on these two cameras; it's the other 3 models that could use some help from the community in order to reach the same stage.

I have yet to commit the emulator bits for M50, but I've got most of the stuff working locally (including file I/O after manually calling the initialization routine from the serial console). For now, you may run the ROM dumper and follow the steps from the "Initial firmware analysis" section in the docs. ROM layout and startup code on M50 are pretty much identical to DIGIC 7 (to the point that some stubs are identical with 200D).

Once you are ready to run this code on your camera, I can prepare a tool to enable the boot flag (for any model from this thread).

I assume the risk of bricking my camera with these tools is low (but not zero)?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12037
  • 5D Mark Free
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #31 on: December 28, 2018, 04:54:59 PM »
That's right. All of the binaries available in this thread were already confirmed to work on at least one other camera, and they were also tested in QEMU (where I could confirm they are reserving memory properly so our binary won't be overwritten by Canon firmware). They do not (at the time of writing) attempt to make any permanent changes to the camera. Sure, that will no longer apply after enabling the boot flag.

However, Canon firmware is, by design, unsafe. Any DryOS task is able to write anywhere in memory, so a programming mistake can - in theory - have devastating effects. Starting from DIGIC 7, we've got a MMU, which currently doesn't do much other than a flat mapping, but can - in theory - be used to implement some sort of protection between ML code and Canon data structures. TLDR: there is a small probability of getting invalid settings written into one of the nonvolatile memories; that's usually recoverable, but I'm unable to guarantee it.

So far, the worst case I've recovered successfully was a Canon firmware update interrupted in the middle (i.e. not caused by ML). Last time I've unbricked a camera was two days ago (an EOS M with invalid video settings; ask Danne for details).

Danne

  • Hero Member
  • *****
  • Posts: 5015
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #32 on: December 28, 2018, 05:17:35 PM »
Last time I've unbricked a camera was two days ago (an EOS M with invalid video settings; ask Danne for details).
Word! My prop value mistake analyzed, narrowed down and fixed to camera original state within the hour...

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #33 on: December 28, 2018, 05:24:31 PM »
So far, the worst case I've recovered successfully was a Canon firmware update interrupted in the middle (i.e. not caused by ML). Last time I've unbricked a camera was two days ago (an EOS M with invalid video settings; ask Danne for details).

that makes me feel a lot better

I'll report back here once I get a chance to read the docs and get set up.

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #34 on: January 02, 2019, 05:15:51 PM »
Docs:
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst

However, that easy coding task mostly applies to models other than M50 and 200D. The current experiments are working out of the box on these two cameras; it's the other 3 models that could use some help from the community in order to reach the same stage.

I have yet to commit the emulator bits for M50, but I've got most of the stuff working locally (including file I/O after manually calling the initialization routine from the serial console). For now, you may run the ROM dumper and follow the steps from the "Initial firmware analysis" section in the docs. ROM layout and startup code on M50 are pretty much identical to DIGIC 7 (to the point that some stubs are identical with 200D).

Once you are ready to run this code on your camera, I can prepare a tool to enable the boot flag (for any model from this thread).

I'm having a bit of trouble with `hg` and `brew` at the moment.  I still want to help, but I need to get back to LA first where I can clean up my Mac.  :)

kenthinson

  • New to the forum
  • *
  • Posts: 2
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #35 on: January 03, 2019, 01:55:02 PM »
Ok progress report and questions for you guys!

Progress.

I have been following:
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst
Step 1, and 2 under installation complete. Got dependencies installed through home-brew.

In addition I dumped my m50 rom files. I got two bin files
540b1d00a2631c6a29a0280d75679440 ROM0.BIN
1e7c58573aa62d016f3b355205d8fed5 ROM1.BIN

Should I continue with step 3? Is qemu ready? or do we need to do some more work first.

Lets get this thing rolling :)

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3503
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #36 on: January 03, 2019, 05:00:49 PM »
I'm having a bit of trouble with `hg` and `brew` at the moment.  I still want to help, but I need to get back to LA first where I can clean up my Mac.  :)

A couple of forum topics that will help get your Mac up and running in a hurry:

Setting up a Magic Lantern development environment on a Macintosh

Magic Lantern development compiler.app (Mac OS)

PM me when you get back to LA if you want to get together. I live nearby in Redondo Beach.

I keep thinking about getting an M50 for my next ML project--once the M2 is up and running.

Should I continue with step 3? Is qemu ready? or do we need to do some more work first.

I just checked and it looks like there isn't an EOSM50 directory yet so it might still be a little early. Though if you want to get started you can create that directory and start figuring out what is needed for the debugmsg.gdb file. The EOSM2 port started from scratch so you can see what was done to get that camera working in QEMU starting around this post in the ML on EOS-M2 topic.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12037
  • 5D Mark Free
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #37 on: January 03, 2019, 06:09:06 PM »
Though if you want to get started you can create that directory and start figuring out what is needed for the debugmsg.gdb file.

Well, that would be quite a bit of duplicated work. Please find here the draft QEMU patches for M50 in their current state. Not double-checked yet, but emulation goes far enough to save a log file on the virtual SD card with these commands in the serial console:
Code: [Select]
akashimorino
mount 2
dumpf

Not sure if that's enough for testing the Canon Basic scripts, but that's all I've got for now.

The M50 also has a serial flash, so one of the next steps is dumping its contents (easy; one way is to adapt this code on the minimal codebase, which is not able to load modules yet, and the other is to work from the "recovery" branch, i.e. directly from bootloader). Probably not a very straightforward task for a newcomer, but it's not very hard either.

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #38 on: January 04, 2019, 12:36:11 AM »
A couple of forum topics that will help get your Mac up and running in a hurry:

Setting up a Magic Lantern development environment on a Macintosh

Magic Lantern development compiler.app (Mac OS)

PM me when you get back to LA if you want to get together. I live nearby in Redondo Beach.

I keep thinking about getting an M50 for my next ML project--once the M2 is up and running.

Thanks for the links.  I'll check it out.

Also, neat.  Maybe we can meet up and you can play with the M50 a bit.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3503
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #39 on: January 04, 2019, 04:46:27 PM »
Please find here the draft QEMU patches for M50 in their current state.

I had no idea it was this far along. Feel like we were just given the password to get into a speakeasy and now it is time to get drunk.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103

leathc

  • New to the forum
  • *
  • Posts: 12
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #40 on: January 05, 2019, 02:51:04 AM »
I had no idea it was this far along. Feel like we were just given the password to get into a speakeasy and now it is time to get drunk.

I'm buyin' if it means I can get my camera to do all the stuff it should be able to do...

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3503
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #41 on: January 06, 2019, 06:35:08 PM »
Applied the qemu patch and got the M50 running. No GUI emulation yet and this is what the default qemu autoexec.bin looks like but hey, it is a start.

5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103

Walter Schulz

  • Hero Member
  • *****
  • Posts: 6569
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #42 on: January 08, 2019, 04:43:20 PM »
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12037
  • 5D Mark Free
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #43 on: January 08, 2019, 05:30:03 PM »
Nope, M50 looks like this:


srsa

  • New to the forum
  • *
  • Posts: 11
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #44 on: January 08, 2019, 05:45:21 PM »
I know what it is, but not spoiling the game.
@a1ex Can it save files yet?

Walter Schulz

  • Hero Member
  • *****
  • Posts: 6569
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #45 on: January 08, 2019, 06:14:04 PM »
Aspect ratio 4:3! Boy, you got us! And we're totally offtopic. Totally!
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

Greg

  • Hero Member
  • *****
  • Posts: 572
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #46 on: January 08, 2019, 06:15:41 PM »
Fujifilm?

Walter Schulz

  • Hero Member
  • *****
  • Posts: 6569
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #47 on: January 08, 2019, 06:16:40 PM »
Nope, it's a Canon. But no DiGiC 7/8 on board!
Those Twitter members will hunt you down, a1ex!

Disclaimer: If I'm right, of course ...
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 1113
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #48 on: January 08, 2019, 06:25:53 PM »
No twitter either over here:

7D Mark II
EOS M3

70D.112 & 100D.101

Greg

  • Hero Member
  • *****
  • Posts: 572
Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)
« Reply #49 on: January 08, 2019, 06:38:08 PM »
SX50 HS