Author Topic: Porting a Canon firmware update  (Read 4360 times)

dfort

  • Hero Member
  • *****
  • Posts: 1798
Porting a Canon firmware update
« on: April 23, 2017, 08:17:23 AM »
Porting a (minor) Canon firmware update


Pretty much every week someone posts something like, "When is Magic Lantern going to be working with X.X.X version firmware?" The answer is usually, "Downgrade to X.X.X -1." Why doesn't ML work with the latest Canon firmware? There are reasons including:
  • There's nothing in the firmware update that you really need.
  • If ML works on X.X.X why take a chance on breaking it?
  • Developers shouldn't waste their time on trivial updates.
  • Because you need a degree in computer science, know ARM processing and spend hundreds of hours reverse engineering to port Magic Lantern.
However, there are also reasons for porting to the latest Canon firmware:
  • Canon recommends using their latest firmware update.
  • The old firmware that ML needs is no longer on the Canon support site so you'll have to get it from a non-official source.
  • Canon may have actually fixed a bug that affects your camera.
  • Because you can.
Let's put the emphasis on you can meaning anyone reading this article should be able to port ML to a Canon firmware update, though it probably shouldn't the first project you attempt. You don't need to know anything about programming or reverse engineering and quite frankly most of what is required to do is grunt work that the main developers aren't really interested in doing. The first thing you need is to set up a development system in order to compile Magic Lantern. There are several ways to do this and there are plenty of tutorials for compiling ML so we won't go into it here.

Note that this article is about doing a relatively minor firmware update like the type Canon released around October of 2016 to fix a problem that affected a couple of lenses. Major updates that add new features or porting to a new camera isn't covered here.

OK--first step is getting ML working on your camera with the supported firmware. This is necessary in order to get firmware dumps, set the camera boot flag and to set the boot flag on the memory cards that you'll be using. You'll find the ROM dumps in the ML/LOGS directory of the card. Save those files. We'll get back to them soon.

You should install ML on a few cards if for nothing else so you'll have cards with the boot flag set. You can also set the card boot flag using a utility like EOScard for Windows and for Mac try MacBoot. I recommend having various different cards with the boot flag set because soon we'll be running something that might not work on certain cards or might corrupt the file system. @a1ex gives a good explanation of what is happening and how to work around it in Reply #1 below. This explains why my trusty 2GB Patriot SD card works like a champ while the much higher performing SanDisk Extreme PRO cards I tried usually failed.

Keep one card without the boot flag in order to do the actual Canon firmware update. Copy both the Canon firmware that is currently working with ML and the version you are upgrading to. That way with one card you can upgrade and downgrade the Canon firmware. If you have several different camera models you can keep all the firmware versions on that one card because the camera will pick out only updates that are valid for that model.

Note that on the 5D3 you can't downgrade from 1.3.X in camera, you need to use EOS Utility. However, what EOS Utility does is to copy the firmware to the card and the camera takes over from there. This process is to discourage users from downgrading. There seems to be a good reason for this because 1.3.X has some added features that might corrupt user settings when downgrading. Simply clearing the Canon settings when downgrading seems to clear up those corruption issues.

Updating the Canon firmware

Here we go--Canon firmware update on card without the boot flag set, check. Update the firmware. Note that this is an article that attempts to cover all ML enabled cameras so I'll be mixing up screenshots from various models.



Dumping the ROM

When you install ML it sets a boot flag on the camera and a boot flag on the card. The camera will always check first to see if the card has the boot flag set and if it is set, attempt to boot off an autoexec.bin file on the card. There are some special autoexec.bin files available on the download page under the heading of ROM dumpers, scroll down the page to find them. You are most like going to need the one labeled Portable. Put it on one of your cards that has the boot flag set and insert the card in your camera. It will probably start working without even turn on the power. You should see something like this:



Or this:



Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. Once again, check out the comment on Reply #1 if you are having trouble getting it to work. The point of this is to get a ROM dump of the new firmware.

If you want to be meticulous about it you should run an MD5 check on your file. When I did it both attempts had the same MD5 value and the dumps matched each other in size so I was confident that I had a good dump to work with. Lazy me. Open up a terminal (Powershell on Windows), navigate to where your ROM dumps and MD5 files are saved and:

Linux

For Linux and probably Cygwin on Windows:
Code: [Select]
md5sum -c *.BIN
Windows

For Windows 8 or higher (thanks Walter):
Code: [Select]
powershell "get-filehash *.BIN -A MD5 | format-list"
Macintosh

A stock Macintosh doesn't come with md5sum installed so you can either create the md5 checksum and compare it with the one generated by the portable rom dumper:
Code: [Select]
md5 ROM1A.BIN
MD5 (ROM1A.BIN) = 8a4d0fbfa6e6759d41b833bf764748fb
cat ROM1A.MD5
8a4d0fbfa6e6759d41b833bf764748fb  ROM1A.BIN
Or you can install md5sum. I use Homebrew:
Code: [Select]
brew install md5sha1sumThis version of md5sum couldn't check all of the dumps at once:
Code: [Select]
md5sum -c *.MD5
Error: --check <filename> cannot be used with additional files
So you'll have to do them one at a time:
Code: [Select]
md5sum -c ROM1A.MD5
ROM1A.BIN: OK

Now you've got several sets of verified ROM dumps, make sure you know which one is which. In most cases you'll only need to work with the ROM1.BIN for the Canon firmware that you started from and the one for the Canon firmware you're porting to.

Congratulations, you just completed the most essential step towards porting Magic Lantern!

If you're in a hurry, curious, stupid or all of the above you'll be tempted to try your current version of ML on the new firmware. Guess which description I fit into?



Disassembly

There are various methods of disassembling the ROM dumps in order to work with them. Everyone seems to have their favorite method. ARMu is quick but only works on Windows--ok it also works on Mac under wine but I haven't figured it out yet. ARM-console is very difficult to set up and use but it is very powerful--this currently my favorite disassembler. There is also a commercial product called IDA but in order to disassemble ARM code you need to buy a license. By far the easiest for beginners is disassemble.pl which is a perl script that needs just a few tweaks to get running properly as explained in this excellent tutorial on finding stubs.

What about QEMU? That could be very useful, especially when porting major firmware updates and new cameras but we're doing just a minor firmware update that can be done with just a few basic tools and a lot of perseverance.

Now it might be tempting to upload and share these ROM dumps and disassemblies but DON'T DO IT! You're holding copyrighted material that is owned by a very large and powerful corporation. You bought the camera so you can look at what is inside it but you aren't supposed to publish "intellectual property" in an open forum like this one. I'll be editing the bits in this post so they don't perfectly match the real disassembly but it should be good enough for instructional purposes.

So let's do this. Assuming disassemble.pl is in the same directory as your ROM1A.BIN file:
Code: [Select]
perl disassemble.pl 0xFF000000 ROM1A.BIN
string dump
create elf file
label scan
0xffff5a4c 
disassemble and string lookup
0xfffffffc 
job complete!

The file you are interested in is ROM1A.BIN.dis. You'll be spending a lot of time together.

Preparing your local repository for the update.

There are lots of ways to work with your local repository. It really depends how you are most comfortable. If you plan to eventually do a pull request you should consider creating a "scratch" or development branch where you can experiment and post all your notes and then when everything is working, copy over your changes to a fresh branch to avoid cluttering up the revision history with trivial commits.

Take a look at some recent firmware updates so you'll know what you are getting yourself into:

EOS M 2.0.2 - 2.0.3
700D 1.1.4 - 1.1.5

This one is somewhat different because it doesn't replace any previous firmware version and is a bit more complex:

5D3 1.3.4

The first thing you'll notice is that it seems that we're throwing out the old and creating new files. That's not really the case, we're renaming the old files but don't do it with the operating system because it defeats the point of keeping everything under version control. Generally I like to work with the SourceTree application but copying directories is one of those things that is probably best done on the command line. There are two directories and one file that needs to be renamed. For example, when going from 1.1.4 to 1.1.5 on the 700D:

The platform directory
Code: [Select]
cd magic-lantern/platform
hg rename 700D.114 700D.115

The installer directory
Code: [Select]
cd magic-lantern/installer
hg rename 700D.114 700D.115

Editing the Source Code

Now for the good news. This step has registered a lot of changes but you'll only need to modify a few of these files. Most of the changes will be happening in the stubs.S file. While it is challenging looking for new stubs or finding stubs for a new camera model, a minor firmware update is rather easy.

Let's start with a very easy change. In magic-lantern/installer/Makefile you'll see this line:

Code: [Select]
SUPPORTED_INSTALLERS := 1100D.105 500D.111 50D.109 550D.109 5D2.212 5D3.113 60D.111 600D.102 650D.104 6D.116 700D.114 EOSM.202 7D.203

Pretty easy to figure out what to change on that line. However, there's something to watch out for when editing a Makefile, you got to make sure that your text editor doesn't change the indents from tabs to spaces. A Makefile needs those tabs.

Now check the pull requests on bitbucket and you'll see several other places where you need to change from the old to the new firmware version. Don't forget to make changes in the modules: adtg_gui, dual_iso, mlv_rec and mlv_lite.

Another rather easy change to make is to remove the old ML-SETUP.FIR file. It won't work with the new firmware and a developer will have to create a new one for you. In the case of a minor firmware update if you did the Canon update with the boot flag set on the camera, you won't have any problems later when you test out your new port. Just don't rush it! There are a few things that need to be done in the right order before attempting to boot into your newly ported ML.

In fact if you want to be extra careful you could take the advice that is in property.c and disable the prop_request_change function. If you're a suspenders and belt kind of guy also go into your platform's internals.h and comment out '#define CONFIG_PROP_REQUEST_CHANGE'. This is highly recommended for new ports because you can do some serious damage. I didn't take that precaution because mine were very minor updates.

Finding where the stubs moved

Now we get into the nitty gritty of finding stubs. We aren't porting a new camera, just a minor firmware update so all the stubs that you should be able to easily look up in the old disassembled ROM1.BIN disassembly should also be in the new disassembly--only in a different location.

If you're working with a DIGIC V the first thing you will notice in the stubs.S file is something like this (700D.114):

Code: [Select]
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...

There is a very good explanation of what this RAM_OFFSET is all about in a1ex's Tutorial: finding stubs. In this case Nanomad left a clue so let's check out what line 0xff0c0098 says in the 700D.114 disassembly. Depending which disassembler you used it might look like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; pointer to sub_FFA5E8B0
ff0c009c: e59f1048 ldr r1, [pc, #72] ; 0xff0c00ec: pointer to 0x1900

or like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b0)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Now let's look at what the 700D.115 disassembly looks like:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b8)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Hooray! RAM_OFFSET changed from 0xFFA5E8B0-0x1900 to 0xFFA5E8B8-0x1900.

Now start going through the rest of the stubs.S file. One trick I discovered is that if you're tackling one of those minor updates Canon pushed in October of 2016, any stub that doesn't use RAM_OFFSET remains unchanged and those with the offset move the same amount in the same direction that RAM_OFFSET changed--for the most part. A few will move. Why? Because there was a few lines of code that were changed somewhere that shifted things around slightly. In any case you should check the address to every stub to make sure. In some cases there's a descriptive string around the stub you're checking into, other times there is nothing to go by except if several lines "look" the same. It becomes sort of a pattern matching game like those cartoons that you try to find the difference between two drawings that look almost identical.

What about those stubs that aren't addresses?

Code: [Select]
/** Task info **/
NSTUB(   0x25024,  task_max)
NSTUB(0xFFA0A0D8 - RAM_OFFSET,  is_taskid_valid)            // AJ_task_trampoline_related_p10
NSTUB(   0x23E14,  current_task)
NSTUB(     0x674,  current_interrupt)                       // in interrupt handler (0x18), where MEM(C0201004) is stored

You can take a chance that it didn't change but it is best to look at the hints, search the forum, read the wiki or post a question how to find these values. Leave those challenging ones until the end because by the time you've gone through all of the other stubs you'll become more of an expert at sleuthing out those stubs.

There are some helpful scripts in the magic-lantern/contrib directory. Try running check-stubs.py to get a report on the differences between the old and new stubs.S files. I posted my results on the 700D.115 pull request.

Make sure you got all of the changed addresses

Not all of the addresses you need to check are in the stubs.S file. The source code for some of the modules also have references to firmware versions and/or addresses that need checking and probably changing. For example, adtg_gui.c, dual_iso.c, mlv_rec.c and mlv_lite.c. By now your pattern matching skills are probably pretty good so good luck hunting them down.

Getting the firmware signature

The next step involves something a little more exciting that pouring over endless lines of disassembly code. You'll need to find the new firmware's signature and for that you'll need to build Magic Lantern, but this isn't a working build--not yet.

Take a look at fw-signature.h. Your updated firmware version is probably not in the list. Now check out boot-hack.c and you'll find this near the beginning of the file:

Code: [Select]
#if defined(CONFIG_HELLO_WORLD)
#include "fw-signature.h"
#endif

That's right, we're going to welcome your firmware update to the world. In order to do that open up config-defines.h and look for this piece of code:

Code: [Select]
/**
 * Enable these for early ports
 */

    /** If CONFIG_EARLY_PORT is defined, only a few things will be enabled (e.g. changing version string) */
    //~ #define CONFIG_EARLY_PORT

    /** Load fonts and print Hello World (disable CONFIG_EARLY_PORT); will not start any other ML tasks, handlers etc. */
    //~ #define CONFIG_HELLO_WORLD
   
    /** Create a developer FIR for enabling the bootflag and dumping the ROM. */
    //~ #define CONFIG_DUMPER_BOOTFLAG

Uncomment the hello world line:

Code: [Select]
#define CONFIG_HELLO_WORLD
Now build Magic Lantern, put it on one of your boot flag enabled cards, insert it into your camera and you should see something like this:



Write it down, put that address in fw-signature.h, comment out '#define CONFIG_HELLO_WORLD' and...

Trying it out

Ok so it should work but something will probably go wrong the first time you try it out. On the 700D.115 some of the fonts were missing. Searching for fonts I found this:

consts.h
Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xFFCF67E4
#define BFNT_BITMAP_OFFSET 0xFFCF972C
#define BFNT_BITMAP_DATA   0xFFCFC674

So it took a bit more effort to find those addresses but magic-lantern/contrib/indy/find_fnt.py found them automatically.

On the 5D3.134 there were some other things that needed attention in consts.h. You would think that constants by definition don't change but like the old saying goes, "the only thing constant is change."

Oh, one last thing--the ML-SETUP.FIR file. Usually that's one of the first things that you need if you're working on a new port but since we're only doing a minor firmware update we can wait until the end. Only one of the main developers can prepare this file for you so you'll have to ask.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Porting a Canon firmware update
« Reply #1 on: April 23, 2017, 11:15:34 AM »
Very nice write-up (and very easy to follow!)

Some minor correction:

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. The point of this is to get a ROM dump of the new firmware. If you want to be meticulous about it you should run an MD5 check on your file.

Here, checking the MD5 is not about being meticulous; the bootloader file I/O routines *are* really stubborn. They may corrupt the filesystem, insert a few or a lot of wrong bytes, lock up the camera, refuse to run on any or all of your cards and so on.

So far, what worked every single time was formatting the card at a much lower capacity (I've used 256MB successfully in many cases). To do this, you can write the SD image that comes with QEMU to your SD or CF card. This guide is helpful; just don't forget to unzip the SD image. This image contains the portable display test and is bootable (so, you can test it in the camera straight away).

Even in this case, checking the MD5 is mandatory. On Linux, this is as simple as:
Code: [Select]
md5sum -c *.MD5

Walter Schulz

  • Hero Member
  • *****
  • Posts: 5643
Re: Porting a Canon firmware update
« Reply #2 on: April 23, 2017, 01:22:26 PM »
Even in this case, checking the MD5 is mandatory. On Linux, this is as simple as:
Code: [Select]
md5sum -c *.MD5

Windows 8 (or higher):
Code: [Select]
powershell get-filehash *.MD5 -A MD5 | format-list

dfort

  • Hero Member
  • *****
  • Posts: 1798
Re: Porting a Canon firmware update
« Reply #3 on: May 13, 2017, 02:36:35 AM »
Ok--finished the tutorial. Next time someone asks when ML will work with the x.x.x firmware update, point them here.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

dfort

  • Hero Member
  • *****
  • Posts: 1798
Re: Porting a Canon firmware update
« Reply #4 on: May 13, 2017, 05:54:25 PM »
Found this list of the recent Canon firmware updates that were released in October 2016. I added which version is currently supported in order to help you decide if you would like to attempt to port a simple firmware update. Those in red bold are the best candidates for this project. (Note that the 7D is behind a couple of updates and might be a bit more challenging.)

EOS 5D Mark III, firmware 1.3.4 -> pending pull request
EOS 6D, firmware 1.1.7 -> 1.1.6
EOS 7D, firmware 2.0.6 -> 2.0.3
EOS 70D, firmware 1.1.2 -> port in progress pending pull request
EOS 60D, firmware 1.1.2 -> 1.1.1
EOS 700D, firmware 1.1.5 -> pending pull request
EOS 650D, firmware 1.0.5 -> 1.0.4
EOS 550D, firmware 1.1.0 -> 1.0.9
EOS 500D, firmware 1.1.2 -> 1.1.1
EOS 1300D, firmware 1.0.2 -> early port maybe in progress?
EOS 1200D, firmware 1.0.2 -> 1.0.1 port in progress
EOS 1100D, firmware 1.0.6 -> 1.0.5
EOS 100D, firmware 1.0.1 -> port in progress pending pull request
EOS M, firmware 2.0.3 -> pending pull request
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

ItsMeLenny

  • Hero Member
  • *****
  • Posts: 870
  • 550D
Re: Porting a Canon firmware update
« Reply #5 on: May 14, 2017, 04:14:20 AM »
I should really attempt the 550D at some point.
This comment is also a note so I can refind this thread.

qqluqq

  • New to the forum
  • *
  • Posts: 7
Re: Porting a Canon firmware update
« Reply #6 on: May 15, 2017, 12:46:01 AM »
I should really attempt the 550D at some point.
This comment is also a note so I can refind this thread.

I already started on the 550D checking stubs atm

qqluqq

  • New to the forum
  • *
  • Posts: 7
Re: Porting a Canon firmware update
« Reply #7 on: May 16, 2017, 08:43:23 PM »
followed the guide
found all adresses

now trying to run hello world but screen just stays black when I try it,
any idea on where I should look to fix that?

code here (not working)
https://bitbucket.org/qqluqq/magic-lantern/branch/550D%20110

also, why do you replace old fw version instead of leaving both in?

Walter Schulz

  • Hero Member
  • *****
  • Posts: 5643
Re: Porting a Canon firmware update
« Reply #8 on: May 16, 2017, 08:50:19 PM »
also, why do you replace old fw version instead of leaving both in?

?

qqluqq

  • New to the forum
  • *
  • Posts: 7
Re: Porting a Canon firmware update
« Reply #9 on: May 16, 2017, 08:58:48 PM »
?

He renames the old version instead of copying, shouldn't both the new and old be in the source?

dfort

  • Hero Member
  • *****
  • Posts: 1798
Re: Porting a Canon firmware update
« Reply #10 on: May 16, 2017, 09:03:40 PM »
also, why do you replace old fw version instead of leaving both in?

It is possible to include both Canon firmware versions but it is easier to support one version at a time. The 5D3 is an exception. The developers didn't want to remove 1.1.3 because 1.2.3 and up has some issues with ML. Minor Canon firmware updates like the one you're doing shouldn't introduce any new bugs, at least not in theory.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Porting a Canon firmware update
« Reply #11 on: May 16, 2017, 09:40:23 PM »
@qqluqq: 550D can be debugged in QEMU (much easier than double-checking everything blindly, hoping to find a needle in the haystack).

Refer to this page for installation and for checking what works and what not.

dfort

  • Hero Member
  • *****
  • Posts: 1798
Re: Porting a Canon firmware update
« Reply #12 on: May 23, 2017, 01:38:13 AM »
Thanks Walter -- edited original post.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*