Porting a (minor) Canon firmware update
Pretty much every week someone posts something like, "When is Magic Lantern going to be working with X.X.X version firmware?" The answer is usually, "Downgrade to X.X.X -1." Why doesn't ML work with the latest Canon firmware? There are reasons including:
- There's nothing in the firmware update that you really need.
- If ML works on X.X.X why take a chance on breaking it?
- Developers shouldn't waste their time on trivial updates.
- Because you need a degree in computer science, know ARM processing and spend hundreds of hours reverse engineering to port Magic Lantern.
However, there are also reasons for porting to the latest Canon firmware:
- Canon recommends using their latest firmware update.
- The old firmware that ML needs is no longer on the Canon support site so you'll have to get it from a non-official source.
- Canon may have actually fixed a bug that affects your camera.
- Because you can.
Let's put the emphasis on
you can meaning anyone reading this article should be able to port ML to a Canon firmware update, though it probably shouldn't the first project you attempt. You don't need to know anything about programming or reverse engineering and quite frankly most of what is required to do is grunt work that the main developers aren't really interested in doing. The first thing you need is to set up a development system in order to compile Magic Lantern. There are several ways to do this and there are plenty of tutorials for compiling ML so we won't go into it here.
Note that this article is about doing a relatively minor firmware update like the type Canon released around October of 2016 to fix a problem that affected a couple of lenses. Major updates that add new features or porting to a new camera isn't covered here.
OK--first step is getting ML working on your camera with the supported firmware. This is necessary in order to get firmware dumps, set the camera boot flag and to set the boot flag on the memory cards that you'll be using. You'll find the ROM dumps in the ML/LOGS directory of the card. Save those files. We'll get back to them soon.
You should install ML on a few cards if for nothing else so you'll have cards with the boot flag set. You can also set the card boot flag using a utility like
EOScard for Windows and for Mac try
MacBoot. I recommend having various different cards with the boot flag set because soon we'll be running something that might not work on certain cards or might corrupt the file system. @a1ex gives a good explanation of what is happening and how to work around it in
Reply #1 below. This explains why my trusty 2GB Patriot SD card works like a champ while the much higher performing SanDisk Extreme PRO cards I tried usually failed.
Keep one card without the boot flag in order to do the actual Canon firmware update. Copy both the Canon firmware that is currently working with ML and the version you are upgrading to. That way with one card you can upgrade and downgrade the Canon firmware. If you have several different camera models you can keep all the firmware versions on that one card because the camera will pick out only updates that are valid for that model.
Note that on the 5D3 you can't downgrade from 1.3.X in camera, you need to use EOS Utility. However, what EOS Utility does is to copy the firmware to the card and the camera takes over from there. This process is to discourage users from downgrading. There seems to be a good reason for this because 1.3.X has some added features that might corrupt user settings when downgrading. Simply clearing the Canon settings when downgrading seems to clear up those corruption issues.
Updating the Canon firmwareHere we go--Canon firmware update on card without the boot flag set, check. Update the firmware. Note that this is an article that attempts to cover all ML enabled cameras so I'll be mixing up screenshots from various models.
Dumping the ROMWhen you install ML it sets a boot flag on the camera and a boot flag on the card. The camera will always check first to see if the card has the boot flag set and if it is set, attempt to boot off an autoexec.bin file on the card. There are some special autoexec.bin files available on the download page under the heading of
ROM dumpers, scroll down the page to find them. You are most like going to need the one labeled
Portable. Put it on one of your cards that has the boot flag set and insert the card in your camera. It will probably start working without even turn on the power. You should see something like this:

Or this:

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. Once again, check out the comment on
Reply #1 if you are having trouble getting it to work. The point of this is to get a ROM dump of the new firmware.
If you want to be meticulous about it you should run an MD5 check on your file. When I did it both attempts had the same MD5 value and the dumps matched each other in size so I was confident that I had a good dump to work with. Lazy me. Open up a terminal (Powershell on Windows), navigate to where your ROM dumps and MD5 files are saved and:
LinuxFor Linux and probably Cygwin on Windows:
md5sum -c *.BIN
WindowsFor Windows 8 or higher (thanks Walter):
powershell "get-filehash *.BIN -A MD5 | format-list"
MacintoshA stock Macintosh doesn't come with md5sum installed so you can either create the md5 checksum and compare it with the one generated by the portable rom dumper:
md5 ROM1A.BIN
MD5 (ROM1A.BIN) = 8a4d0fbfa6e6759d41b833bf764748fb
cat ROM1A.MD5
8a4d0fbfa6e6759d41b833bf764748fb ROM1A.BIN
Or you can install md5sum. I use Homebrew:
brew install md5sha1sum
This version of md5sum couldn't check all of the dumps at once:
md5sum -c *.MD5
Error: --check <filename> cannot be used with additional files
So you'll have to do them one at a time:
md5sum -c ROM1A.MD5
ROM1A.BIN: OK
Now you've got several sets of verified ROM dumps, make sure you know which one is which. In most cases you'll only need to work with the ROM1.BIN for the Canon firmware that you started from and the one for the Canon firmware you're porting to.
Congratulations, you just completed the most essential step towards porting Magic Lantern!If you're in a hurry, curious, stupid or all of the above you'll be tempted to try your current version of ML on the new firmware. Guess which description I fit into?
DisassemblyThere are various methods of disassembling the ROM dumps in order to work with them. Everyone seems to have their favorite method.
ARMu is quick but only works on Windows--ok it also works on Mac under wine but I haven't figured it out yet.
ARM-console is very difficult to set up and use but it is very powerful--this currently my favorite disassembler. There is also a commercial product called
IDA but in order to disassemble ARM code you need to buy a license. By far the easiest for beginners is
disassemble.pl which is a perl script that needs just a few tweaks to get running properly as explained in this excellent
tutorial on finding stubs.
What about QEMU? That could be very useful, especially when porting major firmware updates and new cameras but we're doing just a minor firmware update that can be done with just a few basic tools and a lot of perseverance.
Now it might be tempting to upload and share these ROM dumps and disassemblies but
DON'T DO IT! You're holding copyrighted material that is owned by a very large and powerful corporation. You bought the camera so you can look at what is inside it but you aren't supposed to publish "intellectual property" in an open forum like this one. I'll be editing the bits in this post so they don't perfectly match the real disassembly but it should be good enough for instructional purposes.
So let's do this. Assuming disassemble.pl is in the same directory as your ROM1A.BIN file:
perl disassemble.pl 0xFF000000 ROM1A.BIN
string dump
create elf file
label scan
0xffff5a4c
disassemble and string lookup
0xfffffffc
job complete!
The file you are interested in is
ROM1A.BIN.dis. You'll be spending a lot of time together.
Preparing your local repository for the update.There are lots of ways to work with your local repository. It really depends how you are most comfortable. If you plan to eventually do a pull request you should consider creating a "scratch" or development branch where you can experiment and post all your notes and then when everything is working, copy over your changes to a fresh branch to avoid cluttering up the revision history with
trivial commits.
Take a look at some recent firmware updates so you'll know what you are getting yourself into:
EOS M 2.0.2 - 2.0.3700D 1.1.4 - 1.1.5This one is somewhat different because it doesn't replace any previous firmware version and is a bit more complex:
5D3 1.3.4The first thing you'll notice is that it seems that we're throwing out the old and creating new files. That's not really the case, we're renaming the old files but don't do it with the operating system because it defeats the point of keeping everything under version control. Generally I like to work with the
SourceTree application but copying directories is one of those things that is probably best done on the command line. There are two directories and one file that needs to be renamed. For example, when going from 1.1.4 to 1.1.5 on the 700D:
The platform directory
cd magic-lantern/platform
hg rename 700D.114 700D.115
The installer directory
cd magic-lantern/installer
hg rename 700D.114 700D.115
Editing the Source CodeNow for the good news. This step has registered a lot of changes but you'll only need to modify a few of these files. Most of the changes will be happening in the stubs.S file. While it is challenging looking for new stubs or finding stubs for a new camera model, a minor firmware update is rather easy.
Let's start with a very easy change. In magic-lantern/installer/Makefile you'll see this line:
SUPPORTED_INSTALLERS := 1100D.105 500D.111 50D.109 550D.109 5D2.212 5D3.113 60D.111 600D.102 650D.104 6D.116 700D.114 EOSM.202 7D.203
Pretty easy to figure out what to change on that line. However, there's something to watch out for when editing a Makefile, you got to make sure that your text editor doesn't change the indents from tabs to spaces. A Makefile needs those tabs.
Now check the pull requests on bitbucket and you'll see several other places where you need to change from the old to the new firmware version. Don't forget to make changes in the modules: adtg_gui, dual_iso, mlv_rec and mlv_lite.
Another rather easy change to make is to remove the old ML-SETUP.FIR file. It won't work with the new firmware and a developer will have to create a new one for you. In the case of a minor firmware update if you did the Canon update with the boot flag set on the camera, you won't have any problems later when you test out your new port. Just don't rush it! There are a few things that need to be done in the right order before attempting to boot into your newly ported ML.
In fact if you want to be extra careful you could take the advice that is in property.c and disable the prop_request_change function. If you're a suspenders and belt kind of guy also go into your platform's internals.h and comment out '#define CONFIG_PROP_REQUEST_CHANGE'. This is highly recommended for new ports because you can do some serious damage. I didn't take that precaution because mine were very minor updates.
Finding where the stubs movedNow we get into the nitty gritty of finding stubs. We aren't porting a new camera, just a minor firmware update so all the stubs that you should be able to easily look up in the old disassembled ROM1.BIN disassembly should also be in the new disassembly--only in a different location.
If you're working with a DIGIC V the first thing you will notice in the stubs.S file is something like this (700D.114):
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...
There is a very good explanation of what this RAM_OFFSET is all about in a1ex's
Tutorial: finding stubs. In this case Nanomad left a clue so let's check out what line 0xff0c0098 says in the 700D.114 disassembly. Depending which disassembler you used it might look like this:
ff0c0098: e59f0048 ldr r0, [pc, #72] ; pointer to sub_FFA5E8B0
ff0c009c: e59f1048 ldr r1, [pc, #72] ; 0xff0c00ec: pointer to 0x1900
or like this:
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b0)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)
Now let's look at what the 700D.115 disassembly looks like:
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b8)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)
Hooray! RAM_OFFSET changed from 0xFFA5E8B0-0x1900 to 0xFFA5E8B8-0x1900.
Now start going through the rest of the stubs.S file. One trick I discovered is that if you're tackling one of those minor updates Canon pushed in October of 2016, any stub that doesn't use RAM_OFFSET remains unchanged and those with the offset move the same amount in the same direction that RAM_OFFSET changed--for the most part. A few will move. Why? Because there was a few lines of code that were changed somewhere that shifted things around slightly. In any case you should check the address to every stub to make sure. In some cases there's a descriptive string around the stub you're checking into, other times there is nothing to go by except if several lines "look" the same. It becomes sort of a pattern matching game like those cartoons that you try to find the difference between two drawings that look almost identical.
What about those stubs that aren't addresses?
/** Task info **/
NSTUB( 0x25024, task_max)
NSTUB(0xFFA0A0D8 - RAM_OFFSET, is_taskid_valid) // AJ_task_trampoline_related_p10
NSTUB( 0x23E14, current_task)
NSTUB( 0x674, current_interrupt) // in interrupt handler (0x18), where MEM(C0201004) is stored
You can take a chance that it didn't change but it is best to look at the hints, search the forum, read the wiki or post a question how to find these values. Leave those challenging ones until the end because by the time you've gone through all of the other stubs you'll become more of an expert at sleuthing out those stubs.
There are some helpful scripts in the magic-lantern/contrib directory. Try running check-stubs.py to get a report on the differences between the old and new stubs.S files. I posted my results on the
700D.115 pull request.
Make sure you got all of the changed addressesNot all of the addresses you need to check are in the stubs.S file. The source code for some of the modules also have references to firmware versions and/or addresses that need checking and probably changing. For example, adtg_gui.c, dual_iso.c, mlv_rec.c and mlv_lite.c. By now your pattern matching skills are probably pretty good so good luck hunting them down.
Getting the firmware signatureThe next step involves something a little more exciting that pouring over endless lines of disassembly code. You'll need to find the new firmware's signature and for that you'll need to build Magic Lantern, but this isn't a working build--not yet.
Take a look at fw-signature.h. Your updated firmware version is probably not in the list. Now check out boot-hack.c and you'll find this near the beginning of the file:
#if defined(CONFIG_HELLO_WORLD)
#include "fw-signature.h"
#endif
That's right, we're going to welcome your firmware update to the world. In order to do that open up config-defines.h and look for this piece of code:
/**
* Enable these for early ports
*/
/** If CONFIG_EARLY_PORT is defined, only a few things will be enabled (e.g. changing version string) */
//~ #define CONFIG_EARLY_PORT
/** Load fonts and print Hello World (disable CONFIG_EARLY_PORT); will not start any other ML tasks, handlers etc. */
//~ #define CONFIG_HELLO_WORLD
/** Create a developer FIR for enabling the bootflag and dumping the ROM. */
//~ #define CONFIG_DUMPER_BOOTFLAG
Uncomment the hello world line:
#define CONFIG_HELLO_WORLD
Now build Magic Lantern, put it on one of your boot flag enabled cards, insert it into your camera and you should see something like this:

Write it down, put that address in fw-signature.h, comment out '#define CONFIG_HELLO_WORLD' and...
Trying it outOk so it should work but something will probably go wrong the first time you try it out. On the 700D.115 some of the fonts were missing. Searching for fonts I found this:
consts.h
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES 0xFFCF67E4
#define BFNT_BITMAP_OFFSET 0xFFCF972C
#define BFNT_BITMAP_DATA 0xFFCFC674
So it took a bit more effort to find those addresses but magic-lantern/contrib/indy/find_fnt.py found them automatically.
On the 5D3.134 there were some other things that needed attention in consts.h. You would think that constants by definition don't change but like the old saying goes, "the only thing constant is change."
Oh, one last thing--the ML-SETUP.FIR file. Usually that's one of the first things that you need if you're working on a new port but since we're only doing a minor firmware update we can wait until the end. Only one of the main developers can prepare this file for you so you'll have to ask.
Here are the firmware updates I've been tracking:
EOS 5D Mark III, firmware 1.2.3 -> 1.3.4
pull requestEOS 5D Mark III, firmware 1.2.3 -> 1.3.5
in progressEOS 6D, firmware 1.1.6 -> 1.1.8
pull request See this post.EOS 7D, firmware 2.0.3 -> 2.0.6
pull request Still needs some work.
EOS 60D, firmware 1.1.1 -> 1.1.2
pull request See this post Should be good to merge.EOS 500D, firmware 1.1.1 -> 1.1.2
pull request See this postEOS 550D, firmware 1.0.9 -> 1.1.0
pull request See this post. Some issues noted.
EOS 600D, firmware 1.0.2 -> 1.0.3
pull request See this post. Should be good to merge.EOS 650D, firmware 1.0.4 -> 1.0.5
pull request See this post. Should be good to merge.EOS 1100D, firmware 1.0.5 -> 1.0.6
pull request See this post.EOS M, firmware 2.0.2 -> 2.0.3
unified branch pull request,
crop_rec_4k branch pull request Should be good to merge.EOS M50, firmware 1.0.1 -> 1.0.2
digic6-dumper branch pull request,
Still an early port but it should be good to merge.