Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - dfort

Pages: [1] 2
General Help Q&A / SourceTree for Non-Developers
« on: July 03, 2018, 01:35:08 AM »
Part of the "fun" of being involved with open source projects like Magic Lantern is seeing how it keeps changing. There's a lot more that is going on than just what is on the download pages. You can get a lot more out of ML if you set up a development environment but even if you're a new user with no coding experience, or no interest in learning how to code, you can follow along with what is going on by installing a free app made by the same company that runs -- that's the site where the ML repository lives.

Download and install the SourceTree app which is available for either Mac or Windows. If you're on Linux you're probably more of a power user though there are other graphical front ends to the Mercurial source control management tool that runs under Bitbucket and SourceTree. You can try TortoiseHg if you want something similar to SourceTree in Linux.

To set up Magic Lantern in SourceTree simply select, Clone from URL:

Then take this part of the URL from the Magic Lantern Bitbucket repository:

You can copy and paste from here:

Code: [Select]
This will create a directory called "magic-lantern" in your home directory or anywhere else you want to save the Magic Lantern source code.

Even if you aren't interested in reading the source code there are lots of interesting bits of documentation in the files. You can also switch around the various branches to see what the latest changes were. The best part is that SourceTree will let you know when a new commit has been uploaded so you can read the latest commit messages and update your local repository.

I won't get into how to fork or compile or anything like that in this topic. This is just for users who want want to take their lurking to a higher level than simply reading what's on the forum.

General Development Discussion / lua test_lens_focus()
« on: April 30, 2018, 02:02:17 AM »
What started out with a simple demonstration of how long it takes various lenses to complete the lua lens focus tests turned into something much bigger that probably needed a new topic so here it is.

[EDIT] All these tests were run on the lua_fix branch, not a unified (nightly) build so this is using the newer code base.

I thought it might be interesting to run through all my lenses and post how long it takes to complete the lens test. Since I've also got some EF-M lenses and wanted to run everything on the same body I put on the EF-M to EF adapter along with a plug in power supply on one of my EOSMs and started testing. First issue I came up with was that the Canon default settings puts the camera into Continuous AF mode. This needs to be disabled or the camera and the script will be playing tug of war with the focus.

The next issue I came up with was that some of my lenses didn't work on the EOSM. At first I thought it was because it was because I was using a third party EF-M to EF adapter or because I was using the 2.0.2 firmware and one of the lenses I was testing got a fix with the 2.0.3 update but it is more likely that neither of these were part of the problem. Why did I come up with that conclusion? Because none of the EF-M lenses could pass the lua focus tests.

The lens test starts with a prompt to put the lens in AF mode and to do a half-shutter press. The bulk of the testing happens after the autofocus confirmation so make sure you've got something you can focus on before starting the test.

LensEOSM Time700D Time
Canon EF-S 10-18mm f/4.5-5.6 IS STMNG0:45
Canon EF-S 18-135mm f/3.5-5.6 IS USM   NG1:05
Canon EF-S 17-55mm f/2.8 IS USM   0:501:00
Canon EF 28-105mm f/3.5-4.5 II USM1:001:10
Canon EF 50mm f/1.8 STM3:224:30
Canon EF-M 22mm f/2 STMNGn/a
Canon EF-M 18-55mm f/3.5-5.6 IS STMNGn/a
Canon EF-M 11-22mm f/4.5-5.6 IS STMNGn/a

The reason I started this was because we were testing the 1200D and had problems getting through the focus tests. I have experienced some lenses taking much longer than others to get through the tests but wanted to provide some more specifics so off came the lid on another can of worms.

By the way, while running through these tests sometimes the test wouldn't run the first time. Maybe I didn't hit the half-shutter at the right time or had the camera pointing at something that it couldn't auto focus on, I'm not sure. The lesson here was that if it doesn't run successfully the first time, try it a few more times before reporting that something is broken.

Curious what a successful lua focus test log looks like? This is a run of only the lens focus test.

Code: [Select]
ML/SCRIPTS/API_TEST.LUA - 2018-4-29 15:44:22

Module tests...

Testing lens focus functionality...
Autofocus outside LiveView...
Focus distance: 1300
Autofocus in LiveView...
Please trigger autofocus (half-shutter / AF-ON / * ).
Autofocus triggered.
Autofocus completed.
Focus distance: 1450
Focusing backward...
Focus distance: 655350
Focus motor position: 0
Focusing forward with step size 3, wait=true...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 3, wait=true...
Focus distance: 655350
Focus motor position: 0
Focus range: 5 steps forward, 4 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.
Focusing forward with step size 3, wait=false...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 3, wait=false...
Focus distance: 655350
Focus motor position: 0
Focus range: 8 steps forward, 9 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.
Focusing forward with step size 2, wait=true...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 2, wait=true...
Focus distance: 655350
Focus motor position: 0
Focus range: 25 steps forward, 24 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.
Focusing forward with step size 2, wait=false...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 2, wait=false...
Focus distance: 655350
Focus motor position: 0
Focus range: 41 steps forward, 41 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.
Focusing forward with step size 1, wait=true...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 1, wait=true...
Focus distance: 655350
Focus motor position: 0
Focus range: 95 steps forward, 94 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.
Focusing forward with step size 1, wait=false...
Focus distance: 340
Focus motor position: 0
Focusing backward with step size 1, wait=false...
Focus distance: 655350
Focus motor position: 0
Focus range: 109 steps forward, 115 steps backward.
Motor steps: 0 forward, 0 backward, 0 lost.

Focus test completed.


An unsuccessful run the lens doesn't move and the log doesn't give much indication of what went wrong:

Code: [Select]
ML/SCRIPTS/API_TEST.LUA - 2018-4-29 15:38:31

Module tests...

Testing lens focus functionality...
Focus distance: 160
Autofocus in LiveView...
Please trigger autofocus (half-shutter / AF-ON / * ).
19...18...17...Autofocus triggered.
Autofocus completed.
Focus distance: 1940
Focusing backward...

Edit: the EOS M50 appears to run EOS firmware (other recent models, i.e. M3, M5, M6, M10 and M100, are based on PowerShot firmware). Looking for a volunteer to try the LED blinking test on this camera, too :)


General Development Discussion / EOSM Shutter-Bug reboot
« on: March 11, 2018, 05:04:43 AM »
EOS-M Shutter-Bug

One of the peskiest and long lived Magic Lantern bug is the EOSM shutter-bug. The first report of this bug that I could find was made in 2012. It has been debated to death in the EOS M Alpha shutter-bug discussion and Issue #1893, EOS-M shutter bug on Bitbucket.

Quick workaround - use the one SD card that has been verified to work consistently, the SanDisk 32GB Extreme PRO SDHC UHS-I. A smaller SanDisk Extreme PRO card is ok but don't use anything larger because pretty much all SDXC cards show the shutter-bug.

Better solution, let's try to find a way to squash this bug.

First of all, how to reproduce this bug.

Lens - Only EF-M zoom lenses, both Canon and third party like the Tamron 18-200mm exhibit the shutter-bug. It is probably not the zoom but more likely when the boot process is reading the image stabilization system that is triggering the bug. Maybe, we don't really know. Lenses that have the collapsing feature also exhibit the shutter-bug if the camera is started with the lens extended.

SD Card - Many cards exhibit the bug but there are a few surprises. For example I've got an old 1GB Kodak card that is super slow but works fine. Anything over 32GB seems to show the bug.

Photo Mode - The shutter bug prevents you from taking a still photo. It doesn't affect the silent still module, only regular JPEG and CR2 shots that need a full shutter press. It also has nothing to do with movie recording.

Starting the camera with the Play button makes no difference. Even holding down the INFO button which on the EOSM prevents ML from loading will show the shutter-bug. However, interrupting the startup process by turning the camera on/off/on curing startup will kill the bug if you do it just right. This puts the camera in some sort of a special "StartupCondition" - This is what we know about StartupCondition so far:

StartupCondition : 1, 0 = camera powered on normally, with the ON/OFF button
StartupCondition : 16, 0 = camera powered on via the Play button
StartupCondition : 4, 0 = on/off/on trick that somehow kills the shutter-bug

Another way to kill the shutter-bug is by either twisting the lens on and off in order to break the lens/body connection or if the lens has the collapsing feature simply collapsing and extending the lens also works.

Experienced EOSM ML users have learned to live with the shutter-bug but it continues to surprise new users. Developers have tried multiple times to figure this out but now we have some more powerful tools that we can use.

I'm not sure if this time we'll be able to do it. This will be more of a test of what we've learned over the past several years and what the new tools reveal. Here's a suggestion where to start:

If the issue can be reproduced outside LiveView - for example, with a call("Release") somewhere at startup - that's a good candidate for trying in QEMU. I have some local changes that emulate the photo capture process to some extent - at least it's making the screen black and starts capturing the image. That should be enough for diagnosing the issue, as my understanding is that, when this bug happens, the camera doesn't even start capturing the image (maybe it also locks up?)

We'll need two logs with the above scenario, with the camera starting outside LiveView and trying to capture an image. MPU messages should be enabled (they are by default) - that's the "driving force" behind photo capture (and that's what my local changes are all about - MPU messages from photo capture logs, assigned to different hotkeys). One log should be successful and the other should have the bug.

If there is some workaround (maybe removing the lens and putting it back?), that's also worth logging.

First test - using the io_trace branch I ran a startup log holding down both the Play button so it would start outside of LiveView and the INFO button so it wouldn't load ML. I ran this on both a card that is known to always show the shutter-bug and a SanDisk 32GB Extreme PRO SDHC UHS-I card. Then I looked up some addresses for lens properties and found something interesting:

Code: [Select]
#define PROP_LENS               0x80030011 // info about lens? flags?
card with shutter-bug log
Code: [Select]
card without shutter-bug log
Code: [Select]
Don't know if this is significant. I was careful not to change anything between the tests and the logs are very similar. So if this register holds some lens information or flags, it looks like there is something going on here. Perhaps figuring out how to re-read the lens information/flags at the end of the boot process might be a way to eliminate the shutter-bug?

Share Your Videos / Lunar Eclipse
« on: February 09, 2018, 06:26:12 AM »
We've been having discussion about various timelapse workflows on the 100D topic so I thought I'd share this timelapse I did a few days ago during the lunar eclipse on January 31, A.K.A. Super Blue Blood Moon.

Shot on a 700D in Movie Crop Mode at 2fps with a 135mm Rokinon lens and a Nikon 2x tele extender. Yeah, it was all hacked together.

It looks like a still with a motion effect but it was a timelapse, really! I could actually see it move through the frame as I was setting up the shot.

A while later it started getting light. What looks like a 3/4 moon is actually the full moon with the earth's shadow. Nothing fancy on this one, just a still CR2 with the 135mm, no tele extender.

Camera Emergency Department / EOSM hit with the null pointer bug
« on: January 30, 2018, 05:35:56 PM »
All four of my EOSM's are currently showing the null pointer warning. Apparently I started doing some testing this morning with the one body that wasn't affected and now it is infected. A couple of the bodies were using firmware 2.0.3 so I downgraded to 2.0.2 and it turned out that they also have the null pointer bug.

The story of how it happened is on this post:

Maybe I'm making a mountain out of a mole hill but a few cameras have the same address for two different stubs, SoundDevShutDownIn and StopASIFDMAADC.

I made a list to illustrate the problem--and I do think this is a problem.

Cameras that don't have either the SoundDevShutDownIn or StopASIFDMAADC stub


Cameras that have both the SoundDevShutDownIn and StopASIFDMAADC stubs

NSTUB(0xFF064304,  SoundDevShutDownIn)
NSTUB(0xFF061C08,  StopASIFDMAADC)                          // Called by SoundDevStopIn

NSTUB(0xFF10B0D8,  SoundDevShutDownIn)
NSTUB(0xFF1088E0,  StopASIFDMAADC)  <-- Not Listed in stubs.S

NSTUB(0xFF10B930,  SoundDevShutDownIn)
NSTUB(0xFF109138,  StopASIFDMAADC)                          /* present on 7D.203, 6D.113, EOSM.202 */

Cameras that use the same address for both the SoundDevShutDownIn and StopASIFDMAADC stubs

NSTUB(0xFF11C874,  SoundDevShutDownIn)

NSTUB(0xFF10A920,  SoundDevShutDownIn)                      // Temporarily using address for StopASIFDMAADC to resolve MLV_SND issue
NSTUB(0xFF10A920,  StopASIFDMAADC)                          // Regular -- Stop ASIF ADC - needed for future changes to mlv_snd.c

NSTUB(0xFF112680,  SoundDevShutDownIn)

NSTUB(0xFF11758C,  SoundDevShutDownIn)

Differentiating between these two stubs becomes important in this bit of code in the mlv_snd_stop() function:

Code: [Select]
    /* some models may need this */

Now those platforms that have duplicate addresses are using only StopASIFDMAADC, not the SoundDevShutDownIn. So what happens if we simply do this:

Code: [Select]
/* of course we need to initialize it */
extern WEAK_FUNC(ret_0) int StopASIFDMAADC();
    /* some models may need this */

That way we could put the real SoundDevShutDownIn addresses in the stubs.S for the 6D, EOSM, 100D and 70D. The question is, what will that do to the 7D, 650D and 700D? Also, where else is SoundDevShutDownIn used?

Code: [Select]
#if defined(CONFIG_7D) || defined(CONFIG_6D) || defined(CONFIG_70D) || defined(CONFIG_EOSM)
    /* experimental for 7D now, has to be made generic */
/* Disable Audio */
    void SoundDevShutDownIn();

It is also used in bitrate-6d.c but that section of code is commented out so it doesn't apply.

Seems that now that experimental builds are being compiled and published for anyone willing to test, maybe the crop_rec_4k branch would be a good place to try out a fix? That branch includes the 100D and 70D so it is probably the best place to run an experiment to put the right SoundDevShutDownIn address in the 6D, EOSM, 100D and 70D, modify mlv_snd.c and beep.c to use StopASIFDMAADC and ask testers to report back.

Why fix it if it is working? Because duplicate stub addresses is a rather ugly hack.

Here are a few links to previous discussions of this issue:

And here's a bug report that is affected if you use the "right" SoundDevShutDownIn address in the current mlv_snd code:

By the way, for those of you that didn't understand my subject line, it was taken from a popular TV show named To Tell The Truth. The show first aired in the 1950's where a panel of four celebrities had to correctly identify the contestant (usually with an unusual occupation or experience) from the two imposters.

Then again maybe I chose the imposter and the real SoundDevShutDownIn really is StopASIFDMAADC? That stub is used only once:

Code: [Select]
#ifdef CONFIG_6D
void StopASIFDMAADC();
StopASIFDMAADC(asif_rec_stop_cbr, 0);

General Development Discussion / Porting a Canon firmware update
« on: April 23, 2017, 08:17:23 AM »
Porting a (minor) Canon firmware update

Pretty much every week someone posts something like, "When is Magic Lantern going to be working with X.X.X version firmware?" The answer is usually, "Downgrade to X.X.X -1." Why doesn't ML work with the latest Canon firmware? There are reasons including:
  • There's nothing in the firmware update that you really need.
  • If ML works on X.X.X why take a chance on breaking it?
  • Developers shouldn't waste their time on trivial updates.
  • Because you need a degree in computer science, know ARM processing and spend hundreds of hours reverse engineering to port Magic Lantern.
However, there are also reasons for porting to the latest Canon firmware:
  • Canon recommends using their latest firmware update.
  • The old firmware that ML needs is no longer on the Canon support site so you'll have to get it from a non-official source.
  • Canon may have actually fixed a bug that affects your camera.
  • Because you can.
Let's put the emphasis on you can meaning anyone reading this article should be able to port ML to a Canon firmware update, though it probably shouldn't the first project you attempt. You don't need to know anything about programming or reverse engineering and quite frankly most of what is required to do is grunt work that the main developers aren't really interested in doing. The first thing you need is to set up a development system in order to compile Magic Lantern. There are several ways to do this and there are plenty of tutorials for compiling ML so we won't go into it here.

Note that this article is about doing a relatively minor firmware update like the type Canon released around October of 2016 to fix a problem that affected a couple of lenses. Major updates that add new features or porting to a new camera isn't covered here.

OK--first step is getting ML working on your camera with the supported firmware. This is necessary in order to get firmware dumps, set the camera boot flag and to set the boot flag on the memory cards that you'll be using. You'll find the ROM dumps in the ML/LOGS directory of the card. Save those files. We'll get back to them soon.

You should install ML on a few cards if for nothing else so you'll have cards with the boot flag set. You can also set the card boot flag using a utility like EOScard for Windows and for Mac try MacBoot. I recommend having various different cards with the boot flag set because soon we'll be running something that might not work on certain cards or might corrupt the file system. @a1ex gives a good explanation of what is happening and how to work around it in Reply #1 below. This explains why my trusty 2GB Patriot SD card works like a champ while the much higher performing SanDisk Extreme PRO cards I tried usually failed.

Keep one card without the boot flag in order to do the actual Canon firmware update. Copy both the Canon firmware that is currently working with ML and the version you are upgrading to. That way with one card you can upgrade and downgrade the Canon firmware. If you have several different camera models you can keep all the firmware versions on that one card because the camera will pick out only updates that are valid for that model.

Note that on the 5D3 you can't downgrade from 1.3.X in camera, you need to use EOS Utility. However, what EOS Utility does is to copy the firmware to the card and the camera takes over from there. This process is to discourage users from downgrading. There seems to be a good reason for this because 1.3.X has some added features that might corrupt user settings when downgrading. Simply clearing the Canon settings when downgrading seems to clear up those corruption issues.

Updating the Canon firmware

Here we go--Canon firmware update on card without the boot flag set, check. Update the firmware. Note that this is an article that attempts to cover all ML enabled cameras so I'll be mixing up screenshots from various models.

Dumping the ROM

When you install ML it sets a boot flag on the camera and a boot flag on the card. The camera will always check first to see if the card has the boot flag set and if it is set, attempt to boot off an autoexec.bin file on the card. There are some special autoexec.bin files available on the download page under the heading of ROM dumpers, scroll down the page to find them. You are most like going to need the one labeled Portable. Put it on one of your cards that has the boot flag set and insert the card in your camera. It will probably start working without even turn on the power. You should see something like this:

Or this:

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. Once again, check out the comment on Reply #1 if you are having trouble getting it to work. The point of this is to get a ROM dump of the new firmware.

If you want to be meticulous about it you should run an MD5 check on your file. When I did it both attempts had the same MD5 value and the dumps matched each other in size so I was confident that I had a good dump to work with. Lazy me. Open up a terminal (Powershell on Windows), navigate to where your ROM dumps and MD5 files are saved and:


For Linux and probably Cygwin on Windows:
Code: [Select]
md5sum -c *.BIN

For Windows 8 or higher (thanks Walter):
Code: [Select]
powershell "get-filehash *.BIN -A MD5 | format-list"

A stock Macintosh doesn't come with md5sum installed so you can either create the md5 checksum and compare it with the one generated by the portable rom dumper:
Code: [Select]
MD5 (ROM1A.BIN) = 8a4d0fbfa6e6759d41b833bf764748fb
cat ROM1A.MD5
8a4d0fbfa6e6759d41b833bf764748fb  ROM1A.BIN
Or you can install md5sum. I use Homebrew:
Code: [Select]
brew install md5sha1sumThis version of md5sum couldn't check all of the dumps at once:
Code: [Select]
md5sum -c *.MD5
Error: --check <filename> cannot be used with additional files
So you'll have to do them one at a time:
Code: [Select]
md5sum -c ROM1A.MD5

Now you've got several sets of verified ROM dumps, make sure you know which one is which. In most cases you'll only need to work with the ROM1.BIN for the Canon firmware that you started from and the one for the Canon firmware you're porting to.

Congratulations, you just completed the most essential step towards porting Magic Lantern!

If you're in a hurry, curious, stupid or all of the above you'll be tempted to try your current version of ML on the new firmware. Guess which description I fit into?


There are various methods of disassembling the ROM dumps in order to work with them. Everyone seems to have their favorite method. ARMu is quick but only works on Windows--ok it also works on Mac under wine but I haven't figured it out yet. ARM-console is very difficult to set up and use but it is very powerful--this currently my favorite disassembler. There is also a commercial product called IDA but in order to disassemble ARM code you need to buy a license. By far the easiest for beginners is which is a perl script that needs just a few tweaks to get running properly as explained in this excellent tutorial on finding stubs.

What about QEMU? That could be very useful, especially when porting major firmware updates and new cameras but we're doing just a minor firmware update that can be done with just a few basic tools and a lot of perseverance.

Now it might be tempting to upload and share these ROM dumps and disassemblies but DON'T DO IT! You're holding copyrighted material that is owned by a very large and powerful corporation. You bought the camera so you can look at what is inside it but you aren't supposed to publish "intellectual property" in an open forum like this one. I'll be editing the bits in this post so they don't perfectly match the real disassembly but it should be good enough for instructional purposes.

So let's do this. Assuming is in the same directory as your ROM1A.BIN file:
Code: [Select]
perl 0xFF000000 ROM1A.BIN
string dump
create elf file
label scan
disassemble and string lookup
job complete!

The file you are interested in is ROM1A.BIN.dis. You'll be spending a lot of time together.

Preparing your local repository for the update.

There are lots of ways to work with your local repository. It really depends how you are most comfortable. If you plan to eventually do a pull request you should consider creating a "scratch" or development branch where you can experiment and post all your notes and then when everything is working, copy over your changes to a fresh branch to avoid cluttering up the revision history with trivial commits.

Take a look at some recent firmware updates so you'll know what you are getting yourself into:

EOS M 2.0.2 - 2.0.3
700D 1.1.4 - 1.1.5

This one is somewhat different because it doesn't replace any previous firmware version and is a bit more complex:

5D3 1.3.4

The first thing you'll notice is that it seems that we're throwing out the old and creating new files. That's not really the case, we're renaming the old files but don't do it with the operating system because it defeats the point of keeping everything under version control. Generally I like to work with the SourceTree application but copying directories is one of those things that is probably best done on the command line. There are two directories and one file that needs to be renamed. For example, when going from 1.1.4 to 1.1.5 on the 700D:

The platform directory
Code: [Select]
cd magic-lantern/platform
hg rename 700D.114 700D.115

The installer directory
Code: [Select]
cd magic-lantern/installer
hg rename 700D.114 700D.115

Editing the Source Code

Now for the good news. This step has registered a lot of changes but you'll only need to modify a few of these files. Most of the changes will be happening in the stubs.S file. While it is challenging looking for new stubs or finding stubs for a new camera model, a minor firmware update is rather easy.

Let's start with a very easy change. In magic-lantern/installer/Makefile you'll see this line:

Code: [Select]
SUPPORTED_INSTALLERS := 1100D.105 500D.111 50D.109 550D.109 5D2.212 5D3.113 60D.111 600D.102 650D.104 6D.116 700D.114 EOSM.202 7D.203

Pretty easy to figure out what to change on that line. However, there's something to watch out for when editing a Makefile, you got to make sure that your text editor doesn't change the indents from tabs to spaces. A Makefile needs those tabs.

Now check the pull requests on bitbucket and you'll see several other places where you need to change from the old to the new firmware version. Don't forget to make changes in the modules: adtg_gui, dual_iso, mlv_rec and mlv_lite.

Another rather easy change to make is to remove the old ML-SETUP.FIR file. It won't work with the new firmware and a developer will have to create a new one for you. In the case of a minor firmware update if you did the Canon update with the boot flag set on the camera, you won't have any problems later when you test out your new port. Just don't rush it! There are a few things that need to be done in the right order before attempting to boot into your newly ported ML.

In fact if you want to be extra careful you could take the advice that is in property.c and disable the prop_request_change function. If you're a suspenders and belt kind of guy also go into your platform's internals.h and comment out '#define CONFIG_PROP_REQUEST_CHANGE'. This is highly recommended for new ports because you can do some serious damage. I didn't take that precaution because mine were very minor updates.

Finding where the stubs moved

Now we get into the nitty gritty of finding stubs. We aren't porting a new camera, just a minor firmware update so all the stubs that you should be able to easily look up in the old disassembled ROM1.BIN disassembly should also be in the new disassembly--only in a different location.

If you're working with a DIGIC V the first thing you will notice in the stubs.S file is something like this (700D.114):

Code: [Select]
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...

There is a very good explanation of what this RAM_OFFSET is all about in a1ex's Tutorial: finding stubs. In this case Nanomad left a clue so let's check out what line 0xff0c0098 says in the 700D.114 disassembly. Depending which disassembler you used it might look like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; pointer to sub_FFA5E8B0
ff0c009c: e59f1048 ldr r1, [pc, #72] ; 0xff0c00ec: pointer to 0x1900

or like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b0)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Now let's look at what the 700D.115 disassembly looks like:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b8)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Hooray! RAM_OFFSET changed from 0xFFA5E8B0-0x1900 to 0xFFA5E8B8-0x1900.

Now start going through the rest of the stubs.S file. One trick I discovered is that if you're tackling one of those minor updates Canon pushed in October of 2016, any stub that doesn't use RAM_OFFSET remains unchanged and those with the offset move the same amount in the same direction that RAM_OFFSET changed--for the most part. A few will move. Why? Because there was a few lines of code that were changed somewhere that shifted things around slightly. In any case you should check the address to every stub to make sure. In some cases there's a descriptive string around the stub you're checking into, other times there is nothing to go by except if several lines "look" the same. It becomes sort of a pattern matching game like those cartoons that you try to find the difference between two drawings that look almost identical.

What about those stubs that aren't addresses?

Code: [Select]
/** Task info **/
NSTUB(   0x25024,  task_max)
NSTUB(0xFFA0A0D8 - RAM_OFFSET,  is_taskid_valid)            // AJ_task_trampoline_related_p10
NSTUB(   0x23E14,  current_task)
NSTUB(     0x674,  current_interrupt)                       // in interrupt handler (0x18), where MEM(C0201004) is stored

You can take a chance that it didn't change but it is best to look at the hints, search the forum, read the wiki or post a question how to find these values. Leave those challenging ones until the end because by the time you've gone through all of the other stubs you'll become more of an expert at sleuthing out those stubs.

There are some helpful scripts in the magic-lantern/contrib directory. Try running to get a report on the differences between the old and new stubs.S files. I posted my results on the 700D.115 pull request.

Make sure you got all of the changed addresses

Not all of the addresses you need to check are in the stubs.S file. The source code for some of the modules also have references to firmware versions and/or addresses that need checking and probably changing. For example, adtg_gui.c, dual_iso.c, mlv_rec.c and mlv_lite.c. By now your pattern matching skills are probably pretty good so good luck hunting them down.

Getting the firmware signature

The next step involves something a little more exciting that pouring over endless lines of disassembly code. You'll need to find the new firmware's signature and for that you'll need to build Magic Lantern, but this isn't a working build--not yet.

Take a look at fw-signature.h. Your updated firmware version is probably not in the list. Now check out boot-hack.c and you'll find this near the beginning of the file:

Code: [Select]
#include "fw-signature.h"

That's right, we're going to welcome your firmware update to the world. In order to do that open up config-defines.h and look for this piece of code:

Code: [Select]
 * Enable these for early ports

    /** If CONFIG_EARLY_PORT is defined, only a few things will be enabled (e.g. changing version string) */
    //~ #define CONFIG_EARLY_PORT

    /** Load fonts and print Hello World (disable CONFIG_EARLY_PORT); will not start any other ML tasks, handlers etc. */
    //~ #define CONFIG_HELLO_WORLD
    /** Create a developer FIR for enabling the bootflag and dumping the ROM. */

Uncomment the hello world line:

Code: [Select]
Now build Magic Lantern, put it on one of your boot flag enabled cards, insert it into your camera and you should see something like this:

Write it down, put that address in fw-signature.h, comment out '#define CONFIG_HELLO_WORLD' and...

Trying it out

Ok so it should work but something will probably go wrong the first time you try it out. On the 700D.115 some of the fonts were missing. Searching for fonts I found this:

Code: [Select]
#define BFNT_CHAR_CODES    0xFFCF67E4

So it took a bit more effort to find those addresses but magic-lantern/contrib/indy/ found them automatically.

On the 5D3.134 there were some other things that needed attention in consts.h. You would think that constants by definition don't change but like the old saying goes, "the only thing constant is change."

Oh, one last thing--the ML-SETUP.FIR file. Usually that's one of the first things that you need if you're working on a new port but since we're only doing a minor firmware update we can wait until the end. Only one of the main developers can prepare this file for you so you'll have to ask.

Found this list of the recent Canon firmware updates that were released in October 2016. I added which version is currently supported in order to help you decide if you would like to attempt to port a simple firmware update.

EOS 5D Mark III, firmware 1.2.3 -> 1.3.4 pull request
EOS 5D Mark III, firmware 1.2.3 -> 1.3.5 in progress
EOS 6D, firmware 1.1.6 -> 1.1.8 pull request See this post.
EOS 7D, firmware 2.0.3 -> 2.0.6 pull request Still needs some work.
EOS 60D, firmware 1.1.1 -> 1.1.2 pull request See this post Should be good to merge.
EOS 500D, firmware 1.1.1 -> 1.1.2 pull request See this post
EOS 550D, firmware 1.0.9 -> 1.1.0 pull request See this post. Some issues noted.
EOS 600D, firmware 1.0.2 -> 1.0.3 pull request See this post. Should be good to merge.
EOS 650D, firmware 1.0.4 -> 1.0.5 pull request See this post. Should be good to merge.
EOS 1100D, firmware 1.0.5 -> 1.0.6 pull request See this post.
EOS M, firmware 2.0.2 -> 2.0.3 unified branch pull request, crop_rec_4k branch pull request Should be good to merge.

Scripting API suggestions / Feature Request: lua hook for audio events
« on: March 03, 2017, 06:19:37 AM »
Hope this is the right place to ask for this lua feature request.

There doesn't seem to be any audio functions in the lua API.

I'm mostly interested in running AUDIO_REMOTE_SHOT in lua to trigger either a full shutter press or short and long shutter half-press on an EOSM. The EOSM doesn't have the option of using a wired remote so shutter events have to be done either through the IR remote or the mic input. The lua script should be able to start a process (like a shutter half-press or rack focus or whatever) from an audio cue.

Another advantage of having lua do the shutter half-press is that it resets the power savings timer while AUDIO_REMOTE_SHOT doesn't reset the timer. At least that's the experience with a film scanner project that one user is trying to set up with an EOSM.

Camera-specific discussion / Canon 5D Mark III / 5D3 / Firmware 1.3.4
« on: February 15, 2017, 02:04:51 PM »
Posted a pull request for 5D3.134 firmware support.

  • Because Canon tells us to update our cameras to their latest firmware release.
  • Once you do update Canon says you can't go back (though you can, really).
  • Canon removes prior firmware revisions from their product support web pages so you need to download from unofficial sources in order to use Magic Lantern.
  • There are 5D3 owners out there that might want to try ML but are unwilling to rollback the firmware installed on their cameras.
  • Because it was a challenge and seemed like a worthwhile thing to do.
Ok, that last point was just to make a point--you don't need to be an experienced developer to contribute to Magic Lantern. In fact doing a minor firmware update is a good project for someone who is able to compile the ML code and is willing to put in some time searching through a ROM disassembly looking for changes. It is also a good project to learn some very basic reverse engineering.

What this version is not:
  • If you want the most stable ML experience and don't need the new firmware features, stick with 5D3.113.
  • This update will not replace the 113 and 123 versions--it can live right alongside the previous versions.
With the recently added option to support more than one firmware revision per platform in the ML source code this might be a good opportunity to add experimental releases closer to the Canon firmware revision release dates. Provided of course that it passes the ML stress tests and that enough people test and approve the pull request.

To make testing easier for users to try it out, I'll be posting builds to my bitbucket repository download area until it gets into the Experiments download page. Note that this first very early build doesn't have the ML-SETUP.FIR file yet so make sure your camera bootflag is set before doing the firmware update. To go back to 113 or 123, follow the instructions on the Firmware Update/Downdate? tutorial.

Big thanks to:
  • a1ex for his guidance and encouragement and keeping this project alive.
  • chris_overseas who did the 5D3.133 port which had the last few missing pieces to get this working.
  • dmilligan for his patience answering my noob questions.
  • Danne and DeafEyeJedi my ML buddies.

Just as a footnote: Firmware revision 1.3.4 was released November, 2016 and the 5D300134.FIR file is timestamped September 1, 2016 while the 5D300123.FIR has a timestamp of August 30, 2013 and the 5D300113.FIR has a timestamp of May 28, 2012.

General Development Discussion / 5D3.134 Stubs API test errors
« on: February 14, 2017, 08:15:35 PM »
Hope I'm not overstaying my welcome here but this seems like this is so close. I've been running the Stubs API test and consistently getting 200 errors, but they are all the same error.

Code: [Select]
[FAIL] ABS((m0-m1) - 50*1024) => 0xc800
It seems to be happening in this selftest function:

Code: [Select]
static void stub_test_malloc_n_allocmem()
    // mallocs
    // bypass the memory backend and use low-level calls only for these tests
    // run this test 200 times to check for memory leaks

It maybe pointing to a problem with the memory allocation stubs but I triple checked them against the 123 and the 113 disassemblies and they seem fine, they also pass the test.

Any idea where the problem might be at?

General Development Discussion / Oh where oh where have my fonts gone?
« on: February 14, 2017, 01:09:34 AM »
I've been playing around with getting ML working on the 5D3.134 firmware and have gotten quite far but am stumped trying to find the addresses for the fonts.

According to the 5D3.123 code:

Code: [Select]
#define BFNT_CHAR_CODES    0xF7363A50
#define BFNT_BITMAP_DATA   0xF7369D38

Ok, been there before on the 700D firmware update and treated these just like finding stubs but these addresses aren't in the ROM1.BIN disassembly. I looked up the wiki reference and it pointed to the script which is in the contrib/indy directory and it needs to be run on ROM0.BIN. Ok, so far so good. This is what I got this for the 5D3.123:

Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xff373a2c: FNT
0xff373a30: (+0x04) 0xffd8
0xff373a32: (+0x06) font_width = 40
0xff373a34: (+0x08) charmap_offset = 0x24
0xff373a38: (+0x0c) charmap_size = 0x3174
0xff373a3c: (+0x10) bitmap_size = 0x89f16
0xff373a40: (+0x14) font name = 'HCanonGothic///'
0xff373a50: (+0x24) char_codes[]. 3165 chars
0xff376bc4: (+0x3198) offsets[]. Last offset value = 0x89ee0
0xff379d38: (+0x630c) bitmaps[]
  0xff403c18: (+0x901ec) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xff403c50: FNT
0xff403c54: (+0x04) 0xffd8
0xff403c56: (+0x06) font_width = 40
0xff403c58: (+0x08) charmap_offset = 0x24
0xff403c5c: (+0x0c) charmap_size = 0x188
0xff403c60: (+0x10) bitmap_size = 0x31c4
0xff403c64: (+0x14) font name = 'CanonMonospace'
0xff403c74: (+0x24) char_codes[]. 98 chars
0xff403dfc: (+0x1ac) offsets[]. Last offset value = 0x3142
0xff403f84: (+0x334) bitmaps[]
  0xff4070c6: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

0xffb73a2c: FNT
0xffb73a30: (+0x04) 0xffd8
0xffb73a32: (+0x06) font_width = 40
0xffb73a34: (+0x08) charmap_offset = 0x24
0xffb73a38: (+0x0c) charmap_size = 0x3174
0xffb73a3c: (+0x10) bitmap_size = 0x89f16
0xffb73a40: (+0x14) font name = 'HCanonGothic///'
0xffb73a50: (+0x24) char_codes[]. 3165 chars
0xffb76bc4: (+0x3198) offsets[]. Last offset value = 0x89ee0
0xffb79d38: (+0x630c) bitmaps[]
  0xffc03c18: (+0x901ec) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xffc03c50: FNT
0xffc03c54: (+0x04) 0xffd8
0xffc03c56: (+0x06) font_width = 40
0xffc03c58: (+0x08) charmap_offset = 0x24
0xffc03c5c: (+0x0c) charmap_size = 0x188
0xffc03c60: (+0x10) bitmap_size = 0x31c4
0xffc03c64: (+0x14) font name = 'CanonMonospace'
0xffc03c74: (+0x24) char_codes[]. 98 chars
0xffc03dfc: (+0x1ac) offsets[]. Last offset value = 0x3142
0xffc03f84: (+0x334) bitmaps[]
  0xffc070c6: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

So now I'm more lost than ever. Where are the character codes, bitmap offset and bitmap data and how did someone come up with values of around 0xF736xxxx?

By the way, here's what is reporting on the 5D3.134 ROM0.BIN file:

Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xff373a2c: FNT
0xff373a30: (+0x04) 0xffd8
0xff373a32: (+0x06) font_width = 40
0xff373a34: (+0x08) charmap_offset = 0x24
0xff373a38: (+0x0c) charmap_size = 0x3180
0xff373a3c: (+0x10) bitmap_size = 0x8a18c
0xff373a40: (+0x14) font name = 'HCanonGothic///'
0xff373a50: (+0x24) char_codes[]. 3168 chars
0xff376bd0: (+0x31a4) offsets[]. Last offset value = 0x8a156
0xff379d50: (+0x6324) bitmaps[]
  0xff403ea6: (+0x9047a) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xff403edc: FNT
0xff403ee0: (+0x04) 0xffd8
0xff403ee2: (+0x06) font_width = 40
0xff403ee4: (+0x08) charmap_offset = 0x24
0xff403ee8: (+0x0c) charmap_size = 0x188
0xff403eec: (+0x10) bitmap_size = 0x31c4
0xff403ef0: (+0x14) font name = 'CanonMonospace'
0xff403f00: (+0x24) char_codes[]. 98 chars
0xff404088: (+0x1ac) offsets[]. Last offset value = 0x3142
0xff404210: (+0x334) bitmaps[]
  0xff407352: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

0xffb73a2c: FNT
0xffb73a30: (+0x04) 0xffd8
0xffb73a32: (+0x06) font_width = 40
0xffb73a34: (+0x08) charmap_offset = 0x24
0xffb73a38: (+0x0c) charmap_size = 0x3180
0xffb73a3c: (+0x10) bitmap_size = 0x8a18c
0xffb73a40: (+0x14) font name = 'HCanonGothic///'
0xffb73a50: (+0x24) char_codes[]. 3168 chars
0xffb76bd0: (+0x31a4) offsets[]. Last offset value = 0x8a156
0xffb79d50: (+0x6324) bitmaps[]
  0xffc03ea6: (+0x9047a) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xffc03edc: FNT
0xffc03ee0: (+0x04) 0xffd8
0xffc03ee2: (+0x06) font_width = 40
0xffc03ee4: (+0x08) charmap_offset = 0x24
0xffc03ee8: (+0x0c) charmap_size = 0x188
0xffc03eec: (+0x10) bitmap_size = 0x31c4
0xffc03ef0: (+0x14) font name = 'CanonMonospace'
0xffc03f00: (+0x24) char_codes[]. 98 chars
0xffc04088: (+0x1ac) offsets[]. Last offset value = 0x3142
0xffc04210: (+0x334) bitmaps[]
  0xffc07352: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

So close yet so lost!

Tutorials and Creative Uses / Firmware Update/Downdate?
« on: February 11, 2017, 02:58:02 AM »
How to do a Canon firmware downgrade
(works for upgrade too)

There seems to be a lot of FUD (Fear Uncertainty and Doubt) about changing the firmware on your camera, especially for uninitiated Magic Lantern users who are pointed to a website in Hungary run a guy who goes by the name of Pelican to get older firmware versions. Nothing against Hungary (I'm half-Hungarian) or Pelicans (I live at the beach) but a little anxiety is understandable.

Once you realize that it is extremely unlikely that someone created a FIR file that will cause your camera to send banking information to international cyber terrorists, download the file onto an SD card.

Firmware updates are generally posted in some sort of compressed format often along with the instructions on how to run the firmware update. If you are on a Mac and can only find a Windows version, don't panic. The *.FIR file runs on the camera so it doesn't matter which computer platform you are using. Of course you might download something that looks like this on your Mac:

Again, don't panic. This is a self extracting Windows archive and you can use Stuffit Expander which is in the Applications/Utilities folder of every Mac to expand the archive file.

I've got several cameras and like to play around with different versions of Magic Lantern so I put all of my FIR files on a single card. Make sure that the card doesn't have the boot flag enabled because if your camera also has the boot flag enabled it will just hang when you start up the camera unless you also add a ML autoexec.bin file and then it won't work on multiple platforms. In other words, this will be your dedicated firmware card.

Those CCF14* firmware files are for the 700D/T5i, yeah I know it isn't obvious. And just in case you're thinking it, no the 5D3 firmware won't install on the EOSM. Only the valid choices will show up.

Now when you run the firmware update from the Canon menu using this card, remembering to have the camera in Manual and Still Picture mode, it will run the firmware update process--even if you are doing a firmware downdate. (Is there such a word?) The firmware update will not run if you don't have enough juice left in the battery and some external power adapters supply barely enough voltage to run the camera so it might not work. Best be safe and top off your battery first.

End of tutorial.


Firmware versions in the x.3.x series (currently only affecting 5D3) won't let you downgrade to a prior series. In other words, you can go from 1.3.4 to 1.3.3 but if there is a 1.2.3 or a 1.1.3 firmware on your card you'll get this message:

It is not quite that easy with some of the latest Canon firmware updates. There are warnings about not being able to downgrade and rumors that you need some older 2.x version of Canon EOS Utility to do the downgrade but I found that is not really the case. If you happen to have one of those irrevocable firmware versions on your camera you can use whatever version of EOS Utility you have. Make sure to put a copy of the FIR file on your computer because you'll have to point to it with EOS Utility. Also make sure your camera is in Manual and Still Picture mode or EOS Utility will refuse to do the firmware update.

HA! Just noticed that "WTF Captions" item. Who says Canon engineers don't have a sense of humor?

Ok--back to doing this. You need to have a card in your camera so let's use that dedicated SD card with all the various versions. Here we go:

Hey, what just happened? EOS Utility just transferred the firmware update from your computer onto the card in your camera and you can now disconnect the USB cable. The rest is exactly the same as before. In fact if you have multiple firmware versions on your card you can now have a second chance to decide which version you want to install.

Interestingly the firmware update will not change the status of the camera boot flag so you won't have to re-install ML. If you are curious what a new firmware version looks like, use the Portable ROM dumper and run the ROM1.BIN file through a disassembler and...I'm getting off topic here, porting a firmware update to ML is another tutorial.

General Development Discussion / Running ARM-console on a Mac
« on: December 16, 2016, 01:12:24 AM »
I'm following these instructions:

Seems like I'm so close but stuck on this:

Code: [Select]
You might want to do this:
    D = load_dumps("autoexec")
Traceback (most recent call last):
  File "", line 8, in <module>
  File "/usr/local/lib/python2.7/site-packages/IPython/", line 242, in __call__
  File "/usr/local/lib/python2.7/site-packages/IPython/", line 1836, in embed_mainloop
  File "/usr/local/lib/python2.7/site-packages/IPython/", line 1267, in set_completer_frame
    self.Completer.namespace = self.user_ns
AttributeError: 'InteractiveShell' object has no attribute 'Completer'

Any hints or should I just fire up the old Linux laptop for this?

Is there a reason that it isn't assigned to "0" with manual lenses? I'd like to have the focal length tag in MLV files default to "0" when using manual lenses instead of 50mm.

I looked all over the code and can't figure out where lens_info.focal_len is assigned a value of 50 when a manual non-cpu lens is attached to the body. I did find a place where it is reassigned "0" but it seems like it is there in case a chipped lens reports bogus values:

lens.c line 1533
Code: [Select]
    if (lens_info.focal_len > 1000) // bogus values
        lens_info.focal_len = 0;

Am I'm missing something obvious somewhere?

Feature Requests / Update mlv_dump
« on: October 08, 2016, 03:52:05 PM »
mlv_dump is one of the Magic Lantern stalwart apps and is still useful in many situations although many of the functions have been improved in dmilligan's MLVFS. Some of the "features" in MLVFS could be considered bug fixes that should be implimented in mlv_dump.

I opened an enhancement issue on bitbucket, Issue #2609

First baby step is to change the way dng files are named so that it conforms to MLVFS naming. The justification is because there should be some consistency between ML apps.

Pull request:

Hi fellow Magic Lanterners. I'm working on a short film shot on a 5D3 and standard 1920x1080 MLV. I created proxy files using Danne's MLP. After editing the ProRes proxy files in Premiere I thought I'd try doing the color grading in After Effects linking back to the DNG files. Why AE instead of DaVinci Resolve? Partially because I'm more familiar with AE and also because I'm doing some effects work that I'm more comfortable doing in AE. Anyway--everything was going fine following this tutorial:

Except that after interpreting the footage from 30 to 23.976 it was off a frame or so between the QuickTime file from the Premiere timeline to the DNG image sequence. I'd really like to get this working properly.

By the way, I started out going from MLV to DNG then to a log ProRes file and do the LUT, look and color grading thing on the log ProRes but that seemed to lose far more detail and limits what you can do compared to working with the DNG files.

Here's what my ProRes file looks like -- not much I can do with it before it starts falling apart.

From the DNG I brightened it up, pulled out some blue in the window softened it up and added grain all in ACR.

Again from the DNG, a grittier look also in ACR.

General Chat / Timecode for ML Users (and abusers)
« on: March 24, 2016, 06:09:06 PM »
In the late '90's a friend working as a post production supervisor at Sony Pictures called to hash over a problem. He had a project that was being shot on the brand new Sony DVCAM format. The idea was to shoot four different stories with four cameras without stopping and put them all on the screen at the same time. The plan was to shoot just one take and make no cuts. Problem was, the cameras kept drifting and the timecode wouldn't match up so they were trying to get it to work by cutting out a frame here and there to get it to sync--they were also on their second week of shooting. They finally resolved it on the sixteenth take. The name of the movie?


So if a major studio with the resources of a giant multi-national corporation had problems with timecode there's a good chance you're going to encounter some issues too. Let's get started.

I borrowed this image from an old post on the editingstandards, a long abandoned site started by a couple of very talented assistant editors. I'll keep referring back to it.

Recently I've been having conversations with some Magic Lantern users and developers and it seems that most people here already have a basic understanding of what timecode is (a standard maintained by the Society of Motion Picture and Television Engineers), how it is displayed (hours:minutes:seconds:frames) and some of the ways timecode might be used in an MLV workflow so I won't bore you with the basics. Instead let's jump into the areas where most people seem to having difficulties.

Look at two timecode displays in the upper right of the image. Notice one is in 24fps and is called "Record Time" while the other is counting in 30fps and is called "Tape Time." Now a quiz--what fractional second is the timecode displaying? Well the 30fps timecode is showing 10 frames and the 24fps timecode is showing 8 frames so 10/30 or 8/24 will give you the answer--0.33333... seconds.

This sample was originally shot on 35mm film at 24fps and transferred to NTSC 29.97fps videotape in a telecine bay using a 3:2 pulldown. The timecode on the videotape is non-drop frame. I'll get into why we don't use drop frame timecode a little later but first look at the extra number to the right of the video timecode. It is both a 1 and a 2 interlaced together. These numbers are burned into the separate video fields identified as upper/lower or field 1 and field 2. There are other ways that these fields can be displayed without having to add extra characters to the timecode display. One popular method is to use a period as the delimiter between the seconds and frames for field 1 and a colon for field 2 so in the above example one field would be 01.10 and the next 01:10.

Our cameras have a setting that allows us to shoot either 50p or 60p (actually 59.94) the "p" standing for progressive or full frames so they are not interlaced (limited to 1280x720 on ML supported cameras and up to 1920x1080 the 7D Mark II). So--it is understandable if two fields make up one frame then that one frame is identified as a frame in the timecode but why isn't every progressive frame identified as a frame in the timecode? Some developers think that because we're in the digital age we should be able to change timecode to accommodate this higher framerate material and indeed some systems do stray from the SMPTE specifications but there are reasons why the standard that was established about 50 years ago hasn't changed yet. One reason is because the broadcast signal is still being interlaced. Now I'm not an engineer and have not worked in broadcasting so you might want to run a fact check on that. The bottom line is that even when we work with 50p and 60p media, we have to stick with 25 and 30fps timecode.

General Development Discussion / bitbucket protocol
« on: February 23, 2016, 07:18:31 PM »
I've got a question for the developers who have write and administrative permissions on the Magic Lantern Trammell Hudson bitbucket account.

What is the protocol for getting something merged into the unified branch so that it will be pushed to the nightly builds?

The reason I'm asking is because lately there has been lots of updates on branches and when an important issue is resolved in one of these development branches when does it get merged into unified? We have a situation right now where an issue with the metadata for some cameras wasn't being saved properly in MLV files. That issue was resolved in the mlv_prop_data branch but it was not merged into unified.

Now we have MLV Lite available for testing which is in a development branch called new-raw-format which understandably shouldn't be merged into unified until it is well tested but anyone testing that module with cameras that show focus pixels in raw video cannot test MLV Lite with the focus pixel removal tools recently implimented in MLVFS and MLP without using a custom build from the mlv_prop_data branch. This negates the convenience of dropping in new modules into a nightly build for testing, especially for users that are not able to compile the ML source.

There are also a couple of development branches for the lua module, lua_fix and lua_touch, which are resolving issues with the lua module. At what point are these going to be merged into unified? It seems to me that lua is mainly for experimenters and having frequent updates would be welcome for users who are writing lua scripts.

I've been posting comments on bitbucket about some of these issues and I certainly don't want to continue posting and possibly annoy developers. I'm just asking what is the protocol for requesting that a branch be merged into unified?

Feature Requests / mv1080 on EOSM
« on: February 06, 2016, 04:56:46 PM »
The EOSM cannot currently record in mv1080 mode though it is within the capability of the camera and it seems to be a "possible" Magic Lantern feature request.

I'm wondering if there is any possible way to get mv1080 on the EOSM but I take it that has been tried before.
It wasn't tried too hard ;)

Looking a bit in forum history, it seems like in the very first implementations of raw video, there was a (undocumented) way to do it: and

Maybe I should revisit it, but that would be a different topic.

a1ex explained that there are other ways to capture raw video. Perhaps one of these alternate methods can capture the data after it has gone through the focus pixel annihilation process?
I wouldn't really say there are "other ways" to capture raw video, there are just other possible video modes. Capturing them is the same, you just have to figure out how to get Canon firmware to put the camera in those modes. The EOSM is unique b/c Canon firmware doesn't put it into mv1080 until you start recording (H.264). So if you want to record raw video on mv1080, you have to do it while also recording H.264 at the same time (ML prevents you from doing this now a days, but it was possible in some old versions when raw video just came out). That or do some reverse engineering to figure out how to put the camera in mv1080 without starting recording. All other cameras go into the video mode you selected in the Canon menu immediately, so actually mv1080 is pretty much the "default".

Let me try to sell this feature request by stating some of the reasons I believe why the lowly EOS-M is perhaps just as important to moving Magic Lantern forward as the 5D mark III.

The camera is the least expensive of all the ML capable cameras which makes this the ideal model to experiment without being too concerned about bricking an expensive piece of photographic equipment. It is so affordable that multiple bodies can be had for less than the cost of higher end models. In fact you can get 10 EOSM bodies for the price of one 5D3--I myself have 4 bodies, yeah I'm crazy.

At first the EOSM may seem radically different from the other Canon models because it is a mirrorless but the internals are very similar to the DSLR's. The advantages  include not having to lock up a mirror to get into Live View and a shorter lens flange distance so many lenses can be adapted for use on this camera.

Canon EOSM with C-mount lenses

Though the EOSM may feel like a toy compared to the DSLR's it actually has quite a rugged metal body and can stand up to the riggers of a documentary project I'm working on. I planned to use it as a spare body for a 5D3 setup but it proved capable enough as the 'A' camera. (When I was a professional still photographer back in the day I would never consider going on a job without a backup camera body.)

EOSM documentary rig

The EOSM can also be an ideal replacement for a director's viewfinder and with a PL adapter the camera can be used to look through professional cine lenses. Adding mv1080 also gives the capability of shooting much higher quality tests than with H.254 video.

EOSM with PL mount adapter

Adding mv1080 on the EOSM will also elevate this platform to the same level as the rest of the Magic Lantern capable cameras though filmmakers may want to dress it up a bit in order to be taken seriously.

EOSM Shoulder Rig with Follow Focus

There are disadvantages to the EOSM. One problem for video shooters is the short battery life though extra batteries are very affordable and it is very simple to rig up external power supplies. Another problem is that it responds slower than other cameras in still photo mode though it is just as quick as the other cameras in video mode. Speaking of still photo mode, there's that pesky shutter bug when using Magic Lantern but that's a different topic.

Reverse Engineering / Reverse Engineering Picture Styles
« on: December 07, 2015, 05:50:39 AM »
Ok--here we go:

2016-09-07 v1.4 - Test version to improve highlights detail. (Don't use if you're using zebras to judge exposure.) Dropped logStandard and logFaithful.
2016-09-02 v1.3 - Overall better match
2016-09-01 v1.2 - Combined highlights from v1.0 with shadows from v1.1 and Saturation now defaults to -2
2016-08-31 v1.1 - Small adjustments to correct color shifts in shadows
2016-08-31 v1.0 - Initial release

This was a joint effort between Danne and myself, dfort. For the first release we did the best we could at matching Technicolor CineStyle. The reason we picked CineStyle is because it is in wide use, hasn't been updated since it was initially released in 2011 and it is a .pf2 that cannot be editing using Canon's Picture Style Editor. Ours is in the newer .pf3 file format and is editable. We didn't even put a copyright notice on it so feel free to do with it as you please.

We did a lot of testing and have a fairly high degree of confidence that this can be a direct replacement for CineStyle. This means that all of the luts available for CineStyle can be used. Is it a perfect copy? No, but it is the best we could do using the tools we had at our disposal. Visually it should be impossible to see any difference between CineStyle and logNeutral. Why call it logNeutral? Because it turns out that CineStyle uses the standard Canon Neutral picture style as its base. Only the luma gamma curve has been modified. How do we know this? Let's just say we're pretty sure.

So if there is a logNeutral could there be variations using other standard Canon picture styles? Yes. We found out that the luma gamma curve works with other picture styles. Sure, you could probably adjust the look of the Neutral style in post to match the other picture styles but one of most often mentioned issues with CineStyle is that it is hard to get pleasing skin tones. So why not experiment with the Portrait style as a base? We included all of standard Canon picture styles with this release.

Now with all the excitement over raw video why is a copy of a 5-year old picture style of any significance? It turns out that many people are using CineStyle not only for H.264 (8-bit 4:2:0) but also for HDMI out to an external recorder. In the future perhaps someone can do the same with Log-C or something that will take better advantage of the 8-bit 4:2:2 HDMI output.

Consider using MLP in your workflow for this picture style. Danne continues to update this program and it can handle luts, denoising and conversion to Prores. MLP can also work with H.264 HDR which is a natural for this picture style.

Most of all, try it out and report back. If you don't like it, hey at least you didn't pay for it.

Original post starts here:

Not sure which category to post this under because it is part general chat, part shooting, part post production, part raw and part not...

I've been thinking about picture styles because of a conversation I had with Danne about CineStyle. I call that conversation the "CineStyle Conspiracy Theory." Basically it goes like this:

Ok, let's start out with who developed Cinestyle. Technicolor--not exactly a little fly by night company but one of the most respected companies in the motion picture industry. Interestingly it is a French multinational company, not just a film lab in Hollywood.

When. 2011, that's a year before the 5D Mark III but more importantly also a year before the C100 which has the so-called "log" profile built in. Why did Technicolor create Cinestyle then stopped supporting it? The luts and tools that Technicolor developed are no longer available directly from the company though the webpage is still live and the profile is still available for download.

Looking at the sample footage shot on "Canon Log" from the C100 compared with Cinestyle they look very similar. Same with the GoPro "Protune" but here's a little known fact--Protune was developed by Technicolor. I'm not saying that GoPro footage is great but Protune does allow more flexibility in post in order to more closely match footage shot on other cameras so there is some merit to this.

I have read a lot of pros and cons (mostly cons) on the boards about Cinestyle and mostly praises for the "log" profiles like S-Log, Log C, Canon Log, etc. Log does have advantages and disadvantages. The old saying that you can't get something from nothing certainly applies here. Technically, it may even be inferior to getting everything "right" in the camera but in practice things are a little different.

I can't look into what they did with CineStyle because that profile is locked out of the Canon PictureStyleEditor but I did read up on how to make a picture style and you have to start with a CR2 file. That got me thinking, all this talk about picture styles not affecting raw images--is that really true? I shot some still and MLV files with monochrome and CineStyle then took a look at them in various programs. Guess what, it does make a difference depending on what program you're using. Try it yourself, set the profile to monochrome and shoot something then open it up in Canon's Digital Photo Professional app--that's a free one that you can download from Canon. The raw image is in black and white. Now open it in Photoinfo--I'm not sure where that came from but I think it comes with OS-X. Also in black and white but it is apparently showing the jpeg preview. How about Adobe Camera Raw or RawTherapee? Color! Run exiftool on it and it will show:

Code: [Select]
Picture Style                   : Monochrome
Running exiftool with the -v option will give even more information:

Code: [Select]
  | | | 20) ProcessingInfo (SubDirectory) -->
  | | | + [BinaryData directory, 28 bytes]
  | | | | ToneCurve = 0
  | | | | Sharpness = 3
  | | | | SharpnessFrequency = 0
  | | | | SensorRedLevel = 0
  | | | | SensorBlueLevel = 0
  | | | | WhiteBalanceRed = 0
  | | | | WhiteBalanceBlue = 0
  | | | | WhiteBalance = -1
  | | | | ColorTemperature = 7300
  | | | | PictureStyle = 134
  | | | | DigitalGain = 0
  | | | | WBShiftAB = 0
  | | | | WBShiftGM = 0

What is interesting is running an MLV shot with the monochrome picture profile through mlv_dump with the -mv option will also show some picture style information:

Code: [Select]
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   134
     contrast:   0
     sharpness:  3
     saturation: -559038737
     colortone:  -559038737

There's that same "134" picture style which I guess is the monochrome profile. How about an MLV shot on CineStyle?

Code: [Select]
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   33
     contrast:   -4
     sharpness:  0
     saturation: -2
     colortone:  0

So what is the significance of this? Well it might be possible to make a "real" log picture profile that will create proper log H.264 files. I looked into this and yes it is possible thought it might not be practical. DeafEyeJedi mentioned that the editors where he works prefer wide gamut over log and I'm thinking that CineStyle might just be a hybrid log/flat/wide gamut picture style because there's only so much you can do with 8-bit 4.2.0 files. However, if the picture style can also be used on the raw files then you can actually create a true log like Arri Log C picture style.

Now for the bad news. Picture styles are binary files and the PictureStyleEditor is a rather elementary tool that isn't really suitable for creating a proper log curve. Then of course there's the job of figuring out how to interpret the picture style that is embedded in the raw file. I looked at the CineStyle picture style in a HEX editor and can't make heads or tails of it but there is a lot more information in there than what exiftool is reporting.

Feature Requests / Reassign button for White Balance
« on: December 01, 2015, 11:09:05 PM »
Recently I discovered the ML Auto adjust Kelvin + G/M in a submenu. It would be so cool (and warm and magenta and green) to map a button for this like on the big boys' cinema cameras. Is this possible? Maybe with a lua script?

Thanks to Danne for pointing out this feature to me.

Here's the issue, cr2hdr seems to do fine when converting one dual_iso dng at a time but it sometimes seg faults when doing a batch.

[EDIT: It doesn't have anything to do with batch processing. I had an instance where cr2hdr failed when converting a single dng then succeeded when re-running cr2dng on the same dng.]

Code: [Select]
Horizontal stripe fix...
Segmentation fault: 11

So the failure seemed to be happening on "Full-res reconstruction..."

I've come across this every once in a while but now I have a dual_iso mlv file that is causing this consistently on both my system and Danne's.
(BTW--this came up while testing his new MLPP workflow for OS-X, which is coming along very nicely.)

Ok--this is how to reproduce the issue. Download this mlv file from my dropbox account:

It is 648 MB so I'd like to delete from there it as soon as enough people have had a chance to try it out. (That shot won't win any awards but it seems to consistently reproduce the issue.)

Next convert it to dng using mlv_dump. I included Mac and Windows versions of mlv_dump and cr2hdr in my bitbucket download area so you don't have to go hunting for them. (Use the testing versions.)

Here's a simple way to do it with mlv_dump:
Code: [Select]
mlv_dump --dng M23-1338.MLV
Finally, batch convert these through cr2hdr you can try:
Code: [Select]
cr2hdr *
Code: [Select]
for i in *.dng; do cr2hdr $i; done
Or whatever method you use to batch convert. You can even import the files into Lightroom, select all of them and run them as a batch using kichetof's cr2hdr plugin. Everything I have tried has caused cr2hdr to seg fault on random frames.

So far this has only been tested on OS-X but it would be interesting to see if it also affects Windows and Linux.

General Development Discussion / Dealing with Focus Pixels in raw video
« on: October 22, 2015, 11:09:10 PM »
One possibility is mapping the focus pixels for each camera and then interpolating out those pixels (rather than just chroma-smoothing the entire image). This might fix the dual ISO / focus pixel issue, IDK. It should be faster than chroma-smoothing, and not have any artifacts. It's a lot of work though, you need to create a focus pixel map for each camera and video mode.

So here we go--

Affected cameras: EOSM, 100D, 600D, 650D, 700D, (and more?)
[EDIT: Pretty sure now that the 600D doesn't show focus pixels in raw video.]

Focus pixels, phase detectors, call them what you will when these dots show up on your raw video they are annoying. There have been attempts at clearing them up like the PinkDotRemoval tool that has apparently been abandoned. Chroma smoothing helps but it affects the overall image and it doesn't work for dual iso footage.

I tried to shoot something that shows as many of these pixels as possible by shooting an out of focus white paper with a couple different colors of lights and processing the image with a heavy hand to show the pixels as clearly and dramatically as possible.

Got your attention? This is a frame from an mlv file shot on an EOSM in crop movie mode. It doesn't look quite as bad when shooting regular subjects but it is a problem.

It is possible to get an even clearer view of these pixels by running a frame through dcraw using the -d option which, according to the documentation:
Show the raw data as a grayscale image with no interpolation.

Here is a closeup of that same frame at the top of this post.

Once we know the coordinates of these pixels they can be zapped.

Method 1 - In Camera

The ideal way to deal with them would be in the camera. In fact there is a way that "bad pixels" can be mapped out and apparently it is common practice to map out these faulty pixels when the camera is manufactured or repaired--saves on reject sensors. Assuming that we can't access the code that maps out pixels in the Canon firmware, the developers over at the chdk project have figured out more than one way to do it:

I'm not skilled enough to take what they have done and apply it to Magic Lantern but maybe someone who has the knowledge can drop a few hints?

Method 2 - In Post

Yeah, we can fix it in post. In fact there are several programs that can deal with pixel problems. Pixel Fixer is a dedicated tool, RawTherapee has an option to deal with hot and dead pixels and an interesting Python wrapper for the LibRaw library called rawpy that creates a ".badpixels" file dcraw can use to zap out pixels. This might be a way to create the database of focus pixel maps dmilligan suggested at the top of this post.

I tried using dcraw to get rid of one of the focus pixels on a dual iso frame and found that one pass wouldn't do it. Maybe it is because the process is breaking down by running the file through cr2hdr first? My understanding is that the debayering process is averaging neighboring pixels and in dual iso it seems to "smear" the pixel vertically. In addition, some of these focus pixels have a secondary pixel diagonally across that also needs to be mapped.

Here is what I ended up doing to get rid of just one focus pixel--and the diagonal secondary pixel.

Code: [Select]
1247 121 0
1248 120 0
1248 121 0
1248 125 0
1248 126 0
1248 122 0
1248 123 0
1248 124 0
1248 126 0
1248 122 0
1248 120 0
1248 124 0

Note that I hit the diagonal pixel just once but had to work top and bottom to middle and hit the same pixels several times in different sequences. Think of it as stippling a brush to spot a print--in photography school I spent many hours spotting prints. (By the way, this a map of all the focus pixels for the 650D that was used on the PinkDotRemover tool.)

Here is the result. I circled the area where the hot pixel was located and yes, it looked just like those other ones before running the dng through dcraw.

So if this is the solution then what's the problem?

The problem is that I'd like to get some feedback that this would actually be practical. It looks like there would have to be a different focus pixel map file for every camera and every crop factor (raw image size) and all of this would have to be assembled in a way that users could handle. Danne is the master of assembling scripts that can automate complex processes so it is possible to piece together something with dcraw--but then again is that the best solution? Maybe it would be better to adapt dcraw's "badpixels" code to mlv_dump or cr2hdr or MLVFS?

Just asking for a little guidance here.

General Development Discussion / Compiling Magic Lantern on a Macintosh
« on: October 14, 2015, 02:36:14 AM »
Setting up a Magic Lantern development environment on a Macintosh

For the terminally impatient try Danne's Magic lantern development compiler tool(Mac OS). It includes the quick install script originally written for this tutorial. You can still get just the install script and skip this tutorial--scroll down to the bottom of this first post.

Before we begin did you know that all Macs have development tools already installed yet they are not available to the user? If you have a "virgin" Mac that you have never used for development try this, open up a terminal (located in Applications under Utilities) and type:

Code: [Select]
which gcc
Chances are that it will return /usr/bin/gcc, gcc is the compiler to build Mac binaries and one of the tools that we'll use to build Magic Lantern. So we're all set, right? Ok, not that easy try typing in gcc to launch the compiler:

We don't really need the full Xcode development environment so simply click the highlighted "Install" button, accept the licence to activate the developer tools. If you have already installed Xcode you might still need to install the command tools. Use this terminal command:

Code: [Select]
xcode-select --install
This will install the developer tools in /Library/Developer/CommandLineTools/. These tools need to play nicely with other developer tools required by Magic Lantern. If you are using OS 10.14 (Mojave) or newer there's another step we need to take so all these tools work nicely together:

Code: [Select]
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.*.pkg

Next, there are some additional tools needed to compile Magic Lantern and several ways to install them. The simplest is to use one of the popular package managers like Fink ( MacPorts ( or Homebrew ( Homebrew is the newest of the bunch so we'll be using that. Homebrew installs the tools so they are available in /usr/local/bin which happens to be included in every new Mac's default path. However, your Mac may or may not have a /usr/local directory. Another possibility is that you have a /usr/local but don't have permissions to install anything in there. Do a directory listing to see where you stand:

Rosies-Air:~ Fort$ ls -la /usr
total 8
[email protected]   12 root  wheel    408 Oct 12 11:45 .
drwxr-xr-x    31 root  wheel   1122 Sep 30 21:56 ..
drwxr-xr-x     5 root  wheel    170 Aug 22 18:51 X11
lrwxr-xr-x     1 root  wheel      3 Sep 30 21:22 X11R6 -> X11
drwxr-xr-x     3 root  wheel    102 Aug 26 19:17 adic
drwxr-xr-x  1061 root  wheel  36074 Sep 30 21:28 bin
drwxr-xr-x   258 root  wheel   8772 Oct 12 11:45 include
drwxr-xr-x   285 root  wheel   9690 Oct 12 11:45 lib
drwxr-xr-x   183 root  wheel   6222 Sep 30 21:22 libexec
drwxr-xr-x   243 root  wheel   8262 Sep 30 21:22 sbin
drwxr-xr-x    46 root  wheel   1564 Oct 12 11:45 share
drwxr-xr-x     4 root  wheel    136 Sep 17 00:03 standalone

Notice that I used "ls -la" to get a detailed view of everything in the /usr directory. This system doesn't have a /usr/local directory so we'll have to create one. If you have a /usr/local directory but it is owned by root, you need to change permissions. Try this first:

Code: [Select]
sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
Don't be intimidated by long command lines, you're on a Mac so simply copy and paste it into the terminal window. Note that you will have to enter your password.

If it didn't work you are probably on OS-X 10.11 "El Capitan" or 10.12 "Sierra" or even something newer so you need to temporarily turn off disable System Integrity Protection (SIP) in order to create a /usr/local you can use.

1. Restart the computer, while booting hold down Command-R to boot into recovery mode.
2. Once booted, navigate to the "Utilities > Terminal" in the top menu bar.
3. Enter csrutil disable in the terminal window and hit the return key.
4. Restart the machine and System Integrity Protection will now be disabled.

Now in the terminal enter this command:

Code: [Select]
sudo mkdir /usr/local && sudo chflags norestricted /usr/local && sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
What you are looking for is a /usr/local directory with your name on it:

drwxr-xr-x     2 Fort  admin     68 Oct 12 12:52 local

Don't skip this next step -- go back into recovery mode and this time type csrutil enable in the terminal.

Now that we've got a /usr/local that we can use, install Homebrew. Ok--if you are really itching to get up and running you can now just skip to the bottom of this post and run the quick installation script. Go!

If you want to understand what we're doing any why, keep following this tutorial:

First of all this command will download and run a ruby script to install a basic Homebrew setup:

Code: [Select]
ruby -e "$(curl -fsSL"
Next start adding some of the tools needed to build Magic Lantern.

Code: [Select]
brew install python wget mercurial
What's the deal with Python, isn't it already installed? Yes, but getting the Apple version configured to work with the ML install scripts is more trouble than it is worth so we'll just use the Homebrew version. In addition, we're going to get some python scripts using the package manager that Apple left out of their version of Python.

Code: [Select]
pip install docutils
pip2 install docutils
pip3 install docutils

That's not a mistake--we want all three versions of docutils.

If for some reason you'd rather use Apple's version of Python, I've included instructions on how to set it up at the bottom of this post.

Although the compiler that comes with the Mac can compile ML there are reasons to use a different version. Technically it isn't really gcc but another compiler called "clang" and if you try using it to compile ML you'll probably get a bunch of clang error messages. In addition, Homebrew switched to version 6 of gcc and that's causing some issues with ML so we'll go with good old version 5.

Code: [Select]
brew install [email protected]
Two versions of gcc? Won't that confuse the system? Homebrew is designed to work with the system so the gcc that you just installed is called gcc-5. This requires some special handling that we'll get to a little later.

We're not not done with compilers, we'll need yet another version of gcc to compile binaries that will run on the ARM architecture used in our Canon cameras. To figure out which ARM compiler to get refer to the ML code, specifically: Makefile.user.default -- ARM_PATH=~/gcc-arm-none-eabi-4_8-2013q4 notice that little squiggly mark after the equal sign? That means that the path starts in your home directory. Ok, so much for the lecture let's get it and put everything in its place.

Code: [Select]
cd ~ && wget -c &&  tar -jxf gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2 &&  rm gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
Note that some of the branches will be looking for a different version of the ARM compiler. You can install multiple versions and the build scripts will search out the preferred version. Let's get 2013-q4 because several of the development branches use that version:

Code: [Select]
cd ~ && wget && tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2
Before doing this next step make sure you are in the directory where you want to save the magic-lantern source code.

Finally, time to grab the ML source code and start building. There are plenty of resources on the forum but basically, to get the latest unified branch:

Code: [Select]
hg clone
cd magic-lantern
hg update unified

We're almost ready to build Magic Lantern but first we've got to make one small change first. Go into the magic-lantern directory, find the file named Makefile.user.default and you'll see this block of code:

Code: [Select]
# Host compiler settings
HOST_CC=$(shell which gcc)
HOST_LD=$(shell which ld)
HOST_AR=$(shell which ar)

Now make a new file called Makefile.user and put in just these lines:

Code: [Select]
# Host compiler settings
HOST_AR=$(shell which ar)

Warning: DON'T EDIT Makefile.user.default -- Put the changes you want to make to Makefile.user.default in your Makefile.user file. Think of this as a way to customize the Magic Lantern build environment without modifying the code.

Another area you might want to customize is which modules to include when making a new build. The modules that the developers decided you probably want are in the modules directory in a file called Makefile.modules.default but maybe you'd like to add another one, take for example bulb_nd. To do this without modifying any Magic Lantern code create a file in the modules directory called Makefile.modules.user and put this in it:

Code: [Select]
Now let's build Magic Lantern. Here's the way I do it for the EOSM:

Code: [Select]
cd ~/magic-lantern/platform/EOSM.202/
make clean && make zip

So you want to start hacking away on the code? There's lots of choices when it comes to text editors. You can even use TextEdit though I wouldn't recommend it because you have to turn off all the "smart" features so it doesn't destroy your source code. A better choice is Aquamacs. I have no financial interest if you get it or not. Hey, it is free. What about the editor that comes with Xcode? You can certainly install the full Xcode app that includes a very capable editor though it is quite a beast to master. It is also quite a heavy download. Of course if you are a real propeller head there's all the old favorites like vi, emacs, pico and nano already installed for you.

Some other software you might want to look into if you are going to get serious and contribute back to the Magic Lantern project is SourceTree. It supports the bitbucket repositories used by Magic Lantern and it has a nice graphical interface showing the various branches in ML. You're on a Mac, not everything has to be done on the terminal.

Added bonus --  Compiling Magic Lantern command line tools a.k.a. "Module Helpers"

The Magic Lantrn source code comes with some applications that you can run in the terminal or incorporate them in your own programs. Both the cr2hdr Lightroom plugin and Danne's cr2hdr-r workflows use--surprise, cr2hdr which is a part of the dual_iso module. There is also a raw2dng, mlv_dump and a few others. Your system is already configured to compile these tools. In fact if you make a build for your camera the Makefile script builds a few of them for you. For example, in the terminal go to magic-lantern/modules/dual_iso and type:

Code: [Select]
We didn't give it any instructions so it simply lists a short list of the available options that it has available. It is beyond the scope of this tutorial how to use the various command line tools but search through the forum and wiki for instructions on how to use the command line tools.

If you find something interesting in the modules you'd like to try out like say raw2dng.c and there isn't a binary for it. Check out the Makefile and see if there is a "rule" to build it. In the case of raw2dng simply type:

Code: [Select]
make raw2dng
and presto, you have a Mac binary to play with.

Extra Added bonus --  Cross compile to Windows

Let you in on a little Magic Lantern secret. Most Windows binaries aren't compiled in Windows. Sure, there are developers using Windows systems but they are either running a Linux distribution inside of a "VirtualBox" or Cygwin which is a self contained environment that functions within Windows. Only a MinGW or MinGW-64 Magic Lantern development setup might be considered Windows native. In fact if you want the best environment to work with open source projects like Magic Lantern set up a Linux system but one issue is that although a Linux system can be easily set up to compile Windows binaries it is not so easy to do Mac binaries. Anyway, to make Windows binaries on a Mac you need yet another gcc compiler like the ones maintained by the MinGW project. MinGW is short for Minimalistic GNU for Windows. What does GNU stand for? GNU is a recursive acronym for GNU's Not Unix. If you check out Apple documentation you will see that they claim that OS-X is Unix. Geek alert - instead of getting caught up in techno-babble how about setting up a cross compiler on your Mac already and start building Windows binaries.

You will need a darwin (OS-X) build of the MinGW compiler. Homebrew to the rescue.

Code: [Select]
brew install mingw-w64
This give you a full 32-bit and 64-bit Windows cross compiling setup ready to go. Now to make a Windows binary all you need to do is add the ".exe" file extension when you invoke make. For example:

make cr2hdr.exe

Being able to create Windows binaries on a Mac is so easy that I am including it on the quick installation script at the bottom of this post, though you'll need to uncomment that line first.

Tip - Removing pesky "._" files from your card

Filenames beginning with a dot are "invisible" and normally you won't even know they are there. Every time you copy files from a Mac file system to another type of filesystem it creates these "dot" files to store some information that is not normally used in other filesystems. If you are using lua on a memory card that is formatted as a FAT16 or FAT32 filesystem (usually cards under 64GB), you may see some strange messages on your screen when your camera starts up with the lua module enabled. It isn't a big deal and dmilligan put in some code to ignore files that begin with a dot but it is rather annoying. Apple actually included a utility to clean up this mess. Assuming your card is named EOS_DIGITAL, here's how to run it:

Code: [Select]
dot_clean -m /Volumes/EOS_DIGITAL
The "-m" option also removes files that start with an underbar.

Tip - Uninstalling Homebrew
Code: [Select]
ruby -e "$(curl -fsSL"You might also want to clean out your /usr/local directory for any stragglers. Just remember that if you are on "El Capitan" or "Sierra" and you delete /usr/local you will have to do the Command-R csrutil enable/disable song and dance to get it back.

Tip - Configuring Apple's Python to work with ML

Note: I don't use this and things might have changed so you're on your own if you want to experiment with this.

Makefile.user.default specifies that PYTHON=python2. Of course you can change that or make an alias for python2 but if you are comfortable disabling System Integrity Protection (SIP) go ahead and do that then:

Code: [Select]
ln -s /usr/bin/python /usr/bin/python2
The reason for installing the Homebrew version of Python is because Apple left out a Python utility called "pip" that makes it easy to install docutils. Follow these steps to install pip and docutils:

Code: [Select]
Code: [Select]
sudo -H python
Code: [Select]
sudo -H pip install docutils
I like to do some cleanup. The docutil scripts are owned by root so let's change that and make sure that we've got permissions for everything in /usr/local:
Code: [Select]
sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
You're done with the
Code: [Select]
Quick installation script.

If you have a Mac with a /usr/local directory and would like to get everything up and running in a hurry, you can use Danne's menu driven compiler tool or download just the quick install script from his repository.

Let's download and run the script using only the terminal. First we'll use curl (copy URL) because it is available on a vanilla Mac. This command copies the script from Danne's repository and saves it to a file named

Code: [Select]
curl >
In order to make the file executable you need to change the file's properties. Here's an easy way to do it:

Code: [Select]
chmod +x
Now it is ready to run. To run it from the current current directory you can start it this way:

Code: [Select]
You'll need to answer a few questions, enter your password and make decisions like if you want to install the Windows cross compiler and/or QEMU.

Happy coding!

Pages: [1] 2