Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Topics - dfort

Pages: [1] 2
1
Maybe I'm making a mountain out of a mole hill but a few cameras have the same address for two different stubs, SoundDevShutDownIn and StopASIFDMAADC.

I made a list to illustrate the problem--and I do think this is a problem.

Cameras that don't have either the SoundDevShutDownIn or StopASIFDMAADC stub

5D2
5D3
60D
500D
550D
600D
1100D

Cameras that have both the SoundDevShutDownIn and StopASIFDMAADC stubs

7D:
NSTUB(0xFF064304,  SoundDevShutDownIn)
...
NSTUB(0xFF061C08,  StopASIFDMAADC)                          // Called by SoundDevStopIn

650D:
NSTUB(0xFF10B0D8,  SoundDevShutDownIn)
...
NSTUB(0xFF1088E0,  StopASIFDMAADC)  <-- Not Listed in stubs.S

700D:
NSTUB(0xFF10B930,  SoundDevShutDownIn)
...
NSTUB(0xFF109138,  StopASIFDMAADC)                          /* present on 7D.203, 6D.113, EOSM.202 */

Cameras that use the same address for both the SoundDevShutDownIn and StopASIFDMAADC stubs

6D:
NSTUB(0xFF11C874,  SoundDevShutDownIn)
...
NSTUB(0xFF11C874,  StopASIFDMAADC)

EOSM:
NSTUB(0xFF10A920,  SoundDevShutDownIn)                      // Temporarily using address for StopASIFDMAADC to resolve MLV_SND issue
...
NSTUB(0xFF10A920,  StopASIFDMAADC)                          // Regular -- Stop ASIF ADC - needed for future changes to mlv_snd.c

100D:
NSTUB(0xFF112680,  StopASIFDMAADC)
...
NSTUB(0xFF112680,  SoundDevShutDownIn)

70D:
NSTUB(0xFF11758C,  StopASIFDMAADC)
...
NSTUB(0xFF11758C,  SoundDevShutDownIn)


Differentiating between these two stubs becomes important in this bit of code in the mlv_snd_stop() function:

mlv_snd.c
Code: [Select]
    /* some models may need this */
    SoundDevShutDownIn();
    audio_configure(1);

Now those platforms that have duplicate addresses are using only StopASIFDMAADC, not the SoundDevShutDownIn. So what happens if we simply do this:

Code: [Select]
/* of course we need to initialize it */
extern WEAK_FUNC(ret_0) int StopASIFDMAADC();
...
    /* some models may need this */
    StopASIFDMAADC();
    audio_configure(1);

That way we could put the real SoundDevShutDownIn addresses in the stubs.S for the 6D, EOSM, 100D and 70D. The question is, what will that do to the 7D, 650D and 700D? Also, where else is SoundDevShutDownIn used?

beep.c
Code: [Select]
#if defined(CONFIG_7D) || defined(CONFIG_6D) || defined(CONFIG_70D) || defined(CONFIG_EOSM)
    /* experimental for 7D now, has to be made generic */
/* Disable Audio */
    void SoundDevShutDownIn();
    SoundDevShutDownIn();
#endif

It is also used in bitrate-6d.c but that section of code is commented out so it doesn't apply.

Seems that now that experimental builds are being compiled and published for anyone willing to test, maybe the crop_rec_4k branch would be a good place to try out a fix? That branch includes the 100D and 70D so it is probably the best place to run an experiment to put the right SoundDevShutDownIn address in the 6D, EOSM, 100D and 70D, modify mlv_snd.c and beep.c to use StopASIFDMAADC and ask testers to report back.

Why fix it if it is working? Because duplicate stub addresses is a rather ugly hack.

Here are a few links to previous discussions of this issue:

http://www.magiclantern.fm/forum/index.php?topic=16040.msg190519#msg190519
https://bitbucket.org/hudson/magic-lantern/pull-requests/863/fix-for-audio-issues-on-eos-100d-possibly/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/657/eosm-code-cleanup/diff#comment-9556325

And here's a bug report that is affected if you use the "right" SoundDevShutDownIn address in the current mlv_snd code:

https://bitbucket.org/hudson/magic-lantern/issues/2255/mlv_sound-working-only-on-the-first-video

By the way, for those of you that didn't understand my subject line, it was taken from a popular TV show named To Tell The Truth. The show first aired in the 1950's where a panel of four celebrities had to correctly identify the contestant (usually with an unusual occupation or experience) from the two imposters.

Then again maybe I chose the imposter and the real SoundDevShutDownIn really is StopASIFDMAADC? That stub is used only once:

Code: [Select]
#ifdef CONFIG_6D
void StopASIFDMAADC();
StopASIFDMAADC(asif_rec_stop_cbr, 0);
#endif

2
General Development Discussion / Porting a Canon firmware update
« on: April 23, 2017, 08:17:23 AM »
Porting a (minor) Canon firmware update


Pretty much every week someone posts something like, "When is Magic Lantern going to be working with X.X.X version firmware?" The answer is usually, "Downgrade to X.X.X -1." Why doesn't ML work with the latest Canon firmware? There are reasons including:
  • There's nothing in the firmware update that you really need.
  • If ML works on X.X.X why take a chance on breaking it?
  • Developers shouldn't waste their time on trivial updates.
  • Because you need a degree in computer science, know ARM processing and spend hundreds of hours reverse engineering to port Magic Lantern.
However, there are also reasons for porting to the latest Canon firmware:
  • Canon recommends using their latest firmware update.
  • The old firmware that ML needs is no longer on the Canon support site so you'll have to get it from a non-official source.
  • Canon may have actually fixed a bug that affects your camera.
  • Because you can.
Let's put the emphasis on you can meaning anyone reading this article should be able to port ML to a Canon firmware update, though it probably shouldn't the first project you attempt. You don't need to know anything about programming or reverse engineering and quite frankly most of what is required to do is grunt work that the main developers aren't really interested in doing. The first thing you need is to set up a development system in order to compile Magic Lantern. There are several ways to do this and there are plenty of tutorials for compiling ML so we won't go into it here.

Note that this article is about doing a relatively minor firmware update like the type Canon released around October of 2016 to fix a problem that affected a couple of lenses. Major updates that add new features or porting to a new camera isn't covered here.

OK--first step is getting ML working on your camera with the supported firmware. This is necessary in order to get firmware dumps, set the camera boot flag and to set the boot flag on the memory cards that you'll be using. You'll find the ROM dumps in the ML/LOGS directory of the card. Save those files. We'll get back to them soon.

You should install ML on a few cards if for nothing else so you'll have cards with the boot flag set. You can also set the card boot flag using a utility like EOScard for Windows and for Mac try MacBoot. I recommend having various different cards with the boot flag set because soon we'll be running something that might not work on certain cards or might corrupt the file system. @a1ex gives a good explanation of what is happening and how to work around it in Reply #1 below. This explains why my trusty 2GB Patriot SD card works like a champ while the much higher performing SanDisk Extreme PRO cards I tried usually failed.

Keep one card without the boot flag in order to do the actual Canon firmware update. Copy both the Canon firmware that is currently working with ML and the version you are upgrading to. That way with one card you can upgrade and downgrade the Canon firmware. If you have several different camera models you can keep all the firmware versions on that one card because the camera will pick out only updates that are valid for that model.

Note that on the 5D3 you can't downgrade from 1.3.X in camera, you need to use EOS Utility. However, what EOS Utility does is to copy the firmware to the card and the camera takes over from there. This process is to discourage users from downgrading. There seems to be a good reason for this because 1.3.X has some added features that might corrupt user settings when downgrading. Simply clearing the Canon settings when downgrading seems to clear up those corruption issues.

Updating the Canon firmware

Here we go--Canon firmware update on card without the boot flag set, check. Update the firmware. Note that this is an article that attempts to cover all ML enabled cameras so I'll be mixing up screenshots from various models.



Dumping the ROM

When you install ML it sets a boot flag on the camera and a boot flag on the card. The camera will always check first to see if the card has the boot flag set and if it is set, attempt to boot off an autoexec.bin file on the card. There are some special autoexec.bin files available on the download page under the heading of ROM dumpers, scroll down the page to find them. You are most like going to need the one labeled Portable. Put it on one of your cards that has the boot flag set and insert the card in your camera. It will probably start working without even turn on the power. You should see something like this:



Or this:



Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. Once again, check out the comment on Reply #1 if you are having trouble getting it to work. The point of this is to get a ROM dump of the new firmware.

If you want to be meticulous about it you should run an MD5 check on your file. When I did it both attempts had the same MD5 value and the dumps matched each other in size so I was confident that I had a good dump to work with. Lazy me. Open up a terminal (Powershell on Windows), navigate to where your ROM dumps and MD5 files are saved and:

Linux

For Linux and probably Cygwin on Windows:
Code: [Select]
md5sum -c *.BIN
Windows

For Windows 8 or higher (thanks Walter):
Code: [Select]
powershell "get-filehash *.BIN -A MD5 | format-list"
Macintosh

A stock Macintosh doesn't come with md5sum installed so you can either create the md5 checksum and compare it with the one generated by the portable rom dumper:
Code: [Select]
md5 ROM1A.BIN
MD5 (ROM1A.BIN) = 8a4d0fbfa6e6759d41b833bf764748fb
cat ROM1A.MD5
8a4d0fbfa6e6759d41b833bf764748fb  ROM1A.BIN
Or you can install md5sum. I use Homebrew:
Code: [Select]
brew install md5sha1sumThis version of md5sum couldn't check all of the dumps at once:
Code: [Select]
md5sum -c *.MD5
Error: --check <filename> cannot be used with additional files
So you'll have to do them one at a time:
Code: [Select]
md5sum -c ROM1A.MD5
ROM1A.BIN: OK

Now you've got several sets of verified ROM dumps, make sure you know which one is which. In most cases you'll only need to work with the ROM1.BIN for the Canon firmware that you started from and the one for the Canon firmware you're porting to.

Congratulations, you just completed the most essential step towards porting Magic Lantern!

If you're in a hurry, curious, stupid or all of the above you'll be tempted to try your current version of ML on the new firmware. Guess which description I fit into?



Disassembly

There are various methods of disassembling the ROM dumps in order to work with them. Everyone seems to have their favorite method. ARMu is quick but only works on Windows--ok it also works on Mac under wine but I haven't figured it out yet. ARM-console is very difficult to set up and use but it is very powerful--this currently my favorite disassembler. There is also a commercial product called IDA but in order to disassemble ARM code you need to buy a license. By far the easiest for beginners is disassemble.pl which is a perl script that needs just a few tweaks to get running properly as explained in this excellent tutorial on finding stubs.

What about QEMU? That could be very useful, especially when porting major firmware updates and new cameras but we're doing just a minor firmware update that can be done with just a few basic tools and a lot of perseverance.

Now it might be tempting to upload and share these ROM dumps and disassemblies but DON'T DO IT! You're holding copyrighted material that is owned by a very large and powerful corporation. You bought the camera so you can look at what is inside it but you aren't supposed to publish "intellectual property" in an open forum like this one. I'll be editing the bits in this post so they don't perfectly match the real disassembly but it should be good enough for instructional purposes.

So let's do this. Assuming disassemble.pl is in the same directory as your ROM1A.BIN file:
Code: [Select]
perl disassemble.pl 0xFF000000 ROM1A.BIN
string dump
create elf file
label scan
0xffff5a4c 
disassemble and string lookup
0xfffffffc 
job complete!

The file you are interested in is ROM1A.BIN.dis. You'll be spending a lot of time together.

Preparing your local repository for the update.

There are lots of ways to work with your local repository. It really depends how you are most comfortable. If you plan to eventually do a pull request you should consider creating a "scratch" or development branch where you can experiment and post all your notes and then when everything is working, copy over your changes to a fresh branch to avoid cluttering up the revision history with trivial commits.

Take a look at some recent firmware updates so you'll know what you are getting yourself into:

EOS M 2.0.2 - 2.0.3
700D 1.1.4 - 1.1.5

This one is somewhat different because it doesn't replace any previous firmware version and is a bit more complex:

5D3 1.3.4

The first thing you'll notice is that it seems that we're throwing out the old and creating new files. That's not really the case, we're renaming the old files but don't do it with the operating system because it defeats the point of keeping everything under version control. Generally I like to work with the SourceTree application but copying directories is one of those things that is probably best done on the command line. There are two directories and one file that needs to be renamed. For example, when going from 1.1.4 to 1.1.5 on the 700D:

The platform directory
Code: [Select]
cd magic-lantern/platform
hg rename 700D.114 700D.115

The installer directory
Code: [Select]
cd magic-lantern/installer
hg rename 700D.114 700D.115

Editing the Source Code

Now for the good news. This step has registered a lot of changes but you'll only need to modify a few of these files. Most of the changes will be happening in the stubs.S file. While it is challenging looking for new stubs or finding stubs for a new camera model, a minor firmware update is rather easy.

Let's start with a very easy change. In magic-lantern/installer/Makefile you'll see this line:

Code: [Select]
SUPPORTED_INSTALLERS := 1100D.105 500D.111 50D.109 550D.109 5D2.212 5D3.113 60D.111 600D.102 650D.104 6D.116 700D.114 EOSM.202 7D.203

Pretty easy to figure out what to change on that line. However, there's something to watch out for when editing a Makefile, you got to make sure that your text editor doesn't change the indents from tabs to spaces. A Makefile needs those tabs.

Now check the pull requests on bitbucket and you'll see several other places where you need to change from the old to the new firmware version. Don't forget to make changes in the modules: adtg_gui, dual_iso, mlv_rec and mlv_lite.

Another rather easy change to make is to remove the old ML-SETUP.FIR file. It won't work with the new firmware and a developer will have to create a new one for you. In the case of a minor firmware update if you did the Canon update with the boot flag set on the camera, you won't have any problems later when you test out your new port. Just don't rush it! There are a few things that need to be done in the right order before attempting to boot into your newly ported ML.

In fact if you want to be extra careful you could take the advice that is in property.c and disable the prop_request_change function. If you're a suspenders and belt kind of guy also go into your platform's internals.h and comment out '#define CONFIG_PROP_REQUEST_CHANGE'. This is highly recommended for new ports because you can do some serious damage. I didn't take that precaution because mine were very minor updates.

Finding where the stubs moved

Now we get into the nitty gritty of finding stubs. We aren't porting a new camera, just a minor firmware update so all the stubs that you should be able to easily look up in the old disassembled ROM1.BIN disassembly should also be in the new disassembly--only in a different location.

If you're working with a DIGIC V the first thing you will notice in the stubs.S file is something like this (700D.114):

Code: [Select]
#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...

There is a very good explanation of what this RAM_OFFSET is all about in a1ex's Tutorial: finding stubs. In this case Nanomad left a clue so let's check out what line 0xff0c0098 says in the 700D.114 disassembly. Depending which disassembler you used it might look like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; pointer to sub_FFA5E8B0
ff0c009c: e59f1048 ldr r1, [pc, #72] ; 0xff0c00ec: pointer to 0x1900

or like this:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b0)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Now let's look at what the 700D.115 disassembly looks like:

Code: [Select]
ff0c0098: e59f0048 ldr r0, [pc, #72] ; ff0c00e8: (ffa5e8b8)
ff0c009c: e59f1048 ldr r1, [pc, #72] ; ff0c00ec: (00001900)

Hooray! RAM_OFFSET changed from 0xFFA5E8B0-0x1900 to 0xFFA5E8B8-0x1900.

Now start going through the rest of the stubs.S file. One trick I discovered is that if you're tackling one of those minor updates Canon pushed in October of 2016, any stub that doesn't use RAM_OFFSET remains unchanged and those with the offset move the same amount in the same direction that RAM_OFFSET changed--for the most part. A few will move. Why? Because there was a few lines of code that were changed somewhere that shifted things around slightly. In any case you should check the address to every stub to make sure. In some cases there's a descriptive string around the stub you're checking into, other times there is nothing to go by except if several lines "look" the same. It becomes sort of a pattern matching game like those cartoons that you try to find the difference between two drawings that look almost identical.

What about those stubs that aren't addresses?

Code: [Select]
/** Task info **/
NSTUB(   0x25024,  task_max)
NSTUB(0xFFA0A0D8 - RAM_OFFSET,  is_taskid_valid)            // AJ_task_trampoline_related_p10
NSTUB(   0x23E14,  current_task)
NSTUB(     0x674,  current_interrupt)                       // in interrupt handler (0x18), where MEM(C0201004) is stored

You can take a chance that it didn't change but it is best to look at the hints, search the forum, read the wiki or post a question how to find these values. Leave those challenging ones until the end because by the time you've gone through all of the other stubs you'll become more of an expert at sleuthing out those stubs.

There are some helpful scripts in the magic-lantern/contrib directory. Try running check-stubs.py to get a report on the differences between the old and new stubs.S files. I posted my results on the 700D.115 pull request.

Make sure you got all of the changed addresses

Not all of the addresses you need to check are in the stubs.S file. The source code for some of the modules also have references to firmware versions and/or addresses that need checking and probably changing. For example, adtg_gui.c, dual_iso.c, mlv_rec.c and mlv_lite.c. By now your pattern matching skills are probably pretty good so good luck hunting them down.

Getting the firmware signature

The next step involves something a little more exciting that pouring over endless lines of disassembly code. You'll need to find the new firmware's signature and for that you'll need to build Magic Lantern, but this isn't a working build--not yet.

Take a look at fw-signature.h. Your updated firmware version is probably not in the list. Now check out boot-hack.c and you'll find this near the beginning of the file:

Code: [Select]
#if defined(CONFIG_HELLO_WORLD)
#include "fw-signature.h"
#endif

That's right, we're going to welcome your firmware update to the world. In order to do that open up config-defines.h and look for this piece of code:

Code: [Select]
/**
 * Enable these for early ports
 */

    /** If CONFIG_EARLY_PORT is defined, only a few things will be enabled (e.g. changing version string) */
    //~ #define CONFIG_EARLY_PORT

    /** Load fonts and print Hello World (disable CONFIG_EARLY_PORT); will not start any other ML tasks, handlers etc. */
    //~ #define CONFIG_HELLO_WORLD
   
    /** Create a developer FIR for enabling the bootflag and dumping the ROM. */
    //~ #define CONFIG_DUMPER_BOOTFLAG

Uncomment the hello world line:

Code: [Select]
#define CONFIG_HELLO_WORLD
Now build Magic Lantern, put it on one of your boot flag enabled cards, insert it into your camera and you should see something like this:



Write it down, put that address in fw-signature.h, comment out '#define CONFIG_HELLO_WORLD' and...

Trying it out

Ok so it should work but something will probably go wrong the first time you try it out. On the 700D.115 some of the fonts were missing. Searching for fonts I found this:

consts.h
Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xFFCF67E4
#define BFNT_BITMAP_OFFSET 0xFFCF972C
#define BFNT_BITMAP_DATA   0xFFCFC674

So it took a bit more effort to find those addresses but magic-lantern/contrib/indy/find_fnt.py found them automatically.

On the 5D3.134 there were some other things that needed attention in consts.h. You would think that constants by definition don't change but like the old saying goes, "the only thing constant is change."

Oh, one last thing--the ML-SETUP.FIR file. Usually that's one of the first things that you need if you're working on a new port but since we're only doing a minor firmware update we can wait until the end. Only one of the main developers can prepare this file for you so you'll have to ask.

3
Scripting API suggestions / Feature Request: lua hook for audio events
« on: March 03, 2017, 06:19:37 AM »
Hope this is the right place to ask for this lua feature request.

There doesn't seem to be any audio functions in the lua API.

I'm mostly interested in running AUDIO_REMOTE_SHOT in lua to trigger either a full shutter press or short and long shutter half-press on an EOSM. The EOSM doesn't have the option of using a wired remote so shutter events have to be done either through the IR remote or the mic input. The lua script should be able to start a process (like a shutter half-press or rack focus or whatever) from an audio cue.

Another advantage of having lua do the shutter half-press is that it resets the power savings timer while AUDIO_REMOTE_SHOT doesn't reset the timer. At least that's the experience with a film scanner project that one user is trying to set up with an EOSM.

http://www.magiclantern.fm/forum/index.php?topic=19005.0

4
Camera-specific discussion / Canon 5D Mark III / 5D3 / Firmware 1.3.4
« on: February 15, 2017, 02:04:51 PM »
Posted a pull request for 5D3.134 firmware support.

Why?
  • Because Canon tells us to update our cameras to their latest firmware release.
  • Once you do update Canon says you can't go back (though you can, really).
  • Canon removes prior firmware revisions from their product support web pages so you need to download from unofficial sources in order to use Magic Lantern.
  • There are 5D3 owners out there that might want to try ML but are unwilling to rollback the firmware installed on their cameras.
  • Because it was a challenge and seemed like a worthwhile thing to do.
Ok, that last point was just to make a point--you don't need to be an experienced developer to contribute to Magic Lantern. In fact doing a minor firmware update is a good project for someone who is able to compile the ML code and is willing to put in some time searching through a ROM disassembly looking for changes. It is also a good project to learn some very basic reverse engineering.

What this version is not:
  • If you want the most stable ML experience and don't need the new firmware features, stick with 5D3.113.
  • This update will not replace the 113 and 123 versions--it can live right alongside the previous versions.
With the recently added option to support more than one firmware revision per platform in the ML source code this might be a good opportunity to add experimental releases closer to the Canon firmware revision release dates. Provided of course that it passes the ML stress tests and that enough people test and approve the pull request.

To make testing easier for users to try it out, I'll be posting builds to my bitbucket repository download area until it gets into the Experiments download page. Note that this first very early build doesn't have the ML-SETUP.FIR file yet so make sure your camera bootflag is set before doing the firmware update. To go back to 113 or 123, follow the instructions on the Firmware Update/Downdate? tutorial.

Big thanks to:
  • a1ex for his guidance and encouragement and keeping this project alive.
  • chris_overseas who did the 5D3.133 port which had the last few missing pieces to get this working.
  • dmilligan for his patience answering my noob questions.
  • Danne and DeafEyeJedi my ML buddies.


Just as a footnote: Firmware revision 1.3.4 was released November, 2016 and the 5D300134.FIR file is timestamped September 1, 2016 while the 5D300123.FIR has a timestamp of August 30, 2013 and the 5D300113.FIR has a timestamp of May 28, 2012.


5
General Development Discussion / 5D3.134 Stubs API test errors
« on: February 14, 2017, 08:15:35 PM »
Hope I'm not overstaying my welcome here but this seems like this is so close. I've been running the Stubs API test and consistently getting 200 errors, but they are all the same error.

Code: [Select]
[FAIL] ABS((m0-m1) - 50*1024) => 0xc800
It seems to be happening in this selftest function:

Code: [Select]
static void stub_test_malloc_n_allocmem()
{
    // mallocs
    // bypass the memory backend and use low-level calls only for these tests
    // run this test 200 times to check for memory leaks

It maybe pointing to a problem with the memory allocation stubs but I triple checked them against the 123 and the 113 disassemblies and they seem fine, they also pass the check-stubs.py test.

Any idea where the problem might be at?

6
General Development Discussion / Oh where oh where have my fonts gone?
« on: February 14, 2017, 01:09:34 AM »
I've been playing around with getting ML working on the 5D3.134 firmware and have gotten quite far but am stumped trying to find the addresses for the fonts.

According to the 5D3.123 code:

Code: [Select]
// http://magiclantern.wikia.com/wiki/Fonts
#define BFNT_CHAR_CODES    0xF7363A50
#define BFNT_BITMAP_OFFSET 0xF7366BC4
#define BFNT_BITMAP_DATA   0xF7369D38

Ok, been there before on the 700D firmware update and treated these just like finding stubs but these addresses aren't in the ROM1.BIN disassembly. I looked up the wiki reference and it pointed to the find_fnt.py script which is in the contrib/indy directory and it needs to be run on ROM0.BIN. Ok, so far so good. This is what I got this for the 5D3.123:

Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xff373a2c: FNT
0xff373a30: (+0x04) 0xffd8
0xff373a32: (+0x06) font_width = 40
0xff373a34: (+0x08) charmap_offset = 0x24
0xff373a38: (+0x0c) charmap_size = 0x3174
0xff373a3c: (+0x10) bitmap_size = 0x89f16
0xff373a40: (+0x14) font name = 'HCanonGothic///'
0xff373a50: (+0x24) char_codes[]. 3165 chars
0xff376bc4: (+0x3198) offsets[]. Last offset value = 0x89ee0
0xff379d38: (+0x630c) bitmaps[]
  0xff403c18: (+0x901ec) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xff403c50: FNT
0xff403c54: (+0x04) 0xffd8
0xff403c56: (+0x06) font_width = 40
0xff403c58: (+0x08) charmap_offset = 0x24
0xff403c5c: (+0x0c) charmap_size = 0x188
0xff403c60: (+0x10) bitmap_size = 0x31c4
0xff403c64: (+0x14) font name = 'CanonMonospace'
0xff403c74: (+0x24) char_codes[]. 98 chars
0xff403dfc: (+0x1ac) offsets[]. Last offset value = 0x3142
0xff403f84: (+0x334) bitmaps[]
  0xff4070c6: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

0xffb73a2c: FNT
0xffb73a30: (+0x04) 0xffd8
0xffb73a32: (+0x06) font_width = 40
0xffb73a34: (+0x08) charmap_offset = 0x24
0xffb73a38: (+0x0c) charmap_size = 0x3174
0xffb73a3c: (+0x10) bitmap_size = 0x89f16
0xffb73a40: (+0x14) font name = 'HCanonGothic///'
0xffb73a50: (+0x24) char_codes[]. 3165 chars
0xffb76bc4: (+0x3198) offsets[]. Last offset value = 0x89ee0
0xffb79d38: (+0x630c) bitmaps[]
  0xffc03c18: (+0x901ec) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xffc03c50: FNT
0xffc03c54: (+0x04) 0xffd8
0xffc03c56: (+0x06) font_width = 40
0xffc03c58: (+0x08) charmap_offset = 0x24
0xffc03c5c: (+0x0c) charmap_size = 0x188
0xffc03c60: (+0x10) bitmap_size = 0x31c4
0xffc03c64: (+0x14) font name = 'CanonMonospace'
0xffc03c74: (+0x24) char_codes[]. 98 chars
0xffc03dfc: (+0x1ac) offsets[]. Last offset value = 0x3142
0xffc03f84: (+0x334) bitmaps[]
  0xffc070c6: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

So now I'm more lost than ever. Where are the character codes, bitmap offset and bitmap data and how did someone come up with values of around 0xF736xxxx?

By the way, here's what find_fnt.py is reporting on the 5D3.134 ROM0.BIN file:

Code: [Select]
Find bitmap fonts in Canon DSLR firmwares
Arm.Indy. based on work by Pel, Trammel Hudson and A1ex

0xff373a2c: FNT
0xff373a30: (+0x04) 0xffd8
0xff373a32: (+0x06) font_width = 40
0xff373a34: (+0x08) charmap_offset = 0x24
0xff373a38: (+0x0c) charmap_size = 0x3180
0xff373a3c: (+0x10) bitmap_size = 0x8a18c
0xff373a40: (+0x14) font name = 'HCanonGothic///'
0xff373a50: (+0x24) char_codes[]. 3168 chars
0xff376bd0: (+0x31a4) offsets[]. Last offset value = 0x8a156
0xff379d50: (+0x6324) bitmaps[]
  0xff403ea6: (+0x9047a) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xff403edc: FNT
0xff403ee0: (+0x04) 0xffd8
0xff403ee2: (+0x06) font_width = 40
0xff403ee4: (+0x08) charmap_offset = 0x24
0xff403ee8: (+0x0c) charmap_size = 0x188
0xff403eec: (+0x10) bitmap_size = 0x31c4
0xff403ef0: (+0x14) font name = 'CanonMonospace'
0xff403f00: (+0x24) char_codes[]. 98 chars
0xff404088: (+0x1ac) offsets[]. Last offset value = 0x3142
0xff404210: (+0x334) bitmaps[]
  0xff407352: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

0xffb73a2c: FNT
0xffb73a30: (+0x04) 0xffd8
0xffb73a32: (+0x06) font_width = 40
0xffb73a34: (+0x08) charmap_offset = 0x24
0xffb73a38: (+0x0c) charmap_size = 0x3180
0xffb73a3c: (+0x10) bitmap_size = 0x8a18c
0xffb73a40: (+0x14) font name = 'HCanonGothic///'
0xffb73a50: (+0x24) char_codes[]. 3168 chars
0xffb76bd0: (+0x31a4) offsets[]. Last offset value = 0x8a156
0xffb79d50: (+0x6324) bitmaps[]
  0xffc03ea6: (+0x9047a) last bitmap
  +0x00: bitmap width = 28
  +0x02: bitmap height = 28
  +0x04: char width = 36
  +0x06: X offset = 4
  +0x08: Y offset = 16
    bitmap size = 0x70

0xffc03edc: FNT
0xffc03ee0: (+0x04) 0xffd8
0xffc03ee2: (+0x06) font_width = 40
0xffc03ee4: (+0x08) charmap_offset = 0x24
0xffc03ee8: (+0x0c) charmap_size = 0x188
0xffc03eec: (+0x10) bitmap_size = 0x31c4
0xffc03ef0: (+0x14) font name = 'CanonMonospace'
0xffc03f00: (+0x24) char_codes[]. 98 chars
0xffc04088: (+0x1ac) offsets[]. Last offset value = 0x3142
0xffc04210: (+0x334) bitmaps[]
  0xffc07352: (+0x3476) last bitmap
  +0x00: bitmap width = 22
  +0x02: bitmap height = 22
  +0x04: char width = 22
  +0x06: X offset = 0
  +0x08: Y offset = 0
    bitmap size = 0x42

So close yet so lost!

7
Tutorials and Creative Uses / Firmware Update/Downdate?
« on: February 11, 2017, 02:58:02 AM »
How to do a Canon firmware downgrade
(works for upgrade too)


There seems to be a lot of FUD (Fear Uncertainty and Doubt) about changing the firmware on your camera, especially for uninitiated Magic Lantern users who are pointed to a website in Hungary run a guy who goes by the name of Pelican to get older firmware versions. Nothing against Hungary (I'm half-Hungarian) or Pelicans (I live at the beach) but a little anxiety is understandable.

Once you realize that it is extremely unlikely that someone created a FIR file that will cause your camera to send banking information to international cyber terrorists, download the file onto an SD card.

Firmware updates are generally posted in some sort of compressed format often along with the instructions on how to run the firmware update. If you are on a Mac and can only find a Windows version, don't panic. The *.FIR file runs on the camera so it doesn't matter which computer platform you are using. Of course you might download something that looks like this on your Mac:


Again, don't panic. This is a self extracting Windows archive and you can use Stuffit Expander which is in the Applications/Utilities folder of every Mac to expand the archive file.


I've got several cameras and like to play around with different versions of Magic Lantern so I put all of my FIR files on a single card. Make sure that the card doesn't have the boot flag enabled because if your camera also has the boot flag enabled it will just hang when you start up the camera unless you also add a ML autoexec.bin file and then it won't work on multiple platforms. In other words, this will be your dedicated firmware card.


Those CCF14* firmware files are for the 700D/T5i, yeah I know it isn't obvious. And just in case you're thinking it, no the 5D3 firmware won't install on the EOSM. Only the valid choices will show up.

Now when you run the firmware update from the Canon menu using this card, remembering to have the camera in Manual and Still Picture mode, it will run the firmware update process--even if you are doing a firmware downdate. (Is there such a word?) The firmware update will not run if you don't have enough juice left in the battery and some external power adapters supply barely enough voltage to run the camera so it might not work. Best be safe and top off your battery first.

End of tutorial.

Not!

Firmware versions in the x.3.x series (currently only affecting 5D3) won't let you downgrade to a prior series. In other words, you can go from 1.3.4 to 1.3.3 but if there is a 1.2.3 or a 1.1.3 firmware on your card you'll get this message:


It is not quite that easy with some of the latest Canon firmware updates. There are warnings about not being able to downgrade and rumors that you need some older 2.x version of Canon EOS Utility to do the downgrade but I found that is not really the case. If you happen to have one of those irrevocable firmware versions on your camera you can use whatever version of EOS Utility you have. Make sure to put a copy of the FIR file on your computer because you'll have to point to it with EOS Utility. Also make sure your camera is in Manual and Still Picture mode or EOS Utility will refuse to do the firmware update.


HA! Just noticed that "WTF Captions" item. Who says Canon engineers don't have a sense of humor?

Ok--back to doing this. You need to have a card in your camera so let's use that dedicated SD card with all the various versions. Here we go:



Hey, what just happened? EOS Utility just transferred the firmware update from your computer onto the card in your camera and you can now disconnect the USB cable. The rest is exactly the same as before. In fact if you have multiple firmware versions on your card you can now have a second chance to decide which version you want to install.


Interestingly the firmware update will not change the status of the camera boot flag so you won't have to re-install ML. If you are curious what a new firmware version looks like, use the Portable ROM dumper and run the ROM1.BIN file through a disassembler and...I'm getting off topic here, porting a firmware update to ML is another tutorial.


8
General Development Discussion / Running ARM-console on a Mac
« on: December 16, 2016, 01:12:24 AM »
I'm following these instructions:

http://magiclantern.wikia.com/wiki/Magic_Lantern_Development_on_Mac

Seems like I'm so close but stuck on this:

Code: [Select]
http://magiclantern.wikia.com/wiki/GPL_Tools/ARM_console
You might want to do this:
    D = load_dumps("autoexec")
------------------------------------------------------------
Traceback (most recent call last):
  File "main.py", line 8, in <module>
    console.ipshell()
  File "/usr/local/lib/python2.7/site-packages/IPython/Shell.py", line 242, in __call__
    self.IP.embed_mainloop(banner,local_ns,global_ns,stack_depth=1)
  File "/usr/local/lib/python2.7/site-packages/IPython/iplib.py", line 1836, in embed_mainloop
    self.set_completer_frame()
  File "/usr/local/lib/python2.7/site-packages/IPython/iplib.py", line 1267, in set_completer_frame
    self.Completer.namespace = self.user_ns
AttributeError: 'InteractiveShell' object has no attribute 'Completer'

Any hints or should I just fire up the old Linux laptop for this?

9
Is there a reason that it isn't assigned to "0" with manual lenses? I'd like to have the focal length tag in MLV files default to "0" when using manual lenses instead of 50mm.

I looked all over the code and can't figure out where lens_info.focal_len is assigned a value of 50 when a manual non-cpu lens is attached to the body. I did find a place where it is reassigned "0" but it seems like it is there in case a chipped lens reports bogus values:

lens.c line 1533
Code: [Select]
    if (lens_info.focal_len > 1000) // bogus values
        lens_info.focal_len = 0;

Am I'm missing something obvious somewhere?

10
Feature Requests / Update mlv_dump
« on: October 08, 2016, 03:52:05 PM »
mlv_dump is one of the Magic Lantern stalwart apps and is still useful in many situations although many of the functions have been improved in dmilligan's MLVFS. Some of the "features" in MLVFS could be considered bug fixes that should be implimented in mlv_dump.

I opened an enhancement issue on bitbucket, Issue #2609

First baby step is to change the way dng files are named so that it conforms to MLVFS naming. The justification is because there should be some consistency between ML apps.

Pull request: https://bitbucket.org/hudson/magic-lantern/pull-requests/758/simplify-dng-names/diff

11
Hi fellow Magic Lanterners. I'm working on a short film shot on a 5D3 and standard 1920x1080 MLV. I created proxy files using Danne's MLP. After editing the ProRes proxy files in Premiere I thought I'd try doing the color grading in After Effects linking back to the DNG files. Why AE instead of DaVinci Resolve? Partially because I'm more familiar with AE and also because I'm doing some effects work that I'm more comfortable doing in AE. Anyway--everything was going fine following this tutorial:


Except that after interpreting the footage from 30 to 23.976 it was off a frame or so between the QuickTime file from the Premiere timeline to the DNG image sequence. I'd really like to get this working properly.

By the way, I started out going from MLV to DNG then to a log ProRes file and do the LUT, look and color grading thing on the log ProRes but that seemed to lose far more detail and limits what you can do compared to working with the DNG files.

Here's what my ProRes file looks like -- not much I can do with it before it starts falling apart.



From the DNG I brightened it up, pulled out some blue in the window softened it up and added grain all in ACR.



Again from the DNG, a grittier look also in ACR.


12
General Chat / Timecode for ML Users (and abusers)
« on: March 24, 2016, 06:09:06 PM »
In the late '90's a friend working as a post production supervisor at Sony Pictures called to hash over a problem. He had a project that was being shot on the brand new Sony DVCAM format. The idea was to shoot four different stories with four cameras without stopping and put them all on the screen at the same time. The plan was to shoot just one take and make no cuts. Problem was, the cameras kept drifting and the timecode wouldn't match up so they were trying to get it to work by cutting out a frame here and there to get it to sync--they were also on their second week of shooting. They finally resolved it on the sixteenth take. The name of the movie?

Timecode

So if a major studio with the resources of a giant multi-national corporation had problems with timecode there's a good chance you're going to encounter some issues too. Let's get started.


I borrowed this image from an old post on the editingstandards, a long abandoned site started by a couple of very talented assistant editors. I'll keep referring back to it.

Recently I've been having conversations with some Magic Lantern users and developers and it seems that most people here already have a basic understanding of what timecode is (a standard maintained by the Society of Motion Picture and Television Engineers), how it is displayed (hours:minutes:seconds:frames) and some of the ways timecode might be used in an MLV workflow so I won't bore you with the basics. Instead let's jump into the areas where most people seem to having difficulties.

Look at two timecode displays in the upper right of the image. Notice one is in 24fps and is called "Record Time" while the other is counting in 30fps and is called "Tape Time." Now a quiz--what fractional second is the timecode displaying? Well the 30fps timecode is showing 10 frames and the 24fps timecode is showing 8 frames so 10/30 or 8/24 will give you the answer--0.33333... seconds.

This sample was originally shot on 35mm film at 24fps and transferred to NTSC 29.97fps videotape in a telecine bay using a 3:2 pulldown. The timecode on the videotape is non-drop frame. I'll get into why we don't use drop frame timecode a little later but first look at the extra number to the right of the video timecode. It is both a 1 and a 2 interlaced together. These numbers are burned into the separate video fields identified as upper/lower or field 1 and field 2. There are other ways that these fields can be displayed without having to add extra characters to the timecode display. One popular method is to use a period as the delimiter between the seconds and frames for field 1 and a colon for field 2 so in the above example one field would be 01.10 and the next 01:10.

Our cameras have a setting that allows us to shoot either 50p or 60p (actually 59.94) the "p" standing for progressive or full frames so they are not interlaced (limited to 1280x720 on ML supported cameras and up to 1920x1080 the 7D Mark II). So--it is understandable if two fields make up one frame then that one frame is identified as a frame in the timecode but why isn't every progressive frame identified as a frame in the timecode? Some developers think that because we're in the digital age we should be able to change timecode to accommodate this higher framerate material and indeed some systems do stray from the SMPTE specifications but there are reasons why the standard that was established about 50 years ago hasn't changed yet. One reason is because the broadcast signal is still being interlaced. Now I'm not an engineer and have not worked in broadcasting so you might want to run a fact check on that. The bottom line is that even when we work with 50p and 60p media, we have to stick with 25 and 30fps timecode.

13
General Development Discussion / bitbucket protocol
« on: February 23, 2016, 07:18:31 PM »
I've got a question for the developers who have write and administrative permissions on the Magic Lantern Trammell Hudson bitbucket account.

What is the protocol for getting something merged into the unified branch so that it will be pushed to the nightly builds?

The reason I'm asking is because lately there has been lots of updates on branches and when an important issue is resolved in one of these development branches when does it get merged into unified? We have a situation right now where an issue with the metadata for some cameras wasn't being saved properly in MLV files. That issue was resolved in the mlv_prop_data branch but it was not merged into unified.

Now we have MLV Lite available for testing which is in a development branch called new-raw-format which understandably shouldn't be merged into unified until it is well tested but anyone testing that module with cameras that show focus pixels in raw video cannot test MLV Lite with the focus pixel removal tools recently implimented in MLVFS and MLP without using a custom build from the mlv_prop_data branch. This negates the convenience of dropping in new modules into a nightly build for testing, especially for users that are not able to compile the ML source.

There are also a couple of development branches for the lua module, lua_fix and lua_touch, which are resolving issues with the lua module. At what point are these going to be merged into unified? It seems to me that lua is mainly for experimenters and having frequent updates would be welcome for users who are writing lua scripts.

I've been posting comments on bitbucket about some of these issues and I certainly don't want to continue posting and possibly annoy developers. I'm just asking what is the protocol for requesting that a branch be merged into unified?

https://bitbucket.org/hudson/magic-lantern/issues/2460/700d-t5i-mlv-idnt-metadata-incorrect
https://bitbucket.org/hudson/magic-lantern/commits/3a9c6deedafd66d093b60abc4cc31492f1d87d17#general-comments
https://bitbucket.org/hudson/magic-lantern/pull-requests/684/mlv_rec-eliminate-propad_getpropertydata/diff

14
Feature Requests / mv1080 on EOSM
« on: February 06, 2016, 04:56:46 PM »
The EOSM cannot currently record in mv1080 mode though it is within the capability of the camera and it seems to be a "possible" Magic Lantern feature request.

I'm wondering if there is any possible way to get mv1080 on the EOSM but I take it that has been tried before.
It wasn't tried too hard ;)

Looking a bit in forum history, it seems like in the very first implementations of raw video, there was a (undocumented) way to do it: http://www.magiclantern.fm/forum/index.php?topic=5860 and http://www.magiclantern.fm/forum/index.php?topic=12022

Maybe I should revisit it, but that would be a different topic.

a1ex explained that there are other ways to capture raw video. Perhaps one of these alternate methods can capture the data after it has gone through the focus pixel annihilation process?
I wouldn't really say there are "other ways" to capture raw video, there are just other possible video modes. Capturing them is the same, you just have to figure out how to get Canon firmware to put the camera in those modes. The EOSM is unique b/c Canon firmware doesn't put it into mv1080 until you start recording (H.264). So if you want to record raw video on mv1080, you have to do it while also recording H.264 at the same time (ML prevents you from doing this now a days, but it was possible in some old versions when raw video just came out). That or do some reverse engineering to figure out how to put the camera in mv1080 without starting recording. All other cameras go into the video mode you selected in the Canon menu immediately, so actually mv1080 is pretty much the "default".

Let me try to sell this feature request by stating some of the reasons I believe why the lowly EOS-M is perhaps just as important to moving Magic Lantern forward as the 5D mark III.

The camera is the least expensive of all the ML capable cameras which makes this the ideal model to experiment without being too concerned about bricking an expensive piece of photographic equipment. It is so affordable that multiple bodies can be had for less than the cost of higher end models. In fact you can get 10 EOSM bodies for the price of one 5D3--I myself have 4 bodies, yeah I'm crazy.

At first the EOSM may seem radically different from the other Canon models because it is a mirrorless but the internals are very similar to the DSLR's. The advantages  include not having to lock up a mirror to get into Live View and a shorter lens flange distance so many lenses can be adapted for use on this camera.

Canon EOSM with C-mount lenses

Though the EOSM may feel like a toy compared to the DSLR's it actually has quite a rugged metal body and can stand up to the riggers of a documentary project I'm working on. I planned to use it as a spare body for a 5D3 setup but it proved capable enough as the 'A' camera. (When I was a professional still photographer back in the day I would never consider going on a job without a backup camera body.)

EOSM documentary rig

The EOSM can also be an ideal replacement for a director's viewfinder and with a PL adapter the camera can be used to look through professional cine lenses. Adding mv1080 also gives the capability of shooting much higher quality tests than with H.254 video.

EOSM with PL mount adapter

Adding mv1080 on the EOSM will also elevate this platform to the same level as the rest of the Magic Lantern capable cameras though filmmakers may want to dress it up a bit in order to be taken seriously.

EOSM Shoulder Rig with Follow Focus

There are disadvantages to the EOSM. One problem for video shooters is the short battery life though extra batteries are very affordable and it is very simple to rig up external power supplies. Another problem is that it responds slower than other cameras in still photo mode though it is just as quick as the other cameras in video mode. Speaking of still photo mode, there's that pesky shutter bug when using Magic Lantern but that's a different topic.

15
Reverse Engineering / Reverse Engineering Picture Styles
« on: December 07, 2015, 05:50:39 AM »
Ok--here we go:

2016-09-07 v1.4 - Test version to improve highlights detail. (Don't use if you're using zebras to judge exposure.) Dropped logStandard and logFaithful.
2016-09-02 v1.3 - Overall better match
2016-09-01 v1.2 - Combined highlights from v1.0 with shadows from v1.1 and Saturation now defaults to -2
2016-08-31 v1.1 - Small adjustments to correct color shifts in shadows
2016-08-31 v1.0 - Initial release

This was a joint effort between Danne and myself, dfort. For the first release we did the best we could at matching Technicolor CineStyle. The reason we picked CineStyle is because it is in wide use, hasn't been updated since it was initially released in 2011 and it is a .pf2 that cannot be editing using Canon's Picture Style Editor. Ours is in the newer .pf3 file format and is editable. We didn't even put a copyright notice on it so feel free to do with it as you please.

We did a lot of testing and have a fairly high degree of confidence that this can be a direct replacement for CineStyle. This means that all of the luts available for CineStyle can be used. Is it a perfect copy? No, but it is the best we could do using the tools we had at our disposal. Visually it should be impossible to see any difference between CineStyle and logNeutral. Why call it logNeutral? Because it turns out that CineStyle uses the standard Canon Neutral picture style as its base. Only the luma gamma curve has been modified. How do we know this? Let's just say we're pretty sure.

So if there is a logNeutral could there be variations using other standard Canon picture styles? Yes. We found out that the luma gamma curve works with other picture styles. Sure, you could probably adjust the look of the Neutral style in post to match the other picture styles but one of most often mentioned issues with CineStyle is that it is hard to get pleasing skin tones. So why not experiment with the Portrait style as a base? We included all of standard Canon picture styles with this release.

Now with all the excitement over raw video why is a copy of a 5-year old picture style of any significance? It turns out that many people are using CineStyle not only for H.264 (8-bit 4:2:0) but also for HDMI out to an external recorder. In the future perhaps someone can do the same with Log-C or something that will take better advantage of the 8-bit 4:2:2 HDMI output.

Consider using MLP in your workflow for this picture style. Danne continues to update this program and it can handle luts, denoising and conversion to Prores. MLP can also work with H.264 HDR which is a natural for this picture style.

Most of all, try it out and report back. If you don't like it, hey at least you didn't pay for it.

********************
Original post starts here:
********************

Not sure which category to post this under because it is part general chat, part shooting, part post production, part raw and part not...

I've been thinking about picture styles because of a conversation I had with Danne about CineStyle. I call that conversation the "CineStyle Conspiracy Theory." Basically it goes like this:

Ok, let's start out with who developed Cinestyle. Technicolor--not exactly a little fly by night company but one of the most respected companies in the motion picture industry. Interestingly it is a French multinational company, not just a film lab in Hollywood.

When. 2011, that's a year before the 5D Mark III but more importantly also a year before the C100 which has the so-called "log" profile built in. Why did Technicolor create Cinestyle then stopped supporting it? The luts and tools that Technicolor developed are no longer available directly from the company though the webpage is still live and the profile is still available for download.

Looking at the sample footage shot on "Canon Log" from the C100 compared with Cinestyle they look very similar. Same with the GoPro "Protune" but here's a little known fact--Protune was developed by Technicolor. I'm not saying that GoPro footage is great but Protune does allow more flexibility in post in order to more closely match footage shot on other cameras so there is some merit to this.

I have read a lot of pros and cons (mostly cons) on the boards about Cinestyle and mostly praises for the "log" profiles like S-Log, Log C, Canon Log, etc. Log does have advantages and disadvantages. The old saying that you can't get something from nothing certainly applies here. Technically, it may even be inferior to getting everything "right" in the camera but in practice things are a little different.



I can't look into what they did with CineStyle because that profile is locked out of the Canon PictureStyleEditor but I did read up on how to make a picture style and you have to start with a CR2 file. That got me thinking, all this talk about picture styles not affecting raw images--is that really true? I shot some still and MLV files with monochrome and CineStyle then took a look at them in various programs. Guess what, it does make a difference depending on what program you're using. Try it yourself, set the profile to monochrome and shoot something then open it up in Canon's Digital Photo Professional app--that's a free one that you can download from Canon. The raw image is in black and white. Now open it in Photoinfo--I'm not sure where that came from but I think it comes with OS-X. Also in black and white but it is apparently showing the jpeg preview. How about Adobe Camera Raw or RawTherapee? Color! Run exiftool on it and it will show:

Code: [Select]
Picture Style                   : Monochrome
Running exiftool with the -v option will give even more information:

Code: [Select]
 
  | | | 20) ProcessingInfo (SubDirectory) -->
  | | | + [BinaryData directory, 28 bytes]
  | | | | ToneCurve = 0
  | | | | Sharpness = 3
  | | | | SharpnessFrequency = 0
  | | | | SensorRedLevel = 0
  | | | | SensorBlueLevel = 0
  | | | | WhiteBalanceRed = 0
  | | | | WhiteBalanceBlue = 0
  | | | | WhiteBalance = -1
  | | | | ColorTemperature = 7300
  | | | | PictureStyle = 134
  | | | | DigitalGain = 0
  | | | | WBShiftAB = 0
  | | | | WBShiftGM = 0

What is interesting is running an MLV shot with the monochrome picture profile through mlv_dump with the -mv option will also show some picture style information:

Code: [Select]
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   134
     contrast:   0
     sharpness:  3
     saturation: -559038737
     colortone:  -559038737

There's that same "134" picture style which I guess is the monochrome profile. How about an MLV shot on CineStyle?

Code: [Select]
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   33
     contrast:   -4
     sharpness:  0
     saturation: -2
     colortone:  0

So what is the significance of this? Well it might be possible to make a "real" log picture profile that will create proper log H.264 files. I looked into this and yes it is possible thought it might not be practical. DeafEyeJedi mentioned that the editors where he works prefer wide gamut over log and I'm thinking that CineStyle might just be a hybrid log/flat/wide gamut picture style because there's only so much you can do with 8-bit 4.2.0 files. However, if the picture style can also be used on the raw files then you can actually create a true log like Arri Log C picture style.

Now for the bad news. Picture styles are binary files and the PictureStyleEditor is a rather elementary tool that isn't really suitable for creating a proper log curve. Then of course there's the job of figuring out how to interpret the picture style that is embedded in the raw file. I looked at the CineStyle picture style in a HEX editor and can't make heads or tails of it but there is a lot more information in there than what exiftool is reporting.

16
Feature Requests / Reassign button for White Balance
« on: December 01, 2015, 11:09:05 PM »
Recently I discovered the ML Auto adjust Kelvin + G/M in a submenu. It would be so cool (and warm and magenta and green) to map a button for this like on the big boys' cinema cameras. Is this possible? Maybe with a lua script?

Thanks to Danne for pointing out this feature to me.

17
Here's the issue, cr2hdr seems to do fine when converting one dual_iso dng at a time but it sometimes seg faults when doing a batch.

[EDIT: It doesn't have anything to do with batch processing. I had an instance where cr2hdr failed when converting a single dng then succeeded when re-running cr2dng on the same dng.]

Code: [Select]
Horizontal stripe fix...
Segmentation fault: 11

So the failure seemed to be happening on "Full-res reconstruction..."

I've come across this every once in a while but now I have a dual_iso mlv file that is causing this consistently on both my system and Danne's.
(BTW--this came up while testing his new MLPP workflow for OS-X, which is coming along very nicely.)

Ok--this is how to reproduce the issue. Download this mlv file from my dropbox account:

https://www.dropbox.com/sh/mfouwwdch8uauas/AACMTxQwzTUvGwmcgWYGMNGla?dl=0

It is 648 MB so I'd like to delete from there it as soon as enough people have had a chance to try it out. (That shot won't win any awards but it seems to consistently reproduce the issue.)

Next convert it to dng using mlv_dump. I included Mac and Windows versions of mlv_dump and cr2hdr in my bitbucket download area so you don't have to go hunting for them. (Use the testing versions.)

Here's a simple way to do it with mlv_dump:
Code: [Select]
mlv_dump --dng M23-1338.MLV
Finally, batch convert these through cr2hdr you can try:
Code: [Select]
cr2hdr *
Or
Code: [Select]
for i in *.dng; do cr2hdr $i; done
Or whatever method you use to batch convert. You can even import the files into Lightroom, select all of them and run them as a batch using kichetof's cr2hdr plugin. Everything I have tried has caused cr2hdr to seg fault on random frames.

So far this has only been tested on OS-X but it would be interesting to see if it also affects Windows and Linux.

18
General Development Discussion / Dealing with Focus Pixels in raw video
« on: October 22, 2015, 11:09:10 PM »
@dfort,
One possibility is mapping the focus pixels for each camera and then interpolating out those pixels (rather than just chroma-smoothing the entire image). This might fix the dual ISO / focus pixel issue, IDK. It should be faster than chroma-smoothing, and not have any artifacts. It's a lot of work though, you need to create a focus pixel map for each camera and video mode.

So here we go--

Affected cameras: EOSM, 100D, 600D, 650D, 700D, (and more?)
[EDIT: Pretty sure now that the 600D doesn't show focus pixels in raw video.]

Focus pixels, phase detectors, call them what you will when these dots show up on your raw video they are annoying. There have been attempts at clearing them up like the PinkDotRemoval tool that has apparently been abandoned. Chroma smoothing helps but it affects the overall image and it doesn't work for dual iso footage.

I tried to shoot something that shows as many of these pixels as possible by shooting an out of focus white paper with a couple different colors of lights and processing the image with a heavy hand to show the pixels as clearly and dramatically as possible.


Got your attention? This is a frame from an mlv file shot on an EOSM in crop movie mode. It doesn't look quite as bad when shooting regular subjects but it is a problem.


It is possible to get an even clearer view of these pixels by running a frame through dcraw using the -d option which, according to the documentation:
Quote
-d
Show the raw data as a grayscale image with no interpolation.

Here is a closeup of that same frame at the top of this post.


Once we know the coordinates of these pixels they can be zapped.

Method 1 - In Camera

The ideal way to deal with them would be in the camera. In fact there is a way that "bad pixels" can be mapped out and apparently it is common practice to map out these faulty pixels when the camera is manufactured or repaired--saves on reject sensors. Assuming that we can't access the code that maps out pixels in the Canon firmware, the developers over at the chdk project have figured out more than one way to do it: http://chdk.wikia.com/wiki/Badpixel_removal

I'm not skilled enough to take what they have done and apply it to Magic Lantern but maybe someone who has the knowledge can drop a few hints?

Method 2 - In Post

Yeah, we can fix it in post. In fact there are several programs that can deal with pixel problems. Pixel Fixer is a dedicated tool, RawTherapee has an option to deal with hot and dead pixels and an interesting Python wrapper for the LibRaw library called rawpy that creates a ".badpixels" file dcraw can use to zap out pixels. This might be a way to create the database of focus pixel maps dmilligan suggested at the top of this post.

I tried using dcraw to get rid of one of the focus pixels on a dual iso frame and found that one pass wouldn't do it. Maybe it is because the process is breaking down by running the file through cr2hdr first? My understanding is that the debayering process is averaging neighboring pixels and in dual iso it seems to "smear" the pixel vertically. In addition, some of these focus pixels have a secondary pixel diagonally across that also needs to be mapped.

Here is what I ended up doing to get rid of just one focus pixel--and the diagonal secondary pixel.

Code: [Select]
1247 121 0
1248 120 0
1248 121 0
1248 125 0
1248 126 0
1248 122 0
1248 123 0
1248 124 0
1248 126 0
1248 122 0
1248 120 0
1248 124 0

Note that I hit the diagonal pixel just once but had to work top and bottom to middle and hit the same pixels several times in different sequences. Think of it as stippling a brush to spot a print--in photography school I spent many hours spotting prints. (By the way, this a map of all the focus pixels for the 650D that was used on the PinkDotRemover tool.)

Here is the result. I circled the area where the hot pixel was located and yes, it looked just like those other ones before running the dng through dcraw.


So if this is the solution then what's the problem?

The problem is that I'd like to get some feedback that this would actually be practical. It looks like there would have to be a different focus pixel map file for every camera and every crop factor (raw image size) and all of this would have to be assembled in a way that users could handle. Danne is the master of assembling scripts that can automate complex processes so it is possible to piece together something with dcraw--but then again is that the best solution? Maybe it would be better to adapt dcraw's "badpixels" code to mlv_dump or cr2hdr or MLVFS?

Just asking for a little guidance here.

19
General Development Discussion / Compiling Magic Lantern on a Macintosh
« on: October 14, 2015, 02:36:14 AM »
Setting up a Magic Lantern development environment on a Macintosh


For the terminally impatient there is a script at the bottom of this post that will get you up and running in about 2 minutes.

Before we begin did you know that all Macs have development tools already installed yet they are not available to the user? If you have a "virgin" Mac that you have never used for development try this, open up a terminal (located in Applications under Utilities) and type:

Code: [Select]
which gcc
Chances are that it will return /usr/bin/gcc, gcc is the compiler to build Mac binaries and one of the tools that we'll use to build Magic Lantern. So we're all set, right? Ok, not that easy try typing in gcc to launch the compiler:


We don't really need the full Xcode development environment so simply click the highlighted "Install" button, accept the licence to activate the developer tools. If you have already installed Xcode you might still need to install the command tools. Use this terminal command:

Code: [Select]
xcode-select --install
This will install the developer tools in /Library/Developer/CommandLineTools/ but there are some additional tools needed to compile Magic Lantern and several ways to install them. The simplest is to use one of the popular package managers like Fink (http://www.finkproject.org/) MacPorts (https://www.macports.org/) or Homebrew (http://brew.sh/) Homebrew is the newest of the bunch so we'll be using that. Homebrew installs the tools so they are available in /usr/local/bin which happens to be included in every new Mac's default path. However, your Mac may or may not have a /usr/local directory. Another possibility is that you have a /usr/local but don't have permissions to install anything in there. Do a directory listing to see where you stand:

Quote
Rosies-Air:~ Fort$ ls -la /usr
total 8
[email protected]   12 root  wheel    408 Oct 12 11:45 .
drwxr-xr-x    31 root  wheel   1122 Sep 30 21:56 ..
drwxr-xr-x     5 root  wheel    170 Aug 22 18:51 X11
lrwxr-xr-x     1 root  wheel      3 Sep 30 21:22 X11R6 -> X11
drwxr-xr-x     3 root  wheel    102 Aug 26 19:17 adic
drwxr-xr-x  1061 root  wheel  36074 Sep 30 21:28 bin
drwxr-xr-x   258 root  wheel   8772 Oct 12 11:45 include
drwxr-xr-x   285 root  wheel   9690 Oct 12 11:45 lib
drwxr-xr-x   183 root  wheel   6222 Sep 30 21:22 libexec
drwxr-xr-x   243 root  wheel   8262 Sep 30 21:22 sbin
drwxr-xr-x    46 root  wheel   1564 Oct 12 11:45 share
drwxr-xr-x     4 root  wheel    136 Sep 17 00:03 standalone

Notice that I used "ls -la" to get a detailed view of everything in the /usr directory. This system doesn't have a /usr/local directory so we'll have to create one. If you have a /usr/local directory but it is owned by root, you need to change permissions. Try this first:

Code: [Select]
sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
Don't be intimidated by long command lines, you're on a Mac so simply copy and paste it into the terminal window. Note that you will have to enter your password.

If it didn't work you are probably on OS-X 10.11 "El Capitan" or 10.12 "Sierra" or even something newer so you need to temporarily turn off disable System Integrity Protection (SIP) in order to create a /usr/local you can use.

1. Restart the computer, while booting hold down Command-R to boot into recovery mode.
2. Once booted, navigate to the "Utilities > Terminal" in the top menu bar.
3. Enter csrutil disable in the terminal window and hit the return key.
4. Restart the machine and System Integrity Protection will now be disabled.


Now in the terminal enter this command:

Code: [Select]
sudo mkdir /usr/local && sudo chflags norestricted /usr/local && sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
What you are looking for is a /usr/local directory with your name on it:

Quote
drwxr-xr-x     2 Fort  admin     68 Oct 12 12:52 local

Don't skip this next step -- go back into recovery mode and this time type csrutil enable in the terminal.

Now that we've got a /usr/local that we can use, install Homebrew. Ok--if you are really itching to get up and running you can now just skip to the bottom of this post and run the quick installation script. Go!

If you want to understand what we're doing any why, keep following this tutorial:

First of all this command will download and run a ruby script to install a basic Homebrew setup:

Code: [Select]
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Next start adding some of the tools needed to build Magic Lantern.

Code: [Select]
brew install python wget mercurial
What's the deal with Python, isn't it already installed? Yes, but getting the Apple version configured to work with the ML install scripts is more trouble than it is worth so we'll just use the Homebrew version. In addition, we're going to get some python scripts using the package manager that Apple left out of their version of Python.

Code: [Select]
pip2 install docutils
If for some reason you'd rather use Apple's version of Python, I've included instructions on how to set it up at the bottom of this post.

Although the compiler that comes with the Mac can compile ML there are reasons to use a different version. Technically it isn't really gcc but another compiler called "clang" and if you try using it to compile ML you'll probably get a bunch of clang error messages. In addition, Homebrew switched to version 6 of gcc and that's causing some issues with ML so we'll go with good old version 5.

Code: [Select]
brew install [email protected]
Two versions of gcc? Won't that confuse the system? Homebrew is designed to work with the system so the gcc that you just installed is called gcc-5. This requires some special handling that we'll get to a little later.

We're not not done with compilers, we'll need yet another version of gcc to compile binaries that will run on the ARM architecture used in our Canon cameras. To figure out which ARM compiler to get refer to the ML code, specifically: Makefile.user.default -- ARM_PATH=~/gcc-arm-none-eabi-4_8-2013q4 notice that little squiggly mark after the equal sign? That means that the path starts in your home directory. Ok, so much for the lecture let's get it and put everything in its place.

Code: [Select]
cd ~ && wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2
Before doing this next step make sure you are in the directory where you want to save the magic-lantern source code.

Finally, time to grab the ML source code and start building. There are plenty of resources on the forum but basically, to get the latest unified branch:

Code: [Select]
hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
hg update unified

We're almost ready to build Magic Lantern but first we've got to make one small change first. Go into the magic-lantern directory, find the file named Makefile.user.default and you'll see this block of code:

Code: [Select]
#
# Host compiler settings
#
HOST_CC=$(shell which gcc)
HOST_LD=$(shell which ld)
HOST_AR=$(shell which ar)

Now make a new file called Makefile.user and put in just these lines:

Code: [Select]
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)

Warning: DON'T EDIT Makefile.user.default -- Put the changes you want to make to Makefile.user.default in your Makefile.user file. Think of this as a way to customize the Magic Lantern build environment without modifying the code.

Another area you might want to customize is which modules to include when making a new build. The modules that the developers decided you probably want are in the modules directory in a file called Makefile.modules.default but maybe you'd like to add another one, take for example bulb_nd. To do this without modifying any Magic Lantern code create a file in the modules directory called Makefile.modules.user and put this in it:

Code: [Select]
ML_MODULES_DYNAMIC += bulb_nd
Now let's build Magic Lantern. Here's the way I do it for the EOSM:

Code: [Select]
cd ~/magic-lantern/platform/EOSM.202/
make clean && make zip

So you want to start hacking away on the code? There's lots of choices when it comes to text editors. You can even use TextEdit though I wouldn't recommend it because you have to turn off all the "smart" features so it doesn't destroy your source code. A better choice is Aquamacs. I have no financial interest if you get it or not. Hey, it is free. What about the editor that comes with Xcode? You can certainly install the full Xcode app that includes a very capable editor though it is quite a beast to master. It is also quite a heavy download. Of course if you are a real propeller head there's all the old favorites like vi, emacs, pico and nano already installed for you.

Some other software you might want to look into if you are going to get serious and contribute back to the Magic Lantern project is SourceTree. It supports the bitbucket repositories used by Magic Lantern and it has a nice graphical interface showing the various branches in ML. You're on a Mac, not everything has to be done on the terminal.

Added bonus --  Compiling Magic Lantern command line tools a.k.a. "Module Helpers"

The Magic Lantrn source code comes with some applications that you can run in the terminal or incorporate them in your own programs. Both the cr2hdr Lightroom plugin and Danne's cr2hdr-r workflows use--surprise, cr2hdr which is a part of the dual_iso module. There is also a raw2dng, mlv_dump and a few others. Your system is already configured to compile these tools. In fact if you make a build for your camera the Makefile script builds a few of them for you. For example, in the terminal go to magic-lantern/modules/dual_iso and type:

Code: [Select]
./cr2hdr
We didn't give it any instructions so it simply lists a short list of the available options that it has available. It is beyond the scope of this tutorial how to use the various command line tools but search through the forum and wiki for instructions on how to use the command line tools.

If you find something interesting in the modules you'd like to try out like say raw2dng.c and there isn't a binary for it. Check out the Makefile and see if there is a "rule" to build it. In the case of raw2dng simply type:

Code: [Select]
make raw2dng
and presto, you have a Mac binary to play with.

Extra Added bonus --  Cross compile to Windows

Let you in on a little Magic Lantern secret. Most Windows binaries aren't compiled in Windows. Sure, there are developers using Windows systems but they are either running a Linux distribution inside of a "VirtualBox" or Cygwin which is a self contained environment that functions within Windows. Only a MinGW or MinGW-64 Magic Lantern development setup might be considered Windows native. In fact if you want the best environment to work with open source projects like Magic Lantern set up a Linux system but one issue is that although a Linux system can be easily set up to compile Windows binaries it is not so easy to do Mac binaries. Anyway, to make Windows binaries on a Mac you need yet another gcc compiler like the ones maintained by the MinGW project. MinGW is short for Minimalistic GNU for Windows. What does GNU stand for? GNU is a recursive acronym for GNU's Not Unix. If you check out Apple documentation you will see that they claim that OS-X is Unix. Geek alert - instead of getting caught up in techno-babble how about setting up a cross compiler on your Mac already and start building Windows binaries.

You will need a darwin (OS-X) build of the MinGW compiler. Homebrew to the rescue.

Code: [Select]
brew install mingw-w64
This give you a full 32-bit and 64-bit Windows cross compiling setup ready to go. Now to make a Windows binary all you need to do is add the ".exe" file extension when you invoke make. For example:

Quote
make cr2hdr.exe

Being able to create Windows binaries on a Mac is so easy that I am including it on the quick installation script at the bottom of this post, though you'll need to uncomment that line first.

Tip - Removing pesky "._" files from your card

Filenames beginning with a dot are "invisible" and normally you won't even know they are there. Every time you copy files from a Mac file system to another type of filesystem it creates these "dot" files to store some information that is not normally used in other filesystems. If you are using lua on a memory card that is formatted as a FAT16 or FAT32 filesystem (usually cards under 64GB), you may see some strange messages on your screen when your camera starts up with the lua module enabled. It isn't a big deal and dmilligan put in some code to ignore files that begin with a dot but it is rather annoying. Apple actually included a utility to clean up this mess. Assuming your card is named EOS_DIGITAL, here's how to run it:

Code: [Select]
dot_clean -m /Volumes/EOS_DIGITAL
The "-m" option also removes files that start with an underbar.

Tip - Uninstalling Homebrew
Code: [Select]
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)"You might also want to clean out your /usr/local directory for any stragglers. Just remember that if you are on "El Capitan" or "Sierra" and you delete /usr/local you will have to do the Command-R csrutil enable/disable song and dance to get it back.

Tip - Configuring Apple's Python to work with ML

Note: I don't use this and things might have changed so you're on your own if you want to experiment with this.

Makefile.user.default specifies that PYTHON=python2. Of course you can change that or make an alias for python2 but if you are comfortable disabling System Integrity Protection (SIP) go ahead and do that then:

Code: [Select]
ln -s /usr/bin/python /usr/bin/python2
The reason for installing the Homebrew version of Python is because Apple left out a Python utility called "pip" that makes it easy to install docutils. Follow these steps to install pip and docutils:

Code: [Select]
wget https://bootstrap.pypa.io/get-pip.py
Code: [Select]
sudo -H python get-pip.py
Code: [Select]
sudo -H pip install docutils
I like to do some cleanup. The docutil scripts are owned by root so let's change that and make sure that we've got permissions for everything in /usr/local:
Code: [Select]
sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local
You're done with the get-pip.py
Code: [Select]
rm get-pip.py
Quick installation script.

If you have a Mac with a /usr/local directory and would like to get everything up and running in a hurry copy and paste the following in your terminal. You'll have to enter your password, hit a button, accept a license agreement and maybe hit the return key once but other than that it is pretty much automatic. It will setup your development environment and even download the Magic Lantern source to your home directory. A big thanks to Danne for his scripting skills on this.

Code: [Select]
cat <<'EOF' > mac_ml.sh
#!/bin/sh

#  mac_ml.sh
#
#  Script to almost automatically install
#  and configure a Magic Lantern development
#  environment on a Macintosh.
#
#  Reference:
#  http://magiclantern.fm/forum/index.php?topic=16012.0
#
#  Daniel Fort (dfort) - 2015-10-15
#  Modified 2016-03-17 - took out ugly gcc hack
#  Modified 2017-06-27 - added mingw-w64 for cross compiling Windows binaries
#  Modified 2017-06-28 - check if magic-lantern already exists
#  Modified 2017-07-12 - write a Makefile.user file
#  Modified 2017-09-23 - made mingw-w64 (Windows cross compiler) optional
#  Modified 2017-09-24 - added interactive options for mingw-w64 and QEMU
#  Modified 2017-09-26 - QEMU install.sh now installs dependencies so no need to do it here.
#  Modified 2017-09-29 - Improved handling of QEMU install.
#  Modified 2017-11-16 - Homebrew changed gcc-5 install to [email protected], added gcc-arm-none-eabi-5_4-2016q3
#  Modified 2017-11-17 - added more checks in cases where a development system was already in place
#
#  Thanks to Danne and kichetof and of course a1ex for their input

if ! test -d /usr/local; then
  echo "/usr/local directory not found."
  echo "Follow the steps at the beginning of"
  echo "tutorial to create /usr/local and"
  echo "then re-run this script."
  exit 1
fi

echo "/usr/local found -- "
echo "continuing with installation."

cd ~
if [ ! -d "/Library/Developer/CommandLineTools" ]; then xcode-select --install; fi

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install [email protected] python wget mercurial

pip2 install docutils

if ! test -d ~/gcc-arm-none-eabi-4_8-2013q4; then
    cd ~ && \
    wget -c https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    tar -jxf gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2 && \
    rm gcc-arm-none-eabi-4_8-2013q4-20131218-mac.tar.bz2
fi

if ! test -d ~/gcc-arm-none-eabi-5_4-2016q3; then
    cd ~ && \
    wget -c https://launchpad.net/gcc-arm-embedded/5.0/5-2016-q3-update/+download/gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2 && \
    tar -jxf gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2 && \
    rm gcc-arm-none-eabi-5_4-2016q3-20160926-mac.tar.bz2
fi

if ! test -d ~/magic-lantern; then
    cd ~
    hg clone https://bitbucket.org/hudson/magic-lantern
    cd magic-lantern
    hg update unified
    cat <<'EOT' > Makefile.user
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
EOT
    cd ~
fi

#
# OPTION 1 - Windows cross compiler
#
echo
read -p "Would you like to install the Windows cross compiler mingw-w64? [y/n] "
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "installing mingw-w64"
    brew install mingw-w64
fi

#
# OPTION 2 - QEMU
#
echo
read -p "Would you like to install QEMU? [y/n] "
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    echo "installing QEMU"
    cd ~
    echo 'export PATH=~/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH' >> .profile
    source .profile
    cd magic-lantern
    hg update qemu -C
    cd contrib/qemu/
    ./install.sh
    cd ~/qemu/qemu-2.5.0
    ../configure_eos.sh
    make -j`grep -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`
fi

cd ~
echo
echo "Magic Lantern development environment setup finished."
exit 0
EOF
bash mac_ml.sh && rm mac_ml.sh

20
General Chat / Canon EOS M10
« on: October 13, 2015, 07:26:44 AM »
This one came out with no press release--new EF-M 15-45mm f/3.5-6.3 IS STM Lens too.

http://www.bhphotovideo.com/explora/photography/news/unveiled-canon-beefs-its-mirrorless-line-eos-m10-and-15-45mm-lens

Looks like a cheaper version of the M3. Wonder what happened to M4 through M9. The lens looks like it has a collapsed position like the 11-22mm. That makes killing the shutter-bug easy.


21
General Chat / Magic Lantern on Lynda.com
« on: September 25, 2015, 05:51:16 AM »
Looks like this just went live today:

http://photofocus.com/2015/09/24/hacking-a-canon-camera-with-magic-lantern-menus/

Funny thing is the misinformation like always use the latest Canon firmware (not always true) and HDR video is for raw files, where did they get their information?

In any case, good PR for Magic Lantern.

22
Setting up a Magic Lantern development environment with Cygwin/MinGW-64


Quick start for the terminally impatient:
Download the Cygwin installer. We're going to use the 32-bit installer. Don't worry if you have a 64-bit machine, a 32-bit Magic Lantern development environment works better than a 64-bit Cygwin installation.

setup-x86.exe

Assuming it is in your Downloads directory open a Command Prompt. Note: Don't use the PowerShell, it seems to have a problem with the long command we're using in this tutorial.

Run the installer in the Command Prompt - just copy and paste the following line and it will automatically install the packages needed to compile Magic Lantern.

Code: [Select]
"%UserProfile%"\Downloads\setup-x86.exe -s http://cygwin.mirrors.pair.com -q -P mingw64-i686-gcc-core,gcc-core,make,mercurial,python3-docutils,perl,wget,unzip,zip,diffutils
Open a Cygwin Terminal from your start menu.
Code: [Select]
cd && wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip && mkdir gcc-arm-none-eabi-4_8-2013q4 && unzip -n gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip -d gcc-arm-none-eabi-4_8-2013q4 && rm gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip
That was the ARM toolchain that the unified branch is using but some of the experimental branches are looking for a newer version so let's get that one to.
Code: [Select]
cd && wget https://launchpad.net/gcc-arm-embedded/5.0/5-2016-q3-update/+download/gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip && mkdir gcc-arm-none-eabi-5_4-2016q3 && unzip -n gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip -d gcc-arm-none-eabi-5_4-2016q3 && rm gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip
Now get the Magic Lantern source.
Code: [Select]
hg clone https://bitbucket.org/hudson/magic-lantern && cd magic-lantern && hg update unified && cd
Go!

Cygwin is probably the quickest and easiest way to set up a ML development environment on Windows. Note that you won't be able to use all of the features of the ML build scripts as you can with Linux or to an extent on a Macintosh, though you can still compile an ML nightly zip file and all of command line tools. It used to be that you couldn't create Windows native cr2dng, mlv_dump or other ML command line tools. However, ML Supporter, Hero Member and Cygwin advocate Marsu42 pointed out that it is possible to use the MinGW-w64 gcc compilers with Cygwin to in effect cross compile Windows native binaries.

One caveat before you start. If you've got a space in your Windows user name it doesn't work. I tried coming up with a workaround but nothing worked 100% of the time. The best advice is to create a user name without spaces.

Get the Cygwin Installer, setup-x86.exe (just click on the link to save a few steps), these are the same installers that are on the official Cygwin website.

Just accept the default options for the installer.


The easiest way to find what you want is to type it in the Search field. For example, to get the 32-bit version of the MinGW gcc package which is key to cross compiling the Magic Lantern command line tools:


That's the mingw64-i686-gcc-core package. What if you have a 64-bit system?
Note that most module exe helpers are targeted for x86 code, i.e. even if you run cygwin x64 (which you should on a x64 Windows) you need to use the i686 mingw toolchain.
Please leave feedback if you get mingw64-x86-64-gcc-core working and on which command line tools--it is going to need some tweaks to the ML source code. In the meantime I'm recommending setting up a 32-bit Cygwin even on 64-bit machines. Why? because the command line tools like mlv_dump don't build properly in the Cygwin64 environment.

You will also need some other packages:

Devel
  • gcc-core (Yes, you will need both the MinGW and Cygwin compilers)
  • make
  • mercurial
Python
  • python3-docutils
Perl
  • perl
Web
  • wget
Archive
  • unzip
  • zip
Utils
  • diffutils

There may be some other packages that you might want to install but this should be enough to build the unified branch which is what gets pushed out to the nightly builds.

Once the installer is finished, let it make shortcuts to your startup menu.



Go ahead and launch the Cygwin Terminal.

We're almost there. Canon cameras are obviously not Windows computers so we will need one more compiler to build something that will run on the ARM Architecture used in the camera. To figure out which ARM compiler to get refer to the ML code, specifically: Makefile.user.default -- ARM_PATH=~/gcc-arm-none-eabi-4_8-2013q4 notice that little squiggly mark after the equal sign? That means that the path starts in your home directory. Let's make it easy, just copy/paste the following into your Cygwin Terminal:

Code: [Select]
cd && wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip && mkdir gcc-arm-none-eabi-4_8-2013q4 && unzip -n gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip -d gcc-arm-none-eabi-4_8-2013q4 && rm gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip
Some of the newer experimental branches prefer a newer ARM toolchain so get that one too:

Code: [Select]
cd && wget https://launchpad.net/gcc-arm-embedded/5.0/5-2016-q3-update/+download/gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip && mkdir gcc-arm-none-eabi-5_4-2016q3 && unzip -n gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip -d gcc-arm-none-eabi-5_4-2016q3 && rm gcc-arm-none-eabi-5_4-2016q3-20160926-win32.zip

Finally, time to grab the ML source code and start building. There are plenty of resources on the forum but basically, to get the latest unified branch:

Code: [Select]
hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
hg update unified

Here's the way I do it for the EOSM:

Code: [Select]
cd magic-lantern/platform/EOSM.202/
make zip

So you want to start hacking away on the code? There's lots of choices when it comes to text editors, I would recommend using Notepad++. It is designed to work with source code and you can check if the source has Linux, Mac or Windows line endings--don't mix them up! Here is a screenshot of the interface, notice that posix.c has Windows (CR+LF) line endings. :o


Also consider using the excellent SourceTree app to keep track of your changes and easily switch between branches.



Saving the best for last.

If you want to build one of the command line tools, let's say cr2hdr, and run it on another Windows computer you might look into the ~/magic-lantern/modules/dual_iso directory and maybe (if you already compiled ML for your camera) find a cr2hdr.exe file already built. If you run it from the Cygwin Terminal it will work fine but move it somewhere else and run it with cmd.exe and it will not work if it cannot find the cygwin1.dll. The way around this is to cross compile to Windows. It may seem strange, after all aren't you already in Windows? Not really, Cygwin sets up its own environment to mimic a posix (a.k.a. Linux) system.

Here's how to compile mlv_dump:

Code: [Select]
cd ~/magic-lantern/modules/mlv_rec
Code: [Select]
make mlv_dump.exe
By adding the ".exe" file extension the Magic Lantern Makefile(s) will use the MinGW gcc compiler and create a native Windows binary. I have tested this with cr2hdr, mlv_dump and raw2dng. There are other command line tools, some of which I don't know what they do and can't get a working build for them. That seems to be pretty much standard operating procedure in open source development. Happy hacking!



It may seem that creating Windows programs that do not rely on the cygwin1.dll was either not possible or not permitted because of this F.A.Q. item in the official Cygwin website:
  • Can I build a Cygwin program that does not require cygwin1.dll at runtime?

    No. If your program uses the Cygwin API, then your executable cannot run without cygwin1.dll. In particular, it is not possible to statically link with a Cygwin library to obtain an independent, self-contained executable.

    If this is an issue because you intend to distribute your Cygwin application, then you had better read and understand https://cygwin.com/licensing.html, which explains the licensing options. Unless you purchase a special commercial license from Red Hat, then your Cygwin application must be Open Source.

ML command line tools aren't really using the Cygwin API (Application Programming Interface) and MinGW has been using the Win32 API for years so there shouldn't be any legal issues using a setup like this as long as everything is kept open source. However, keep in mind that I'm not a legal expert.

23
Many have tried and failed -- and so have I but this is soooo close.

Cut to the chase -- There is an issue with xor_chk in a Windows msys/MinGW build environment that prevents it from creating a working autoexec.bin. This isn't anything new, it was noted back in the original xor_chk pull request.

There is an open bug report on the issue that points back to this discussion.

Before anyone says to give it up and use Cygwin, Linux, Mac or even Linux in a virtual box on Windows please let me give you the back story. I was compiling some Mac binaries of the command line tools, cr2hdr, mlv_dump and raw2dng from dmilligan's ml_dng fork to help testers who wanted to try it out but couldn't compile the code on their own. Then a few Windows users asked if someone could supply them some binaries too. Looking at this as a challenge, I dusted off my old 32-bit Windows/Linux dual-boot laptop and gave it a go. Cygwin was super easy, but the .exe files compiled in that environment required cygwin1.dll and that wasn't very practical for distribution. Cross compiling in Linux worked but I had a 32-bit machine and testers wanted 64-bit binaries. I finally settled on cross compiling on my MacBookPro which worked like a charm.

In the process I spent more time that I should have trying to get the msys/MinGW setup working and I was eventually able to compile the command line tools with just a few Windows specific changes to a couple of files. Seeing that there might be some value in this I posted these pull requests:

lv_rec: use fseeko/fseeko64 depending on platform as in mlv_dump.c.

Modified readme2modulestrings.py so that it will work in a MinGW Windows environment.

At this point I did something crazy--I tried compiling ML for my camera and it worked! Well, sort of. The autoexec.bin didn't work. It looked fine, I mean it was the right size. There were a few differences when comparing it to a working autoexec.bin byte by byte and everything else (modules, etc.) worked. Eventually I tracked down the problem to xor_chk.exe. At first the only quirk seemed to be that "make clean" wouldn't remove it but there was something else very wrong here. Copying over the xor_chk.exe along with cygwin1.dll from the Cygwin build environment I had on the same computer and re-running the platform build created a working autoexec.bin.

Ok so the problem is narrowed down to just this one file, apparently the only one that the host system needs to build in order to complete a working autoexec.bin. Everything else is handled by the ARM cross compiler.

So why is it important to work on this? Well it does give users and developers more options. Ok, developers will still prefer a POSIX compliant environment but setting up msys/MinGW is almost as easy as Cygwin and much easier than running a Linux virtual machine on Windows. The original project at www.mingw.org may be outdated but there is the msys2 project and an easy installer for it here that can be used with the MinGW-w64 - for 32 and 64 bit Windows project that leapfrogs what is available in Cygwin. In addition, the compiled command line tools will run on a stock Windows machine without any extra .dll files--which is where this whole thing started.

24
General Development Discussion / MinGw for Windows command line tools
« on: August 16, 2015, 10:19:47 PM »
I set up a MinGw environment on a Windows system just to compile the command line tools so there shouldn't be a need to set up Virtual Box. There was no problem building mlv_dump but I'm having problems with cr2hdr and raw2dng. Any clues what's going on here? One issue might be that this is a 32-bit system and maybe these tools need a 64-bit compiler?

raw2dng:
Code: [Select]
[ HOST_CC  ]   raw2dng
[ HOST_CC  ]   raw2dng
raw2dng.c: In function 'main':
raw2dng.c:107:5: warning: unknown conversion type character 'z' in format [-Wformat=]
     if (sizeof(lv_rec_file_footer_t) != 192) FAIL("sizeof(lv_rec_file_footer_t) = %zu, should be 192", sizeof(lv_rec_file_footer_t));
     ^
raw2dng.c:107:5: warning: too many arguments for format [-Wformat-extra-args]
raw2dng.c:109:5: warning: implicit declaration of function 'fseeko' [-Wimplicit-function-declaration]
     fseeko(fi, -192, SEEK_END);
     ^
[ HOST_CC  ]   raw2dng
raw2dng.o:raw2dng.c:(.text.startup+0x62): undefined reference to `fseeko'
collect2.exe: error: ld returned 1 exit status
make: *** [raw2dng] Error 1

cr2hdr:
Code: [Select]
Updated HGVERSION
[ README   ]   module_strings.h
LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}
{node|short}
{author|user}
{desc|strip|firstline}'

'LC_TIME' is not recognized as an internal or external command,

operable program or batch file.


(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x024A2710>)
make: *** [module_strings.h] Error 1

25
General Development Discussion / Need help cleaning up a Bitbucket mess
« on: August 04, 2015, 09:44:37 AM »
There is a strange branch in bitbucket with my name on it:

daniel_fort/added-sounddevshutdownin-1435698205057

I must have made a mistake on a June 30 pull request and just noticed it. This wasn't a problem with the two previous pull requests I made following these instructions.

There are some instructions about deleting branches with a "git push" command but I have never used git before and don't want mess things up even worse.

What should I do?  :'(

Pages: [1] 2