Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - a1ex

Pages: [1] 2 3 ... 331
Camera-specific discussion / Re: ML on EOS-M2
« on: Today at 08:40:40 AM »
You need two *equal* halves, without any approximation.

SFDUMP.BIN can probably be used from 700D or EOSM as well, but these cameras have a smaller size. Padding it with zeros may or may not work. How to get it? It's covered in QEMU's install script.

Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 11:15:20 PM »
Split it in two halves: first ROM0, second ROM1.

Back then, it was much easier to patch Canon code to dump everything in one file, so it covers both ROMs. For this reason,
older QEMU versions used to require the ROMs to be joined like this (but it was changed to allow using the backup ROM copies already saved by ML on the cards under ML/LOGS, and to allow more unusual ROM configurations).

Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 09:23:14 PM »
The dumper linked above runs on the main firmware (in contrast with the portable ROM dumper, which runs from bootloader), so it has no issues with card sizes, but it's limited to DIGIC 4 and 5 models. Does it help if you follow the instructions in the linked post? (it's different from the portable dumper you are used to)

You can start from the latest firmware.

Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 12:17:11 PM »
I'll start with a quick start guide for latest QEMU and IDA. First step is to assume it's very similar to EOSM, so after installing QEMU, head over to qemu/qemu-2.5.0/hw/eos/model_list.c and copy the basic entries from the section for EOSM:

Code: [Select]
        .name                   = "EOSM2",
        .digic_version          = 5,

The others are likely model-specific, so let's not assume too much from the beginning. Let's try:

Code: [Select]
./ EOSM2,firmware="boot=0"
FFFF3F50: MCR p15,0,Rd,cr6,cr0,0:  946_PRBS0 <- 0x3F       (00000000 - FFFFFFFF, 0x100000000)
FFFF3F58: MCR p15,0,Rd,cr6,cr1,0:  946_PRBS1 <- 0x3D       (00000000 - 7FFFFFFF, 0x80000000)
FFFF3F60: MCR p15,0,Rd,cr6,cr2,0:  946_PRBS2 <- 0xE0000039 (E0000000 - FFFFFFFF, 0x20000000)
FFFF3F68: MCR p15,0,Rd,cr6,cr3,0:  946_PRBS3 <- 0xC0000039 (C0000000 - DFFFFFFF, 0x20000000)
FFFF3F70: MCR p15,0,Rd,cr6,cr4,0:  946_PRBS4 <- 0xFF00002F (FF000000 - FFFFFFFF, 0x1000000)
FFFF3F78: MCR p15,0,Rd,cr6,cr5,0:  946_PRBS5 <- 0x37       (00000000 - 0FFFFFFF, 0x10000000)
FFFF3F80: MCR p15,0,Rd,cr6,cr6,0:  946_PRBS6 <- 0xF700002F (F7000000 - F7FFFFFF, 0x1000000)
128K Sector FROM From BL 0xffff
[SF] InstallSerialFlash 6 0xc022c0d4 0x0 0x1000000 1

OK, so we've got a rough idea about its memory map (same as other DIGIC 4 models). We also noticed it has a serial flash of size 0x1000000. This is similar to 100D (lookup serial_flash_size in model_list.c), so it may be a good idea to reuse the serial flash data from this model:
Code: [Select]
        .serial_flash_size      = 0x1000000,

Unfortunately, this step doesn't change anything in the emulation. Let's look at the next messages:
Code: [Select]
    10:    46.336 [PROPAD] ERROR Not Exist Valid ComboPackages!! 0x10000
    14:    56.576 [STARTUP] InitializeIntercom
    42:   722.688 ERROR [DL] ########## DL ERROR!!!!! ###########
- Many errors regarding missing properties. These are either stored in ROM, or in the serial flash, or they come from the MPU. The ones from ROM should not pose any problems, since we have the complete ROM. For the others, we did not see any sign of activity, so we have to dig in to find out whether anything is different (usually these communication channels may use different registers, depending on model, but overall the communication protocol is the same).
- DL ERROR: if you look in EOSM/debugmsg.gdb, you'll find a way to get past this message (by skipping the initialization of "DL", whatever that is).
- The debug messages are not very informative. To get more insights, find out a few essential functions (such as task_create and DebugMsg) and place them in EOSM2/debugmsg.gdb (look at other models for how the declarations should look like).

OK, so at this point we have to disassemble the firmware. If you use IDA, the initial steps would be:
- load ROM1.BIN
- select ARM processor, architecture ARMv5TEJ (we have ARMv5TE on all models from DIGIC 2 to DIGIC 5)
- uncheck 'Delete instructions with no xrefs' (we have plenty of those)
- uncheck 'Perform no-return analysis' (IDA usually gets it wrong)
- RAM section: 0, size 0x100000 (we can always add more later)
- ROM start address: 0xFF000000
- ROM size: 0xFFFFFC (off-by-one bug in IDA; I have an older version, so YMMV)
- File load address / size: same as ROM.

Now that our binary is loaded, go to 0xFFFF0000 and press "c" to mark this as code. This is the bootloader start address (aka HIVECS in ARM documentation). Next step is to find where the main firmware starts (where the code path leaves the bootloader). It's easy to assume similarity with other models, but let's try to find it from scratch (just to show off the latest QEMU tool):

Code: [Select]
./ EOSM2,firmware="boot=0" -d calls
call 0xFFFF0FCC(0, ffffdfff, 2000, ffff0b20)                                     at [ffff09c8:0]
 call 0x100000(0, 11836c, 11836c, 100000)                                        at [ffff1010:ffff09cc]

OK, so the bootloader runs from RAM.

Where does the bootloader end? When it stops executing from its address range (near 0xFFFF0000 or 0x100000):

Code: [Select]
  return 0 to 0x10013C                                                           at [1022f4:ffff1014]
 return 1 to 0xFFFF1014                                                          at [1000bc:ffff09cc]
return 0 to 0xFFFF09CC                                                           at [ffff101c:0]
Warning: R10 not restored (0xa -> 0x1)                                           at [ffff101c:ffff09cc]
PC jump? 0xF80C0000 lr=ffff09cc                                                  at [ffff0a04:ffff09cc]
0xffff0a04:  e1a0f000      mov pc, r0
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
0xf80c0000:  e59ff0c4      ldr pc, [pc, #196] ; 0xfffffffff80c00cc

Ta-da! Same as other DIGIC 5 models: ROM1 startup code is at 0xF80C0000, but code from main firmware expects to run from a mirror copy (details). So, our ROMBASEADDR is 0xFF0C0000.

Let's go there in IDA and mark this section as code. At this point, IDA already recognized a few functions, but let's get some more from the QEMU execution trace:

Code: [Select]
./ EOSM2,firmware="boot=0" -d idc
EOSM2.idc saved.

Now load this IDC script into IDA, or convert it for your favorite disassembler, and start looking at the functions called during the execution in QEMU:
Code: [Select]
PC jump? 0xFF0C000C lr=ffff09cc                                                  at [f80c0000:ffff09cc]
call 0xFF0C1BD4(1000, 698, eeeeeeee, 1000)                                       at [ff0c0dbc:0]
 call 0x866B4(f88, 74, eeeeeeee, 1000)                                           at [ff0c1be4:ff0c0dc0]
 return ffc to 0xFF0C1BE8                                                        at [866f4:ff0c0dc0]
 call 0x3168(f88, ff0c57a4, 0, 0)                                                at [ff0c1c80:ff0c0dc0]

0xFF0C1BD4 is cstart, 0x866B4 is bzero32, 0x3168 must be create_init_task and 0xFF0C13A4 must be init_task. Some of these functions are called from RAM, so we'll have to identify where its contents come from (what is copied where). TODO.

Next step, for the emulation, would be to find the MPU registers (near InitializeIntercom), mpu_send/mpu_recv stubs may also help, and we also need to check the serial flash communication.

For executing user code on the camera, next step would be to reserve memory for our application. This is usually done by shrinking the "malloc" buffer. Take a look at boot-hack.c from the "qemu" branch (as it's a bit more verbose and better explained). These debug messages can be seen on the QEMU-boot-check page.

To be continued.

General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: Yesterday at 10:31:51 AM »
A summary of the recent IRC discussions.

Not finding an easy way to spit out debug messages when debugging it running under qemu.

To print debug info from ML to the QEMU console, there is qprintf (in qemu-utils.c). I'd like to turn this into a "standard" debugging API, making it available anywhere in the source code (so it won't get compiled in the regular binary, but activated with CONFIG_QEMU=y). Halfway done on the "qemu" branch.

To control QEMU's verbosity, try running with "-d help" (there are many options). Note: most of these are on QEMU 2.5.0 (qemu branch in our tree). They are not ported to 2.9.0, where not all the basics are working properly yet.

Question:  Where is it we are trying to locate to ?

RELOCADDR is in RAM (our modified copy of Canon's startup code). On 1300D, main firmware starts at FE0C0000, not FE010000, so we'll relocate the startup code from there, until being able to replace their init_task with our version. Once there, we can launch our own task(s) alongside Canon's.

I can see the HIJACK_TASK_ADDR is around here on the 60D (0x1a20 to be exact), so I'm wondering if this is a table of tasks / interrupt vectors or something ?

HIJACK_TASK_ADDR is probably the same as CURRENT_TASK in GDB and current_task in stubs.S (pointer to the current task structure - see tasks.h).

In 1300D/debugmsg.gdb:
Code: [Select]
macro define CURRENT_TASK 0x31170
macro define CURRENT_ISR  (*(int*)0x31174 ? (*(int*)0x640) >> 2 : 0)

Also, to see tasks starting:
Code: [Select]
# this is valid on all firmware versions
b *0x38FC

# this one is for firmware 1.0.1
b *0xFE11D6B4

This debugmsg.gdb is committed on the "qemu" branch, but I've only tested on firmware 1.0.1 (only noticed there's a newer firmware available after committing).

hello world

Code: [Select]
23:22 < KennetRunner> or is it at a stage where I run hello world on my camera ?
23:24 < alexML> well, you can override the image buffer address before Canon GUI initializes the display (so ML code has
                something to draw on)
23:24 < alexML> ML gets the buffer from bmp_vram_info[1].vram2 (where bmp_vram_info is a stub)
23:25 < alexML> rather than waiting for this to get valid (nonzero), just set it to something outside the normal RAM range
                (e.g. 0x50000000 should be fine) and set QEMU to display from the same address
23:26 < alexML> this should give a hello world from the minimal platform, before getting the GUI working on the vanilla
23:26 < KennetRunner> I'll have a bash at that then...
23:27 < alexML> in qemu, look at s->disp.bmp_vram

I've also set up a job on the build server for QEMU 1300D tests, where you can find some startup logs with various levels of detail (firmware 1.0.1 for now):

Code: [Select]
SD: CMD12 in a wrong state
[SDIO] Error

That's fine, I get those too. I'm not sure if this indicates an emulation bug / incomplete model, or it's just how the (simplified) SD driver used in bootloader is supposed to behave (note: CMD12 is STOP_TRANSMISSION).

In the main firmware, you'll get a similar error about CMD1; this one is OK, as it appears to be the way Canon code probes for MMC cards. Regular SD cards are probably not supposed to reply to CMD1 outside the SPI transfer mode (at least that's my understanding), so the SD emulation backend prints some messages. The full conversation can be watched with -d sdcf (or -d sdcf,io for more details) and cross-checked with the SD specification.

If the reader is familiar with SD protocol, I'd welcome any insights (in particular, for the UHS-I initialization sequence).

Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?

To avoid corrupting the data on the SD image, it's best to prevent starting the emulation if the image is mounted by the user (as there will be two processes wanting to write on the same card image, without knowing about each other). This is done on Linux by checking /proc/mounts (losetup is used with "loopback" devices - that is, when mounting an image as a filesystem). Don't know how this works on Mac, and don't know whether my method is portable across other Linux distributions either.

The other issues appear to be (more or less) just annoyances (as they don't print what the user expects to see) and probably easy to fix.

Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 09:43:20 AM »
Seems like this one should be one of the "easier" ports.

Definitely, as it's plain old DIGIC 4 (unlike 1300D, which looks like a mix between D4, 5 and 6).

The boot flag is "traditionally" enabled after getting the Hello World working. There are two flavors: the minimal target (which requires a tiny set of stubs and only compiles a few small files) and the regular one (CONFIG_HELLO_WORLD in the full source code, which also requires file I/O for loading fonts, and a bunch of other initializations). Once the second version is working, CONFIG_DUMPER_BOOTFLAG will be straightforward to compile and run. The "user-friendly" installer comes a bit later, as it requires some more stubs.

The boot flag can be also enabled earlier, from the bootloader context; we did that on the DIGIC 2 and 3 models, blindly (without having the display available for debugging or user feedback). Now that we have display access from bootloader, that's a valid option as well (though I wasn't comfortable doing this step on 7D Mark II).

Until enabling the boot flag, you'll need my assistance to run a binary on the camera. You can run them in QEMU though, by setting "boot=1" though.

Will explain how to get the Hello World on models that don't boot to Canon GUI on the 1300D thread, as it was covered on IRC a few days ago.

General Help Q&A / Re: Lua vs ML data
« on: May 23, 2017, 08:02:22 PM »
Any test case we can use for cross-checking the math?

Code: [Select]
    else if(!strcmp(key,"value")) lua_pushnumber(L, lens_info.aperture / 10.0);
    else if(!strcmp(key, "focus_distance")) lua_pushinteger(L, lens_info.focus_dist * 10);
    else if(!strcmp(key, "focal_length")) lua_pushinteger(L, lens_info.focal_len);

Nice. This is the cocoa interface, right? Do you have menus that allow you to switch between serial console, VGA and so on?

BTW, after this commit it should install cleanly on clang and/or Mac.

Scripting Corner / Re: Lua Scripting (
« on: May 21, 2017, 01:53:08 AM »
Suspecting some stack corruption from lua_getfield (which appears to require a lua_pop, but that doesn't happen everywhere, see e.g. lua_dryos.c), I've started to dig inside the Lua C API (which I still don't really understand) and got some questions:

1) Is there some easy way to check the stack usage of our C functions? (for example, to make sure each lua_getfield call is paired with a lua_pop, but not only)

I can imagine some API test that calls every such function 1000 times or so to check whether it will overflow the stack, but that doesn't look very nice.

The LUA_FIELD_* macros appear to address this purpose, but they are not used everywhere; as a first step, I'd refactor all the calls to lua_getfield to use similar macros. But I suspect there are more instances of similar behavior, not obvious from the function names.

2) I've also stumbled upon the LUA_PARAM* macros. The Lua manual recommends luaL_checkinteger/number/string/whatever for the same purpose. Any reason we use those long macros, instead of the standard functions?

3) Most of the C tables allow setting arbitrary fields besides the predefined ones (but not all - font doesn't). What is the rationale behind this? Does any of the existing scripts make use of this feature?

I'm tempted to make them read-only, so it would catch typos when accessing these fields, rather than creating new fields that have no effect. Example: you could write in a script: lv.emabled = true (typo) and with the current behavior, you'll probably spend some time figuring out why this doesn't enter LiveView.

4) Most of the fields (from various objects) appear in 3 places (and it's easy for them to become inconsistent, as it doesn't give a compiler error, and I couldn't come up with a way to check them in api_test.lua either). For example, in lua_battery, all the fields (level, id, performance etc) appear in luaCB_battery_index, in luaCB_battery_newindex (on the long line with strcmp) and on lua_battery_fields. Any suggestions for reducing these repetitions?

Camera-specific discussion / Re: Canon 1200D
« on: May 20, 2017, 08:47:44 PM »
Does it crash in the default configuration? (after resetting ML to defaults)

BTW, this kind of crash might depend on the files you are transferring from the card (could be their number, size, or even the contents of some particular file, or a combination of factors). Therefore, to reproduce it, you might have to copy the images back on the card. Is this true? (in other words, can you reproduce the issue regardless of how many files you have on the card?)

Scripting Q&A / Re: GUI state triaging
« on: May 19, 2017, 08:27:15 PM »
That would only give info about "major" GUI changes on Canon's side (and the constants are model-dependent, though pretty much compatible across DIGIC 4 and 5 models). Small dialogs (such as ISO) are not covered here.

For the latter, I use display_idle(). In Lua, this is exposed as display.idle(), although these should be probably all grouped into some sort of GUI routines (along with SetGUIRequestMode, for example) and abstract some model-specific differences. That's where the GUI emulation in QEMU is going to be useful.

A sensel cannot be read out at two different ISOs at the same time (so you are going to have some missing data in either spatial or temporal domain). In the first case you'll get aliasing; in the second one, you'll get motion artifacts.

This might be doable though:

Very likely a Lua bug. How "randomly" do you experience it? Is it a transient error that goes away after a second or so? (I assume not, since you could take a screenshot)

Searching this error message gives

To me, it looks like qemu didn't finish compiling.

General Help Q&A / Re: 6D battery drains while camera is off
« on: May 17, 2017, 09:19:20 PM »
You (or other 6D user who has this issue) will have to do some more detective work to narrow it down.

1) Find a quick way to tell whether the battery is draining or not. If 10 minutes give a noticeable difference, that's fine. An ammeter (if you don't mind hijacking an external power supply) should be even quicker. If you have to take the battery out before starting the camera again, that's a very clear sign.

2) Save your ML configuration (ML/SETTINGS directory on the card), then reset ML to defaults. Do you still experience battery drain?

3) Try in-between configurations (between ML default and your usual one) until you find the setting that causes battery drain.

BTW I use Visual Studio Code for my Lua scripting and I must say I simply tab to make it 'look good', eg indents.

Looks like this should help:

General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: May 17, 2017, 08:00:20 PM »
-singlestep does not produce visible results by itself - it affects the way QEMU translate the guest code (that is, a TranslationBlock will contain only one guest instruction). The program will still run just like before, maybe with a minor speed penalty.

The speed penalty is minor because TranslationBlock's are chained (linked), so an execution "step" will include more guest instructions. To prevent this chaining, you can also pass "-d nochain"; this mix of flags does have a noticeable speed penalty, but it's very helpful when writing analysis code on top of QEMU.

If you are trying to print all the instructions, as they are executed, try:
Code: [Select]
./ 1300D,firmware="boot=1" -singlestep -d nochain,exec [...]

If you are OK with printing each instruction as it's translated (that is, only the first time the emulator encounters it), you get a massive speed boost by omitting nochain.

If you want to run it step by step, you can do so with GDB commands. You can place a breakpoint where autoexec.bin loads (0x800000) and run it step by step from there. It's very slow that way - I prefer collecting larger logs.

You can also toggle logging options from the QEMU monitor console (e.g. during a breakpoint set in gdb), but it's a bit of a hassle. Can probably be scripted (e.g. start logging with options X, Y, Z once the PC register reached address Q). If I need such triggers, I just hardcode them somewhere in the TB exec hook; for example:
Code: [Select]
static void tb_exec_cb(void *opaque, CPUState *cpu, TranslationBlock *tb)
    if (tb->pc == 0x800000) {
        qemu_loglevel |= CPU_LOG_EXEC | EOS_LOG_IO;

PrefetchAbort sounds like the code likely jumped to some invalid memory address.

Yes, exactly there.

To specify the C++ compiler, try this:

Code: [Select]
CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

It almost compiles with clang, except for a tiny function which you can safely comment out, and some warnings. Will look into them.

For SDL, try --disable-sdl.

Camera-specific discussion / Re: Canon 700D / T5i
« on: May 17, 2017, 03:44:40 PM »
That means, it's very likely a hardware issue. Not much to do at this point - just for curiosity, you can now run the startup logger as well, but I don't think it will be of any help.

Camera-specific discussion / Re: Canon 700D / T5i
« on: May 17, 2017, 03:22:34 PM »
If the shutter makes noise with the portable display test on the card, that means the card is not bootable (in other words, the test was not loaded). After loading*) and displaying the test screen, it should lock up, forcing you to take the battery out in order to get any more signs of life (such as LED blinks) outside the viewfinder.

*) at this step, a short LED blink may or may not be visible, depending on camera model; would be nice if another 700D owner could check.

The viewfinder is controlled by another processor, and whatever it does during the display test (no signs at all, busy, or whatever) is model-dependent, not controllable by ML, and thus not relevant for the test.

You can also try this (just curious why the pictures are not saved to the card):

(it would blink the LED for the first 30 seconds, then save a log file that contains all the debug messages from Canon)

Comment out the affected line; plain make is fine.

Or, try something like this:

Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Compiling without SDL should be fine (it's used that way on the build server, where it runs without GUI).

Pages: [1] 2 3 ... 331