Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - a1ex

Pages: [1] 2 3 ... 340
1
I've kept my 5D3 in LiveView for many hours, recording raw video over and over for testing purposes. Here's an example (30 hours, with external power supply). Don't remember the temperature, but it didn't trigger any warnings.

Want me to repeat the experiment and take some test images before and after?

2
Last time I've checked (read: 1-2 years ago), the audio meters should disappear when there's no sound recorded (be it in H.264 or mlv_snd). Is this no longer the case?

Headphone output is not currently under ML control.

4
Camera-specific discussion / Re: ML on EOS-M2
« on: July 24, 2017, 03:03:36 PM »
as discussed here I failed to use AllocateMemory pool but:

I'm ready If you want to give it another go.

Got it working in QEMU, on both 6D and 100D (they are very similar, and 700D is from the same group). EOSM2 is just a tiny bit different.

You'll probably need new-dryos-task-hooks for EOSM2 as well. Hopefully this time I'll get the confirmation from 6D and 70D in order to merge it in mainline :D

5
Camera-specific discussion / Re: ML on EOS-M2
« on: July 23, 2017, 08:36:46 AM »
In order to follow along on my disassembly I'm doing this:

Code: [Select]
0x7ABC + RAM_OFFSET

That's exactly the reason I've listed the RAM stubs in the ROM_ADDR - RAM_OFFSET format in stubs.S - so one can just copy the ROM address and look it up in a ROM disassembly, without requiring the RAM segments to be set up. I should probably include the RAM address in comments as well - a job for the stubs formatting scripts.

Quote
Ok--this works but how did you know to give r0 = 4 and r1 = 8?

Just picked two RAM addresses that I thought they won't be used in this routine. Since in some cases, null pointers are used for optional output arguments, I've chosen non-zero pointers. Addresses 4 and 8 are actually exception vectors - Undefined instruction and Software interrupt. Our routine is not going to trigger these, and they are easy to type.

x/2 4 means 2 items from address 4, and by default, an item is a 32-bit integer. Therefore, it's printing the memory contents at 4 and 8 (our pointers).

Alternatives:
- allocate on the stack (below $sp) - probably the safest option if you debug a real target
- use some high address that's unlikely to be used (anything above 0x10000000 on a 256M camera will do, since QEMU uses 512M for all models, to keep it simple)
- find some other address unlikely to be used (e.g. in the middle of some RAM string, or some other statically allocated variable)
- pick something random and repeat until it succeeds (emulation only!)

Quote
Code: [Select]
#define MOV_R1_0xC2A000_INSTR 0xe3a0???? // mov r1, 0xc2a000

You don't need to hardcode a MOV here - it uses PC-relative addressing. Look at ff0c3204.

Will take a closer look later, also for 100D, since now I have the tools to do so :D

6
Camera-specific discussion / Re: ML on EOS-M2
« on: July 22, 2017, 05:51:57 PM »
Quote
So other cameras don't need this or is the 500D supported better in QEMU than any other camera?

500D was the first to get menu navigation in QEMU, and right now there are still some things that only work there (such as changing display color, or the memcheck tool). Also, in some places such as photo capture, Canon code is simpler (easier to understand).

Quote
ff0c30a4:    ldr r1, [pc, #344] ; ff0c3204: (00c2a000)   <-- end address??
ff0c30a8:    ldr r0, [pc, #344] ; ff0c3208: (0046c000)   <-- start address??
...
ff0c30c4:    bl loc_ff146d3c <-- allocatememory_init_pool?

Almost there - start and end address are correct, but allocatememory_init_pool is not.

During a function call, R0-R3 are not preserved; in particular, R0 is also used for return value. Therefore, whatever was loaded in R0 and R1 are parameters for the function call that follows immediately, but not for the subsequent functions.

Therefore, allocatememory_init_pool is the one called right after loading the parameters into registers:
Code: [Select]
FF0C30AC                 BL      0x7ABC <-- this one

Note that both 0x81F0 (AllocateMemory) and 0x7ABC (allocatememory_init_pool) reference the same memory address (0x90CC8). The latter (0x7ABC) writes something to this address (the result of 0x79FC), and this call contains a solid hint about the meaning of R0 and R1: start and end. Why? Because 0x79FC receives R0 = start and R1 = end-start, so the latter must be size:
Code: [Select]
00007ABC
...                      ; nothing else touching R0=start and R1=end
00007AD0                 SUB     R1, R1, R0
00007AD4                 BL      sub_79FC

In other words, the AllocateMemory pool is from 0x46c000 to 0xc2a000. Note that, unlike my previous comments (je me suis trompé), this is not the system memory area:

Code: [Select]
(gdb) set $pc=0xFF0C77EC
System Memory Information
  Start Address       = 0x0018ac08
  End Address         = 0x002aac00
  Total Size          = 0x0011fff8 (  1179640)
  Allocated Size      = 0x00062328 (   402216)
  Allocated Peak      = 0x00064b30 (   412464)
  Allocated Count     = 0x00000051 (       81)
  Free Size           = 0x000bdcd0 (   777424)
  Free Block Max Size = 0x000bb240 (   766528)
  Free Block Count    = 0x00000003 (        3)

I have a feeling this might be where DryOS allocates stack space for its tasks. Let's check:
Code: [Select]
arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
...
Current stack: [18af38-18acb8]
...
Current stack: [1ed9b0-1ec9b0]

Confirmed. Finding the allocator for the system memory would be good for cameras with very little RAM (such as 1100D).

Back to our case - AllocateMemory uses a different block of memory (not sys_mem), so I'd better update the wrong comments in the source code.

Let's try to find out how much free space we have (GetMemoryInformation). From mem.c, we know this about it:
Code: [Select]
extern int GetMemoryInformation(int* total, int* free);

So, it takes 2 parameters, pointers to 32-bit integers. It will write the total amount into the first integer, and the amount of free space into the second integer.

Let's call it from gdb. Since we don't care about code execution after this function returns, we can just pick two arbitrary addresses to store the two integers (never do this on real hardware!)
Code: [Select]
(gdb) set $r0 = 4
(gdb) set $r1 = 8
(gdb) set $pc = 0xFFD25774
(gdb) b *$lr
Breakpoint 2 at 0xff352220
(gdb) c
Continuing.

Breakpoint 2, 0xff352220 in ?? ()
(gdb) x/2 4
0x4: 0x7be000 0x3148d0

Cross-checking: 0xc2a000 - 0x46c000 = 0x7be000.

So, that's what we have to patch in order to load ML.

TODO: I'd like to update 100D to use the "classic" boot process as well.

7
Camera-specific discussion / Re: Canon 70D (Beta-4a / 26th Oct)
« on: July 20, 2017, 05:53:56 PM »
Is there any obvious setting other than the Global Draw option that I should be checking?

If there's any precondition you need to set, it should be printed at the bottom of screen and the menu should be grayed out. Otherwise, it's a bug.

8
Reverse Engineering / Re: Firmware Dumping and Loading into IDA
« on: July 20, 2017, 08:09:08 AM »
Updated settings for current dumps:

- load ROM1.BIN
- [DIGIC 2...5] select ARM processor, architecture ARMv5TEJ (we have ARMv5TE on all models from DIGIC 2 to DIGIC 5)
- [DIGIC 6...7] select ARM processsor, architecture ARMv7 A&R (most likely Cortex R4 on D6 and A9 on D7)
- uncheck 'Delete instructions with no xrefs' (we have plenty of those)
- uncheck 'Perform no-return analysis' (IDA usually gets it wrong)
- RAM section: 0, size 0x100000 (we can always add more later)
- ROM start address: 0xFF000000 for 16MB dumps, 0xFE000000 for 32MB dumps, ROMBASEADDR for old non-fullsize dumps
- ROM size: 0xFFFFFC or 0x1FFFFFC (off-by-one bug in IDA; I have an older version, so YMMV)
- File load address / size: same as ROM.
- [DIGIC 2...5] go to FFFF0000 and press C (that's the bootloader)
- [DIGIC 2...5] go to ROMBASEADDR (usually FF810000 or FF010000 or FF0C0000) and press C (that's the main firmware)
- [DIGIC 6] [optional, may slow down IDA]: Load Additional Binary, ROM1.BIN, loading segment 0, offset FC000000, num. bytes 0x1FFFFFC (only if your IDA has the off-by-one bug)
- [DIGIC 6] go to FC000000 or FE000000 to find the start address (32-bit); go there and press C (that's the bootloader)
- [DIGIC 6] go to ROMBASEADDR (usually FE0A0000) and press C (that's the main firmware)

After loading the ROM, you will most likely need to load additional blobs, copied by Canon firmware from ROM into RAM and executed from there. You can find these by running the ROM in QEMU with "-d romcpy" (or just look them up in the QEMU test suite log).
Code: [Select]
./run_canon_fw.sh 60D,firmware="boot=0" -d romcpy

Then, run the firmware in QEMU to get an initial IDC script with all the functions called during emulation:
Code: [Select]
./run_canon_fw.sh 60D,firmware="boot=0" -d idc
...
60D.idc saved.

Getting around:
- find out where the bootloader code jumps to main firmware
- follow the startup code in main firmware until you find "cstart" (it calls bzero32, then passes init_task as parameter to create_init_task)
- look up something that resembles a debug message, passed as third argument (in R2) to a function called thousands of times (that's DebugMsg)
- look up something that resembles a task name (e.g. GuiMainTask, CSMgrTask) and find task_create
- look up AllocateMemory (usually prints NG AllocateMemory and calls GetMemoryInformation on failure)
- look up the "Illegal inter" string and find out who calls it (you'll find the interrupt handler; there's one for each DryOS core in the firmware)
- look up some more functions with the Finding Stubs tutorial.

You should now have the basics to start browsing around. Have fun!

9
General Help Q&A / Re: Field monitor for photography?
« on: July 20, 2017, 07:15:58 AM »
Have you used the build linked on that page? (it won't work with the regular nightly)

10
General Help Q&A / Re: Field monitor for photography?
« on: July 19, 2017, 09:07:16 PM »
Best guess: it probably can be solved by preventing the display from being turned off.

You can start the investigation by using the startup-log build to find out what happens when the monitor is turned off; you may post the logs here.

11
Camera-specific discussion / Re: Canon 80D
« on: July 19, 2017, 06:17:48 PM »
Looks mostly fine. In the second box, try Loading segment 0, Loading offset 0xFE000000, and (not sure if this is needed with latest versions) number of bytes 0x1FFFFFC.

12
Camera-specific discussion / Re: Canon 100D / SL1 (Beta-4b)
« on: July 19, 2017, 05:26:23 PM »
Right - and I trust this method better than Canon's implementation, so if it works, let's go with that.

I have a feeling the 70D might require it as well.

Side note: okay, so now it's clear it takes some weeks (up to one month) for something obviously broken (as in, not working at all) on a widely used feature in an experimental build, to be reported. Just in case you might be wondering why the progress is slow.

13
Camera-specific discussion / Re: Canon 80D
« on: July 19, 2017, 05:12:16 PM »
No, the ROM dump loads at FE000000 and FC000000. The ROM is mirrored and code runs from both addresses, so you can either define two projects if you have a slow PC like mine, or just one (using the additional binary option in IDA) if you have a fast one like emklap's ;)

FE0A0000 is where you will find the main firmware after loading the ROM. The initialization code starts at FC000008, and the bootloader (LILO-style, prints "BootLoder" on the UART char by char) starts at FE020000. To start the main firmware, the bootloader writes 0 to 0xD20C0084, then jumps to FE0A0000.

There are a few additional blobs copied from ROM to RAM; you can get them from QEMU if you run it with "-d romcpy" (or just look them up in the QEMU test suite log).

15
Camera-specific discussion / Re: Canon 100D / SL1 (Beta-4b)
« on: July 19, 2017, 12:07:39 PM »
Correct - are there any reasons to keep it disabled?

16
Camera-specific discussion / Re: Canon 80D
« on: July 18, 2017, 11:17:22 PM »
QEMU does not emulate cache behavior, other than a few status registers.

It's probably safe to perform these tests on the camera. However, at this point, you'll need some help from us to sign the binaries you want to run (currently the binaries can be run either as autoexec.bin or as FIR; the former needs a boot flag set in the ROM and the latter must be signed). To my knowledge, CHDK had trouble running the Canon firmware from a firmware update on most (if not all) digic 6 models, so we might have to enable the boot flag from the bootloader context, in order to run autoexec.bin. Traditionally, we enable the boot flag from main firmware, but we did enable it from bootloader on VxWorks models. It's probably best to try on a less expensive D6 model first; if anyone is willing to take the risk, I can prepare a FIR for enabling the boot flag in this way.

Having some other ARMv7 device to test things could be helpful, too.

17
Camera-specific discussion / Re: Canon 80D
« on: July 18, 2017, 09:56:31 PM »
The display backend changed significantly on DIGIC 6 - look it up on CHDK. I'm not sure of the pixel format used outside bootloader - first we need to run something on the camera alongside the main firmware (e.g. LED blinking).

Then we need to figure out how to print things on the screen; it will be probably similar to CHDK, but I'm not sure yet.

Only after doing this step we can see whether we need bmp_vram_info and LCD_Palette, or maybe some replacement.

The CanonGothic string is not present in the 80D ROM. No big deal - Canon fonts are not used in many places in our code.

BTW, the pixel format used in the bootloader is 8 BPP, palette-based; the palette is specified as YUV, but the U/V components are unsigned now (they were signed on previous cameras). See QEMU source and disp_direct.c on the recovery branch.

18
Sorry, no experience with Sigma lenses.

19
So is the next step adapting Canon EF lenses to the computer?

Joking aside, you've probably noticed the Debug -> Lens Info menu is actually usable on some models. This info comes from MPU messages.

That means, you can use the startup-log builds while starting the camera with various lenses, and find out which parts of the log are lens-specific.

If we want to log things such as changing focus distance or focal length in LiveView (where these are reported back to the ARM CPU), we have a small issue: the log buffer fills up really quickly in LiveView. It may make sense to prepare a different set of builds, with reduced verbosity (or maybe skip logging unimportant messages once e.g. half of the buffer is full). Will look into that.

I should also make the mpu_send/recv stubs mandatory, as they are now known to work identically in all models, from digic 2 to digic 6 (not sure about 7 yet). And I'll need such logs from all cameras anyway.

20
Scripting Q&A / Re: Programmatically changing image quality setting?
« on: July 18, 2017, 05:12:00 PM »
See the discussion from https://bitbucket.org/hudson/magic-lantern/pull-requests/753/bad-settings-warning/

(in a nutshell: this is a setting where Canon code doesn't do proper validation, so it may end up saving the setting into some nonvolatile memory, then giving ERR70, even on subsequent boots without ML; these settings are model-specific, therefore we need to whitelist the valid settings for each model)

21
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: July 18, 2017, 12:09:07 AM »
Let me know if I can help in any way.

Of course. However, I'm afraid you'll have to... well... read the previous posts.

In particular, go to http://builds.magiclantern.fm/ and scroll to "Your camera is not listed?"

If you are waiting for me to port ML, it might not be the best choice. I'm providing tools, walkthroughs, tutorials, advice and so on, other community members did their part (in particular, this tutorial is very helpful), but it's up to somebody who owns the camera to go through all this and complete the port.

I expect this to be one of the easiest cameras for porting ML (it's DIGIC 4, but has some things borrowed from both D5 and D6). 1200D and EOSM2 are marginally easier, but that's just because the emulator is able to display the GUI.

22
First of all, 5D4 has mostly Thumb code, and ARM-console won't help with that (it's ancient stuff I wrote before getting an IDA license, ARM only). Look on CHDK forum for suggestions (from there, I've tried capdis, but only for double-checking small snippets of code where I didn't trust the output from IDA).

Many stubs are easily identified from strings; chris_overseas found some (look up his repo). Running the firmware in QEMU will save an IDC with what functions got called during that session. Currently the emulation works for the bootloader (which is ARM code) and it also runs a tiny part of the main firmware.

Memory layout is similar to other D6 models (look up my notes on e.g. 7D2 or 80D threads). Or just read it from QEMU logs.

First puzzle to solve: how to jump to main firmware from autoexec.bin? Note the 5D4 has two cores, and I believe we have to do something about it somehow.

Next step (which is easy, will probably work in QEMU, but not on the real hardware) would be to adapt the 7D2/80D startup code (which... works in QEMU, but not on the real hardware).

Third step would be making the startup code work on the real hardware (likely as easy as copying a routine from CHDK, discussed on 80D thread). Since I don't have a D6 model yet, I didn't feel any rush to try it.

After that, porting ML should be more or less straightforward (I don't expect major issues, other than some special handling for Thumb stubs).

For the first puzzle (which was solved for single-digic D6 models, and was a non-issue on earlier single-digic models), I've made some limited progress understanding the IPC protocol (communication between the two cores); if you (or anyone else) is interested in continuing this investigation, I can publish them in the current state.

23
Shoot preparation / Re: Stuck at 640x480 30FPS on Canon 50D
« on: July 17, 2017, 08:11:46 PM »
When all else fails, have a look at the source code:
Code: [Select]
    {
        .name       = "Movie Record",
        .priv       = &lv_movie_select,
        .select     = lv_movie_toggle,
        .select_Q   = lv_movie_size_toggle,
        .update     = lv_movie_print,
        .help       = "Enable movie recording on 50D :) ",
        .depends_on = DEP_LIVEVIEW,
    },

This is ancient code (2011), so patches to improve usability are welcome ;)

24
Camera-specific discussion / Re: ML on EOS-M2
« on: July 17, 2017, 06:15:22 PM »
You may wonder - why are all these messages doubled?
Code: [Select]
[    PowerMgr:ff222b94 ] (80:16) --- Common Top ----
   186:  3179.520 [RSC] --- Common Top ----

The first format is written by "-d debugmsg" (we are printing the arguments to DebugMsg and some context - task and address where it was called from).

The second format is written to UART (possibly one of these pins) by Canon code - they do that for important messages (whatever that means - we don't know much about it).

You can redirect the UART messages to a file (e.g. "-serial file:uart.log"). The bootloader menu (printed e.g. if you start from a bootable card without autoexec.bin present) also goes to the serial port - you can interact with it from the console if you use "-serial stdio". These are standard QEMU options - you can use them on any other guest OS that uses a serial port.

TODO: the serial console is also usable while running the firmware (and you can call these info functions from there - example). The emulation is probably missing an interrupt or some status bits on the UART in order to accept incoming characters.

25
Camera-specific discussion / Re: ML on EOS-M2
« on: July 17, 2017, 12:39:54 AM »
Now how did you run that?

As simple as...

Code: [Select]
./run_canon_fw.sh EOSM2,firmware="boot=0" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
CTRL-C
(gdb) set $pc = 0xff222b64
(gdb) continue

Quote
Is that two ways of looking at the same thing?
Yes.

Quote
Before and after memory allocation?
No.

Quote
Ok--so what are we looking at here?

Besides standard C malloc (aka user_mem in DryOS), and a similar allocator named AllocateMemory (sys_mem - nope, that's a third one), which both live in a tiny space of about 12MB (the gray bar at the left - 13MB on most models), for the memory-hungry stuff, Canon code uses a bunch of hardcoded regions (which may change according to operating modes). The graph shows these regions (top: scaled to show their real size, bottom: scaled so you can read the text).

Some of these regions are managed by allocators (shoot_malloc, srm_malloc), but it's not very clear which and when. It's one thing in photo mode and another in LiveView. Consider modes like video recording, raw developing, direct print (anyone uses it?) or wifi operation, and it starts to get really complicated. The safest way is to rely on these allocators when a large amount of memory is needed, and free it as soon as you no longer need it (otherwise you may not be able to change between some of the above modes). That's why the memory backend avoids shoot/srm allocators until it has no other choice. That's also the explanation for the second way to trigger ERR70 described in the stack trace PR.

In some cases, there are memory areas that are never used by Canon code (they are marked as full-height gray bars). We can double-check whether these are really unused (by making sure nobody writes to them while operating the camera in all possible modes we can think of), and if they are really free, we can take them. Caveat: they might be using stuff that's not listed in this log (as parts of that log are also hardcoded, so there's nothing to make sure what's in the log actually matches reality - they could have printed any numbers). On 60D, a region identified as free in this graph was later found to be actually used by Canon code (see the RscMgr PR, where we ran into trouble because of that). So, take this graph with a grain of salt.

For ML, we only need about 512K to load autoexec.bin (compare that to 128MB or even 64 for the 1000D). The other stuff (modules, work memory, whatever) are allocated dynamically, which is less troublesome, as the memory backend figures out which allocators to use, and lets you configure additional ones. But the initial code (autoexec.bin) has to be loaded statically, at some predefined address (unless you relocate it).

Of course, we can move everything to modules and end up with a very small static binary. It's a little more difficult than copy/paste though. Or you can try to compile as Thumb (good luck figuring it out).

Or, we can disable some features. You can even comment out features.h, CONFIG_RAW_* and fix up whatever gives compilation errors - that way, you'll get just the menu with a few debug items inside. It's best to do that anyway, and enable features one by one - otherwise you may end up with lots of possibly broken features (as it's really hard to test every single thing - I used to do that manually years ago), and if it's not something really widely used, you get a bug report after some months or years (even if the feature is not working at all).

Exercise: draw a similar memory map for DryOS (the first ~13 MB), based on the info from this post. Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

After finishing this, you will have both the Hello World working and a very good understanding of how the memory is organized.

Pages: [1] 2 3 ... 340