autoexec.bin on Canon XF605

[ponguin]

XF605's UART (see https://www.magiclantern.fm/forum/index.php?topic=26807.msg242477#msg242477) provides, inter alia, the EnableBootDisk and DisableBootDisk commands in an eventproc shell.

From first inspection, Canon XF605's bootloader appears to support booting via autoexec.bin, if the "bootflag" is set.
on XF605, bootflag resides at 0xE3FF8004 in ROM (and 0xDFFE2084 in TCM).

This thread shall be about building a "ML style" autoexec.bin that will facilitate development of custom code for XF605.

[ponguin]

Not exactly "ML style", yet heavily inspired by ML and chdk (big thanks especially to reyalp, kitor and names_are_hard!), here a rather unpolished, highly experimental proof-of-concept that works around the PushAF bug of Canon XF605 stock firmware:
https://chdk.setepontos.com/index.php?action=dlattach;topic=14803.0;attach=18111

20230915_workaround_pushaf_bug.zip.asc

Code: [Select]

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQTuqjGte1kC77493zByb9iJKwGigAUCZQTGyQAKCRByb9iJKwGi
gPz8AP9EFSsDRlV0WWxZVNnKd9E8/H2i6mOFK+rzRIAbYyvV2AEAuZY5sCANCLtq
dleLZ7A5Nr0v6l09fAGsKIK8cue7FgE=
=taaw
-----END PGP SIGNATURE-----
This version supports the following XF605 firmware versions:

Code: [Select]

    FW    "real version"  displayed as
  100390   1.0.0(3.9.0)     1.0.0.1
  100392   1.0.0(3.9.2)     1.0.0.1
  101416   1.0.0(4.1.6)     1.0.1.1
Compile
E.g., to compile for the oldest supported firmware version, use:

Code: [Select]

make FW=100390
The "clean" target can be run with any valid firmware version, e.g.:

Code: [Select]

make clean FW=100390
How does it work?
The Makefile compiles two binaries: resident.bin and afterwards autoexec.bin, which contains resident.bin as binary blob.

When a XF605 with BOOTFLAG enabled is switched on, the first-stage bootloader checks both SD-card slots whether an SD-card is present and whether it is marked bootable. If that is the case, it assumes that autoexec.bin exists on it and loads its content into memory (at 0x00800000) and jumps there (assuming valid thumb code).

autoexec.bin is created from start_autoexec.S, autoexec.c, crc32.c and the corresponding stubs_*.S file.

start_autoexec.S places a single jump instruction in the .init section which is designated to comprise the very beginning of autoexec.bin. Furthermore, it stores resident.bin as binary blob alongside its length and some easy-to-spot marker.

autoexec.c contains the code being jumped to from start_autoexec.S and is entered in our_main. It checks for a compatible first-stage bootloader, second-stage bootloader and ICU firmware version by calculating and comparing the CRC32 of the respective flash memory region. If possible, commenting debug strings are output to the serial console using first-stage bootloader functions.
It copies the resident.bin blob to a lower RAM address, choosen such that the binary blob ends close to the address that marks the end of what appears to be the heap used by the original firmware.
It appends a copy of the position-independent code that comprises the original second-stage bootloader. The second-stage bootloader copy gets patched to return to the resident.bin copy instead of directly advancing into the code it copies into RAM.
Finally the patched second-stage bootloader copy gets executed and finally jumps to resident.bin.

resident.bin is created from start_resident.S, resident.c and the corresponding stubs_*.S file.

start_resident.S places a single jump instruction in the .init section which is designated to comprise the very beginning of resident.bin. Furthermore, it defines a label called _posts_to_swhack_msgq_hook which marks a "function" carefully written in assembler.

resident.c contains the code being jumped to from start_resident.S and is entered in on_return_from_canon. It patches the code that the second-stage bootloader copied from flash to RAM: The heap size is being reduced such that the carefully placed resident.bin copy is never overwritten by the original firmware. Furthermore, it hooks into a button-handling function of the original firmware and redirects the control flow to the "function" carefully written in assembler. It does so by moving eight bytes from the target function containing only position-independent instructions into placeholders within the destination function and overwriting the source with a jump there. The destination function essentially adds a call to a C function in resident.c containing logic to work around the PushAF bug.

The workaround consists in overriding AF_mode while PushAF is being pressed if AF-boosted MF would be in action otherwise.

[names_are_hard]

Hey, nice, congratulations!  It should be much easier to diagnose problems and extend functionality now you have a working boot process!  Shame I don't have an XF605 to test with

Is the code available?  I'd like to compare the process, especially against our early support for Digic X cams, which seemed to share many features.  We have some significant problems remaining there, I'd be happy to take any improvements you've made.

Notes for ML devs: the XF605 is unique amongst cams I've examined, in that almost the entire ROM is copied to RAM before use.  So you can modify code very extensively, with no special effort!  Possibly this is a Digic DV thing, they might have a lot more ram spare?  Or perhaps they have RAM much faster than ROM, so it's a worthwhile step for perf.

[ponguin]

Hi! 

Full source code is available via the link at the end of the first paragraph of the last post.

Regarding "working boot process": this workaround gets by without creating any tasks. So there is still some significant step missing until this can be considered anywhere close to simple CHDK or ML examples.

Copying ROM to RAM might be due to speed, latency or consistent access time. All three reasons would make perfectly sense to me on a camcorder 

When trying to boot the workaround's autoexec.bin in qemu, it seems as if the SD-card emulation needs to be improved for actually making the first-stage bootloader find and load autoexec.bin. This would provide gdb-multiarch access and help a lot for further development Memory regions 0xD2B100xx and 0xD2B200xx seem highly related – does that ring any bell when comparing to Digic X? Executing XF605 autoexec.bin in qemu currently requires remarkable manual/scripted help from gdb-multiarch... :/

The first-stage bootloader is probably very similar to Digic X, so has there been by chance any progress in improving qemu from that direction?

[kitor]

When trying to boot the workaround's autoexec.bin in qemu, it seems as if the SD-card emulation needs to be improved for actually making the first-stage bootloader find and load autoexec.bin.

I remember doing some patches for SD code to work with R6 ROM. But it failed anyway very early in Canons init task.

I think I sent you already my diff for QEMU2.5 and Digic X. If not, ping me on Monday.