First of all, 5D4 has mostly Thumb code, and ARM-console won't help with that (it's ancient stuff I wrote before getting an IDA license, ARM only). Look on CHDK forum for suggestions (from there, I've tried capdis, but only for double-checking small snippets of code where
I didn't trust the output from IDA).
Thanks a1ex. Ah, I knew about the Thumb code but didn't realise ARM-console was not the way to go with it.
Thanks for the advice. I tried capdis and it worked but disassembled without text strings when I tried it.
I had better luck with the GPL Disassembling advice at CHDK and the disassemblev7.pl from there produced a disassembly with text strings. I now have disassembled code which agrees with the stubs found by chris_overseas. I have found a few more stubs and have registered at bitbucket. I will fork the ML repo once I have figured out the bitbucket workflow ( chris_overseas has given me his ideas for what to do at bitbucket ).
Many stubs are easily identified from strings; chris_overseas found some (look up his repo). Running the firmware in QEMU will save an IDC with what functions got called during that session. Currently the emulation works for the bootloader (which is ARM code) and it also runs a tiny part of the main firmware.
Yes, I had seen the initial stubs work from chris_overseas and it was that which inspired me to have a go to see what I might be able to do. I have been in contact with Chris and he has been very helpful.
I am yet to get QEMU up and running but that is my next step.
First puzzle to solve: how to jump to main firmware from autoexec.bin? Note the 5D4 has two cores, and I believe we have to do something about it somehow.
Next step (which is easy, will probably work in QEMU, but not on the real hardware) would be to adapt the 7D2/80D startup code (which... works in QEMU, but not on the real hardware).
Third step would be making the startup code work on the real hardware (likely as easy as copying a routine from CHDK, discussed on 80D thread). Since I don't have a D6 model yet, I didn't feel any rush to try it.
After that, porting ML should be more or less straightforward (I don't expect major issues, other than some special handling for Thumb stubs).
For the first puzzle (which was solved for single-digic D6 models, and was a non-issue on earlier single-digic models), I've made some limited progress understanding the IPC protocol (communication between the two cores); if you (or anyone else) is interested in continuing this investigation, I can publish them in the current state.
I would be happy to look at the jump to main firmware from autoexec.bin problem to see if there is anything I can figure out. So if you can publish your work on the IPC protocol, please do.