DIGIC 7 development (200D/SL2, 800D/T7i, 77D, 6D2)

[a1ex]

Some progress:
- verified and committed the 77D stubs from @aprofiti and @calle2010; nice job!
- DISP_SetUpdateOSDVram (suggested by @jack001214) was, indeed, the keyword for figuring out the display buffer address on all DIGIC 6/7/8 models
- committed the Hello World code from @names_are_hard (200D) and @chris_overseas (5D4) and cleaned it up a bit (hopefully still working)
- some definitions useful for porting
- dedicated stub macros for ARM/Thumb code and also for data (less error-prone)
- various other code cleanups (and some robots programmed to perform them)

To compile the minimal Hello World code (expected to work on DIGIC 4/5/6/7/8, most models):

Code: [Select]

hg up digic6-dumper -C
cd minimal/hello-world
make MODEL=77D clean
make MODEL=77D install  # or install_qemu, but won't work out of the box


77D FIR for enabling the boot flag: BOOT_77D.FIR (edit: confirmed by calle2010)

This will modify your camera (details)

Have fun!

[turtius]

Great News! Future seems bright for these camera's!

Also, (if you dont mind) can you briefly explain the process which took place too find this address when the memory region is  not 'cache-able'?
Thanks,

[a1ex]

Also, (if you dont mind) can you briefly explain the process which took place too find this address when the memory region is  not 'cache-able'?

Not sure I understand the question. Generally, an address below 0x40000000 is cacheable (i.e. any access to that memory goes through the CPU cache). An address above 0x40000000 (until 0xC0000000, but that limit is only reached on the EOS R for now) is uncacheable, i.e. memory access is done bypassing the CPU cache, i.e. reading/writing directly into physical memory.

Memory address 0x01234567 and 0x41234567 refer to the same physical address; the former is cacheable, the latter is not.

Regarding the display buffers - the code after DISP_SetUpdateOSDVram flips between two image buffers (given as pointers to MARV structures, documented earlier on the 80D thread). There is a data structure with 3 pointers:

Code: [Select]

struct bmp_vram_info
{
    struct MARV * vram1;        /* one of the two bitmap buffers - statically allocated? */
    struct MARV * vram2;        /* the other bitmap buffer */
    struct MARV * back_vram;    /* we need to write to the other one */
};

The first two fields are constant (confirmed from various RAM dumps), and the third one points to one of the first two. In the code that follows the OSDVram message, the third pointer is flipped after configuring the hardware registers. That means, it doesn't point to the currently displayed buffer, but to the "back" one, for some reason. This code pattern was present on all DIGIC 6/7/8 models I've checked.

Basically, this pseudocode:

Code: [Select]

DebugMsg(4, 3, "DISP_SetUpdateOSDVram(%#x)(%d)", MEM(0x1238), ...)
DISP_SetUpdateOSDVram(...)
if MEM(0x1238) == MEM(0x1230):      # third field same as first one?
    MEM(0x1238) = MEM(0x1234)       # yes, switch to the second one
else:
    MEM(0x1238) = MEM(0x1230)       # no, switch to the first one

So, the bmp_vram_info would be 0x1230 in this hypothetical case.

In ASM:

Code: [Select]

  LDR   R3, [R5,#0x1C]      ; VRAM pointer (to be printed - first arg after the format string)
  ...
  BLX   DebugMsg            ; DISP_SetUpdateOSDVram(...)
  BL    DISP_SetUpdateOSDVram
  LDR   R1, [R5,#0x1C]      ; VRAM pointer
  LDR   R0, [R5,#0x14]      ; VRAM1
  CMP   R1, R0              ; is it pointing to VRAM1?
  BNE   store               ; no, flip to VRAM1
  LDR   R0, [R5,#0x18]      ; yes, flip to VRAM2
store:
  STR   R0, [R5,#0x1C]      ; store the new VRAM pointer

Notice Canon's data structure is a bit larger than the 3 fields I've declared; didn't check the other fields yet, but they might be useful. The number of fields before these 3 VRAM pointers is model-specific, but I'm trying to write generic (portable) code whenever possible.

[calle2010]

77D FIR for enabling the boot flag: BOOT_77D.FIR

I have loaded BOOT_77D.FIR on an SD card formatted in camera, executed the firmware update process and ... it worked! 





I could compile minimal-d78.c for 77D.102 from commit 537045b7424c and the AUTOEXEC.BIN worked on the camera.

I've got another dump of my ROMs and this output, which must come from call("dumpf"):
https://bitbucket.org/snippets/calle2010/Ke4GB8#file-LOG000.LOG
Camera continued to work while the dumps were made and afterwards...

@Alex: You mentioned that you have a fix for struct fio_file. Are you going to commit this, too?

I guess the next step for 77D is to capture the log files and then try to get Hello, world running?

[a1ex]

I guess the next step for 77D is to capture the log files and then try to get Hello, world running?

Hello World is expected to run out of the box, without any tweaking (well, unless I've mistyped the stub).

@Alex: You mentioned that you have a fix for struct fio_file. Are you going to commit this, too?

Done.

I must be doing something wrong in my workflow:

- figuring out the changes to struct fio_file (by trial and error, printf-based debugging): about 15-30 minutes
- writing some reasonably clean test code to cover these changes, and manually testing it on 5 models in QEMU: about 1 hour
- including it in the QEMU test suite and making sure it passes on all other EOS models: a couple of days...

And after all that, it still doesn't pass the tests every single time I run the suite, but that's caused by imperfect emulation (some nondeterministic issue emulating the SD card on all DIGIC 7 models).

And I'm still wondering whether the tests will pass on the build server (watch it live, it will take an hour or so to run all the tests; the newly added test for FindFirst/FindNext/FindClose takes 4 minutes on my local machine).

Maybe I'm misunderstanding what TDD is all about

[calle2010]

Thank you, Alex!

Maybe I'm misunderstanding what TDD is all about

I was told in the last millenium that testing takes 10 times longer than development. I guess this hasn't changed. Only that you now can make a machine doing it over and over again, saving us from all the headaches! 

I haven't even started to dive into the testing machinery that you built...

[a1ex]

Right. And programming that machine takes 10 times longer than actual feature development.

It's still way faster than asking users to test every single "nitpick" change - see e.g. all these backend refactors made to have a sane Lua API. On some models, the feedback is immediate, i.e. in hours or the next day. On others... I get feedback after several months, and that's when I discover that basic stuff - such as mere ML menu navigation - is broken (but working in QEMU because of an incorrect assumption of mine). And that's just because of my desire to include in mainline only what's known to work on all models, without breaking existing features.

Wish I could fully test the entire ML feature set in QEMU at the press of a button, even if it takes a day or two (as it will be done by a robot). Currently, the tests running on the build server are covering only some basic things - whether it boots, whether one can browse the menus and other similar nitpicks.

[turtius]

Notice Canon's data structure is a bit larger than the 3 fields I've declared;

Yes i found my self nitpicking each log and found the same result against the ram dump.

That means, it doesn't point to the currently displayed buffer, but to the "back" one, for some reason. This code pattern was present on all DIGIC 6/7/8 models I've checked.

Ah! So now i see, when i was changing that region it affected a 'part' of the display which was probably the live view from the camera.

[calle2010]

I played with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP.

With commit 6a3ca36c I get a very clean DEBUGMSG.LOG with only "used/maybe unused" log entries.

I tried it two times. Then I found two memory areas with 6MB each consistently mark as "maybe unused".
Main difference between the two tries was that in the menu I clicked an item to setup a photobook. It seemed to read the pictures on the card (nearly 100 from previous experiments), so this try showed many more memory areas as used.

I'm a bit disappointed that I couldn't find a larger chunk of memory. I think logging will be very limited with only 6MB.

My procedure for the experiment:

Steps:
- put the card with autoexec.bin and close the card door
- wait for the LED to go dark
- start camera in M or P mode, High Speed Shutter enabled
- startup will take a couple of seconds
- wait for three seconds (LED blinks)

Do all of this within 50 seconds:
- open menu and do two or three steps
- take one picture in photo mode
- take one picture in LV mode
- take some seconds of video (in my case 8s video snapshot function since I don't use any other video function)
- go to image playback mode, zoom into one of the photos
- take pictures until buffer is full

- Important: Wait until LED blinks 3/4 times and all pictures are written to the card (may overlap)

Only until LED is dark for a minute:
- Switch off camera
- wait five seconds
- open card door
- wait five seconds
- take card, get DEBUGMSG.LOG

These are the two areas. All others are smaller:

Code: [Select]

5A400000-5A4FFFFF: maybe unused
5A500000-5A5FFFFF: maybe unused
5A600000-5A6FFFFF: maybe unused
5A700000-5A7FFFFF: maybe unused
5A800000-5A8FFFFF: maybe unused
5A900000-5A9FFFFF: maybe unused

69600000-696FFFFF: maybe unused
69700000-697FFFFF: maybe unused
69800000-698FFFFF: maybe unused
69900000-699FFFFF: maybe unused
69A00000-69AFFFFF: maybe unused
69B00000-69BFFFFF: maybe unused


[a1ex]

- take pictures until buffer is full

Assuming you are not going to log a full burst, you can skip this, for these reasons:
- a full burst will allocate as much memory as it can
- these hardcoded addresses are not used in "production" code, only for initial logging experiments
- you are not going to log a full burst anyway
- memory used for image processing is well away from DryOS internal data structures (so, no bad outcome expected, should we happen to overwrite this)

Taking 2-3 pictures in a row should be OK, just in case, but a full burst is probably overkill.

Also, once the firmware is started, one can use the "exmem" allocators to get large chunks of memory, so this trick is pretty much for startup logging only.

[calle2010]

Ol, Alex.  I will try again and not use the photobook function either

[chris_overseas]

I'm a bit disappointed that I couldn't find a larger chunk of memory. I think logging will be very limited with only 6MB.

Don't be too disappointed by this, I had the same problem with the 5D4. I created a workaround using a ring buffer that gets periodically flushed and hence allows continuous logging. With this I managed to get 45MB+ log files with 8MB of buffer.

It is still a bit experimental and not yet merged into the ML repo, but if you're willing to get your hands dirty with the code you can try merging in the relevant parts from my repo. The initial commit was this one but there were a few more improvements and fixes added later, so you'll probably want to cherry pick those bits too if you want to give it a try. Note also that it was only hacked together for the 5D4, but should be trivial enough to use on other cameras.

[calle2010]

Sounds great, Chris! If there is a need I will try it.

I found one area with 14MB which seems to be pretty stable, even with burst pictures, and three more areas with more than 40MB which should be good for lighter workloads.

Code: [Select]

/* area at 43700000 doesn't seem to be overwritten by burst pictures */
buf = (void *) 0x43700000           /* 43700000-444FFFFF = 14MB, 4BA00000 - 4E2FFFFF = 41MB, 4E400000 - 50DFFFFF = 42MB, 72A00000 - 753FFFFF = 42MB */
buf_size = 14 * 1024*1024;

I wonder if it is better to use the uncacheable address space, as defined for the 80D? Cause the log is likely to be written from both CPUs.
Still confused by what is cacheable and what not.

[buro341]

Just got the 77d. 
Very glad to see people working to get ML working on this camera!

thnx!

[calle2010]

hello-world on 77D:







In liveview mode and while recording a movie it blinks once per second. During menu display it gets wiped as soon as a new screen is shown. I think this is expected. Also at shutdown it overlays the sensor clean animation, blinking.

[buro341]

and hello world shows that with altering some code, ML will work on 77d?
anyways, I love that people are working on the 77D, it really needs ML to produce great video imo.

[calle2010]

From what I understand this shows that we can run code on the camera next to Canon's firmware and some people much smarter than I have figured out how to modify the screen buffer.

If you need ML to produce great video then you should not hold your breadth while waiting for it...

[aprofiti]

Good! Giving m50 able to navigate menu, should be only a matter of time before it can also be archived on d7 cameras.
Archiving MMIO logs could also help to enhance emulation in the future.

I don't have a 77d, so my help likely end here; nevertheless the development is already in good hands.

P.S: Looks like only 6D2 and 800D are missing the party... a1ex, any roms dump available?

[names_are_hard]

Hey guys! I recently bought a 200D and am quite stoked at the development on this forum! Would just like to know if there is anything I could do to help?

What kind of programming experience do you have?  I have a 200D and am slowing porting magiclantern.  There is not enough done yet to safely try running magiclantern, but if you are happy with C, ARM assembly, or reverse engineering, there's lots you could help with.

[turtius]

I have a 200D and am slowing porting magiclantern

How far did you get? I mean i am goofing around with my 200D too with the RAM dumps etc .If there was a list or some updates i'd follow it.

Also, any decoded buttons 'codes' are found yet?

[calle2010]

Would be great if we could share progress here in the forum and also have the links to our repositories.

Mine is: https://bitbucket.org/calle2010/magic-lantern-77d/commits/all (updated)
I will share whatever is halfway working for me.

I know this one with WIP for 200D: https://bitbucket.org/stephen-e/ml_200d/commits/all

Finally it probably would be good to have pull requests for Alex for any substantial progress.

I'm not very fond of IRC, especially because I think a lot of communication is lost when one is offline. No idea how to stay online for 24h. Is anybody interested in using Slack or similar for chatting? At least for the DIGIC 7 stuff, not as a replacement for IRC.

[chris_overseas]

My 5D4 work-in-progress repo is here: https://bitbucket.org/chris_miller/ml-fork/commits/all

It's digic 6+ of course but has some similar challenges to the ones faced with digic 7 so may well still be helpful.

[names_are_hard]

This is a useful starting place:
https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst?fileviewer=file-view-default#rst-header-adding-support-for-a-new-camera-model

The Finding Stubs tutorial is also handy:
https://www.magiclantern.fm/forum/index.php?topic=12177.0

IRC is a good place to ask "how do I find out about X?".

The stephen-e repo is mine, I am working on stub hunting currently.  There's about 80 stubs I haven't tried to locate yet, when I've taken a first pass through that I'll need to ask more questions.  If anyone with a 200D finds stubs further down in stubs.S them my "UNSURE or untested" section, let me know and I can add them.  Currently I have the hack hello world working, and the official CONFIG_HELLO_WORLD Magiclantern *building*, but it won't run; it will need some of the stubs that I haven't yet found.  I guess it might take 40 hours to try and find the 80 I know are currently wrong.  Then maybe another 40 to find the ones that are hard to find?  Then I don't know what is next; I guess maybe correcting the ML source code for assumptions it makes that aren't true for Digic 7.  I have noticed that some functions seem to have a changed number of parameters, perhaps that will become important.

If you can't afford IDA Pro + decompiler (about $4000), then I highly recommend Ghidra.  It's free, and has very similar features.  If you've not used something like that before I can get you started.  This lets you better compare a known good camera with a new camera, so you can find the equivalent functions in the new version.

You can stay logged into IRC permanently by using an IRC Proxy / Bouncer, then your IRC client connects to that.  ZNC looks alive, I've not tried it.

[calle2010]

That's a very good summary. I decided to give up on improving the emulation for now. This task is beyond my knowledge and I think I make no progress:
I tried to create 77D MPU spells but ailed because many important spells are not in the log from my camera and I don't know why.
I tried to analyze and debug the WaitPU1 issue but do not understand enough of DryOS semaphores, sequencers and other things to make any sense of it. I believe this blocks the firmware startup sequence
I have no clue how to fix QEMU for the new display GPU. I thought about having a breakpoint, dumping the VRAM every second to a file and convert and display outside of QEMU...  but as long as the emulation is stuck it doesn't make much sense.

So I think I will follow your path, trying to find the relevant stubs for ML Hello World. As the two models are very similar we should be able to benefit from each other.

I use Ghidra, too.  I wrote a small Getting Started for the 77D ROMs. It should be nearly identical for the other Digic 7 cameras: https://github.com/calle2010/magic-lantern-77d-vagrant/blob/master/ghidra.md
Im happy for feedback and tips since this is the first time I use a tool like this.

[names_are_hard]

It's a nice Ghidra guide, I like it!  I made some changes but git push gave me a 403, so I guess I can't submit a pull request?  I'm not very good with git, maybe it's something else.  I put my edits here: https://pastebin.com/vP5TNPt1

There are two things that I think will make your life much easier - I think you have the "language" wrong, and should be using ARM Cortex 32 little endian, not ARM v7. Second, you shouldn't load each ROM as a file in your Ghidra project (I made this mistake too).  This makes each file separate and disassembly can't see stuff in the other files, so you will get lots of broken refs.  Instead, load the main ROM, then use File -> Add To Program.  This puts all the files in the same address space and disassembly works much better.

Third...  I should really write a Ghidra script that reads stubs.S and disassembles and labels every code address.