DIGIC 7 development (200D/SL2, 800D/T7i, 77D, 6D2)

Started by feedrail, June 12, 2017, 07:05:50 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

names_are_hard

Thought I'd do a not very exciting update, so people know I haven't given up :)  I am mostly on the Discord these days, but will summarise things every so often:

- I took a step back and got a lot more familiar with the ML codebase
- I've merged a few ML branches in my git repo, meaning I don't need to keep switching branches to keep things in sync
- I updated Qemu to 4.2 (still some problems, but easier to build than old Qemu, and better in some ways)
- it is now easier to run stable test code on 200D

a1ex

Quote from: names_are_hard on December 17, 2020, 01:21:44 AM
- I took a step back and got a lot more familiar with the ML codebase

This is actually the step I'm most excited about :)

Yes, being able to discuss the low-level details to somebody who actually knows what they are talking about, makes a huge difference for me.

Quote
- I updated Qemu to 4.2 (still some problems, but easier to build than old Qemu, and better in some ways)

This one is also a significant update, as QEMU will be an essential piece of puzzle in supporting recent models (read: DIGIC 4+/6/7/8/X). There are significant - likely massive - amounts of low-level work ahead, but, without the ability to run automated tests for every single supported camera model (whether in emulator or... somehow on real hardware), I don't see any other reasonable way to support ML on all of these new models.

The development kit from @coon will be another essential piece of puzzle - besides exploring camera internals, it will allow debugging ML in real-time while running on real hardware, i.e. exactly where names_are_hard stumbled at his previous attempt. And I'm sure we'll find plenty other good uses for it in the near future.

Chapeau 8)

names_are_hard

Hah, I'm glad someone likes it :)  Maybe we should have a planning session soon, with the other active people.

Full agreement around automated testing.  The more complex the problem, the more important reducing human testing is.

Danne

Lowlevel work. I personally would donate to see work expansion here. Deepest respect.

nikfreak

[size=8pt]70D.112 & 100D.101[/size]

Maix

Hows the development going?? (why'd you delete this before admin?)


names_are_hard

Diagnostics will now be easier:

Debug contacts, biro for scale:


Post surgery (waiting on nicer connectors but I got impatient, this ugly way will do for now):


I only tore off one pad.  Your sacrifice will not be forgotten, pad number 5.

wlfbbz


names_are_hard

Did this a while back but forgot to update:

Tiny wires:


Bonus serial ports:


Debugging in action:

names_are_hard

After a few failed attempts, I have worked out the magic required to get a CONFIG_HELLO_WORLD build working in platform dir.  There's the minor bug that it stops the screen turning on, but you can't get everything right first time.  The code is running to hello_world(), LED blinks prove it.

This is a major step, I think it's the first public build for Digic >= 7 that is using the real build system.  Still many steps to go.

Current WIP is here:

https://github.com/reticulatedpines/magiclantern_simplified/tree/200d_hello_world_fix

names_are_hard

Blank screen bug is fixed by a short sleep to, presumably, let the display get initialised.

names_are_hard

Improved CONFIG_HELLO_WORLD platform/200D build:



uart_printf() works too, between that and onscreen debugging, it should be easier than before to work out what that annoying TouchUtility error is.

Not recommended unless you know what you're doing, but the code is here: https://github.com/reticulatedpines/magiclantern_simplified/commit/051714b613644a6f0d04f0618049d41b2a2318b1

It should be fairly easy to adapt to other DIGIC 6, 7 and 8 cams.

coco_goat

Quote from: names_are_hard on March 01, 2021, 08:27:25 AM
Improved CONFIG_HELLO_WORLD platform/200D build:

Yay, thank you for your work, names_are_hard! I'm trying to run this on my 200D, no luck so far.

- I've downloaded BOOT200D.FIR from this post https://www.magiclantern.fm/forum/index.php?topic=19737.msg212603#msg212603
- Enabled boot flag (I guess. "BOOT=-1" is okay?)



- Made my card bootable



- Compiled autoexec.bin


> .../magiclantern_simplified/platform/200D.101/git rev-parse --short HEAD
051714b61

> .../magiclantern_simplified/platform/200D.101/make
Using ~/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gcc (preferred).
[ VERSION  ]   ../../platform/200D.101/version.bin
......... strip .........
[ LD       ]   magiclantern
[ OBJCOPY  ]   magiclantern.bin
[ STAT     ]   magiclantern.bin
magiclantern.bin: 210700 bytes
[ CC       ]   reboot.o
[ CC       ]   disp_direct.o
[ CC       ]   font_direct.o
[ CC       ]   footer.o
[ LD       ]   autoexec
[ XOR_CHK  ]   ../../build_tools/xor_chk
[ OBJCOPY  ]   autoexec.bin
[ XOR_CHK  ]   autoexec.bin

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000100 0x001b6300 0x001b6300 0x3370c 0x3f5fc RWE 0x100

Tip: to compile faster, try one of these:
    make -j4
    export MAKEFLAGS='-j4'

[ SYMBOLS  ]   magiclantern.sym
[ CP       ]   200D_101.sym


- Dropped autoexec.bin on my card (just autoexec.bin, without ML directory)

... and my camera freezes right after boot.
I see "sensor cleaning" animation and that's all, it does not respond to keys, or, at best, reboots.
What I'm doing wrong? If I'm doing everything right, could you please share your autoexec.bin? Or maybe I can somehow dump some logs to card?

UPD: It actually works. ML directory was mandatory :D



And even ML itself is kinda trying to work, hehe, throws some asserts:



ML ASSERT:
0
at ../../src/bmp.c:69 (BMP_VRAM_START), task console_task
lv:0 mode:0

console_task stack: 2c3580 [2c36d0-2c26d0]
0xUNKNOWN  @ 1b6b7b:2c3580

Magic Lantern version : Nightly.2021Mar03.200D101
Mercurial changeset   : NO_HG
Built on 2021-03-03 21:43:07 UTC by username@tp.
Free Memory  : 0K + 2759K


ML ASSERT:
0
at ../../src/bmp.c:69 (BMP_VRAM_START), task notifybox_task
lv:0 mode:0

notifybox_task stack: 2ad448 [2ad690-2ac690]
0xUNKNOWN  @ 1b6b7b:2ad448

Magic Lantern version : Nightly.2021Mar03.200D101
Mercurial changeset   : NO_HG
Built on 2021-03-03 21:43:07 UTC by username@tp.
Free Memory  : 0K + 2760K

c_joerg

EOS R

names_are_hard

@coco_goat - welcome aboard, glad you got it working!  I will warn you, it's almost entirely untested.  Chances of it breaking something are much higher than real ML.  It looks like you tried to do a non-CONFIG_HELLO_WORLD build.  That's especially untested and increases risk as it will try to use more functionality.  People willing to work with the code and try and understand what's going on are definitely welcome though!

Consider joining the Discord, you can get quicker feedback there and I don't post everything to the forums, only the stuff that's more generally interesting.

T7i owner

Quote from: names_are_hard on March 04, 2021, 02:50:17 PM
Consider joining the Discord, you can get quicker feedback there and I don't post everything to the forums, only the stuff that's more generally interesting.
Anyway I can join the discord?

I would love to test stuff out, meanwhile I'll still work on my C learning skills ;)

names_are_hard

Scroll up, hit the Discord link.  It's not at a stage where things really need testing, and I don't recommend anyone run the code in my repo without understanding it first.

coco_goat

Quote from: names_are_hard on March 04, 2021, 02:50:17 PM
@coco_goat - welcome aboard, glad you got it working!  I will warn you, it's almost entirely untested.  Chances of it breaking something are much higher than real ML.  It looks like you tried to do a non-CONFIG_HELLO_WORLD build.  That's especially untested and increases risk as it will try to use more functionality.  People willing to work with the code and try and understand what's going on are definitely welcome though!

Consider joining the Discord, you can get quicker feedback there and I don't post everything to the forums, only the stuff that's more generally interesting.

Thanks, I'll visit you someday :) I had some experience in reversing x86 code with OllyDdg and tried to dig a bit 200D firmware in IDA/Ghidra, but it looks quite hazy so far. Toolchain itself is very inconvenient, e.g. I have no idea how to quickly load 200 ROM fragments produced by romcpy.sh to different addresses, what's preventing running 200D FW in QEMU, what are that gdb patches (and how to use gdb at all, lol), how ML bootstraps itself, like where does control go after after my_big_init_task... etc. But at least it's now builds!

--

BTW have you guys seen this? https://github.com/turtiustrek/magiclantern_simplified
Someone ported avrcraft to 200D! (it's Minecraft server, not client)


Walter Schulz

Yes, we've seen this. Turtius is quite active in ML Discord ...

names_are_hard

I'd say the toolchain is not truly inconvenient, it's more that the problem is hard - and the toolchain isn't documented as well as it could be.  Once you've worked out how to use it, it's good for the normal problems we're trying to solve.

QuoteI have no idea how to quickly load 200 ROM fragments produced by romcpy.sh to different addresses
I used dd to put romcpy output that had similar addresses into one file, then mapped those few combined files into memory space for my 200D project.  One way to do that last part in Ghidra is the Display Memory Map icon.

Quotewhat's preventing running 200D FW in QEMU
Not quite clear what you mean here, it does run.  It doesn't emulate fully (or even very far).  And of course, improving Qemu emulation is a much needed task; there are always camera specific behaviours that need to be understood so Qemu can be made to handle them.

Quotehow ML bootstraps itself, like where does control go after after my_big_init_task
https://a1ex.magiclantern.fm/bleeding-edge/ml-startup.png
By a1ex, from this thread:
https://www.magiclantern.fm/forum/index.php?topic=23641.0


fgervais

I'm trying to estimate my chances of making the 800d port on my side before I buy the unit.

Not sure how you guys normally exchange files but I'm leaving a dropbox file request link here in case somebody could drop the 800d ROM:

https://www.dropbox.com/request/lNqRrP4SW4a10SUflsGQ

names_are_hard

What background do you have in reverse engineering / embedded development / C / ARM asm?  If you're strong in any one of those, your chances of eventually completing a port are high, if you are persistent.  But I would expect it to take months / years.

If you want to use ML, get a cam that's supported.  If you want to port ML, that's great!  We will help you learn what you need to get started.  Expect it to take a long time.  Most people that start, give up.

Porting becomes significantly easier if there is a supported cam for the same generation of Digic.

fgervais

Yes on the technical side of things I'm fine. I can definitely do the port if my life would depend on it ;) but I'd like to start looking at the firmware and see if this would fit in one of my free time slice.

If you want to know more about me out of curiosity, this page pretty much sums it up:

https://fgervais.github.io/

names_are_hard

Cool, you'd probably enjoy it.  We have a few active devs at the moment, we're okay on the reversing side and general HW understanding, but not skilled at ARM internals.  A1ex has good ARM knowledge but, currently not enough time.

Let me give you a quick summary of some random points:
- Can run arbitrary code, from C, on real cams (this should include 800D).  This can be triggered interactively when Canon code (DryOS) is running.
- Can emulate cams to varying degrees in a custom version of Qemu.  Old cams have full GUI menus and can even simulate taking pictures, etc.  New cams (including 800D) don't emulate that far but can do early init before getting stuck.  Not crashed: they go to an idle state, waiting for tasks from a queue that is empty.  Somewhat earlier I see Canon code checking a struct in mem that is not initialised.  On real hardware it is, so whatever inits that is a missing piece, though perhaps not the direct cause.
- With a little work for the physical connection, you should be able to get UART access to the cam, includes an interactive shell.  There are different shells for different processors.
- ROMs are quite friendly to work with, no obfuscation and they contain many debug strings, including an assert function
- Overall structure of Canon OS, DryOS, is fairly well understood.  RTOS, task based via a queue, interrupts.  Some of the messaging protocols between peripherals are understood.
- ML abstracts camera internals to some degree.  There is good support for a range of Digic 4 and 5 cameras.  800D is Digic 7, getting ML working on Digic 7 is mostly working out how 7 has diverged from 4&5 and updating ML to support this.  Structures used by Canon APIs may have new fields, HW abstraction in ML may need updating for new co-processors, etc.