DIGIC 7 development (200D/SL2, 800D/T7i, 77D, 6D2)

Started by feedrail, June 12, 2017, 07:05:50 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

EKchan


names_are_hard

Hello and welcome!
Quote from: hkisgg on November 23, 2019, 06:02:25 PM
I have seen posts on finding stubs. I need help with that if you kindly show me how to get stubs file(s) and 1 example of finding stubs(if there are threads or posts). I can help with the development of ML for Canon 77D. (Totally noob, I'm sorry if anything wrong)
Here's a good start on stub hunting:
https://www.magiclantern.fm/forum/index.php?topic=12177.0

If you don't have IDA, you can use Ghidra.  Calle2010 was working on a 77D and has a repo with instructions for getting started with Ghidra, as well as a repo for his 77D porting work (which has some stubs found):
https://github.com/calle2010/magic-lantern-77d-vagrant/blob/master/ghidra.md
https://bitbucket.org/calle2010/magic-lantern-77d/commits/all

EKchan - what experience do you have with C, assembly, or reverse engineering?

names_are_hard

I have made useful debugging progress.  Something is flakey with task code, which can cause weird behaviour / crashes at unpredictable times after doing the task.  This was very hard to understand due to the delayed effect.

I think it's probably related to the significant rework Calle2010 did around tasks.  I'd seen that before and forgotten about it.  I can't find a long explanation for why he did that.  To do with task_create not returning a pointer, it reads like, but beyond that I don't know.
https://bitbucket.org/hudson/magic-lantern/pull-requests/958/replace-is_taskid_valid-with-direct-access/diff

So, next up is understanding that change and porting to mine.  For now I've disabled all tasks in my_big_init_task(), which gets me further in my memory hooking job, but disabling the tasks means I don't get crash logs...

a1ex

Yes, these task updates should be applied to all DryOS models (including the old ones). In particular, they will be very useful for DIGIC 4+, as this part of Canon code is shared with (or backported from?) DIGIC 6/7/8.

names_are_hard

Merged them, with some tweaks.  Doesn't seem to be the cause of the prior bug.  Still useful later I imagine.  It does seem to have exposed another bug, but it's so strange I'm not sure.  Maybe something generally related to memory pressure, or bad memory layout?

In vram.c, vram_update_luts(), which gets called indirectly from call_init_funcs(), there's a loop that initialises a cache of screen coordinates:

Quote
    int y_times_BMPPITCH_cache[BMP_H_PLUS - BMP_H_MINUS];
[...]
    for (int y = BMP_H_PLUS - 1; y < BMP_H_PLUS; y++)
    {
        //~ bm2hd_r_cache[y - BMP_H_MINUS] = BM2HD_Ru(y);
        y_times_BMPPITCH_cache[y - BMP_H_MINUS] = y * BMPPITCH;
    }

This crashes.  If I restrict the range of the writes, it doesn't (not sure what the minimum restriction is).  It doesn't look like it can write outside the array to me, which would be the obvious cause.  It's an immediate crash, no crash log and the cam is unresponsive, have to pull battery.

Am I missing something?  Memory alignment (I'd expect y_times_BMPPITCH_cache to be int aligned and for that to be okay)?  Could sizeof(int) be 2 due to Thumb and this does something weird (y * BMPPITCH would overflow)?  I tried increasing ml_reserved_mem by far more than the size of y_times_BMPPITCH_cache and that didn't make a difference.

hkisgg

Quote from: names_are_hard on December 14, 2019, 02:51:11 AM
Hello and welcome!Here's a good start on stub hunting:
https://www.magiclantern.fm/forum/index.php?topic=12177.0

Ok sir, I will start with this thread and I was busy with shooting for a short film lately sorry for the late reply.

Quote from: names_are_hard on December 14, 2019, 02:51:11 AM
If you don't have IDA, you can use Ghidra.  Calle2010 was working on a 77D and has a repo with instructions for getting started with Ghidra, as well as a repo for his 77D porting work (which has some stubs found):
I just started using Qemu. I'm learning to use it. I guess I can jump to Ghidra to start learning and working on it. Is it ok if I PM you if I'm stuck anywhere in this process?   
Thank you for your kind reply.
:) HK

dedmen

Hi! I'm the new guy.
I wanted to use my 200D as a webcam, but the resolution it sends in live view feels too low so I thought I'll just hack the firmware.
I dumped the rom, decompiled most of it in IDA, and made myself a autoexec build of magic lantern from stephen's bitbucket repo.

Now theres the question, how I can set the boot flag to let it execute my autoexec, the 200d firmware dumper sadly is compiled without the bootflag set enabled, and the FIR compile tools are not available.
I already found the FIR crypto code, but I don't really want to go to the effort to reverse it.

Is there a easy-ish way to simply enable the bootflag?
I assume simply running ML build for another camera won't work.

I found the FROMUTILITY MENU which apparently can set bootflag, but I'm not sure how to activate that.
I assume its USB serial stuff, but that seems like too much effort, I don't really want to figure out their whole support, debug access stuff.
I know which flags I need to set on which address to enable the command mode... mh maybe I'll figure out their whole debug stuff afterall..
Yeah its UART, I don't want to rip off the rubber to get to it, finding some software vuln to get in seems easier.

(in case its not obvious, I'm actually constantly pausing while writing this post, while i continue reversing stuff, in case it sounds a bit chaotic.)

Okey already found a stack buffer overflow in PTP, I can probably just use that to get code execution and set the bootflag myself.
Or I can just run the "run autoexec.bin" function directly from there, ignoring the bootflag check...

I'll probably invest more time into helping getting ML running on the 200D, but I have two weeks of vacation coming up, and am currently quite busy so.. no promises.
I will most likely not even finish this stuff before I leave.

names_are_hard

@hkisgg - feel free to PM me with questions.

@dedmen - you can enable the bootflag with the special firmware update Alex posted earlier in this thread:
https://www.magiclantern.fm/forum/index.php?topic=19737.msg212603#msg212603

It's a flash update, so it's persistent, but it just enables the flag.  You'll also need to set your card bootable.  Be aware that if you put in a bootable card without autoexec.bin it hangs the camera.  Non-bootables are fine.

I'm currently working on some major changes to my repo, I think I was working with an older version of Unified branch (I probably messed up some hg command).  Working through a 30k line patch file seeing what's sensible to merge in :(  So, don't trust my repo to be from a sensible base.  Welcome aboard and good luck!

Akaei

I recently bought a T7i/800D.  It's my first decent camera since my AE1 back in the 80s.  I didn't realize until way too late about the file size restriction which limits maximum video duration/quality.   I just wanted to chime in with my appreciate that someone is working on Magic Lantern solutions for the T7i.  I'm looking forward to it.

Walter Schulz

File size restriction on 800D? AFAIK the only restriction is 29:59 recording duration. See manual page 277.

kitor

Too many Canon cameras.
If you have a dead R, RP, 250D mainboard (e.g. after camera repair) and want to donate for experiments, I'll cover shipping costs.

Walter Schulz

Which will generate chunk files < 4 GB but doesn't affect recording duration.

dedmen

Quoteyou can enable the bootflag with the special firmware update
wow, now I feel great for spending like.. 12 hours overnight on working on my exploit. :D

QuoteI think I was working with an older version of Unified branch
Yeah, I would also appreciate updating qemu with the one from main qemu branch, gave me quite a headache to get running.

Is there a more direct communication channel? Like Discord/Slack or smth? Not sure about what specific things I could be working on.
Yeah need to verify stubs, but not sure which from the stubs file are missing and required, and which need verification.
And as this is the first version I'm working on, I don't have references to older version patterns to compare.

names_are_hard

Quote from: dedmen on January 04, 2020, 04:52:11 AM
wow, now I feel great for spending like.. 12 hours overnight on working on my exploit. :D
Pff, just read every page on every forum post before you start ...  ::)

Quote
Yeah, I would also appreciate updating qemu with the one from main qemu branch, gave me quite a headache to get running.
Done 27k lines of merge, 3k left.  Should finish this weekend.  Then I "just" need to fix the things I inevitably broke.

Quote
Is there a more direct communication channel? Like Discord/Slack or smth?
The main devs (not me) use IRC.  They're not always online but will get back to any questions you have.  I tried using it for a while but I don't have a 24/7 computer to keep IRC active, and IRC servers have no chat history, so it was very frustrating.
If you start a Discord for 200D discussions I'd join it.

Quote
Not sure about what specific things I could be working on.
My repo works well enough on 200D to allow you to run arbitrary code (you can print to screen and log stuff to files so it's pretty useful), but it crashes part way through ML initialisation for reasons I don't yet understand.  This commit is probably good: https://bitbucket.org/stephen-e/ml_200d/commits/74004ec2c30c8b7253dfb50c598c81333aca8b63
Take a look at src/boot-hack.c, my_big_init_task().  The info_led_blink() lines should let you find when it currently crashes.  I tend to put "SJE" near code where I've made significant (probably hacky) changes.  You should get detailed logs on the cam, sometimes you even get a crash log, depends how broken that exact commit is (didn't check it was a good one, just guessing).
Trying to diagnose the crash would be most helpful to me.  I've asked around and nobody here seems to know what it might be.  So maybe it's hard, or maybe nobody noticed my questions.  The crash log gives line of code (in Canon's source, that we don't have), it's pretty easy from strings to work out where this is in the ROM.  I was working on doing some jump hooks around that location so I can see why it's failing an assert (multiple possible causes).  My jump hooks fail for unknown reasons (alignment?  Bad thumb mode?  page permissions?  There's no log).  That's on hold until I bring my repo up to a more modern version (because...  **** Mercurial's stupid branch model).

Quote
Yeah need to verify stubs, but not sure which from the stubs file are missing and required, and which need verification.
And as this is the first version I'm working on, I don't have references to older version patterns to compare.
The stubs in my repo are at least mostly good.  ML boots and gets a fair way through initialisation.  If you look in my platform/200D/stubs.S you'll see I've divided the stubs into sections based on my confidence that I've found the right function.  Some I am not confident on at all (but does the code currently go far enough that it matters?  Unknown).  So you could work on those if you wanted.  Probably a good way to get familiar with the ROM code.
If you DM me I can send you my Ghidra project.

Kestis

I'd like to shoot better quality video with my 800D. Maybe I could help somehow to make it happen sooner?
I have some experience writing C code on hobby basis for Arduino and AVR microcontrollers.

By the way. How much progress has been made with Digic 7 development?

srsa

Assuming these questions are still unanswered, I'll try to answer some of them.
Quote from: names_are_hard on December 15, 2019, 09:08:08 AM
In vram.c, vram_update_luts(), which gets called indirectly from call_init_funcs(), there's a loop that initialises a cache of screen coordinates:
A disassembled version of the compiled code could give more hints. The reason of crash is not necessarily related to that piece of code, it could be that RAM gets corrupted somewhere (stack, heap, ML code/data).
QuoteMemory alignment
Should not be a problem from C (unless you try hard) and this core seems to cope with unaligned words anyway.
QuoteCould sizeof(int) be 2 due to Thumb
No. It's 32 bits, regardless of ARM/Thumb.

QuoteI was working on doing some jump hooks around that location
Is that platform/200D.101/jump_hooks.c?


names_are_hard

@Kestis - you could probably help!  On 200D I can run arbitrary code, but it tends to crash.  Probably true for the other Digic 7.  Need to debug why.  You can print to the cam screen and log to the card to find out.  If you want to help out, get digic6-dumper branch working on your cam, then see if you can find enough stubs so my repo works on 800D.  That's not enough description so let me know if you want any help :)

@srsa - good idea about disassembling, stupidly I had done this in other areas but not for the code I was seeing the problem in!  Good to confirm mem align and thumb int size.  Yes, jump_hooks.c.  That will probably be changing soon, have done a lot of work updating the base branch I'm working from.

Progress report: merged a bunch of code from a branch Danne is working on.  I didn't realise, but the main Unified is about 2 or 3 years old, so I was missing a lot of stuff.  During the merge I realised I need to find a stub for task_trampoline (wasn't needed in the old Unified I was working from).  That has a reasonable chance to be related to where I'm seeing the crash.  Merge is complete but build is broken as now it depends on that stub.

nikfreak

That progress report sounds good.

Quote from: names_are_hard on January 17, 2020, 03:10:32 AM
That has a reasonable chance to be related to where I'm seeing the crash.  Merge is complete but build is broken as now it depends on that stub.

+1. Don't give up.
[size=8pt]70D.112 & 100D.101[/size]

a1ex

Quote from: names_are_hard on January 17, 2020, 03:10:32 AM
Progress report: merged a bunch of code from a branch Danne is working on.  I didn't realise, but the main Unified is about 2 or 3 years old, so I was missing a lot of stuff.

For what to merge, please re-read reply #230. It's probably best to start directly from digic6-dumper; at least, the code restructuring to allow booting on different digic generations, and also the new stub formats, are changes I consider useful for all new ports. That branch is also up to date with most other backend changes, that you also want.

The crop_rec_4k branch contains some highly experimental changes that break the compatibility with many cameras. It's also behind in other backend areas. So, while it's probably the most tempting one for users, and actively worked on by non-core developers, it's probably not the best choice for a new port, if you ask me.

Danne's branches are even more experimental, and they contain mostly unreviewed code. I haven't played with it yet, but I highly doubt I'll include these changes verbatim into the main codebase. See also this.

Unless you are betting on me completely retiring from the project - in that case, feel free to ignore my advice. That's probably a sensible thing to do, as I couldn't (yet) find a job that pays the bills, while also leaving me with some time for hobbies. I'm still wondering what you might be doing for close to $20k a month - I could work for 2 months per year, with a job like that ;)

names_are_hard

Thanks!

Nik, the build is only a tiny bit broken, the merge was *much* more painful.

Alex, yes, I recall #230 :)  I have some of that done.  But I only just realised the names are branch names in Hudson repo!  qemu, new-dryos-task-hooks, lua_fix and patchmgr are all branches that should be merged into unified?  I have taken most of the code around tasks, patch stuff and qemu I think, but not cleanly from those branches (didn't understand hg or ML codebase well enough, could probably do it now).  And I improved the tasks stuff, it was overcomplicated I think, the task struct on Digic 7 just needed two fields added then everything lined up nicely.

I had no way of *testing* these changes as the only cam I had was 200D, so I didn't make PRs.  Now I have an M2, which runs ML fine (but not Unified I think, that doesn't support M2).  If I can get those branches merged and it runs on M2, is that an okay level of testing for PRs to be worthwhile?  I assumed that the goal of Unified is one branch that runs on all supported cameras, with as many working features possible per cam, conditionally included?  Is that correct?  That would make the most sense to me and was what I was working towards (with a focus on 200D for a challenge).  I dislike how splintered cam support is - it makes finding the "right" build a challenge for new people and generates much extra effort in support questions.  Why not one branch that builds for any camera that we know works?

They don't pay me 20k a month, but if I was going to quit my job to be an ML dev full time I would want a pay rise for the extra risk :)

Danne

@names_are_hard
Working for eosm2? To what extent? freezefree recordings around raw? Any build, code you could share? Got my eosm2 collecting dust over here  :-*
@a1ex, please do not retire :(. Are we out of options really?


names_are_hard

M2: it runs, has menus you can navigate and you can take pictures.  Beyond that, I haven't tested.  When I say working I mostly mean "not crashing".  This at least lets me test if I totally broke builds for another cam when trying to get 200D working.  Or when trying to merge branches, now I know I can be useful doing that, maybe!

Danne


nikfreak

Just checked #230 and from my experiences I can tell you that each and every instruction @a1ex provided me was as precise as possible. That said, I sometimes needed to read it twice or thrice until I was able to follow it.

Quote from: names_are_hard on January 17, 2020, 03:38:03 PM
Thanks!

Nik, the build is only a tiny bit broken, the merge was *much* more painful.

Alex, yes, I recall #230 :)  I have some of that done.  But I only just realised the names are branch names in Hudson repo!  qemu, new-dryos-task-hooks, lua_fix and patchmgr are all branches that should be merged into unified? 


Yes, I know that pain, drove me crazy handling "unsupported" branches.  As you realised there are lots of branches that should be merged into unified. Becoming familar with them requires you to compile and try out on your cam. You'l need some time but I can just confirm what @a1ex said in #230. You already did some major progress IMO. Please continue.
[size=8pt]70D.112 & 100D.101[/size]