Magic Lantern GDBstub for remote debugging

[g3gg0]

I promised to have a look at how to add a GDB stub into magic lantern.
looked quite simple to do, so i implemented it.


as it is a quite big thing, please understand that i cannot publish a full guide to this tool.
also it is (very) far from perfect.
see it as a tool that helps you to breakpoint into a specific function, get the memory/stack/register contents and
*eventually* continuing the execuion.
latter wont work reliable on the camera, as it is some complex realtime system.
we cannot simply halt some tasks without side effects like ERR70 or even total lockup.

but i was able to test some basic breakpointing, changing registers and continuing in some test task.
in canon tasks it did sometimes lock up everything.

the way how i implemented it does not perfectly match how gdb frontends expect it.
that would mean we would not simply set a BP wherever we want (no interrupts!) and wait for some task(s) being stalled.
i intentionally decided not to do it like they expect as i had some special goals using that code.
i want it to be a swiss tool to e.g. hook code anywhere in ROM for testing purpose or capturing registers etc.

nevertheless it may help with inspecting "what are the parameters to this function?"
you can also add "watchpoints" that are breakpoints that trigger once and save the registers.
this feature is not yet available via GDB interface, but you must call the functions manually using
   gdb_add_bkpt(uint32_t address, GDB_BKPT_FLAG_WATCHPOINT | GDB_BKPT_FLAG_ARMED)
and then read the captured registers from gdb_breakpoints[pos].ctx
but you can very simple extend the gdb.c to allow setting such watchpoints from your favorite GDB frontend.


be warned - you will have to repower your camera very often

here it is:
magic lantern code: http://upload.g3gg0.de/pub_files/8e155ddcc88ee690cd07b6c2da365807/gdbstub.zip
ptpcam: use the one inside the contrib folder of the repository (as of 2012-10-16)
(Merged) updated tasks.h: http://upload.g3gg0.de/pub_files/4015304003c3c336e66f651e1418439e/tasks.h
(Merged) ptpcam patch: http://upload.g3gg0.de/pub_files/92879a741f5b8863da832ca8fe9327db/gdb.patch

how it works:
 * it adds two new PTP messages PTP_CHDK_GDBStub_Download and PTP_CHDK_GDBStub_Upload (defined for 600D)
 * "void gdb_setup()" is starting the processing task that handles GDB serial commands - call it from e.g. run_test
 * ptpcam has 3 new commands "gdb s" to send gdb serial commands, "gdb r" to receive the response and "gdbproxy" to forward gdb commands between a network socket and the camera (for remote debugging using e.g. IDA)
 * breakpoints are set using cache hacking - i place a undefined instruction that raises UNDEFINED interrupt
 * in UNDEFINED interrupt, i stall the running process using continuous msleep(1) and *store* the process context
 * when that process is resumed, i use another UNDEF instruction and the handler to *restore* the process context where it was stalled

important notes:
 * it is not possible to continue a process as long the breakpoint is still active. deactivate the BP, add another one behind the current PC, continue and then set the first one again
 * there is no "single step" functionality - the camera will do nothing
 * some tools (like IDA) update newly set breakpoints when e.g. "single step"ing and they fetch the current registers - so consdier "single step" as some kind of "refresh" or "sync"
 * do not (!) set breakpoints in interrupts. that wont work.
 * if you just continue execution in e.g. IDA without setting any breakpoint, IDA will wait. and wait. and wait. ... until you kill the network connection by exiting ptpcam or killing IDA (this break could also be done with a menu in ML to break the wait in in gdb.c:1159)
 * this might also happen if the "continue" command failed for some reason (i warned you that it wont work reliable )

how to debug:
 * call "gdb_setup()"
 * start "ptpcam --chdk"
 * enter "gdbproxy"
 * connect to localhost:23946 using your favorite debugger
 * you will see all the registers 0x0000..
 * set a breakpoint in the function you want to debug
 * "sync" using single step to activate the breakpoint
 * "sync" again to get current process registers - they will change if breakpoint is reached
 * you must now disable the triggered breakpoint and set a new one where you want to stop again
 * "continue" execution until BP is reached
 * some frontends like IDA allow "step over" commands that automatically set a breakpoint after the current instruction - this is supported of course

example code that displays all breakpoints and registers on-screen:

Code: [Select]


    while(1)
    {
        uint32_t line = 0;
        uint32_t bp = 0;
       
        bmp_printf(FONT_MED, 0, line++ * 20, "exc %08X, l 0x%08X", gdb_exceptions_handled, loops++);
       
        for(bp = 0; bp < GDB_BKPT_COUNT; bp++)
        {
            if(gdb_breakpoints[bp].flags & GDB_BKPT_FLAG_ENABLED)
            {
                uint32_t reg = 0;
           
                bmp_printf(FONT_MED, 0, line++ * 20, "BP#%d 0x%08X flags 0x%1X hits %d", bp, gdb_breakpoints[bp].address, gdb_breakpoints[bp].flags, gdb_breakpoints[bp].hitcount);
                for(reg = 0; reg < 15; reg+=2)
                {
                    bmp_printf(FONT_MED, 0, line++ * 20, "R%02d %08X R%02d 0x%08X", reg, gdb_breakpoints[bp].ctx[reg], reg+1, gdb_breakpoints[bp].ctx[reg+1]);
                }
                bmp_printf(FONT_MED, 0, line++ * 20, "CPSR %08X", gdb_breakpoints[bp].ctx[16]);
            }
        }       
        msleep(100);
    }

feel free to improve that tool. maybe it will get good enough to get into mainline tree somewhen
 

[ilguercio]

What are the uses for this?
I don't mind restarting my camera, i have a wall adapter anyway, so if you need testers in general...

[nanomad]

You can use it debug most canon functions (to discover paramters, register (variables) values and such)

[jplxpto]

THANK YOU! THANK YOU! THANK YOU! THANK YOU!

I'll test and post my comments...

[nanomad]

Hi, I'm trying to make GDB work on the 1100D... here's what I did so far:

- Replaced ptp-chdk.(h|c) with your version
- Ignored tasks.h as it was already merged
- Ignored cache_hacks.h as the version in the repo seems newer
- Ignored gdb.(h|c) for the same reason

Now all I need to find is the current_task_ctx and next_task_ctx in the firmwared. Do you have any suggestions?

[jplxpto]

I also like to know the answer to this question

Thank  you

BR

[nanomad]

I've imported ptpcam into ML repository, patched with g3gg0 patch. It works fine here (GDB not tested)

[jplxpto]

I've imported ptpcam into ML repository, patched with g3gg0 patch. It works fine here (GDB not tested)

Thank you!!!

[g3gg0]

ah okay but one change request.
the gdb.c features two main use cases

a) gdbstub for debugging via gdb protcol over PTP and
b) useful functions like gdb_add_watchpoint() to inject a hook function at any flash address, that
gets called everytime when ARM processor executes specified instruction

feature a) is good to be selectable, but needs OS functions like task_create and PTP integration
feature b) is also good to be selectable and is target-independent (no OS calls, no stubs needed) and very helpful

so can we make it selectable in 2 different ways?

CONFIG_GDB (generic debugging routines)
  will add gdb.c only and make watchpoint funcs accessible

CONFIG_GDBSTUB (GNU debugger compatible stub via PTP)
  will enable task_create in gdb_setup and the functions starting from gdb_send_append (undo changeset be31d5eb6daf, gdb.c)
  will automatically enable CONFIG_GDB

CONFIG_GDB_PTP is replaced by CONFIG_GDBSTUB then.

what do you think?

[nanomad]

Got it, should be in by tomorrow

[jplxpto]

sounds good!

[g3gg0]

thanks

here some example code with hints when using watchpoints.

Code: [Select]

/* when our hook gets executed, it captures the call stack into this string */
char stack_trace[1024];
/* here we see the maximum malloc size */
int max_size = 0;

/* this is our hook. we want it to get called whenever 'alloc_dma_memory' is called */
void alloc_dma_hook(breakpoint_t *bkpt)
{
    /* here we check R0 if it is larger than the current maximum. ctx[2] would be R2. thats possible up to R15 */
    if(bkpt->ctx[0] > max_size)
    {
        /* when yes, capture the call stack. this *must* be done in the hook. that cannot be done later as stack gets overwritten when we return */
        max_size = bkpt->ctx[0];
        strcpy(stack_trace, gdb_get_callstack(bkpt));
    }
    /* we could now even patch the registers if we want. but don't patch R15 please. else this simple watchpoint feature wouldn't trigger everytime, but only every second time */
}
 
void run_test()
{
    uint32_t loops = 0;

    /* install ARM exception hook for breakpoint (we use an invalid instruction) */
    if(!gdb_setup())
    {
        return;
    }

    /* so now add a watchpoint at alloc_dma_memory() */
    gdb_add_watchpoint(&alloc_dma_memory, 0, &alloc_dma_hook);

    /*
        note:
        a watchpoint sets TWO breakpoints. one at the instruction you say and one behind.
        when the first breakpoint is reached, its deactivated and the second gets activated.
        then the hook is executed (if != NULL). after that execution is continued and we
        instantly reach the next breakpoint as the second breakpoint is placed there automatically.
        then the second breakpoint gets disabled and the first one enabled again.
        (short, the second is just to re-enable the first breakpoint)

        reason: we can *not* execute an instruction where a breakpoint is set
        and if we remove the breakpoint, well.. then the breakpoint is removed :)
        its comparable to an air lock
    */
   
    while(1)
    {
        uint32_t line = 0;
        uint32_t bp = 0;

        /* counting numbers are always fun */
        bmp_printf(FONT_MED, 0, line++ * 20, "exc %08X, l 0x%08X", gdb_exceptions_handled, loops++);
       
        for(bp = 0; bp < GDB_BKPT_COUNT; bp++)
        {
            /* just print the watchpoint, not the second breakpoint needed for re-enabling (we call it 'linked' watchpoint) */
            if((gdb_breakpoints[bp].flags & GDB_BKPT_FLAG_WATCHPOINT) && !(gdb_breakpoints[bp].flags & GDB_BKPT_FLAG_WATCHPOINT_LINK))
            {
                uint32_t reg = 0;
           
                bmp_printf(FONT_MED, 0, line++ * 20, "BP#%d 0x%08X flags 0x%2X hits %d", bp, gdb_breakpoints[bp].address, gdb_breakpoints[bp].flags, gdb_breakpoints[bp].hitcount);
                for(reg = 0; reg < 15; reg+=2)
                {
                    bmp_printf(FONT_MED, 0, line++ * 20, "R%02d %08X R%02d 0x%08X", reg, gdb_breakpoints[bp].ctx[reg], reg+1, gdb_breakpoints[bp].ctx[reg+1]);
                }
                bmp_printf(FONT_MED, 0, line++ * 20, "CPSR %08X", gdb_breakpoints[bp].ctx[16]);
                bmp_printf(FONT_MED, 0, line++ * 20, stack_trace);
            }
        }
       
        msleep(100);
    }
}


[jplxpto]

thank you

i think you can put this code at debug.c

[g3gg0]

some special case are VxWorks cameras.
when scanning the binary for instruction cache flushing functions, i found a lot.
here is some code to disable that for 40D v1.1.1.

when we dont patch this, its likely that our flash patches get deleted.
(to be verified)

#if defined(CONFIG_40D)
  /* this is a evil hack to disable instruction cache clearing */
  cache_fake(0xFF810220, 0xE1A00000, TYPE_ICACHE);
  cache_fake(0xFFD654E0, 0xE1A00000, TYPE_ICACHE);
  cache_fake(0xFFD654EC, 0xE1A00000, TYPE_ICACHE);
  cache_fake(0xFFD65560, 0xE1A00000, TYPE_ICACHE);
  cache_fake(0xFFFFDEF40, 0xE1A00000, TYPE_ICACHE);
  cache_fake(0xFFFFDF04, 0xE1A00000, TYPE_ICACHE);
#endif

[jplxpto]

Hi, I'm trying to make GDB work on the 1100D... here's what I did so far:

- Replaced ptp-chdk.(h|c) with your version
- Ignored tasks.h as it was already merged
- Ignored cache_hacks.h as the version in the repo seems newer
- Ignored gdb.(h|c) for the same reason

Now all I need to find is the current_task_ctx and next_task_ctx in the firmwared. Do you have any suggestions?

Hello nanomad

I checked that there was something wrong.
The g3gg0 had added support for gdb that is not in the repository. * Ptp_chdk_gdb_ missing functions. They are on file gdb.patch.