PicoC -> TCC

[a1ex]

This thingie (TCC) can compile C programs to ARM code directly on the camera, so you will be able to run your C scripts at native speed. Image processing anyone?

First test: https://bitbucket.org/hudson/magic-lantern/commits/f8aab146bf6c

I've also ran this image processing script. It was almost instant, whereas picoc took around 1 minute.

// 5D Mark II only
// for others, look in platform/YOUR.CAMERA/consts.h for YUV422_LV_BUFFER_DISPLAY_ADDR
short * buf = *(short**)0x2900;

for (int i = 0; i < 480; i++)
{
    for (int j = 0; j < 720; j++)
    {
        // for each pixel, keep chroma, set luma to 0x80
        buf[i*720+j] = (buf[i*720+j] & 0x00FF) | 0x8000;
    }
}

RAM usage: the compiler takes 150K as ARM and 115K as Thumb. It's probably needed only at the compilation stage, so I think it can be loaded as a plugin, and unloaded as soon as compilation is done. If that's true, it will be a lot more memory-friendly than PicoC.

On desktop, tcc seems to be roughly as fast as gcc -O0. Didn't ran any benchmarks yet.

Stay tuned

[SDX]

Holy ..

A1ex, you are truly awesome. I can't wait to get the time to play with that..
I still have a bunch of ideas in my head which I couldn't realize with PicoC due to performance issues.

[Marsu42]

RAM usage: the compiler takes 150K as ARM and 115K as Thumb.

But that still means that 60D users are out of luck or have to disable a lot of features? PicoC uses 57k on my 60d... but sounds great anyway, shame on Canon for being short on memory

[sztupy2k]

We need a plugin system

[g3gg0]

we have one, but i am not convinced it is good enough.
i am thinking about a new plugin system that makes (nearly) all functions accessible from
within the plugin without having to declare/change all functions.

this concept can be re-used for scripts too.

[sztupy2k]

we have one, but i am not convinced it is good enough.
i am thinking about a new plugin system that makes (nearly) all functions accessible from
within the plugin without having to declare/change all functions.

this concept can be re-used for scripts too.

My idea was basically the same: make all functions accessible to the plugins. I started a pilot a year ago how to make the menu system accessible to plugins (so they could simply add themselves there), but didn't really had time to finish it (and trunk is now much more different than it was a year ago )

[g3gg0]

my current idea:

autoexec.bin loads a minimal loader that understands to read ELF (alex' TCC port seems to support exactly what we need)
this loader loads MAGIC.ELF contains the real magic lantern with all symbols etc.
symbol table (function name + address) is kept in memory.

we later can load another ELF and TCC will resolve the symbols.
same for scripts - they can use the same table for addresses. (still the prototypes must be put in header files)

additional feature: ML can load into any RAM, just like the PIC approach, just less complex (using standards like ELF/.o)

[g3gg0]

yep, its working basically.

i compiled some code using GCC into an ELF (.o file).
then i loaded it in camera using TCC, added some symbols and executed it.

[g3gg0]

it works, i can load ELF files into magic lantern.
to test the current implementation, copy magiclantern.sym (new file) into your card root as "magic.sym".
it must match to the current autoexec.bin and copied again when you compile again.
(we can later build a SHA1 hash over the file and check it for consistency)

then compile this code using gcc:

Code: [Select]

int main()
{
    msleep(500);
    printf("Hiding Console\n");
   
    for(int i = 0; i < 5; i++)
    {
        printf(".\n");
        msleep(500);
    }
    console_hide();
    msleep(1000);
    console_show();
    printf("Test finished\n");
    return 42;
}

i used this command line:
~/gcc-arm-2012.q1/bin/arm-none-eabi-gcc -c test.c -o test.elf -nostdlib -std=c99
(yeah, we should declare the functions we use the compiler will tell us several times)

then copy the test.elf into CF card root and execute run_test

[Michael Zöller]

That's wonderful Great work!

[g3gg0]

problem of this approach is this:

(my understanding)
ELF file format supports several object file types. these are the relevant:
ET_REL - relocatable
ET_EXEC - executable
ET_DYN - shared object

When compiling C files into .o files, no address allocation is done.
this .o will be in ET_REL format, that just contains all objects in a binary representation. (compiled code, variables, arrays in binary etc)
all these objects are something like sections. one section for every single object (function, variable etc)

the linker loads several files in this format and merges them into a file of format ET_EXEC where (in our case)
the address of every single object is allocated and the objects are merged into sections they were intended for.
e.g. all functions which were are merged into a single section named .text and variables into .data.

if compiled + linked as shared object, we have ET_DYN file that is again different.
it is intended for DLLs/.so files and require different handling.


TCC's linker code only supports ET_REL as it is a "linker" or ET_DYN but seems to rely on OS functions to load a DLL.
not sure if it can load such DYN objects at all on our cameras. i tried it and it simply didnt work.
(i should try that in an emulator and check where it is failing)
so i am left with ET_REL as only working solution.

this has some disadvantage.
it is meant as intermediate file in a build process that will get linked to other objects into a final executable or a shared library.
so you can not simply compile several .c files into one .o that can be loaded in ML as single plugin (at least, i dont know how )
well, the linux kernel uses exactly the same method for its drivers (.ko), so it might be still a good choice.

but i hoped to load .so files (ET_DYN) which is the cleanest solution imho.


anyway, i will stick to ET_REL .o files now, hoping that it wont cause too much trouble.


EDIT:
well, using LD with param -r we can output relocateable objects as the linux kernel does.
this sounds like a good solution.
at the moment i am investigating how to add plugins during runtime.
now i can add them all at once, then relocate all those .o's and then run functions.

[jplxpto]

Guys I'm surprised by these developments. It will be a great improvement if the ML supports dynamic loading of ELF files.

I'm anxious to try it.

[jplxpto]

Perhaps in the near future, we can use the GDB stubs to upload small programs and load them dynamically.
Maybe in the future we can even upload and launch the ML.

What do you think about this this my idea?

[a1ex]

Made some progress in porting TCC as scripting engine.

What works:
- scripting GUI from PicoC (as a module)
- compiler is loaded on demand (from tcc.mo), compiles the script, then gets unloaded. So, the memory usage gets high only for a split-second (which is very good IMO). The only thing that stays loaded all the time is the scripting GUI (which is a module, you can unload that too if you don't use it).
- hello world is working with minor source code tweaks

What doesn't work:
- scripting library is not there (so existing scripts won't run)
- stopping scripts doesn't work (this maybe can be hacked somehow).
- floating point doesn't work. Here we have a little problem: TCC only emits code for a hardware floating point unit (which we don't have).

Relevant message from TCC mailing list, regarding floating point support: http://lists.nongnu.org/archive/html/tinycc-devel/2006-10/msg00055.html

Example code:

Code: [Select]

float add(float a, float b)
{
    return a + b;
}

GCC output:

Code: [Select]

.text:00000000 add
.text:00000000                 STMFD   SP!, {R3,LR}
.text:00000004                 MOV     R3, R0
.text:00000008                 MOV     R0, R1
.text:0000000C                 MOV     R1, R3
.text:00000010                 BL      __aeabi_fadd
.text:00000014                 LDMFD   SP!, {R3,PC}

TCC output:

Code: [Select]

ROM:00BF3A60                 MOV     R12, SP         ; Rd = Op2
ROM:00BF3A64                 STMFD   SP!, {R0,R1}    ; Store Block to Memory
ROM:00BF3A68                 STMFD   SP!, {R11,R12,LR} ; Store Block to Memory
ROM:00BF3A6C                 MOV     R11, SP         ; Rd = Op2
ROM:00BF3A70                 NOP                     ; No Operation
ROM:00BF3A74                 LDC     p1, c0, [R11,#var_s10] ; Load Coprocessor Register
ROM:00BF3A78                 LDC     p1, c1, [R11,#var_sC] ; Load Coprocessor Register
ROM:00BF3A7C                 CDP     p1, 0, c0,c0,c1, 0 ; Coprocessor Data Processing
ROM:00BF3A80                 NOP                     ; No Operation
ROM:00BF3A84                 LDMIA   R11, {R11,SP,PC} ; Load Block from Memory

So, unless somebody changes the TCC backend to output soft-float code (quite hard), using TCC as a scripting engine is not going to be fun...

[g3gg0]

cool.
... or just add FPU emulation, which isnt very hard.

[g3gg0]

we should pull the latest changes from http://repo.or.cz/w/tinycc.git/tree
there are some float and register fixes for ARM targets which might be helpful.

already diffed our tcc module and the git version and there are a few minor conflicts.

[a1ex]

as a stills shooter, I know the video 'stuff' gets the attention.

I use ML primarily for stills, but yes, the video stuff gets a lot of discussion.

Nice to know there's still demand for the scripting engine (it just moved a little higher on my todo list).

[engardeknave]

Nice to know there's still demand for the scripting engine (it just moved a little higher on my todo list).

I want scripts soooo bad.

[a1ex]

Another point higher. I expect TCC to have no more problems with floating point math thanks to g3gg0, though I have yet to try...

[engardeknave]

If TCC works the way I think, then it will dramatically diminish the barriers to entry into the world of ML coding. This will surely lure in a good number developers.

[a1ex]

Well... you will place a plain old C program in the scripts directory, and TCC will compile and run it at (nearly) full speed. I'm thinking at a lightweight API, no module tags & prop handlers, but a quick way to code simple things. Maybe even assign scripts to certain events (like button press, mode change, idk yet, just throwing some ideas).

* nearly = it won't do the same optimizations as GCC.

I'm even thinking to have a really easy way for organizing them in menu. Either infer it from directory structure (e.g. place something in directory Scripts/Shoot and the script appears in the Shoot menu), or a tag in the header.

Also, the new menu caret option can be the default option for script parameters.

[Marsu42]

I expect TCC to have no more problems with floating point math thanks to g3gg0, though I have yet to try...

If you try, try to find a working compiler line for newer gcc versions - the -fpu=fpa is obsoleted, and I couldn't find a replacement to make it compile at all :-o

[gerk.raisen]

Another point higher. I expect TCC to have no more problems with floating point math thanks to g3gg0, though I have yet to try...

One point up also for me?
I'm thinking about it every night

[engardeknave]

I saw some mentions of TCC in the change logs. Does this mean there's module with script functionality somewhere?

[a1ex]

Right now, TCC is used in the core for loading the modules. It's stripped-down, so you can't do any scripting with the core version.