DIGIC 7 development (200D/SL2, 800D/T7i, 77D, 6D2)

[names_are_hard]

It can boot to Canon menus now, but often hangs afterwards.  One definite problem is initialising fonts; 200D (and other digic7 I think? Can't find this on the forum, maybe it was IRC) uses Opentype, not bitmap.  This makes our init error (it's not possible to give a valid address for BFNT_CHAR_CODES to bfnt_ok()).  Digic6-dumper uses some fixed bitmap font I believe, I'll port that next as the easiest option.

At some point I suppose we want an opentype renderer, or some one time step to convert those to bitmap, dump to ML dir, and ML init could load them to mem.

[names_are_hard]

Reliable booting to Canon menus.  Okay-for-now workaround for the font problem (conditionally skip the Canon font initialisation and only use the RBF fonts on the card).  Means you can't do early logging to the display, but I've changed the logging to disk code so it makes sequential numbered DBG00000.LOG files.

@kitor pointed me at some vram changes in DIGIC7 that were very useful - hello_world() now prints to display...  but the output is junk.  Need to change some consts to get the offset into display right (maybe just BMP_VRAM_START) and do colour correction since DIGIC7 uses YUV display.  Then see why the font output is garbled.

Anyone know if the RBF font routines require valid Canon bitmap fonts?  I'd guess not, looks like we use Canon fonts as fallback, but I don't really know.

I think this commit is mildly broken and needs an msleep(2000) ish before log_finish() in boot-hack.c (or the menus won't load before hello_world() starts and blocks; this means back screen won't display).  Too lazy to retest and push, local build works, I'll push something else soon enough:
https://bitbucket.org/stephen-e/ml_200d/commits/2fffa3bbda8980fdc3ce1c3fe1a8323ddaee45f4

[kitor]

Won't bmp_putpixel_fast from bmp.h be a problem here? It's used by font_draw_char from frb_font.c:

Code: [Select]

inline void bmp_putpixel_fast(uint8_t * const bvram, int x, int y, uint8_t color)
{
    /* vxvorks skipped */
    bvram[x + y * BMPPITCH] = color;
    /* 500d skipped ;) */
}
It's definitely not YUV-aware.

What about replacing it with code based on disp_set_pixel from minimal.c in digic6-dumper?

[names_are_hard]

Yes, bmp_putpixel_fast() is part of the problem.  disp_set_pixel() also wants fixing in unified, which isn't as simple as I expected, a bunch of symbols are in different places in unified vs digic6-dumper.

I haven't looked much at how bmp_putpixel_fast() is using its colour param.  It's an enum (eg, COLOR_GREEN1 in bmp.h), but seems to get written direct to vram, so I'm missing something.  Not immediately obvious how to map to YUV (maybe it's not needed, map the YUV to the color params first?).

[kitor]

I see some discussion about YUV stuff on chdk forum -> https://chdk.setepontos.com/index.php?topic=11316.120

As for "simply replacing" - from what I understand now, YUV buffers have also alpha channel, so replacing rgb values with yuv ones will produce werid results (mentioned in chdk thread too). And Canon seemed to change this twice - once to UYVY + separate opacity data that's in 200D (according to comments in minimal.c - and by "adapted from names_are_hard" i believe you are aware of that ) and then to UYVYAA (5D4, M50, R...)

For now I would just replace bmp_putpixel_fast with implementation based on CONFIG_DIGIC_678 part of disp_set_pixel in digic6-dumper/minimal/hello-world/minimal.c, completly ignoring bvram and converting 8 bit input into 32 by some bitwise operations (?, i was never good at optimizing low-level stuff) since now it's hardcoded to white. This surely will be order of magnitude slower, but if that takes us into ML menu, i guess that's good idea to fix/optimize it after we know that we can do GUI stuff.

As for

disp_set_pixel() also wants fixing in unified, which isn't as simple as I expected, a bunch of symbols are in different places in unified vs digic6-dumper.

I thought that this is used only from BL context, so not required at this stage:

Code: [Select]

grep -irw disp_set_pixel ./
./src/font_direct.c:    disp_set_pixel(x, y, c);
./src/disp_direct.c:void disp_set_pixel(uint32_t x, uint32_t y, uint32_t color)
./src/minimal.c:void disp_set_pixel(int x, int y, int c)
./src/disp_direct.h:void disp_set_pixel(uint32_t x, uint32_t y, uint32_t color);


[a1ex]

a bunch of symbols are in different places in unified vs digic6-dumper.

Yeah, that's because I did some changes in digic6-dumper in order to accommodate DIGIC 6/7/8 models, in particular, the refactoring to allow different boot methods, or the one to allow different kind of stubs (such as ARM or Thumb code). Indeed, that branch went a little beyond "just" a simple dumper stage.

The digic6-dumper branch also contains a couple of experimental backends, that are not yet in unified:
- qemu (pretty much ready for merging if you ask me)
- new-dryos-task-hooks (also pretty much ready, depends on qemu, but there may be surprises)
- lua_fix (needs testing on currently supported cameras; depends on qemu, new-dryos-task-hooks, 100D and some others; basically good, but many model-specific changes to review; currently broken on at least 5D2 and 500D)
- patchmgr (breaks 7D, may break some reworked features, needs cleanup, no dependencies)

So, I'd say the logical path to include DIGIC 6/7/8 into unified would be to merge the above branches (and their dependencies) first. There's also calle2010's task rework that needs to be included, as it should be applied to all DIGIC 4/5/6/7/8 models. Unfortunately, a straw broke the camel’s back, so... I'm very sorry these things are not done yet

In any case, I see your efforts as major progress - sometimes, reinventing the wheel can be very helpful to understand how things work.

The code is kind of ugly [...]

That's also my issue - I can easily bang out code that does the job, but turning it into something clean and maintainable, with comments and without breaking existing stuff, and with readable history... requires a lot of extra effort from my side. Maybe it's worth it in the long run, I don't know. I don't have any experience with maintaining large codebases, besides ML.

[names_are_hard]

kitor: I don't think bmp_putpixel_fast() is called with rgb values at all.  Look at the values when it's called, and the constants in bmp.h, eg, COLOR_WHITE is 1, COLOR_BLACK is 2.  Indexed colour maybe.  I need to understand this better before I can change it.  Probably won't have much time to work on it before the weekend.

I don't know what disp_set_pixel() is used for, but it's the easier thing to port, so I may as well do both.  I guess this is quite a long way from a menu   This is a CONFIG_HELLO_WORLD build, it bails out very early.  Still, having fonts will make debugging much faster.

Alex: sadly I have taken a horrible mix of your changes as I've noticed them or found them useful!  All the ugliness in my branch is my fault.  Your suggestion on merging the various branches makes sense to me, but I really struggled with hg.  I lost changes multiple times.  That's why I moved the whole thing to git, but that makes clean merges harder.  Again, my fault.  If it ever gets nice enough to merge back into real unified I'll make an hg repo.  And if I get up to say, menus, I'll be able to notice if I broke something trying to merge in other things to my code.  Currently it's all broken, can't tell if I did a bad merge

There's no need to apologise for not having magically created enough hours to do my work for me!  It's nobodies work really, we're all volunteers, mostly because we want cool toys for our own cameras I guess.  Besides, I wouldn't have started working on this if I didn't enjoy these low-level puzzles.  And yes, definitely learning more doing things the hard way, which is fine by me.

I don't have a reliable opinion on the shape of the code yet.  I can't tell which parts are actually ugly, and what is simply my confusion.  Probably mostly the latter!  Generally I'd say make it work first, then make it reliable, then, maybe, pretty.  I've reviewed millions of lines of C and seen much, much worse (in commercial products...).

[kitor]

kitor: I don't think bmp_putpixel_fast() is called with rgb values at all.  Look at the values when it's called, and the constants in bmp.h, eg, COLOR_WHITE is 1, COLOR_BLACK is 2.  Indexed colour maybe.  I need to understand this better before I can change it.  Probably won't have much time to work on it before the weekend.

You are right! I looked at uint_8t, saw color definitions and just assumed it without looking at the numbers. Lesson learned once again: don't assume things 
https://magiclantern.fandom.com/wiki/VRAM/BMP
https://chdk.fandom.com/wiki/Frame_buffers -> "Bitmap (or Overlay)"
https://chdk.fandom.com/wiki/Palette
Indeed that's an indexed palette, i guess that it was more consistent between EOS cameras than on PowerShot models, but this apply:

Code: [Select]

#define COLOR_TRANSPARENT_GRAY  0x14 // not portable, old cameras show it as magenta
Since CHDK is running on D6+ cameras, I checked their code (on dev trunk) and while their functions have a bit different definitions, core/gui_draw.c seems to have all the magic implemented that needs to be ported here.

[esinem]

As a non-techie user, I assume this geek talk means an 800D/T7i version is on the way? Does anyone have any idea when a download is likely to become available for your average idiot like me? :-)

[Walter Schulz]

Answered in
Top of page -> User Guide -> FAQ -> Troll Questions

[totalmichel]

Answered in
Top of page -> User Guide -> FAQ -> Troll Questions

👏👏👏

[names_are_hard]

@kitor - those CHDK links were very helpful, thank you!  It *used* to be indexed, but now it's not.  That's good, I can ignore the indexed code and simply replace it (ifdef for > Digic 5).

@esinem - I don't know that anyone is working on an 800D port.  It will still be several months, minimum, before I have 200D working.  Each camera needs individual work.  Porting between the various Digic7 cams should be easier, so if 200D gets usable, progress on the others may speed up.  Or I may get bored, or get stuck, and stop and it'll never get done.  Who can say?

[totalmichel]

@kitor - those CHDK links were very helpful, thank you!  It *used* to be indexed, but now it's not.  That's good, I can ignore the indexed code and simply replace it (ifdef for > Digic 5).

@esinem - I don't know that anyone is working on an 800D port.  It will still be several months, minimum, before I have 200D working.  Each camera needs individual work.  Porting between the various Digic7 cams should be easier, so if 200D gets usable, progress on the others may speed up.  Or I may get bored, or get stuck, and stop and it'll never get done.  Who can say?

Dont give up 🙂 (on a more serius tone, good job, im a programer, i know the struggle)
I'm waiting for you to complete the 200D to  *try* adapting it to the 6D2 😝
But im nowhere good enough to figure out the things you're doing.
Maybe it will work, or maybe ill brick my camera, who knows...

[names_are_hard]

Learning other people's code is hard

If you want a probably-low-risk task for porting to 6D2, this commit would be good:
https://bitbucket.org/stephen-e/ml_200d/commits/2fffa3bbda8980fdc3ce1c3fe1a8323ddaee45f4
You would need to find the right values for stubs.S and consts.h (Ghidra, or IDA Pro), if you get enough of them right it will let your cam boot while also loading a small amount of ML code (and let you blink the LED for debugging, search for info_led_blink).  I would recommend getting Alex's digic6-dumper branch working first, if you haven't done that already.

If you want a probably-considered-easy task which is completely safe and will help (force...) you to learn the code, you could get this commit building:
https://bitbucket.org/stephen-e/ml_200d/commits/6ce440ec21564070cdddc7783d50c254aca39093

I'm stuck on that.  You don't need to run any code, only fix the makefiles (or headers?  includes?) so platform/200D.101 will build.  That work is needed to make the bootloader able to display things.  I'm giving up on it for now, it's not very important, the bootloader already finishes.  I'm moving to bmp_putpixel_fast(), if I can get that working I can use output on the display for debugging.

[names_are_hard]

It's (more) alive! This is quite nice, it proves I don't need to get opentype working to get a usable interface.  Haven't tried to track down the green bars yet, I guess I left some weird debug code in.

[nikfreak]

congrats! Keep up your good work.

[kitor]

Now that's what i call progress!
Need to find some time and finally dig into your repository.

Inb4 "so when download be available?" -> answer from a few posts ago apply exactly the same now.

[names_are_hard]

I hope this should mean it's fairly easy to get CONFIG_HELLO_WORLD building on other Digic 7 cams!  Let me know your progress!

I fixed the green bars, it was a stupid ifdef mistake.  Next up, I have found that disabling CONFIG_HELLO_WORLD and enabling CONFIG_EARLY_PORT is broken, it doesn't compile.  Even doing the obvious fix (a required function definition was ifdef'd out) the cam doesn't boot as far.  Don't know what's going on there, haven't investigated.  I have CONFIG_PROP_REQUEST_CHANGE *disabled* so I decided to try without CONFIG_EARLY_PORT, and this is better - obviously doesn't really work but further than a CONFIG_HELLO_WORLD build.  It's currently failing in call_init_funcs().  I can see which function is crashing via debug logs.

VERY IMPORTANT: if you are trying this, ensure CONFIG_PROP_REQUEST_CHANGE is not defined.  This would be controlled in platform/200D.101/internals.h or equivalent.  Trying to write to properties is risky, can brick cam.

In unrelated news I decided to actually use my cam today and got this lucky shot: https://imgur.com/a/zltkJjv
They're skittish critters, I am learning how to sneak up on them.  Tamron 60mm F/2 macro.  Can't post in Share Photos, it's not using ML

[OlRivrRat]

                 @ N_A_H

        Nice Shot ~ 1 thing I've noticed about Dragon Flies is that they are very repetitive creatures,

so if @1st They get away just be patient & They soon, most likely, will return ~

[names_are_hard]

Yes, you're right, they often land back on the same leaf, facing the same direction.  Funny little guys

Re 200D porting, I have made some more progress.  I had to fake out one of the funcs that call_init_funcs() wants to setup, a zebra func.  That was giving me weird hangs, at a short but not consistent time afterwards.  With that disabled, it gets far enough to reach ml_started = 1, in my_big_init_task().  So, ML is officially ported?

ML is getting far enough that it prints a warning: "Camera was not shutdown cleanly.  Skipping module loading" - pretty good.  I'm not sure how to proceed now, what's the best way?  The Canon UI is unresponsive, often the camera locks up.  Sometimes it doesn't and half press will give you exposure info.  Sometimes the off switch does a clean shutdown, generally not.  I'm sure some of my consts and stubs will be wrong.

I have a decent log covering up to ml_started = 1:
https://drive.google.com/open?id=1QXRSHMbn1SFUs41wGDFhoZBVbcKEp3U9

Any ideas?  I have a lot of things I can try, I'm wondering if there is an obvious route for someone with experience.

[nikfreak]

... I'm sure some of my consts and stubs will be wrong.

...

Any ideas?  I have a lot of things I can try, I'm wondering if there is an obvious route for someone with experience.

Not really followed your post history but it sounds like you should focus on consts / stubs fixing first. Running the stubtest would be a good way but it shouldn't work atm. For me it was rather easy to do consts / stubs "matching" when porting Digic 5 by comparing ROM dumps with same generaton cameras. Drop me a PM if you don't already have some digic5 dumps to compare. I can't tell if it could help with digic 6+. Your situation is rather a difficult one as you got no fully working D6+ port available to do the stubs / consts matching.

[aprofiti]

What do you get if you try "unified" helloworld with digic6 branch?
Maybe there are some changes from a1ex useful to make it stable enough.

The Canon UI is unresponsive, often the camera locks up.  Sometimes it doesn't and half press will give you exposure info.  Sometimes the off switch does a clean shutdown, generally not.

I had experience of GUI lockup when updating 5DC port to more recent codebase; If I was quick enough I could as example enter the canon menu before ml override the task responsabile for buttons handling.

Don't remember exactly how I solved but probably was some missing initializzation in the custom ml_init for vxworks cameras. I had to doublecheck with the "unified" code to notice what was missing to load gui.

Also was missing initializzation of memory allocator...

ML is getting far enough that it prints a warning: "Camera was not shutdown cleanly. Skipping module loading" - pretty good.

Maybe here it is hanging on the first startup of Ml (after copy files to card, wiping previous settings) locking the camera and skipping module initializzation on the next round because of the undeleted module lock file in the card...

On vxworks it try to scan the directory to list all modules but doesn't iterate thought, so  camera appears as freezed...

Try to notice what is happening with fresh a install and see if the lock file is still present after shutdown.
If you want to disable modules, just comment module_task creation in ml_big_init

[names_are_hard]

it sounds like you should focus on consts / stubs fixing first

I'd like to fix them, do you know how I'd find which ones are broken?  Qemu for Digic7 doesn't get very far.  Is there some pattern to look for in debug logs from real cam that shows it's a stubs / consts problem?  I think 90%+ will be right for stubs, maybe 80% for consts.  I've got some other roms to work from and it's surprising how close the code is even for old cams.

What do you get if you try "unified" helloworld with digic6 branch?

If I'm understanding you right, I've sort of done this.  digic6-dumper runs fine, but it doesn't have much functionality.  I've ported the parts that looked relevant into unified branch, fixed some stuff and can now run a full build of ML far enough to get to the point described above.  CONFIG_HELLO_WORLD on unified branch is stable (but again, isn't running much code).

If I was quick enough I could as example enter the canon menu before ml override the task responsabile for buttons handling

This may be the problem!  I am pretty sure I don't have any button consts found, so could it simply be ML is hijacking all the button handlers and breaking them?  That would be good news.

because of the undeleted module lock file in the card

I can check this too, thanks.  Should be easy to disable the module loading for now, I can't imagine it works yet.

Thanks both for the help!

[names_are_hard]

Found a nice missing piece, I had incorrect DRYOS_ASSERT_HANDLER in consts.h, so I wasn't generating CRASH00.LOG.

[nikfreak]

Nice one. Now you just may also use several lines of bmp_printf (e.g test1, test2, test3, test4...) to write some text onto the screen and use this approch to locate code which causes trouble. Afterwards it's easy to see which stubs / consts are used around / within that code. You then either uncomment that code and see where code execution reaches (next bmp_printf) or better fix the stubs / consts first. I used this approach a lot and even still do in vb6 when i get stuck in one of my private projects

Next, enable features one by one and use same approach...