Compositors, layers, contexts in RGB and YUV - How Digic 7(6?)+ draw GUI

[kitor]

Preamble

Up to not-so-long-ago everybody* thought that Digic 6+ cameras render GUI in YUV. In 80D thread, @a1ex documented MARV structures that referenced two UYVY VRAM surfaces. More of that in Chapter 0.

In last December @coon noticed that EOS RP seems to have RGBA VRAMs. This got somehow unnoticed, and he was unable to mess with them anyway.

About a month ago, after I tried to make GUI rendering work on R in YUV and failed successfully. I thought "There's no way that Canon renders GUI in YUV". The next day I found the same RGBA layers that coon noticed, with whole infrastructure related to that. This is Chapter 2 and Chapter 3.

@names_are_hard started looking into his 200D ROM and found some similar things to R/RP. It was lacking a lot of stuff, but he was able to locate one RGBA layer. I bought 200D to investigate this, and... this goes into Chapter 1

Finally @names_are_hard made a nice compatibility layer for existing indexed RGB code and I integrated compositor support PoC for EOSR. For TL;DR see Chapter 5.1

*Future me: See Chapter 6

[kitor]

Chapter 0 - MARV backwards is VRAM

Some very basic details on how D7+ (D6+?) store VRAM information:

MARV structure
Code based on vram.h in digic6_dumper branch, updated to my research.

Code Select Expand


struct MARV
{
    uint32_t signature;         /* MARV - VRAM reversed */
    void * bitmap_data;         /* Pointer to buffer representing actual VRAM draw surface*/
    uint8_t * opacity_data;     /* Pointer to buffer representing opacity data in case of UYVY + separate alpha VRAM  */
#ifdef CONFIG_DIGIC_X
    uint32_t unk1;              /* unknown, so far always 0xFFFFFFFF in the wild */
    uint32_t ibus_lo;           /* Lower 32 bits of IBus address */
    uint32_t ibus_hi;           /* Upper 32 bits of IBus address */
#endif
    uint32_t flags;             /* Bitmask (?) representing type of color used in buffers above */
    uint32_t width;             /* X resolution of buffer */
    uint32_t height;            /* Y resolution of buffer */
    void * pmem;                /* pointer to PMEM (Permanent Memory) structure */
};

//IBus on Digic X is a new memory interface (with new Resource Manager). This is not yet understood.


MARV is a structure that describes VRAM parameters.

color
Seems to be a bitmask? There are multiple possible values referenced across R, RP, 200D, R6 roms, but only three seems to be used:
0x05040100  - 32 bit RGBA
0x01040280  - 32 bit UYVY + separate alpha buffer (opacity_data)
0x11060200  - 40 bit UYVYAA (alpha stored in main buffer)

Other possible values (but unknown meaning), pseudocode based on XimrSetInputLayerMARV in 200D. This may explain why we think it is a bitmask.
Don't mind ID column, this is just an internal ID used by some functions from Chapter 1.

Code Select Expand

         Color     ID    Comment
if       0x05040100   0x5   RGBA - this one is used by RGB layers on 200D, R, RP, R6
elif  <  0x05040101
{
    if   0x01040280   0x1   UYVY + separate alpha buffer (opacity_data)
    if   0x02010100   0x2
    if   0x03000000   0x3
    if   0x04020100   0x4
    else              0x0
{
else
{
    if   0x06040100   0x6
    if   0x07020100   0xA 
    else              0x0
}

Update
color / flags field contains at least 4 information, starting from MSB:

Code Select Expand

8 bits: Seems to be related to "ID" from table above. Possibly Ximr type ID for color space.
8 bits: Bits per pixel?
8 bits: Subsampling factor?
1 bit : Flag if structure utilize that "separate opacity layer"
7 bits: Unknown / not referenced in any code I analyzed.

"Bits per pixel" and "Subsampling factor" are used as follows in CreateMARV:

Code Select Expand

BMP_VRAM_SIZE = fast_division(width * bits_per_pixel, subsampling_factor);
BMP_VRAM_SIZE = BMP_VRAM_SIZE * height;

So for example:

Code Select Expand


0x11060200 UYVYAA:
0x11: type, 0x06: bytes per pixel, 0x02: subsampling factor, no bit flag set for separate alpha layer

0x01040280 UYVY + alpha:
0x01: type, 0x04: bytes per pixel, 0x02: subsampling factor, 0x80 (10000000b) - use separate alpha

0x05040100 RGBA:
0x05: type, 0x04: bytes per pixel, 0x01: subsampling factor, no bit flag for separate alpha layer


On EOSR i was able to conduct a few experiments:
0x03000000 - nothing is drawn on screen.
0x04020100 - YUV without alpha channel.
0x02010100 - Also YUV, with half the data per pixel from above.

This may differ per Zico firmware. On early Digic 6 PowerShots bitmask format is slightly different.

Resolution
Resolution of buffer doesn't need to match physical display resolution. Resolution of different buffers doesn't need to match others 

bmp_vram_info
bmp as in bitmap, not as in BMP file type

Code Select Expand

struct bmp_vram_info
{
    struct MARV * vram1;        /* one of the two bitmap buffers - statically allocated? */
    struct MARV * vram2;        /* the other bitmap buffer */
    struct MARV * back_vram;    /* we need to write to the other one - but! */
};

The code above is again sourced from digic6_dumper. This was considered to be a structure describing YUV VRAMs - double buffering is done by Canon code.
This structure name is a bit misleading for two reasons:
- first, as you already know - RGBA buffers were found and this struct describes only "output" YUV buffers
- 2nd, in reality it is just a part of bigger structure that holds more details related to renderer - like WINSYS semaphore or some flags for triggering redraw.

Later I will refer to full structure as just vram_info. I won't post exact struct as it seems to be different per camera.

Surprise: RP / 250D / R6 have no back_vram pointer. This breaks digic6-dumper YUV drawing. More on that in Chapter 4.

[kitor]

Chapter 1 - Let's get some Context - Ximr / DIGIC 7

Note: This chapter may also apply to Digic 6. I had no chance to analyze any Digic 6 ROM, but i know XimrExe is referenced in those too.

Future me: I dug into M10 (Digic 6, Powershot codebase) on CHDK forums. Ximr functions are very similar (in fact, nearly identical - earlier implementation) to the description below. RefreshVrmsSurface() and other non-Ximr named functions are specific to EOS codebase. For more details on PowerShots look into links from Chapter 6

Even more future me: 750D and 5D4 confirmed to use just "earlier" implementation, our rendering code so far works well.

RefreshVrmsSurface()
Let's start with this function. It can be found by string referencing that name. It will be important in D8+ too.

Function belongs to GUI code, and is used to redraw screen to YUV buffers after RGBA buffers has changed.
It will also swap YUV buffers as last step.

Wait, what? Yup, I don't know where to start, so I let this just sink in

On the very beginning it has this code:

Code Select Expand

if (ximr_initialized == 0) {
  Initialize_Ximr();
  ximr_initialized = 1;
}

Ximr
Ximr refers to family of "lower level" functions that utilize GPU (ZICO core) to merge input buffers into output buffer.
From now on, I will refer to input buffers as layers and output buffer as Output Chunk.

Ximr Context
Ximr family of functions uses structure known as Ximr Context.
Exact structure seems to be tied to DIGIC generation (size 0x300 on D7, 0x31C on D8, 0x7A0 on DX).
Structure contains a set of data that will be transferred to ZICO core for the rendering to happen.

Initialize_Ximr()
Helper function that will prepare empty `Ximr Context` structure and then set some details about:
- Output Chunk MARV, scaling, offsets, etc..
- Layers MARV, offsets, color parameters, enabled or not, etc...

This function is just a helper to call real Ximr functions, specific to camera. From this we know that 200d has only one Layer and Ximr Context is capable of having up to 8 of them:

Code Select Expand

XimrSetLayerVisibility(&XimrContext,0,1);
XimrSetInputLayerMARV(&XimrContext,0,bmp_vram_info.rgb_vram,0);
//...
XimrSetLayerVisibility(&XimrContext,1,0);
XimrSetLayerVisibility(&XimrContext,2,0);
//...
XimrSetLayerVisibility(&XimrContext,7,0);

XimrExe(XimrContext* pXC)
Function that sends structure to ZICO for rendering. Call is easy to notice inside of RefreshVrmsSurface() as it is done just after obtaining WINSYS semaphore.



Chapter 1.1 - Drawing in RGB
You may have noticed in Initialize_Ximr:

Code Select Expand

XimrSetInputLayerMARV(&XimrContext,0,bmp_vram_info.rgb_vram,0);
This leads us directly into Canon RGBA buffer (the one GUI is rendered into). Canon code would update this buffer, set refresh_display_needed flag to 1 and call RefreshVrmsSurface, which will tirgger re-render if flag is set.

Early tests by just stealing this buffer and calling RefreshVrmsSurface were not exactly successful. Probably due to a race with Canon code - remember WINSYS semaphore I mentioned earlier?

@names_are_hard and/or @turtius quickly noticed that we may go another way and just take a shortcut. This got implemented:

Code Select Expand


//draw on RGBA VRAM first
take_semaphore(winsys_sem, 0);
XimrExe((void *)XIMR_CONTEXT);
give_semaphore(winsys_sem);


...and it was successful. In fact, this is a part of how current builds render GUI on 200D. See @names_are_hard video:

[gifv]https://i.imgur.com/UP8gqR0.mp4[/gifv]



Chapter 1.2 - Layers. Onions have layers. Ximr has layers.
With Initialize_Ximr function calls it was easy to spot what is used by Canon code to setup "Layer 0".

In fact, with just a few calls (except creating and allocating own MARV which I skipped there):

Code Select Expand


XimrSetInputLayerMARV        (XimrContext, layerId, pNewLayer, 0);
XimrSetInputLayer_unk        (XimrContext, layerId, 0, 3, 0xff);
XimrSetInputLayerColorParams (XimrContext, layerId, pNewLayer, 1, 1);
XimrSetInputLayerVisibility  (XimrContext, layerId, 1);
XimrSetInputLayerDimensions  (XimrContext, layerId, off_x, off_y, w, h, 0, 0);


I was able to setup own drawing surface, independent from Canon code as Layer 1.

In such a case, one could need to call redraw like Canon code does:

Code Select Expand


void surface_redraw()
{
    display_refresh_needed = 1;
    sync_caches();
    RefreshVrmsSurface();
}

...but it also works with pure XimrExe call mentioned earlier.

Those are excerpts from my Ximr integration PoC: Github link
Interesting part is that on Digic 7 only, buffer itself must be aligned to 256 blocks - otherwise it will be rendered shifted (misaligned).

As you can see on the movie below, my layer is "above" Canon GUI. As RGBA layers support transparency and Canon GUI code doesn't know they exists, there's no interference with that.

[kitor]

Chapter 2 - Thinking in Layers - Ximr Context Maker - DIGIC 8+

Note: This apply to R, RP, R6.
M50 is Digic 8, but it uses code very similar to 200D, without XCM. Maybe this is R series only feature?
Update 2021-06: XCM is confirmed on 250D, code looks like RP/R6

This is where the story really started for me. Output above is a result of executing XCMStatus command in EvShell of EOS R:

Code Select Expand


--- XimrContextMaker 0x00bce800 ---
[Input Layer] No:0, Enabled:1
  VRMS Addr:0x02e18500, pImg:0x02c1e100, pAlp:0x00000000, W:960, H:540
  Color:=0x05040100, Range:FULL
  srcX:0120, srcY:0030, srcW:0720, srcH:0480, dstX:0000, dstY:0000
[Input Layer] No:1, Enabled:1
  VRMS Addr:0x03012a00, pImg:0x02e18600, pAlp:0x00000000, W:960, H:540
  Color:=0x05040100, Range:FULL
  srcX:0120, srcY:0030, srcW:0720, srcH:0480, dstX:0000, dstY:0000
[Input Layer] No:2, Enabled:0
[Input Layer] No:3, Enabled:0
[Input Layer] No:4, Enabled:0
[Input Layer] No:5, Enabled:0

[OutputChunk] No:0, Enabled:1
  VRMS Addr: 0x412ff900, pImg: 0x41100100, pAlp: 0x00000000, W:1024, H:682
  Color:=0x11060200
  LayerEnabledBit:ffffffff, isActualSize:0, isSetToContext:1
  Output Offset X:0000, Y:0000
  Resize fromX:0000, fromY:0000, outputW:1024, outputH:0682
  Horizontal 64/45, filter:0, Vertical 341/240, filter:0
[OutputChunk] No:1, Enabled:0
[OutputChunk] No:2, Enabled:0
[OutputChunk] No:3, Enabled:0

[LUT] Addr:0x00000000 Type:LUT_RANGE_FULL
  LUT is not enabled.

[Ximr Context] No:0, Addr:0x00bce810, Size:0x0000031c, Eanabled:1
ADD     :+0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +a +b +c +d +e +f
--------------------------------------------------------
  bce810:00 00 00 00 d0 02 e0 01 00 08 00 01 01 01 00 00
  bce820:4d 41 52 56 00 01 10 41 00 00 00 00 00 02 06 11
//kitor: hex output truncated


You will notice two layers already there. First one is for Canon GUI, 2nd is used for Canon LV overlays (focus box, etc).

Ximr Context Maker

I referenced to Ximr* family of functions as "lower level" for a reason. EOS R brings up a whole new family of functions: XCM*

Ximr functions exists for direct creation / manipulation of XimrContext structure.

Ximr Context Maker as name may spoil - is a tool for making (and managing) multiple XimrContext structures.

Ximr Context Maker uses it's own data structures (i will refer to it as pXCM from now). They hold:
- header containing LUT type, address, number of XimrContext
- up to 3 XimrContext structures
- structure containing settings for up to 6 Input Layers
- structure containing settings for up to 4 Output Chunks
- and probably more / decomp has messy offsets as it treats whole pXCM as single structure.

EOS R and RP use one pXCM with one XimrContext, two Input Layers and one Output Chunk.

Future me: R6/Digic X has different number of layers, chunks supported. And bring new features to XCM and Ximr. Not tested on real camera yet.

Changes to RefreshVrmsSurface
XCM replaces (encapsulates) almost all direct Ximr calls. The only "direct" call left is XimrExe in RefreshVrmsSurface.
Functions are more user oriented.

Instead of using XimrContext directly, Canon code calls setXimrParameter and XCM_MakeContext before XimrExe is executed.

Why XCM?
XCM handles a few things behind the scenes. While on 200D there are basically two possible output chunk resolutions - one for internal LCD, one for HDMI; EOS R series has EVF.

Fun part is that EVF is... addressed vertically (rotated 90 degrees). XCM handles scaling and rotation transparently.
Any attempt to directly write into YUV OutputChunk resulted in stretched image in EVF. While using RGB layers, ML GUI just works - on LCD, EVF and HDMI, without changing any resolutions as input layers stays the same all the time.

[kitor]

Chapter 3 - Interfacing with XCM, EOS R

Future me: This applies to EOS R only. RP, 250D and R6 seems to use single struct for setup. See Chapter 4.
This makes R outlier, but it is kind of expected as it was the first model to include this feature.

In theory, to create a new layer with XCM, one would need to call just:

Code Select Expand


//of course first create MARV* pNewLayer
void *pOutChunk = XCM_GetOutputChunk(XimrContext, 0);
XCM_SetSourceArea(XimrContext, newLayerID, -BMP_W_MINUS, -BMP_H_MINUS, bmp_w, bmp_h);
XCM_SetSourceSurface(XimrContext, newLayerID, pNewLayer);
XOC_SetLayerEnable(pOutChunk, newLayerID, pNewLayer);


This would be similar to pure Ximr. But there's a slight issue with that.

While XCM handles everything internally, RefreshVrmsSurface is not a part of XCM, but a part of GUI. And since GUI now use layers (one for GUI, one for overlays), it means it wants to have control over XCM.

EOS R has three arrays related to that, quote from compositor.c:

Code Select Expand


/*
* RENDERER_LayersArr    holds MARV struct pointers to layers created by
*                       RENDERER_InitializeScreen(). Those are layers created
*                       by Canon GUI renderer code.
*                       SaveVRAM on EvShell uses this array for enumeration.
* VMIX_LayersArr        Like above, but used by VMIX for XCM rendering setup.
*                       Entries from array above are copied in (what I called)
*                       Initialize_Renedrer() after RENDERER_InitializeScreen()
*                       and VMIX_InitializeScreen() are done.
* VMIX_LayersEnableArr  Controls if layer with given ID is displayed or not.
*                       Set up just after XCM_LayersArr is populated.
*/
extern struct MARV *RENDERER_LayersArr[XCM_MAX_LAYERS];
extern struct MARV *VMIX_LayersArr[XCM_MAX_LAYERS];
extern uint32_t     VMIX_LayersEnableArr[XCM_MAX_LAYERS];


You will find references to those structures in Initialize_Renedrer function (search for RENDERER_InitializeScreen and go to the only place it is referenced)

setXimrParameter() call in each RefreshVrmsSurface run will use those arrays to update configuration. This means even if we interface directly with XCM, it will be overwritten on each redraw anyway.

Code for EOS R is thus very simple, just add those layers to GUI structures:

Code Select Expand


//example assumes that MARV * pNewLayer is already created.
//add new layer to compositor layers array
RENDERER_LayersArr[newLayerID] = pNewLayer;
VMIX_LayersArr[newLayerID] = pNewLayer;

//enable new layer - just in case (all were enabled by default on R180)
VMIX_LayersEnableArr[newLayerID] = 1;

[kitor]

Chapter 4 - Interfacing with XCM - EOS RP / 250D / R6

Note: Based on RP and R6 ROMs. But as R6 XCM almost identical to RP and 250D (except stuff described in this chapter) I think it will apply to R5 too.
I believe this will be 'common' implementation, with just R as an outlier (see chapter 3)

Edit 2023-07: Now tested on RP+. Doesn't work on Digic X models, but that's due to a new memory management that we don't understand

RefreshVrmsSurface() / VMIX_TransferRectangleToVram()
First of all, RP dropped back_vram pointer from vram_info. RefreshVrmsSurface seems to simply swap two buffers as one of last steps on each call.

Around 850D, function got renamed to VMIX_TransferRectangleToVram which makes sense in terms of other VMIX_* family functions.

Structures
If you look at RENDERER_InitializeScreen in RP and R6, you'll find that it uses different structures than R. Now there's just one array of structures for input layers.

Code Select Expand


struct RENDERER_LayerInfo
{
    MARV     *pMARV;
    uint32_t  width;
    uint32_t  height;
}


Adding layers
Then later in Initialize_Renedrer, for each of those layers XCM_SetSourceSurface, XOC_SetLayerEnable and XCM_SetSourceArea are called (see Chapter 3).
850D and Digic X models have extra call to XCM_SetColorMatrixType which DNE on previous models (but it is set to 0 anyway).

It looks that RefreshVrmsSurface() will not mess with layers anymore during redraw (like it does on R).

Thus IMO to setup a new layer on RP/R6 it will be enough to call

Code Select Expand


XCM_SetSourceSurface(_pXCM, newLayerID, pNewLayer);
XCM_SetSourceArea(_pXCM, newLayerID, -BMP_W_MINUS, -BMP_H_MINUS, BMP_W_PLUS + BMP_W_MINUS, BMP_H_PLUS + BMP_H_MINUS);
XOC_SetLayerEnable(0,0,newLayerID,1);
// Around 850D "ColorMatrix" setup was added
XCM_SetColorMatrixType(0, newLayerID, 0);


and maybe add it to array of RENDERER_LayerInfo structures.

MARV on R6 / Digic X
Due to this additional "ColorMatrix" stuff, it seems that Digic X has additional fields in MARV struct - see Chapter 0. I guessed those offsets/sizes based on XCMStatus code. One would need to take real dump for RGB MARV from camera and see what is held in those fields.

[kitor]

Chapter 5 - Drawing in RGBA - summary

Drawing over Canon layers - any generation (tested on 200D, R)
Take Canon GUI layer (first on the list or only layer - depending on generation).
Draw over Canon layer. Use XimrExe method from Chapter 1 to redraw.

Drawing on own layers - Pure Ximr (cameras without XCM - tested on 200D)
Setup layers using pure Ximr - see Chapter 1.
Draw over your layer. Use surface_redraw or XimrExe method from Chapter 1 to redraw screen.

Drawing on own layers - cameras with XCM - tested on R
Setup layers via XCM or specific structures - see Chapter 2 and Chapter 3.
Draw over your layer. Use surface_redraw or XimrExe method from Chapter 1 to redraw screen.



Chapter 5.1 - How we did it on 200D / R

Code will hopefully soon be merged to dev branch in @names_are_hard magiclantern_simplified repository.

Long story short: @names_are_hard prepared our own indexed RGB layer (not related to any stuff from this thread). It is used as compatibility layer for existing Magic Lantern code, which expect indexed RGB buffer.
This layer is then drawn into RGBA buffer with use of indexed RGB LUT, and XimrExe is used to redraw screen.

This is known as FEATURE_VRAM_RGBA. When only this feature is set, code will directly use Canon GUI RGBA layer.

There's additional FEATURE_COMPOSITOR_XCM. It integrates EOSR support from Chapter 3 and exposes our own RGBA layer to indexed RGB compatibility code.

On the movie below 200D draws directly to Canon GUI layer and R draws over our own RGBA layer

[kitor]

Chapter 6 - Digic 6 and CHDK
And then, a few days later, @Lorenzo33324 on Discord sent a link to CHDK Digic 6 porting thread:
https://chdk.setepontos.com/index.php?topic=11316.80
https://chdk.setepontos.com/index.php?topic=12788.0

I'd be interested in seeing that new drawing system. I was thinking about using the RGBA buffer(s), but have not attempted to make an implementation.

This means two things:
- Digic 6 RGBA confirmed
- We were 4 years behind and 3 steps ahead CHDK

Chapter 6.1 - EOS M10
My findings on CHDK forum:
https://chdk.setepontos.com/index.php?topic=12788.msg145641#msg145641

Interesting bits:
- different color "flags" content in MARV struct
- 8 layers possible like on Digic 7
- different (both in size and content) Ximr Context structures on different Digic 6 PowerShot base cameras, seems that Zico firmware was changed mid-generation
- M10 Ximr Context in terms of layer definition and structure size is 1:1 to 200D. But other stuff differs (e.g. Output Chunk configuration).
- "API" / Ximr family of functions seems to have the same interface on M10 like 200D, just internals (structure) differs.

[kitor]

Chapter 6.2 - Digic 6 EOS

I finally analyzed D6 EOS rom.
Looks very much like predecessor of Digic 7. refreshVrmsSurface has much less code, most notable difference is missing vsync semaphore around XimrExe call.

750D has ability to run 7 layers (odd, but at least init code initializes layer 0 and disables layers 1-6). Color flags looks like those I saw on M10.

Nothing was tested on real camera yet.

Tested, works just like D7/D8, without use of mentioned semaphore. I'm not sure if Vsync is achieved any other way, or they just didn't implement it in D6.

[g3gg0]

indeed very impressive work kitor!

[kitor]

Interesting part is that GUI as far as Digic 8 is still drawn in indexed 8 bit. Drawing indexed resources into RGBA buffer is hardware accelerated via Zico core (there are functions to draw some primitives as well as sprites)
After I found palettes, and class of functions used to upload/swap palettes (they use stack of palette IDs to swap them) @names_are_hard was able to brute force change GUI palette on 200D.

There's a big chance we can use Zico to convert our "fake" indexed RGB buffer (one were we draw ML GUI now) to RGBA, however it was not yet explored that far.

[kitor]

Recently I did some tests with EGL capabilities of ZICO (GPU) core. Quote from R/RP thread:

Code Select Expand


  mzrm eglGetDisplay 0
  mzrm eglInitialize 1 2720200 2720204
  mzrm eglGetError
cmd status      EGL_SUCCESS
EGL Version     1.3 // via eglInitialize()
EGL_VENDOR      TAKUMI Corporation
EGL_VERSION     1.3 TAKUMI Corporation
EGL_EXTENSIONS  NONE
EGL_CLIENT_APIS OpenGL_ES OpenVG

Fun fact: Zico core can do OpenGL ES and EGL 1.3. There's no implementation ICU side, but at least EGL is implemented on Zico. After some digging I was able to initialize EGL and query capability strings.
Not sure if that will go anywhere (OpenGL seems missing in Zico firmware even though hardware is capable of that), I never worked with OpenGL before so I don't know if EGL is enough to do anything interesting.
There's OpenVG support too.

This was a good excuse to dig into ICU-Zico communication.

Fun fact, EGL/OpenVG while exposed via mzrm (ICU - Zico comm) it is cleary not tested (?) by Canon. Some OpenVG functions returs Zico code pointers (Xtensa code in Zico private memory region) to ARM side...
There is at least part of OpenGL implementation in Zico firmware, but wrappers are completely missing from Zico RPC table.

Anyway, EGL tests ended in a discovery!

Up to now we rendered GUI in indexed RGB (like old models do) and then converted it using a naive code (well, a loop over a buffer + LUT table) to RGBA that new generations use.
We theoreticized since the very beginning that ZICO (gpu core) could be probably used to accelerate that, since Canon also still keeps whole gui resources in indexed RGB.

A few days ago I was able to achieve this. Full double-buffered ML menu with hardware indexed rgb -> RGBA rendering on Zico.

https://www.youtube.com/watch?v=q5aps3qHU_w

While old method felt "laggy" and was prone to rendering glitches, this is so damn fast. Feels exactly like using 5D3.

This is all thanks to a single Zico function - GrypDsCoreDrawImageToVramForEqualPhase.

Caveats? Of course they are caveats Camera likes to randomly lock up (literally lock, UART output hangs mid-print) on Canon main menu enter.
This happens on R, I wasn't able to trigger it on 80D (D6) and 77D (D7). No other models were tested yet.

Also it is unable to render full buffer at once. Buffer is 960x540px (out of which 720x480 is the visible area on LCD). I was able to get at max 960x480px renedred at once, so I just split rendering into a two calls - for upper and lower half of indexed buffer.

Also on 80D and 77D menu is still laggy, we get like 5FPS refresh rate while R seems to do at least 20FPS. I switched LV frame rate to power saving mode and it was still as fast.

Dirty code (EGL tests, ZICO rendering + double buffering, etc) is on my dirty branch: https://github.com/kitor/magiclantern_simplified/tree/EOSR_zico_EGL

[70MM13]

Fascinating stuff! Thanks for the update.

Silly question: does this potentially expose computational resources that could somehow be used for arbitrary purposes?

Either way, nice find!

[kitor]

Yes and no. ICUs have secondary cores, delegated to some specific tasks.
D6 has ARM core called OMAR. Likely used for tasks like EEKO on D5
>= D6 have Zico core - Xtensa arch, delegated for GPU offload)
>= D7 have Lime / CCLime - Xtensa arch, offloading of network stuff
>= D8 have (forgot the name) - again Xtensa arch, seems to be used for offloading cr2/cr3/etc encoding/decoding.

We can control firmware loaded there, but it will be a non-trivial task. We are more likely to use available functionality to accelerate/offload GUI rendering than anything else.
Zico is interesting as it has direct access mapped 1:1 at least to lower 1GB of ICU address space. There's some private memory at 0x80000000 and then again shared one at 0xBFF...... (used for RPC)