Author Topic: Compositors, layers, contexts in RGB and YUV - How Digic 7(6?)+ draw GUI  (Read 555 times)

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Preamble

Up to not-so-long-ago everybody* thought that Digic 6+ cameras render GUI in YUV. In 80D thread, @a1ex documented MARV structures that referenced two UYVY VRAM surfaces. More of that in Chapter 0.

In last December @coon noticed that EOS RP seems to have RGBA VRAMs. This got somehow unnoticed, and he was unable to mess with them anyway.

About a month ago, after I tried to make GUI rendering work on R in YUV and failed successfully. I thought "There's no way that Canon renders GUI in YUV". The next day I found the same RGBA layers that coon noticed, with whole infrastructure related to that. This is Chapter 2 and Chapter 3.

@names_are_hard started looking into his 200D ROM and found some similar things to R/RP. It was lacking a lot of stuff, but he was able to locate one RGBA layer. I bought 200D to investigate this, and... this goes into Chapter 1

Finally @names_are_hard made a nice compatibility layer for existing indexed RGB code and I integrated compositor support PoC for EOSR. For TL;DR see Chapter 5.1

*Future me: See Chapter 6 :)
EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 0 - MARV backwards is VRAM

Some very basic details on how D7+ (D6+?) store VRAM information:

MARV structure
Code based on vram.h in digic6_dumper branch, updated to my research.

Code: [Select]
struct MARV
{
    uint32_t signature;         /* MARV - VRAM reversed */
    void * bitmap_data;         /* Pointer to buffer representing actual VRAM draw surface*/
    uint8_t * opacity_data;     /* Pointer to buffer representing opacity data in case of UYVY + separate alpha VRAM  */
    uint32_t color;             /* Bitmask (?) representing color type */
#ifdef CONFIG_DIGIC_X
    uint32_t unk1;              /* New values seems to be related to a bit different color handling */
    uint32_t unk2;
    uint32_t unk3;
#endif
    uint32_t width;             /* X resolution of buffer */
    uint32_t height;            /* Y resolution of buffer */
    void * pmem;                /* pointer to PMEM (Permanent Memory) structure */
};

MARV is a structure that describes VRAM parameters.

color
Seems to be a bitmask? There are multiple possible values referenced across R, RP, 200D, R6 roms, but only three seems to be used:
0x05040100  - 32 bit RGBA
0x01040280  - 32 bit UYVY + separate alpha buffer (opacity_data)
0x11060200  - 40 bit UYVYAA (alpha stored in main buffer)

Other possible values (but unknown meaning), pseudocode based on XimrSetInputLayerMARV in 200D. This may explain why we think it is a bitmask.
Don't mind ID column, this is just an internal ID used by some functions from Chapter 1.
Code: [Select]
         Color     ID    Comment
if       0x05040100   0x5   RGBA - this one is used by RGB layers on 200D, R, RP, R6
elif  <  0x05040101
{
    if   0x01040280   0x1   UYVY + separate alpha buffer (opacity_data)
    if   0x02010100   0x2
    if   0x03000000   0x3
    if   0x04020100   0x4
    else              0x0
{
else
{
    if   0x06040100   0x6
    if   0x07020100   0xA 
    else              0x0
}

Update
color / flags field contains at least 4 information, starting from MSB:

Code: [Select]
8 bits: Seems to be related to "ID" from table above. Possibly Ximr type ID for color space.
8 bits: Bits per pixel?
8 bits: Subsampling factor?
1 bit : Flag if structure utilize that "separate opacity layer"
7 bits: Unknown / not referenced in any code I analyzed.

"Bits per pixel" and "Subsampling factor" are used as follows in CreateMARV:

Code: [Select]
BMP_VRAM_SIZE = fast_division(width * bits_per_pixel, subsampling_factor);
BMP_VRAM_SIZE = BMP_VRAM_SIZE * height;

So for example:
Code: [Select]
0x11060200 UYVYAA:
0x11: type, 0x06: bytes per pixel, 0x02: subsampling factor, no bit flag set for separate alpha layer

0x01040280 UYVY + alpha:
0x01: type, 0x04: bytes per pixel, 0x02: subsampling factor, 0x80 (10000000b) - use separate alpha

0x05040100 RGBA:
0x05: type, 0x04: bytes per pixel, 0x01: subsampling factor, no bit flag for separate alpha layer

On EOSR i was able to conduct a few experiments:
0x03000000 - nothing is drawn on screen.
0x04020100 - YUV without alpha channel.
0x02010100 - Also YUV, with half the data per pixel from above.

Resolution
Resolution of buffer doesn't need to match physical display resolution. Resolution of different buffers doesn't need to match others  :)

bmp_vram_info
bmp as in bitmap, not as in BMP file type
Code: [Select]
struct bmp_vram_info
{
    struct MARV * vram1;        /* one of the two bitmap buffers - statically allocated? */
    struct MARV * vram2;        /* the other bitmap buffer */
    struct MARV * back_vram;    /* we need to write to the other one */
};

The code above is again sourced from digic6_dumper. This was considered to be a structure describing YUV VRAMs - double buffering is done by Canon code.
This structure name is a bit misleading for two reasons:
- first, as you already know - RGBA buffers were found and this struct describes only "output" YUV buffers
- 2nd, in reality it is just a part of bigger structure that holds more details related to renderer - like WINSYS semaphore or some flags for triggering redraw.

Later I will refer to full structure as just vram_info. I won't post exact struct as it seems to be different per camera.

Surprise: RP and R6 have no back_vram pointer. More on that in Chapter 4.
EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 1 - Let's get some Context - Ximr / DIGIC 7

Note: This chapter may also apply to Digic 6. I had no chance to analyze any Digic 6 ROM, but i know XimrExe is referenced in those too.

RefreshVrmsSurface()
Let's start with this function. It can be found by string referencing that name. It will be important in D8+ too.

Function belongs to GUI code, and is used to redraw screen to YUV buffers after RGBA buffers has changed.
It will also swap YUV buffers as last step.

Wait, what? Yup, I don't know where to start, so I let this just sink in :)

On the very beginning it has this code:
Code: [Select]
if (ximr_initialized == 0) {
  Initialize_Ximr();
  ximr_initialized = 1;
}

Ximr
Ximr refers to family of "lower level" functions that utilize GPU (ZICO core) to merge input buffers into output buffer.
From now on, I will refer to input buffers as layers and output buffer as Output Chunk.

Ximr Context
Ximr family of functions uses structure known as Ximr Context.
Exact structure seems to be tied to DIGIC generation (size 0x300 on D7, 0x31C on D8, 0x7A0 on DX).
Structure contains a set of data that will be transferred to ZICO core for the rendering to happen.

Initialize_Ximr()
Helper function that will prepare empty `Ximr Context` structure and then set some details about:
- Output Chunk MARV, scaling, offsets, etc..
- Layers MARV, offsets, color parameters, enabled or not, etc...

This function is just a helper to call real Ximr functions, specific to camera. From this we know that 200d has only one Layer and Ximr Context is capable of having up to 8 of them:
Code: [Select]
XimrSetLayerVisibility(&XimrContext,0,1);
XimrSetInputLayerMARV(&XimrContext,0,bmp_vram_info.rgb_vram,0);
//...
XimrSetLayerVisibility(&XimrContext,1,0);
XimrSetLayerVisibility(&XimrContext,2,0);
//...
XimrSetLayerVisibility(&XimrContext,7,0);

XimrExe(XimrContext* pXC)
Function that sends structure to ZICO for rendering. Call is easy to notice inside of RefreshVrmsSurface() as it is done just after obtaining WINSYS semaphore.



Chapter 1.1 - Drawing in RGB
You may have noticed in Initialize_Ximr:
Code: [Select]
XimrSetInputLayerMARV(&XimrContext,0,bmp_vram_info.rgb_vram,0);This leads us directly into Canon RGBA buffer (the one GUI is rendered into). Canon code would update this buffer, set refresh_display_needed flag to 1 and call RefreshVrmsSurface, which will tirgger re-render if flag is set.

Early tests by just stealing this buffer and calling RefreshVrmsSurface were not exactly successful. Probably due to a race with Canon code - remember WINSYS semaphore I mentioned earlier?

@names_are_hard and/or @turtius quickly noticed that we may go another way and just take a shortcut. This got implemented:
Code: [Select]
//draw on RGBA VRAM first
take_semaphore(winsys_sem, 0);
XimrExe((void *)XIMR_CONTEXT);
give_semaphore(winsys_sem);

...and it was successful. In fact, this is a part of how current builds render GUI on 200D. See @names_are_hard video:




Chapter 1.2 - Layers. Onions have layers. Ximr has layers.
With Initialize_Ximr function calls it was easy to spot what is used by Canon code to setup "Layer 0".

In fact, with just a few calls (except creating and allocating own MARV which I skipped there):
Code: [Select]
XimrSetInputLayerMARV        (XimrContext, layerId, pNewLayer, 0);
XimrSetInputLayer_unk        (XimrContext, layerId, 0, 3, 0xff);
XimrSetInputLayerColorParams (XimrContext, layerId, pNewLayer, 1, 1);
XimrSetInputLayerVisibility  (XimrContext, layerId, 1);
XimrSetInputLayerDimensions  (XimrContext, layerId, off_x, off_y, w, h, 0, 0);

I was able to setup own drawing surface, independent from Canon code as Layer 1.

In such a case, one could need to call redraw like Canon code does:
Code: [Select]
void surface_redraw()
{
    display_refresh_needed = 1;
    sync_caches();
    RefreshVrmsSurface();
}
...but it also works with pure XimrExe call mentioned earlier.

Those are excerpts from my Ximr integration PoC: Github link
Interesting part is that on Digic 7 only, buffer itself must be aligned to 256 blocks - otherwise it will be rendered shifted (misaligned).

As you can see on the movie below, my layer is "above" Canon GUI. As RGBA layers support transparency and Canon GUI code doesn't know they exists, there's no interference with that.

EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 2 - Thinking in Layers - Ximr Context Maker - DIGIC 8+

Note: This apply to R, RP, R6.
M50 is Digic 8, but it uses code very similar to 200D, without XCM. Maybe this is R series only feature?


This is where the story really started for me. Output above is a result of executing XCMStatus command in EvShell of EOS R:

Code: [Select]
--- XimrContextMaker 0x00bce800 ---
[Input Layer] No:0, Enabled:1
  VRMS Addr:0x02e18500, pImg:0x02c1e100, pAlp:0x00000000, W:960, H:540
  Color:=0x05040100, Range:FULL
  srcX:0120, srcY:0030, srcW:0720, srcH:0480, dstX:0000, dstY:0000
[Input Layer] No:1, Enabled:1
  VRMS Addr:0x03012a00, pImg:0x02e18600, pAlp:0x00000000, W:960, H:540
  Color:=0x05040100, Range:FULL
  srcX:0120, srcY:0030, srcW:0720, srcH:0480, dstX:0000, dstY:0000
[Input Layer] No:2, Enabled:0
[Input Layer] No:3, Enabled:0
[Input Layer] No:4, Enabled:0
[Input Layer] No:5, Enabled:0

[OutputChunk] No:0, Enabled:1
  VRMS Addr: 0x412ff900, pImg: 0x41100100, pAlp: 0x00000000, W:1024, H:682
  Color:=0x11060200
  LayerEnabledBit:ffffffff, isActualSize:0, isSetToContext:1
  Output Offset X:0000, Y:0000
  Resize fromX:0000, fromY:0000, outputW:1024, outputH:0682
  Horizontal 64/45, filter:0, Vertical 341/240, filter:0
[OutputChunk] No:1, Enabled:0
[OutputChunk] No:2, Enabled:0
[OutputChunk] No:3, Enabled:0

[LUT] Addr:0x00000000 Type:LUT_RANGE_FULL
  LUT is not enabled.

[Ximr Context] No:0, Addr:0x00bce810, Size:0x0000031c, Eanabled:1
ADD     :+0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +a +b +c +d +e +f
--------------------------------------------------------
  bce810:00 00 00 00 d0 02 e0 01 00 08 00 01 01 01 00 00
  bce820:4d 41 52 56 00 01 10 41 00 00 00 00 00 02 06 11
//kitor: hex output truncated

You will notice two layers already there. First one is for Canon GUI, 2nd is used for Canon LV overlays (focus box, etc).

Ximr Context Maker

I referenced to Ximr* family of functions as "lower level" for a reason. EOS R brings up a whole new family of functions: XCM*

Ximr functions exists for direct creation / manipulation of XimrContext structure.

Ximr Context Maker as name may spoil - is a tool for making (and managing) multiple XimrContext structures.

Ximr Context Maker uses it's own data structures (i will refer to it as pXCM from now). They hold:
- header containing LUT type, address, number of XimrContext
- up to 3 XimrContext structures
- structure containing settings for up to 6 Input Layers
- structure containing settings for up to 4 Output Chunks
- and probably more / decomp has messy offsets as it treats whole pXCM as single structure.

EOS R and RP use one pXCM with one XimrContext, two Input Layers and one Output Chunk.

Changes to RefreshVrmsSurface
XCM replaces (encapsulates) almost all direct Ximr calls. The only "direct" call left is XimrExe in RefreshVrmsSurface.
Functions are more user oriented.

Instead of using XimrContext directly, Canon code calls setXimrParameter and XCM_MakeContext before XimrExe is executed.

Why XCM?
XCM handles a few things behind the scenes. While on 200D there are basically two possible output chunk resolutions - one for internal LCD, one for HDMI; EOS R series has EVF.

Fun part is that EVF is... addressed vertically (rotated 90 degrees). XCM handles scaling and rotation transparently.
Any attempt to directly write into YUV OutputChunk resulted in scratched image in EVF. While using RGB layers, ML GUI just works - on LCD, EVF and HDMI, without changing any resolutions as input layers stays the same all the time.
EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 3 - Interfacing with XCM, EOS R

In theory, to create a new layer with XCM, one would need to call just:
Code: [Select]
//of course first create MARV* pNewLayer
void *pOutChunk = XCM_GetOutputChunk(XimrContext, 0);
XCM_SetSourceArea(XimrContext, newLayerID, -BMP_W_MINUS, -BMP_H_MINUS, bmp_w, bmp_h);
XCM_SetSourceSurface(XimrContext, newLayerID, pNewLayer);
XOC_SetLayerEnable(pOutChunk, newLayerID, pNewLayer);

This would be similar to pure Ximr. But there's a slight issue with that.

While XCM handles everything internally, RefreshVrmsSurface is not a part of XCM, but a part of GUI. And since GUI now use layers (one for GUI, one for overlays), it means it wants to have control over XCM.

EOS R has three arrays related to that, quote from compositor.c:
Code: [Select]
/*
 * XCM_RendererLayersArr holds MARV struct pointers to layers created by
 *                       RENDERER_InitializeScreen(). Those are layers created
 *                       by Canon GUI renderer code.
 *                       SaveVRAM on EvShell uses this array for enumeration.
 * XCM_LayersArr         Like above, but used by XCM for rendering setup.
 *                       Entries from array above are copied in (what I called)
 *                       Initialize_Renedrer() after RENDERER_InitializeScreen()
 *                       and VMIX_InitializeScreen() are done.
 * XCM_LayersEnableArr   Controls if layer with given ID is displayed or not.
 *                       Set up just after XCM_LayersArr is populated.
 */
extern struct MARV *XCM_RendererLayersArr[XCM_MAX_LAYERS];
extern struct MARV *XCM_LayersArr[XCM_MAX_LAYERS];
extern uint32_t     XCM_LayersEnableArr[XCM_MAX_LAYERS];

You will find references to those structures in Initialize_Renedrer function (search for RENDERER_InitializeScreen and go to the only place it is referenced)

Self note: as those structures are GUI code, not XCM - I should rename them in compositor.c and stubs

setXimrParameter() call in each RefreshVrmsSurface run will use those arrays to update configuration. This means even if we interface directly with XCM, it will be overwritten on each redraw anyway.

Code for EOS R is thus very simple, just add those layers to GUI structures:
Code: [Select]
//assumes that MARV * pNewLayer is already created.
//add new layer to compositor layers array
XCM_LayersArr[newLayerID] = pNewLayer;
XCM_RendererLayersArr[newLayerID] = pNewLayer;
//enable new layer - just in case (all were enabled by default on R180)
XCM_LayersEnableArr[newLayerID] = 1;

EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 4 - Interfacing with XCM - EOS RP/R6

Note: Based on RP and R6 ROMs. But as R6 XCM almost identical to RP (except stuff described in this chapter) I think it will apply to R5 too.
This is untested, I hope @coon will verify it soon on his RP and I'll be able to update this section.


RefreshVrmsSurface() / VMIX_TransferRectangleToVram()
First of all, RP dropped back_vram pointer from vram_info. RefreshVrmsSurface seems to simply swap two buffers as one of last steps on each call.

In EOS R6 rom, function got renamed to VMIX_TransferRectangleToVram which makes sense in terms of other VMIX_* family functions.

Structures
If you look at RENDERER_InitializeScreen in RP and R6, you'll find that it uses different structures than R. Now there's just one array of structures for input layers.

Code: [Select]
struct RENDERER_LayerInfo
{
    MARV     *pMARV;
    uint32_t  width;
    uint32_t  height;
}

Adding layers
Then later in Initialize_Renedrer, for each of those layers XCM_SetSourceSurface, XOC_SetLayerEnable and XCM_SetSourceArea are called (see Chapter 3). R6 has additional call to XCM_SetColorMatrixType which is new for Digic X (but it is set to 0 anyway).

It looks that RefreshVrmsSurface() will not mess with layers anymore during redraw (like it does on R).

Thus IMO to setup a new layer on RP/R6 it will be enough to call
Code: [Select]
void *pOutChunk = XCM_GetOutputChunk(XimrContext, 0);
XCM_SetSourceArea(XimrContext, newLayerID, -BMP_W_MINUS, -BMP_H_MINUS, bmp_w, bmp_h);
XCM_SetSourceSurface(XimrContext, newLayerID, pNewLayer);
XOC_SetLayerEnable(pOutChunk, newLayerID, pNewLayer);
#ifdef CONFIG_DIGIC_X
XCM_SetColorMatrixType(0, newLayerID);
#endif

and maybe add it to array of RENDERER_LayerInfo structures.

MARV on R6 / Digic X
Due to this additional "ColorMatrix" stuff, it seems that Digic X has additional fields in MARV struct - see Chapter 0. I guessed those offsets/sizes based on XCMStatus code. One would need to take real dump for RGB MARV from camera and see what is held in those fields.
EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 5 - Drawing in RGBA - summary

Drawing over Canon layers - any generation (tested on 200D, R)
Take Canon GUI layer (first on the list or only layer - depending on generation).
Draw over Canon layer. Use XimrExe method from Chapter 1 to redraw.

Drawing on own layers - Pure Ximr (cameras without XCM - tested on 200D)
Setup layers using pure Ximr - see Chapter 1.
Draw over your layer. Use surface_redraw or XimrExe method from Chapter 1 to redraw screen.

Drawing on own layers - cameras with XCM - tested on R
Setup layers via XCM or specific structures - see Chapter 2 and Chapter 3.
Draw over your layer. Use surface_redraw or XimrExe method from Chapter 1 to redraw screen.



Chapter 5.1 - How we did it on 200D / R

Code will hopefully soon be merged to dev branch in @names_are_hard magiclantern_simplified repository.

Long story short: @names_are_hard prepared our own indexed RGB layer (not related to any stuff from this thread). It is used as compatibility layer for existing Magic Lantern code, which expect indexed RGB buffer.
This layer is then drawn into RGBA buffer with use of indexed RGB LUT, and XimrExe is used to redraw screen.

This is known as FEATURE_VRAM_RGBA. When only this feature is set, code will directly use Canon GUI RGBA layer.

There's additional FEATURE_COMPOSITOR_XCM. It integrates EOSR support from Chapter 3 and exposes our own RGBA layer to indexed RGB compatibility code.

On the movie below 200D draws directly to Canon GUI layer and R draws over our own RGBA layer :)

EOS R, 200d

kitor

  • Developer
  • Member
  • *****
  • Posts: 220
Chapter 6 - Digic 6 and CHDK
And then, a few days later, @Lorenzo33324 on Discord sent a link to CHDK Digic 6 porting thread:
https://chdk.setepontos.com/index.php?topic=11316.80
https://chdk.setepontos.com/index.php?topic=12788.0

Quote from: srsa
I'd be interested in seeing that new drawing system. I was thinking about using the RGBA buffer(s), but have not attempted to make an implementation.

This means two things:
- Digic 6 RGBA confirmed
- We are 4 years behind and 3 steps ahead CHDK ;)

Chapter 6.1 - EOS M10
My findings on CHDK forum:
https://chdk.setepontos.com/index.php?topic=12788.msg145641#msg145641

Interesting bits:
- different color "flags" content in MARV struct
- 8 layers possible like on Digic 7
- different (both in size and content) Ximr Context structures on different cameras, seems that Zico firmware was changed mid-generation
- M10 Ximr Context in terms of layer definition and structure size is 1:1 to 200D. But other stuff differs (e.g. Output Chunk configuration).
- "API" / Ximr family of functions seems to have the same interface on M10 like 200D, just internals (structure) differs.
EOS R, 200d