Messages

It may not be safe to run the Omar code from ICU.  These cores are similar but different ARM architectures.  The code may not be compatible with ICU.  I also don't know if this code is truly PIC.  Arm code tends to be mostly PIC by default, but this is no guarantee it all is.  And, calling shamem_read() from ICU may well give incorrect results - assuming it's intended to be read from Omar, the masking may be the reverse of what you expect.

Some of the EDMAC routines obtain kernel locks before proceeding.  Since the Omar core is running an entirely different instance of the OS, I wouldn't expect these to be shared locks with the ICU.  This is untested.  And, since the kernel locks probably aren't shared between different OS instances, the cam will crash if you contest for resources on different cores.

The Omar 0x8000_6XXX code is populated by a copy from fe6X_YYYY rom region.  See the struct created in fe0ad874().  After the copy, the memory is visible at 0xdfXXXXXX on 7D2, or 0x8000_XXXX on Omar - I haven't tested writing the 7D2 side so I don't know if this is a mirror (my guess) or a copy.

Triggering arbitrary code from Omar looks easy, so this may be the safer route.  That way we know code is running in the intended context.  If you can prove all the code is appropriate and safe to run from ICU context that's fine too.  Feels like more work to me.  It also seems fairly likely there is an existing RPC mechanism.

The shamem info is useful to me, thanks.  I've not worked with this before, it doesn't seem to exist on D7 and above.

fe0ad874 is load_Omar().  This copies (somehow, indirectly) Omar firmware from 7D2 master, and does some other region setup.

I would guess 0xdff0_0000 is mirrored to 0x8000_0000, 0xdff4_0800 to 0x8000_0800.  However, my quick tests on phys cam don't make sense.  Maybe I did something wrong, maybe my assumptions are wrong.  Either way, it seemed potentially useful enough that I wanted to share before fully understanding it.

Bear in mind, we think the Omar core is Digic 5: ARMv5t.  D6 is ARMv7-R.  I know that's not forward compatible: -R has hardware division ops.  Can't remember if it's backwards compatible.  I think so?  Running Omar code from ICU may be dumb for a bunch of other reasons too, but you know that

EDIT: mistake identified.  I'd got two comments in the wrong places in a struct.

Code Select Expand


omar_dst_01 0xdff00000        // this is mapped as Omar atcm at 0x0
icu_src_01  code at 0xfe6afd04
len_01      0x6c80
omar_dst_02 0xdff40800        // this is mapped as Omar btcm at 8000_0800
icu_src_02  code at 0xfe6b698c
len_02      0xa0f0
omar_dst_03 0x1ac0000
icu_src_03  0xfe6c0a84
len_03      0x95c8
omar_dst_04 0x1ae0000
icu_src_04  0xfe6ca054
len_04      0x1f2ea8


So, all the Omar EDMAC func code should be readable (and writable!) in the dff40800 region.  Good luck flushing Omar's icache, I guess.

Nice, glad it's useful to you!  I've been looking for these for ages (more the 200D ones really), and now I know how to do it, it's not even that hard to find.  But the docs for what anything means are so fragmentary

The EDMAC stuff you're talking about helps me, it's easier to understand the wiki page now: https://magiclantern.fandom.com/wiki/Register_Map#EDMAC

It would make sense to me if you could read back the edmac_info region values, I'll try that.  I want to map out usage, I assume channels get remapped when in different modes / purposes.  I'm reading up on EDMAC (again) and intend to run tests on 200D - Digic 7 doesn't have Omar, and we can patch rom via MMU, so it's easier.  The code in edmac-memcpy.c has a lot of very ugly hard-coded values, as does the module.  I will look at making these general, should help me understand how things work, and it would be good to have these working on modern cams.

Re running code on Omar, I can think of a few ways.  The cleanest would be finding an existing Canon RPC mechanism.  It feels likely this exists (probably the code near fe0a30b2 is doing RPC with Omar).  If it doesn't, or we can't find it, since Omar is running DryOS, and ICU supplies the firmware it runs, it wouldn't be that hard to create our own RPC.  Bilal pointed me at some experiments on Eeko RPC here: https://www.magiclantern.fm/forum/index.php?topic=13408.msg175842#msg175842  Might be useful?  It's not clear how it triggers code in shared mem to run, presumably via the 0xc0XX_YYYY registers, but it's not explained.

So, we have options.  Since I don't know how to use EDMAC yet perhaps it makes sense for you to get it working however you want, and I can try and make a cleaner RPC method if needed.  Presumably 5D4 is similar, would be nice to have it somewhat general.

lv_save_raw is new to me.  The function just sets a flag, and indeed forum search suggests "There's a debug flag (lv_save_raw) that enables this".  I see that CONFIG_EDMAC_RAW_SLURP is supposed to be "better" than lv_save_raw.  Trying to understand this from reading forum posts is so incredibly slow.  I'm glad you're still around to keep the old knowledge alive

Possibly related...  take a look at 7d2 master rom1, fe25570a.  This is clearly Omar init related and looks to be doing something EDMAC.

I believe 13dfe() is send_msg_to_Omar(), there's a similar ComForOmar on the other side.  No idea on the message contents (refs Postman, so probably the same system, whatever that is).

I have identified the following stubs for 200D, 1.0.1:

35392 StartEDmac
35486 edmac_set_addr
35506 edmac_set_size
3649c ConnectWriteEDmac
364f0 ConnectReadEDmac
3678e RegisterEDmacCompleteCBR

The EngDrv functions don't seem to exist here, instead registers are written to directly.  shamem_read() also appears missing, probably for similar reason.

After a lot of reading very old forum posts, and much messing about in Ghidra, I now have some important stubs found for 7D2 1.1.2.  These are all on Omar:

8000626a EngDrvOut
800062a4 EngDrvBits
800062ba EngDrvOuts
80006298 shamem_read
800063c2 StartEDmac
80006498 edmac_set_addr
800064c2 edmac_set_size
80006bfe ConnectWriteEDmac
80006c10 ConnectReadEDmac
80006c42 RegisterEDmacCompleteCBR

Alex's notes say edmac_set_addr() and edmac_set_size() together act like SetEDmac, I haven't checked that.  If true, we can wrap these together to provide that function.

As is, we cannot call these functions from ICU.  The 0x8000_XXXX address space does not contain the same content if you read it from ICU vs Omar.

Heder - are any other stubs required to attempt reading from e.g. the sensor?  To be safe, we need to find unused channels - anything else?

After further testing on real cam to confirm I understand it, here's an illustrative copy example.

Code Select Expand


    // Region size 128kB == 0x20000, we can factor to:
    // 0x4 * 0x10 grid, each tile 0x80 * 0x10;
    // 0x4 * 0x10 * 0x80 * 0x10 == 0x20000
    struct edmac_info region = {
        .off1a = 0,
        .off1b = 0,
        .off2a = 0,
        .off2b = 0,
        .off3 = 0,
        .xa = 0x80, // "regular" col width
        .xb = 0x80, // "irregular" col width
        .ya = 0xf, // implies 0x10 high
        .yb = 0xf, // 0x10 high
        .xn = 0xf, // 0xf regular cols + 1 xb col
        .yn = 0x3 // 3 normal rows + 1 yb row
    };


It might seem more obvious to not use xb or yb at all; just have 10 normal columns and 4 normal rows.  I agree, but the hardware doesn't.

If you have xb == 0, your copy will silently fail.

yb is more tricky.  In general, ya and yb are treated as holding a number 1 higher than their values, but if yn is 0, ya is not used. Yb is always used, meaning if you keep it 0, you get a single row added to the end of your copy, of length (xa * xn) * (xb).

[Some examples:]

I am exploring EDMAC on modern cams.  I found this thread useful, but hard to understand.   Here's my attempt for a useful diagram:



xa and ya specify "regular" tiles, which can be repeated, with the number of repeats controlled by xn and yn.

xb and yb specify "irregular" / "remainder" tiles, only appearing in one column or row, or none if the respective xb or yb is 0.

Memory is linear, accessed left-to-right, top-to-bottom.

I would assume this design was chosen as it allows specifying regions of arbitrary size, while also allowing most of the transfers to occur using tiles of a convenient width for the underlying copy mechanism (which is presumably aligned to some power of two).

Note that I'm ignoring the offsets.  You don't need them if you're copying a contiguous region of memory, and they complicate the diagram significantly.  Check the patent for details, should you need this.  In broad terms, you can specify an offset between any of the rows or columns, and that offset can be positive (you are now copying a grid of disconnected regions), or negative (you are copying a grid of partially overlapping regions).

Because you can provide a different edmac_info struct for the dst and src, this allows some kinds of transforms to happen very efficiently (likely bottle-necked only by ram speed).  The example given in the patent is for handling data from a unit that produces RGB data as three separate channels (each at a different memory location), which can be combined together using such a copy operation.  For the source, specify 3 rows or columns, with offsets meaning you're reading from each channel, for the destination, choose a different geometry.

Further complicating things, there are various restrictions on what edmac_info configs are valid.  I don't know all of these.  Alex implies some in his comments, but also talks about some configs that aren't valid on my test cam, a 200D.

Some examples:
You can specify a single rectangle with only xb and yb, and if so, xa and ya can be 0.  But yb must be at least 1, and this implies 2 rows: there's an implicit first row, so 0 => 1, but this doesn't work if xb is the only non-zero value.
If you have xa and xn non-zero, but everything else zero - your copy will silently fail.
If you have xa, ya, xn, yn all non-zero, but everything else zero - your copy will silently fail.
If you have xa, xn, xb non-zero, all else zero, this works!
Possibly this means you must have a non-zero xb.

There are limits on the maximums for each variable, which can be quite low, perhaps somewhere around 4000.  But it depends on which variable, and I haven't checked this thoroughly.  xb works up to at least 65536.  But in essence, each tile should be somewhat small.

I played around with this and now understand it more.  I haven't been able to do direct comparison between EDMAC on modern digic and fast dma_memcpy(), for complicated reasons.

Heder isn't using dma_memcpy() the *function*, he's using the underlying physical DMA controller in a similar manner to how the code tries to do it, but via direct access.  I suspect the reason you're doing that is because dma_memcpy() doesn't work on 7D2?  At least, that's what I find: it always fails for me, taking the "Nothing DMA ch now!" path, which I think means it fails to find a free DMA channel, possibly because they haven't been initialised.

7D2 has EDMAC, and it lives on Omar core.  https://www.magiclantern.fm/forum/index.php?topic=26249.msg245605#msg245605

If we want to use EDMAC resources that are sometimes used by *any* DryOS instance, we may have to go through Omar, or we're going to get into trouble with races / contention of physical resources.  If we can find channels / ports that are never used by DryOS then we can probably do stuff direct from ICU.  Or if there's some signalling method, RPC or interrupts etc to synchronise across cores.

D7 has a very different dma_memcpy() to older gens.  The functionality is the same, but the code is doing things in a very different way, no use of digic registers are easily found.  Might exist deeper into the code, I haven't spent a lot of time on it (EDMAC works for memory copies here).

Given that dma_memcpy() looks deliberately not working on 7D2 ICU, but does work on Digic 7, maybe this functionality is supposed to happen via Omar on D6?  It's not clear to me.

7D2 has the same EDMAC code for mem_to_mem_edmac_copy(), but it lives on the Omar core.  Specifically, at 0x1b0e150.  I don't currently know how to run code on Omar, so I haven't tested this in the same way as 200D.  Possibly, since they're both ARM cores, I can just run the code from ICU...  but I'm not sure how memory mapping interacts here.  The code tries to get kernel locks at various points, and Omar is running a different instance of DryOS.  Feels like avoiding races on the locks is a pain, as well as avoiding contention accessing the physical resources - unless there's some RPC layer to synchronise the different OS instances.

Have you measured it, or are you guessing?  I haven't measured, but I'd guess after being read, each file is cached, and no writes need to happen to the card.  If so, the card reads won't be a bottleneck, and it'll be the same speed either way.  If the output file is written to the card, that would suck, but I assume that's optional.

Why should it be slower than copying *the same files* off the cards, and then processing?  The same data needs to be read from the same place.

You mean transcoding straight from cards to the drive?

Sure, why not?

Fastest solution would be creating a location with symlinks and then transcode from there.

Why is this faster than having MLVApp work with the files direct from the drives?  It's the same thing but the user doesn't have to do anything.

Actually already exists some scripted simple workflows around this.

Yes, but why can't MLVApp do this automatically?  We know it can't *now*, but that's why it's a feature request.

Edit: It is faster copying MLV content first to a ssd drive before transcoding though.

Is it?  Why?  Either way it has to read all the files from the card.

We chatted about this on discord, and since it was a feature request for MLVApp, suggested it get filed on forums.

The problem here is that huyct666 has card spanning enabled, and MLVApp can't process the files without them first being moved into a single dir.  That means copying all files to a single location, which is slow.  It would be a clear improvement if that step could be avoided.

Possibly this could be achieved by having MLVApp support an optional second dir (during import?  I don't use it), or auto-detecting the files are card-spanned (I don't know if this is possible). 

Great, thanks a lot, good write up!  So this is an improvement on an old system.  Sounds useful and will be interesting to see how it interacts with EDMAC - can we use both at once, do we hit ram limits, etc?

Doesn't look like we use dma_memcpy() much, but there are a few places.  I'll see if I can improve mem benchmark module with this change, and test on a few cams.

Interesting and seems good progress!

I've had the dma_memcpy stub found for 200D for a long time but the copy speed isn't very high.  It would really help if you could link to the commit where you make these changes (what am I supposed to do with the DMA_BASE info?  How do I use this?).  Given the clues, maybe this is CONFIG_EDMAC_MEMCPY related?  I've avoided that code so far because it wants a lot of EDMAC stubs found.  I've never worked with D45 cams, so I don't know how EDMAC is supposed to work when doing a port.  The more you can share about *how* you're finding things, the better chances I'll have to understand how to do similar on other cams.

7D2 has some of the same strings I'm finding useful in 200D, but not all of them.  A large difference is that they are mostly in the Omar code.  Omar is another ARM core, which the ICU initialises by copying part of the ROM over.  It's Thumb capable.  I suspect ShtVfx processing is all done on Omar.  So you may need to look there for EDMAC usage.  I'm fairly sure the EDMAC hardware has the same interface as D45.  This has been discussed on Discord before, you should find recent references if you search for Omar.

E.g. see "ShtVfxMultiShotPathState" - this exists in 200D main rom, doesn't exist in 7D2 main rom, but is in Omar instead.  E.g. e008f84a() on 200D is 01af4ae8() on 7D2 Omar.  Most of the MultiShot functions seem to use EDMAC.

Yup, this is the same on 200D.  On that cam, there's an array starting at 0x66264, where each element is something like: {uint32_t channel_addr, uint32_t ModeInfo}.

I think it's the same EDMAC device, or at least the API is very similar, mapped to a different address.  There's a partial write up in my post here:
https://www.magiclantern.fm/forum/index.php?topic=26249.msg245565#msg245565

In Kitor's dumps of the tables from various cams you can see the 0xd0004000 range is used on various D7 and D8 cams.

Please note most EDMAC related functions now take a struct, which holds two channels, rather than two functions being used, each taking one channel.  This means (I think!) that there are no direct equivalents to the EDMAC stubs for D45 cams.  One such struct is roughly described here:
https://github.com/reticulatedpines/magiclantern_simplified/commit/7a67edc5011d27c7080c6c940385328ab3aae82e#diff-9450d43dd0c51c7b781f01100a2d37e9ef1903d6a16fcf51be40b03f3d8aed38R693

Comparing 650d to 200d, I see for example that c0f2b000 is used in the same way d002b000 is for 200d.  So quite probably just the base address of the device has changed.  I don't have experience working with EDMAC on older cams.  Do you know of a good writeup anywhere?  I'm finding lots of similarities, but I don't have a good idea of what to do with them.

I have EDMAC working on 200D.  Currently, I am limited to mem->mem copies.  However, this is a significant step towards general control of EDMAC on modern cams, a requirement for using most of the cool hw accelerators, as well as raw video.

Code Select Expand


  2183:  6795.981 Attempting EDMAC mem->mem copy
  2288:  6815.774 Pre-copy, *dst, *src: 0x0, 0xa5a5a5a5
  2292:  6821.866 Post-copy, *dst, *src: 0xa5a5a5a5, 0xa5a5a5a5
  2293:  6821.877 ms time for size 0x200000: 6
  2356:  7916.659 dst / src content equal :)


My understanding is not yet complete, but my work proves:
- EDMAC unit exists on Digic 7.
- EDMAC unit can be used by ML!
- the same core struct, edmac_info, is used on this unit.  It's either the same, or has similar/same interface.  We know this is a Canon unit due to a patent matching strings in the rom.
- channel numbers used for this op are 0x11 and 0x2b.  The same numbers are (or can be) used for the same functions on D4 and D5.

Mem->mem transfer speed is at least 550MB/s, so we shouldn't be bottlenecked on ram.

Not intended as a "real" commit, so it will disappear at some point, but this shows how the transfer is done, with a bunch of descriptive comments in debug.c:
https://github.com/reticulatedpines/magiclantern_simplified/commit/7a67edc5011d27c7080c6c940385328ab3aae82e
(when it does disappear it should mean better, cleaner EDMAC code has hit dev branch, so please ping me to get the link updated)

I'll write this up more cleanly over the coming days.

ML checks the existing firmware version and will refuse to run on a non-matching version: you must use 1.1.1.

Regrettably there was no conspiracy to hide crop_rec on 5D2 from you 

It appears to be compatible with the 5d2 since it mentions the 5d2 in its code.

This code won't do anything on 5D2.  See the function crop_rec_init().  There is no camera specific code for 5D2.

Always bear in mind, that XF605 is a camcorder and not a target DSLR/DSLM of ML!

I did a blind port of ML to XF605: https://github.com/reticulatedpines/magiclantern_simplified/tree/xf605

This is untested on phys cam and caution should be used.  The code around the bootloader has been tested in qemu and works for me...  later code cannot be tested in qemu because it doesn't emul far enough.  If you test it, please let me know how it goes

From experience with other cams, I would expect this to be safe to run on phys cam.  It might crash, but it is probably safe.  I put some uart_printf() logging in init.c inside CONFIG_XF605 guards, which will give some idea of how far through the boot process it gets.

In general, this feels like a standard Digic X cam, until you get quite high up in the GUI layers.  I would guess the windowing system is the same, but what is drawn with it is different.  Makes sense, since it has a video oriented GUI.  It does have some LiveView strings though, so I'd guess the code has DSLR heritage.  I saw no fundamental reasons why ML wouldn't run fine on this cam.  I'm not sure the standard ML menus would be very useful, so we might want to split that up, menu-dv.c or something.  But all the code we have for drawing menus, storing settings, interacting with tasks etc would still be useful.

Some notes:

Lens info struct is definitely not correct.  I made it the right size, but don't know the names or offsets of fields.

DryOS asserts have two potential handlers on this cam!  One at 0x4004, one at 0x4000.
ML hijacks one handler, assuming there is only one.  I haven't investigated what the second handler does (GUI handler maybe?)

I couldn't find CancelDateTimer() so I hacked this out in source for XF605 only.  Would be nice to find it.

All the button codes in platform/XF605.101/gui.h are junk, so you won't be able to enter ML menus.  The easiest way to find these is log them from uart, see e.g. here in menu.c:

6258 // SJE useful for logging buttons
6259 //    DryosDebugMsg(0, 15, "event->param 0x%x", event->param);

Last note, not relevant to the port, just something interesting I found, XF605 (fw 1.0.1) appears to have a GUI service mode, see "WINSYS_CreateDialogBox (StartServiceModeMainApp)", 1eac6046(), 1eac5de4().

It's likely triggered by an event:

    if (param_2 == 0x10000052) {
      DryosDebugMsg(0x82,3,"IDLEHandler(Common) : PRESS_SERVICE_ON",0x1c);
      FUN_1eac6046();
      return 0;
    }

I don't know the buttons on this cam, possibly there's one labelled Service?  If not, this is probably a synthetic event.  You could create it via code, might be interesting.

Good to hear it's pretty stable for you.

SRM works differently (and badly) on Digic 7, where I have spent most time.  Haven't looked at it at all on 6.  Consequently I don't know much about this system.

I'd have to see the code to guess at a cause for the "don't click me" behaviour.  Fork my repo and push a branch?

Oh yeah, I remember that now - this could be forced by not letting the AF lock, if I'm thinking of the same thing.

I guess it depends how you want to debug   For that cam I don't have a cable made up, so I mostly used Qemu or bmp_printf().  Not ideal.

For intervalometer, that makes sense, I misunderstood the context.  Yes, you'll have to find new stuff for enabling features.

Are you finding it stable so far?  I haven't tested much at all with 7D2, just did enough work to get barebones GUI working and benchmarks.

Looking forward to a "new" feature   Nice to have someone else joining in on these new cams - and hopefully intervalometer will be an easy port to other models.

It should be possible - but you're replying to a thread from 2017 so most likely you won't get a reply from anyone that knows the code.

The Lua API has some docs, here: https://builds.magiclantern.fm/lua_api/

I think you can get it to make a noise (dependent on camera support).