Messages

Several years of learning ML code, trying to understand fragmentary docs, and in some cases writing the docs myself, 200D has raw video:

Bugs / quirks / limitations:
- only 14 bit is tested
- lossless modes are listed and definitely not supported, these should be hidden (this is a problem with mlv_lite code)
- mlv_lite has several hard-coded defaults that don't make sense for 200D (I think we need to make defaults detect cam model and choose behaviour intelligently)
- it records at 19.051 fps.  I have a few guesses as to why and will fix this
- 1280 * 720 is continuous, higher resolutions are not (200D SD interface is limited to about 40MB/s)
- MLV metadata is wrong in some places (bunch of stuff filled in as 0 or MAX_INT, it's a division by zero bug in ML code)

Code is available: https://github.com/reticulatedpines/magiclantern_simplified/compare/dev...200d_raw_draft
That's the changes compared to main branch; changes are fairly small!  This is a good place to look if you're interested in porting raw video to other Digic 7 models.

No direct links to builds yet, but if you ask in discord and have a good reason for using something that certainly hasn't been tested well, I might hand one over.  Otherwise you can wait till I've cleaned it up.

More cameras are in the works...

Nope, because I had no idea I should look there.  This code is really poorly organised.  There's different sensor size values in at least three different files, and they all interact.  Feels like a simpler system must be possible.  Why are some per cam values in platform/XXD, where you can find them, some are in src/random_file.c, where you can't, and some are in modules/random/random.c?  Ugh.

Thank you for coming back and teaching me this impossible ancient knowledge

The per cam stuff in raw.c should probably be moved into platform, right?

It's a code problem in mlv, not sure where yet.  It's producing an output file that claims to be 1920 * 1080, but contains data that is 2096 * 1164 (the correct size of the image).  mlvdump, and mlvapp, seem to be trusting the 1920 * 1080 fields, but filling the display from the 2096 wide data, so everything is the wrong colours and skewed.  I can see both values inside the file:


https://i.imgur.com/XR89GXZ.png

Time to learn mlv format and fix the code I guess.

If you never used IRC (or never knew that ML devs used IRC!), then you definitely don't need to care about Discord.  It is better for quick things, so feel free to use it if you just want to ask a question and get an answer (it's like WhatsApp or similar in this regard).  Can also do screensharing / voice chat, sometimes useful.

I like the 7D2.  Good build quality, metal body, even has weather-sealing.  It feels like an APS-C sized 5D3.  It's faster with card spanning than 5D3.  I'd recommend it if you see one at a good price

5D2 plans also make sense to investigate.  I'd especially appreciate somebody with more experience working with the hardware, who could try to verify my theory that the reslock structures are a linked list holding (block num, device ID) pairs.  If I'm right, it means the way ML locks devices is wrong.  So if some cams don't work the way they "should", that could be the cause.  And the read_edmacs[] / write_edmacs[] can be simplified quite a bit.  We might also be able to determine what individual devices are for - the current way we do this means we're using the wrong numbers!

Interesting HDMI strings   Good luck!

Separating out code / repo stuff.

Any help or advice would appreciated , i'm not big Linux user mainly PC(windows7 some windows 10 if i have to)
and some mac. its been about 3 years since i did any development so just need some refreshing 

The repo I maintain should work on old cams, and has many branches merged.  It contains crop_rec, lua_fix and other branches: https://github.com/reticulatedpines/magiclantern_simplified/commits

I mention this because some of the work I've done means this repo is much easier to build on modern systems.  You don't need to find gcc 4.8.  You can use any gcc from 8 to 13.  Ubuntu is an okay distro to use, Debian is better (this makes some things easier with qemu testing).  It compiles cleanly, with no warnings and no errors.  If any compilation warnings are introduced, I fix them.

Unlike the heptapod repo, I want to keep the number of active branches very low: only "dev".  All other branches are temporary, with the intention they will be deleted or merged into dev.  I found using heptapod very confusing - hard to find what you wanted.  Hard to know if a given branch would build, or what features it had.

I've also done similar work with qemu code, making this easier to build, and easier to run.  No patching things, no needing weird old versions of tools.

If you want to try this repo, I will happily talk you through how to get it working.  It is likely to become the official repo.  Nobody with access to the current official repo is active anymore, and making a working dev environment with it is difficult.

[5D2]

After seeing considerable work on D6 (7d mrkii heder)  &  D7& D8(names_are_hard)   
Its inspired me to finish some unfinished magic lantern business on the 5D2 & 50D
I find that i have some free now (6-8 weeks , after that Drag racing Season(ProMod's) start around Mid May (canada)
being i build race car chassis & parts (I'm in the metal fabricator business) i need to be at the track & late nights in the shop 

Cool, would be good to have you back

That all for now , i hope all development continues on the forum for everyone to see publicly , i don't like when the ML discussion is behind closed doors

Keeping devs involved is the most important thing for me, so I'd like to understand this more.  I'm not aware of anything being closed up, apart from the way Alex used to work.  Is there anything in particular that you don't like?  Maybe we can imake things better.

I would say the dev work has been more public since Alex left than before.  Alex did a lot of work, but he didn't like sharing things until they were finished.  We had to relearn lots of things about D678 cams because it wasn't documented anywhere.  E.g. the M50 "fishy" build - the source for this was never published, or even shared with other devs.  This was a largely working Digic 8 build, I could have learned a lot from it.  Instead I had to work it all out myself.

See updates from me since at least 2020, here:
https://www.magiclantern.fm/forum/index.php?topic=19737.350
Kitor on edmac, here:
https://www.magiclantern.fm/forum/index.php?topic=26249.0

Summaries of work here and here:
https://www.magiclantern.fm/forum/index.php?topic=26814.0
https://www.magiclantern.fm/forum/index.php?topic=26852.0

Also see a large number of commits over the last few years, all open   https://github.com/reticulatedpines/magiclantern_simplified/commits

Discord is just as open as the forums, anyone can join.  If I posted all my small findings to the forums it would be useless spam.  Instead, I post summaries, with links to interesting commits.  We treat Discord as useful for quick discussion, but useless as an archive, so we document things properly on forum and in the repo.

PS I did lots of work on D6 too   Heder started from a working ML build for 7D2, because I did that port.  750D and 5D4, too.  There's been lots of background work getting ML booting on many new cams!

I'm cleaning the old hack repo and making a new clean one. Updating to the newest mlv_lite and creating a version with direct buffering so all can use this if they need it. Also inforcing that both ICU(Marius) and Omar cores uses shamem, should make fps override easy to use. Next milestone will be around 7th of Marts. The biggest issue atm is missing edmac api, as all the functions are located on Omar, but I know for a fact that ICU(Marius) does use edmac channels (firmware string "mossy") so it's possible to use edmac from ICU(Marius). Also I will look into card spamming on mlv_lite as this could enable 3k@14bit recording ~ 195 MiB/s ..

Progress sounds good!

I have 200D working with modern mlv_lite, with normal buffering.  So you may want to do that part last, as I am still intending to port your 7D2 code in a similar way.  When I have 200D working cleanly with nice code, porting 7D2 should be easy enough.  Unfortunately, I still have some last problem to work out; the video data captured from 200D is scrambled when mlv_lite is used.  I think perhaps some bad offset or width calculation - if I manually capture the buffer the data is correct.  There are lots of hard-coded constants, it's been quite tedious to debug things

I'm not forcing you to use discord but it would make things easier to coordinate   You don't have to install anything, it works from a browser.

Not very surprising, but worth documenting: if you read the ram address associated with an edmac mmio, when the transfer is in progress this address updates during the transfer.  When it's finished, it remains at the last address.  You can therefore monitor the percentage progress if you want.

I'm currently trying to work out how to determine if a channel is active or not.  dma_state field is not enough.  From what I've logged so far, it would make sense if there's a way to lock channels, but I haven't found a mechanism.

Some edmac related functions on R.  Not all strictly edmac related, but often found in that kind of code.

RegisterEDmacCompleteCBR: e0535a82
edmac_set_addr: e05364b6  (probably...  wants checking)
_engio_write: e06b92b4
wait_for_any_event_flags: e053a3bc
wait_for_all_event_flags: e053a43c
unset_event_flag_bits: e053a5e6
register_interrupt: e0554504
edmac_set_dma_flags: e05369b8
_edmac_set_dma_flags: e06a37a8
edmac_pwrmng_sleep_devices_maybe: e0671770
edmac_pwrmng_wake_devices_maybe: e0671742

_engio_write() is probably at 13ea0.  Untested.  Testing it is potentially risky, I assume you already know that, heder

This is related to engio_write()!  I didn't recognise the code because we've never found that function for digic 7.  I might be misinterpreting whether it's writing into the array, or writing through pointers stored in the array - the (addr, val) pairs are very similar to engio_write() buffers used on 5D3 (and I assume, other cams).  I need to look at this in more depth.

_engio_write() itself is probably at e050d6dc.

As noted here, I have some useful edmac / raw progress on 200D: https://www.magiclantern.fm/forum/index.php?topic=13746.msg245918#msg245918

I've now got mlv_lite meeting all deps for 200D (this doesn't mean much, just that all required symbols are exported - some of these I am faking for now).  Trying to run it causes problems, multiple in fact.  I've fixed some serious bugs in the module, and surrounding code (that in theory affect all cams, but may not be possible to hit in practice).  We have very bad discipline around checking for null pointers before use, which on D7 and up causes an exception, whereas D456 you can get away with it.  Canon code has also been changed due to this, they have added many more checks   One of the problems is the FIO_* functions often take char *filename, and Canon code checks this now, and asserts.  ML can sometimes pass in null pointers.

One oddity is worth noting.  Some MMIO addresses have changed from Digic 5, we've known this for a while.  For D5 0xc0f06014 is FPS_REGISTER_B (which is hard-coded into the source, a painful decision that I must fix).  On 7D2, this has moved to 0xd0006014.  But on 200D, while the *value* 0xd0006014 is used, it's not used as an address.  Instead, there's an array of (addr, val) pairs, 0xd0006014 is one of the addresses, and the code has some utility functions to search the array, and either update the val from a global (obtained via a function I am calling get_fps_register_b_val()), or return the val.  See e.g. e032b54a().

It's quite weird.  I suppose it might be a more space efficient way to do shamem_read(), if the region was sparsely populated?  This complicates things in at least two ways: there's no fixed offset for FPS_REGISTER_B, I may have to refactor all accesses into a function call.  Second, on 200D we can't assume 0xd00X_YYYY constants are MMIO access, which may break many assumptions in ML code.

Thanks, that does help me understand your opinion, but it's just that, an opinion.  It's fine if you prefer 3k crop.  Claiming the 5k mode is "a bit above 1080p" is still untrue.

Why would anyone think a value  1664 is bigger than 2500 ?

Why indeed, when it's the first time you've mentioned this number, and it's not relevant?

zenny actual true res of 5k mode in canon m is lower than cropped 2.5k mode, its a bit above 1080p res

This is not true.  Possibly it might be true if you explained what you meant.

Cleanly adapting modern cams to the existing code will require significant changes.

An example from edmac.c (there are many examples I could use!):

Code Select Expand


119 uint32_t edmac_get_base(uint32_t channel)
120 {
121     if (channel >= NUM_EDMAC_CHANNELS)
122     {
123         return -1;
124     }
125
126     uint32_t bases[] = { 0xC0F04000, 0xC0F26000, 0xC0F30000 };
127     uint32_t edmac_block = channel >> 4;
128     uint32_t edmac_num = channel & 0x0F;
129
130     return bases[edmac_block] + (edmac_num << 8);
131 }
132
133 static uint32_t edmac_get_block(uint32_t reg)
134 {
135     switch (reg & 0xFFFFF000)
136     {
137         case 0xC0F04000: return 0;
138         case 0xC0F26000: return 1;
139         case 0xC0F30000: return 2;
140         default: return 0xFFFFFFFF;
141     }
142 }


This hardcodes the same magic addresses twice, in different places.  The addresses don't hold for new cams.  And new cams use 4 regions, not 3.  Edmac code doesn't use structs, but uses raw offsets to base addresses, with no explanations.  It's really ugly code, with no docs or explanations in the code (some on the wiki).

Changing the old code needs to be done carefully, or it will fail on old cams.  Changing the old code must be done, or it can't work on new cams.

I could hack in a quick version for 200d (or 7d2), but I'd greatly prefer taking the time to fix things so any cam can use the same code, probably with some new platform/XXD constants defined.

Probably not important - I suspect heder did what I did for this first test: handheld it and dumped data direct to disk with FIO_WriteFile().  Mine all look the same, it's not fast enough to avoid tearing.  Heder's real code uses a faster method, I assume no problem there.

Fixed.  I can scrape 14-bit packed raw data.  I had some bug in the python I was using to unpack to 16-bit.  Still not sure what it is, rewrote it in C.

https://i.imgur.com/LklrBm3.png

I'll now try and get 7D2 and 200D working with modern mlv_lite.

Congratulations on your horrible ancient distro

I see.  Those repos are old (or, for Danne's, based on old code) and don't expect to use python3 at all.  That's why they're not using the rst2html you have installed.  Almost certainly the linux you're using is much too modern for them to build successfully.  There will be other problems, I couldn't recommend that combination at all, you might encounter serious bugs.

Obvious choices:
- don't use these old repos
- downgrade your linux to about 2016 era, that may build it
- use my repo, it builds on modern systems (but is largely untested on your cam and may not have features you want)

Looks sensible enough then.  Which repo is it you're trying to build?

Okay, what's this give you?

Code Select Expand


head /usr/bin/rst2html


I don't know.  Is rst2html in your path?  You will see that error if it isn't.

What does this show you?

Code Select Expand

which rst2html

Thanks, I haven't monitored that field changing, worth a try.  I'm very likely going to refactor the code to stop using those ugly consts for edmac MMIO.  The struct looks like this:

Code Select Expand


413 struct edmac_mmio
414 {
415     uint32_t dma_state;
416     uint32_t dma_flags;
417     uint32_t ram_dst_offset;
418     uint32_t yn_xn;
419     uint32_t yb_xb;
420     uint32_t ya_xa;
421     uint32_t off1b;
422     uint32_t off2b;
423     uint32_t off1a;
424     uint32_t off2a;
425     uint32_t off3;
426     uint32_t unk_01;
427     uint32_t irq_reason;
428     uint32_t irq_related;
429     uint32_t unk_02;
430     uint32_t unk_03;
431     uint32_t fencing_related_maybe;
432     uint32_t unk_04[0x2f]; // padding?  Probably some but not all.
433 };
434 SIZE_CHECK_STRUCT(edmac_mmio, 0x100);


You can then cast the MMIO "channel" addresses.  Much nicer checking mmio->yb_xb.

I may have found the buffer used by lv_save_raw via static analysis (there's a pointer global, get's populated in LV).  I've dumped image data from it and it's clearly real, but I don't understand the format.  It might be packed 14-bit bayer, but if so I'm missing some detail required to decode it to a sane image, colours are all wrong.

Could you share a dump of that mem region, for a known good capture?

You are missing rst2html from your path.  If you're using Debian, this will install it:

Code Select Expand


apt install python3-docutils


This is explained a few posts up in this thread.  No, you must not create it manually.

Dont mind vsync, it's not critical for now.

Good to know.

1. Does "lv_save_raw" exist as a string in ghidra ?. It might also have changed name, like on the 7D2 I cant find any more "fdump" its now called "fdumpall" or "oldfdumpall", search for "save_raw" and investigate those places.

Yes, the same function exists, I've called it.  I've had it marked in my 200D project for over a year.  It doesn't do what you describe (on this cam).  As in my previous post: running call("lv_save_raw") does not trigger any new usage of channels or devices that I can see, and there is no bayer data saved to in-use edmac buffers.

That's why I'm asking questions about how you see edmac usage change when *you* call it.  So I can try to find the buffer *without* looking for the bayer data directly, since that doesn't happen here (or, perhaps the buffer is reused quickly and my logging isn't fast enough, etc).