Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - names_are_hard

#1
Found a quite interesting shell command, EngMirrorDump.  This takes an integer, and returns info on a shamem region.  This includes the name of the subsystem (engine), and address offset in shamem.  It also dumps content to card.  Results from 200D.  The full list is long, but these at least are interesting and show it's useful to us:

EngMirrorDump 3   (CHANNEL) Addr:4000,Size:1000,Flag:0
EngMirrorDump 29  (CHANNEL) Addr:26000,Size:1000,Flag:0
EngMirrorDump 37  (CHANNEL) Addr:30000,Size:1000,Flag:0
EngMirrorDump 65  (CHANNEL) Addr:57000,Size:1000,Flag:0
EngMirrorDump 66  (CHANNEL) Addr:58000,Size:1000,Flag:0

These are the EDMAC channel ranges.

Full list:
EngMirrorDump 0   (PWRCNT/SETTER) Addr:1000,Size:1000,Flag:0
EngMirrorDump 1   (KAISER) Addr:2000,Size:1000,Flag:0
EngMirrorDump 2   (SHAREMEM) Addr:3000,Size:1000,Flag:0
EngMirrorDump 3   (CHANNEL) Addr:4000,Size:1000,Flag:0
EngMirrorDump 4   (CHNSW) Addr:5000,Size:1000,Flag:0
EngMirrorDump 5   (HEAD) Addr:6000,Size:1000,Flag:0
EngMirrorDump 6   (HEAD) Addr:7000,Size:1000,Flag:0
EngMirrorDump 7   (PEPPER) Addr:8000,Size:1000,Flag:0
EngMirrorDump 8   (FAUST) Addr:9000,Size:1000,Flag:0
EngMirrorDump 9   (WOMBAT) Addr:a000,Size:1000,Flag:0
EngMirrorDump 10  (WOMBAT) Addr:b000,Size:1000,Flag:0
EngMirrorDump 11  (WOMBAT) Addr:c000,Size:1000,Flag:0
EngMirrorDump 12  (AFFINE) Addr:10000,Size:1000,Flag:0
EngMirrorDump 13  (FIXER) Addr:11000,Size:1000,Flag:0
EngMirrorDump 14  (TAIWAN) Addr:12000,Size:1000,Flag:0
EngMirrorDump 15  (RABBIT) Addr:13000,Size:1000,Flag:0
EngMirrorDump 16  (POSTER/SARIDON) Addr:15000,Size:1000,Flag:0
EngMirrorDump 17  (HISTORY) Addr:16000,Size:1000,Flag:0
EngMirrorDump 18  (PCFG) Addr:18000,Size:1000,Flag:0
EngMirrorDump 19  (WOMBAT) Addr:19000,Size:1000,Flag:0
EngMirrorDump 20  (LUCKY) Addr:1a000,Size:1000,Flag:0
EngMirrorDump 21  (DANCING) Addr:1b000,Size:1000,Flag:0
EngMirrorDump 22  (DANCING) Addr:1c000,Size:1000,Flag:0
EngMirrorDump 23  (PONY) Addr:1d000,Size:1000,Flag:0
EngMirrorDump 24  (LOTUS) Addr:1e000,Size:1000,Flag:0
EngMirrorDump 25  (LUCKY) Addr:20000,Size:1000,Flag:0
EngMirrorDump 26  (LUCKY) Addr:22000,Size:1000,Flag:0
EngMirrorDump 27  (CPUIF) Addr:24000,Size:1000,Flag:0
EngMirrorDump 28  (LUCKY) Addr:25000,Size:1000,Flag:0
EngMirrorDump 29  (CHANNEL) Addr:26000,Size:1000,Flag:0
EngMirrorDump 30  (DISTER) Addr:28000,Size:1000,Flag:0
EngMirrorDump 31  (SARIDON2) Addr:29000,Size:1000,Flag:0
EngMirrorDump 32  (QUARK) Addr:2a000,Size:1000,Flag:0
EngMirrorDump 33  (SUMMA) Addr:2b000,Size:1000,Flag:0
EngMirrorDump 34  (SMAP) Addr:2d000,Size:1000,Flag:0
EngMirrorDump 35  (OHYEAR) Addr:2e000,Size:1000,Flag:0
EngMirrorDump 36  (MICROU) Addr:2f000,Size:1000,Flag:0
EngMirrorDump 37  (CHANNEL) Addr:30000,Size:1000,Flag:0
EngMirrorDump 38  (MOSSY) Addr:34000,Size:1000,Flag:0
EngMirrorDump 39  (PEPPER) Addr:37000,Size:1000,Flag:0
EngMirrorDump 40  (SHREK) Addr:38000,Size:1000,Flag:0
EngMirrorDump 41  (SUSAN_A) Addr:39000,Size:1000,Flag:0
EngMirrorDump 42  (SUSAN_B) Addr:3a000,Size:1000,Flag:0
EngMirrorDump 43  (LUCKY) Addr:3c000,Size:1000,Flag:0
EngMirrorDump 44  (LUCKY) Addr:3e000,Size:1000,Flag:0
EngMirrorDump 45  (LUCKY) Addr:40000,Size:1000,Flag:0
EngMirrorDump 46  (LTKIDS) Addr:42000,Size:1000,Flag:0
EngMirrorDump 47  (SUSAN_A) Addr:43000,Size:1000,Flag:0
EngMirrorDump 48  (PEPPER) Addr:45000,Size:1000,Flag:0
EngMirrorDump 49  (BAUST) Addr:46000,Size:1000,Flag:0
EngMirrorDump 50  (HAIZEN) Addr:47000,Size:1000,Flag:0
EngMirrorDump 51  (HISTORY2) Addr:48000,Size:1000,Flag:0
EngMirrorDump 52  (HISTORY2) Addr:49000,Size:1000,Flag:0
EngMirrorDump 53  (HISTORY2) Addr:4a000,Size:1000,Flag:0
EngMirrorDump 54  (HISTORY2) Addr:4b000,Size:1000,Flag:0
EngMirrorDump 55  (CUMULO) Addr:4c000,Size:1000,Flag:0
EngMirrorDump 56  (CAPTAIN) Addr:4d000,Size:1000,Flag:0
EngMirrorDump 57  (CDM) Addr:4e000,Size:1000,Flag:0
EngMirrorDump 58  (WEAVER) Addr:4f000,Size:1000,Flag:0
EngMirrorDump 59  (WEAVER) Addr:50000,Size:1000,Flag:0
EngMirrorDump 60  (VERSARCH) Addr:51000,Size:1000,Flag:0
EngMirrorDump 61  (OPTIMUS) Addr:52000,Size:1000,Flag:0
EngMirrorDump 62  (ELISION) Addr:53000,Size:1000,Flag:0
EngMirrorDump 63  (COMPASS) Addr:54000,Size:1000,Flag:0
EngMirrorDump 64  (PURE) Addr:55000,Size:1000,Flag:0
EngMirrorDump 65  (CHANNEL) Addr:57000,Size:1000,Flag:0
EngMirrorDump 66  (CHANNEL) Addr:58000,Size:1000,Flag:0
EngMirrorDump 67  (ECHIZEN) Addr:59000,Size:1000,Flag:0
EngMirrorDump 68  (ECHIZEN) Addr:5a000,Size:1000,Flag:0
EngMirrorDump 69  (ECHIZEN) Addr:5b000,Size:1000,Flag:0
EngMirrorDump 70  (ECHIGO) Addr:5c000,Size:1000,Flag:0
EngMirrorDump 71  (ECHIGO) Addr:5d000,Size:1000,Flag:0
EngMirrorDump 72  (ECHIGO) Addr:5e000,Size:1000,Flag:0
EngMirrorDump 73  (POSTER) Addr:5f000,Size:1000,Flag:0
EngMirrorDump 74  (BIZEN) Addr:60000,Size:1000,Flag:0
EngMirrorDump 75  (BIZEN) Addr:61000,Size:1000,Flag:0
EngMirrorDump 76  (BIZEN) Addr:62000,Size:1000,Flag:0
EngMirrorDump 77  (BINGO) Addr:63000,Size:1000,Flag:0
EngMirrorDump 78  (BINGO) Addr:64000,Size:1000,Flag:0
EngMirrorDump 79  (BINGO) Addr:65000,Size:1000,Flag:0
EngMirrorDump 80  (COBALT) Addr:66000,Size:1000,Flag:0
EngMirrorDump 81  (COBALT_2) Addr:67000,Size:1000,Flag:0
EngMirrorDump 82  (CUMULO_2) Addr:68000,Size:1000,Flag:0
EngMirrorDump 83  (DAFIGARO) Addr:69000,Size:1000,Flag:0
EngMirrorDump 84  (ELISION_2) Addr:6a000,Size:1000,Flag:0
EngMirrorDump 85  (LUCKY) Addr:6b000,Size:1000,Flag:0
EngMirrorDump 86  (LUCKY) Addr:6c000,Size:1000,Flag:0
EngMirrorDump 87  (PEPPER) Addr:6d000,Size:1000,Flag:0
EngMirrorDump 88  (SADIRS) Addr:6e000,Size:1000,Flag:0
EngMirrorDump 89  (SIBORE) Addr:6f000,Size:1000,Flag:0
EngMirrorDump 90  (POSTER) Addr:70000,Size:1000,Flag:0
EngMirrorDump 91  (XRESON) Addr:71000,Size:1000,Flag:0
EngMirrorDump 92  (OPTIMUS_2) Addr:72000,Size:1000,Flag:0
EngMirrorDump 93  (SUB_CTL_A_0) Addr:73000,Size:1000,Flag:0
EngMirrorDump 94  (SUB_CTL_A_1) Addr:74000,Size:1000,Flag:0
EngMirrorDump 95  (SUB_CTL_A_2) Addr:75000,Size:1000,Flag:0
EngMirrorDump 96  (SUB_CTL_B_12) Addr:76000,Size:1000,Flag:0
EngMirrorDump 97  (SUB_CTL_B_3) Addr:77000,Size:1000,Flag:0
EngMirrorDump 98  (SUB_CTL_C) Addr:78000,Size:1000,Flag:0
EngMirrorDump 99  (SUB_CTL_D_13) Addr:79000,Size:1000,Flag:0
EngMirrorDump 100 (SUB_CTL_D_2) Addr:7a000,Size:1000,Flag:0
EngMirrorDump 100 (SUB_CTL_D_2) Addr:7a000,Size:1000,Flag:0
EngMirrorDump 101 (SUB_CTL_D_4) Addr:7b000,Size:1000,Flag:0
EngMirrorDump 102 (SUB_CTL_E_1) Addr:7c000,Size:1000,Flag:0
EngMirrorDump 103 (SUB_CTL_E_2) Addr:7d000,Size:1000,Flag:0
EngMirrorDump 104 (SUB_CTL_F) Addr:7e000,Size:1000,Flag:0
EngMirrorDump 105 (SUB_CTL_G) Addr:7f000,Size:1000,Flag:0
EngMirrorDump 106 (SUB_CTL_H) Addr:80000,Size:1000,Flag:0
EngMirrorDump 107 (SUB_CTL_K) Addr:81000,Size:1000,Flag:0
EngMirrorDump 108 (SUB_CTL_M) Addr:82000,Size:1000,Flag:0
EngMirrorDump 109 (ELISION_MULTI) Addr:83000,Size:1000,Flag:0
EngMirrorDump 110 (CUMULO_MULTI) Addr:84000,Size:1000,Flag:0
EngMirrorDump 111 (OPTIMUS_MULTI) Addr:85000,Size:1000,Flag:0
EngMirrorDump 112 (COBALT_MULTI) Addr:86000,Size:1000,Flag:0
EngMirrorDump 113 (ECHIZEN2_MULTI) Addr:87000,Size:1000,Flag:0
EngMirrorDump 114 (ECHIZEN3_MULTI) Addr:88000,Size:1000,Flag:0
EngMirrorDump 115 (ECHIGO2_MULTI) Addr:89000,Size:1000,Flag:0
EngMirrorDump 116 (ECHIGO3_MULTI) Addr:8a000,Size:1000,Flag:0
EngMirrorDump 116 (ECHIGO3_MULTI) Addr:8a000,Size:1000,Flag:0
EngMirrorDump 117 (LUCKY1_MULTI) Addr:8b000,Size:1000,Flag:0
EngMirrorDump 118 (LUCKY2_MULTI) Addr:8c000,Size:1000,Flag:0
EngMirrorDump 119 (LUCKY3_MULTI) Addr:8d000,Size:1000,Flag:0

If you know the name of an Engine, and want to monitor MMIO usage, this lets you find where it occurs.
#2
Sorry, it's not that easy :)
#3
Camera-specific Development / Re: Canon 7D Mark II
March 21, 2024, 03:53:00 PM
Progress does not happen at a steady speed :)

I'm still documenting DMA and EDMAC stuff, got distracted by not having working tools for benchmarks.

I know you got DMA working directly, but did you ever have dma_memcpy() work on 7D2?  Or related functions, there's an async one for example.
#4
Do you mean add features to the build?  You'll need to know C, and have time to learn the code, and Ghidra, to understand the rom.  What do you know so far?

The 6D2 does not have a CF slot.
#5
Okay, good to learn about another major bug.  Well, you know, "good".

Obviously this is something that should be fixed.  Especially since on modern cams, trying to allocate all the SRM buffers triggers an OS assert and therefore crashes.

Looking here in exmem.c, this looks like a badly designed API by us:
283 struct memSuite *_shoot_malloc_suite(size_t size)
284 {
285     //qprintf("_shoot_malloc_suite(%x)\n", size);
286
287     if(size)
288     {
289         /* allocate exact memory size */
290         return shoot_malloc_suite_int(size);
291     }
292     else
293     {
294         /* allocate as much as we can */
295         size_t max_size = shoot_malloc_autodetect();
296         struct memSuite * hSuite = shoot_malloc_suite_int(max_size);
297         shoot_full_suite = hSuite;
298         return hSuite;
299     }
300 }

The max_size guessing code should probably be split out into its own function.  Then if you want to allocate a very large amount, you call that first, and pass the size to shoot_malloc_suite().  But now you can subtract some margin first, and avoid exhausting the pool.  I imagine it won't be that simple, but the current design doesn't even let you try!

This is an interesting topic, but it's also off topic for this thread.  Please create a new one if more discussion is useful.
#6
QuoteYes! But on the EOS M, the screenshot fumction only works to capture the ML Menus. Raw video needs to be turned off, so False colors and other overlays are not recorded.

Are you only checking using screenshots from benchmark?  Have you tried screenshots from Debug menu?

Debug menu does this:
take_screenshot(SCREENSHOT_FILENAME_AUTO, SCREENSHOT_BMP | SCREENSHOT_YUV);

Bench module code does this:
take_screenshot("bench%d.bmp", SCREENSHOT_BMP);

The screenshot code can capture both.  Benchmark code is asking it to not copy LV data.
#7
Not sure what you mean.  Like this?

#8
Updates for 2023 Oct - 2024 Mar
Last commit considered: https://github.com/reticulatedpines/magiclantern_simplified/commit/8ce0e1903594b0c8cd4ff9a7712f5066a9394b3c

A long time since my last update!  There was significant Real Life, but also, it took me quite a lot of time to get raw video working on 200D.

Headline changes
More in details below, but here's the bigger or funner pieces that made it in.

- Raw video for 200D, 7D2, R (in branches, no, you can't use this yet, but probably next month)
- Screenshots, including from benchmarks, are now .bmp, not .ppm.  Much more convenient
- Benchmarks are now much faster on slow cards / cams

Details!
1100D build was broken due to the low memory limits on this cam.  Some features removed to make it work again.

7D2 had a hard to track down bug due to being very close to the limit of allowed tasks.  Increased this limit during early boot process.

EDMAC stubs found for 7D2 and 200D.  Kitor has found these for R as well (maybe RP too?  I haven't been keeping track of external branches).  This is extremely important for new cams :)  In dev branch, there's code for EDMAC mem -> mem copies on 200D.  We have raw video working on a few cams, but this code is in branches.  Support is not clean yet, we need to integrate the different approaches on different Digic gens, and make the code more general, before it can be nicely merged into dev.

Significant bugs in ML were found and fixed during this work.  Lots of null pointer bugs in the old code, since the old cams don't crash if you read (or write!) through a null pointer.  MMU cams are not so forgiving.

Related to EDMAC, I refactored some FPS_REGISTER code to make it easier for D678X cams.  These cams use a different address range for this MMIO region, and in some cases different ways of accessing.  We'll need to do more of this, since the old ML code often hard-codes MMIO addresses, in files in src/, when in reality these are per cam constants - and different from D678X cams.

Created a tool for automatically determining image format and displaying it with correct proportions, given a buffer of data.  Works pretty well, can handle RGB, YUV and Bayer in the formats commonly found in cam memory.  See "src/tools/image/image_buffer_guesser/display_buf.py -h"

Moved all of contrib to tools.  Contrib had become a bad name.  Removed some unused code, too, including qemu, this has had its own repo for a long time now: https://github.com/reticulatedpines/qemu-eos/tree/qemu-eos-v4.2.1/magiclantern

More Dev Guide work.  This is slow work, but I think it will eventually be very helpful.  This update adds an overview about hardware: https://github.com/reticulatedpines/magiclantern_simplified/blob/eb84a464e5994a036f22dfa61fa95f885b684bf7/developer_guide/02_00_hardware.md

6D2 support finally tested on a physical cam.  I added this years ago and nobody volunteered.  Now we have a new dev / tester who has updated this cam to newer firmware, and fixed some bugs in my blind port.  Thanks!

Some improvements to benchmarks.  These are now time boxed, that is, they don't write a fixed amount of data, but will stop early if enough time has passed.  On fast cams / cards, this is the same behaviour as before.  On slow cams / cards, this makes the tests much faster to complete, with (so far) no observed problems with the results.

Screenshots: these now save in BMP format, not PPM.  This means...  you can view them just by opening the card.
#9
6D2 1.1.1 now has limited support.  Our one dev / tester (unsure if they have an account here, CrypticKlippo on Discord), suggests it's stable.  It's the standard "has ML GUI, has no features" early build.  A good place to work from to add things.

https://github.com/reticulatedpines/magiclantern_simplified/commit/d16c293288737193bc785f09e8f8cd86a888feb6
#10
Great, let me know if you have any problems.  The docs are very incomplete.  The docs generally for ML are quite awkward, sadly.  They're split over many different sources, quality varies.  Finding what you need is often difficult.

Hence why I'm trying to make a coherent dev guide, but I'm expecting this to take a long time.
#11
Raw Video / Re: Recording Raw on a DJI Gimbal
March 11, 2024, 01:29:16 AM
ML probably logged the errors to the card, that would also be useful.

And of course a pic of the errors shown on screen.
#12
Thanks, that I can try and copy :)  Won't be for a while though, got a bunch of tasks queued up.
#13
It's expected to work.

If you explain why it doesn't work for you, perhaps we can help.
#14
Thanks, but that doesn't really mean anything to me, and doesn't explain how I could test what difference dual ISO makes.
#15
Quote from: iaburn on March 08, 2024, 10:54:46 PMI did it to compare the maximum dynamic range achievable with magic lantern, because 200D is basically ISO invariant and wouldn't benefit from dual ISO, but the older cams do

I have a 200D.  What's the evidence that it's "ISO invariant"?  How would I measure this, and how would I measure dual ISO performance?

I have dual ISO working on 200D but don't know that much about photography, being more of a dev, so don't know how to test the results.
#16
Quote from: magiclantern.oimu5 on March 08, 2024, 07:03:46 PMYes, programming, reverse engineering, I have a background in embedded, firmware, virtualization development.

Good timing, we have someone who is currently updating the partially-supported ROM version from 1.0.5 to 1.1.1.  I am checking the changes currently.

I think it's likely that 6D2 can quickly be brought up to the same level of support as 200D, it's the same digic gen.  That would mean getting raw video on 6D2 is quite plausible.  On 200D, dual ISO works, too, as well as network support.

Discord is probably the easiest way to get you quickly setup.  But I'm happy to help here.  You will want this repo: https://github.com/reticulatedpines/magiclantern_simplified/commits

It should build easily on modern linux, I use Debian Testing.  We don't have a good checker for pre-reqs, missing rst2html is a common problem, for that you want python3-docutils.

Currently, it's fw 1.0.5.  1.1.1 is the new target, I expect my checks will be completed and 1.1.1 support merged within a week.
#17
General Help Q&A / Re: Micro SD Cards
March 07, 2024, 02:52:24 PM
Micro SD vs SD card, both are the same.  Real shooting experience.

There is no way for us to replicate what you are seeing because you don't give any information, so it's not useful to anyone.  If you list what cards you use, in what conditions, people could check.
#18
It's a commercial product, why ask us?  Ask them, the website is listed and has a contact address.
#19
Share Your Videos / Re: Canon 70D Test Footage
March 03, 2024, 03:40:33 PM
Thanks for the appreciation :)

Work on older cams isn't wasted, ML tries to be a framework that allows the same code to work on many cameras.  It's harder to port to a new generation of cams so less people attempt it.  However, this is most of what I work on.

When new cams are added (e.g. see recent work on 200D, 7D2, R), features that work on old cams can be enabled.  This does require additional work, but less than if the features hadn't been made at all!
#20
QuoteIn that case, can you share me the right one
It's by design that module_strings.h is generated locally.  This is a slightly strange design, but that's the way it's supposed to work.  Giving you the file is not fixing the problem (and you already have the file, since you got it building on another system).

QuoteI have not given up. Just waiting for your inputs.
The problem here is that I want to fix this new error, but I can't reproduce it and don't have the time to install voidlinux and then hope I get the same broken config as you.  If you can determine the cause, I will fix it.

Quotevoidlinux has not broken in the last four years
I have nothing against bleeding edge distros - but I know from many years of experience that they are bad choices as build environments.  You will run into all kinds of difficulties due to cutting edge tools or libraries.  Essentially, you are asking me to support voidlinux as a build environment, and I don't want to, because I know it's going to cost me lots of effort for minimal gain.  You're choosing to use the weird environment, you have the responsibility to understand it.  I use Debian Testing or Stable.  For dev work, boring is good.

Personally I wouldn't recommend Ubuntu as a dev env, either, it's designed more for consumer usage.  Probably easier to get working than voidlinux though :)

The M build errors should go in a separate thread (you're missing some dev lib, possibly libc6-dev).
#21
From the module_strings.h file it's clear that a) it has bad content and b) where the code must stop early in order to generate that content.  I couldn't work out a reason why it might stop there, it doesn't look possible to me.

Given I can't reproduce the problem and you've given up, I will too.  I'm glad you got it working, but be warned it's likely to stop working on voidlinux; these kind of distros are likely to update some library the binary depends on in a breaking way.
#22
This suggests the content of module_strings.h is bad.  Please put the file on pastebin or similar and send me the link.
#23
Quote from: zenny on February 24, 2024, 07:17:02 AMDidn't go well at all

I need more info than that in order to fix it :)

You've helped me find one real problem, which is now fixed for everyone.  If there are more problems remaining, we can fix those too.  This helps everyone who uses ML.

I have a binary that works for me, but that doesn't mean it will work on your system, because it's designed to load the libraries it was built against.  Yours will differ.  Sometimes this works, sometimes it doesn't.
#24
Ah, good, we have found the real error:

modules/dual_iso/../html2text.py:630: SyntaxWarning: invalid escape sequence '\s'
  data = re.sub('\s+', ' ', data)

You are using a version of Python too modern for the build system.  A change in 3.12 has introduced this error:
https://docs.python.org/3/whatsnew/3.12.html
A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning,
instead of DeprecationWarning. For example, re.compile("\d+\.\d+") now emits a SyntaxWarning
 ("\d" is an invalid escape sequence, use raw strings for regular expression:
 re.compile(r"\d+\.\d+")). In a future Python version, SyntaxError will eventually be raised,
 instead of SyntaxWarning. (Contributed by Victor Stinner in gh-98401.)

Using a bleeding edge distro for doing builds may not be a good idea, you'll hit these kinds of problems fairly frequently.

Fixed: https://github.com/reticulatedpines/magiclantern_simplified/commit/09d3fddeb7f462d1dd61098e55abc1b81dd6433d
#25
That shouldn't be relevant here.  You were trying to build an old repo that expects old tooling, and there was confusion around python2 and python3.

If zenny followed my instructions, they're using a modern repo that works fine with python3.