Proposal for managing modules

[a1ex]

1) Thinking to add a :Category: field to module metadata (in the README files). Something along these lines:

raw-video:
mlv_rec mlv_snd mlv_play raw_rec bolt_rec

raw-photo:
dual_iso deflick ettr pic_view silent (mini_iso?)

// well... ettr, dual iso, mini_iso work for video too, but I'd say the primary target is photo; other ideas?

debug:
mem_chk mem_prot trace raw_diag (benchmrk? selftest?)

research:
adtg_log adtg_gui cf_acc iso_test iso_regs lv_rec mem_spy mrc_dump smpte

input:
ime_base ime_rot ime_std ime_null (ime_morse?)

misc:
file_man io_crypt tr_wifi (bitrate?)

misc-photo:
adv_int autoexpo auto_iso dot_tune filepref (fstack?)

games:
arkanoid

scripting:
script tcc fpu_emu tinypy

Let me know if you can think of better categories, or I've missed something. I've also included modules that are not yet done, but might appear in the near (or far) future.

I'm also considering tags instead of categories (so one module might have more than one tag). Advantages? Disadvantages?

2) What about treating modules like apps from users? That means, provide a central place for users to download these modules without requiring them to be included in the main ML repo.

This way, Marsu can keep developing auto_iso without me dissecting his code, Pravdomil can tweak autoexpo without having to submit a pull request to get it updated in the nightly builds, 1% can share his bitrate code as a module that can run on top of normal ML, dmilligan can continue his work on adv_int without pressure and so on. We can setup the nightly builds page to build each module from its own repository.

Of course, whenever you require core changes to support these modules, these are much easier to review when you submit them via pull requests.

3) To further lower the entry barrier for development, I'm also thinking about source modules, besides scripts (that is, just drop the .c source file in the camera in the modules directory, like you are doing now with .mo files, and have it compiled on the fly by TCC). I'm not 100% sure it's doable, but I'd like to know if there is interest in this direction. With this implemented, anybody with basic C skills can tweak their modules without requiring an ARM toolchain. It will probably cause confusion (as in, what's the difference between these source modules and scripts?), and the source modules will probably run a bit slower than current .mo files, but it's just an idea for discussion.

Thoughts?

[ItsMeLenny]

1
I myself wouldn't be a fan so much of the categories. I always find it difficult to find the thing I'm looking for because I never know what category it's in.
(Maybe if there was the ability to switch between category view and list all modules view).
However putting debug modules in a category probably wouldn't be a bad idea. But then you have modules both in and out of categories which might be considered in consistent.

2
Modules as apps isn't a bad idea. Then if they don't get update they don't need to be downloaded.
Another program on top of that could be built for the computer that checks what modules are available, what modules need updating.
Basically one click downloads, one click updates.

3
It certainly wouldn't be considered a bad feature.

[Audionut]

Tags are a good idea.  This way the raw-photo modules could be tagged as raw-video also.

At the top of the module menu you could have a selection for module type, which then only displays modules with that tag.  I wouldn't even put it in a sub menu or anything, just indented below the type selection.  Sort of like the modified menu, except of course, you would only have the one main selection, with all the modules from the tag type indented below.

My only concern with opening up modules in the way you describe, is people using this system to deliver malicious code.  Personally, I would prefer some code review?

[engardeknave]

I think functionality should be as modular as possible. I don't know how much more difficult that makes things technically, I'm just speaking in terms of getting people involved and making development less centralized. This gets the people who are very invested in certain features working on those features.

There are so many different photography niches, but not all of them have a technically skilled developer for whom the obstacles of getting started are easily overcome. As far as I can tell, the strongest coders are mostly interested in bugs, stars, and lamps.

[gerk.raisen]

+1 for the tags.

A lot more flexible than simple categories


+2 also for source modules.
I already wrote a very close idea to the forum some times ago

Can't wait !

[chris_overseas]

1) Adding categories or tags to the metadata sounds like a good idea and your category suggestions all seem sensible. My question is, how do you see that metadata being used and how will it affect the UI? Perhaps display a different icon for each module based on the category? Group modules by category (perhaps in submenus)? Change the module's location in the menu system somehow?

2) I like what you're trying to achieve here. My main concern would be that I'd have less trust in the modules (in terms of both safety & quality) because they were no longer being code reviewed by the core developers. Perhaps a commenting/rating system would help address that. Safeguards would also need to be put in place to ensure modules could only load for the correct camera and firmware versions.

3) Sounds good, just not sure it would be worth the effort required and the potential confusion given it targets a relatively small use-case.

[Marsu42]

1. If you go for categories, at least two are missing:
* usability: Probably most of what I do & will submit sooner or later - hotkeys, little automatizations, ...
* exposure: autoexpo, auto_iso ... maybe too little for an own category, but I wouldn't know where to put these in your current list.

2. [edit] The decision not to merge modules, but offload to some other distribution system is not just a technical decision, but a social one.

[g3gg0]

1)
+1

stability tags please too.
 - experiment
 - development
 - staging
 - release

2)
i think we already discussed the concept of a module database.

using tr_wifi and (internal experiment) mod_mgr i already set up a prototype to check feasability.
tr_wifi uses transcend wifi cards for HTTP access, mod_mgr uses http://ml.g3gg0.de/ for in-camera module updating.
see e.g.
 - http://ml.g3gg0.de/modules/io_crypt.mo/latest_date to get the date/time of last update of trace.mo
 - http://ml.g3gg0.de/modules/io_crypt.mo/io_crypt.mo to get the latest module or
 - browse http://ml.g3gg0.de/modules/io_crypt.mo/ to see previous versions too
 - simply uploading a .mo via http://ml.g3gg0.de/ will update all references

i like that idea and it is already working as experiment, far from usability. that needs a nifty Web-UI too.
(final: use hg/svn for versioning and allow different "package sources" like debian)

3)
which was one of the main reasons, why tcc went into a module.
iirc our goal was: modules either as .mo or as .c will be the same except for performance. (assuming gcc optimizes better than tcc)

[Marsu42]

stability tags please too.
 - experiment
 - development
 - staging
 - release

+1, I already suggested this some time ago but to no avail, in addition to a tag specifying on what *models* this module is tested on and thus "release" usability can be ensured.

[a1ex]

I already suggested this some time ago but to no avail

Sorry, I'm not able to allocate 72 hours daily for ML, no need to rant.

Stability: how should I evaluate it?

For example, let's take ETTR. For me it works most of the time, still has small hiccups that I couldn't diagnose yet, but on EOS-M for example it was reported to be kinda broken. So it's kinda camera-specific and subjective.

I'm thinking to implement some sort of usage logs (that is, ML would save on your card a usage.log file containing the setting you used, modules loaded, log crashes and so on), and when you download the next nightly, you also have a upload box where you send us these logs, along with (optional) comments about how each feature worked (you would get a feedback form generated from your usage log - e.g. contain only the things you had enabled).

With enough data (say over a few months), I might be able to write some analysis tools and do graphs about testing coverage, what features get used and what gets never touched, how often it crashes, and I might even find hints about what modules/features might cause the crashes.

So, with these usage logs, I hope to find a somewhat objective metric of stability. E.g. if say feature/module X ran for 1000 hours with no major hiccups, it's probably fine. If most (or all) test runs of module Y resulted in crash or abnormal shutdown, there's probably something wrong with it. If feature Z was not used for more than 5 minutes in 3 months (5 minutes being total usage time over all users), it's a candidate for removal.

This won't catch bugs that happen once in a blue moon (like many race conditions or memory overflows), but at least should give a better idea about their presence and frequency.

[g3gg0]

i.e. categorize into 3 "support" levels.
 - proven stable (dev has this model and it shows no misbehavior)
 - usable (seems to work on this model, at least there were reports, hickups may happen)
 - untested (might work, expect anything)
 - unsupported (proven not working)

and add this tag for every model (see propvalues.h definitions wich are also used in EXIF for model identification) separately.
we could use the module macros as we have now for that.

MODULE_SUPPORTLEVELS()
    MODULE_SUPPORTLEVEL(MODEL_EOS_7D, SUPPORT_NONE)
    MODULE_SUPPORTLEVEL(MODEL_EOS_5D3, SUPPORT_STABLE)
 ...

[a1ex]

@g3gg0: I like this one (and the color coding too). Untagged would default to untested, right?

I'd prefer to tag them in the README file, what about something like this:

Code: [Select]

:Stable: 5D3 7D EOSM
:Usable: 5D2 550D
:Not working: 500D

and everything else defaults to untested?

[g3gg0]

right, we already put that annotation stuff into .rst.
then there, with "C level implementation" being like above, generated from .rst
(i think .rst -> .c is also still using the macros for description we had before, right?)

when someone adds a .c only module, are these old annotations we had as macros then still valid?
or does he have to supply a .rst?

[a1ex]

Camera-specific tags: yes, your suggestion about implementing is more elegant than just copying the strings and doing some parsing on the fly, but we would have to maintain a list of MODEL_EOS_xxD constants; I'm not a big fan of these lists, I prefer to reuse the camera_model_short strings, as they are quite readable.

C-only modules: didn't think yet about how to annotate them (the RST parser is a Python script). Good point, will think about it.

Now, combining with the idea from Audionut, what about filtering the displayed modules like this?

Code: [Select]

static struct menu_entry module_menu[] = {
    {
        .name = "Show modules",
        .priv = &show_modules,
        .choices = CHOICES("All", "Stable", "Usable & stable", "Untested", "Photo", "Video", "RAW", "Exposure", "Diagnostics", "Experiments"),
        .help = "Choose what modules to display:",
        .help2 =
            "All: show all modules\n"
            "Stable: developer has this model and it shows no misbehavior\n"
            "Usable: seems to work on this camera, hiccups may happen\n"
            "Untested: might work, expect anything\n"
            "Photo: show modules focused on stills shooting\n"
            "Video: show modules focused on video shooting\n"
            "RAW: show modules focused on RAW shooting (photo or video)\n"
            "Exposure: show modules that help with exposure (photo or video)\n"
            "Diagnostics: show modules intended for troubleshooting\n"
            "Experiments: show developer's experiments and research code\n",
    },
    MODULE_ENTRY(0)
    MODULE_ENTRY(1)
    MODULE_ENTRY(2)
    ...

I've mixed stability ratings with tags for the following reasons;
- to have a single entry in menu
- troubleshooting and experiments don't really need stability ratings, do they?

And, of course, the tag list is open, and each tag also has a description visible in the menu.

So, the modules could be tagged in the README, besides stability ratings, as:

Code: [Select]

:Tags: exposure raw photo
:Tags: experiment video

and so on.

[Audionut]

Can you define stable, unstable, untested, all?

Define stable, show tags photo, exposure.

This way when you select this or that tag, you also only get stable/unstable/untested/all modules.

[a1ex]

Like this?

Code: [Select]

    {
        .name = "Filter by stability",
        .priv = &stability_filter,
        .choices = CHOICES("Stable", "Usable (or better)", "Untested (or better)", "All"),
        .help = "Choose what modules to display, by stability rating:",
        .help2 =
            "Stable: developer has this model and it shows no misbehavior\n"
            "Usable: seems to work on this model, hiccups may happen\n"
            "Untested: might work, expect anything\n"
            "All: show all modules regardless of stability rating\n"
    },
    {
        .name = "Filter by tags",
        .priv = &tag_filter,
        .choices = CHOICES("All", "Photo", "Video", "RAW", "Exposure", "Diagnostics", "Experiments", "Untagged"),
        .help = "Choose what modules to display, by tags:",
        .help2 =
            "All: show all modules regardless of their tags\n"
            "Photo: show modules focused on stills shooting\n"
            "Video: show modules focused on video shooting\n"
            "RAW: show modules focused on RAW shooting (photo or video)\n"
            "Exposure: show modules that help with exposure (photo or video)\n"
            "Diagnostics: show modules intended for troubleshooting\n"
            "Experiments: show developer's experiments and research code\n",
            "Untagged: show modules without tags in README.rst (please add tags)\n"
    },
    MODULE_ENTRY(0)
    MODULE_ENTRY(1)
    MODULE_ENTRY(2)
    ...


[Audionut]

Yeah that's better 

This way the stability of the module always has some priority.

edit:  Hmmm, not sure if we would then need the corresponding tags also?  Since we already define this in the other menu option.  The "all" tag would still be needed to show all module tag types.

Code: [Select]

    {
        .name = "Filter by stability",
        .priv = &stability_filter,
        .choices = CHOICES("Stable", "Usable (or better)", "Untested (or better)", "All"),
        .help = "Choose what modules to display, by stability rating:",
        .help2 =
            "Stable: developer has this model and it shows no misbehavior\n"
            "Usable: seems to work on this model, hiccups may happen\n"
            "Untested: might work, expect anything\n"
            "All: show all modules\n"
    },
    {
        .name = "Filter by tags",
        .priv = &tag_filter,
        .choices = CHOICES("All", "Photo", "Video", "RAW", "Exposure", "Diagnostics", "Experiments", "Untagged"),
        .help = "Choose what modules to display, by tags:",
        .help2 =
            "All: show all modules\n"
            "Photo: show modules focused on stills shooting\n"
            "Video: show modules focused on video shooting\n"
            "RAW: show modules focused on RAW shooting (photo or video)\n"
            "Exposure: show modules that help with exposure (photo or video)\n"
            "Diagnostics: show modules intended for troubleshooting\n"
            "Experiments: show developer's experiments and research code\n",
            "Untagged: show untagged modules only\n"
    },
    MODULE_ENTRY(0)
    MODULE_ENTRY(1)
    MODULE_ENTRY(2)
    ...
Ok, I see you removed them from the choices.

[a1ex]

Updated the code to clarify the meaning of "All" (it's actually some sort of "OFF", since it turns off that filter).

[Audionut]

- troubleshooting and experiments don't really need stability ratings, do they?

I would say no.  This way we don't create a condition where stability is defined as stable, resulting in no modules showing with tag selection as experimental.

[a1ex]

So, to be clear, if tag selection is "experimental", I should ignore the stability rating?

(or should I move the "experimental" tag to stability choices? this also doesn't make sense per-camera, because an experiment is experiment for all cameras...)

[Audionut]

Choices would probably be more suitable.  Since by definition, experimental = unknown stability type.

Per-camera, well if the author knows that some things are currently only defined for specific camera, then model type would still be useful?

[a1ex]

If something is not working for a specific camera, there's the :Not working: field, which is a stability rating.

[g3gg0]

some other thoughts:
beside the "category selection", add one option for loading modules:
 Allow unsafe plugins: YES/NO  (WARNING: Some experimental modules may crash your cameras)

process the modules in that order:
 - if a module is tagged as "known not working" for a model, it will be hidden (extra menu point "incompatible modules" may show/summarize the modules that were hidden due to that?)
 - when a module is not considered "stable" for your model, it will be hidden/grayed out if "unsafe" isn't enabled.
 - depending on the stability level, the module is colored in green/blue/yellow or some warning for that level is added

so the proposed category tags are used for visual filtering and the stability tags are used for functional filtering/coloring.

rookie users only see the modules that are stable for their camera.
advanced users can enable the "usable" and "unstable" modules from menu after accepting a "i agree" nag screen.
devs can alter module's "not working" tags if they want to test/port it on their cameras.

[Marsu42]

Sorry, I'm not able to allocate 72 hours daily for ML, no need to rant.

Sorry, I specifically  tried to convey that I am not ranting (and certainly not at you), what I meant to say is that this topic was discussed before and could be picked up. But I read how the feeling of the core devs is, so that's also something.

Stability: how should I evaluate it?

I don't think too much statistics need to put into this as long as it's "production" (tested for a sufficient time w/o bugs), "beta" (works on the given camera model, but might produce bugs), "alpha" (somebody got this running on a specic model, but has known bugs) and "untested".

[a1ex]

Sure, but I'd like an objective criteria for evaluation.

Basically, I want to avoid questions like why my module is rated as "alpha" and some other module is rated as "beta"? So, one point of the usage log proposal is to minimize the subjective factor.

The second point is to automate this task. Once we have some ratings, who's going to keep them updated?

The feature matrix is updated at compile time (which is great if you ask me). I want something similar for ratings, if that's not science-fiction.