https://en.wikipedia.org/wiki/Null_pointerThe null pointer test checks whether ML can
detect this particular error and save a crash log (rather than ignoring it silently). These errors may come from both ML code (where they should be fixed), or from Canon code (these are harder to investigate, so we have chosen to
ignore them for now; we used to whitelist them, but there were just too many of them, so we gave up).
Older builds marked as successful simply did not run the tests that are now failing (so they are not safer in any way, just less tested).
Please note the QEMU tests are just barely scratching the surface - less than 10 features out of several hundreds are actually tested like this. Things are improving, but manpower is an issue, so any help is welcome; however, most of the interesting features require emulating things like photo capture and LiveView (so the tests that can be run right now are pretty limited - menu navigation, DryOS internals and so on). (Don't know where to start?
Right here; the tests are just bash scripts around QEMU).
BTW, here's an animation (
5D3.gif - 10MB) that shows some script-driven ML menu navigation (it takes screenshots and compares their hashes to know where to stop). Just navigating each top-level menu option - without even trying to change the submenu settings - takes 15 minutes for the script, at 2 keystrokes per second. This does not test any functionality - it covers just plain menu navigation (top-level settings only). The linked animation is a WIP for
this proposal.