Automated tests for nightly builds in QEMU

[a1ex]

Another pipe dream came true - this time, a dream of mine.

Have you noticed a bunch of screenshots on the nightly builds page?

Were you wondering what's up with them?



These screenshots are created on the build server, by emulating the very builds available for download, unmodified, in QEMU.

In other words, most of the nightly builds are no longer 100% untested when they appear on the download page

This is not an overnight development - it's built upon all these years of fiddling with QEMU. A short while ago I couldn't give a good answer regarding the usefulness of the emulator - now you can see it live.

Right now there are only a few tests, with OCR-based menu navigation (using tesseract):

1) navigate to Debug -> Free Memory and take a screenshot from there
2) load the Lua module and run the Hello World script
3) load the file_man module and browse the filesystem
4) play the first 3 levels of the Sokoban game (lua_fix only; example for 1200D)

TODO:
- add more tests (easy, but time-consuming)
- emulate more camera components (e.g. image playback to be able to test ML overlays)
- check code coverage
- diff the screenshots
- nicer reports

For now, have fun watching the testing script playing Sokoban in QEMU



Emulation log

If you are wondering what's the point of testing this game: it covers many backend items, such as menu navigation, module loading, script config files, making sure keys are not missed randomly during script execution, checking whether the camera has enough memory to run scripts - most of these are real bugs found on some camera models from the current nightly builds.

At least, these tests will catch the long-standing issue of some camera models running out of memory, thus not being able to boot. Not very funny for a build considered somewhat stable...

And the emulation is still pretty limited, so I'm just adding tests for what works


A while ago I've got the suggestion to use openQA, but I'm still wrapping my head around it. If you can show how it could save us from reinventing the wheel, I'm all ears.

[dmilligan]

This is quite an accomplishment. Congratulations!

[DeafEyeJedi]

Now this is it. Finally you have pulled the trigger on pushing me closer to attempting to get into QEMU. Thanks for this and congrats for yet another pipe dream accomplishment @a1ex!

[g3gg0]

also congratulations from me. thats reealllly a lot of work you put into this.

[kichetof]

You did it! Automated QC for our DSLR! All my Congrats!

[tonij]

I predict ML starts thinking for itself and hides forever in the cameras very core

[oneo]

hi guys hope this helps with your test results...
loaded 2 x 5D Mk1  works fine
loaded 7/8 x 350d works fine
loaded 1 x EOS M  works fine
loaded 1 x 7D  works fine
loaded 1 550D works fine  (had to reload it a couple of times, maybee sd) now good
loaded 1 x 100D  problem with DOF Bracket .... corrupted all on sd.   had to set camera to middle ie f12-16 then seemed ok

great work guys

[Audionut]

also congratulations from me. thats reealllly a lot of work you put into this.

+1

Very exciting to see all of these backend things coming to fruition.  These things don't come with all of the accolades that shiny new toys for users to play with generate.  So here's another +1 on behalf of all of the users that didn't get a shiny new toy to play with, and don't understand the significance of these developments. 

[a1ex]

Queued for automatic testing: 6D, 70D, 5D3.1.2.3.

edit: they ran overnight; almost worked