Magic Lantern Forum

Developing Magic Lantern => General Development => Topic started by: a1ex on September 19, 2017, 10:35:56 PM

Title: Automated tests for nightly builds in QEMU
Post by: a1ex on September 19, 2017, 10:35:56 PM
Another pipe dream came true :) - this time, a dream of mine.

Have you noticed a bunch of screenshots on the nightly builds page?

Were you wondering what's up with them?

(https://builds.magiclantern.fm/jenkins/job/QEMU-nightly-tests/90/artifact/qemu/magiclantern-Nightly.2017Sep15.5D2212/2-freemem.png) (https://builds.magiclantern.fm/jenkins/job/QEMU-nightly-tests/189/artifact/qemu/magiclantern-Nightly.2017Sep19.1200D101/3-lua-hello.png) (https://builds.magiclantern.fm/jenkins/job/QEMU-nightly-tests/189/artifact/qemu/magiclantern-Nightly.2017Sep19.1200D101/7-sokoban-l1-sol.png)

These screenshots are created on the build server, by emulating the very builds available for download, unmodified, in QEMU.

In other words, most of the nightly builds are no longer 100% untested when they appear on the download page :)

This is not an overnight development - it's built upon all these years of fiddling with QEMU (http://www.magiclantern.fm/forum/index.php?topic=2864). A short while ago I couldn't give a good answer regarding the usefulness of the emulator (http://www.magiclantern.fm/forum/index.php?topic=2864.msg169970#msg169970) - now you can see it live.

Right now there are only a few tests, with OCR-based menu navigation (using tesseract (https://github.com/tesseract-ocr/tesseract)):

1) navigate to Debug -> Free Memory and take a screenshot from there
2) load the Lua module and run the Hello World script
3) load the file_man module and browse the filesystem
4) play the first 3 levels of the Sokoban game (lua_fix only; example for 1200D (https://builds.magiclantern.fm/1200D-101.html))

TODO:
- add more tests (easy, but time-consuming)
- emulate more camera components (e.g. image playback to be able to test ML overlays)
- check code coverage
- diff the screenshots (http://www.magiclantern.fm/forum/index.php?topic=12396.0)
- nicer reports

For now, have fun watching the testing script playing Sokoban in QEMU :)

(https://builds.magiclantern.fm/jenkins/view/QEMU%20nightly%20tests/job/QEMU-nightly-tests/189/artifact/qemu/magiclantern-Nightly.2017Sep19.1200D101/sokoban.gif)

Emulation log (https://builds.magiclantern.fm/jenkins/job/QEMU-nightly-tests/189/consoleText)

If you are wondering what's the point of testing this game: it covers many backend items, such as menu navigation, module loading, script config files, making sure keys are not missed randomly during script execution, checking whether the camera has enough memory to run scripts - most of these are real bugs found on some camera models from the current nightly builds.

At least, these tests will catch the long-standing issue of some camera models running out of memory, thus not being able to boot. Not very funny for a build considered somewhat stable...

And the emulation is still pretty limited, so I'm just adding tests for what works :)




A while ago I've got the suggestion to use openQA (http://open.qa/), but I'm still wrapping my head around it. If you can show how it could save us from reinventing the wheel, I'm all ears.
Title: Re: Automated tests for nightly builds in QEMU
Post by: dmilligan on September 20, 2017, 04:12:17 AM
This is quite an accomplishment. Congratulations!
Title: Re: Automated tests for nightly builds in QEMU
Post by: DeafEyeJedi on September 20, 2017, 08:04:25 AM
Now this is it. Finally you have pulled the trigger on pushing me closer to attempting to get into QEMU. Thanks for this and congrats for yet another pipe dream accomplishment @a1ex!
Title: Re: Automated tests for nightly builds in QEMU
Post by: g3gg0 on September 20, 2017, 11:15:48 PM
also congratulations from me. thats reealllly a lot of work you put into this.
Title: Re: Automated tests for nightly builds in QEMU
Post by: kichetof on September 21, 2017, 08:36:58 AM
You did it! Automated QC for our DSLR! All my Congrats! 8)
Title: Re: Automated tests for nightly builds in QEMU
Post by: tonij on September 21, 2017, 10:24:28 AM
I predict ML starts thinking for itself and hides forever in the cameras very core
Title: Re: Automated tests for nightly builds in QEMU
Post by: oneo on October 10, 2017, 08:36:52 PM
hi guys hope this helps with your test results...
loaded 2 x 5D Mk1  works fine
loaded 7/8 x 350d works fine
loaded 1 x EOS M  works fine
loaded 1 x 7D  works fine
loaded 1 550D works fine  (had to reload it a couple of times, maybee sd) now good
loaded 1 x 100D  problem with DOF Bracket .... corrupted all on sd.   had to set camera to middle ie f12-16 then seemed ok

great work guys
Title: Re: Automated tests for nightly builds in QEMU
Post by: Audionut on October 11, 2017, 05:52:59 AM
Quote from: g3gg0 on September 20, 2017, 11:15:48 PM
also congratulations from me. thats reealllly a lot of work you put into this.

+1

Very exciting to see all of these backend things coming to fruition.  These things don't come with all of the accolades that shiny new toys for users to play with generate.  So here's another +1 on behalf of all of the users that didn't get a shiny new toy to play with, and don't understand the significance of these developments.  :)
Title: Re: Automated tests for nightly builds in QEMU
Post by: a1ex on November 11, 2017, 01:52:39 AM
Queued for automatic testing: 6D, 70D, 5D3.1.2.3.

edit: they ran overnight; almost worked :)