Introduction
With the Aikau framework for Share construction we have created a large number of widgets with some underlying code from Dojo’s UI Library, Dijit. It makes sense to automatically test the Aikau widgets in isolation and therefore we have created a test framework using Intern (http://theintern.io/) to do just that. The tests run quite fast and with predictable results. It is now easy to see if a recent addition or update to an existing widget has caused a bug or regression.
To perform tests we have written a number of test web scripts that contain widget models. The widget models are usually quite simple but occasionally have a more complex structure to expose edge case behaviours. Some test web scripts also contain intentionally broken models to test the catching of errors. For testing purposes the test web scripts are run using Jetty as we do not need any of the more complex overhead from Share.
Process Overview
To run a suite of tests we do the following:
- Launch an instance of an Ubuntu virtual machine using Vagrant
The virtual machine provides a set of test browsers, drivers for those browsers and a running instance of Selenium that Intern can target - Run up the test web scripts using Jetty
We do not need all of Share so a basic Jetty server is fine here - Run tests against the test web scripts using Intern
Each of the tests will be run according to the Intern configuration and against the browsers defined and requested therein
Step 1 - Prerequisites
The testing framework for Aikau makes use of a number of technologies and you will need the following free downloads to proceed. Please install with the default settings:
I am assuming that you can use a command line and have a vague idea of what a virtual machine and Vagrant are. If not, please read about it first: http://www.vagrantup.com/
Step 2 - Installation
Having downloaded and installed a current version of the Alfresco code base, open a command prompt and navigate to the following Alfresco directory:
{alfresco}/code/root/projects/slingshotRun the following command to install Node dependencies:
>> npm install
If you’re interested to know what is being installed you can look at file package.json in the directory above which contains a list of the dependencies. Once you have installed all of the components required for the test framework, you should be able to launch a virtual machine with vagrant for the first time. The first time you run this process it may be slow as it has a lot to download:
>> g vup
Note: This command is ‘g’ for grunt and ‘vup’ for vagrant up.
When the ‘g vup’ command has completed, if it has been successful you should be able to observe the Selenium instance running here:
http://192.168.56.4:4444/wd/hub/static/resource/hub.html
The selenium console should look something like this:
Step 3 - Running the suite of tests
With an instance of Selenium available on the virtual machine, you should now be able to run Intern by issuing the following command:
>> g test
Note: This command is ‘g’ for grunt and ‘test’ for run intern test suite (virtual machine).
This command first checks if the test application server is running and launches it if not. Once it is happy that the server has started completely it will proceed to launch the Intern test runner. You should see the test suite run through a large number of tests (about 110 tests run twice for two different browsers at time of writing) and log them to the console. Hopefully they will all pass.
This is the sort of output you should expect to see once the initialisation steps have been performed:
...
>> Starting 'AccessibilityMenuTest' on chrome
>> Test page for 'AccessibilityMenuTest' loaded successfully
>> AccessibilityMenuTest: Find the menu element
>> AccessibilityMenuTest: Find the heading text
>> AccessibilityMenuTest: Find the menu items
>> AccessibilityMenuTest: Find the first target
>> AccessibilityMenuTest: Find the second target
>> AccessibilityMenuTest: Find the first menu link - which links the first target
>> AccessibilityMenuTest: Hit the browser with a sequence of different accesskey combinations and the letter 's' for a nav skip
>> Starting 'SemanticWrapperMixinTest' on chrome
>> Test page for 'SemanticWrapperMixinTest' loaded successfully
>> SemanticWrapperMixinTest: Check NO_WRAPPER dom is correct
>> SemanticWrapperMixinTest: Check GOOD_WRAPPER dom is correct
>> SemanticWrapperMixinTest: Check BAD_WRAPPER dom is correct
>> SemanticWrapperMixinTest: Check LEFT_AND_RIGHT_WRAPPER dom is correct
>> Starting 'Pie Chart Test' on chrome
>> Test page for 'Pie Chart Test' loaded successfully
>> Starting 'PublishPayloadMixinTest' on chrome
>> Test page for 'PublishPayloadMixinTest' loaded successfully
...
Note: When a test run is complete the Jetty server is left running. If there are any failures it is possible to immediately investigate if something catastrophic has occurred by simply observing the test web script in a browser.
Useful commands
There are several grunt commands that can be used with Vagrant, the test application server and Intern. Here are the ones we've already seen and a few more, all of which should be run from the directory shown above:
| Command | Function |
|---|---|
| g vup | Short for ‘vagrant up’ this will launch a virtual machine instance with Vagrant |
| g vdown | Short for ‘vagrant down’ this will stop a running instance of a Vagrant virtual machine |
| g vclean | Short for ‘vagrant clean’ this will delete an existing instance of a Vagrant virtual machine |
| g test | Run up an instance of the test application server and run the Intern test suite against it |
| g shell:startTestApp | Start the test application server |
| g shell:stopTestApp | Stop the test application server if it is running |
| g nt | Short for ‘new test’ this command will restart the Vagrant virtual machine, restart the test application server and finally run the Intern test suite against them |
| g utd | Short for ‘update test deployment’ this command will bring down the test application server, rebuild slingshot and then relaunch the test application server with any file modifications that have been made |
Adding a test
If you wanted to add a test of your own there are three steps to the process:
- Create a test web script
- Create an Intern test
- Add the test to the Intern Suites.js file
Create a test web script
Test web scripts all live in this location or a sub-directory of it:
{alfresco}/code/root/projects/slingshot/tests/testApp/WEB-INF/classes/alfresco/site-webscripts/alfrescoEach web script requires three files - a JavaScript file, a Freemarker template file and an xml description file. There are examples in the directory that can be looked at and copied. As with web scripts in the main applications the format of the file names is important. With a correctly written test web script you should be able to view the test model in action at a URL such as:
http://localhost:8089/aikau/page/tp/ws/AccessibilityMenu
This example test web script has in it's model an AccessibilityMenu widget. It isn't very pretty as rendered here but it isn't supposed to be.
Create an Intern test
The actual test files are created here or in a sub-directory of it:
{alfresco}/code/root/projects/slingshot/tests/alfrescoIntern tests are written in JavaScript using a promise-based dependency called Leadfoot which is provided by SitePen, the company who wrote Intern itself. You can read the Leadfoot documentation here. Strategies for writing Selenium tests are complex and I'm not going to investigate them here. Needless-to-say, one emulates the behaviour of an individual using a browser to interrogate the test web script as rendered.
The specific way in which an Intern test addresses a test web script can be seen if any of the existing tests is viewed. Pay close attention to this part of most tests which is the point at which the test web script is loaded:
...
var browser = this.remote;
var testname = 'AccessibilityMenuTest';
return TestCommon.loadTestWebScript(this.remote, '/AccessibilityMenu', testname)
...
Add the test to the Suites.js file
Tests that should be run are all listed in this file in the array 'baseFunctionalSuites':
{alfresco}/code/root/projects/slingshot/tests/config/Suites.jsNext steps
The testing framework supports testing against a local instance of Selenium rather than a VM instance as described above. It is also possible to run a coverage report on the test framework to indicate the amount of the code base being interrogated. The details of these alternative scenarios will follow in another post.
1 Comment
