Welcome to part III in my TestLink tutorial series. In the previous tutorial we managed to create a Test Plan which we can execute manually and have a nice report on what requirements are met and where we still need to polish stuff to get the project up to the promised standards.
Manual testing is fine for a small project with only a handful of testcases. When things get serious and you have hundreds of tests to check the status of hundreds of software requirements it is nearly impossible to keep up, even with a whole QA team supporting you. This is where Jenkins comes in.
Many companies already use Jenkins to automate the testing process. However, there is a missing link between those test results and the requirements of a project. Some plugins try to close the gap, RobotFramework for instance, but they work from test cases up to requirements (or tags in RF’s case). Ideally what we want is the following:
- Have a set of requirements for the project and subsets for each release
- Have a Test Plan for each release that has a subset of all relevant tests for this release
- Have Jenkins execute the testcases in this Test Plan
- Have Jenkins return the results to the management system
- Have an always up to date report with the status of every requirement for this release
It sounds like a lot of work, but thanks to a Jenkins plugin for TestLink it’s fairly easy to accomplish. Let me tell you how.
First of all you will need a running Jenkins server. It is available on the Jenkins CI website. In this tutorial we’ll not be going into detail on how to set up such a server. It is pretty straightforward and there are many helpful guides available on the internet.
Once that’s done we need Jenkins to grab us the TestLink Plugin. Next up is configuring the plugin, visit the “Configure System” page in the Manage Jenkins section. You’ll find the following:
Make sure your URL is set properly. The plugin still uses a location older versions of TestLink used. To obtain your own Developer Key head over to TestLink and go to your account settings page (the person with a pencil icon). Click “Generate a new key” in the API Interface section and copy paste this string into the Dev key field on the Jenkins configuration page.
The next step is creating a Jenkins job. In the Build section, add the “Invoke TestLink” action. The action consists of (at least) three steps.
In this step you select your active TestLink server, tell the plugin what project to use, what plan to execute and what to call the build it will automatically add to the test plan in TestLink. In our case here’s what it should look like:
There are several (advanced) options in this step. For simplicity’s sake I’ll be using only the Single Build Steps which execute only once, before executing the Iterative Test Build Steps. The latter step is usually where you will invoke your test execution e.g. by passing a command line argument based on the test’s name, id or custom field (we’ll come back to this). Just to show the output we put some echo statements for now:
Result Seeking Strategy
Common testing frameworks (JUnit, NUnit, …) save the exeuction results in xml files. Somehow we need Jenkins to push all test results back to TestLink. The mapping is quite difficult as either system has no notion of the other. That’s why we have result seeking strategies. You can map a test case in TestLink to a result xml file in Jenkins by some unique field, e.g. Method or Class name.
This is where the Custom Field comes into play, which will be explained in just a minute. Let’s first look at a possible configuration:
With all of this configured, let’s give the job a test run. Hit the Build Now button, if everything went well you should see a Console Output like this:
Started by user anonymous Preparing TestLink client API. Using TestLink URL: http://localhost/tl_tut/lib/api/xmlrpc/v1/xmlrpc.php Found 3 automated test cases in TestLink. Sorting automated test cases by TestLink test plan execution order. Executing single Build Steps. Executing TL plan Version 0.1, part of the Angry Canaries project Executing iterative Build Steps. Running test 21 New Area Creation Running test 23 Obstacle placement Running test 19 Game Over Screen Audio Looking for the test results of TestLink test cases. Looking for test results in JUnit classes by its name. Found 3 test result(s). Finished: SUCCESS
Note that I stripped out some lines to keep the log short.
Jenkins fetched the active testcases in our test plan and ran the echo we wanted. Note that there are no results pushed back to TestLink yet, for that we need a custom field.
As mentioned before, these fields can be used to provide Jenkins with additional information about the test case as well as give it the opportunity to map the test result to a test case in TestLink.
Let’s create such a field in TestLink. Head over to the Define Custom Fields page. I always like to define a custom field specifically for the linking of TestLink and Jenkins, hence the name in the following config:
Now click the Add to Current Test Project button to enable it immediately. You can manage custom fields afterwards for your project by going to the Assign custom fields page.
For now, go over the couple of test cases we created and copy paste their title into their TL-Jenkins field. Also make sure to add TL-Jenkins to the TestLink Configuration and Result Seeking Strategy custom field configuration in your Jenkins job.
Actual Test Execution
This is the part where you’d use your test framework to execute your tests. However to understand how TestLink and Jenkins work together I’ll be using a simple Python script that accepts the Test Case Name as a command line argument and pass/fails it randomly.
import sys import xml.etree.ElementTree as ET from random import randint tc_result = False tc_name = sys.argv print "Test Framework will now execute " + tc_name random_nr = randint(0,1) if random_nr == 1: tc_result = True print "The test result was " + str(tc_result) root = ET.Element('testsuite') child = ET.Element('testcase', name=tc_name, classname=tc_name, time=str(10)) root.append(child) if tc_result is False: result = ET.Element('failure', message="Error 101") result.text = "We randomly failed!" child.append(result) et = ET.ElementTree(root) xmlname = "junit-"+ tc_name + ".xml" et.write(xmlname) print "Written result to " + xmlname
Save the above Python script to executeTest.py and put it somewhere Jenkins will be able to access it. Here’s how my Iterative Test Build Steps configuration looks like now:
Execute your job and go to its status page, it should look something like this:
Looking at the Console Output you can see the plugin found a mapping for the executed testcases:
[...] Running test 19 Game Over Screen Audio + python /Users/Shared/executeTest.py 'Game Over Screen Audio' Test Framework will now execute Game Over Screen Audio The test result was False Written result to junit-Game Over Screen Audio.xml Looking for the test results of TestLink test cases. Looking for test results in JUnit classes by its name. Updating TestLink test cases. Updating TestLink test cases. Updating TestLink test cases. Found 3 test result(s). There are failed tests, setting the build result as UNSTABLE. Build step 'Invoke TestLink' changed build result to UNSTABLE Finished: UNSTABLE
Hopping over to TestLink, check out the Requirement Based Report, you’ll see that a build has been added to the dropdown box, to see the results of that particular build that Jenkins ran for you, select it and hit Apply.
Notice that “Title Screen Audio” is set to Not Run, this is because in an earlier tutorial we have set this test case to manual execution, this is why Jenkins will not execute it.
That’s all there is to it. Now you can create test plans, add test cases to it and tell Jenkins to execute it. Results will come back to TestLink to give you a nice overview in the Requirement Based Report.
In the next tutorial I’ll explain how to use Mantis as an issue tracker together with TestLink. Stay tuned!