Welcome to part III in my TestLink tutorial series. In the previous tutorial we managed to create a Test Plan which we can execute manually and have a nice report on what requirements are met and where we still need to polish stuff to get the project up to the promised standards.

Manual testing is fine for a small project with only a handful of testcases. When things get serious and you have hundreds of tests to check the status of hundreds of software requirements it is nearly impossible to keep up, even with a whole QA team supporting you. This is where Jenkins comes in.

Many companies already use Jenkins to automate the testing process. However, there is a missing link between those test results and the requirements of a project. Some plugins try to close the gap, RobotFramework for instance, but they work from test cases up to requirements (or tags in RF’s case). Ideally what we want is the following:

1. Have a set of requirements for the project and subsets for each release
2. Have a Test Plan for each release that has a subset of all relevant tests for this release
3. Have Jenkins execute the testcases in this Test Plan
4. Have Jenkins return the results to the management system
5. Have an always up to date report with the status of every requirement for this release

It sounds like a lot of work, but thanks to a Jenkins plugin for TestLink it’s fairly easy to accomplish. Let me tell you how.

First of all you will need a running Jenkins server. It is available on the Jenkins CI website. In this tutorial we’ll not be going into detail on how to set up such a server. It is pretty straightforward and there are many helpful guides available on the internet.

Once that’s done we need Jenkins to grab us the TestLink Plugin. Next up is configuring the plugin, visit the “Configure System” page in the Manage Jenkins section. You’ll find the following:

TestLink Plugin Config

Make sure your URL is set properly. The plugin still uses a location older versions of TestLink used. To obtain your own Developer Key head over to TestLink and go to your account settings page (the person with a pencil icon). Click “Generate a new key” in the API Interface section and copy paste this string into the Dev key field on the Jenkins configuration page.

The next step is creating a Jenkins job. In the Build section, add the “Invoke TestLink” action. The action consists of (at least) three steps.

TestLink Configuration

In this step you select your active TestLink server, tell the plugin what project to use, what plan to execute and what to call the build it will automatically add to the test plan in TestLink. In our case here’s what it should look like:

TestLink Configuration step

Test Execution

There are several (advanced) options in this step. For simplicity’s sake I’ll be using only the Single Build Steps which execute only once, before executing the Iterative Test Build Steps. The latter step is usually where you will invoke your test execution e.g. by passing a command line argument based on the test’s name, id or custom field (we’ll come back to this). Just to show the output we put some echo statements for now:

Test Execution step

Result Seeking Strategy

Common testing frameworks (JUnit, NUnit, …) save the exeuction results in xml files. Somehow we need Jenkins to push all test results back to TestLink. The mapping is quite difficult as either system has no notion of the other. That’s why we have result seeking strategies. You can map a test case in TestLink to a result xml file in Jenkins by some unique field, e.g. Method or Class name.

This is where the Custom Field comes into play, which will be explained in just a minute. Let’s first look at a possible configuration:

Result Seeking Strategy

Execution

With all of this configured, let’s give the job a test run. Hit the Build Now button, if everything went well you should see a Console Output like this:

Started by user anonymous
Preparing TestLink client API.
Using TestLink URL: http://localhost/tl_tut/lib/api/xmlrpc/v1/xmlrpc.php
Found 3 automated test cases in TestLink.
Sorting automated test cases by TestLink test plan execution order.

Executing single Build Steps.

Executing TL plan Version 0.1, part of the Angry Canaries project
Executing iterative Build Steps.

Running test 21 New Area Creation
Running test 23 Obstacle placement
Running test 19 Game Over Screen Audio
Looking for the test results of TestLink test cases.
Looking for test results in JUnit classes by its name.
Found 3 test result(s).

Finished: SUCCESS

Note that I stripped out some lines to keep the log short.

Jenkins fetched the active testcases in our test plan and ran the echo we wanted. Note that there are no results pushed back to TestLink yet, for that we need a custom field.

Custom Fields

As mentioned before, these fields can be used to provide Jenkins with additional information about the test case as well as give it the opportunity to map the test result to a test case in TestLink.

Let’s create such a field in TestLink. Head over to the Define Custom Fields page. I always like to define a custom field specifically for the linking of TestLink and Jenkins, hence the name in the following config:

Custom Field

Now click the Add to Current Test Project button to enable it immediately. You can manage custom fields afterwards for your project by going to the Assign custom fields page.

For now, go over the couple of test cases we created and copy paste their title into their TL-Jenkins field. Also make sure to add TL-Jenkins to the TestLink Configuration and Result Seeking Strategy custom field configuration in your Jenkins job.

Actual Test Execution

This is the part where you’d use your test framework to execute your tests. However to understand how TestLink and Jenkins work together I’ll be using a simple Python script that accepts the Test Case Name as a command line argument and pass/fails it randomly.

import sys
import xml.etree.ElementTree as ET
from random import randint
 
tc_result = False
tc_name = sys.argv[1]
 
print "Test Framework will now execute " + tc_name
 
random_nr = randint(0,1)
 
if random_nr == 1:
 tc_result = True
 
print "The test result was " + str(tc_result)
 
root = ET.Element('testsuite')
child = ET.Element('testcase', name=tc_name, classname=tc_name, time=str(10))
root.append(child)
if tc_result is False:
 result = ET.Element('failure', message="Error 101")
 result.text = "We randomly failed!"
 child.append(result)
 
et = ET.ElementTree(root)
xmlname = "junit-"+ tc_name + ".xml"
et.write(xmlname)
print "Written result to " + xmlname

Save the above Python script to executeTest.py and put it somewhere Jenkins will be able to access it. Here’s how my Iterative Test Build Steps configuration looks like now:

Iterative build step

Execute your job and go to its status page, it should look something like this:

Jenkins Job status

Looking at the Console Output you can see the plugin found a mapping for the executed testcases:

[...]
Running test 19 Game Over Screen Audio
+ python /Users/Shared/executeTest.py 'Game Over Screen Audio'
Test Framework will now execute Game Over Screen Audio
The test result was False
Written result to junit-Game Over Screen Audio.xml
Looking for the test results of TestLink test cases.

Looking for test results in JUnit classes by its name.

Updating TestLink test cases.

Updating TestLink test cases.

Updating TestLink test cases.

Found 3 test result(s).

There are failed tests, setting the build result as UNSTABLE.
Build step 'Invoke TestLink' changed build result to UNSTABLE
Finished: UNSTABLE

Hopping over to TestLink, check out the Requirement Based Report, you’ll see that a build has been added to the dropdown box, to see the results of that particular build that Jenkins ran for you, select it and hit Apply.

Requirement based Report

Notice that “Title Screen Audio” is set to Not Run, this is because in an earlier tutorial we have set this test case to manual execution, this is why Jenkins will not execute it.

Conclusion

That’s all there is to it. Now you can create test plans, add test cases to it and tell Jenkins to execute it. Results will come back to TestLink to give you a nice overview in the Requirement Based Report.

In the next tutorial I’ll explain how to use Mantis as an issue tracker together with TestLink. Stay tuned!