Jun 21

Test Management using TestLink – Pt III

Welcome to part III in my TestLink tutorial series. In the previous tutorial we managed to create a Test Plan which we can execute manually and have a nice report on what requirements are met and where we still need to polish stuff to get the project up to the promised standards.

Manual testing is fine for a small project with only a handful of testcases. When things get serious and you have hundreds of tests to check the status of hundreds of software requirements it is nearly impossible to keep up, even with a whole QA team supporting you. This is where Jenkins comes in.

Many companies already use Jenkins to automate the testing process. However, there is a missing link between those test results and the requirements of a project. Some plugins try to close the gap, RobotFramework for instance, but they work from test cases up to requirements (or tags in RF’s case). Ideally what we want is the following:

  1. Have a set of requirements for the project and subsets for each release
  2. Have a Test Plan for each release that has a subset of all relevant tests for this release
  3. Have Jenkins execute the testcases in this Test Plan
  4. Have Jenkins return the results to the management system
  5. Have an always up to date report with the status of every requirement for this release

It sounds like a lot of work, but thanks to a Jenkins plugin for TestLink it’s fairly easy to accomplish. Let me tell you how.

First of all you will need a running Jenkins server. It is available on the Jenkins CI website. In this tutorial we’ll not be going into detail on how to set up such a server. It is pretty straightforward and there are many helpful guides available on the internet.

Once that’s done we need Jenkins to grab us the TestLink Plugin. Next up is configuring the plugin, visit the “Configure System” page in the Manage Jenkins section. You’ll find the following:

TestLink Plugin Config

TestLink Plugin Config

Make sure your URL is set properly. The plugin still uses a location older versions of TestLink used. To obtain your own Developer Key head over to TestLink and go to your account settings page (the person with a pencil icon). Click “Generate a new key” in the API Interface section and copy paste this string into the Dev key field on the Jenkins configuration page.

The next step is creating a Jenkins job. In the Build section, add the “Invoke TestLink” action. The action consists of (at least) three steps.

TestLink Configuration

In this step you select your active TestLink server, tell the plugin what project to use, what plan to execute and what to call the build it will automatically add to the test plan in TestLink. In our case here’s what it should look like:

TestLink Configuration step

TestLink Configuration step

Test Execution

There are several (advanced) options in this step. For simplicity’s sake I’ll be using only the Single Build Steps which execute only once, before executing the Iterative Test Build Steps. The latter step is usually where you will invoke your test execution e.g. by passing a command line argument based on the test’s name, id or custom field (we’ll come back to this). Just to show the output we put some echo statements for now:

Test Execution step

Test Execution step

Result Seeking Strategy

Common testing frameworks (JUnit, NUnit, …) save the exeuction results in xml files. Somehow we need Jenkins to push all test results back to TestLink. The mapping is quite difficult as either system has no notion of the other. That’s why we have result seeking strategies. You can map a test case in TestLink to a result xml file in Jenkins by some unique field, e.g. Method or Class name.

This is where the Custom Field comes into play, which will be explained in just a minute. Let’s first look at a possible configuration:

Result Seeking Strategy

Result Seeking Strategy

Execution

With all of this configured, let’s give the job a test run. Hit the Build Now button, if everything went well you should see a Console Output like this:

Started by user anonymous
Preparing TestLink client API.
Using TestLink URL: http://localhost/tl_tut/lib/api/xmlrpc/v1/xmlrpc.php
Found 3 automated test cases in TestLink.
Sorting automated test cases by TestLink test plan execution order.

Executing single Build Steps.

Executing TL plan Version 0.1, part of the Angry Canaries project
Executing iterative Build Steps.

Running test 21 New Area Creation
Running test 23 Obstacle placement
Running test 19 Game Over Screen Audio
Looking for the test results of TestLink test cases.
Looking for test results in JUnit classes by its name.
Found 3 test result(s).

Finished: SUCCESS

Note that I stripped out some lines to keep the log short.

Jenkins fetched the active testcases in our test plan and ran the echo we wanted. Note that there are no results pushed back to TestLink yet, for that we need a custom field.

Custom Fields

As mentioned before, these fields can be used to provide Jenkins with additional information about the test case as well as give it the opportunity to map the test result to a test case in TestLink.

Let’s create such a field in TestLink. Head over to the Define Custom Fields page. I always like to define a custom field specifically for the linking of TestLink and Jenkins, hence the name in the following config:

Custom Field

Custom Field

Now click the Add to Current Test Project button to enable it immediately. You can manage custom fields afterwards for your project by going to the Assign custom fields page.

For now, go over the couple of test cases we created and copy paste their title into their TL-Jenkins field. Also make sure to add TL-Jenkins to the TestLink Configuration and Result Seeking Strategy custom field configuration in your Jenkins job.

Actual Test Execution

This is the part where you’d use your test framework to execute your tests. However to understand how TestLink and Jenkins work together I’ll be using a simple Python script that accepts the Test Case Name as a command line argument and pass/fails it randomly.

import sys
import xml.etree.ElementTree as ET
from random import randint
 
tc_result = False
tc_name = sys.argv[1]
 
print "Test Framework will now execute " + tc_name
 
random_nr = randint(0,1)
 
if random_nr == 1:
 tc_result = True
 
print "The test result was " + str(tc_result)
 
root = ET.Element('testsuite')
child = ET.Element('testcase', name=tc_name, classname=tc_name, time=str(10))
root.append(child)
if tc_result is False:
 result = ET.Element('failure', message="Error 101")
 result.text = "We randomly failed!"
 child.append(result)
 
et = ET.ElementTree(root)
xmlname = "junit-"+ tc_name + ".xml"
et.write(xmlname)
print "Written result to " + xmlname

Save the above Python script to executeTest.py and put it somewhere Jenkins will be able to access it. Here’s how my Iterative Test Build Steps configuration looks like now:

Iterative build step

Iterative build step

Execute your job and go to its status page, it should look something like this:

Jenkins Job status

Jenkins Job status

Looking at the Console Output you can see the plugin found a mapping for the executed testcases:

[...]
Running test 19 Game Over Screen Audio
+ python /Users/Shared/executeTest.py 'Game Over Screen Audio'
Test Framework will now execute Game Over Screen Audio
The test result was False
Written result to junit-Game Over Screen Audio.xml
Looking for the test results of TestLink test cases.

Looking for test results in JUnit classes by its name.

Updating TestLink test cases.

Updating TestLink test cases.

Updating TestLink test cases.

Found 3 test result(s).

There are failed tests, setting the build result as UNSTABLE.
Build step 'Invoke TestLink' changed build result to UNSTABLE
Finished: UNSTABLE

Hopping over to TestLink, check out the Requirement Based Report, you’ll see that a build has been added to the dropdown box, to see the results of that particular build that Jenkins ran for you, select it and hit Apply.

Requirement based Report

Requirement based Report

Notice that “Title Screen Audio” is set to Not Run, this is because in an earlier tutorial we have set this test case to manual execution, this is why Jenkins will not execute it.

Conclusion

That’s all there is to it. Now you can create test plans, add test cases to it and tell Jenkins to execute it. Results will come back to TestLink to give you a nice overview in the Requirement Based Report.

In the next tutorial I’ll explain how to use Mantis as an issue tracker together with TestLink. Stay tuned!

May 09

Test Management using TestLink – Pt I

I’ve been using TestLink for a while now and would like to share some of the knowledge I’ve gained during this period. In this tutorial series I’ll tackle the whole process from start to finish to set up a system that uses TestLink for Requirements and Test Project/Plan/Case management along with Test Plan execution using Jenkins.

Topics we’re going to handle include

  • Setting up TestLink
  • Creating content in TestLink (Projects, Plans, Requirements, Test Cases, …)
  • Issue Tracker integration
  • Pulling your Test Plan into Jenkins for test executions
  • Pushing the results back to TestLink

What I’ll be using in this tutorial is

If you do not have access to a webhost that has MySQL (or MSSQL) and an up to date version of PHP you’ll also want to install a webserver on your own PC (or your Jenkins machine) for experimenting purposes. In case you are not familiar with setting up a webserver or database I’d recommend you get

It doesn’t really matter what version of the software mentioned you get. Jenkins seems to release update very frequently while TestLink gets about a new release per quarter. Things might be slightly different in newer versions but it shouldn’t be a problem to follow the tutorial.

Let’s get started installing TestLink! Surf to the location of your TestLink instance, if you’re doing this on your own computer it’s probably something along the lines of http://localhost/testlink1_9_13/, you’ll be greeted by this page

TestLink Install

If you’re doing a fresh install just hit the “New Installation” link and agree to the license agreement. The following page checks if your setup meets the requirements. If you’re running this on your own computer you might need to resolve write access issues before being able to continue.

TL Install requirment check

Next we need to configure the database access. Select which DBMS you want to use (I’ll be using MySQL in this tutorial). Enter a user with enough “admin” rights and fill in a username and password which TestLink will use to access the database (this is NOT your login info for TestLink itself. Default credentials are admin/admin). Let’s continue!

Install report

In the case that TestLink could not write your config file, you’ll have to manually create the file and paste the given contents into it. Now everything is set up, browse to the root TestLink URL and you’ll be presented with a login screen.

Note that you can do a whole lot of configuration in TestLink but it’s not accessible in the GUI. You need to open up config.inc.php in a text editor and manually change the values there. I suggest you scroll through it and have a look what you can do. To be safe, make a backup of the file before you start editting it.

Some default behaviour I liked changing included the following

  • Change the path of the upload directory and log files
 $g_repositoryPath = '/var/testlink/upload_area/'; /* unix example */
 $tlCfg->log_path = '/var/testlink/logs/'; /* unix example */
  • Do not hide the available actions on pages by changing the value to inline
 $tlCfg->gui->op_area_display->test_spec_container = 'none'; // ''
 $tlCfg->gui->op_area_display->test_case = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req_spec_container = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req = 'none'; // 'inline'

There are many, many more things that are possibly a good idea to change depending on your needs, just keep in mind that all of these settings are in this file for future reference.

Congratulations! You’ve just installed and configured TestLink. When you log in you’ll be asked to create a Test Project before you can continue. I’ll be talking about this and more in the next tutorial.

Dec 29

Traceability and requirements coverage with TestLink and Jenkins

If you’re working on a software/hardware project chances are you have a list of requirements that have to be checked off by running (automated) tests that confirm the requirement is met.

I’ve seen a lot of projects using Jenkins to automatically build and compile the software and run tests to check which existing testcases pass and which are failing with every new push to the code repository. That’s great, in the blink of an eye you see if something broke: if the number of failing testcases rises you might want to take a closer look for regression issues.

The problem with only using Jenkins is traceability.
Imagine this: you have 450 testcases, they all check a piece of the functionality of your SW/HW. If a couple of them fail, can you immediately say which requirement is not met? Unless you have a very structured way of ordering your test suites this probably is not the case. This is where TestLink comes into play.

Requirement management with TestLink

TestLink is a test and requirements management framework, completely opensource and written in PHP. It has an API which you can use even if you’re using custom or proprietary tools to execute your testcases. With said API it is very easy to return test results to TestLink automatically.

I had to deal with this kind of situation before, a whole bunch of testcases that did not link back to any requirements. That’s when I decided to expand upon the existing Jenkins configuration by adding TestLink to the mix.

A list of requirements was available which was imported into my TestLink set up. Using a quickly whipped together Python script I converted the proprietary testset XML files into something that could be imported into TestLink. A couple of minutes later I had a list of requirements and a (huge) amount of testcase entries. Manually adding requirements to each and every testcase was long and tedious work, luckily you only have to do it once. I have to admit that the TestLink user interface has been stuck in the nillies and could benefit from an overhaul but once you get used to it, it’s pretty straightforward.

Combining Jenkins and TestLink

Next up was using Jenkins to feed the results back to TestLink. The Jenkins configuration was set in such a way that a build job is spawned when a change in the repo was found. After the build job is successful a second job is spawned that executes all testsets using an in-house framework that generates jUnit XML output. To be able to match the jUnit XML files to the testcases in TestLink I used a handy Jenkins plugin.

The matching criterium is simple, using the testcase name in the “result seeking strategy” of the plugin I generated a custom field value for each testcase in TestLink which was a simplified version of the testcase name. Whenever the plugin matches this custom field value with the testcase name in the jUnit output it updates the result in TestLink.

In the build job configuration you also have to specify a build number, with the Copy Artifacts plugin I could use an environment variable that contains the Jenkins build number of the job that compiled the code. This way you can run your test job multiple times even if there is no new version of your software. TestLink will create a record for each run of a testcase so you can quickly check its health using the execution history and see the progress over time with each build number.

Conclusion

Using these tools I can quickly have TestLink generate a report that gives me an overview of the health per testsuite, how much % of the requirements have been covered and how much of that has tests that pass. It also allows you to create reports (e.g. in PDF format) to present to your project or QA manager and so much more!

To be fair, I haven’t used the full power of TestLink yet. You can manage what tests get executed for every build or milestone and have the Jenkins plugin use that info to trigger execution of those testcases. I’ve not used this functionality yet however.

It’s a first step, going from 400 testcases with no traceability to having a requirement management framework that allows you to easily follow up on requirements coverage. If you’re having trouble keeping your requirements and testcases organized I’d recommend you give TestLink a go.