Jun 21

Test Management using TestLink – Pt III

Welcome to part III in my TestLink tutorial series. In the previous tutorial we managed to create a Test Plan which we can execute manually and have a nice report on what requirements are met and where we still need to polish stuff to get the project up to the promised standards.

Manual testing is fine for a small project with only a handful of testcases. When things get serious and you have hundreds of tests to check the status of hundreds of software requirements it is nearly impossible to keep up, even with a whole QA team supporting you. This is where Jenkins comes in.

Many companies already use Jenkins to automate the testing process. However, there is a missing link between those test results and the requirements of a project. Some plugins try to close the gap, RobotFramework for instance, but they work from test cases up to requirements (or tags in RF’s case). Ideally what we want is the following:

  1. Have a set of requirements for the project and subsets for each release
  2. Have a Test Plan for each release that has a subset of all relevant tests for this release
  3. Have Jenkins execute the testcases in this Test Plan
  4. Have Jenkins return the results to the management system
  5. Have an always up to date report with the status of every requirement for this release

It sounds like a lot of work, but thanks to a Jenkins plugin for TestLink it’s fairly easy to accomplish. Let me tell you how.

First of all you will need a running Jenkins server. It is available on the Jenkins CI website. In this tutorial we’ll not be going into detail on how to set up such a server. It is pretty straightforward and there are many helpful guides available on the internet.

Once that’s done we need Jenkins to grab us the TestLink Plugin. Next up is configuring the plugin, visit the “Configure System” page in the Manage Jenkins section. You’ll find the following:

TestLink Plugin Config

TestLink Plugin Config

Make sure your URL is set properly. The plugin still uses a location older versions of TestLink used. To obtain your own Developer Key head over to TestLink and go to your account settings page (the person with a pencil icon). Click “Generate a new key” in the API Interface section and copy paste this string into the Dev key field on the Jenkins configuration page.

The next step is creating a Jenkins job. In the Build section, add the “Invoke TestLink” action. The action consists of (at least) three steps.

TestLink Configuration

In this step you select your active TestLink server, tell the plugin what project to use, what plan to execute and what to call the build it will automatically add to the test plan in TestLink. In our case here’s what it should look like:

TestLink Configuration step

TestLink Configuration step

Test Execution

There are several (advanced) options in this step. For simplicity’s sake I’ll be using only the Single Build Steps which execute only once, before executing the Iterative Test Build Steps. The latter step is usually where you will invoke your test execution e.g. by passing a command line argument based on the test’s name, id or custom field (we’ll come back to this). Just to show the output we put some echo statements for now:

Test Execution step

Test Execution step

Result Seeking Strategy

Common testing frameworks (JUnit, NUnit, …) save the exeuction results in xml files. Somehow we need Jenkins to push all test results back to TestLink. The mapping is quite difficult as either system has no notion of the other. That’s why we have result seeking strategies. You can map a test case in TestLink to a result xml file in Jenkins by some unique field, e.g. Method or Class name.

This is where the Custom Field comes into play, which will be explained in just a minute. Let’s first look at a possible configuration:

Result Seeking Strategy

Result Seeking Strategy

Execution

With all of this configured, let’s give the job a test run. Hit the Build Now button, if everything went well you should see a Console Output like this:

Started by user anonymous
Preparing TestLink client API.
Using TestLink URL: http://localhost/tl_tut/lib/api/xmlrpc/v1/xmlrpc.php
Found 3 automated test cases in TestLink.
Sorting automated test cases by TestLink test plan execution order.

Executing single Build Steps.

Executing TL plan Version 0.1, part of the Angry Canaries project
Executing iterative Build Steps.

Running test 21 New Area Creation
Running test 23 Obstacle placement
Running test 19 Game Over Screen Audio
Looking for the test results of TestLink test cases.
Looking for test results in JUnit classes by its name.
Found 3 test result(s).

Finished: SUCCESS

Note that I stripped out some lines to keep the log short.

Jenkins fetched the active testcases in our test plan and ran the echo we wanted. Note that there are no results pushed back to TestLink yet, for that we need a custom field.

Custom Fields

As mentioned before, these fields can be used to provide Jenkins with additional information about the test case as well as give it the opportunity to map the test result to a test case in TestLink.

Let’s create such a field in TestLink. Head over to the Define Custom Fields page. I always like to define a custom field specifically for the linking of TestLink and Jenkins, hence the name in the following config:

Custom Field

Custom Field

Now click the Add to Current Test Project button to enable it immediately. You can manage custom fields afterwards for your project by going to the Assign custom fields page.

For now, go over the couple of test cases we created and copy paste their title into their TL-Jenkins field. Also make sure to add TL-Jenkins to the TestLink Configuration and Result Seeking Strategy custom field configuration in your Jenkins job.

Actual Test Execution

This is the part where you’d use your test framework to execute your tests. However to understand how TestLink and Jenkins work together I’ll be using a simple Python script that accepts the Test Case Name as a command line argument and pass/fails it randomly.

import sys
import xml.etree.ElementTree as ET
from random import randint
 
tc_result = False
tc_name = sys.argv[1]
 
print "Test Framework will now execute " + tc_name
 
random_nr = randint(0,1)
 
if random_nr == 1:
 tc_result = True
 
print "The test result was " + str(tc_result)
 
root = ET.Element('testsuite')
child = ET.Element('testcase', name=tc_name, classname=tc_name, time=str(10))
root.append(child)
if tc_result is False:
 result = ET.Element('failure', message="Error 101")
 result.text = "We randomly failed!"
 child.append(result)
 
et = ET.ElementTree(root)
xmlname = "junit-"+ tc_name + ".xml"
et.write(xmlname)
print "Written result to " + xmlname

Save the above Python script to executeTest.py and put it somewhere Jenkins will be able to access it. Here’s how my Iterative Test Build Steps configuration looks like now:

Iterative build step

Iterative build step

Execute your job and go to its status page, it should look something like this:

Jenkins Job status

Jenkins Job status

Looking at the Console Output you can see the plugin found a mapping for the executed testcases:

[...]
Running test 19 Game Over Screen Audio
+ python /Users/Shared/executeTest.py 'Game Over Screen Audio'
Test Framework will now execute Game Over Screen Audio
The test result was False
Written result to junit-Game Over Screen Audio.xml
Looking for the test results of TestLink test cases.

Looking for test results in JUnit classes by its name.

Updating TestLink test cases.

Updating TestLink test cases.

Updating TestLink test cases.

Found 3 test result(s).

There are failed tests, setting the build result as UNSTABLE.
Build step 'Invoke TestLink' changed build result to UNSTABLE
Finished: UNSTABLE

Hopping over to TestLink, check out the Requirement Based Report, you’ll see that a build has been added to the dropdown box, to see the results of that particular build that Jenkins ran for you, select it and hit Apply.

Requirement based Report

Requirement based Report

Notice that “Title Screen Audio” is set to Not Run, this is because in an earlier tutorial we have set this test case to manual execution, this is why Jenkins will not execute it.

Conclusion

That’s all there is to it. Now you can create test plans, add test cases to it and tell Jenkins to execute it. Results will come back to TestLink to give you a nice overview in the Requirement Based Report.

In the next tutorial I’ll explain how to use Mantis as an issue tracker together with TestLink. Stay tuned!

May 14

Test Management using TestLink – Pt II

In the previous part of the tutorial we installed and configured our TestLink installation. Now it’s time to create a Test Project and its requirements.

Browse to your TestLink URL and log in. As you see, TestLink needs you to create at least one project before you can start working with it.

Test Project Creation

Some information on the “Enhanced features”

  • Requirements: Check if you want to use requirements tracing
  • Testing Priority: Check if you want to have testcases have priority over others
  • Test Automation: Check if you want to have an API available (e.g. for Jenkins or Python scripts)
  • Inventory: Check if you want to have a section where you can track servers, items, …

The next step is to input the requirements for your product. Click on Requirement Specification

Requirements are grouped by Specifications in TestLink, you could think of them as folders to group requirements that belong together. For the purpose of this tutorial let’s make two Specifications, called Level Designer and Audiovisual.

Now click on Audiovisual in the document tree on the left. To create a requirement click on the create button under Requirement Operations (hit the cogwheel if the buttons are not visible).

Manually adding requirements is a tedious process, many projects have documents or spreadsheets that contain the requirements. TestLink supports importing requirements in CSV or Excel files. Let’s create a couple of requirements manually for now though. This is what I ended up with:

Requirements Tree

 Congratulations! You’ve just accomplished requirements traceability in your project, well, the first step of it at least. It’s a good start and now you’ll need a Test Plan with Test Cases that can be executed to see if those requirements are being met or not.

Hop over to Test Plan Management and let’s create our first Test Plan.

Test Plan creation

Okay we’re nearly there on the management side of things. Let’s continue by creation a couple of Test Cases. TestLink allows you to generate Test Cases based upon the Requirements if you would like to do that. However, in this tutorial we’ll go full manual for now, visit Test Specification.

Here we find something that is most likely familiar to you if you have been involved in software testing. The use of Test Suites that hold several relevant Test Cases. These can also be imported from an XML file with the correct structure. You can find out more on how to arrange your Test Suites and Test Cases in such an XML by reading the File Formats Documentation document in TestLink. Let’s create a couple manually for now:

Test Case List

You might think this new Test Case creation form is pretty limited in what you can specify. However when you click on a Test Case in the tree on the left you can supply more information such as execution type (manual or automatic, select the latter if you’d like Jenkins to execute it later on) and execution steps.

Test Case Detailed

Now the most important part, linking these Test Cases to the Requirements we created earlier. This will allow for Requirements Traceability once we’re going to execute these Test Cases. So let’s check out the Add Requirements page.

Here you can either bulk assign requirements to tests in a Test Suite or manually pick them for a selected Test Case. Let’s mix and match the ones we’ve made. It’s pretty obvious to do, an example:

Requirement Assignment

Note that you can also assign test cases to Requirements in two other places, namely in the Edit Requirement view and the Edit Test Case view.

The second last step we need to take is assigning these Test Cases to our previously created Test Plan, so that when we want to execute the Test Plan we get a list of all the TCs we need to process. Click on Add/Remove Test Cases on the main page. Select a Test Suite and enable the TCs in the list. Active Test Cases for the selected plan will be highlighted.

Add/Remove TCs

You need a defined version of your software to test against. In TestLink these are called Builds (cfr Jenkins builds). So the last thing we need to do now is add a build in TestLink, do so by going to Builds / Releases.

Build Creation

Phew! That was quite a bit of work don’t you think? But we’re there! We can now have fun testing our product, using our test plan as a guide and executing test cases based on the steps we’ve defined in each of them.

In the next tutorial I’ll explain how to automate test case execution using Jenkins. But for now have a look at Metrics Dashboard and Test Reports and Metrics > Requirement Based Report

Requirements Based ReportNeat isn’t it? We get a nice overview of all our requirements, all linked test cases and whether they have been run, have passed or have failed. I do have to say that it is unfortunate that you cannot add Requirements to a Test Plan. The requirements that show up here are the ones which have test cases that are active in the selected Test Plan.

You do need to be careful that if you have a set of requirements for a Test Plan that each and every of those requirements have at least one test case linked to them that is active in that particular Test Plan else it will not show up in this overview at all.

Next time is sure going to be interesting… Test automation using Jenkins based on your Test Plans in TestLink!

May 09

Test Management using TestLink – Pt I

I’ve been using TestLink for a while now and would like to share some of the knowledge I’ve gained during this period. In this tutorial series I’ll tackle the whole process from start to finish to set up a system that uses TestLink for Requirements and Test Project/Plan/Case management along with Test Plan execution using Jenkins.

Topics we’re going to handle include

  • Setting up TestLink
  • Creating content in TestLink (Projects, Plans, Requirements, Test Cases, …)
  • Issue Tracker integration
  • Pulling your Test Plan into Jenkins for test executions
  • Pushing the results back to TestLink

What I’ll be using in this tutorial is

If you do not have access to a webhost that has MySQL (or MSSQL) and an up to date version of PHP you’ll also want to install a webserver on your own PC (or your Jenkins machine) for experimenting purposes. In case you are not familiar with setting up a webserver or database I’d recommend you get

It doesn’t really matter what version of the software mentioned you get. Jenkins seems to release update very frequently while TestLink gets about a new release per quarter. Things might be slightly different in newer versions but it shouldn’t be a problem to follow the tutorial.

Let’s get started installing TestLink! Surf to the location of your TestLink instance, if you’re doing this on your own computer it’s probably something along the lines of http://localhost/testlink1_9_13/, you’ll be greeted by this page

TestLink Install

If you’re doing a fresh install just hit the “New Installation” link and agree to the license agreement. The following page checks if your setup meets the requirements. If you’re running this on your own computer you might need to resolve write access issues before being able to continue.

TL Install requirment check

Next we need to configure the database access. Select which DBMS you want to use (I’ll be using MySQL in this tutorial). Enter a user with enough “admin” rights and fill in a username and password which TestLink will use to access the database (this is NOT your login info for TestLink itself. Default credentials are admin/admin). Let’s continue!

Install report

In the case that TestLink could not write your config file, you’ll have to manually create the file and paste the given contents into it. Now everything is set up, browse to the root TestLink URL and you’ll be presented with a login screen.

Note that you can do a whole lot of configuration in TestLink but it’s not accessible in the GUI. You need to open up config.inc.php in a text editor and manually change the values there. I suggest you scroll through it and have a look what you can do. To be safe, make a backup of the file before you start editting it.

Some default behaviour I liked changing included the following

  • Change the path of the upload directory and log files
 $g_repositoryPath = '/var/testlink/upload_area/'; /* unix example */
 $tlCfg->log_path = '/var/testlink/logs/'; /* unix example */
  • Do not hide the available actions on pages by changing the value to inline
 $tlCfg->gui->op_area_display->test_spec_container = 'none'; // ''
 $tlCfg->gui->op_area_display->test_case = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req_spec_container = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req = 'none'; // 'inline'

There are many, many more things that are possibly a good idea to change depending on your needs, just keep in mind that all of these settings are in this file for future reference.

Congratulations! You’ve just installed and configured TestLink. When you log in you’ll be asked to create a Test Project before you can continue. I’ll be talking about this and more in the next tutorial.