Tag Archives: requirements

May 14

Test Management using TestLink – Pt II

Posted on by

In the previous part of the tutorial we installed and configured our TestLink installation. Now it’s time to create a Test Project and its requirements.

Browse to your TestLink URL and log in. As you see, TestLink needs you to create at least one project before you can start working with it.

Test Project Creation

Some information on the “Enhanced features”

  • Requirements: Check if you want to use requirements tracing
  • Testing Priority: Check if you want to have testcases have priority over others
  • Test Automation: Check if you want to have an API available (e.g. for Jenkins or Python scripts)
  • Inventory: Check if you want to have a section where you can track servers, items, …

The next step is to input the requirements for your product. Click on Requirement Specification

Requirements are grouped by Specifications in TestLink, you could think of them as folders to group requirements that belong together. For the purpose of this tutorial let’s make two Specifications, called Level Designer and Audiovisual.

Now click on Audiovisual in the document tree on the left. To create a requirement click on the create button under Requirement Operations (hit the cogwheel if the buttons are not visible).

Manually adding requirements is a tedious process, many projects have documents or spreadsheets that contain the requirements. TestLink supports importing requirements in CSV or Excel files. Let’s create a couple of requirements manually for now though. This is what I ended up with:

Requirements Tree

 Congratulations! You’ve just accomplished requirements traceability in your project, well, the first step of it at least. It’s a good start and now you’ll need a Test Plan with Test Cases that can be executed to see if those requirements are being met or not.

Hop over to Test Plan Management and let’s create our first Test Plan.

Test Plan creation

Okay we’re nearly there on the management side of things. Let’s continue by creation a couple of Test Cases. TestLink allows you to generate Test Cases based upon the Requirements if you would like to do that. However, in this tutorial we’ll go full manual for now, visit Test Specification.

Here we find something that is most likely familiar to you if you have been involved in software testing. The use of Test Suites that hold several relevant Test Cases. These can also be imported from an XML file with the correct structure. You can find out more on how to arrange your Test Suites and Test Cases in such an XML by reading the File Formats Documentation document in TestLink. Let’s create a couple manually for now:

Test Case List

You might think this new Test Case creation form is pretty limited in what you can specify. However when you click on a Test Case in the tree on the left you can supply more information such as execution type (manual or automatic, select the latter if you’d like Jenkins to execute it later on) and execution steps.

Test Case Detailed

Now the most important part, linking these Test Cases to the Requirements we created earlier. This will allow for Requirements Traceability once we’re going to execute these Test Cases. So let’s check out the Add Requirements page.

Here you can either bulk assign requirements to tests in a Test Suite or manually pick them for a selected Test Case. Let’s mix and match the ones we’ve made. It’s pretty obvious to do, an example:

Requirement Assignment

Note that you can also assign test cases to Requirements in two other places, namely in the Edit Requirement view and the Edit Test Case view.

The second last step we need to take is assigning these Test Cases to our previously created Test Plan, so that when we want to execute the Test Plan we get a list of all the TCs we need to process. Click on Add/Remove Test Cases on the main page. Select a Test Suite and enable the TCs in the list. Active Test Cases for the selected plan will be highlighted.

Add/Remove TCs

You need a defined version of your software to test against. In TestLink these are called Builds (cfr Jenkins builds). So the last thing we need to do now is add a build in TestLink, do so by going to Builds / Releases.

Build Creation

Phew! That was quite a bit of work don’t you think? But we’re there! We can now have fun testing our product, using our test plan as a guide and executing test cases based on the steps we’ve defined in each of them.

In the next tutorial I’ll explain how to automate test case execution using Jenkins. But for now have a look at Metrics Dashboard and Test Reports and Metrics > Requirement Based Report

Requirements Based ReportNeat isn’t it? We get a nice overview of all our requirements, all linked test cases and whether they have been run, have passed or have failed. I do have to say that it is unfortunate that you cannot add Requirements to a Test Plan. The requirements that show up here are the ones which have test cases that are active in the selected Test Plan.

You do need to be careful that if you have a set of requirements for a Test Plan that each and every of those requirements have at least one test case linked to them that is active in that particular Test Plan else it will not show up in this overview at all.

Next time is sure going to be interesting… Test automation using Jenkins based on your Test Plans in TestLink!

May 09

Test Management using TestLink – Pt I

Posted on by

I’ve been using TestLink for a while now and would like to share some of the knowledge I’ve gained during this period. In this tutorial series I’ll tackle the whole process from start to finish to set up a system that uses TestLink for Requirements and Test Project/Plan/Case management along with Test Plan execution using Jenkins.

Topics we’re going to handle include

  • Setting up TestLink
  • Creating content in TestLink (Projects, Plans, Requirements, Test Cases, …)
  • Issue Tracker integration
  • Pulling your Test Plan into Jenkins for test executions
  • Pushing the results back to TestLink

What I’ll be using in this tutorial is

If you do not have access to a webhost that has MySQL (or MSSQL) and an up to date version of PHP you’ll also want to install a webserver on your own PC (or your Jenkins machine) for experimenting purposes. In case you are not familiar with setting up a webserver or database I’d recommend you get

It doesn’t really matter what version of the software mentioned you get. Jenkins seems to release update very frequently while TestLink gets about a new release per quarter. Things might be slightly different in newer versions but it shouldn’t be a problem to follow the tutorial.

Let’s get started installing TestLink! Surf to the location of your TestLink instance, if you’re doing this on your own computer it’s probably something along the lines of http://localhost/testlink1_9_13/, you’ll be greeted by this page

TestLink Install

If you’re doing a fresh install just hit the “New Installation” link and agree to the license agreement. The following page checks if your setup meets the requirements. If you’re running this on your own computer you might need to resolve write access issues before being able to continue.

TL Install requirment check

Next we need to configure the database access. Select which DBMS you want to use (I’ll be using MySQL in this tutorial). Enter a user with enough “admin” rights and fill in a username and password which TestLink will use to access the database (this is NOT your login info for TestLink itself. Default credentials are admin/admin). Let’s continue!

Install report

In the case that TestLink could not write your config file, you’ll have to manually create the file and paste the given contents into it. Now everything is set up, browse to the root TestLink URL and you’ll be presented with a login screen.

Note that you can do a whole lot of configuration in TestLink but it’s not accessible in the GUI. You need to open up config.inc.php in a text editor and manually change the values there. I suggest you scroll through it and have a look what you can do. To be safe, make a backup of the file before you start editting it.

Some default behaviour I liked changing included the following

  • Change the path of the upload directory and log files
 $g_repositoryPath = '/var/testlink/upload_area/'; /* unix example */
 $tlCfg->log_path = '/var/testlink/logs/'; /* unix example */
  • Do not hide the available actions on pages by changing the value to inline
 $tlCfg->gui->op_area_display->test_spec_container = 'none'; // ''
 $tlCfg->gui->op_area_display->test_case = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req_spec_container = 'none'; // 'inline'
 $tlCfg->gui->op_area_display->req = 'none'; // 'inline'

There are many, many more things that are possibly a good idea to change depending on your needs, just keep in mind that all of these settings are in this file for future reference.

Congratulations! You’ve just installed and configured TestLink. When you log in you’ll be asked to create a Test Project before you can continue. I’ll be talking about this and more in the next tutorial.

Dec 29

Traceability and requirements coverage with TestLink and Jenkins

Posted on by

If you’re working on a software/hardware project chances are you have a list of requirements that have to be checked off by running (automated) tests that confirm the requirement is met.

I’ve seen a lot of projects using Jenkins to automatically build and compile the software and run tests to check which existing testcases pass and which are failing with every new push to the code repository. That’s great, in the blink of an eye you see if something broke: if the number of failing testcases rises you might want to take a closer look for regression issues.

The problem with only using Jenkins is traceability.
Imagine this: you have 450 testcases, they all check a piece of the functionality of your SW/HW. If a couple of them fail, can you immediately say which requirement is not met? Unless you have a very structured way of ordering your test suites this probably is not the case. This is where TestLink comes into play.

Requirement management with TestLink

TestLink is a test and requirements management framework, completely opensource and written in PHP. It has an API which you can use even if you’re using custom or proprietary tools to execute your testcases. With said API it is very easy to return test results to TestLink automatically.

I had to deal with this kind of situation before, a whole bunch of testcases that did not link back to any requirements. That’s when I decided to expand upon the existing Jenkins configuration by adding TestLink to the mix.

A list of requirements was available which was imported into my TestLink set up. Using a quickly whipped together Python script I converted the proprietary testset XML files into something that could be imported into TestLink. A couple of minutes later I had a list of requirements and a (huge) amount of testcase entries. Manually adding requirements to each and every testcase was long and tedious work, luckily you only have to do it once. I have to admit that the TestLink user interface has been stuck in the nillies and could benefit from an overhaul but once you get used to it, it’s pretty straightforward.

Combining Jenkins and TestLink

Next up was using Jenkins to feed the results back to TestLink. The Jenkins configuration was set in such a way that a build job is spawned when a change in the repo was found. After the build job is successful a second job is spawned that executes all testsets using an in-house framework that generates jUnit XML output. To be able to match the jUnit XML files to the testcases in TestLink I used a handy Jenkins plugin.

The matching criterium is simple, using the testcase name in the “result seeking strategy” of the plugin I generated a custom field value for each testcase in TestLink which was a simplified version of the testcase name. Whenever the plugin matches this custom field value with the testcase name in the jUnit output it updates the result in TestLink.

In the build job configuration you also have to specify a build number, with the Copy Artifacts plugin I could use an environment variable that contains the Jenkins build number of the job that compiled the code. This way you can run your test job multiple times even if there is no new version of your software. TestLink will create a record for each run of a testcase so you can quickly check its health using the execution history and see the progress over time with each build number.

Conclusion

Using these tools I can quickly have TestLink generate a report that gives me an overview of the health per testsuite, how much % of the requirements have been covered and how much of that has tests that pass. It also allows you to create reports (e.g. in PDF format) to present to your project or QA manager and so much more!

To be fair, I haven’t used the full power of TestLink yet. You can manage what tests get executed for every build or milestone and have the Jenkins plugin use that info to trigger execution of those testcases. I’ve not used this functionality yet however.

It’s a first step, going from 400 testcases with no traceability to having a requirement management framework that allows you to easily follow up on requirements coverage. If you’re having trouble keeping your requirements and testcases organized I’d recommend you give TestLink a go.