If you’re working on a software/hardware project chances are you have a list of requirements that have to be checked off by running (automated) tests that confirm the requirement is met.

I’ve seen a lot of projects using Jenkins to automatically build and compile the software and run tests to check which existing testcases pass and which are failing with every new push to the code repository. That’s great, in the blink of an eye you see if something broke: if the number of failing testcases rises you might want to take a closer look for regression issues.

The problem with only using Jenkins is traceability.
Imagine this: you have 450 testcases, they all check a piece of the functionality of your SW/HW. If a couple of them fail, can you immediately say which requirement is not met? Unless you have a very structured way of ordering your test suites this probably is not the case. This is where TestLink comes into play.

Requirement management with TestLink

TestLink is a test and requirements management framework, completely opensource and written in PHP. It has an API which you can use even if you’re using custom or proprietary tools to execute your testcases. With said API it is very easy to return test results to TestLink automatically.

I had to deal with this kind of situation before, a whole bunch of testcases that did not link back to any requirements. That’s when I decided to expand upon the existing Jenkins configuration by adding TestLink to the mix.

A list of requirements was available which was imported into my TestLink set up. Using a quickly whipped together Python script I converted the proprietary testset XML files into something that could be imported into TestLink. A couple of minutes later I had a list of requirements and a (huge) amount of testcase entries. Manually adding requirements to each and every testcase was long and tedious work, luckily you only have to do it once. I have to admit that the TestLink user interface has been stuck in the nillies and could benefit from an overhaul but once you get used to it, it’s pretty straightforward.

Combining Jenkins and TestLink

Next up was using Jenkins to feed the results back to TestLink. The Jenkins configuration was set in such a way that a build job is spawned when a change in the repo was found. After the build job is successful a second job is spawned that executes all testsets using an in-house framework that generates jUnit XML output. To be able to match the jUnit XML files to the testcases in TestLink I used a handy Jenkins plugin.

The matching criterium is simple, using the testcase name in the “result seeking strategy” of the plugin I generated a custom field value for each testcase in TestLink which was a simplified version of the testcase name. Whenever the plugin matches this custom field value with the testcase name in the jUnit output it updates the result in TestLink.

In the build job configuration you also have to specify a build number, with the Copy Artifacts plugin I could use an environment variable that contains the Jenkins build number of the job that compiled the code. This way you can run your test job multiple times even if there is no new version of your software. TestLink will create a record for each run of a testcase so you can quickly check its health using the execution history and see the progress over time with each build number.

Conclusion

Using these tools I can quickly have TestLink generate a report that gives me an overview of the health per testsuite, how much % of the requirements have been covered and how much of that has tests that pass. It also allows you to create reports (e.g. in PDF format) to present to your project or QA manager and so much more!

To be fair, I haven’t used the full power of TestLink yet. You can manage what tests get executed for every build or milestone and have the Jenkins plugin use that info to trigger execution of those testcases. I’ve not used this functionality yet however.

It’s a first step, going from 400 testcases with no traceability to having a requirement management framework that allows you to easily follow up on requirements coverage. If you’re having trouble keeping your requirements and testcases organized I’d recommend you give TestLink a go.