Today, the use of source code quality indicators has become essential in software development. Amongst the most prominent is that of unit testing with “code coverage” indicators. This is used to find out the proportion of source code tested by unit testing. But of what quality are these tests and how relevant are they?
An objective of unit testing is to verify that following a change in the source code, the software continues to operate according to specifications. This is sometimes called regression testing. The problem is, although my testing might cover a majority of the source code, how am I guaranteed that the software will still operate correctly? How can I justify the relevance of my unit tests if they might not be helping?
Enter the concept of "mutation testing". The principle is this: if we arbitrarily change the code to be tested by creating a "mutant" version of this code, are my tests able to detect this change?
In java, implementing "mutation analysis" comes from altering the byte-code of the class to be tested, then creating a mutant. This mutant class is then scrutinized by unit tests. If the tests fail i.e. the difference was found, it is estimated that the mutant was "killed". If the tests pass i.e. the difference was undetected, then the mutant has "survived". The goal is that when tested for mutation, zero or very few mutants should survive, thus showing my tests are relevant because they have detected code changes.
In a Java / Maven environment, the implementation of mutation tests is simple. They have no direct impact on the test code or testing. It is performed externally by a Maven plugin (org.pitest: pitest-maven), not the developer, and they can also be integrated in Sonar indicators, a source code quality analysis tool.
Don’t hesitate to go on a mutant hunt to improve both the quality of your testing and of your applications.
Jacques Desmazieres is Managing Consultant, Java E5 J2EE Solutions at blue-infinity