Hunting for Mutants

No, this isn’t an article about a recent zombie game release. Rather, it’s a more serious subject concerning the quality of source code in software applications. How can we test the robustness of source code and better guarantee its quality?


Today, the use of source code quality indicators has become essential in software development.  Amongst the most prominent is that of unit testing with “code coverage” indicators. This is used to find out the proportion of source code tested by unit testing. But of what quality are these tests and how relevant are they?

An objective of unit testing is to verify that following a change in the source code, the software continues to operate according to specifications. This is sometimes called regression testing. The problem is, although my testing might cover a majority of the source code, how am I guaranteed that the software will still operate correctly? How can I justify the relevance of my unit tests if they might not be helping?

Enter the concept of "mutation testing".  The principle is this: if we arbitrarily change the code to be tested by creating a "mutant" version of this code, are my tests able to detect this change?

In java, implementing "mutation analysis" comes from altering the byte-code of the class to be tested, then creating a mutant. This mutant class is then scrutinized by unit tests. If the tests fail i.e. the difference was found, it is estimated that the mutant was "killed". If the tests pass i.e. the difference was undetected, then the mutant has "survived".  The goal is that when tested for mutation, zero or very few mutants should survive, thus showing my tests are relevant because they have detected code changes.

In a Java / Maven environment, the implementation of mutation tests is simple. They have no direct impact on the test code or testing. It is performed externally by a Maven plugin (org.pitest: pitest-maven), not the developer, and they can also be integrated in Sonar indicators, a source code quality analysis tool.

Don’t hesitate to go on a mutant hunt to improve both the quality of your testing and of your applications.

Jacques Desmazieres is Managing Consultant, Java E5 J2EE Solutions at blue-infinity

Digital news!

Are you as digitally addicted as we are? We can supply you with a regular dose of digital news. Simply sign-up, or click to Follow us online.

About the author

Jacques Desmazières
Jacques Desmazières

J2EE/Java senior consultant. Expert in Software architecture and technical design, and involved in the b-i quality commitee. Before working for blue-infinity he was the technical director of a banking software editor on the French Riviera and also spend some years as a system integration consultant.