State of Mutation Testing

Abstract

Testing a solution is important in order to confidently deliver the best work. Mutation testing helps in this process by accurately describing the quality of the tests being used. A problem can arise when using mutation testing since it is inherently computationally intensive. This is because the computational load scales with complexity. Mutation testing is easiest to use in smaller projects. However, the bar for what counts as a smaller project continues to be lowered as more and better ways of mutation testing are developed. From products currently available to even the largest of corporations joining in the research and deployment of mutation testing for IT solutions.

Introduction

Mutation testing is considered to be the best way to gauge the efficacy of a given test suite. This is because it checks if the given test suite would detect faults introduced by the mutation test. As more faults are checked the confidence in the results gained from the mutation test increases. There exists an inherent problem with mutation testing resulting from the way it works; an ever-growing set of faults need to be checked and the amount of work to run a mutation test increases with the scope of a project. This has been the reason mutation testing is not widely adopted. But this is beginning to change with the creation of solid and robust solutions and big corporations like Google LLC. investing and applying mutation testing on the largest of scales.

The stigma around mutation testing needs to fall, in order for this to happen more people need to know the current state, advantages and how to implement mutation testing.

Related works

Goran Petrovic (2021) discusses possible improvements to make mutation testing applicable for the workflow at Google LLC. He acknowledges the possibilities mutation testing may provide and argues that major features need to be added to existing solutions for them to use mutation testing at Google LLC.

While I do agree that major improvements can be made, I do not see the lack of these features as a dealbreaker. As most organizations and projects do not have the large scope researched by Goran Petrovic (2021).

ThoughtWorks (2020) confirms this belief, they have recently added Stryker (Nico Jansen, 2021) to their technology radar. ThoughtWorks states they have successfully used Stryker and by extension mutation testing for their JavaScript, C# and Scala projects.

The success of modern mutation testing solutions has been in the air for a while since most of the major technological hurdles for mutation testing were laid out and solved before the start of the new millennium (A. J. Offutt, 1992).

Analysis

In my professional experience I have seen many successful projects running on top of Stryker (Nico Jansen, 2021). The current problem described by Goran (2021) is that mutation testing does not scale to larger projects. While I do see the point he is making, I think he is overestimating the performance costs since his point of reference is that of one of the largest IT providers. However, this does not change the fact that the changes proposed by Goran (2021) are at fault or should be thrown away as I see them as a viable path for the future of mutation testing.

The biggest takeaway from the research done by Goran Petrovic (2021) is the fact that mutation testing should only test the cases where mutants are expected to be found. This is addressed by only checking code changes and by using mutant suppression to keep the number of active mutants productive, increasing the odds of finding a useful survived mutant. These facts combined lower the computational load of mutation testing and should make it usable for the largest of corporations, Goran (2021).

Another way to improve the adoption of mutation testing is to improve the ease of use. Current solutions for mutation testing allow the user to configure a lot of options (Nico Jansen, 2021). This is to improve performance for larger projects. But instead of helping people adopt this technology I think it has kept it from being used by the small teams that could gain the most from the current implementations. As they do not encounter the performance disparity encountered with larger projects.

Conclusion

The current implementation of mutation testing is adequate for most wanting to implement it in their workflow. It is true that there is room to expand the scope and speed of mutation testing but even these factors are becoming less of a problem when looking at the solutions currently available. The future improvements proposed by Goran Petrovic (2021) will make mutation testing accessible to larger teams and projects, since they do not impact the applicability for smaller teams this should not hold them back from implementing this new technology.

Recommendation

Mutation testing is currently applicable for most IT providers. For those too large to efficiently use mutation testing it is worth watching the release and updates of new solutions as the largest corporations are embracing this technology.

References

Goran Petrovic, Marko Ivankovic, Gordon Fraser, René Just (2021, February 26). Practical Mutation Testing at Scale. Retrieved 6 June 2021, from
https://arxiv.org/pdf/2102.11378.pdf

A. Jefferson Offutt, Roy P.Pargas, Scott V.Fichter, Prashant K. Khambekar (1992). Mutation Testing of Software Using a MIMD Computer. Retrieved 6 June 2021, from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.5112&rep=rep1&type=pdf

ThoughtWorks, inc. (2020, October 28). Technology Radar: Tools. Retrieved 6 June 2021, from
https://www.thoughtworks.com/radar/tools/stryker

Nico Jansen (2021, May 15). Stryker for JavaScript. Retrieved 6 June 2021, from
https://stryker-mutator.io/blog