Pattern: Mutation Testing

Intent

• To measure sufficiency of given tests to detect (possible) bugs.
• To find new, untested cases.

Also Known As

Motivation

It's really hard to understand how good a given set of tests is. A common sense observation says that a test is good if even slight change in program functionality (regression) leads to a test failure. In mutation testing the original program source code is being changed (mutated) somehow and tests are run. After many different mutations introduced a resulting tests failure rate might be considered as useful test quality metrics.

Code mutations can also be used for finding new, untested cases of functionality.

While doing random mutations is possible, many tools perform much simpler ones, but apply them consistently over large code base. Possible mutations might include:

• stubbing a method with default return value
• randomly changing constants
• randomly substituting boolean expression with a constant value

Applicability

Example

Related Patterns

• [[Code Coverage]] also tries to measure tests quality
• In contrast to [[Testing By Poking Around]] mutation testing doesn't generate random data, but (randomly) changes the source code.

Related Antipatterns

References

• http://c2.com/cgi/wiki?MutationTesting

ToDo

• add references to tools (ruby)