Mutation testing
Coverage says a line ran. Mutation testing says the line mattered: change it slightly, and see whether any test notices. A mutant that survives is either an untested behaviour or a line that does not need to exist — both worth knowing.
test-generator-mutate --lib lib --tests t --json mutation.json --min-score 85
Operators
| Operator | Mutates | Catches |
|---|---|---|
BooleanNegation | A boolean expression or a returned boolean, including return $x; → return !$x; | Tests that check truthiness at all, rather than only that a call succeeded. |
ConditionalInversion | if/unless/while/until conditions, negated in place | Branches where only one side is exercised. |
NumericBoundary | A numeric literal, nudged to the value either side of it | Off-by-one handling and inclusive-versus-exclusive comparisons. |
ReturnUndef | A return expression, replaced by undef | Callers that never check for a failed or empty result. |
Four operators is a deliberate choice. They are the ones whose survivors usually mean something, and whose mutants rarely fail for structural reasons. A mutant that does not compile, or that changes a signature, is noise in a report a human has to read.
How a mutant is applied
The mutated source is written into the project's own lib/
file, the tests run, and the original is restored — always, including when
prove crashes. A workspace copy is used only for the syntax
check.
This is not the obvious design, and the reason is worth knowing if you
modify it. Many CPAN modules put use lib "$Bin/../lib" near
the top of a test file. use lib unshifts onto @INC
at compile time, which happens after PERL5LIB has been
applied — so a harness pointed at a mutated workspace copy still loads the
original library, every mutant survives, and the reported kill rate is 0%.
Writing to lib/ means every resolution path sees the mutant.
The restore happens in the same process that writes the mutant. Kill it
between the two and the file is left mutated. git status lib/
will show it; git restore lib/ puts it back. Running
mutation testing only on a clean tree makes that obvious.
Per-mutant runs use prove -l -r. The baseline run uses
prove -Mblib -l -r, deliberately, so the baseline sees what
the full suite sees while the per-mutant runs are not shadowed by the
original code in blib/lib.
Reading the result
| Term | Meaning |
|---|---|
| Killed | A test failed with the mutant applied. The behaviour is covered. |
| Survived | Every test passed. Nothing asserts this behaviour. |
| Skipped | The line is inside a MUTANT_SKIP block, or the file was excluded. |
| Difficulty | HIGH, MEDIUM or LOW — how hard the survivor is likely to be to kill, used to order the work and to decide what to generate. |
| Score | Killed as a percentage of mutants that were actually run. |
Fast mode
mutation_level => 'full' is the default and tries every
mutant. 'fast' deduplicates first: several mutants on one
line usually produce the same behaviour, and one good test kills all of
them. The fast list is a subset of the full one, so it is the right mode
for a pull request and the full mode for a nightly run.
Skipping a line
Some code must not be mutated, because a mutant does not fail the suite — it hangs it. Mark the region with two anchored comments:
## MUTANT_SKIP_BEGIN
return 1 if $n <= 1;
## MUTANT_SKIP_END
The two cases this exists for:
- Recursion base cases. Negating the guard that stops a
recursive function produces infinite recursion, and the per-mutant
proverun has no timeout, so the whole run stalls. - Process management. Lines calling
kill,waitpidorforkcan hang or emitEND-block warnings instead of failing cleanly.
Markers are matched only as whole comment lines. A mismatched or nested pair is fatal rather than silently ignored, because a skip that stops halfway is how a run turns into an afternoon.
The feedback loop
A survivor is a statement that a test is missing. The pipeline tries to write that test:
mutation survivor
│
▼
SchemaExtractor reads the enclosing sub
│
▼
schema augmented with the mutant's boundary value
│
▼
written to t/conf/
│
▼
t/fuzz.t picks it up and runs it
│
▼
mutation killed on the next run
Three flags drive it, and they are additive:
| Flag | Produces |
|---|---|
--generate_mutant_tests=DIR |
t/mutant_YYYYMMDD_HHMMSS.t — TODO stubs for HIGH and MEDIUM survivors, with the suggested boundary value, the enclosing subroutine and environment hints. LOW survivors get a comment only. Several mutants on one line collapse into one stub, because one good test kills them all. |
--generate_test=mutant |
For each NUM_BOUNDARY survivor, extract the enclosing routine's schema and write a runnable t/conf/mutant_MODULE_FUNCTION_*.yml, with the boundary value and one value either side added. Falls back to a TODO stub when the schema cannot be parsed, the enclosing sub cannot be found, or confidence is very_low/none. |
--generate_fuzz |
Copies every existing schema in t/conf/ that has matching survivors, merges their boundary values in, and writes t/conf/mutant_fuzz_*.yml. The originals are never modified, and files that already start with mutant_fuzz_ are skipped so augmentation cannot cascade. |
test-generator-index \
--generate_mutant_tests=t \
--generate_test=mutant \
--generate_fuzz
The shipped dashboard.yml workflow runs all three and commits
the new files, so a repository accumulates tests that were provoked by its
own mutation results. That is the closest thing here to a test suite that
improves without being written.
Feeding survivors straight into a harness
The same report can be handed to the harness generator, with no intermediate schema file:
test-generator-mutate --lib lib --tests t --json mutation.json
fuzz-harness-generator --mutation-results mutation.json -o t/fuzz.t schema.yml
Each surviving numeric-boundary mutant in the schema's module is traced
to the comparison it mutated, and that comparison becomes a
min or max on the matching input field, so the
harness generates values either side of the boundary the suite missed.
A constraint already in the schema is never replaced, and survivors
that do not describe a numeric comparison are listed for review, since
they need a hand-written test. From Perl, pass
mutation_results => $file to generate().
Sometimes the mutant is equivalent: no input can distinguish it from the original. Equivalent mutants are unavoidable in mutation testing, and a kill rate of 100% is usually a sign that the source has been shaped around the operators rather than that the tests are complete. Use the score as a trend, not a target.