Code coverage tells you a line ran. It doesn’t tell you whether you tried the inputs that matter, or whether anything would notice if the line were wrong. Those are two separate gaps, and there’s an old technique for each: property-based testing generates the inputs you didn’t think of, mutation testing breaks your code to see if a test complains.
Both are decades older than my interest in them, and what changed isn’t the techniques. It’s that generating code got cheap while checking it didn’t. Here’s how they work on .NET and Angular, what they cost, and why each one is the check on the other.
Here’s a test that passes, covers the line it’s about, and proves almost nothing:
[Fact]
public void A_customer_who_joined_before_2020_gets_ten_percent_off()
{
// Arrange
var order = new Order(subtotal: 100m, customerSince: 2019);
// Act
var result = _pricing.Apply(order);
// Assert
Assert.NotNull(result);
Assert.True(result.Total > 0);
}Line coverage is delighted, and the name states exactly what the behaviour is supposed to be.
Two faults, and they’re independent.
The case that decides the answer isn’t here. One case per test is fine. The problem is the case nobody wrote: Apply turns on customerSince < 2020, so the input that settles it is 2020 itself, and 2019 is just the happy path.
Nothing in the suite knows the boundary is missing. A suite of examples is a list of the cases somebody or something thought of, and the bugs live in the ones they didn’t. Handing the enumeration to a model doesn’t fix that: it reaches for 2019 for the same reason a tired developer does, because the happy path is the obvious case. You get the same gap, filled in faster.
It asserts almost nothing. The name promises ten per cent off; the body checks the result isn’t null and the total is above zero. Delete the loyalty branch and this stays green. The name is the specification and the assertions are the evidence, and nothing in your toolchain compares the two.
Fix one and the other remains. A thousand generated inputs checked against Assert.NotNull prove nothing; a razor-sharp assertion about 2019 still says nothing about 2020. And coverage sees neither, because the 2019 test already executes every line the 2020 test would.
So there are two questions to ask of a test suite, and a technique for each:
| Question | Technique | What it does |
|---|---|---|
| Did you think of the cases that matter? | Property-based testing | explores the input space instead of your memory, and shrinks a failure to the smallest example |
| Would anything notice if the code were wrong? | Mutation testing | breaks the code on purpose and checks that a test goes red |
The interesting part, which I’ll get to, is that each one answers a question the other one raises.
Property-based testing: stop choosing the inputs
An example-based test states one fact: given this input, expect that output. A property states a rule that should hold for every input, and the library goes looking for a counterexample.
Properties are easier to find than the word suggests, once you stop hunting for a grand invariant. The useful ones are boring:
- Round-trip. Parse then render gets you back where you started.
- Order independence. Merging A into B equals merging B into A.
- Idempotence. Doing it twice is the same as doing it once.
- Totality. Nothing you can pass it makes it throw. It returns an answer for every input.
- Oracle. The fast implementation agrees with the obvious slow one, which is the property you get for free whenever you optimise something or replace it.
That’s a working shorthand rather than a taxonomy; John Hughes’ How to Specify it! has the systematic version.[1] The one route worth adding here is metamorphic: two related calls return related results, which gets you a property even for a function whose exact output you could never predict.
Metamorphic is what the pricing example from the top of this post needs, because it states something true about Apply without restating the discount rule. The library here is CsCheck, which is a package reference and nothing else:
[Fact]
public void Joining_earlier_never_costs_more()
{
// Arrange: generators, not examples
var cases = Gen.Select(Gen.Decimal[0, 10_000],
Gen.Int[1990, 2030],
Gen.Int[1990, 2030])
.Where((subtotal, earlier, later) => earlier <= later);
// Act and Assert: one rule, checked against every case generated
cases.Sample((subtotal, earlier, later) =>
_pricing.Apply(new Order(subtotal, earlier)).Total
<= _pricing.Apply(new Order(subtotal, later)).Total,
iter: 2_000);
}There’s no Assert because Sample is the assertion: it runs the lambda against each generated case and throws on the first one that comes back false. iter is how many cases to try, which is the dial the next section is about, and it defaults to 100. Where throws away the pairs where the years are the wrong way round. One thing to know before your first run: Sample uses every core by default, which is free speed on a pure function and a source of phantom failures on anything that isn’t thread-safe, and threads: 1 turns it off.
Notice what the property doesn’t say. It never mentions 2020 or ten per cent, so it isn’t the implementation written out a second time. That’s the trap waiting for anyone writing properties over business rules. It says something weaker and more durable: loyalty never costs you money. Invert the comparison inside Apply and every generated pair straddling the boundary fails at once.
It also does the thing the opening test couldn’t, though not in the way you’d expect. The generator doesn’t go looking for 2020, and it would be a poor bet if it did: pick a range of a couple of hundred years instead of forty and the odds of drawing that exact value collapse.
It doesn’t need to. It only needs one pair that straddles 2020, and those odds move the other way, rising towards one case in two as the range widens. The property fails on such a pair, and then shrinking walks both years inward to the simplest pair that still fails, which parks later on the first year where the behaviour changes. The generator supplies a failure somewhere in a half-infinite space, and the shrinker turns it into the number nobody wrote down. Enumerating cases by hand means guessing the boundary in order to test it. This doesn’t.
And it’s honest about its limits. This property will not catch < turning into <=. Loyalty still never costs more either way, so the test stays green while the boundary is wrong by a year. That isn’t a flaw in the property, it’s the edge of what a property reaches, and it’s where the second half of this post picks up.
None of this is specific to .NET. The TypeScript equivalent is fast-check, and here is the same property again:
test('joining earlier never costs more', () => {
fc.assert(
// Arrange: arbitraries, not values
fc.property(
fc.integer({ min: 0, max: 10_000 }),
fc.integer({ min: 1990, max: 2030 }),
fc.integer({ min: 1990, max: 2030 }),
(subtotal, earlier, later) => {
fc.pre(earlier <= later);
// Act
const a = apply({ subtotal, customerSince: earlier });
const b = apply({ subtotal, customerSince: later });
// Assert
expect(a.total).toBeLessThanOrEqual(b.total);
},
),
{ numRuns: 200 },
);
});Same three parts: generators instead of values, a rule instead of an expectation, and numRuns where the C# version has iter. fc.pre throws away the pairs where the years are the wrong way round, the way Where does above. Constraining the generator is usually better than discarding from it, because every discard is a run you paid for and didn’t use, and discards blunt the shrinker. A precondition you can’t express as a generator is what fc.pre is for. It drops into Vitest or Jest as an ordinary test, so there’s no separate runner to adopt.
What it costs
Six property-based tests bought 15,000 generated cases, ten times as many checks as the other 1,473 tests in that project put together. That cost 150 milliseconds of a 5.2 second suite, and only because the code underneath them is pure.
The dial that keeps it cheap is cases per property, not the number of properties. Both libraries default to 100, which is a reasonable commit-time number and nowhere near a night’s worth. Drop 2,000 to 200 and the suite gets ten times cheaper while every property still holds or doesn’t, which is a dial unit tests don’t have. So the shape worth aiming for is a low count on every commit and a nightly run that turns the number up, with a fresh seed each time so the suite keeps exploring long after it was written. The properties to move to that nightly job first are the ones that stopped being pure functions: a property-based test that touches a database pays its setup cost two thousand times.
Shrinking is the feature
Random input is the easy half. The half that makes it usable is shrinking: when a property-based test fails, the library doesn’t hand you the 4,000-character string that broke it. It searches for the smallest input that still fails.
One of mine failed on a generated array of a few hundred numbers and shrank to [-28]. A single element. You can hold that in your head. A counterexample you can’t reason about is barely better than a stack trace.
Every failure also prints a replay seed, something like CsCheck_Seed=1Jtg91Ucwiaf, so a failure that appeared once can be run again deliberately. That answers the obvious objection to random testing in CI, and it’s the reason to resist pinning a global seed for determinism: a fixed seed turns a property suite into a slow example suite.
The characteristic find is an unstated assumption
You expect this to find crashes. What it mostly finds is a disagreement between what you believed the code did and what it actually does, and about half the time the code turns out to be the one that’s right.
Here is mine. I wrote a property saying a text diff round-trips: render a diff, read it back, get the same text. It failed in seconds, on the pair ("beta\n", ""). The diff works in lines and deliberately normalises a trailing newline away, so a change that only adds or removes a final newline shows up as no change at all. The code was right and the property was wrong, and the behaviour it tripped over had never been written down anywhere. Somebody now has to decide whether a newline-only change should really be invisible in review.
Where it fits, and where it doesn’t
Property-based testing fits pure functions. It doesn’t fit code whose job is to talk to a database or somebody else’s API, where the hard work is arranging the world rather than checking a result.
A quick rule of thumb: if a test needs twenty lines of setup before it reaches the thing it checks, don’t reach for a property. That rules out most Angular components, and leaves the plain TypeScript behind them: store reducers and signal-store updaters, which are pure (state, action) => state functions and the best targets in the codebase, plus route guards, DTO mappers and ValidatorFn implementations. That’s where property-based testing belongs anyway.
There’s a cheap tell for where to start: a test file filling up with [InlineData] rows. If you find yourself adding cases one at a time to cover a little more of the input space, you’re already doing by hand what a generator does for you, and that’s the moment a property-based test starts to pay for itself.
My own properties are all C# so far. The TypeScript above is the same rule in fast-check’s shape, and a store reducer is where I’d start on that side.
Mutation testing: stop trusting the assertion
You can generate a million inputs and still assert nothing about them. That’s the question the first technique can’t reach.
Mutation testing answers it by attacking the code instead of the inputs. Take the code under test, make a small change, and run the suite:
// original
if (order.CustomerSince < 2020)
// mutant: boundary
if (order.CustomerSince <= 2020)
// mutant: negation
if (order.CustomerSince >= 2020)
// mutant: statement removal
// (the whole branch body deleted)Each is a mutant, a deliberately broken version of your production code:
- A test fails, so the mutant is killed. Something noticed. Good.
- All tests pass, so the mutant survived. You changed production behaviour and the suite shrugged.
The mutation score is the percentage killed. Unlike a coverage percentage it’s a direct statement about your assertions: if this code were wrong in this specific way, would we find out?
Two other statuses matter when reading a report. No coverage means no test executes the line at all, which is the ordinary coverage gap arriving by another route. Timeout means the mutant caused an infinite loop and the runner gave up, which counts as killed, because something definitely noticed. A runner that times out because the build agent was busy rather than because the mutant looped will quietly flatter your score, so a timeout count that moves around between runs is worth a look.
Three more statuses are bookkeeping rather than signal. A mutant that doesn’t compile is invalid and drops out of the score entirely, and on .NET that’s routine rather than exceptional: Roslyn will happily generate mutants the compiler then rejects, and a real report is full of them. Runtime error and ignored work the same way. So the denominator is smaller than the mutant count, and that’s normal.
A surviving mutant is a concrete, reproducible claim: here is a change to your code that nothing catches. Sometimes that’s a missing assertion. Sometimes the code is unreachable and should go. Occasionally the mutant is equivalent, meaning it doesn’t change behaviour at all, which is the genuinely annoying category and the reason chasing a 100% score is a mistake.
What mutation testing costs
Mutation testing is wasteful by construction: for each of N mutants you run the suite. Every mature runner has the same answer, per-test coverage, which works out first which tests touch which lines and then runs only those against each mutant. Whether you get that optimisation is the whole story of what this costs you.
It isn’t a knob you forgot to turn, which is what makes the slow regime hard to spot. Stryker.NET asks for per-test coverage by default, and where the capture doesn’t work it falls back without changing the result: same mutants, same score, same exit code. Only the clock tells you. Run it once with --verbosity debug and grep the log for coverage if you want to know which regime you’re in before you spend an afternoon finding out the slow way.
The two regimes aren’t a few per cent apart. In the slow one your bill is roughly suite duration times mutant count, so a few seconds per mutant is fine for the hundred you get from one folder and an overnight job for the ten thousand a real solution produces before filtering. Where capture works, the same run finishes while you fetch a coffee.
On Angular, check what’s actually being mutated
The .NET story above is mostly about time. On Angular the more useful question is what ends up in the denominator at all, because two things quietly shrink it.
Stryker mutates TypeScript, not templates. Every @if, @for, [class.active] and | async in an .html file is invisible to it. In a signals-era component a good deal of the real branching lives exactly there, so it isn’t that those mutants survive. They’re never generated. The score can’t see them and the report doesn’t tell you they’re missing.
And a mutant only counts if the tests covering it can run. Angular has moved to Vitest through @angular/build:unit-test, and specs that lean on that builder don’t necessarily run under Stryker’s own runner. Whatever they covered comes back as no coverage, which reads like a gap in your tests when it’s really a gap in the harness.
Put those together and a first Angular run can hand you a perfectly respectable percentage computed over a minority of the decisions in your application, with nothing in the report saying so. That’s the same failure as filtering the suite, arriving through a different door, which is the next section. The number to read first is the mutant count, not the score.
So point mutate where the logic is plain TypeScript and the tests already run headless: reducers and signal-store updaters, route guards, HTTP interceptors, DTO mappers, ValidatorFn implementations. The same list as for properties, for the same reason.
Two settings will save you an afternoon. Inline template: strings and decorator metadata get mutated as string literals, so selectors and route paths turn into noise you can drop with mutator.excludedMutations. And .stryker-tmp belongs in your .gitignore before the first run, not after it.
Scope the mutants, never the tests
A full mutation run is slow enough that you’ll want to make it smaller. There are two ways to do that, and one of them silently invalidates the result.
Narrowing what gets mutated is the safe one. Point mutate at fewer files and you get fewer mutants, each still judged by the whole suite, so every number still means what it says.
Narrowing what gets run is not. This is the --filter on dotnet test, the fdescribe somebody left in a spec file, the CI job that runs one project’s tests to save minutes. Filter the suite down and you can remove the very tests that would have killed the mutants you kept. Those mutants survive because nothing was left to catch them, so the score you get back describes your filter rather than your code. A score of zero looks alarming enough to investigate; the dangerous case removes only some of the killing tests and returns a plausible number that means nothing.
So: make the run smaller by mutating less, never by testing less.
What to do with a surviving mutant
A surviving mutant is structurally a review comment: a file, a line, and a specific claim that nothing noticed a change. So the obvious move is to file survivors as comments on the pull request, where line anchoring and threads already exist.
Don’t. It falls over on density. Eight survivors in one file are eight unanswered comments, arriving with the same visual weight as “this concurrency assumption is wrong”, and the count at the top of the review stops meaning what it meant.
Google ran into this at a scale nobody else has, and their number is worse than it first sounds: even after sampling had already cut the volume down, developers classified 85% of the mutants they were shown as unproductive. Not 85% of everything a naive run would produce. 85% of what survived a deliberate effort to show them less.
What fixed it was suppressing harder, not explaining better. Their selection cut the median mutants generated per change from 820 to 7, the system caps what it surfaces at seven times the number of files in the change, and a median change now puts two live mutants in front of a human. That’s what took the share developers judged productive from 15% to 89%.[2]
Machine findings nobody has to answer belong in the rendering rather than the conversation. A gutter mark, a margin, a summary line: somewhere the eye can skip. Promote the few that need a human answer and cap that number. You don’t have to build the viewer yourself either. Stryker publishes mutation-testing-elements, reusable web components that render any report following its schema, which is what both the .NET and JavaScript runners emit.
One rule outranks the layout: “no mutant was tried here” has to look different from “mutants were tried and all died”. A result and the absence of a result are not the same fact, and a missing report parses beautifully as no surviving mutants.
They compose, and that’s the real argument
Go back to the property from earlier. It reached 2020 without anybody naming it, and it still couldn’t catch < turning into <=. The property got to the input. It had nothing to say about whether the code handled it correctly.
Now look at the first mutant in the list above. It is exactly that: < flipped to <=. Mutation testing doesn’t need to think of the boundary, because it changes the code at the boundary and waits for something to go red.
That’s the composition. Property-based tests reach inputs nobody enumerated; mutants probe decisions no assertion pinned down. Neither covers the other’s blind spot by accident, and on this one line of code you can watch each of them fail at precisely the thing the other catches.
Neither of them will tell you whether 2020 was the right year. Stryker.NET has no numeric-literal mutator, so 2020 never becomes 2021, and a property about loyalty has no opinion about when the policy started. Both techniques check that the code does what the code says. Checking it against what the business decided is still a job for a person.
A short history, because none of this is new
It’s tempting to present both of these as a response to AI writing our tests. They aren’t. One is older than most people reading this, and the other turns twenty-six this year.
Mutation testing was proposed in 1971, in a student paper by Richard Lipton. The field proper starts late that decade, with Hamlet in 1977[3] and then the paper everybody cites, DeMillo, Lipton and Sayward’s “Hints on Test Data Selection: Help for the Practicing Programmer” in 1978.[4] Working tools followed by the end of the decade.[5]
That work rests on two assumptions which still carry it. The competent programmer hypothesis says programs are usually close to correct, so the faults worth hunting are small ones. The coupling effect says a suite that catches small deliberate faults tends to catch the real, larger ones too. Both are empirical claims rather than theorems, and together they are why flipping a single operator is a sensible proxy for “would you catch a real bug”.
Property-based testing arrived in 2000, when Claessen and Hughes published QuickCheck at ICFP.[6] It’s a short, readable paper, and the ideas transferred almost intact: properties written as ordinary functions, generators you can compose, counterexamples shrunk to something legible. It has since been ported to something like forty languages, fast-check and CsCheck among them.
So what changed isn’t the techniques. It’s the price of the thing they check. Generating code got radically cheaper and validating it didn’t. Reviewing a line still costs what it always did; what moved is that more code arrives, in bigger changes, and the cheap proxies we used to decide “this is fine” got less informative, because the same process that writes the code writes the tests and optimises for the signal you’re watching. A human writing a test that asserts nothing has made a mistake. A model writing one has done exactly what it was asked.
Getting them running
Neither asks much of you. Property-based testing is a package reference, mutation testing is a short config file and a command, and two keys in the TypeScript config are worth knowing about because they decide whether you get a result at all.
Property-based testing. CsCheck is a package reference and nothing else, fast-check a dev dependency. Neither needs a runner, a base class or a config file. The two properties above are the whole pattern.
Mutation testing on .NET. Stryker.NET installs as a local tool:
dotnet new tool-manifest
dotnet tool install dotnet-stryker
dotnet strykerEveryone who clones the repo afterwards runs dotnet tool restore instead of installing again, and that’s the line that belongs in your CI script rather than this one.
stryker-config.json goes in your test project’s folder, and that’s the directory you run dotnet stryker from:
{
"stryker-config": {
"solution": "../MyApp.sln",
"project": "MyApp.csproj",
"mutate": ["**/Pricing/**/*.cs"],
"reporters": ["json", "html"],
"report-file-name": "mutation"
}
}project names the project under test rather than the test project, which is the one to get right first. solution stops being optional as soon as more than one project references the code you’re mutating.
On TypeScript, StrykerJS is a dev dependency plus a runner plugin. Installing the plugin does nothing on its own: testRunner is the setting that switches it on, and it’s also the setting that decides which cost regime you land in. In stryker.conf.json:
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"testRunner": "vitest",
"coverageAnalysis": "perTest",
"checkers": ["typescript"],
"mutate": ["src/app/**/*.ts", "!src/app/**/*.spec.ts"],
"reporters": ["json", "html"]
}Leave testRunner out and it falls back to command, which shells out to npm test. On a stock Angular workspace that’s ng test in watch mode, so the run never finishes and you go looking for the bug in Stryker. It also drops coverageAnalysis to off, which is the slow regime from earlier, reached by accident rather than by choice. checkers earns its place too: TypeScript mutants that don’t compile turn the suite red for a type error, and without the checker Stryker reads that red as a kill. Your score goes up because your code stopped type-checking.
mutate is the single most important setting in either config. Pointing it at your whole solution on day one produces a number you’ll wait a long time for and then ignore. Point it at one folder of real business logic and read that result properly.
Where I’d start
If you’ve never used either, do them in this order, because the cost curves are completely different.
- Write three properties this week. Find your file with the most hand-written cases and add a round-trip, an order-independence or an idempotence property alongside them. Don’t delete the rows: they’re usually named regression cases tied to a bug somebody shipped, and that history is worth keeping. The first falsification usually teaches you something about your own code rather than finding a bug.
- Falsify each one on purpose. Break the code under it, watch it fail, then undo it. That’s the only way to know the test can fail at all.
- Time a scoped mutation run on each stack. Which cost regime you’re in is the only input that matters, and it’s an afternoon to find out rather than a project.
- Where capture works, run mutation on changed files. That’s
--sinceon Stryker.NET and--incrementalon StrykerJS. Both want real git history, so a pipeline that shallow-clones will need telling otherwise. Where capture doesn’t work, schedule it, scoped, with a wall clock. - Assert on the number of things actually tested. A run that tested nothing still exits zero, and a missing report parses beautifully as no surviving mutants. Count the mutants in the report and fail the step yourself if that count is zero or the file isn’t there. Absence and success have to look different to whatever reads the result.
- Show the mutation score, don’t gate on it. A threshold is a number you’ll be tempted to lower, and equivalent mutants mean the last few per cent aren’t worth chasing.
- Surface almost nothing. Google’s number is the one to remember: 85% judged unproductive, and that was already after cutting the volume down.
Both techniques answer the same underlying question, and it’s the one the test at the top of this post couldn’t: does a green suite mean anything?
John Hughes, “How to Specify it! A Guide to Writing Properties of Pure Functions”, Trends in Functional Programming (TFP) 2019. The five routes are validity (operations return well-formed results), postconditions (the return value relates to the arguments of one call), metamorphic, inductive (relate a call to the same call on a smaller input) and model-based (compare against a simpler abstract implementation). Johannes Link has ported the whole paper to Java and jqwik if Haskell isn’t your first language. ↩︎
Goran Petrović, Marko Ivanković, Gordon Fraser and René Just, “Practical Mutation Testing at Scale”, arXiv:2102.11378, 2021, published as “Practical Mutation Testing at Scale: A View from Google” in IEEE Transactions on Software Engineering 47(10). The paper is explicit that the 85% held “even when applying sampling techniques to substantially reduce the number of mutants”, and cites it to Petrović, Ivanković, Kurtz, Ammann and Just, “An industrial application of mutation testing: lessons, challenges, and research directions”, Mutation 2018. The 820-to-7 figure is the median number of mutants generated per change under arid-1-per-line selection; the cap of seven per file and the median of two live mutants per changelist are separate numbers from the same paper. See also “State of Mutation Testing at Google”, ICSE-SEIP 2018. ↩︎
Richard G. Hamlet, “Testing Programs with the Aid of a Compiler”, IEEE Transactions on Software Engineering, SE-3(4), 1977. ↩︎
Richard A. DeMillo, Richard J. Lipton and Frederick G. Sayward, “Hints on Test Data Selection: Help for the Practicing Programmer”, IEEE Computer, 11(4):34-41, 1978. Both this and the Hamlet paper sit behind IEEE’s paywall; search the titles rather than trusting a link. ↩︎
Timothy A. Budd, Mutation Analysis of Program Test Data, PhD thesis, Yale University, 1980, usually cited as the first full mutation system. A working prototype came earlier: Budd and Sayward’s Users Guide to the Pilot Mutation System (Yale, 1977) and the PIMS manual for FORTRAN IV (Georgia Tech, 1979) both predate the thesis, so “the first tool in 1980” is shorthand rather than the record. The history in this section follows Yue Jia and Mark Harman’s survey, “An Analysis and Survey of the Development of Mutation Testing”, IEEE Transactions on Software Engineering, 37(5), 2011, which is the best single starting point if you want the full lineage. ↩︎
Koen Claessen and John Hughes, “QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs”, ICFP 2000, pages 268-279. ↩︎