Why Code Coverage Is a Vanity Metric (and What to Measure Instead)
Coverage measures what your tests ran, not what they verified. See the research, and the four metrics that actually predict whether your tests catch bugs.
What you’ll learn
- Why a high coverage number measures execution, not verification
- What the research says about coverage and bug-catching effectiveness
- Four metrics that actually predict whether your suite catches regressions
- How to measure the coverage that keeps bugs out of production
Your team hit 90% code coverage last quarter and everyone celebrated. Then a checkout bug shipped to production and cost you a weekend.
The coverage number was never lying to you. You were reading it wrong. Code coverage measures the percentage of your code that runs while your tests execute. It says nothing about whether those tests checked that the code did the right thing. A line can run, return garbage, and still count as covered.
I spent 13 years building mobile infrastructure at Square, Facebook, and Instacart, and I watched good teams run the same play over and over. They pushed the coverage number up, shipped on the confidence it bought them, and got burned by a bug the tests never actually checked for. Coverage is a vanity metric. It climbs, it feels like progress, and it does not predict the one thing you care about, whether your tests catch bugs before your customers do.
Nobody’s arguing you should stop measuring coverage. The fix is to stop treating a diagnostic as a grade. So here is how coverage fools you, what happened when researchers tested the assumption head-on, and the four metrics I would put on the dashboard in its place.
100% Coverage, Zero Assertions
The fastest way to see the problem is to break it. You can write a test suite that hits 100% coverage and verifies absolutely nothing. Call every function, execute every branch, and assert nothing about the results. Coverage tools report a perfect score because their job is to track which lines executed, not whether anything was checked.
Assertion-free tests aren’t a hypothetical. They’re one of the most common ways coverage gets gamed under deadline pressure. The test runs the code, the coverage report turns green, and the moment the function returns a wrong value, nothing fails.
Coverage answers “did this line run during a test?” It never answers “did the test confirm this line is correct?” Those are different questions, and only the second one keeps bugs out of production.
Coverage and quality drift apart for exactly this reason. Two suites can both report 85%. One asserts precise expected outputs on every critical path. The other calls the same functions and shrugs. The dashboard shows an identical number, and the bug counts will not be identical. The metric cannot tell the difference between a test that verifies and a test that merely visits. That blind spot is the whole problem.
When Coverage Becomes a Target
Coverage is genuinely useful in one direction. Martin Fowler put it precisely in his essay on test coverage: “Test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are.” Uncovered code is a real signal. It points at a flow nobody tested, and chasing that signal makes your suite better.
The metric breaks the moment you invert it and turn the percentage into a goal. Fowler names the trap in the same essay: “If you make a certain level of coverage a target, people will try to attain it. The trouble is that high coverage numbers are too easy to reach with low quality testing.” Anthropologist Marilyn Strathern captured the general pattern in 1997, in what’s now known as Goodhart’s Law: “when a measure becomes a target, it ceases to be a good measure.”
Mandate a number and watch what your team starts optimizing for instead of bugs:
- Cheap lines beat risky ones. Getters, boilerplate, and happy paths rack up coverage fast. The tangled conditional branches where bugs actually hide are slow to test. A target rewards the wrong trade every time.
- Assertions turn optional. The quickest route to green is a test that runs the line and checks nothing. The percentage cannot tell that apart from a test that verifies, so under deadline pressure the empty version wins.
- The last stretch gets padded. The final ten to twenty points are the most expensive and least valuable, so they fill up with token tests written to clear the bar, not to catch a regression.
- Refactoring gets punished. Delete dead code and your percentage drops, so people keep cruft alive to protect the number. The metric now fights the cleanup it should reward.
- Green becomes the whole game. Once the number is mandated, the team optimizes for reporting it, and the goal quietly slides from catching bugs to hitting a quota.
Fowler quotes Brian Marick on the distinction that saves you: “I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.” Expect coverage as a byproduct of thorough testing and it stays honest. Require a number and your test suite becomes theater.
The Research: Coverage Doesn’t Predict Bugs
This isn’t just an opinion formed from watching teams. It has been measured. Laura Inozemtseva and Reid Holmes tested the assumption directly in their ICSE 2014 paper, “Coverage Is Not Strongly Correlated with Test Suite Effectiveness.” They studied five large Java projects (Apache POI, Closure, HSQLDB, JFreeChart, and Joda Time), each over 100,000 lines of code, and used mutation testing to measure how many real faults each suite actually caught.
Once you control for the size of the test suite, their data shows only a low to moderate correlation between coverage and a suite’s ability to detect faults, and stronger coverage criteria like branch coverage didn’t help. Suites with more tests catch more bugs, which is unsurprising, but the coverage percentage itself adds little predictive power on its own. The paper won an ACM Distinguished Paper award and was named a most influential paper a decade later at ICSE 2024. This is settled ground, not a hot take.
Here is what coverage does and does not tell you, side by side.
| Question you care about | Does line coverage answer it? | What actually answers it |
|---|---|---|
| Did my tests execute this code? | Yes | Line / branch coverage |
| Did my tests verify this code is correct? | No | Mutation score, assertion quality |
| Are my most important user flows protected? | No | Critical-path coverage |
| How many bugs still reach production? | No | Escaped-defect rate |
| How fast does my suite catch a regression? | No | Mean time to detection |
Line coverage answers exactly one of five questions worth asking. Treating it as your headline number means grading your tests on the one dimension that says the least about whether they work.
What to Measure Instead
Kill the vanity metric and you need real ones. These four measure outcomes rather than execution, and together they tell you whether your suite actually protects the product.
-
Mutation score. Mutation testing injects small faults into your code, such as flipping
>to>=or replacing a return value, then reruns your tests. If the tests still pass, the mutant survived and your suite would have missed that bug in real life. Your mutation score is the percentage of injected faults your tests kill. It measures fault-detection directly, which is the thing coverage only pretends to measure. Tools like PIT (pitest) automate it for Java, with equivalents across most languages. -
Escaped-defect rate. Count the bugs that reach production per release, ideally normalized by change volume. It’s the ground truth your coverage number is supposed to be a proxy for. If coverage climbs while escaped defects hold steady, the coverage gains are noise.
-
Critical-path coverage. Not all lines are equal. Ninety percent overall coverage means nothing if checkout, authentication, and payments sit in the untested 10%. Track coverage weighted by the flows that generate revenue and carry risk, and treat a gap on a critical path as a release blocker regardless of the global number.
-
Mean time to detection. How long between a regression entering the codebase and a test catching it? A suite that catches breaks in minutes on every commit is worth more than one that hits a higher percentage but only runs nightly. Speed of detection is a quality property that line coverage is completely blind to.
None of these graph as cleanly as a single coverage percentage. That’s precisely why they’re better. They resist gaming because you cannot fake catching a bug you did not catch.
How Pie Tests What Matters
Every metric above shares one premise. What counts is behavior verified, not lines visited. Pie is built on it. Instead of instrumenting your source code and tallying the lines your tests happen to touch, Pie runs your actual product the way a user does and checks whether it works.
It Tests Real Journeys, Not Lines
Autonomous discovery explores your app the way a user would and maps the flows that generate revenue and carry risk. Then it writes tests for those flows. You stop asking what percentage of lines ran and start asking whether checkout still works, which is the question that predicts a production incident.
It Verifies Outcomes, Not Execution
A green coverage report only proves a line ran. A Pie test asserts that the flow produced the right result, so a function that runs and returns garbage fails the test instead of quietly padding your number. The assertion gap that makes 100% coverage meaningless disappears.
It Holds Up When Your App Changes
The coverage you build by hand rots the first time someone renames a button. Self-healing tests adapt to UI changes instead of breaking, so your protection carries across releases rather than decaying between them.
Pie is not bumping a percentage on a dashboard. It is testing your product, and that is the only kind of coverage that keeps a bug off your customers’ screens.
Measure What Ships, Not What Ran
We built Pie because we were tired of dashboards that felt like safety and weren’t. A coverage number tells you your tests ran. Verified behavior tells you your product works. Only one of those is worth shipping on.
Coverage still earns its keep as a diagnostic. Use it to find the flows nobody tested, then stop staring at the percentage. The teams that ship clean aren’t the ones with the highest number on the wall. They know their critical paths are verified, their regressions get caught in minutes, and their tests fail the moment the product breaks.
Stop optimizing the metric that feels like progress. Start measuring the one that decides whether your customers find the bug before you do.
Measure coverage that catches bugs
See how Pie tests your real user flows end to end, so what your tests catch is what counts.
Book a DemoFrequently Asked Questions
13 years building mobile infrastructure at Square, Facebook, and Instacart. Now building the QA platform he wished existed the whole time. LinkedIn →