Coverage and the eval-set composition problem
1. What coverage is
Coverage is how well your eval set represents the real world.
If your eval set has 50 cases and they all look like the easy ones, your accuracy number is going to be high and meaningless. The model isn't being tested on the things it actually struggles with. You're grading the model on the part of the job it can already do.
Coverage is the question "what cases are missing from this set." It's the most important question in eval design and the most often skipped.
2. The 90% billing problem
A real situation, made up of real things that happen.
You ship the ticket extractor. The first eval set you write has 50 cases. You wrote them in an afternoon, mostly using examples from your own scratch buffer plus a few from the support inbox. You feel good about it.
You run the set. Accuracy is 0.92. You ship the prompt change.
A week later, the technical team complains. The extractor is calling everything "billing." You check production. About 40% of incoming tickets are technical issues, and the model is misclassifying a third of them.
You go look at your eval set. Of your 50 cases, 45 are billing tickets — because billing tickets were what you happened to have on hand. The model was crushing billing. You never tested it seriously on technical. Your 0.92 was real, but it was 0.92 on a population that didn't match production.
This is the composition problem. The set looked fine because the headline number was high. The composition was wrong, and the headline number hid it.
3. The two dimensions of coverage
Coverage has two parts. They're worth separating.
Class coverage. Are all the categories represented? If your feature has four output categories, your eval set needs cases for each of the four. The split doesn't have to match production exactly, but if a class is 30% of production traffic and 1% of your eval set, you're not measuring how the model does on that class.
Failure-mode coverage. Within each class, are the hard versions of that class represented? Easy billing tickets are different from billing tickets that are also account issues. Easy technical tickets are different from technical tickets with prompt-injection attempts buried in them. Coverage means including the gnarly cases, not just the textbook ones.
Most teams remember class coverage and forget failure-mode coverage. Their eval set looks balanced in the count column but is still soft, because every case in every class is a easy one.
4. What the eval set should look like
A reasonable starting target, for a feature like the ticket extractor:
| Class | Easy cases | Hard cases | Total |
|---|---|---|---|
| billing | 4 | 4 | 8 |
| technical | 4 | 4 | 8 |
| account | 4 | 4 | 8 |
| general | 4 | 4 | 8 |
| ambiguous (multi-class) | — | 6 | 6 |
| adversarial (prompt injection, etc.) | — | 4 | 4 |
| Total | 16 | 26 | 42 |
About 40 cases. Roughly half easy, half hard. The hard half is where you actually learn whether the model is improving.
Notice the last two rows. Multi-class and adversarial cases don't fit cleanly into one category — they're the cases that broke for you in production once and you don't want them to break again.
5. The mismatch question
Coverage isn't only about balance within the eval set. It's also about how the eval set compares to production traffic.
A few useful questions to ask, every month:
- What's the class distribution in production this month? Has it shifted?
- Are there new failure modes coming in that aren't in the eval set?
- Are some classes in the eval set over-represented compared to production?
If production shifts and your eval set doesn't, your numbers stop reflecting reality. The model can be improving on the eval set while getting worse in production. That's the worst version of being wrong about being right.
6. The honest size
People ask "how many cases should my eval set have." The honest answer is: enough to cover the classes and failure modes you care about, with at least three cases per failure mode.
Three is the minimum because one is anecdote and two is coincidence. With three you start to see whether the model handles the failure mode consistently or just got lucky once.
For a feature with four classes and a handful of known failure modes, that's typically 30 to 100 cases. You don't need thousands. You don't need a labeling team. You need enough cases that every category and every gnarly situation has at least three rows, and you need to keep adding rows as new failure modes show up.
7. What's next
The next unit is about the three failure modes that get confused with each other in eval conversations: regression, degradation, and drift. Same symptom (numbers go down). Different causes. Different fixes.