The five-then-fifty rule

1. The wrong question

When teams decide to build an eval set, they ask the wrong question first.

"How many cases do we need?"

They think 100. They think 500. They look at a research paper's eval suite with 10,000 examples and quietly close the laptop. Then they don't build the eval set at all, because the number is too big.

The right first number is five. The right next number is fifty. The right number after that depends on what the first fifty taught you.

2. Why five

Five cases isn't a stopgap. It's the size at which an eval set starts being useful.

The first case forces you to write the schema down. The second forces you to decide what "correct" means for at least one field. The third makes you pick something edge. By the fourth you've written more about how the feature is supposed to behave than the original ticket described. The fifth is where you find the first thing that's broken.

You don't need 100 cases to find the first bug. You need 5. Most teams find at least one real defect in the first 5 cases they write, including the people who were sure their feature already worked.

type EvalCase = {
id: string;
input: string;
expected: Partial<TicketExtraction>;
category: "happy" | "edge" | "adversarial" | "drift";
};

const starter: EvalCase[] = [
/* 2 happy, 1 edge, 1 adversarial, 1 drift — see previous unit */
];

// Run it. Look at the failures. Fix one thing. Run it again.
// That loop is the whole job at this stage.

Five cases run in seconds. You can iterate on the prompt twenty times in an afternoon. You can't do that with 500 cases unless you've built infrastructure first, and you shouldn't be building infrastructure before you've written your first case.

3. Why fifty

Five tells you whether the feature works on cases you can hold in your head. Fifty tells you whether it works on cases you can't.

At five cases, you remember every one. You can recite each input. You know which one tends to fail. The eval set is a mental checklist with a runner attached.

At fifty, that breaks. You stop remembering individual cases. You start trusting the aggregate number. The eval has become a measurement instead of a memory aid. That's the shift you're paying for.

Fifty is also the size where category coverage gets real. Five cases can fake balance across four categories. Fifty can't fake it. You'll see whether you actually have eight adversarial cases or whether you have one, three times.

4. Why not 500

Going from 50 to 500 takes 10x the effort. It does not give you 10x the information.

Past a few dozen cases, most new ones repeat patterns you already covered. The marginal case adds a tiny amount of signal. The marginal case also adds maintenance cost: when the schema changes, when the prompt changes, when the model changes, every case has to be re-checked. That cost scales linearly. The signal does not.

For most product features, 50 is where you stop. The next bug you find won't be discovered by case 51 of an offline eval. It'll be discovered in production, by a user, and added to the eval set after.

5. The 80% claim

Five cases will catch about 80% of the bugs you would find by writing 50. That's not a measured number — it's a working rule from people who have built a lot of eval sets. The point of the claim is that the curve bends fast.

It bends because the cheap, dumb, deterministic bugs all surface immediately. Schema fields you forgot. Prompts that don't mention the priority field. Models that return strings where you expected enums. These don't need 50 examples to find. They need 1.

The remaining 20% is the long tail — the rare edge case, the specific phrasing that throws the model off, the input shape you didn't think of. That's what fifty cases buy you. Not "more confidence in the same answer." A different class of bugs.

6. The practical version

Write 5 cases this afternoon. Two happy, one edge, one adversarial, one drift. Run them. Fix what's broken.

Get to 50 over the next month, as you ship and as production shows you what you missed. Don't try to write 50 in one sitting. They won't be honest cases. They'll be cases shaped like cases.

After 50, stop adding for its own sake. Add a new case when production gives you a real failure. That's the only signal worth growing on.

7. Next

Next unit: where the cases come from. The honest answer is that the best ones aren't invented, they're already in your logs.