Where to source real test cases

1. The temptation

When you sit down to write your first eval case, the temptation is to make one up.

You imagine a representative ticket. You write it. It's clean. The category is obvious. The customer email is well-formed. The summary writes itself. Five of those and you've got an eval set.

Don't.

Invented cases are biased toward what's easy to imagine. They sound like the docs example. They're missing the typos, the multiple-issue tangles, the angry all-caps subject lines, the legal-style threats, the bot-generated spam, the customers who write in three languages in one paragraph. Real input is messier than the version of it you can picture from your desk.

The best test cases aren't invented. They come from somewhere they already existed. There are four honest sources.

2. Source 1: Your own production logs

If the feature is live, the logs are the source. If it isn't live yet, you have the predecessor's logs — the human-handled tickets the feature is going to replace.

This is the single best source. It's the actual distribution. It already has the typos, the multi-issue tickets, the angry customers. You don't have to imagine what users send. You can read what they sent yesterday.

async function sampleForEval(days = 30): Promise<EvalCase[]> {
const tickets = await ticketsTable.findRecent({ days });

// Don't just take the first N. Stratify across categories so you don't
// end up with 90% billing because billing is your loudest segment.
const buckets = groupBy(tickets, t => t.assignedCategory);

return Object.values(buckets).flatMap(bucket =>
  sample(bucket, 3).map(toEvalCase)
);
}

One catch. Production logs have customer data. The eval set will get checked into a repo. Strip the personally identifying parts before you save the case. The email becomes a fake address. The name in the body becomes "the customer". The order ID becomes a placeholder.

3. Source 2: Your bug tracker

The second source is the issues your users actually reported. Bug tickets. Support escalations. The Slack thread that starts with "wait, that's not right."

Every one of those is, by definition, a case the system got wrong in a way somebody noticed. If you don't have a regression test for it, the same bug will come back.

For the ticket extractor, this looks like: every time a teammate said "the extractor labelled this as billing but it's clearly technical," that conversation has the input text and the wrong output. Add it to the eval set. The expected output is what your teammate wished the model had produced.

4. Source 3: Historical pull requests

The third source is your own commit history. Every time someone changed the prompt, changed the schema, changed the model — read the PR description and the diff. The PR usually starts with "we noticed X" or "case Y was being mishandled". That's a test case.

If the PR description says "the model was sometimes returning 'urgent' for tickets that had the word 'urgent' in the subject line even when the body was a non-issue," you have the input shape. Write the case. Save it. Now the next person who touches the prompt knows that case has to keep passing.

5. Source 4: Your worst customer interactions

The tickets that ruined someone's day are also the tickets that matter most. The customer who threatened legal. The bug that went viral on social media. The outage that started with a misclassified support ticket.

These are rare. You can't fill a 50-case eval set with them. But you can keep one or two as permanent fixtures. If the model ever fails on the input shape that led to last year's worst outage, you want to know before it ships, not after.

This is also where most of your adversarial coverage will come from. The hostile inputs you saw in reality are more honest than the ones a security review imagines.

6. What "honest" means

A test case is honest when its input came from somewhere outside your imagination, and its expected output was decided by someone who understood the feature's job — not just rewritten from what the model already produced.

The second part matters. A common mistake is to take a production input, run the model on it, and save the model's output as the expected. That's not an eval. That's a snapshot of the model's current behaviour. If the model is currently wrong, you've locked in wrong.

// Wrong: locks in whatever the model says today
async function captureCase(input: string): Promise<EvalCase> {
const output = await extract(input);
return { input, expected: output, category: "happy" };
}

// Right: a human decides what the right answer is, separately
async function captureCase(input: string, expected: TicketExtraction): Promise<EvalCase> {
return { input, expected, category: classifyForEvalCoverage(input) };
}

The right answer is the one a careful human picks, not the one the system currently emits. If you can't tell which is which, the eval set isn't going to catch the model regressing toward "currently wrong" either.

7. The starter sourcing recipe

Pick 30 minutes. Open the production log table. Pull the most recent 100 tickets. Randomly sample 5 that span the four major categories. Strip identifying info. Write down what the correct extraction should be — not what the model produces, what you think the right answer is.

That's your starter set. It took half an hour. It's already more honest than 90% of the synthetic eval sets shipped at companies twice your size.

8. Next

Next unit: what to do when there isn't one right answer. The summary field has many valid forms. Exact match doesn't work. You need other moves.