Picking the right type for the question
1. The decision
You have a feature. You have an eval set. You have four types of eval to choose from. Which type goes on which field?
Most teams get this wrong in one of two directions. They over-engineer — they reach for LLM-as-judge on fields where exact match would have worked. Or they under-engineer — they use exact match on fields where it can't work, and pretend the failures aren't real.
The right answer comes from the question you're asking about the field, not from the field's data type. This unit gives you the decision table.
2. The decision table
Read it from left to right. The question shape gives you the type. The type gives you the cost. The cost is what you're paying to get an honest grade.
| Question shape | Recommended eval type | Cost per case | Time per case |
|---|---|---|---|
| "Is this exactly the expected value?" | Exact match | Free | Microseconds |
| "Is this in the expected set?" | Exact match against a set | Free | Microseconds |
| "Is the format right?" | Schema validation + exact match | Free | Milliseconds |
| "Is this list mostly the expected list?" | Set-comparison (precision, recall, F1) | Free | Milliseconds |
| "Does this mean roughly the same thing?" | Embedding similarity | Cheap embedding call | Tens of ms |
| "Does this mean exactly the same thing?" | Paraphrase classifier or LLM judge | One model call | Seconds |
| "Does this follow a rubric I can write down?" | LLM-as-judge with rubric | One model call | Seconds |
| "Does this feel natural / on-brand?" | Human sampling | Person-time | Minutes |
| "Is this safe to ship to a user?" | Human eval, every case | Person-time | Minutes |
3. Walking the ticket extractor
The four fields of TicketExtraction cover most rows of the table.
interface TicketExtraction {
category: "billing" | "technical" | "account" | "general";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string;
}
// Field-by-field eval plan:
const plan = {
category: { type: "exact-match", question: "Is this exactly the expected enum?" },
priority: { type: "exact-match", question: "Is this exactly the expected enum?" },
customer_email: { type: "exact-match-normalized", question: "Is this the expected email, case-insensitive?" },
summary: { type: "llm-as-judge + human-sample", question: "Is this faithful, specific, well-formed, and natural?" },
};Three fields are exact match. One field needs the heavier machinery. That ratio is normal — most product features have a small number of fuzzy fields buried in a lot of crisp structure. The eval cost is dominated by the fuzzy ones.
4. Common wrong picks
Four shapes show up repeatedly when teams pick the wrong eval type.
Exact match on free text. Treating the summary like a category. Fails every honest paraphrase. The pass rate becomes a measure of how close the model's wording matches the eval author's wording, not whether the model understood the ticket.
LLM-as-judge on enums. Asking a judge model "is this the right category?" when you could have written actual.category === expected.category. Slower, more expensive, and the judge can be wrong on a question the equals sign couldn't be wrong on.
Embedding similarity on lists. Treating a list of tags as one text blob, embedding it, and comparing. You lose the per-item structure. A list with three right tags and one wrong tag scores similarly to a list with one right tag and three wrong tags. Use set comparison.
Human eval on volume. Trying to grade every case in a 500-case eval set with humans. The eval set stops being run. The numbers go stale. Use humans for sampling and calibration, not for the bulk.
5. The cost question
For every field, ask three things. What's the cheapest type that gives an honest grade? What does the failure mode of that type cost you? Can you afford to pair it with something heavier as a sanity check?
For category, the cheapest honest type is exact match. The failure mode of exact match (no false flexibility) doesn't cost you anything because the enum is well-defined. No pairing needed.
For summary, the cheapest honest type is embedding similarity. Its failure mode (rates faithful and hallucinated summaries similarly) is dangerous. Pair it with an LLM-as-judge faithfulness check and a 1% human sampling layer.
That's the layered approach. The cheap grader runs every commit. The middle grader runs nightly. The human layer runs monthly. Each layer catches what the cheaper one missed.
| Layer | Runs | Catches |
|---|---|---|
| Exact match on crisp fields | Every commit | Schema regressions, wrong enum, broken normalization |
| Embedding similarity on summary | Every commit (it's fast) | Wildly off-topic summaries |
| LLM-as-judge on summary | Nightly or pre-deploy | Hallucinated claims, missing specificity |
| Human sampling on 1% of production | Monthly | Tone, naturalness, drift you didn't predict |
6. The honest defaults
Default to the cheapest type that fits the question. Add heavier layers when you need them, not before.
For a new feature with no eval yet: exact match on every crisp field, embedding similarity on every fuzzy field, and a manual reading of 5 cases. That's a real starter eval. It takes an afternoon. It catches the dumb bugs before they ship.
For a mature feature with stable inputs: exact match for crisp fields, judge with calibrated prompt for fuzzy fields, human sampling for what no judge can grade. That's the steady-state setup.
You add layers as you find the failure modes the current setup misses. Not before. Building infrastructure for failure modes you've never seen is how teams burn three months and ship nothing.
7. The end of chapter 4
You now know the four types of eval, when each fits, and how to pick one per field. You also know how to design the eval set itself, from chapter 3.
That's the load-bearing center of the course. The remaining chapters cover reading the numbers an eval set produces, running evals in production, and where to take this next.
If you stopped here, you'd have enough to build a real first eval for a real feature in an afternoon. That's the bar this course was trying to clear.
8. Next
Chapter 5 covers reading the numbers. Accuracy. Precision. Recall. F1. What they mean for a ticket extractor, in plain terms, with the same recurring example.