Exact-match evals — when the answer is the answer

1. The simplest eval

The simplest eval is the equals sign.

You have the model's output. You have what the output should be. You compare them. If they match, the case passes. If they don't, it fails. No threshold. No rubric. No second model judging the first.

This is exact-match evaluation. It's the cheapest, fastest, and most reliable type of eval that exists. When it's the right tool, nothing beats it.

2. Where it fits

Look at the ticket extractor again.

interface TicketExtraction {
category: "billing" | "technical" | "account" | "general";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string;
}

Three of the four fields have crisp right answers.

category is one of four values. The right one is one specific value. === works.

priority is one of four values. Same.

customer_email is one specific string. Normalize the case, strip whitespace, then ===. Same.

The fourth field, summary, doesn't fit. Exact match would fail honest paraphrases. That field needs a different type of eval, covered next unit.

3. The code

Exact match is short enough to fit in your head.

type FieldGrade = { pass: boolean; field: string; reason?: string };

function gradeCategory(actual: TicketExtraction, expected: TicketExtraction): FieldGrade {
return actual.category === expected.category
  ? { pass: true, field: "category" }
  : { pass: false, field: "category", reason: `got ${actual.category}, wanted ${expected.category}` };
}

function gradePriority(actual: TicketExtraction, expected: TicketExtraction): FieldGrade {
return actual.priority === expected.priority
  ? { pass: true, field: "priority" }
  : { pass: false, field: "priority", reason: `got ${actual.priority}, wanted ${expected.priority}` };
}

function gradeEmail(actual: TicketExtraction, expected: TicketExtraction): FieldGrade {
const a = actual.customer_email.trim().toLowerCase();
const e = expected.customer_email.trim().toLowerCase();
return a === e
  ? { pass: true, field: "customer_email" }
  : { pass: false, field: "customer_email", reason: `got ${a}, wanted ${e}` };
}

That's the whole machinery. No external library. No model call. No human in the loop. It runs in microseconds and gives a clear answer on every case.

4. Why it's underrated

Engineers reach past exact match too fast. They see "AI eval" and assume the answer must involve embeddings, a judge model, or a vendor product.

For enum fields and structured IDs, none of that is true. Exact match is the right tool. It's deterministic. It's free. It produces actionable failure messages — "got billing, wanted technical" tells you exactly what to fix. A judge model would tell you "the answer was incorrect" and charge you for it.

When exact match works, use it. The fact that it's boring doesn't mean it's wrong.

5. Where it breaks

Exact match fails in three predictable places.

Open-ended fields. The summary field has no single right answer. === against "Customer reports a duplicate charge" fails the equally good "Two charges hit the customer's card for one invoice." Don't use exact match here.

Fields with valid variations. Customer emails sometimes appear with mixed case. Phone numbers with different formattings. Dates in different formats. Normalize first. If you can't normalize to a canonical form, exact match isn't the right type.

Fields where partial credit matters. A list field where the model produces four of the five correct tags. Exact match scores this as a failure. A set-comparison metric scores it as 80%. If you care about the difference, exact match isn't enough.

6. The aggregate score

Exact-match grading produces a per-field pass/fail. You aggregate over the eval set to get a per-field accuracy.

type CaseResult = { id: string; grades: FieldGrade[] };

function fieldAccuracy(results: CaseResult[], field: string): number {
const relevant = results.flatMap(r => r.grades.filter(g => g.field === field));
const passed = relevant.filter(g => g.pass).length;
return passed / relevant.length;
}

// Example: { category: 0.96, priority: 0.88, customer_email: 1.00 }
// Each is interpretable on its own. Average them and you've thrown away information.

Keep per-field numbers, not just a single aggregate. The aggregate hides which field is failing. If the model gets category 96% right and priority 88% right, you need to know that. A single 92% tells you nothing useful.

7. The honest closer

Most teams' first eval is exact match on one field. That's the right starting point. The work isn't picking the right type, it's picking the right field to start with — usually the enum field whose value most affects what happens downstream.

For the ticket extractor, that's category. Misclassify a billing ticket as technical, and it goes to the wrong team. Get the summary slightly wrong, and a human reads it five seconds later anyway. Grade the field that matters most first.

8. Next

Next unit: the summary field. Exact match can't grade it. Semantic match can, with caveats worth knowing.