What to do when 'right answer' is fuzzy

1. Two kinds of field

Look at the ticket extractor.

interface TicketExtraction {
category: "billing" | "technical" | "account" | "general";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string;
}

Two of these fields are crisp. Two are fuzzy.

category is one of four values. The right answer is one of four values. You can compare with ===. Same for priority. Same for customer_email, which has one correct form.

summary is different. There is no single correct summary. "Customer reports a duplicate charge" and "Two charges hit the customer's card for one invoice" are both fine. A grader would accept either. Comparing with === would fail both.

Most real features have at least one fuzzy field. This unit is about how to evaluate it without pretending it's crisp.

2. Move 1: Define what makes a good answer

The first move is to write down what makes a summary good, before you write a single test case for it.

Three properties usually cover it:

• Length: not a paragraph, not a fragment. One sentence, roughly 8 to 25 words.
• Faithfulness: every claim in the summary appears in the ticket. No invented details.
• Specificity: the summary mentions the issue, not just the category. "Customer wants a refund" beats "Customer has a billing question."

These aren't a rubric yet. They're a list of what you're going to grade on. Writing them down forces you to decide what "good" means before the model gets a vote.

interface SummaryRubric {
lengthOk: boolean;      // 8 to 25 words
faithful: boolean;      // no invented claims
specific: boolean;      // names the actual issue
}

function gradeSummary(input: string, summary: string): SummaryRubric {
return {
  lengthOk: wordCount(summary) >= 8 && wordCount(summary) <= 25,
  faithful: noClaimsOutsideInput(input, summary),
  specific: namesIssueNotJustCategory(summary),
};
}

noClaimsOutsideInput and namesIssueNotJustCategory aren't trivial. The next chapter on eval types covers how to implement them. For now, the point is that you've named the three properties you care about, so the grader has something to grade against.

3. Move 2: Allow multiple expected answers

If two summaries are both fine, the test case should accept both.

type FuzzyEvalCase = {
input: string;
expected: {
  category: TicketExtraction["category"];
  priority: TicketExtraction["priority"];
  customer_email: string;
  acceptableSummaries: string[]; // any of these is fine
};
};

const case1: FuzzyEvalCase = {
input: "/* duplicate charge ticket */",
expected: {
  category: "billing",
  priority: "medium",
  customer_email: "alex@acme.io",
  acceptableSummaries: [
    "Customer reports a duplicate charge on the May invoice.",
    "Two charges hit the customer's card for one invoice; customer wants the duplicate refunded.",
    "Duplicate billing on May invoice; refund requested.",
  ],
},
};

If the model's summary matches any one of these, the case passes. This is a step better than === against a single string. It's still a step worse than rubric grading, because the acceptable set can't list every paraphrase.

Use this when the acceptable set is genuinely small — three to five surface forms. When it's open-ended, move to a rubric.

4. Move 3: Score against a rubric, not exact match

The third move is to stop comparing strings and start scoring against the rubric you wrote in step one.

For each case, the grade for the summary field isn't pass or fail. It's three sub-grades: length ok, faithful, specific. The aggregate is "the case passes if all three sub-grades pass."

This gives you something useful when the model fails. Instead of "the summary was wrong," you see "the summary was the right length and faithful, but not specific enough." That's actionable. You can change the prompt to ask for more specificity. You can't change anything in response to "the summary was wrong."

5. Move 4: Sample and bound the unmeasurable

Some quality is genuinely not measurable in an automated way. Whether a summary reads naturally. Whether the tone matches the brand. Whether a human reading the dashboard would feel informed.

You don't get to skip these by pretending they don't exist. You also don't get to pretend you can measure them automatically when you can't.

The honest move is sampling. Take 5% of production outputs. Have a human grade them on the unmeasurable dimensions. Compute the rate at which production fails on those dimensions. That number is real, even though the underlying property is fuzzy.

The eval set can't catch every bad summary. The sampling layer catches the class of bad summary that the eval set can't define. Together they bound the problem from both sides.

6. Putting the moves together

For the ticket extractor:

• category: exact match against the enum. Move 1 only — define the enum, compare with ===.
• priority: exact match against the enum. Same.
• customer_email: exact match, normalized to lowercase.
• summary: rubric scoring on length, faithfulness, specificity, plus sampling 5% of production for the tone/naturalness dimensions.

Three of the four fields are crisp and finish in five minutes. The fourth needs the rubric. You spend most of the eval-design time on the one field that needs it. That's correct. That's where the uncertainty actually lives.

7. Next

Next unit: even a good eval set decays. The world changes. Categories shift. New products appear. What to do about it.