Semantic-match evals — when the answer has many shapes

1. The summary problem

Three of the extractor's four fields are crisp. The fourth isn't.

interface TicketExtraction {
category: "billing" | "technical" | "account" | "general";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string; // <-- this one
}

// Acceptable summaries for one duplicate-charge ticket:
const acceptable = [
"Customer reports a duplicate charge on the May invoice.",
"Two charges hit the customer's card for one invoice; customer wants the duplicate refunded.",
"Duplicate billing on May invoice; refund requested.",
];

All three are correct. They have different words, different lengths, different sentence shapes. Exact match passes one of them and fails the other two. That's not a useful grade.

Semantic match is the family of techniques that score "are these two pieces of text saying the same thing?" without requiring the same words.

2. Three common approaches

There are three approaches you'll see in real codebases. Each has a different cost-and-accuracy tradeoff.

Embedding distance. Turn both strings into vectors. Compute the distance. Close means similar meaning.

Overlap metrics (ROUGE-style). Count overlapping n-grams between the strings. More overlap means more similar.

Paraphrase classification. Use a model trained to judge "are these two sentences paraphrases of each other?" and read its output.

3. Embedding distance

The most common semantic-match approach. The embedding model turns each summary into a vector. You compare vectors with cosine similarity.

async function gradeSummaryEmbedding(
actual: string,
acceptable: string[],
threshold = 0.85
): Promise<FieldGrade> {
const actualVec = await embed(actual);
const acceptableVecs = await Promise.all(acceptable.map(embed));

const bestScore = Math.max(
  ...acceptableVecs.map(v => cosineSimilarity(actualVec, v))
);

return bestScore >= threshold
  ? { pass: true, field: "summary" }
  : { pass: false, field: "summary", reason: `best similarity ${bestScore.toFixed(2)} below ${threshold}` };
}

Two things to watch out for. The threshold is empirical — 0.85 is a starting guess, not a derived constant. Calibrate it against cases you've graded by hand. Second, embedding similarity rewards "talks about the same things" more than "claims the same things." A faithful summary and a hallucinated summary can score similarly if they mention the same nouns.

4. Overlap metrics

ROUGE-style metrics count n-gram overlap. ROUGE-L counts the longest common subsequence. ROUGE-1 counts overlapping single words. These were built for summarization research and they're cheap to compute.

function rouge1(actual: string, expected: string): number {
const actualTokens = new Set(tokenize(actual.toLowerCase()));
const expectedTokens = new Set(tokenize(expected.toLowerCase()));

const overlap = [...actualTokens].filter(t => expectedTokens.has(t)).length;
const recall = overlap / expectedTokens.size;
const precision = overlap / actualTokens.size;

return (2 * recall * precision) / (recall + precision || 1);
}

Overlap metrics are fast and deterministic. They also reward verbose summaries that include many of the expected words even when the meaning is wrong. "The customer mentioned May, charge, invoice, refund, card, duplicate" would score high without being a coherent sentence.

Use overlap metrics as a coarse filter, not as the only grade. If a summary scores 0.05, something is clearly off. If it scores 0.6, you don't know much.

5. Paraphrase classification

A specialized model trained on "are these two sentences paraphrases?" returns a probability. You set a threshold on the probability.

This is the most accurate of the three, and the most expensive. It's a model call per comparison. It runs slower than embedding similarity and much slower than ROUGE.

For an eval set of 50 cases with 3 acceptable summaries each, that's 150 model calls per eval run. Acceptable for nightly or pre-deploy runs. Not acceptable for tight loops.

6. The tradeoff

The honest tradeoff is the one stated in the heading: cheap and approximate, or expensive and accurate. There's no free middle. Pick the layer that matches the stakes of the field.

7. The faithfulness gap

Semantic match has a blind spot worth naming.

A summary that's similar in meaning is not necessarily faithful. A summary that says "Customer wants a refund for May and June charges" is semantically close to "Customer wants a refund for the May charge." The June claim is invented. Embedding similarity will rate them close. ROUGE will share most tokens. Both will pass the threshold.

The model has hallucinated, and the eval missed it. That's the failure mode you should worry about most with semantic match.

Two mitigations. One: pair semantic match with a faithfulness check, which is usually an LLM-as-judge call with a different prompt (covered next unit). Two: sample a fraction of "passed" cases for human review, watching specifically for invented content.

8. The starter recipe

For the ticket extractor's summary field:

• Use embedding similarity with a hand-calibrated threshold (start at 0.85, adjust based on 20 hand-graded cases).
• Run a ROUGE-1 sanity check as a cheap pre-filter — if the score is below 0.1, fail without bothering with embeddings.
• Sample 10% of passing cases weekly for human review on faithfulness.

This is more machinery than exact match. It's still less than most teams build, and it covers the field. The next unit covers the heavier option: using an LLM as the judge.

9. Next

Next unit: LLM-as-judge. When semantic match isn't enough and you need a grader that can read the rubric.