LLM-as-judge — when there's no other option

1. The escape hatch

For some fields, exact match doesn't fit and semantic match isn't enough.

The summary field in the ticket extractor is the canonical case. You care about whether the summary is faithful — every claim it makes is grounded in the original ticket. You care about whether it's specific enough — it names the actual issue, not just the category. Neither property fits a string comparison. Neither fits cosine similarity.

What you want is a careful reader who can apply a rubric. The cheapest careful reader you can hire at scale is another model.

This is LLM-as-judge. The model judges the model's output. It's the escape hatch when nothing simpler works, and it has its own pitfalls worth naming.

2. The shape

A judge is a prompt that takes the input, the output to grade, and the rubric, and returns a structured score.

const JUDGE_PROMPT = `
You are grading a summary of a customer support ticket.

The original ticket:
---
{ticket}
---

The summary to grade:
---
{summary}
---

Score the summary on three dimensions. For each, answer pass or fail and one short reason.

1. Faithfulness: every claim in the summary appears in the ticket. No invented details.
2. Specificity: the summary names the actual issue, not just the category.
3. Length: the summary is between 8 and 25 words and is one sentence.

Return JSON with shape: { faithful: { pass, reason }, specific: { pass, reason }, length: { pass, reason } }
`;

The judge runs as a structured call. You parse the output. The case passes if all three dimensions pass.

type JudgeVerdict = {
faithful: { pass: boolean; reason: string };
specific: { pass: boolean; reason: string };
length: { pass: boolean; reason: string };
};

async function judgeSummary(ticket: string, summary: string): Promise<FieldGrade> {
const verdict = await callJudge<JudgeVerdict>(
  JUDGE_PROMPT
    .replace("{ticket}", ticket)
    .replace("{summary}", summary)
);

const allPass = verdict.faithful.pass && verdict.specific.pass && verdict.length.pass;
return allPass
  ? { pass: true, field: "summary" }
  : {
      pass: false,
      field: "summary",
      reason: Object.entries(verdict)
        .filter(([, v]) => !v.pass)
        .map(([k, v]) => `${k}: ${v.reason}`)
        .join("; "),
    };
}

When this works, it's powerful. The grader can see things string comparison can't. It can catch invented details. It can flag specificity. It produces an interpretable failure message.

When it doesn't work, the failure mode is the part you have to know about.

3. Pitfall one: the judge has biases

The judge is a model. The model has biases. The biases now affect every eval grade you produce.

The most documented bias is position bias. When asked to compare two answers, judges prefer whichever was shown first. This affects A/B comparisons more than single-answer grading, but it's there.

There's also length bias. Judges rate longer answers more favourably, especially when the rubric isn't tight. A verbose, slightly wrong summary will often beat a terse, correct one in the judge's score.

And there's style bias. A summary written in the judge model's preferred phrasing tends to score higher than one in a different style, even if both are equally correct.

4. Pitfall two: the judge can be gamed

If you use a judge in production — say, the same judge prompt is used to filter responses before they ship — you create a new attack surface.

A user (or the model itself, during fine-tuning) can learn to produce outputs that score well on the judge regardless of correctness. The classic shape is wordy, confident, and structurally clean. It hits the judge's biases. It doesn't necessarily hit the underlying property.

This matters even when you're not under attack. The same dynamic plays out internally. You tweak the prompt. The judge's score goes up. You ship. Production users notice no improvement. The judge was easier to please than the users.

5. Pitfall three: judges agree with confident-sounding answers

A judge presented with a confidently-phrased wrong answer often grades it higher than a hedging right answer.

"The customer is definitely reporting a duplicate charge on the May invoice and requesting an immediate refund" scores higher than "The customer seems to be reporting a charge issue, possibly involving the May invoice." The first answer is more specific-sounding. It's not necessarily more correct. The judge can't reliably tell the difference.

This is the most insidious failure. The judge is wrong in the direction of the model's worst habit — overconfident hallucination.

6. When to use it anyway

You use LLM-as-judge when the alternatives are worse. For most fuzzy fields, the alternatives are: don't measure (worse), measure with overlap metrics that miss the point (worse), or pay a human to grade every case (expensive enough that you stop doing it).

Use it for fuzzy quality properties on fields where the cheaper grades don't fit. Use it knowing the bias profile. Calibrate it against human grades at least once before you trust it. Re-calibrate when the prompt or the judge model changes.

For the ticket extractor's summary field, the judge is the right type, paired with a sampling layer for the dimensions even the judge can't catch.

7. The calibration step

Calibrating the judge is a one-time-per-prompt task that pays for itself fast.

Pick 30 cases. Grade each one by hand on the same rubric the judge uses. Run the judge on the same 30 cases. Compare the human grade with the judge grade case by case.

If the judge agrees with you on 28 of 30, the judge is usable. If it agrees on 20 of 30, the judge prompt needs work — usually a clearer rubric, or splitting one dimension into two. If it agrees on fewer than 20, the property you're trying to grade is probably not gradable with this judge at all, and you need either a different judge model or a different evaluation type.

The 30 cases become a permanent fixture. Every time you change the judge prompt or the judge model, re-run the calibration. If agreement drops, the change isn't safe yet.

8. Next

Next unit: when even the judge isn't enough, and humans have to grade. Where the line is and how to set up the labeling protocol.