Human evals — when the stakes earn it

1. When humans are unavoidable

Three of the eval types so far run in code. Exact match, semantic match, LLM-as-judge — all of them grade cases without anyone reading the output. That's the point. You get to run a thousand evals overnight.

For some properties, no automated grader is good enough. The model can't reliably tell. The embedding can't reliably tell. Exact match isn't defined. The only honest answer is to have a human read each output and grade it.

You don't reach for human evals first. You reach for them when the alternatives have failed, the stakes are high enough to pay for a slower grader, and the property you need to grade is one only a person can judge.

2. The cost

Human evals cost real money and real time.

A trained grader can do a careful eval on a single ticket extraction in maybe two to four minutes. Five tickets is fifteen minutes. Fifty tickets is two and a half hours. Five hundred tickets is two weeks of one person's time.

That's per pass. Every time you re-run the eval — after a prompt change, after a model swap, after a schema change — the human cost runs again. Automated graders amortize. Human graders don't.

The numbers are why most teams can't run human evals on every change. They run them quarterly, or before major releases, or as the calibration layer for a cheaper automated grader.

3. The latency

Even when the budget is there, the calendar isn't.

Automated evals finish in minutes. Human evals finish whenever the grader gets to them. If a grader is part-time, or shared across projects, or based in a different timezone, the round-trip from "code change committed" to "graded eval available" is days, not minutes.

That latency changes how you use the eval. You can't iterate on a prompt in a tight loop if each iteration takes three days of grading. You batch changes. You only request a human eval after the automated layer has filtered out the obviously broken candidates. The human layer becomes the final gate, not the developer's feedback signal.

4. The labeling protocol

The biggest predictor of whether human evals produce useful numbers is the protocol, not the grader. Two careful graders following the same loose protocol will disagree wildly. Two casual graders following a tight protocol will largely agree.

A workable protocol has four parts.

A written rubric. Same rubric the automated graders would use, but in more detail than a prompt can hold. For the summary field: a paragraph each on what faithful, specific, and well-formed mean, with two passing examples and two failing examples per dimension.

A grading sheet, not a conversation. Each case gets a structured form. Three checkboxes for the three dimensions. A free-text reason field. The grader fills the form and moves on. They don't decide what to grade.

Calibration before the real cases. Every grader runs the same five training cases first, where the right answers are pre-decided. If their grades don't match the training answers, they retrain before scoring real cases. This prevents one grader silently using a different rubric.

Two graders on every case, at least at first. Inter-rater agreement is the only honest signal that the protocol is tight enough. If two graders disagree on 30% of cases, the protocol is broken, not the data.

type HumanGrade = {
caseId: string;
graderId: string;
field: "summary";
faithful: { pass: boolean; note: string };
specific: { pass: boolean; note: string };
wellFormed: { pass: boolean; note: string };
gradedAt: string;
// The case overall passes only if all three sub-grades pass.
};

type CaseAgreement = {
caseId: string;
grades: HumanGrade[]; // typically 2
agreed: boolean;       // both graders reached the same overall pass/fail
};

The agreement record matters as much as the grade record. Cases where graders disagree get re-graded after a calibration session, not silently averaged.

5. Inter-rater agreement

If you have two graders on every case, you can compute how often they agree. That number is the floor on how trustworthy the human eval is.

If two trained graders agree on 95% of cases, the rubric is tight and the data is high-signal. If they agree on 70%, half the disagreements are noise, and the eval's pass rate is mostly measuring which grader saw the case.

The standard metric here is Cohen's kappa, which adjusts for the agreement you'd expect by chance. A kappa above 0.8 is good. Below 0.6 means the rubric isn't well-defined and you should fix the rubric before trusting any number from this eval.

You don't need to compute kappa to start. You need to compute it before you cite a pass rate. "95% pass rate" with 70% inter-rater agreement is roughly meaningless.

6. The 80/20 of when to use human evals

Most of the time you don't need human evals at all. The 20% of times you do:

• Safety-critical outputs. Medical advice. Legal triage. Anything where a wrong answer hurts a user. The model's confidence is uncorrelated with correctness in these areas, and the cost of being wrong dwarfs the cost of grading.
• Calibration for an automated grader. You can't trust a judge model without comparing it to human grades on a sample. The sample is small (30 cases) and you only do it once per prompt version, but it's the only way to ground the automated number.
• Production sampling for the unmeasurable dimensions. The tone of a summary. Whether the response feels natural. Properties no judge has the cultural context to grade. Sample 1% of production, grade it monthly.

For most product features, including the ticket extractor, the right setup is automated grading for the bulk of the eval set, plus a small human sampling layer that catches what automation can't.

7. The honest closer

The trap with human evals is treating them as the prestige option — slow and expensive and therefore correct. They aren't automatically correct. A loose human eval with no protocol is worse signal than a calibrated automated grader, and you pay a hundred times more for it.

The value of a human eval is in the rigour, not the humans. If you can't afford the rigour, you can't afford the eval. Spend the budget on the calibration sample for the automated layer instead. That's the path that actually scales.

8. Next

Last unit: how to decide which type of eval fits which question. A decision table you can use the next time you stare at a field and don't know how to grade it.