The maintenance question — evals decay too
1. The unspoken assumption
The unspoken assumption in most eval discussions is that an eval set, once built, is done.
You build the 50 cases. You wire them into CI. You watch the pass rate. Months go by. The pass rate stays at 96%. Everything is fine.
Except the eval set stopped describing reality eight months ago, and nobody noticed.
This is the part most teams skip. Building the eval set takes a week. Maintaining it takes forever. And forever turns out to be the expensive part.
2. What decays
Three things drift, and the eval set has to drift with them.
The inputs drift. Your product gets new features. Last year you had three plans, now you have seven. The "AI Copilot" feature your customers ask about didn't exist when you wrote the eval set. Tickets about it weren't in the production logs you sampled from.
The schema drifts. The category enum had four values. Now product wants six. The extractor's interface changed. Every test case's expected output for that field is now stale.
The model drifts. The provider releases a new version. Your team picks a different model. The same prompt produces a different shape of answer. Cases that passed last week fail in a way that isn't a regression, it's a model swap.
// Last year:
interface TicketExtractionV1 {
category: "billing" | "technical" | "account" | "general";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string;
}
// This year:
interface TicketExtractionV2 {
category: "billing" | "technical" | "account" | "general" | "ai_features" | "enterprise";
priority: "low" | "medium" | "high" | "urgent";
customer_email: string;
summary: string;
affected_product: string | null; // new field
}
// Every case from last year is missing affected_product
// and may have a wrong category for ai_features tickets.The shape moved. The eval set is now lying.
3. The silent lying problem
The reason this matters: an out-of-date eval set doesn't error. It passes. Passing rates stay high. The dashboard stays green. The team feels confident.
Meanwhile, in production, the AI Copilot tickets are being misclassified at 40% and nobody is checking, because the offline eval set has zero AI Copilot tickets in it and never failed on one.
An eval set untouched for a year is not protecting you. It's giving you false comfort about a system whose input distribution it no longer matches.
| State | What the pass rate tells you |
|---|---|
| Eval set current, pass rate high | The system handles the cases you currently care about |
| Eval set current, pass rate dropping | Something regressed against the cases you currently care about |
| Eval set stale, pass rate high | You are passing on what reality used to look like |
| Eval set stale, pass rate dropping | Even the old reality is breaking, and the new one is unknown |
The top-left and bottom-right rows are interpretable. The bottom-left row is the silent lie. It's the most common state of mature eval sets.
4. The maintenance cadence
A useful default is three layers.
Continuous. Every time a new production bug surfaces in the feature, a case for it goes into the eval set. Same week. No exceptions. This is the rule that keeps the set in touch with reality at all.
Quarterly. Once a quarter, somebody sits down for two hours and reviews the eval set. Are any cases obsolete? Are any product areas un-represented? Did the schema change? Are the expected outputs still what we'd want today?
Major-change-triggered. Any time the schema changes, the prompt changes meaningfully, or the underlying model changes — review the whole set before trusting the next pass rate. A pass rate after a model swap means nothing until you know which cases are still valid tests.
type EvalCaseV2 = {
id: string;
input: string;
expected: TicketExtractionV2;
category: "happy" | "edge" | "adversarial" | "drift";
// Maintenance metadata
createdAt: string; // YYYY-MM-DD
lastReviewed: string; // YYYY-MM-DD
schemaVersion: number; // bumped when interface changes
sourcedFrom: "production" | "bug_tracker" | "pr_history" | "manual";
};
// At quarterly review: list cases where lastReviewed is older than 6 months.
// At schema change: list cases where schemaVersion is below current.The metadata is what makes the review tractable. Without it, the quarterly review is "read 50 cases and try to remember when each was written." With it, the review is a query.
5. What triggers an unscheduled refresh
Three triggers should force a refresh outside the cadence:
- A user-visible incident where the feature got something wrong. Add the case. Done.
- A new product feature ships and the extractor will see tickets about it. Add three drift cases now, before the misclassifications start showing up.
- A model swap. Even a minor version. Re-run the whole set, look at the new failures honestly, decide which are real regressions and which are the new model being differently correct.
If none of these happen for six months, you're due for a quarterly review anyway. Don't let the eval set sit untouched longer than that.
6. The honest closer
Maintenance is the unglamorous part. Nobody gets promoted for keeping the eval set current. The cases you add in the quarterly review will look like the cases that were there before. The work feels redundant.
It isn't. The reason your competitors have shipped AI features that quietly degrade for months is that they did the building part and skipped the maintenance part. You don't have to make the same trade.
7. Next
Chapter 4 starts the second half. You know how to design the set. Next: how to actually score each case. The types of eval, what each is good for, what each can't do.