Specifying what 'good' means → Vibe to Spec

1. The thing you can't eval

You can't measure whether the ticket extractor is "good" until you've decided what good means. Category right? Priority right? Email pulled out? Summary readable? All of those? Some of those?

Every eval set is downstream of a specification. If the spec is fuzzy, the eval is fuzzy, and the system you build against the eval will be fuzzy too. Most teams who think they have an eval problem actually have a specification problem one step upstream.

2. What [[vibe-to-spec]] adds

[[vibe-to-spec]] is about the specification layer. How to turn a fuzzy product idea ("the ticket extractor should be helpful") into a precise statement of what the output should look like, what it should never do, and how you'd know if it was wrong.

The course teaches the skill of specifying with the LLM in the loop — using the model to sharpen the spec rather than to skip writing one. The eval set comes out of the spec cleanly once the spec exists. Without the spec, the eval set is your guess at what good means, and that's the part that drifts.

If the units in this course felt like they kept asking "what are you grading against," that course is where the answer comes from.