Sketch to Eval

8 chapters.

A short, free course on evaluating AI outputs. From the smallest eval set that's still useful to designing for adversarial cases, reading regression vs noise, and running evals once real users are involved. Read-only, with TypeScript code samples grounded in a single recurring example: extracting structured data from customer support tickets.

8 chapters · 34 units

1. Ch 0

   Start here

   Psychological · light build

   What this is. Who it's for. The honest test for whether to keep reading.

   2 units

   • 0.0What this is
   • 0.1The honest test
   Start chapter 0 →
2. Ch 1

   What an eval is, actually

   PEAK · most weight

   Strip the framework noise. An eval at its smallest is a table you can read in fifteen seconds.

   5 units

   • 1.0The question every AI feature has
   • 1.1What an eval is — and what it isn't
   • 1.2The five-by-three table: the simplest eval
   • 1.3Why 'looks right' isn't measurement
   • 1.4The eval mindset
   Start chapter 1 →
3. Ch 2

   The vocabulary

   Medium intensity

   Accuracy, recall, precision, drift. The words you need to talk about quality without sounding vague.

   5 units

   • 2.0Accuracy, recall, precision — in plain English
   • 2.1Coverage and the eval-set composition problem
   • 2.2Regression vs degradation vs drift
   • 2.3Ground truth — and the labeling problem nobody warns you about
   • 2.4Eval set vs test set vs production traffic
   Start chapter 2 →
4. Ch 3

   Designing your eval set

   PEAK · most weight

   The single most practical chapter. Where your eval set actually comes from, and what goes in.

   5 units

   • 3.0The four categories of test case
   • 3.1The five-then-fifty rule
   • 3.2Where to source real test cases
   • 3.3What to do when 'right answer' is fuzzy
   • 3.4The maintenance question — evals decay too
   Start chapter 3 →
5. Ch 4

   The types of eval

   Medium intensity

   Exact-match, semantic, LLM-as-judge, human. Pick the cheapest one that's still honest.

   5 units

   • 4.0Exact-match evals — when the answer is the answer
   • 4.1Semantic-match evals — when the answer has many shapes
   • 4.2LLM-as-judge — when there's no other option
   • 4.3Human evals — when the stakes earn it
   • 4.4Picking the right type for the question
   Start chapter 4 →
6. Ch 5

   Reading the numbers

   Medium intensity

   Once you have eval results, the new question is which ones matter and which are noise.

   4 units

   • 5.0The dashboard problem
   • 5.1Comparing two prompt versions, the right way
   • 5.2Real regression vs sampling noise
   • 5.3Knowing when to ship and when to wait
   Start chapter 5 →
7. Ch 6

   Evals in production

   Emotional high point

   The hardest mode. Continuous evals, shadow evals, drift detection, disagreement with your users.

   4 units

   • 6.0Continuous evals — running them on every change
   • 6.1The shadow eval — running the new version on real traffic
   • 6.2Detecting drift after launch
   • 6.3When users disagree with your eval
   Start chapter 6 →
8. Ch 7

   Where to go next

   Medium intensity

   The eval mindset stays with you. Honest pointers to the next courses in the catalog.

   4 units

   • 7.0The patterns of evaluation → API to Architecture
   • 7.1Evals before you ship → Prompt to Production
   • 7.2Specifying what 'good' means → Vibe to Spec
   • 7.3The eval mindset, going forward
   Start chapter 7 →