Eval-Driven Loops

An eval is a fixed set of test cases with answers you decided before running anything, plus a scorer that turns a system's output into a number. Run it, keep the number; change something — a prompt, a model tier, a skill — run it again; compare. That's the entire mechanism, and it is the difference between two ways of working that look identical from the outside: engineering (changes proposed, tested, kept or reverted on evidence) and vibing (change something, eyeball three outputs, feel good). Every lesson in this course that says "measure" lands here. This one teaches how to build a suite that's honest, the three rules that keep its number meaning something, and how to make the loop cheap enough that you actually run it.

Why eyeballing fails — specifically

Not because your judgement is poor. Because your sample is: you look at the cases you just touched (where the change was aimed, so it looks great) while the regression lands in the cases you didn't re-check. And you evaluate at the moment of maximum generosity — right after doing the work. An eval is nothing more exotic than a fixed, honest sample plus a memory: the same twenty cases every time, scored the same way, so "better" means better on a constant yardstick rather than on this afternoon's mood and this afternoon's subset.

Building one is less work than its reputation:

• Twenty real cases beat zero perfect ones. Pull them from history — real PRs for a review prompt, real tickets for triage, real documents for extraction. Synthetic cases test your imagination; real cases test the task.
• Write expected outcomes by hand, in advance. Never derive them from the system under test — that defines current behavior as correct and scores you 100 forever, a perfectly vacuous instrument. (This mistake is common, invisible, and fatal; check for it in any suite you inherit.)
• Score checkable properties. Exact-match where outputs are structured. Where they're prose, score the verifiable core — did it flag the planted bug? name the right file? include the required sections? — not resemblance to a golden paragraph. If you must judge prose quality, a model can grade against a rubric, but keep the graded properties concrete (lesson 28 explains why judgement can inform a score but must not be a gate).

The three rules that keep the number honest

1. Abstention is not error. Build the scorer to distinguish "the system declined" from "the system was wrong" — three counts: correct, wrong, abstained. If declining scores like failing, you've made guessing the rational strategy (a guess sometimes lands; a refusal never does), and your metric now punishes exactly the behavior — honest uncertainty — that you want from anything autonomous (lessons 17, 18). How you trade wrong against abstained is a real product decision; a scorer that collapses them makes that decision for you, silently, in the wrong direction.

2. Hold out a slice from day one. Split the cases: a seen set you tune against, and a held-out set you only ever score. The moment you iterate on a prompt to fix failing cases — which is the entire point of having a suite — you begin fitting the suite: wording that nails those particular twenty examples rather than the task they sample. Seen-score rises; nothing real improved; and from inside, fitting looks exactly like progress. The held-out slice is your tell: seen climbing while held-out stalls means you're memorizing, not learning. Two disciplines keep the tell alive: never tune on held-out failures (studying them converts them to seen data — when you must diagnose, retire the slice and cut a fresh one), and check the gap every time you compare, not just when suspicious. This mechanism becomes the entire safety system of lesson 25, where the tuner is a machine that fits suites faster than you do.

3. A score is meaningless without the suite's identity. "We went from 71 to 84" is information only if it's the same cases and the same scorer. The cheapest way to show progress — usually by accident, mid-refactor — is adding easy cases or softening scoring. So: version the suite (a hash over cases + expected answers + scorer config does it), report every score with its version, and treat cross-version comparisons as the apples-to-oranges they are. When lesson 28 wires the score into CI as a merge gate, this rule is what prevents "make the number go up" from quietly becoming "edit the yardstick."

Making it pay

Score the system, not the model. The unit under test is your prompt + skill + pipeline at the tier it actually runs — which makes the eval the instrument that settles lesson 10's routing question with numbers: run the suite on both tiers, read two scores, route on evidence. Same instrument answers "did the cheaper thinking budget hurt?" (lesson 23) and "does the memory file help?" (lesson 24) — one suite, many arguments settled.

Make the loop cheap enough to be habitual. Twenty cases through a cheap model is pennies; headless invocation (lesson 18) makes it one command; lesson 28 makes it automatic on every push. The eval you run on every meaningful change is worth ten times the eval you run quarterly — the entire value is in the tight loop.

Grow it from failures. Every production miss that annoys you becomes a case. The suite slowly accumulates your system's actual failure modes rather than your imagined ones — which is also what makes a mature suite precious and worth versioning carefully: it's crystallized experience.

Read small deltas with humility. A 20-case suite moves in 5-point steps; +3 is noise wearing a trend costume. The suite's resolution bounds what it can detect (lesson 23 makes this precise), and respecting that bound is what separates measurement from numerology.

The eval doesn't tell you everything. It tells you the same thing every time — and that constancy is the foundation everything in the last third of this course stands on.