Eval-Driven Loops

An eval — short for evaluation suite — is a fixed set of inputs, the answers you decided were correct before running anything, and a scorer that turns a system's output into a number. Run it against your system today, keep the number, change something, run it again. The difference is the only evidence you will ever have that the change helped.

That is all it is. The reason it earns a unit rather than a paragraph is that an eval suite has two jobs which pull against each other:

• report a number that moves when quality moves
• avoid becoming the thing that gets optimised

The second is why a held-out slice — cases the tuning process never sees — exists from the first day rather than being added later when somebody starts tuning. A suite the optimiser can read is a suite the optimiser will fit, and fitting it looks exactly like progress.

A correction, first

I had this term filed as impossible to build here, on the grounds that scoring needs model output and this repository has no model access. That was wrong, and the error is worth naming because it is the common one.

An eval suite does not measure a model. It measures a system whose output can be scored against known-correct answers. This repository has one: the extractor. Cases, a scorer, a held-out slice and a baseline are all buildable with nothing stochastic anywhere.

What a deterministic system under test does not exercise is variance. The same input scores identically on every run, so nothing here says anything about flakiness, sampling or temperature. That is the honest limit, and it is stated in the receipt rather than discovered by a reader six weeks later. The structure — cases, scorer, held-out slice, baseline, suite identity — is what transfers to a system that does vary.

The failure it fixes

Without a suite, "did that change help?" is answered by looking at three outputs and forming an impression. The impression is generous, because you just did the work. Changes land, the system drifts, and the drift is invisible because there was never a number to drift from.

With a suite but no held-out slice, something worse happens. The number goes up on every change, because the fastest way to raise a score is to fit the cases being scored. A prompt tuned until the twenty examples in the suite pass is a prompt that knows twenty examples. It is indistinguishable, from the inside, from a prompt that got better — and the score is what you will quote in the pull request.

Break it first

npx tsx scripts/terms/demo-21.ts

The demo builds a suite of twenty cases from the corpus, holds out every fourth one, scores a deliberately naive baseline, then runs a second "system" that has simply memorised the answers to every case it was allowed to see.

suite a4a20539bf69fbc0: 20 cases, 5 held out

baseline   seen 60.0%  held-out 60.0%  gap 0.0pp
optimised  seen 100.0%  held-out 60.0%  gap 40.0pp

  seen moved     +40.0pp
  held-out moved +0.0pp
  verdict: overfit

A perfect seen score, and the system learned nothing. Reported as a single number — the one that goes up, the one people quote — this is a triumph. The held-out column is the entire difference between knowing that and not.

The corpus has to be frozen

The suite is built from twenty documents in eval-corpus/, and the first version read docs/ directly — the course's own documentation. Every unit written changed the corpus, which changed the expected answers, which changed the suite hash, which made term 28's committed baseline stale on the commit after it was recorded.

That is this unit's own lesson arriving from the other direction. A score is meaningless without the identity of the suite it was measured on, and a suite whose corpus is a live directory has no identity. Freezing it is not tidiness; it is the minimum condition for any of these numbers to mean anything.

The mechanism

src/evals.ts is four functions and a constant.

scoreOn(system, cases) returns correct, abstained and wrong as three separate counts. That separation is the scoring philosophy of the whole repository, and it is load-bearing here: a system that says "I could not read this" is behaving correctly, and one that guesses is not. A scorer that lumps abstention in with error rewards guessing, because a guess sometimes lands and an abstention never does. The demo's last line is the proof — a system that answers nothing at all scores 0 wrong, 36 abstained, 4 correct, not forty errors.

A correction, made later. The first version of scoreOn counted every null answer as an abstention without asking what the expected answer was. That made a case whose correct answer is an abstention permanently unscoreable: answering it right moved nothing, so any suite containing unreadable fields had an accuracy ceiling below 1 and the number could not be read as a share of anything. It was found while building term 25 — the optimiser there stalled with one expected-abstention left, no proposal could ever move the score, and the loop looked like it had converged honestly when it had simply been denied the last step. The overfitting that term is entirely about never appeared. The rule now: when the expected answer is an abstention, abstaining is correct and answering is wrong; when the expected answer is a value, abstaining is neither. Three tests pin all three cases, and the figures above are post-correction.

split(cases) and report(system, cases) score both slices and always return both. report also computes the gap and flags it past GAP_LIMIT = 0.15. Reporting the seen score alone is the failure the function exists to prevent.

suiteHash(cases) is a stable identity for the suite: case ids, held-out flags and expected answers, sorted and hashed. A score without it is meaningless. Adding one easy case raises the number while nothing about the system improved, which is the cheapest way to show progress and the hardest to notice later. The demo does exactly that, comparing a system against itself with one easy case appended:

same system, one easy case added: verdict suite-changed (suite f2448cf36df5d15a)

compare(before, after, cases, previousHash) produces the verdict, and the ordering inside it is the design. suite-changed outranks everything. If the cases moved, the two numbers are not comparable, and saying anything else about them is a lie with arithmetic in it. overfit comes next, then the held-out delta — never the seen delta — decides improved, regressed or unchanged.

The baseline that tied

The first version of this demo used the real extractor as its baseline. The memorising optimiser then scored exactly the same on the seen slice, and the demo printed a tidy zero-point gap.

The reason is worth the paragraph: the expected answers were derived from the extractor itself, so the extractor was already at 100% by construction. There was no headroom for a memoriser to occupy, and therefore nothing for the held-out slice to catch. A demo about detecting overfitting that cannot produce any overfitting is a vacuous check with a chart on it.

The fix was a genuinely naive baseline — read a heading, never attempt a claim — which leaves real room to improve, and the demo now asserts that the optimiser's seen score actually rose before checking whether held-out caught it. If the trap cannot spring, the demo fails.

Break it first (mutations)

mutation                                       tests                 demo
abstention counted as error                    2 failed | 16 passed  exit 0
a changed suite compared anyway                1 failed | 17 passed  exit 1
held-out scored on the seen slice              4 failed | 14 passed  exit 1
suite hash ignores expected answers            1 failed | 17 passed  exit 0
an expected abstention is unscoreable again    1 failed | 17 passed  exit 0

Three of five are caught only by tests, which is the point of running both columns. "Abstention counted as error" leaves the demo green because the overfitting story is unaffected — the harm is a scoring policy that quietly rewards guessing, and only a test asserting wrong === 0 for a total abstainer sees it.

"Suite hash ignores expected answers" did not bite the first time it was tried. The claim in the paragraph above — edit an answer, keep the id, the hash still matches — was in this unit before any test pinned it. Three tests covered the hash (reordering, adding a case, moving a case across the held-out line) and not one of them touched an expected answer, so the invariant the unit described was unenforced. It is the twelve-instance pattern this course is about, found in the course's own material: a check that reads correctly and checks nothing. The test exists now and the mutation fails as the table says.

The receipt

artifacts/21-evals.json carries the suite hash, both slices for both systems, the padded-suite verdict, the abstainer's score, and the stated limit about variance. The hash is in the receipt so a future comparison can check "the score went up" against "the suite did not change" without trusting anyone's memory.

Where it overlaps

Line G is feedback loops, and this is the one every other loop in tier 5 reports into. Term 24 consolidates memory and needs a way to tell whether consolidation helped; term 25's self-improving loop is an optimiser and is therefore exactly the thing the held-out slice is defending against; term 27 repairs tests and needs a number that says the repair was real. Without this term those three are vibes with a build step, which is why interstitial III sits between this and 25.

Line F is determinism wrapping stochasticity. A suite is the deterministic harness placed around a system that may not be, and everything about it — fixed cases, decided-in-advance answers, a hashed identity — exists to make a varying thing measurable.

Line D is execution surface. An eval that only runs when someone remembers is an eval that stops running; term 18 makes it headless and term 28 makes it a gate.

Sharp edges

1. A held-out slice leaks the moment you look at it. Reading held-out failures to decide what to fix converts them into seen cases, silently and permanently. The discipline is to hold out cases and to resist diagnosing with them; when you must, retire the slice and cut a new one.

2. Small suites make large-looking gaps. Five held-out cases at two fields each is ten held-out scored fields, so one field is 10pp of movement. GAP_LIMIT = 0.15 is a threshold chosen for this suite's size, not a universal constant, and it should move when the suite grows.

3. Deriving expected answers from the system under test scores 100% by construction. It is the mistake that produced the tying baseline above, and it is easy to make because the system is right there and the answers are tedious to write by hand.

4. A verdict of unchanged is not the same as "no effect". It means the held-out accuracy did not move at this suite's resolution. With twenty cases the resolution is coarse, and small real regressions hide inside it.

5. Suite identity must include the held-out flags. Moving a case from held-out to seen changes nothing about the corpus and everything about what the comparison means. suiteHash includes the flag for that reason.