Self-Improving Loops

A self-improving loop points an agent at its own configuration: run the eval (lesson 21), propose a change to the prompt or pipeline, keep the change if the score rises, repeat. Mechanically it's hill-climbing — sixty years old, genuinely effective — and with headless runs (lesson 18) it's a few dozen lines of glue. It is also the single most dangerous pattern in this course to run naively, because of one specific phenomenon whose defining property is that it looks exactly like success. This lesson is one-third mechanism, two-thirds guardrail — and it opens with the non-negotiable prerequisite: a real eval with a held-out slice. No suite, no loop. An optimizer without honest measurement isn't automation; it's a random walk with an invoice.

Why the naive loop destroys itself

The loop optimizes the score, and the score is a proxy for what you want. Given any route to a higher number that doesn't require solving the task — and there is always such a route — the loop will find it. Not from malice; from economics: cheap routes are found first, and the cheapest route to fixing a failing eval case is almost never generalization. Concretely, watch what an optimizer does with a failing case: it adds wording that handles that case — a special-case clause, an example mirroring its input, a phrase that pattern-matches its shape. The seen-score climbs beautifully. Generalization doesn't move at all.

And here is the property that makes this a trap rather than a tradeoff: from inside the loop's own metrics, memorizing the suite and mastering the task are indistinguishable. Same rising number. Same green dashboard. Same satisfying trajectory. The chart you'd show your team looks identical in both worlds — and only one of those worlds ships something better.

Expect the characteristic timeline, too. Early iterations make real improvements — honest gains are also cheap at first, so the loop takes them. The drift begins when honest gains thin and case-fitting becomes the cheapest remaining move. A loop that "worked great for two weeks, then plateaued" usually didn't stop improving — it switched strategies, and the switch is invisible from inside.

The guardrail: two slices and a gap

Lesson 21's held-out slice is the entire safety mechanism, promoted from good practice to load-bearing structure. The optimizer tunes against the seen cases only. Every generation, score both slices, and watch one derived number — the gap:

• Seen and held-out rising together → real improvement. Keep climbing.
• Seen rising while held-out stalls (the gap widening) → the loop is memorizing. The "gains" are counterfeit. Stop, and keep the last generation where both moved.

Three rules keep the guardrail from quietly dissolving, and each one closes a specific hole:

1. The optimizer never selects on the held-out score. The temptation is enormous — it's right there, and it's your honest number. Use it for selection and it becomes a second training set: after enough generations of "keep whatever raises held-out," it's as fitted as the seen set, only slower and less visibly — and you've spent your only tell. The held-out slice answers one binary question per generation — keep going or stop — and nothing else. Even that one bit is a slow leak, which is why long-running loops eventually retire slices and cut fresh ones (lesson 21's retirement rule, on a schedule).

2. The stop is automated. A gap threshold — say, seen exceeding held-out by more than the suite's noise for two consecutive generations — halts the loop, and the halt stays halted until a human looks (lesson 17's halt-and-stay-halted, same reasoning: a stop rule that requires a human watching a chart is a stop rule that fires late, and a system that clears its own stops will, at the worst time).

3. A human merges the winner. The loop proposes the improved configuration as a diff, with before/after scores on both slices attached (lesson 28's propose-don't-dispose, applied to the loop's own output). Configuration rewritten by an optimizer and reviewed by no one is how you discover in production what the suite failed to measure — because the suite's blind spots are, precisely, where an optimizer under pressure will put the bodies.

Run it like a tool, not a pet

Where the loop genuinely pays: high-volume, well-measured, stable workflows — the triage prompt running five hundred times a day, the extraction pipeline with a case-rich suite grown from real failures (lesson 21). A weekend of loop time buying a few real points on work at that scale is excellent economics, and the strong-model- proposes / cheap-model-executes split (lesson 10) makes the loop itself cheap to run.

Where it doesn't pay, and saying so is the professional judgement:

• Thin suites. Twenty cases will be fitted by Tuesday; the loop will spend its entire life in the counterfeit regime.
• Fast-moving tasks. You're optimizing toward last month's distribution; the drift outruns the gains.
• Suites that miss what matters. The loop maximizes exactly what you measure. If tone, safety, or latency isn't scored, the loop will cheerfully spend them to buy what is. The suite's quality is the ceiling on the entire exercise.

Which yields the ordering rule that closes the lesson: grow the suite first, then optimize against it — never the reverse. A rich, honest, versioned suite makes even a dumb optimizer useful; a thin suite makes the smartest optimizer a liability with a dashboard.