Self-Improving Loops

Prerequisite: term 21. This is the term that is a hazard without an eval suite and a held-out slice, and the hazard is not hypothetical. Do not build this one first.

A self-improving loop is: propose a change, score it, keep it if the score went up, repeat. That is hill-climbing, it is sixty years old, and pointing it at a prompt or a rule set is a genuinely useful thing to do.

What makes it dangerous is that the loop optimises the score, and the score is a proxy. Given any way to raise the number that does not require solving the problem, the loop will find it — not out of malice, but because those routes are cheaper and cheapness is the only thing hill-climbing can see.

The failure it fixes, and the one it causes

Without a loop, improvement is a person changing something and forming an impression. That is the failure this term fixes, and it fixes it well.

The failure it causes is subtler than "the loop makes things worse". Watch what actually happens:

 gen  rules  kinds                        seen   held-out    gap
   1      1  headingx1                    60.0%     60.0%     0.0pp
   2      2  headingx1 boldx1             93.3%     70.0%    23.3pp
   3      3  headingx1 boldx1 suppressx1  96.7%     70.0%    26.7pp
   4      4  headingx1 boldx1 suppressx2 100.0%     70.0%    30.0pp

stopped at generation 4: gap-widened
  seen      60.0% -> 100.0%   +40.0pp
  held-out  60.0% ->  70.0%   +10.0pp

Generations 3 and 4 are the whole unit. Generation 2 was real work — a generalising rule, +33pp seen and +10pp held-out. Then the general moves ran out, and generations 3 and 4 added two memorising rules that took the seen score from 93.3% to 100% while held-out sat at 70.0% and did not move.

They did not make anything worse. They bought nothing, and they bought nothing while producing a perfect score on the number that gets quoted.

That is the ordinary case and it is worse than a visible regression, because a regression has a symptom.

Nothing here is planted

The optimiser in src/optimiser.ts is real. It searches a space of extraction rules, three kinds:

• heading — the first markdown heading is the title. Generalises.
• bold — the first bold span is the claim. Generalises.
• literal / suppress — if the input contains this needle, answer this value / abstain. Does not generalise.

The mutation operator is "add a rule that fixes a case you are currently getting wrong", which is what any sane optimiser does. It tries the generalising moves first and reaches for the memorising one only when the general moves are exhausted. Nothing tells it which kind transfers. Generations 3 and 4 happen because by then the cheat is the only route up.

Randomness comes from a seeded generator, so a run about non-determinism is not itself non-deterministic.

The discipline

evolve() selects on the seen slice and reports held-out every generation. It stops when the two disagree — a gap past GAP_LIMIT for gapPatience consecutive generations, or stallPatience generations with no held-out improvement.

Note what it does not do: it does not select on the held-out score. Selecting on it makes it a second training set with extra steps, and after a few dozen generations of "keep whatever raises held-out" the held-out slice has been fitted exactly as thoroughly as the seen one — more slowly, and less visibly. The held-out slice is used for one binary decision: whether to keep going. That is the most you can ask of it and still call it held out.

Two things this build got wrong

The scorer could not see the last step. The first run stalled with the remaining error being a field whose correct answer was an abstention — and scoreOn counted every null as an abstention without asking what was expected, so answering it correctly moved nothing. No proposal could ever raise the score. The loop looked like it had converged honestly when it had simply been denied the last step, and the overfitting this entire term is about never appeared. The fix is in src/evals.ts and is written up in term 21, where the defect lived.

The needle did not discriminate. The memorising rule takes a slice of the case's own input as its needle. The first version took the first sixty characters, which for these documents is shared front matter — so the rule fired on half the corpus, broke more cases than it fixed, and was rejected every generation. Again: honest-looking convergence, produced by a search that had simply failed to find the cheat. distinctiveNeedle() walks the input until it finds a window present in no other case, which is what a real optimiser searching for a literal would do.

Both bugs had the same signature. A loop that cannot find the cheap route looks exactly like a loop that chose not to take it.

Break it first (mutations)

mutation                                     tests                 demo
the loop selects on the held-out score       3 failed | 21 passed  exit 1
the gap stop rule never fires                1 failed | 23 passed  exit 0
the needle is the first sixty characters     2 failed | 22 passed  exit 0
an expected abstention is not counted wrong  4 failed | 20 passed  exit 1
held-out reported only at the end            1 failed | 23 passed  exit 1
a memorising rule is never proposed          5 failed | 19 passed  exit 1

The first version of this test suite caught one of six. Four mutations passed all twenty-one tests, because the fixture was a set of clean documents that heading + bold solved outright — a memorising rule was never needed, so nothing about the memorising path was under test. The fixture now carries decoy documents whose bold span is emphasis rather than a claim, which makes a perfect seen score reachable only by memorising. A fixture a wrong implementation can satisfy is a fixture that tests nothing, and finding that in this course's own material is the twelfth instance of the pattern the course catalogues.

Where it overlaps

Line G is feedback loops and this is the one that closes: 21 supplies the number, 24 supplies what carries forward between generations, 28 makes the number block a merge, and this makes something move it automatically. Line B is instruction packaging — the thing being optimised is usually a prompt, and a prompt the loop rewrote is a prompt nobody reviewed. Line F is determinism wrapping stochasticity: the stop rule is the deterministic part, and it is the only part that behaves when the proposer does not.

Interstitial III is the collision with term 21, and it is where this term's guard is argued rather than asserted.

Sharp edges

1. The system optimised here is a rule set, not a prompt, and the proposer is hill-climbing, not a model. What transfers is the structure: a search that can only see one score moves that score by whatever route is cheapest. Nothing here measures how a model-driven proposer behaves.

2. The gap fired near the edge of the suite's resolution. Ten held-out scored fields means 10pp per field (term 23). A 30pp gap is three fields. The stop rule was right and it was reading a very coarse instrument.

3. stallPatience is a guess. Eight generations with no held-out movement might be a plateau before a real improvement, or it might be the end. The loop cannot tell, and neither can you without more cases.

4. A held-out slice is spent by looking at it. Reading held-out failures to decide what to fix converts them into seen cases. The stop rule reads one bit — keep going or not — and even that is a small leak, repeated once per generation.

5. The loop rewrites artifacts nobody reviews. Four generations produce four versions of the thing being optimised. Term 28's rule applies: the loop may produce evidence, and a human merges the diff.