Self-Healing Tests

"Self-healing tests" names an appealing idea — when the suite breaks, an agent fixes it automatically — that is either genuinely valuable or quietly catastrophic depending on one design decision: what the agent is allowed to touch. The industry-default version, auto-update the failing assertion until the suite passes, is the catastrophic one. This lesson draws the line precisely, then builds the version that works: automated triage with a hard wall around the one thing repair must never edit.

The anatomy that makes the line obvious

A test is two parts with entirely different meanings:

• The arrangement: setup, fixtures, selectors, waits, imports — the machinery for exercising the system. Incidental. Drifts constantly as the system evolves.
• The assertion: what must be true — the encoded specification. The reason the file exists.

When a test fails, one of two stories is true: the arrangement drifted (the button moved, the fixture aged, the selector broke), or the system regressed. An agent authorized to edit assertions resolves both stories the same way — it makes the test agree with the code. For drift, that's a repair. For a regression, it just deleted your specification and certified the bug: the suite goes green, the defect ships, and the diff that did it looks exactly like routine maintenance. A suite that cannot stay red under pressure isn't a safety net; it's a green lamp wired to the on position.

The rot compounds socially before anyone notices it technically: once "heal" is the automatic response to red, red stops meaning anything to humans either — the same retry-culture decay lesson 28 describes for flaky gates, here applied to the specifications themselves. The suite still runs, still costs money, and informs no one.

The defensible version: triage, then bounded repair

The correct target for agent effort is diagnosis and arrangement — never specification. The pipeline, built entirely from lessons you have:

Step 1 — classify on evidence, with "unknown" allowed. On failure, a headless run (lesson 18) gathers facts: did the failure reproduce on a clean re-run? did the triggering commit touch the code under test? is the error environmental (missing binary, network, resource)? From the evidence, a classification:

Class	Evidence signature
flaky	fails, then passes on re-run of identical code
environmental	error names something outside the repo
arrangement drift	consistent failure; code under test unchanged
probable regression	consistent failure; code under test changed
unknown	evidence thin or conflicting

Two details carry the integrity of the whole scheme. Insist on the re-run: one execution cannot distinguish flake from fault — zero re-runs means any flakiness verdict is a guess wearing a label. And let unknown exist: a classifier forced to always emit a confident class converts low-evidence failures into confident mislabels, and everything downstream inherits the error. Unknown routes to a human; that's what it's for. (Honest abstention over confident wrongness — lesson 21's scoring philosophy, reappearing as pipeline design.)

Step 2 — route by class. Probable regressions escalate to a human, loudly — that is the suite succeeding at its one job. Flakes get quarantined and ticketed: quarantine is honest deferral — visible, tracked, temporary — but a quarantine list nobody burns down is deletion on layaway, so the ticket is not optional. Environmental failures page whoever owns the environment. Only arrangement drift proceeds to automated repair.

Step 3 — repair the arrangement; propose the diff. Update the selector, regenerate the fixture, fix the import — assertion untouched, specification intact — and the change arrives as a PR with the diagnosis attached (propose-don't-dispose, lessons 17 and 28). A human merges. The gate is not ceremony: classification is probabilistic, and someday a real regression will be misfiled as drift — the untouched-assertion rule plus the human merge are what bound that error's cost to "someone reviews a wrongheaded PR" instead of "a bug gets certified into the spec."

And apply lesson 15's reflex to the pipeline itself: don't ask the repair agent to leave assertions alone — deny it. A CI check that rejects any repair-labeled commit touching assertion lines turns the policy from a hope into a property, and (fail-closed, as always) a repair diff the checker can't parse is a repair diff it rejects.

Measure the triager like the judgement system it is

The classifier is an agent making probabilistic calls, which makes it exactly the kind of system lesson 21 exists for: keep a scored set of past failures with known causes, and check the router against it before trusting it with your weekends — then keep growing that set from live mistakes. Teams that do this honestly converge on the same boring, excellent truth: most failures are flakes and drift, the automation absorbs them beautifully, and the rare true regression reaches a human faster than before — no longer buried under thirty broken selectors. That's the actual prize. Not tests that heal themselves: a suite where red reliably means something, and the something reaches a person with the noise already cleared.