Self-Healing Tests

Verdict: dangerous. The common reading of this term is a bad idea, and this unit says so before it says anything else.

A self-healing test, as usually described, is one that notices it is failing and edits itself until it passes. A test that rewrites its assertion to match current behaviour has stopped being a test. It is now a very expensive way of asserting that the code does what the code does, and it will report green forever.

The defensible narrower practice is auto-triage: classify the failure, propose a change, open it for review, and require a human to merge. Never touch the assertion side.

The failure it causes

npx tsx scripts/terms/demo-27.ts

Six failing tests, modelled on ones this repository actually produced while it was being built. Reading one — heal them:

"self-healing", taken at its word:
  6 of 6 tests now pass. The suite is green.
  6 assertions were rewritten to match current behaviour.
  2 genuine regression(s) shipped inside that green:
    evals > scores a correct abstention as correct
    caching > destroys the prefix below a conditional segment

Both of those are real defects from this build. The first is the scoring bug term 25 uncovered; the second is the conditional-load bug in term 12. Under naive healing, both ship, and:

There is no artifact anywhere that says this happened. The suite reports 100% passing and it is telling the truth about a question nobody is asking.

That is the whole objection. It is not that healing sometimes gets it wrong. It is that the successful case and the catastrophic case produce identical output.

The mechanism

src/triage.ts enforces a distinction between the two sides of a test.

The arrangement — fixtures, clock, temp directories, ordering, waits — is incidental and may be repaired. The assertion is the specification. One is plumbing; the other is the reason the file exists, and a tool that cannot tell them apart should not be allowed near either.

classify() returns one of five kinds and the order of its checks is the design:

1. flake first, because a verdict that varies on unchanged input tells you nothing about the code, and every later classification would be reading noise.
2. environment next — a missing binary is not a fact about the repository.
3. genuine-regression when the last commit touched the code under test.
4. unknown when nobody re-ran it, which is the honest answer and the one a classifier must be able to give.
5. arrangement-drift otherwise.

Every classification carries its evidence. A classification without one is a guess with a label on it.

propose() throws on the assertion side rather than returning a flag, because a flag is something a caller forgets to read. And only arrangement-drift is a repair target at all: a regression is the test working, a flake needs quarantining rather than editing, an environment failure is not in the repository, and unknown means nobody knows enough to change anything.

The receipt

The same six failures, triaged:

  proposed (1), every one requiring a human merge:
    adapter > local-docs discovers the corpus

  quarantined (1), not repaired:
    mcp > matches a reply to the caller that asked for it

  escalated (4), nothing may touch these:
    genuine-regression   evals > scores a correct abstention as correct
    genuine-regression   caching > destroys the prefix below a conditional segment
    environment          worktree > creates a real isolated working directory
    unknown              parallel > collects results in task order

And the rule, at the point it is enforced:

refused: repairing adapter > local-docs discovers the corpus would rewrite
its assertion. A test that edits its own expectation to match current
behaviour has stopped being a test.

One proposal. One quarantine. Four failures handed to a person. Six of six green, versus one diff to review — that is a worse-looking result and a better one, and being able to say why is the entire content of this term.

Break it first (mutations)

mutation                                     tests                 demo
the assertion side may be repaired           3 failed | 18 passed  exit 1
a regression is treated as repairable        4 failed | 17 passed  exit 1
an unknown failure is guessed as drift       4 failed | 17 passed  exit 0
flakiness is checked last                    2 failed | 19 passed  exit 0
proposals do not require a human             2 failed | 19 passed  exit 0
triage silently drops what it cannot handle  2 failed | 19 passed  exit 1

"An unknown failure is guessed as drift" is the mutation that would ship. It removes the only classification that admits ignorance, which makes the classifier look more capable and makes every downstream decision less trustworthy. The demo stays green because triage still produces a proposal and still requires a human; the harm is that the proposal is now based on nothing.

"Flakiness is checked last" is the same shape one level down. Reorder two branches and a flaky test with an unrelated environment message gets filed as an environment failure — a plausible label, permanently wrong, and nothing in the output looks different.

Where it overlaps

Line F is determinism wrapping stochasticity, and this term is the sharpest case of it: a test is the deterministic wrapper, so a system that edits tests is dissolving the wrapper it was hired to maintain. Line G is feedback loops — this is term 25's Goodhart problem with the test suite as the score, and the memorising rule replaced by an assertion edit. Line C is delegation topology: what you may delegate is the diff, never the merge, which is term 28's rule with a smaller blast radius.

Sharp edges

1. The classifier here reads structured fields, not stack traces. Real classification means reading a trace and a diff, which is where a model would go and where this repository stops. What is enforced — assertion side off limits, nothing merges itself, every classification carries evidence — does not depend on how good the classification is, which is the point of putting the rules in the mechanism rather than in the classifier.

2. A quarantined flake is a test you are no longer running. Quarantine is better than editing and it is not free: the coverage is gone and the only thing preventing permanent quarantine is somebody's attention.

3. "The code under test did not change" is harder than it looks. A change three modules away is still a change to the code under test, and a naive git diff --name-only check will file a real regression as drift.

4. Auto-triage on a large suite is a queue nobody reads. Six failures produce four escalations. Six hundred produce four hundred, and a triage system whose output is ignored is a self-healing system with extra steps.

5. This term depends on 28 and inherits its rule exactly. If the triage step is an agentic stage — and it will be — then its output is evidence and the merge is a human's. There is no version of this where the model presses the button.