Agent CI/CD

Continuous integration is the pipeline that runs on every push and decides whether the change may merge. Putting a model inside it is the last rung of the execution-surface ladder: term 18 removed the human from a run, term 17 removed them from the decision to start one, and this makes the run's output a verdict that blocks other people's work.

That last word is the entire term. A verdict that differs between two runs of the same commit is not a verdict, and everything below follows from taking that seriously.

The rule

An agentic stage may produce evidence. It may not be the gate.

A model reads, judges, drafts, summarises, and writes an artifact. A deterministic check reads that artifact and decides. The moment a model's own output is the pass/fail, you have a gate whose behaviour is a distribution.

The failure it fixes

The symptom is a red build that goes green on retry.

What happens next is the actual damage, and it is social rather than technical. People learn to hit retry. Once retrying is the normal response to a failure, the pipeline is still running and no longer gating anything — a red build has become a delay rather than a signal. Nobody decided this and nobody can point at the moment it happened.

The second failure is quieter. An eval suite wired into CI needs a threshold, and a threshold gets written as a number. When the build goes red, the cheapest fix in the world is to edit the number, or to edit the suite. Both turn the build green. Neither is visible as anything except a one-line diff in a config file that nobody reviews closely.

Break it first

npx tsx scripts/terms/demo-28.ts

The demo wires term 21's real eval suite into a pipeline and puts a deliberately unstable agentic stage next to it. Nine attempts, unchanged input:

the agentic stage, nine attempts on unchanged input:
  verdicts        P P F P P F P P F
  distinct        2 verdict(s), 9 distinct evidence digest(s)
  stable as a gate: false
  a build that goes green on retry 67% of the time is not a gate, it is a coin.

Note the second line carefully. Nine distinct evidence digests and two distinct verdicts. The evidence varying is fine — that is what a model does, and demanding byte-identical prose from one is a fight you lose. The verdict varying is not. stability() checks the second and ignores the first, and that distinction is what makes agentic stages usable at all.

The mechanism

src/pipeline.ts, four pieces.

Stage kinds. deterministic and agentic are different types, and assertGateable() throws on the second. It refuses even a stage whose verdict has been stable a hundred times running, because the rule is about the kind, not about observed behaviour — a rule that only fires once instability has been observed fires after the damage.

Stability. stability(stage, attempts) runs a stage repeatedly against unchanged input and reports whether the verdict was stable. It is a sample, and the sample size is a design decision: two attempts against a stage that turns over every third run reports stable, which is a failing test in the suite rather than a footnote here.

Baselines. A Baseline is a committed artifact — a score, the suite hash it was measured on, a tolerance, and the commit that recorded it. Four gate calls from the demo:

  the same system, re-measured           pass              merge=true
  a system that answers nothing          regressed         merge=false  -50.0pp below the baseline
  a higher score on an edited suite      suite-changed     merge=false  the two scores are not comparable
  a first run with nothing to compare to baseline-missing  merge=false  cannot tell improvement from regression

Row three is the one that matters. 99% on an edited suite blocks the merge. The score went up and the build stays red, because a changed suite makes the two numbers incomparable — so it is not a passing build with a caveat, it is a build that cannot be judged. Re-baselining becomes an explicit commit somebody reviews, which is the only defence against the threshold quietly sliding down.

Budget. runPipeline sums every stage's estimate and refuses the whole run before the first stage executes:

  estimated 45000 tokens exceeds the per-run ceiling of 30000;
  refused before the first stage ran
  CI runs on every push, and the number of pushes is not yours to choose.

Summing matters. Four stages of 300 tokens are each under a 1000 ceiling and together are over it, so a per-stage check waves through a run that costs 1200.

The rule, made structural

runPipeline records an agentic stage's outcome and never lets it set failedAt. The demo's last block rigs the reviewer to object:

the shape that ships, under the same ceiling:
  eval                             deterministic  ran=true ok=true
  agentic-review(evidence-only)    agentic        ran=true ok=false
  check-evidence                   deterministic  ran=true ok=true
  estimated 12000 tokens, merge=true
  the agentic stage said ok=false and the merge still passed.
  Its verdict is evidence, not a gate.

If a stage of that kind could halt the pipeline, assertGateable would be a suggestion sitting beside a mechanism that ignores it — which is term 08's finding exactly: priced on every turn, enforcing nothing. The rule has to be in the runner, not in a README.

Break it first (mutations)

mutation                                     tests                 demo
a changed suite passes anyway                1 failed | 20 passed  exit 1
an agentic stage may be the gate             2 failed | 19 passed  exit 0
a failing agentic stage blocks the merge     1 failed | 20 passed  exit 1
budget checked per stage, not summed         2 failed | 19 passed  exit 0
stability compares evidence, not verdicts    4 failed | 17 passed  exit 0
a missing baseline is treated as a pass      1 failed | 20 passed  exit 0

Four of six survive the demo. "Stability compares evidence, not verdicts" is the one I would actually have written first: it looks stricter, it is trivially satisfied by a deterministic stage, and it rejects every agentic stage on earth including the good ones. "A missing baseline is treated as a pass" is the same shape as a vacuous check — the first run of a new pipeline is always green, and it is green because nothing was compared.

Where it overlaps

Line F is determinism wrapping stochasticity and this is its strongest statement in the course: the wrapper is not a nicety, it is the only thing that makes the stochastic part admissible near a merge button. Line D is the execution surface at its far end — 6, 18, 17, 28. Line G is feedback loops: term 21 built the number, this makes the number consequential, and once it is consequential term 25's warning about optimising against it becomes urgent rather than theoretical. Line H is isolation, because a CI agent that writes outside its workspace has done it to a shared runner.

Sharp edges

1. Retry is a feature request from the pipeline you built. If people ask for automatic retries on the agentic stage, the honest reading is that the stage is a gate and should not be.

2. The stability sample is not proof. Nine attempts catch a 1-in-3 flip and miss a 1-in-50. A stage that passes stability() is a stage you have not yet caught, and it is still refused as a gate on kind alone for exactly that reason.

3. A tolerance is a commitment to accept some regression. The demo uses 2pp. That is not "no regressions"; it is "regressions up to 2pp merge silently, forever, cumulatively." Ten of them are 20pp.

4. The token estimate is an estimate. estimatedTokens is stated per stage. A budget enforced against a wrong estimate refuses the wrong runs, and nothing here measures the real spend — term 3's calibration is the nearest thing, and it is a model too.

5. The agentic stage is a scripted flip, not a model. It reproduces the shape of a varying verdict on unchanged input, which is the only property the design depends on. It says nothing about how often a real model disagrees with itself, and that rate is not measurable here.