23. Ultrathink

Every lesson in this course builds one system: a stablecoin reserve attestation registry that discovers issuers, reads the PDFs they publish, and refuses to publish a field it could not read. What it is and where this lesson sits.

Verdict: thin. It is a knob, not an architecture — a token budget for reasoning. Raise it and the model is allowed to spend more before answering. The real practice is one sentence: spend more where the branching factor is high, then check whether the spend moved the eval score.

The part worth code is the checking, because almost nobody does it and the reason is arithmetic rather than laziness. A suite has a resolution — the smallest change it can distinguish from no change — and that is set by the number of scored fields, not by anything about the system:

the suite: 20 cases, 40 scored fields (10 held out)
  smallest change it can see, whole suite : 2.5pp
  smallest change it can see, held out    : 10.0pp

  +0.5pp   INVISIBLE  200 scored fields would be needed to see it, against 10 here
  +1.0pp   INVISIBLE  100 scored fields would be needed
  +3.0pp   INVISIBLE   34 scored fields would be needed
  +10.0pp  visible    at least one scored field of 10
  -2.0pp   INVISIBLE   50 scored fields would be needed

The regression row is the dangerous one. A 2pp drop is equally invisible, so an unmeasurable improvement and an unmeasurable regression report identically.

detectable() returns false meaning "this suite cannot distinguish this from no change" — not "no change". Those are different statements and only the first is supported.

And the trade, priced:

a 8000-token reasoning budget over 200 turns a day at $15/M:
  $24.00 a day, for an effect this suite cannot measure
  the same tokens would build 1777 more suite cases, at which point it could

Sharp edge

No reasoning budget was varied here and none could be: this repository has no model access, and its system under test is deterministic, so the score would not move at any budget. What is measured is the suite — how small a change it can represent — which is a property of the suite alone and holds for any system under test.

Break it first

mutation                                  tests                 demo
resolution counts cases, not fields       1 failed | 12 passed  exit 0
undetectable is reported as no change     3 failed | 10 passed  exit 1
held-out judged against the whole suite   1 failed | 12 passed  exit 0

Two of three leave the demo green. A resolution computed from cases rather than scored fields is off by exactly the number of fields per case, which is invisible until somebody adds a third field.

A personal teaching project, in development. The registry it builds is a teaching artifact and is not an assurance opinion about any issuer. Not affiliated with or endorsed by any employer, or by any vendor whose tools it describes. Where a unit depends on a specific flag or path, it names the version it was verified against. All units.