Model Routing

Routing means sending a task to a cheaper model when a cheaper model will do. The pitch is that this is the largest easy saving available, and on the cost side the pitch is correct.

It is also the dangerous half, and the danger is not subtle. Routing is a decision with two inputs: what a model costs, and whether it is good enough for the task. Only one of those is measurable without running the task both ways.

So "route to the cheap model, save 90%" is not an incomplete recommendation. It is a wrong one — the half that causes damage, delivered with the confidence of the half that does not. A unit measuring only the saving would be teaching precisely that, which is why this one does something else.

What is measured, and what is not

The cost side is real, and it is here:

30 tasks from domain local-docs
  all on large  $0.2828
  all on small  $0.0151
  routing everything cheap saves $0.2677 (94.7%)

The quality side is the reader's to supply, from their own eval on their own tasks. decide() takes cheapIsSufficient and returns unverified when it is null:

with no quality measurement : unverified
  routing would save $0.009045 on this task, and that is one of two inputs.
  Whether small is good enough for it has not been measured. Settle it by
  running the task both ways against a held-out set and comparing the
  outputs, which is term 21.
with the reader's eval, pass: route-cheap
with the reader's eval, fail: route-expensive

UNVERIFIED is on the recommendation, not on a number. Every figure above is measured. What is missing is the authority to advise, and the unit says so in the return value rather than in a footnote.

The demo exits non-zero if a recommendation is ever produced from cost alone.

Build it

src/routing.ts prices a task against two published list prices — named and dated, because they move — and refuses to collapse two inputs into one. The saving is still reported while refusing, because withholding a measured number would be its own dishonesty.

Break it first

mutation                                       tests                  demo
recommends on cost alone                       2 failed | 9 passed    exit 1
output tokens never priced                     1 failed | 10 passed   exit 0
declines without saying what would settle it   1 failed | 10 passed   exit 0

Two of those rows only read that way after the tests were fixed.

The output-pricing test originally asserted that a cost was positive and that the cheap model was cheaper. Both stay true if output tokens are dropped entirely, so the cost model could quietly become half a model. It now compares two tasks identical except for expected output.

The third mutation was a bad one before it was a good one: replacing the words "Settle it by running" left "held-out set" in the string the test greps for, so it failed to fail for a reason that had nothing to do with the code.

Where it overlaps

Line A is context economics, and this is the rung where the cost of a token stops being uniform. Line G is feedback loops: the missing input is a measurement, and term 21 is where the repository learns to take it. This unit is the clearest case in the course of a decision that looks like one variable and is two, which is why it sits next to the eval line rather than inside it.