Tokens

Models don't read words or characters — they read tokens: chunks of a few characters produced by byte-pair encoding, an algorithm that builds its vocabulary from the most frequent character sequences in training data. Tokens are the unit in which context windows are measured (lesson 2), bills are priced, and latency accrues. You'll never compute one by hand, but a working feel for them — what's cheap, what's expensive, and how estimates go wrong — explains a dozen things about agent behavior and cost that otherwise look arbitrary.

The intuition: frequency is cheapness

BPE gives common sequences short encodings. the is one token; function (with its leading space) is one token; implementation is one or two. Rare sequences fragment: an unusual identifier, a foreign word, a hash. The consequence is a pricing structure over kinds of text:

Content	Rough rate	Why
English prose	~4 chars/token, ~0.75 words/token	training data is full of it
Typical code	denser per meaning	brackets, operators, indentation all bill
JSON	notoriously heavy	every quote, brace, and repeated key is paid for, per row
Hashes, UUIDs, base64	~1 char/token	maximally rare sequences

Concrete versions of that table, because the numbers make it stick: a thousand words of prose ≈ 1,300 tokens. A myVeryDescriptiveVariableName can cost 5–7 tokens every time it appears. A wide table serialized as JSON can cost several times the same data as CSV — the keys repeat on every row, and every one of those repetitions bills. A single base64-embedded image in a file an agent reads can quietly consume tens of thousands of tokens. Logs full of timestamps and hex are the classic silent budget-killer, which is why lesson 2's advice about taming tool output is a token lesson in disguise.

Two second-order facts worth knowing. Whitespace is real: deeply indented code pays for its indentation, every level, every line. And tokenizers differ by model family — a budget tuned against one model's tokenizer is only approximately right for another's, so calibrate against the one you actually bill against.

Why input dominates everything

In an agent session, the model reads perhaps fifty times more than it writes: your files, its tool output, the accumulated transcript, all re-sent every turn (lesson 2). So when a bill surprises you, the cause is almost never verbose answers — it's input volume: the 30,000 tokens of file contents and test logs riding along with every request, priced per model tier (lesson 10), discounted only where caching applies (lesson 12). This is also why the highest-value token optimizations are structural, not editorial:

• Trim what gets loaded. "Read these three files" versus "read the whole repo" is a 50x input difference at identical task quality.
• Prefer references to payloads. Passing a filename and letting the model read on demand beats pasting content that might not be needed — the idea that matures into lesson 20.
• Exclude token-hostile files. Minified bundles, lock files, vendored deps, giant fixtures. One accidentally-read package-lock.json can out-cost the rest of the session. Tell agents what not to read (CLAUDE.md is the right home — lesson 8).
• Shape agent-to-agent traffic. When workers report to an orchestrator (lesson 22), the format of their reports is a real cost decision — terse structured summaries, not JSON-of-everything.

And the optimization not to make: shaving your prompt from 60 tokens to 45 while the session carries 80,000 tokens of file reads. Wording thrift is bikeshedding; loading discipline is engineering.

Estimation: calibrate or be lied to

Everything serious you'll build later — budgets (lesson 5), refusal thresholds, batch cost projections (lesson 19), routing decisions (lesson 10) — sits on a token estimate. The honest way to get one:

1. Take a real sample of your content — your code, your logs, your docs. Not lorem ipsum; the stuff you'll actually process.
2. Count it with the real tokenizer (Anthropic's API has a count-tokens endpoint; open-source tokenizer libraries approximate well for planning purposes).
3. Derive your own chars-per-token ratio per content type, and add a small headroom factor (5–10%) for variance.

Generic ratios miss badly on atypical content — the chart on this page shows a naive estimator running 44% high on real documents until calibrated, which matters more than it sounds, because estimator error has a direction and both directions cost you. Underestimate and budgets blow through. Overestimate — the direction that feels safe — and every budget check refuses work it could easily afford: an agent that declines a task, a batch that won't launch, capacity planning that buys double. "Conservative" estimation isn't conservative when the estimate gates action; it's just wrong in the expensive-quiet direction instead of the expensive-loud one.

The five-minute habits

• Glance at cost/usage readouts (Claude Code reports them) until your intuition calibrates — you should be able to guess a session's token count within 2x before looking.
• When designing anything that serializes data for a model, do the ten-second format comparison: JSON vs CSV vs aligned text for this payload. It's routinely a 2–4x difference.
• When an agent will read logs, decide the truncation at the source: --reporter=dot, tail -30, --quiet. Tokens not generated are the only tokens that are truly free.
• Label every token number you write down as measured or estimated. Estimates drift, tokenizers change, content mixes shift — the label is what future-you needs to know whether to trust it.

That last habit generalizes into the course's whole measurement discipline (lesson 21): numbers earn trust by how they were produced, and token counts are your first practice at keeping the pedigree attached.