Background Agents

A background agent is one that starts itself — on a schedule, a webhook, an event — does its work, and reports. Nightly dependency triage, a morning test-flake report, auto-labeling new issues, a weekly docs-drift check: the class of work that's valuable, boring, and never urgent enough to do by hand. The scheduling is trivial — cron is fifty years old; every CI system has a schedule: trigger; Claude Code runs headless from any of them (lesson 18 covers the invocation). Everything non-trivial about background agents comes from one fact: nobody is watching. Unattended failure has three shapes attended work never taught you to fear, plus a stop-rule discipline and an authority boundary. That's this lesson.

Failure shape one: overlap

Monday's run hangs on a slow network call. Tuesday's fires anyway. Now two agents, same job, same files — lesson 13's shared-directory corruption, on a timer, with no human around to notice the pile-up. By Thursday there are three.

Every scheduled agent needs a concurrency answer designed, not defaulted: a lockfile the run takes and releases; your scheduler's built-in guard (GitHub Actions' concurrency: key does this in two lines); or a fresh worktree per run (lesson 13) so overlapping runs at least can't corrupt each other. And the policy question stated explicitly: when the previous run is still alive, do you skip this firing, queue it, or kill the stale run? For most maintenance jobs, skip is correct — a missed night costs almost nothing; an interleaved night costs an investigation. Write the choice down in the job itself.

Failure shape two: drift

A run fails halfway and leaves partial state — half-updated files, a held lock, an abandoned branch, a temp directory. The next run starts from that wreckage and compounds it. By Friday the job fails for reasons four days removed from the original cause, and the investigation has to dig through the sediment of every intermediate failure.

The cure is a lifecycle rule with no third state: every run starts from known state and ends in one of exactly two conditions — completed, or cleaned up. Fresh clone or fresh worktree at start; a cleanup path (trap handlers, a finally, the scheduler's post-step) that runs even on failure. "Failed messily" is not an allowed exit. This is also an argument for making runs idempotent where possible: a job that can safely re-run from the top is a job whose drift surface is near zero.

Failure shape three: silence

The job broke in March. It's discovered in May. Nothing alerted, because the thing that would have noticed is the person who isn't there. Two disciplines close the gap, and the second is the one everyone misses:

Failures announce themselves somewhere humans already look — Slack, an issue, email. Not a log file nobody tails; logs are where alerts go to be archived.

Success leaves evidence too. A heartbeat artifact — "ran at 02:00, examined 34 PRs, acted on 3" — is what distinguishes quiet because fine from quiet because dead since March. Without it, the two are literally indistinguishable, and you'll only learn which one you had when the work it was doing silently stops being done. The rule: silence must never be ambiguous. (Lesson 18's structured run artifacts are the natural heartbeat; this is one more reason every headless run should leave one.)

Stop rules: retry versus runaway

Attended, a failing job gets a human sigh and a fix. Unattended, the retry policy is the judgement, so write it with the care you'd want at 2am:

• Allow N consecutive failures, then halt. Three is a sane default.
• Count consecutive, not cumulative. A cumulative counter eventually halts every healthy job that fails occasionally — a job failing once a fortnight is fine forever, and a counter that doesn't reset on success will kill it by June. This bug is subtle, common, and always discovered as "why did the healthy job stop?"
• Halted stays halted until a human clears it. The halt is a good outcome: a loop that would have burned money reproducing the same failure nightly has converted itself into a visible, stationary problem. The schedule may not restart it; only a person, having looked, may. (Auto-resume defeats the entire mechanism — it turns "halt" back into "retry with extra steps.")

Stack the money guards underneath (lesson 5): per-run budget refused up front, monthly cap at the provider — scheduled × runaway is exactly the multiplication that turns a bug into an invoice.

What deserves a schedule, and how much authority

Three admission criteria: the work is genuinely recurring (one-offs get sessions); it's tolerant of a missed run (if a skipped night hurts, you're building production infrastructure — give it lesson 28's discipline, not a cron line); and it's bounded in authority.

That last one is the deepest design choice, and the principle is worth engraving: an unattended agent proposes; humans (or gated pipelines) dispose. Open the PR — don't merge it. File the issue — don't close others'. Draft the reply — don't send it. Hooks enforce the boundary mechanically (lesson 15); the halt rule bounds the blast radius; propose-don't-dispose bounds the blast consequence — because some run, someday, will be wrong, and the design question is never "how do we prevent all wrong runs" but "what does the wrong run cost." Design for that one run, and the other three hundred are free.

Done right, background agents are the best deal in the course: an evening of setup that runs for a year is hundreds of hours of attention you never spend. The craft is entirely in the guards — overlap, drift, silence, stop, authority — every one cheap on the day you build the job, every one expensive on the week you learn why it was missing.