Prompt-drift check — three agents in two days told a story about themselves their capabilities contradicted

[serge-ivo]

The recurring failure

Three separate incidents in two days, all the same shape — a prompt string asserting something
about the agent that its resolved capabilities contradict:

	What the prompt said	What was true
#254	"NEVER claim you personally ran commands… you do not drive the engine"	start_work was in BASE; the agent had just used it
#255	"attach a repository in the Repo tab"	the agent declares surfaces: ["coding"] — it has no Repo tab
#247	Coder docs described a tmux pane	the engine is a child process; tmux was removed in June

Each was fixed individually. The class is untouched, and the codebase already diagnosed it — from
toolBlurbFor:

Describing tools the agent doesn't have is not cosmetic: it tells the model a story about itself
that its actual tool set contradicts, and the model believes the story.

That comment was written after Local Repo Chat refused to read a repo it could already read. It
is the same lesson, and it keeps being learned one incident at a time.

Why it keeps happening

Capability is data — toolNamesFor(capabilities), capabilities.surfaces,
SELF_WRITABLE_FIELDS. Claims about capability are hardcoded English in agent-think.ts.
Prose cannot be type-checked, so it drifts the moment the data changes, and the drift is invisible
until an agent says something wrong to a user.

There are currently 8+ prompt lines naming UI locations ("the Coding tab", "the Repo tab") that
are not derived from capabilities.surfaces. None are wrong today. That is exactly what was true
of the tmux lines in June.

Proposal: a prompt-drift check, in the shape the repo already uses

scripts/docs-drift.mjs does this for documentation — 8 checks, including "MCP tool table: 124
registered, 124 documented, exact match". instance-config.test.ts does it for a code pattern,
scanning the tree for raw UPDATE agent_instances SET config because "a reviewer cannot be
expected to catch the seventeenth by eye".

Same idea for prompts. Build the system prompt for a set of representative capability fixtures and
assert:

• No surface named that the fixture does not declare. A ["coding"] agent's prompt must never
  contain "Repo tab". This alone would have caught [bug] A Repo Coder is not told it owns a repo — ownership lives in a memory string, and it cites tabs it does not have #255.
• No tool named that toolNamesFor did not grant — and, the inverse that bit us, no denial of
  a tool that WAS granted (start_work present ⇒ the prompt must not say it cannot act). [bug] The prompt still forbids what start_work does — a Coder denied real work and told the user to redo it #254 was
  the inverse direction, which is the one nobody thinks to check.
• Every UI location mentioned resolves to a surface in the registry — the same check
  docs-drift already makes for /docs/<slug>/ links.

Fixtures worth pinning: a Repo Coder (coding, drive:false), Repo Chat (repo), a cloud-only
chat agent (no surfaces, no runtime), and the legacy Coder.

Why this is worth a ticket rather than more care

Care was applied. Every one of those three lines was written by someone who understood the system,
and #254's was written as a correctness fix. The lines went stale because capability moved, not
because anyone was careless — and that is precisely the situation a mechanical check exists for.

Verification

• Adding a tool to BASE without updating the prompt fails the check with the contradicting line
  named.
• Renaming or removing a console surface fails any prompt that still names it.
• The check runs in CI next to docs:drift.

[serge-ivo]

Shipped in a39af44.

The guard — workers/api/src/lib/prompt-claims.ts + prompt-claims.test.ts. It adjudicates claims keyed to a named capability, and nothing else:

kind	the claim	checked against
tab	"the tab"	tabsFor(capabilities)
tool	a catalog tool name	toolNamesFor(capabilities)
denial	"you cannot run shell commands", "You are READ-ONLY", "you do not drive the engine"	predicates over the resolved SelfModel — the inverse direction #254/#318 needed
runtime	pags up, "tmux", "the indexed code"	capabilities.runtime, the tmux surface, hasCodeIndex

Two halves: a semantic check that builds the real prompt for the five first-party agents across every runtime branch and asserts zero violations, and a ratchet over the prompt modules' string literals that pins the exact set of hardcoded claims (counts included) so a new one lands in review.

What it cannot see, stated in the module header rather than implied: free-text advice with no named capability; whether a tab the agent has is the right one; sub-locations that are not tabs (KB sub-tabs, sections — reported unadjudicated, never failed); text assembled at runtime from D1/DO rows; prompts built outside the scanned modules.

Lexing. promptTextOf is the exact inverse of source-guard.ts: here the subject IS the string literals, and of the ~14 tab mentions in workers/api/src nearly all are comments explaining these incidents. Single-token literals are skipped — "tmux" as a surface id is an identifier, not a sentence.

Enabling change. Every capability claim moved out of agent-think.ts into pure modules (lib/agent-style-prompt.ts); a test asserts the file now hardcodes none, so the one place a unit test cannot call contains nothing to check. agent-think.ts 894 → 828, pin updated.

Three live violations found on the first run, all fixed here:

• tabsFor never learned about the Stats tab the console shipped in [stats] Console Stats surface — four card kinds, inline SVG, one page-level window #311 — every agent was told its tab list "exactly" with Stats missing, while get_stats' own description points at it.
• executionAuthorityPrompt pointed a Coder Lead at subordinate_status, which the Lead does not declare — [bug] check_work makes a supervisor deny work it just delegated — the #254 guard misfires on delegators #318's fix shipped with the over-reporting half of the same bug in it.
• the coding STYLE block told every coding agent to file issues with github_create_issue; only the Repo Coder declares the GitHub connector.

Verified by mutation — each watched go red, then reverted: #254 (denial made unconditional), #255 ("Repo tab" hardcoded back), #247 (a tmux pane in the Coder prompt), #318 (subordinate_status named unconditionally), plus a new hardcoded "Coding tab" literal in agent-think.ts against the ratchet.

CI status: no CI run verified this. GitHub Actions is in a declared major outage. Verified locally: 4233 unit tests, pnpm -r typecheck, biome check, docs:drift, and the file-size ratchet all pass.