core: pressing Stop inside a 429 backoff window still runs the member turn, and the answer is then thrown away #1679

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: d9cda3a7db614ff98d56abf5162cf3fd
Branch: dev/1-stop-during-backoff
Base: 11a4be7
At: 2026-10-29T11:20:24Z

What happens

When a provider round returns 429, core/src/host/turns.js asks the member again after a backoff instead of spending the turn (that is #1672, live in 1d7c47b1). The backoff timer is not consulted when the person presses Stop: a round stopped inside the backoff window still calls the member's model (and runs its tools) a moment later, and then throws the answer away because the round is gone.

Observed on live main 11a4be75c5b6148d4a1b05fa308ecebe7e3a337f with the probe below (2026-10-29T11:40:04Z, exit 1):

callsAfterStop=1
retried=[{"role":"room","rateLimited":true,"text":"[Group chat \"Ops\" · the provider rate limited the last call, so it is repeated] | …"}]
cancelled=0
result.replies=[]

So: one invisible model call plus whatever tools it runs, for an answer nobody ever sees — during the exact window where the person pressed Stop because they were waiting.

Why

Code reading of core/src/host/turns.js at 11a4be75 (the file is 42 324 bytes, blob edc7532a8a85):

• :621-623 suppresses the retry when the round was already stopped: rateLimited = … && !(info?.role === 'room' && stoppedRoomTurns.has(info.exchangeId)).
• :624-637 schedules the retry: const timer = setTimeout(() => { rateLimitRetries.set(id, { spent: spent + 1 }); retryRateLimited(id, info, origin)… }, wait) — the callback re-checks nothing; stopRoomTurn never clears it. rateLimitRetries is keyed by session only ({ spent, timer }), so clearing from the stop path would first need the exchange id put into that map.
• stopRoomTurn (:167-180) cancels a running turn (entry?.turn !== undefined) or removes the queued inbox copies of that exchange. During a backoff window the turn has already ended and the retry has not been delivered yet, so neither branch matches: cancelled=0, and the stopped exchange id survives in stoppedRoomTurns.
• :711 then drops the answer: const stopped = stoppedRoomTurns.delete(info.exchangeId) → !stopped is false → no lateRoomReply, and the round's waiter was already resolved when the person stopped.

This is a gap in new behaviour, not a regression of old behaviour: 1db1037a (the parent of 1d7c47b1) has no RATE_LIMIT_BACKOFF_MS and no rateLimitRetries at all — the same probe exits 2 there with probe failed: not reached: the backoff was scheduled.

Reproduction

Root at 11a4be75c5b6148d4a1b05fa308ecebe7e3a337f, Node ≥22; cd core && npm ci && npm run harness once. The probe writes only to mkdtemp and removes it.

cd core
# save the probe below as p153-stop-retry.mjs, then:
node --import ./test/harness-resolve.mjs p153-stop-retry.mjs; echo "exit=$?"
# exit 1 = a stopped round asked the member again (defect present), exit 0 = Stop held

Probe (real apply() of the host, real group_relay tool, real participant.cancel, 300 ms backoff):

// P-153 probe: does a round the user stopped still run the member turn that a 429 put on
// a backoff timer?  exit 0 = stop wins, exit 1 = the member works after Stop.
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { apply } from './index.js'
import { fakeContext } from './test/fake-ctx.js'

const seed = () => ({
  version: 1,
  mainBotIds: ['a'],
  bots: {
    a: { id: 'a', name: 'Chief', role: 'Boss', sessionId: 'a', createdAt: 1 },
    b: { id: 'b', name: 'Writer', role: 'Write', sessionId: 'b', createdAt: 2 },
  },
  rooms: { r1: { id: 'r1', name: 'Ops', members: ['a', 'b'], mode: 'admin', admin: 'b', named: true, sessions: { b: 'gb' }, log: [], seq: 0, seen: {} } },
  agents: {},
  exchanges: [],
  exchangeTurns: {},
  questions: {},
  prefs: {},
  revision: 0,
})

const home = mkdtempSync(join(tmpdir(), 'ds-bot-stop-retry-'))
writeFileSync(join(home, 'state.json'), JSON.stringify(seed()))
const ctx = fakeContext()
const warns = []
ctx.logger = { warn: (...p) => warns.push(p.map(x => (x instanceof Error ? x.message : String(x))).join(' ')), info() {} }
const agents = new Map()
const agentOf = (sessionId) => {
  if (!agents.has(sessionId)) {
    const calls = []
    const held = []
    const inbox = { nextStep: [], nextTurn: held, remove(id) { const at = held.findIndex(m => m.id === id); if (at === -1) return false; held.splice(at, 1); return true } }
    const keep = (m) => { calls.push(m); held.push(m) }
    agents.set(sessionId, { session: { id: sessionId }, calls, cancelled: [], inbox, followup: keep, steer: keep, cancel(r) { this.cancelled.push(r) } })
  }
  return agents.get(sessionId)
}
Object.assign(ctx, {
  llm: { listProviders: () => [], listModels: async () => [], resolveModelInfo: async () => ({}) },
  workspaceRegistry: { get: () => undefined, resolveByPath: async () => undefined, create: async path => ({ id: 'w1', path }), archiveSession: async () => {} },
  sessionController: { create: async () => ({ sessionId: 's1' }), resolveAgent: async sessionId => ({ agent: agentOf(sessionId) }), selectModel: async () => {} },
  sessionTitle: { rename: async () => {} },
  sessions: { flush: async () => {} },
})
apply(ctx, { home, rateLimitBackoffMs: [300, 300, 300] })
let seq = 0
const emit = (id, type, data) => {
  for (const listener of ctx.listeners.get('session/event') ?? []) {
    if (listener.length >= 2) listener({ id }, { type, seq: ++seq, data })
  }
}
const tool = (name, args, sessionId, exec = {}) => ctx.registeredTools.get(name).execute(args, { agent: { id: sessionId }, ...exec })
const settle = async (ok, label) => {
  const until = Date.now() + 4000
  while (!ok() && Date.now() < until) await new Promise(r => setImmediate(r))
  if (!ok()) throw new Error(`not reached: ${label}`)
}
try {
  await tool('list_bots', {}, 'a')
  const stop = new AbortController()
  let result
  void tool('group_relay', {}, 'r1', { callId: 'c1', signal: stop.signal }).then(v => { result = JSON.parse(v) })
  await settle(() => agentOf('gb').calls.length === 1, 'the room turn delivered')
  const first = agentOf('gb').calls[0]
  agentOf('gb').inbox.nextTurn.length = 0
  emit('gb', 'turn/start', { turn: 1 })
  emit('gb', 'user/message', { content: first.content, source: first.source })
  emit('gb', 'turn/end', { turn: 1, reason: { kind: 'error', error: { code: 'PROVIDER_ERROR', status: 429, message: '429: Rate limit exceeded for api_key. Limit type: requests. Current limit: 50, Remaining: 0.' } } })
  await settle(() => warns.some(w => /rate limited .*asking again/.test(w)), 'the backoff was scheduled')
  const before = agentOf('gb').calls.length
  stop.abort()                        // the person presses Stop on the round, inside the backoff window
  await new Promise(r => setTimeout(r, 900))
  const after = agentOf('gb').calls.length
  const retried = agentOf('gb').calls.slice(before).map(m => ({ role: m.source?.role, rateLimited: m.source?.rateLimited, text: m.content?.[0]?.text?.split('\n').slice(0, 2).join(' | ') }))
  console.log(JSON.stringify({ finalRole: first.source?.role, result: result?.replies, callsAfterStop: after - before, retried, cancelled: agentOf('gb').cancelled.length }, null, 2))
  if (after > before) {
    console.log('DEFECT: the stopped round asked the member again after Stop')
    process.exitCode = 1
  } else {
    console.log('OK: Stop held; no turn was asked again')
  }
} catch (error) {
  console.error('probe failed:', error.message)
  process.exitCode = 2
} finally {
  await ctx.dispose()
  rmSync(home, { recursive: true, force: true, maxRetries: 5 })
}

Acceptance

Each as its own test, all observable without reading private internals:

1. Stop inside the backoff window (rateLimitBackoffMs:[300,300,300]): group_relay starts a round → the member's turn ends with a 429 → the warn asking again in 300 ms appears → round.signal.abort() (= the room's Stop) → wait 900 ms. Expected callsAfterStop=0, retried=[], cancelled=0, result.replies=[], probe exits 0 printing OK: Stop held; no turn was asked again.
2. Stop before the 429 turn/end (freezes today's :622 guard, must not regress): abort first, then let the same 429 turn/end arrive → callsAfterStop=0 and zero asking again warnings.
3. The timer does not leak into the next round (unit test with mock timers in core/test/rate-limit-retry.test.js): 429 → backoff scheduled → stopRoomTurn(sessionId, exchangeId) → let the 300 ms timer fire → then run a normal turn/start + turn/end (completed) on the same session. Expected: no deliver({rateLimited:true}) for the stopped exchange, and the normal round is delivered exactly once with no rateLimited flag.
4. An own-chat 429 without any Stop still retries (do not switch #1672 off): own-chat turn ends with 429, no stop → when the backoff fires, retried.length=1, retried[0].role="retry", retried[0].rateLimited=true.
5. Only if the same fix covers stopMemberWork: same scenario but the person uses the group's "stop this member's work" instead. Today that path resolves to participant.cancel({ sessionId }) with no exchangeId (core/src/host/groups.js:1283-1293, the call at :1289; the stop path that does carry an exchangeId is :552 / :894), the member is already working:false in snapshot (core/src/host/room-participant.js:100-102), so the stop returns false and never reaches the timer. Expected after the fix: stopMemberWork returns true and callsAfterStop=0. If this entrance is not part of the change, say so and it becomes a follow-up issue — do not silently claim it.

Scope and notes

• One host file plus tests: core/src/host/turns.js. The check that holds is inside the setTimeout callback (:627-634), where info is in scope: if info?.role === 'room' && stoppedRoomTurns.has(info.exchangeId), forget the rate-limit entry and return without calling retryRateLimited. That reuses existing state and needs no map change.
• Clearing the timer from stopRoomTurn alone is not sufficient: rateLimitRetries is keyed by session and stores { spent, timer } (:636), so the stop path cannot tell which exchange the timer belongs to, and the stopMemberWork entrance never reaches it at all.
• Keep #1672's retry semantics intact for everything else (retryRateLimited, rateLimitRetries, the retried guard, the role === 'room' && stoppedRoomTurns.has(...) guards at :622 and :647).
• core/src/host/turns.js is mirrored under patches/ds-bot/** (#1672 touched patches/ds-bot/0001 and 0005): run ds-bot-patches generate and land the regenerated mirror in the same commit; check must print ok 10 patches apply cleanly in order and ok core/ equals ds-bot/ + patches/ds-bot/.
• docs/subsystems/core-host.md documents the retry rule #1672 added. If this change does not alter that paragraph, the tip commit must carry a Docs-not-needed: trailer with the reason — a trailer in the PR body alone fails arch-docs, turns tests-ok red before the suite runs, and costs a review round (see #1675 / comment 6494515948). If the paragraph does need a clause, change the doc instead.
• No client change expected; if the build regenerates patches/ds-bot/0008-client-bundle.patch, say so in the PR body.
• Related but separate: the picked-up probe numbers and the P-154 half (no visible "retrying in N s" while a backoff is pending) are not this issue.

Verification expected in the PR

• Red first on 11a4be75: the new tests against the old turns.js must fail (paste the failing test names/lines).
• TZ=UTC ARCH_DOCS_BASE=11a4be75 npm run verify exit 0 with the per-suite counts.
• The probe above prints OK: Stop held; no turn was asked again and exits 0 against the fixed tree.