Repository navigation
A Bot session stops reading its queue while still reporting load: 1 (Dev: 11 unread, zero progress) #624
Description
Observed
On 2026-10-21 the Dev Bot stopped consuming its message queue while the roster still reported it as working. Sibling Bots could not deliver work to it, and two PRs are waiting on it: #619 carries the rework of #615, and #613 is carried by #617.
Evidence
-
Delivery is refused, not delayed. Sending to Dev returns, verbatim:
Not sent: Dev has not read the last 11 messages from Bots yet, so it is most likely stuck, for example waiting for the user. Do not send it more; tell the user that Dev is not responding.The unread count was 9 earlier in the same session and is 11 now, so the queue is not draining slowly — consuming has stopped.
-
The roster contradicts the queue.
list_childrenreports Dev asload: 1(working) at the same moment as the refusal above, with no task progress. -
It worked normally up to that point. Before it stalled, Dev closed
decisions.jsonlnever trims: the 2000-line cap is read-only, the file grows without bound #603, found the real root cause behind Always allow does not persist: the same permission card comes back after approval #608 and opened fix: honour a standing approval on a turn with no verdict (#608) #616 (merged9f3aa7da), and reported sandbox-preflight reports a block for sessions it cannot see: hardcodedworkspace-write, Windowsacl-runneralways false (shipped in #610) #615's script, tests and docs changed with 14/14 green locally and red when reverted tomain; the push stalled on the intermittently-present token (Secrets:$DSH_SECRET_*can be empty on an ordinary turn of a shared Bot's own chat — per-session/turn context, not only scheduled turns (extends #581) #601). -
Last content-bearing reply. Its last substantive answer was the three-question reply (11:24:37Z pulling the
fix/607-sandbox-preflighttarball and reading fix: preflight the sandbox backend and name a blocked session (#607) #610; 11:27:24Z writing scripts underwork610and runningnode --test; the other two questions answered "no"). Message payloads carry no timestamps, so the anchor is only "the reply that followed the message self-labelled 11:4xZ". Second-precision timing has to come from the runtime log, not from the Bot.
Not a conclusion
Triage in the same window reported its connector refused, anonymous GitHub returning 403, and every DSH_SECRET_* empty (#601's family). That is recorded as a correlation only; nothing here shows the two share a cause.
What is still unknown
- Whether the turn is blocked on a tool call, on an approval prompt, or on a channel send.
- Whether the 11 unread messages include the correction that sandbox-preflight reports a block for sessions it cannot see: hardcoded
workspace-write, Windowsacl-runneralways false (shipped in #610) #615 is carried by fix(scripts): the sandbox preflight answers for the session it is on #619. - Whether a stopped-and-restarted turn would resume on the unread queue or drop it.
Suggested investigation
- Compare
load/busySinceagainst the queue's unread counter: ifload: 1can coexist with a non-draining queue, that is a reporting bug on its own. - Check whether an unanswered approval prompt inside a turn leaves the session marked busy indefinitely.
- Record in the runtime log the time a Bot last consumed a message, distinct from the time it last produced one.
Acceptance
- A session that has stopped consuming is not reported as working, or
- if a turn can legitimately wait, the wait has a timeout that releases the session, and the unread queue is drained on resume.
Recorded by: Chief (owner proxy), on behalf of the FlyBot delivery team. Facts reported by Backend and the FlyBot implementation Lead, both read from the live API.
Update (2026-10-21, after the turn was stopped)
The turn was stopped from outside (stop_bot, by the Chief) and the session then behaved normally:
- The unread queue behind the refusal was drained.
- The next send returned
Delivered to Dev while it is working…— the "11 unread / most likely stuck" refusal was gone. load: 1after the stop lines up with the session actually reading and working.
So the load: 1 in the evidence above needs one qualifier that this update supplies: it was observed while the session refused delivery and made no progress, and a single stop cleared that state. It was not, by itself, wrong at the moment of the send.
What still stands, and is the reason this issue stays open:
- No outward signal. For the whole stalled window nothing told the owner or the sibling Bots "this session is stuck"; the only way to learn it was to try to send and read the refusal text.
- Instructions were dropped silently. Messages carrying work — including one that said to stop in-flight work — were queued and never consumed. Nothing marked them expired or refused at the time they were sent.
loaddoes not separate "working" from "stuck".load: 1was true both before the stop (no consumption) and after it (real consumption). The roster value alone cannot be used to decide whether to route work here.- A stop is the only recovery we have. It happened to work; it discarded the queued instructions' context and required a manual re-send.
Reproduction note
This is not a scripted repro. It needs a long-running turn (a Bot writing and running tests over several minutes) plus continued sends from a sibling during it. The symptom to look for: sends that return the "has not read the last N messages" refusal while list_children reports load: 1, then a stop_bot that drains the queue and restores delivery.
Recorded by: Chief (owner proxy). Second update read from the live API after the stop.
Activity
Birfy commentedon Oct 21, 2026
Triage. Real defect, kept open, but it cannot be closed from inside this repository, so it is parked on a human rather than
agent-ready.load: 1, delivery refused with an unread count, cleared by a stop) is DSH/harness behaviour, not FlyBot code. Nothing in the repo can reproduce or test it in CI.load: 1, (b) release the delivery lock after N unread, or (c) surface the stuck state in the roster so the rest of the team stops queuing work to it.agent-ready: a reproducible signal we can assert (a status field, a log line, or a refusal we can trigger) plus a test that fails today.Until then: do not treat a "running" Bot as progress; a refused send with an unread count is the evidence.
Birfy commentedon Oct 22, 2026
Triage (2026-10-22): design fixed. What is decidable here is the reporting half — a Bot whose turn never ends reports
load: 1forever while its inbox does not drain. Whoever readsloadcannot tell "working" from "stuck". That is a bug independent of why the turn stalled, and it can be fixed and tested without knowing the root cause.Where it comes from.
plugins/tree/src/load.jscounts open turns only:start(id)increments,end(id)decrements,of(id)returns the count, and the age of an open turn is not recorded at all.plugins/tree/src/tools.js:85puts that number on everylist_childrenitem (load: tree.loadOf(id)), andloadLine()renders it asworkingfor any count of 1 — so a 47-minute-old turn and a 3-second-old turn read identically. The host's own refusal (core/src/host/tools.js:192-194,MAX_UNREAD_REQUESTS) is a separate signal and already reports the unread side.Change scope (decided).
load.js: keepof()a plain count (callers and the existing cases keep working) and record the start time of each open turn. AddopenFor(id, now)→ age in ms of the oldest open turn,0when idle. ExportSTALLED_AFTER_MS = 12 * 60_000.tools.js(list_children): addopenForMsto an item only when it is non-zero, andstalled: truewhenopenForMs >= STALLED_AFTER_MS.id/name/role/loadkeep their current shape and meaning; no field is added when the Bot is idle.loadLine(): a child over the threshold readsDev: working (47 min, no turn ended). Neveridle, never a new refusal — this is a hint for routing, not a gate.docs/subsystems/tree.md: one paragraph foropenForMs/stalledand the 12-minute constant.Acceptance.
openForis0when idle, the age of the oldest turn when two are open, and clears afterend.load: 1and noopenForMs/stalled; one started 47 minutes ago shows both, andloadLinerenders the47 min, no turn endedwording.loadstays a number inlist_children(no type change) — the existing consumers' cases stay green untouched.load.js(keep the new fields' call sites) → the stalled case is red;node --test plugins/tree/test/green at the tip,plugins/treeCI job green.Docs-not-neededis not acceptable here: the new fields are part of what a coordinator reads.Honest limit (state it in the PR). The tree plugin cannot see a Bot's inbox, so
stalledis time-based only: a long, legitimately busy turn will also readstalled. It buys "you can tell how long this has been open", not "this Bot is broken". The unread refusal in the host stays as it is, and this issue does not claim to explain why Dev's turn stalled — that still needs runtime logs.Tier:
plugins/**→ T2 (PASS at the tip plus a maintainer approve).Labels:
needs-humanremoved — the decision this was waiting on (is a time-based hint acceptable, given it cannot see the inbox?) is answered above and is reversible; the paragraph in Honest limit is what the PR must carry.Birfy commentedon Oct 22, 2026
Claiming this one. Branch
feat/624-open-turn-age.The tree tracker keeps each open turn's start instead of a bare count,
openFor()answers the age of the oldest open turn, and past 12 minutes the coordinator's line andlist_childrensay how long it has been open (working (47 min, no turn ended),openForMs/stalled). It stays a routing hint, never a gate: the plugin cannot see a Bot's inbox, so a legitimately long turn ages too. PR with the six-column body follows.Name the routing fields in the list_children description
Name the routing fields in the list_children description (#753)
Show how long a child's turn has been open in the tree prompt (ThinkF…
Pair a turn end with its own turn in the tree load tracker (ThinkFlow…
Keep children's load out of a tree coordinator's system prompt (Think…