Tower/shellper: architect session-ID collisions + non-recovering crash loops after workspace restart #1224

Activity

1. waleedkadous

   added

   area/towerArea: Tower server / agent farm CLIArea: Tower server / agent farm CLI

   on Jul 22, 2026
2. waleedkadous commented on Jul 22, 2026

   

   waleedkadous

   on Jul 22, 2026

   ContributorAuthor

   More actions

   Forensic timeline from the reporting workspace (all times PDT, 2026-07-22; ~8 architects + ~15 builders; architect names anonymized as A/B/C).

   Setup

   • ~04:22 — full Tower restart. Main architect shellper up 04:22:41 (pid 37635).
   • 04:24:07–08 — restore wave via afx workspace add-architect: six sibling shellpers (pids 39739, 39773, 39812, 39831, 39849, 39982 — including the three that later failed).
   • 04:51:04 — one more architect added (pid 61274, session 89f3b4c0-a6d7-47cb-b71a-7cc38c0627a3).

   Incident 1 — architect A (restored, then re-added): session-ID reuse collision with a dead sibling's remnant

   • 05:21:41 — a sibling's afx send to A returned NOT_FOUND: first silent death — first-wave shellper 39982 already gone from the registry.
   • ~05:21:59 — re-add of A spawned shellper pid 18907, whose args carried --session-id 166e1c25-fd66-46ed-a81c-5dfcf2295efd (persisted ID re-fed on respawn).
   • ~06:10 — that terminal crash-looping on Session ID 166e1c25… already in use. At investigation time, 18907 was the only process referencing that session; A absent from the registry; A's builder alive.
   • Inference: the re-add reused the persisted session ID while some remnant of the dead first-wave instance still held it (holder no longer identifiable by then). Plain kill of 18907 + fresh add-architect at 06:36 (pid 61353) → stable since.

   Incident 2 — architect B: pure silent vanish

   • First-wave shellper 39849 (started 04:24:08) simply gone by ~06:30 — no crash loop, no registry entry, no zombie. Re-add (pid 83869) → stable.

   Incident 3 — architect C (richest data): foreign holder, then wedge-after-free

   • ~06:56 — user accidentally started a foreign interactive claude on their own tty s119 (pid 97841, wrong auth), which claimed C's session 89f3b4c0…. C's registered shellper 61274 crash-looped with already in use.
   • ~07:0x — user quit the foreign process. Key datapoint: with the session demonstrably free (zero processes referencing it), 61274 still did not recover — observed child pids 98525 then 166, each dying <8s after spawn, across 60+ seconds of watching.
   • Meanwhile C had silently deregistered: afx workspace remove-architect C → not found while pid 61274 was alive and looping.
   • ~07:12 — plain kill 61274 + add-architect → shellper 4599, child 4600 immediately stable (24s+ uptime at first check).

   Pattern summary

   • 2× session-ID reuse collision (one self-inflicted stale remnant, one foreign process) — consistent with session IDs being persisted per architect-name and re-fed to claude on respawn → fix (a).
   • 1× wedge-after-free — crash loop persists with the session demonstrably free → fix (b) is the cure for the loop itself, and the wedge suggests the failure isn't (only) the session collision.
   • 3× silent deregistration with process state diverged from the registry (both directions) → fix (c).

   Reporter is willing to run live diagnostics on the next occurrence.
3. waleedkadous commented on Jul 22, 2026

   

   waleedkadous

   on Jul 22, 2026

   ContributorAuthor

   More actions

   On it! Working on a fix now.
4. waleedkadous commented on Jul 22, 2026

   

   waleedkadous

   on Jul 22, 2026

   ContributorAuthor

   More actions

   Live recurrence + recovery datapoint (maintainer's machine, 2026-07-22 ~14:00Z): the reporting workspace's default main architect hit the same pattern — deregistered from Tower while its original shellper (the pid from the timeline's 04:22 restart) was still alive and holding the session, ~9.5h after the workspace restart. Recovery differed from the named-architect cases: afx workspace add-architect --name main is refused ('reserved for the workspace's default architect'), so the path is: verify the zombie via the untracked-PID diff (running shellpers − GET /api/terminals), plain kill it, then re-run afx workspace start from the workspace root — which recreates the default architect (new shellper registered, child stable immediately, no crash loop). Fix (c) should account for the default architect having a different re-registration path than added architects.
5. waleedkadous

   added 9 commits that reference this issue on Jul 22, 2026

   [Bugfix #1224] Fix: skip resume when a live process holds the archite…

   ...

   0f7fb4b

   [Bugfix #1224] Test: live-holder guard in resolveArchitectLaunch

   ...

   a5e1699

   [Bugfix #1224] docs: builder thread — investigation, fix, validation

   9055c80

   [Bugfix #1224] docs: thread — CMAP results and pr gate

   761e1d9

   [Bugfix #1224] Fix: mint-or-reclaim architect sessions + symptom-B li…

   ...

   f28f330

   [Bugfix #1224] Test: mint-or-reclaim classifier, never-kill-foreign, …

   ...

   f7d1324

   [Bugfix #1224] docs: thread — iteration 2 (mint-or-reclaim + symptom B)

   2b75b9b

   [Bugfix #1224] Fix: confirm death after SIGKILL in reapShellpers; no …

   ...

   5c9b4cb

   [Bugfix #1224] docs: thread — CMAP iter2 (reap-race fix + rebuttal)

   894a8bc
6. amrmelsayed commented on Jul 22, 2026

   

   amrmelsayed

   on Jul 22, 2026

   Collaborator

   More actions

   Investigation findings from main architect (2026-07-23) — three datapoints for the in-flight fix (PR #1225)

   1. Empirical: the collision does not reproduce on current claude (2.1.216) in interactive mode. Test on this machine: instance A launched with --session-id <uuid>, conversation materialized (jsonl on disk, holder alive and responding); instance B launched with the same --session-id, same cwd — B started cleanly to the REPL and answered prompts, no 'Session ID already in use' at startup or at message time. So the instant-exit collision the adopter hit is claude-version- or mode-dependent, not a current universal behavior. Implication: PR #1225's ps-scan guard is the right shape — it doesn't depend on claude's collision behavior at all — but the fix should not assume the error string/exit behavior as a stable contract for tests or detection. (Method: tmux + remain-on-exit; happy to share exact steps.)

   2. A deregistered-but-alive shellper is never reaped by ANY existing path — relevant to fix (c) scope. The Symptom-B zombie (live shellper, no registry row) survives even the next Tower restart: killOrphanedShellpers (session-manager.ts) probes each candidate's socket and skips kill when the socket is responsive ('reality trumps SQLite' — correct for the #1198 incident it was hardened against, but it means a healthy-socket zombie whose rows were deleted is permanently invisible to the sweeper). shellper-main.ts meanwhile deliberately stays alive after child exit waiting for a SPAWN frame that, post-deregistration, never comes. So both divergence directions currently have no automated reconciliation; fix (c) needs an explicit 'live shellper with no registry row → adopt or terminate' decision somewhere.

   3. The crash-loop breaker (#1149) has two coverage gaps in this exact scenario. (i) crashLoopFallback is only wired when resumed && storedSessionId && fallback — a fresh-mint launch gets none (defensible today, but worth noting if (b) is revisited). (ii) The breaker counts only exit.code !== 0 exits — if any claude version/mode exits 0 on the collision (unverified; see datapoint 1's version-dependence), the breaker is blind to it while restartCount still burns toward maxRestarts, after which removeDeadSession + the ptySession exit handler produce exactly the Symptom-B divergence (row deleted, process alive). Same exit-code-semantics territory as #1214.

   Also for the record: #1224 is orthogonal to the recently merged #1198/#1204 (reconnect-error swallowing) and #1215/#1218 (replay-wait stall) — launch-args layer vs connection/replay layers; neither caused nor mitigates this.
7. waleedkadous

   closed this as completedin #1225on Jul 22, 2026
8. waleedkadous

   added a commit that references this issue on Jul 22, 2026

   Merge pull request #1225 from cluesmith/builder/bugfix-1224

   ...

   8e4ad34
9. waleedkadous commented on Jul 22, 2026

   

   waleedkadous

   on Jul 22, 2026

   ContributorAuthor

   More actions

   ✅ Resolved — PR #1225 merged to main (2026-07-22 22:50Z)

   Fixed via the BUGFIX protocol (builder bugfix-1224).

   Root cause

   resolveArchitectLaunch resumed a stored architect session id whenever its transcript existed on disk (#1145 ownership check), but never checked whether a live process still held it. After a workspace restart a stale shellper remnant (or an unrelated foreground claude) keeps the id, so claude --resume <id> dies with "Session ID is already in use" and the shellper crash-loops forever. The existence check couldn't catch it: a held session's transcript exists precisely because the holder is writing to it.

   What shipped (maps to all three reported fixes + symptoms A & B)

   • (a) Live-holder aware resume + JSON-argv detection. sessionIdNeedles matches both the claude child's shell argv (--session-id <id> / --resume <id> / = forms) and the shellper parent's config-JSON form ("--session-id","<id>") — the durable evidence while a crash-looping remnant's child is dead between respawns.
   • Mint-or-RECLAIM. A holder verifiably our own superseded shellper (shellper-main.js + matching session id + cwd + CODEV_ARCHITECT_NAME) is reaped (process-group SIGTERM→SIGKILL, death confirmed) and the conversation resumed. A foreign holder (e.g. the user's own tty claude) is never touched → mint fresh. Wired into add-architect and main's launch path.
   • (b) Crash-loop breaker. Give-up (SessionManager maxRestarts) now reaps the shellper husk and logs the dying child's exit code/signal + stderr (builds on the existing Tower recovery: Claude architect crash-loops on stale --resume with no user escape #1149 fallback).
   • (c) Registry/process consistency (symptom B). remove-architect reaps a live-process-without-a-registry-row zombie instead of failing not found; give-up leaves no row-gone/process-alive divergence.

   Recovery for the reported incidents is now automatic: incident 1 (self stale remnant) → reclaim+resume; incident 3 foreign holder → mint fresh, holder untouched.

   Honest caveat

   The wedge-after-free datapoint (children dying <8s with the session demonstrably free) is captured-for-diagnosis (give-up now logs exit reason + reaps the husk so the loop can't persist), not definitively root-caused — claude's own error surfaces via the PTY/ring buffer, not shellper stderr. The offer to run live diagnostics on the next occurrence would confirm it.

   Stats

   Metric	Value
   PR	#1225 (regular merge)
   Net diff	+1023 / −9 across 9 files
   Code	architect-session-holder.ts (new), tower-utils.ts, tower-instances.ts, session-manager.ts
   Tests	architect-session-holder.test.ts (new) + tower-utils.test.ts + session-manager.test.ts — full suite 3583 pass / 0 fail
   Review	2 iterations. CMAP: Claude APPROVE(HIGH) ×2; Codex APPROVE(HIGH) → REQUEST_CHANGES (fixed a real post-SIGKILL death-confirmation race)
   Gates	pr (human-approved)
   CI	all 6 checks green
10. waleedkadous

    added a commit that references this issue on Jul 25, 2026

    [Bugfix #1241] Fix: only auto-restart the agent on unnatural exits

    ...

    44809eb