You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Orphaned shellper processes accumulate again (regression of #389): af cleanup / Tower restart leave untracked shellpers running #1007
Orphaned shellper-main.js processes are accumulating again — a regression of #389 (and the earlier #294 / #341 / #324, all CLOSED). On the dev box right now there are 51 live shellper processes but Tower's authoritative registry (GET /api/terminals, which is global across all workspaces) only tracks 27. That leaves 24 orphans — running shellpers with no Tower session attached in any workspace, all reparented to PPID 1 (their original Tower instance died).
Forensics (2026-06-06)
Method — subtract Tower's global live set from the running shellper set:
node-25 zombies + two stray /bin/bash inside live builder worktrees (bugfix-2135, bugfix-2164)
Old codev session
1
pid was a node-25 claude session
Two distinct leak sources
af cleanup doesn't kill the shellper — the original af cleanup doesn't kill shellper processes — orphaned processes accumulate #389 defect. Worktree + branch + Tower row removed, but the process survives (e.g. codev bugfix-985: worktree deleted, yet Tower still lists pid 31931 as a live session — a stale-but-tracked row, a sibling of this bug).
Tower restart abandons old shellpers — the node-25→node-26 migration restarted Tower; the old node-25 shellpers detached (PPID 1) instead of being reaped. This is the Shellper processes do not survive Tower restart despite detached:true #324 failure mode resurfacing. Most of the 24 orphans are node-25.
E2E reconnect tests also leak their fixture shellpers (14 of the 24) — the test harness should tear these down.
Expected
af cleanup kills the shellper process group (and children), not just the Tower row.
Tower startup reaps shellpers from a prior instance that no client will ever reattach to (or they should self-terminate when their parent Tower dies — detached:true should not mean "outlive Tower forever").
E2E reconnect tests clean up their spawned shellpers.
Add a reconciliation/GC pass: any shellper-main.js not present in /api/terminals is an orphan and can be reaped.
Detection one-liner
# orphans = running shellpers not in Tower's global registry
comm -23 \
<(ps -eo pid,command | grep shellper-main.js | grep -v grep | awk '{print $1}'| sort) \
<(curl -s http://127.0.0.1:4100/api/terminals | python3 -c 'import sys,json;[print(t["pid"]) for t in json.load(sys.stdin)["terminals"]]'| sort)
Fleet census 2026-08-02 (multi-workspace Tower host, prompted by a sibling workspace's audit): 201 live shellper-main.js processes, 80 tracked in /api/terminals → 121 orphans.
Breakdown (one label per PID, via the running-minus-tracked method this issue documents):
7 × shellpers wrapping real claude/zsh sessions (4 claude --resume, 2 in a stale builder worktree, 1 architect session for an inactive workspace) — held for human confirmation before killing, per the live-architect scar
The sibling workspace additionally observed: Tower tabs surviving builders whose worktrees are deleted (see #1296/#1083), a respawn loop minting ~12 shellpers for one dead worktree (#1296's pattern), and a worktree-type registry row no cleanup flag can target (#1101/#1102).
Activity
waleedkadous commentedon Aug 2, 2026
Fleet census 2026-08-02 (multi-workspace Tower host, prompted by a sibling workspace's audit): 201 live shellper-main.js processes, 80 tracked in /api/terminals → 121 orphans.
Breakdown (one label per PID, via the running-minus-tracked method this issue documents):
~/.agent-farm/test-workspaces/codev-reconnect-*) → filed Shellper reconnect test suite leaks its fixture shellpers: test-workspaces never tear down #1337/bin/echo ready-probewrappers, cwd=/tmp → filed Tower readiness-probe shellper wrappers are never reaped after the probe exits #1336/bin/sleep 300wrappers, cwd=/tmp (probe-adjacent)spir-0042-feature-nametest fixtureclaude/zshsessions (4claude --resume, 2 in a stale builder worktree, 1 architect session for an inactive workspace) — held for human confirmation before killing, per the live-architect scarThe sibling workspace additionally observed: Tower tabs surviving builders whose worktrees are deleted (see #1296/#1083), a respawn loop minting ~12 shellpers for one dead worktree (#1296's pattern), and a
worktree-type registry row no cleanup flag can target (#1101/#1102).