Repository navigation
Tower serves API requests before internal wiring completes — terminal DELETE returns spurious 404 during startup window #1261
Description
Problem
Tower opens its HTTP port and accepts requests before its internal dependency wiring is complete. Requests that need _deps (set by initInstances()) during that window fail closed. Concretely: DELETE /api/terminals/:id returns 404 for a terminal that exists — while GET on the same id returns 200, because the GET path doesn't touch _deps.
Root cause
The entire boot sequence lives inside the server.listen() callback (tower-server.ts:375): stale-socket cleanup → session reconcile → orphan kill → husk sweep (#1227) → session-log retention scan (#1238/#1243) → and only then initInstances() (tower-server.ts:575). killTerminalWithShellper() (tower-instances.ts:799) starts with if (!_deps) return false, which the DELETE route (tower-routes.ts:842) translates into 404 Session not found.
The race predates 3.2.4, but the husk sweep and retention scan added in the 3.2.4 window run between port-open and deps-wiring, so the window now scales with disk state: with ~1,400 session-log files (2.2 GB) it exceeds 100ms, enough that afx tower start && <immediate afx command> can hit it. Discovered when tower-terminals.e2e.test.ts > DELETE > kills and removes a terminal began failing deterministically on a log-heavy dev machine while CI (fresh runners, no logs) stayed green. Instrumentation confirmed _deps was null at DELETE time; a 400ms delay before DELETE makes the test pass.
Suggested fix
- Routes that need
_depsshould return 503 with Retry-After ("Tower is starting up") instead of 404/false — generalizing the patternstopInstancealready uses ("Tower is still starting up. Try again shortly."). - Structurally: defer
server.listen()until afterinitInstances()(or hold requests until init completes). Note the ordering comments in the boot sequence — some steps must run beforeinitInstances()reads architect state — so this needs care.
(1) is small and safe; (2) closes the window for good if it proves compatible with the ordering constraints.
Testing notes
The existing DELETE e2e test covers the regression once the window is closed (it currently only fails on machines with large session-log volume — CI won't catch it until the fix makes readiness deterministic).
Activity
waleedkadous commentedon Jul 26, 2026
On it! Working on a fix now.
[Bugfix #1261] Fix: hold Tower API requests until boot wiring completes
[Bugfix #1261] Test: regression coverage for the startup readiness wi…
[Bugfix #1261] Record CMAP verdicts in builder thread
Merge pull request #1263 from cluesmith/builder/bugfix-1261
waleedkadous commentedon Jul 27, 2026
Fixed in PR #1263 (merge commit
757656a).Completion statistics
tower-server.ts,tower-instances.ts,tower-routes.tstower-startup-readiness.e2e.test.ts") + 1 unit (tower-routes.test.ts`)send-integration.e2e.test.tsafterAll10s teardown timeout, all 168 tests passing)What was wrong
The whole boot sequence lived inside the
server.listen()callback withinitInstances()— which sets_deps— as its last step.killTerminalWithShellper()returns a barefalsewhen_depsis null, and the DELETE route read that as "not found".Two findings beyond the original report:
afx tower start's readiness signal was itself dishonest. It polls/api/statusfor a 200, which the pre-fix Tower returned during the window becausegetInstances()returns[]when_depsis null. That is what madeafx tower start && <immediate afx command>racy.What changed
bootSequence()callsmarkBootComplete(). A boot exceeding 20s answers 503 +Retry-Afterrather than hanging clients.initInstances()/initCron()moved ahead of the husk sweep (Codev process fleet drives macOS memory-pressure kills: stranded shellper husks accumulate per restart + no idle-fleet memory policy #1227), the session-log retention scan (agent-farm: PTY session logs have no rotation or retention policy #1238) andinitTunnel(), which now run post-readiness. Readiness should depend on wiring, not housekeeping; and gating it on a remote tunnel connect would make an unreachable cloud endpoint look like a broken local Tower.instancesReady()lets the_deps-dependent delete paths answer 503 instead of 404, or a lying 204 on the workspace tab-delete path that discards the kill result.arch.md's canonical boot-order table was stale as of this change; it is updated, with a new "Boot Readiness Gate (#1261)" section stating the invariant for future startup work: anything a request handler depends on must be wired beforemarkBootComplete(), anything else goes after it.Known adjacent case, deliberately left open
WebSocket upgrades bypass the gate (
setupUpgradeHandlerattaches its own'upgrade'listener). Not reachable in practice — every client makes an HTTP call before opening a socket — and the #997whenStartupReconcileSettled()barrier already covers the WS terminal-upgrade routes for reconcile's output. Worth a follow-up issue if you want it closed explicitly.