fix(relay): stale Relay workers no longer flood a reopened thread

[rynfar]

Reopening a thread after a Pylon restart filled the end of the timeline with dozens of "Ran 1 subagent · 1 idle" cards for subagent runs that had finished hours earlier.

The Relay worker bridge's startup receipt scan walks the whole activity history and adopts every Relay MCP tool call it finds. On the first boot after the bridge shipped, that bound 284 long-finished jobs spanning 207 old turns in a four-second burst. Their Relay files were long gone, so every one of them failed observation and published a task.progress row — status: "idle", "Relay observer unavailable" — attributed to its original turn but stamped with the current time. Clients render each as a subagent card.

Confirmed against a live projection: all 286 relay.binding rows in the database and all 284 relay-observer-unavailable:* rows shared a single one-minute window at server startup.

An outage row now only explains a worker the thread already started, so a job that never projected anything stays quiet. An outage that has already been announced also pushes its recheck back out instead of spawning the Relay CLI on every sweep for the life of the process.

The existing assertion that a never-projected binding opens an idle row is inverted deliberately: nothing was ever shown for that worker, so there is nothing to downgrade.

Verification: vp test run apps/server/src/orchestration/RelayWorkerBridge.test.ts apps/server/src/orchestration/ThreadBackgroundLiveness.test.ts apps/server/src/orchestration/Layers/ProjectionSnapshotQuery.test.ts — 52 passed. New regression test fails without the source change. vp run -F t3 typecheck and lint on the changed files are clean. No UI code changed.

Model: Claude Opus 5 (1M context), harness: Claude Code in Pylon.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
pylon-marketing	Ignored	Preview	Sep 25, 2026 5:30am UTC

[vercel]

Deployment failed for project pylon-marketing with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/pylon-code?upgradeToPro=build-rate-limit

[rynfar]

Added the projection cleanup for installs that already booted the earlier build.

Migration 063_ClearUnprojectedRelayOutages drops relay-observer-unavailable:* rows whose task has no task.started in the same thread, and keeps the ones that explain a worker the thread really ran. Projection only — the event history is untouched, matching 062_RepairStoppedSessionRequests.

Dry-run against the affected projection: 286 of 286 outage rows in the database match the predicate, i.e. every one of them is an orphan and none are legitimate.

vp test run apps/server/src/persistence — 32 files, 75 passed. vp run -F t3 typecheck clean.

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	14.0 KiB	14.0 KiB	+5 B (+0.0%)	15.1 KiB	✅
Codex	Thread snapshot wire	7.2 KiB	7.2 KiB	−5 B (−0.1%)	7.3 KiB	✅
Codex	Live turn WebSocket wire	6.7 KiB	6.7 KiB	+10 B (+0.1%)	7.8 KiB	✅
Codex	Live turn WebSocket decoded	58.0 KiB	58.0 KiB	0 B (0.0%)	66.4 KiB	✅
Codex	Live turn messages	9	9	0 (0.0%)	21	✅
Claude	Total thread wire	14.0 KiB	14.0 KiB	−14 B (−0.1%)	15.1 KiB	✅
Claude	Thread snapshot wire	7.2 KiB	7.2 KiB	−10 B (−0.1%)	7.3 KiB	✅
Claude	Live turn WebSocket wire	6.7 KiB	6.7 KiB	−4 B (−0.1%)	7.8 KiB	✅
Claude	Live turn WebSocket decoded	58.9 KiB	58.9 KiB	0 B (0.0%)	66.4 KiB	✅
Claude	Live turn messages	9	9	0 (0.0%)	21	✅

Baseline: fd36791 · PR result: a4c6f09 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 115.6 KiB
• Claude decoded thread snapshot: 116.3 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[rynfar]

Blocking review findings at head 69729d0:

1. The exact-head Check job fails typecheck in RelayWorkerBridge.test.ts: SqlClient.SqlClient["Service"] is not a type with a Service property (TS2339), and the resulting fake-engine effect has unknown requirement/error channels (TS377030). Please fix and rerun final-head CI.
2. Migration 063 deletes an outage whenever there is no task.started, even if that same thread/task already has a non-outage task.progress. The new migration test explicitly creates this case (relay-progress:job-historical) and expects the outage removed. foldSubagentActivities creates an agent from a progress-only row, so this can leave a previously visible worker without its outage status. The bridge's hasProjectedStart check has the same blind spot. Please retain/explain outages for any previously projected worker activity, or prove why progress-only Relay rows cannot occur; add a regression for that case.

The historical never-projected binding case is valuable and should stay quiet. I have not merged this PR or run its migration against live data.

[rynfar]

Rebased onto the latest pylon and closed the remaining reliability gap in this path.

An outage that had already been announced returned early without pushing its recheck back out, so every such binding kept a Relay CLI spawn on each sweep — 284 spawns a sweep, indefinitely, on the affected projection. Now a worker that may still return has its recheck pushed out, and a binding is retired outright once Relay itself reports the job gone and nothing about it ever reached the thread. Relay being absent says nothing about the job, so that case keeps the binding and only slows the recheck. The four copies of the teardown are now one helper.

Steady-state Relay CLI spawns for adopted history go from continuous to zero; boot still re-checks each adopted binding once.

vp test run apps/server/src/orchestration apps/server/src/persistence — 87 files, 829 passed. vp run -F t3 typecheck clean.

Companion fix for the second, unrelated defect in the same screenshots (background shell tasks stamped as agents on their terminal row): #853.

[rynfar]

Follow-up to the blocking review at 69729d0: pushed 245db4d on the PR branch after merging current pylon. The bridge now treats any non-outage task activity (including progress-only rows) as evidence that a worker appeared; migration 063 retains its outage in that case and still clears outages for never-projected historical jobs. Added regression coverage for both the migration and a progress-only worker after bridge restart. Focused bridge/migration tests: 9 passed; server typecheck passed. Waiting for this exact head’s CI before merge. No live-data migration was run.

[rynfar]

Closed the last open item rather than leaving it as a known cost, and picked up the hasProjectedWorkerActivity correction already on this branch (a worker whose only projected rows were task.progress would have been silently retired under the task.started rule — that is a better predicate, so I rebased onto it rather than around it).

Retirement is now durable. It only lived in memory, so every job the environment had ever run was re-adopted and re-probed at each start — a cost that grows with history and never settles. I had called that "bounded, one-time per restart"; it wasn't.

It is now recorded on the binding row itself. projection_thread_activities upserts on activity_id, so this writes no new row, needs no new activity kind, and stays out of the transcript. Boot skips a retired binding unless an activation was recorded after the watermark; a resume outranks the retirement, wakes the worker, then answers to the ordinary rules again so a finished one is not re-probed forever.

The watermark is a row id, not a timestamp — this file already orders recovery that way, and same-millisecond writes are common (the test clock exposed exactly that).

For an environment with 284 adopted historical jobs: 852 Relay CLI spawns at every start → 852 once, then zero.

vp test run apps/server/src/orchestration apps/server/src/persistence — 87 files, 831 passed. vp run -F t3 typecheck clean.

[rynfar]

The exact-head code CI failure in Test Server 2 is an unrelated Prime daemon test that uses a 5 ms timeout for both private-socket cleanup and the later post-kill receipt assertion. I isolated the test reliability fix in #855; its focused suite and server typecheck pass. Once #855 is green and merged, I will sync this Relay branch to that base and require a fresh green run here before merging.

[rynfar]

Synced onto pylon after #855 merged, preserving the reviewed progress-only worker and retirement-watermark fixes. The merged tree passes 10 focused Relay bridge/migration tests and server typecheck. New exact-head CI is running on a4c6f09; I will hold the merge until that run is green.