Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Summary
Follow-up to #1064 / PR #1065. That fix resolved the original false-positive reaping (the StuckRunReaper no longer kills workers that are provably still doing work — confirmed: the old Reaped by scheduler error is gone and runs now survive 7–22 min instead of being killed at ~3 min). However, re-running the migrated scope-doc#51 runs in integration surfaced a distinct, genuine failure: workers stall during heavy cargo build steps, the liveness heartbeat goes stale (>120s), and the run is correctly failed via the new redelivery guard with:
Worker presumed dead (instance=… pod=coder-acp-copilot-…, last heartbeat 18Xs ago, threshold 120s); queue message redelivered while run was in 'processing' state.
Unlike #1064 this is not a false positive — the worker genuinely stops emitting events and heartbeats, and no work continues after the death declaration. Root cause is resource (CPU) starvation of the worker's Node.js event loop during the Rust workspace compile.
Evidence (6 of 9 retried runs, integration, build af9c80e)
| Run | Runtime | Stale heartbeat at death | Pod |
|---|---|---|---|
| b86a3373 | 20.8 min | 184s | …pr2lb |
| 7e76f73d | 18.8 min | 184s | …4glvg |
| 822bde3b | 15.8 min | 183s | …rbvj6 |
| 6f1ca384 | 11.0 min | 182s | …wh5jk |
| b4121a45 | 9.3 min | 183s | …wh5jk |
| f77498a7 | 7.5 min | 181s | …wh5jk |
(The other 3 retried runs — e1ede0fd, dea61104, afaa975c — ran the full agent phase, 12–22 min, and reached the judge, failing only on an unrelated Copilot model rate limit. So runs can now complete the agent phase under the fix.)
Smoking-gun timeline (b86a3373, from run.jsonl)
| Time (PDT) | Event |
|---|---|
| 6:11:01 | "Check Rust compiler version" completes |
| 6:11:01 → 6:25:19 | ~14 min of light work (thinking, small edits) — heartbeat survives fine, no stall |
| 6:27:46 | Agent starts "Compile the Rust workspace" (full multi-crate cargo build) |
| 6:28:49 | Redelivery guard: "Duplicate queue message re-deferred — original worker still heart-beating (last beat 63s ago)" — heartbeat already lagging (cadence is 15s) |
| 6:30:50 | Run failed: "Worker presumed dead … last heartbeat 184s ago" — last successful beat = 6:27:46, the exact moment compile started. Zero liveness ticks during the compile. |
The heartbeat dies the instant the full workspace compile begins, and survives all lighter phases (including a 14-min gap). Every one of the 6 reaches ~181–184s stale (just over the 120s threshold) before failing. Pod …wh5jk accounts for 3 of 6, suggesting per-node CPU contention.
Root cause
The agent's cargo build/rustc processes run inside the main coder-acp-copilot container, sharing its cgroup budget with the Node.js worker that emits the heartbeat. From deploy/base/workers/coder-acp-copilot.yaml:
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "1", memory: 1Gi }A multi-crate Rust workspace compile fans out to one rustc per core and pins CPU to the 1-core ceiling. Node's event loop is CPU-throttled, so the 15s liveness setInterval (in packages/shared/src/queue/visibility-heartbeat.ts) can't fire within the 120s window → "presumed dead". The gradual lag ("63s ago" → "184s ago") is consistent with CPU throttling rather than an instant death. (Rust compiles are also memory-heavy, so an OOM-kill against the 1Gi limit is a secondary candidate — same resource-exhaustion class.)
Note PR #1065 already decoupled the liveness loop from the queue updateMessage call, but both loops still live on the main event loop, which CPU starvation blocks regardless.
Suggested fixes (any combination)
- Raise the main container's resource limits (CPU and memory) so a
cargo/rustccompile can't starve the Node worker — e.g. bump CPU limit well above1and memory above1Gifor thecoder-acp-copilot(and likely other coder) containers. - Cap build parallelism to leave headroom for the event loop — e.g. set
CARGO_BUILD_JOBS/--jobsbelow the CPU limit. - Move the liveness write off the main thread (dedicated worker thread / sidecar / separate process) so a CPU-bound child process can no longer delay the heartbeat timer. This is the most robust fix since it removes the event-loop dependency entirely.
- Consider raising
SCOPE_RUN_HEARTBEAT_STALE_MS/ redelivery threshold above 120s for compile-heavy workloads as a stopgap, but this only widens the window — it doesn't fix the starvation. - Confirm CPU-throttle vs OOM via pod metrics /
kubectl logs coder-acp-copilot-*(look for missing/lateLiveness heartbeat tick #Nlines vs an OOMKilled restart) for one reaped run (e.g. b86a3373, runId from/runs/b86a3373-e432-493d-a153-a09567c52cb3).
Impact
Compile-heavy benchmark runs (Rust workspaces in particular) still fail mid-flight in integration — now via "Worker presumed dead" instead of the old reaper message. The detection is correct, but the underlying resource limit makes these runs unable to complete, wasting full agent executions including multi-minute compiles.
Environment
- Integration:
https://msscope-int.azurewebsites.net - Worker container
coder-acp-copilot, buildaf9c80e(post-fix(scheduler): stop StuckRunReaper from reaping healthy runs (#1064) #1065), limits CPU1/ mem1Gi - Heartbeat: 15s liveness
setInterval; reaper/redelivery threshold 120s - Affected runs originated from the scope-doc#51 migration retry.
Related
- Follow-up to StuckRunReaper false-positives: healthy long-running runs reaped on stale heartbeat (integration) #1064 (false-positive reaping) — fixed by fix(scheduler): stop StuckRunReaper from reaping healthy runs (#1064) #1065. This issue is the remaining, genuine stall the fix exposed.
Activity