Skip to content

Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067

Closed

Description

Original author: Cedric Vidal (@cedricvidal)

Summary

Follow-up to #1064 / PR #1065. That fix resolved the original false-positive reaping (the StuckRunReaper no longer kills workers that are provably still doing work — confirmed: the old Reaped by scheduler error is gone and runs now survive 7–22 min instead of being killed at ~3 min). However, re-running the migrated scope-doc#51 runs in integration surfaced a distinct, genuine failure: workers stall during heavy cargo build steps, the liveness heartbeat goes stale (>120s), and the run is correctly failed via the new redelivery guard with:

Worker presumed dead (instance=… pod=coder-acp-copilot-…, last heartbeat 18Xs ago, threshold 120s); queue message redelivered while run was in 'processing' state.

Unlike #1064 this is not a false positive — the worker genuinely stops emitting events and heartbeats, and no work continues after the death declaration. Root cause is resource (CPU) starvation of the worker's Node.js event loop during the Rust workspace compile.

Evidence (6 of 9 retried runs, integration, build af9c80e)

Run Runtime Stale heartbeat at death Pod
b86a3373 20.8 min 184s …pr2lb
7e76f73d 18.8 min 184s …4glvg
822bde3b 15.8 min 183s …rbvj6
6f1ca384 11.0 min 182s …wh5jk
b4121a45 9.3 min 183s …wh5jk
f77498a7 7.5 min 181s …wh5jk

(The other 3 retried runs — e1ede0fd, dea61104, afaa975c — ran the full agent phase, 12–22 min, and reached the judge, failing only on an unrelated Copilot model rate limit. So runs can now complete the agent phase under the fix.)

Smoking-gun timeline (b86a3373, from run.jsonl)

Time (PDT) Event
6:11:01 "Check Rust compiler version" completes
6:11:01 → 6:25:19 ~14 min of light work (thinking, small edits) — heartbeat survives fine, no stall
6:27:46 Agent starts "Compile the Rust workspace" (full multi-crate cargo build)
6:28:49 Redelivery guard: "Duplicate queue message re-deferred — original worker still heart-beating (last beat 63s ago)" — heartbeat already lagging (cadence is 15s)
6:30:50 Run failed: "Worker presumed dead … last heartbeat 184s ago" — last successful beat = 6:27:46, the exact moment compile started. Zero liveness ticks during the compile.

The heartbeat dies the instant the full workspace compile begins, and survives all lighter phases (including a 14-min gap). Every one of the 6 reaches ~181–184s stale (just over the 120s threshold) before failing. Pod …wh5jk accounts for 3 of 6, suggesting per-node CPU contention.

Root cause

The agent's cargo build/rustc processes run inside the main coder-acp-copilot container, sharing its cgroup budget with the Node.js worker that emits the heartbeat. From deploy/base/workers/coder-acp-copilot.yaml:

resources:
  requests: { cpu: 250m, memory: 512Mi }
  limits:   { cpu: "1",  memory: 1Gi }

A multi-crate Rust workspace compile fans out to one rustc per core and pins CPU to the 1-core ceiling. Node's event loop is CPU-throttled, so the 15s liveness setInterval (in packages/shared/src/queue/visibility-heartbeat.ts) can't fire within the 120s window → "presumed dead". The gradual lag ("63s ago" → "184s ago") is consistent with CPU throttling rather than an instant death. (Rust compiles are also memory-heavy, so an OOM-kill against the 1Gi limit is a secondary candidate — same resource-exhaustion class.)

Note PR #1065 already decoupled the liveness loop from the queue updateMessage call, but both loops still live on the main event loop, which CPU starvation blocks regardless.

Suggested fixes (any combination)

  • Raise the main container's resource limits (CPU and memory) so a cargo/rustc compile can't starve the Node worker — e.g. bump CPU limit well above 1 and memory above 1Gi for the coder-acp-copilot (and likely other coder) containers.
  • Cap build parallelism to leave headroom for the event loop — e.g. set CARGO_BUILD_JOBS / --jobs below the CPU limit.
  • Move the liveness write off the main thread (dedicated worker thread / sidecar / separate process) so a CPU-bound child process can no longer delay the heartbeat timer. This is the most robust fix since it removes the event-loop dependency entirely.
  • Consider raising SCOPE_RUN_HEARTBEAT_STALE_MS / redelivery threshold above 120s for compile-heavy workloads as a stopgap, but this only widens the window — it doesn't fix the starvation.
  • Confirm CPU-throttle vs OOM via pod metrics / kubectl logs coder-acp-copilot-* (look for missing/late Liveness heartbeat tick #N lines vs an OOMKilled restart) for one reaped run (e.g. b86a3373, runId from /runs/b86a3373-e432-493d-a153-a09567c52cb3).

Impact

Compile-heavy benchmark runs (Rust workspaces in particular) still fail mid-flight in integration — now via "Worker presumed dead" instead of the old reaper message. The detection is correct, but the underlying resource limit makes these runs unable to complete, wasting full agent executions including multi-minute compiles.

Environment

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureCloud resources, hosting, networking, and shared infrastructure.area: workerCoding agent worker runtime, task execution, and lifecycle.type: bugIncorrect behavior or a regression that needs fixing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions