Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Summary
All GitHub Copilot CLI worker runs (11/11 recent) fail in integration and prod with:
Worker presumed dead (instance=… pod=coder-acp-copilot-…, last heartbeat >120s ago, threshold 120s); queue message redelivered while run was in 'processing' state.
Root-caused as a memory/OOM problem: the main coder-acp-copilot container is capped at limits.memory: 1Gi, but real per-run demand is ~1.3–2.1 GiB. The kernel OOM-killer fires, the Copilot CLI subprocess is Killed, and the worker's Node.js event loop stalls under the OOM/GC pressure, freezing its 15s liveness setInterval → the run is reaped as "presumed dead".
This is the memory sibling of #1067. #1067 is narrowly scoped to Rust cargo build CPU starvation (worker survives, throttled); this issue is the general, workload-independent OOM at the 1Gi memory ceiling that affects every run regardless of task.
Evidence — local repro of prod run d1916e3f-47d9-453b-a94b-14d565f96506
Reproduced the prod run in docker-compose with the worker pinned to prod's exact limits (cpu: 1 / mem: 1Gi):
| Scenario | Peak memory | Outcome |
|---|---|---|
| 1 GiB (prod parity) | 1014 MiB / 99% → ceiling | Kernel OOM-killer fires (literal Killed in logs; OOMKilled=true; memory cliff 1014 MiB → 340 MiB). Copilot CLI subprocess dies; worker fails ACP agent process exited unexpectedly (code 1) at ~41s. |
| 4 GiB (control) | 2.09 GiB | Zero OOM, run completes healthily. |
Locally the OOM surfaces as an immediate subprocess kill; in prod the OOM/GC stall instead lands on the worker's Node event loop, freezing the heartbeat timer → "Worker presumed dead".
Root cause
deploy/base/workers/coder-acp-copilot.yaml, main coder-acp-copilot container:
resources:
requests: { cpu: 250m, memory: 512Mi }
limits: { cpu: "1", memory: 1Gi } # ← too low; real demand ~1.3–2.1 GiBThis is a memory problem, not CPU — CPU stays at the 1 limit and is not the bottleneck for typical (non-compile) runs.
Fix
PR #1247 raises the main container to requests.memory: 1Gi / limits.memory: 2.3Gi (~10% headroom over the observed 2.09 GiB peak). The int/prod overlays only patch image tags, so this base change propagates to both environments. The PR also mirrors prod K8s CPU/memory limits into docker-compose*.yml so local repros surface the same OOM instead of hiding it behind an unbounded container.
Distinction from #1067
| #1067 | this issue | |
|---|---|---|
| trigger | heavy cargo build |
any run (11/11), no special workload |
| cause | 1-core CPU starvation throttles heartbeat timer | memory exhaustion → OOM-kill / event-loop stall |
| container fate | survives (throttled), no restart | Copilot CLI subprocess Killed, worker exits |
| remedy | raise CPU / cap -j / off-thread heartbeat |
raise memory 1Gi → 2.3Gi |
Both share the "Worker presumed dead" symptom and the "raise the container's resource limits" family of remedies; #1247's memory bump also partially satisfies #1067's memory-as-secondary-candidate note.
Impact
Every Copilot CLI benchmark run fails mid-flight in integration and prod (11/11 recent), wasting full agent executions. The Copilot worker is effectively unusable until the memory limit is raised. High severity.
Environment
- Worker container
coder-acp-copilot, int/prod limitscpu: 1/mem: 1Gi - Heartbeat: 15s liveness
setInterval; reaper/redelivery threshold 120s - Repro: docker-compose, worker pinned to
1Gi(OOMs at ~41s) vs4Gi(healthy, 2.09 GiB peak)
Related
- Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067 — CPU-starvation sibling (scoped to Rust compiles)
- Fixed by fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247
Activity