Skip to content

Copilot worker OOM-killed at 1Gi memory limit → systemic "Worker presumed dead" failures (all runs) #1249

Closed

Description

Original author: Cedric Vidal (@cedricvidal)

Summary

All GitHub Copilot CLI worker runs (11/11 recent) fail in integration and prod with:

Worker presumed dead (instance=… pod=coder-acp-copilot-…, last heartbeat >120s ago, threshold 120s); queue message redelivered while run was in 'processing' state.

Root-caused as a memory/OOM problem: the main coder-acp-copilot container is capped at limits.memory: 1Gi, but real per-run demand is ~1.3–2.1 GiB. The kernel OOM-killer fires, the Copilot CLI subprocess is Killed, and the worker's Node.js event loop stalls under the OOM/GC pressure, freezing its 15s liveness setInterval → the run is reaped as "presumed dead".

This is the memory sibling of #1067. #1067 is narrowly scoped to Rust cargo build CPU starvation (worker survives, throttled); this issue is the general, workload-independent OOM at the 1Gi memory ceiling that affects every run regardless of task.

Evidence — local repro of prod run d1916e3f-47d9-453b-a94b-14d565f96506

Reproduced the prod run in docker-compose with the worker pinned to prod's exact limits (cpu: 1 / mem: 1Gi):

Scenario Peak memory Outcome
1 GiB (prod parity) 1014 MiB / 99% → ceiling Kernel OOM-killer fires (literal Killed in logs; OOMKilled=true; memory cliff 1014 MiB → 340 MiB). Copilot CLI subprocess dies; worker fails ACP agent process exited unexpectedly (code 1) at ~41s.
4 GiB (control) 2.09 GiB Zero OOM, run completes healthily.

Locally the OOM surfaces as an immediate subprocess kill; in prod the OOM/GC stall instead lands on the worker's Node event loop, freezing the heartbeat timer → "Worker presumed dead".

Root cause

deploy/base/workers/coder-acp-copilot.yaml, main coder-acp-copilot container:

resources:
  requests: { cpu: 250m, memory: 512Mi }
  limits:   { cpu: "1",  memory: 1Gi }   # ← too low; real demand ~1.3–2.1 GiB

This is a memory problem, not CPU — CPU stays at the 1 limit and is not the bottleneck for typical (non-compile) runs.

Fix

PR #1247 raises the main container to requests.memory: 1Gi / limits.memory: 2.3Gi (~10% headroom over the observed 2.09 GiB peak). The int/prod overlays only patch image tags, so this base change propagates to both environments. The PR also mirrors prod K8s CPU/memory limits into docker-compose*.yml so local repros surface the same OOM instead of hiding it behind an unbounded container.

Distinction from #1067

#1067 this issue
trigger heavy cargo build any run (11/11), no special workload
cause 1-core CPU starvation throttles heartbeat timer memory exhaustion → OOM-kill / event-loop stall
container fate survives (throttled), no restart Copilot CLI subprocess Killed, worker exits
remedy raise CPU / cap -j / off-thread heartbeat raise memory 1Gi → 2.3Gi

Both share the "Worker presumed dead" symptom and the "raise the container's resource limits" family of remedies; #1247's memory bump also partially satisfies #1067's memory-as-secondary-candidate note.

Impact

Every Copilot CLI benchmark run fails mid-flight in integration and prod (11/11 recent), wasting full agent executions. The Copilot worker is effectively unusable until the memory limit is raised. High severity.

Environment

  • Worker container coder-acp-copilot, int/prod limits cpu: 1 / mem: 1Gi
  • Heartbeat: 15s liveness setInterval; reaper/redelivery threshold 120s
  • Repro: docker-compose, worker pinned to 1Gi (OOMs at ~41s) vs 4Gi (healthy, 2.09 GiB peak)

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureCloud resources, hosting, networking, and shared infrastructure.area: workerCoding agent worker runtime, task execution, and lifecycle.type: bugIncorrect behavior or a regression that needs fixing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions