Skip to content

fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures - #1247

Closed
Cedric Vidal (cedricvidal) wants to merge 2 commits into
mainfrom
cedricvidal-microsoft-raise-copilot-worker-memory
Closed

Cedric Vidal (cedricvidal) wants to merge 2 commits into
mainfrom
cedricvidal-microsoft-raise-copilot-worker-memory

Conversation

@cedricvidal

Copy link
Copy Markdown
Collaborator

Original author: Cedric Vidal (@cedricvidal)

Fixes #1249 — the systemic Copilot-worker OOM "presumed dead" failure. Related: #1067 (the CPU-starvation sibling, scoped to Rust compiles).

Problem

All GitHub Copilot CLI worker runs (11/11 recent) have been failing in integration and prod with:

Worker presumed dead ... last heartbeat >120s ago

Root cause

Reproduced prod run d1916e3f-47d9-453b-a94b-14d565f96506 locally in docker-compose with the worker pinned to prod's exact limits (cpu: 1 / mem: 1Gi):

  • At 1 GiB (repro): container memory climbs to the ceiling (1014 MiB / 99%), the kernel OOM-killer fires (literal Killed in container logs; memory cliff 1014 MiB → 340 MiB), the Copilot CLI subprocess dies, and the worker fails with ACP agent process exited unexpectedly (code 1) at ~41s. In prod the OOM/GC stall lands on the worker's Node event loop instead, freezing its setInterval heartbeats → the run is marked "Worker presumed dead".
  • Control at 4 GiB: the identical task peaks at 2.09 GiB with zero OOM and runs healthily.

Conclusion: the worker's real per-run memory demand is ~1.3–2.1 GiB, but the main container is capped at 1Gi. This is a memory problem, not CPU.

Change

Single file: deploy/base/workers/coder-acp-copilot.yaml. On the main coder-acp-copilot container only (sidecars mcp-gateway/devproxy untouched, CPU untouched):

before after
requests.memory 512Mi 1Gi
limits.memory 1Gi 2.3Gi

2.3Gi gives ~10% headroom over the observed 2.09 GiB peak. The int/prod overlays only patch image tags (not resources), so this base change propagates to both environments.

Verification

  • git diff confirms only the main container's memory requests/limits changed (CPU and sidecars untouched).
  • kubectl kustomize deploy/overlays/integration and deploy/overlays/prod both render the coder-acp-copilot Deployment's main container with requests.memory: 1Gi and limits.memory: 2.3Gi.
  • docs/architecture/deployment.md does not document worker resource limits, so no doc change was needed.

Note / possible follow-up

The separate coder-acp-copilot-windows worker (deploy/base/workers/coder-acp-copilot-windows.yaml) still has 512Mi/1Gi. It was not part of this repro and is intentionally left unchanged here; if it exhibits the same OOM symptom it can be bumped in a follow-up.

Relationship to #1067

#1067 tracks a narrower variant — heavy Rust cargo build starving the event loop via the 1-core CPU limit (worker survives, throttled). #1249 (this PR's fix) is the general memory/OOM failure at the 1Gi ceiling that affects every run regardless of workload. The local repro here resolves #1067's open "confirm CPU-throttle vs OOM" question in favor of OOM/memory, and this memory bump also partially satisfies #1067's memory-as-secondary-candidate remedy. CPU is deliberately left at 1 since it is not the bottleneck for the systemic failure.


Local dev: mirror prod K8s limits in Docker Compose

So local repros surface the same OOM behavior we just root-caused (and so dev resource usage reflects prod), the Compose stack now carries prod-accurate resource limits.

Base docker-compose.yml — added top-level v2 cpus: + mem_limit: (and mem_reservation: mirroring prod requests where prod brackets one) to every service with a prod Deployment analog, sourced 1:1 from deploy/base/*.yaml / deploy/base/workers/*.yaml (overlays only patch image tags, so base = int and prod). Docker's m/g suffixes are binary, so they map 1:1 to K8s Mi/Gi. We use the top-level keys (not deploy.resources.limits, which docker compose up ignores outside swarm — verified via docker inspect).

compose service cpus mem_limit (mem_reservation)
api 0.5 512m 256m
gateway 0.5 512m 128m
judge 1 1g 512m
portal 0.2 128m 64m
scheduler 0.2 256m 128m
token-manager 0.2 256m 128m
model-scanner-copilot 0.25 256m —
model-scanner-anthropic 0.25 256m —
post-processor 1 1g —
report-generator 1 1g —
db-migrate 0.5 256m —
coder-acp-copilot 1 2.3g 1g
coder-acp-claude-code 1 1g 512m
coder-vscode-web 2 4g 1g
coder-vscode-electron-driver-ext 2 4g 1g
devproxy-copilot / mcp-gateway-copilot 0.5 512m —
devproxy-claude-code / mcp-gateway-claude-code 0.5 512m —
mcp-gateway-vscode-electron 0.5 512m —

The copilot worker sits at 2.3g / 1g, matching this PR's manifest change — so a local run reproduces the exact OOM ceiling instead of hiding it behind an unbounded container.

Excluded (no prod Deployment analog — managed services in prod — or short-lived init/registration jobs): mongodb, redis, azurite, lowkey-vault, azurite-init, devproxy-*-init, register-agents, register-version-post-processor.

Dev-target overrides (docker-compose.dev.yml)

The base limits mirror the compiled prod runtime, but the dev build target runs memory-hungrier processes: tsx + in-container tsc --watch (api/scheduler/token-manager) and the Vite dev server + esbuild dependency-optimization (portal). I boot-tested the dev stack and measured actual usage, then added mem_limit overrides only where the dev watcher exceeds the prod cap:

service prod base steady dev usage override reason
api 512m ~530 MiB 1g tsc --watch exceeds 512m
scheduler 256m ~515 MiB 1g watcher baseline ≫ 256m
token-manager 256m ~519 MiB 1g watcher baseline ≫ 256m
portal 128m ~563 MiB + esbuild burst 1g OOM-killed at 512m (confirmed OOMKilled=true); Vite dep-optimization spikes
judge 1g ~512 MiB — fits, no override
coder-acp-copilot 2.3g ~531 MiB idle — left un-overridden so run-time OOM ceiling is preserved
report-generator / post-processor 1g ~watcher baseline — fit within 1g

The base (prod-accurate) values were not loosened to accommodate dev — the override layer keeps docker-compose.yml a faithful mirror of prod.

Compose validation

  • docker compose -f docker-compose.yml config and ... -f docker-compose.yml -f docker-compose.dev.yml config both parse cleanly; rendered byte values match prod exactly (e.g. copilot mem_limit: 2469606195 = 2.3 GiB).
  • Boot-tested the dev stack (api, judge, scheduler, token-manager, portal + infra) for ~3 min: all stable, OOMKilled=false, Restarts=0 — except portal at the initial 512m override, which was OOMKilled=true during Vite's esbuild optimization burst, so its override was raised to 1g and re-verified stable (563 MiB, survives the burst).
  • Full docker:dev:copilot stack boot-tested end-to-end: all applied cgroup limits match prod exactly (copilot 2.30 GiB / 1 cpu, dev overrides at 1 GiB, sidecars 512 MiB); copilot worker booted healthy (MongoDB connected, queue ensured) idle at ~542 MiB with full 2.3 GiB headroom; no OOM/restarts across the run.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Add prod-accurate `cpus` + `mem_limit` (and `mem_reservation` mirroring K8s
requests for the primary app/worker services) to every compose service that has
a prod Deployment/Job analog, so local dev reflects prod resource constraints.

Values are sourced from deploy/base/*.yaml and deploy/base/workers/*.yaml (the
int/prod overlays only patch image tags, so base = both environments). The
coder-acp-copilot worker is set to mem_limit 2.3g / mem_reservation 1g to match
the raised manifest limit in this PR, so local repros surface the same OOM
ceiling that caused the "Worker presumed dead" failures.

Uses the top-level v2 keys `mem_limit`/`cpus` (enforced by `docker compose up`),
not `deploy.resources.limits` (ignored outside swarm). Infra with no prod analog
(mongodb, redis, azurite, lowkey-vault) and short-lived init/registration
containers are intentionally left unconstrained.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The base docker-compose.yml mirrors prod K8s runtime limits, but the dev
build target runs tsx + in-container tsc --watch (api/scheduler/token-manager)
and the Vite dev server + esbuild dep-optimization (portal), which need more
memory than the compiled prod runtime.

Empirically boot-tested: Node dev watchers converge to ~510-560 MiB steady,
and portal's esbuild optimization burst OOM-kills at 512m. Add 1g dev overrides
for api, scheduler, token-manager, and portal. judge/report-generator/
post-processor keep their 1g base limit (fits). The copilot worker keeps its
2.3g base limit un-overridden so local repros still surface the OOM ceiling
that causes the 'Worker presumed dead' failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead")

1 participant