Repository navigation
fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures - #1247
Closed
Cedric Vidal (cedricvidal) wants to merge 2 commits into
Closed
Cedric Vidal (cedricvidal) wants to merge 2 commits into
Cedric Vidal (cedricvidal) wants to merge 2 commits into
Conversation
Add prod-accurate `cpus` + `mem_limit` (and `mem_reservation` mirroring K8s requests for the primary app/worker services) to every compose service that has a prod Deployment/Job analog, so local dev reflects prod resource constraints. Values are sourced from deploy/base/*.yaml and deploy/base/workers/*.yaml (the int/prod overlays only patch image tags, so base = both environments). The coder-acp-copilot worker is set to mem_limit 2.3g / mem_reservation 1g to match the raised manifest limit in this PR, so local repros surface the same OOM ceiling that caused the "Worker presumed dead" failures. Uses the top-level v2 keys `mem_limit`/`cpus` (enforced by `docker compose up`), not `deploy.resources.limits` (ignored outside swarm). Infra with no prod analog (mongodb, redis, azurite, lowkey-vault) and short-lived init/registration containers are intentionally left unconstrained. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The base docker-compose.yml mirrors prod K8s runtime limits, but the dev build target runs tsx + in-container tsc --watch (api/scheduler/token-manager) and the Vite dev server + esbuild dep-optimization (portal), which need more memory than the compiled prod runtime. Empirically boot-tested: Node dev watchers converge to ~510-560 MiB steady, and portal's esbuild optimization burst OOM-kills at 512m. Add 1g dev overrides for api, scheduler, token-manager, and portal. judge/report-generator/ post-processor keep their 1g base limit (fits). The copilot worker keeps its 2.3g base limit un-overridden so local repros still surface the OOM ceiling that causes the 'Worker presumed dead' failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original author: Cedric Vidal (@cedricvidal)
Problem
All GitHub Copilot CLI worker runs (11/11 recent) have been failing in integration and prod with:
Root cause
Reproduced prod run
d1916e3f-47d9-453b-a94b-14d565f96506locally in docker-compose with the worker pinned to prod's exact limits (cpu: 1/mem: 1Gi):Killedin container logs; memory cliff 1014 MiB → 340 MiB), the Copilot CLI subprocess dies, and the worker fails withACP agent process exited unexpectedly (code 1)at ~41s. In prod the OOM/GC stall lands on the worker's Node event loop instead, freezing itssetIntervalheartbeats → the run is marked "Worker presumed dead".Conclusion: the worker's real per-run memory demand is ~1.3–2.1 GiB, but the main container is capped at
1Gi. This is a memory problem, not CPU.Change
Single file:
deploy/base/workers/coder-acp-copilot.yaml. On the maincoder-acp-copilotcontainer only (sidecarsmcp-gateway/devproxyuntouched, CPU untouched):requests.memory512Mi1Gilimits.memory1Gi2.3Gi2.3Gigives ~10% headroom over the observed 2.09 GiB peak. The int/prod overlays only patch image tags (not resources), so this base change propagates to both environments.Verification
git diffconfirms only the main container's memoryrequests/limitschanged (CPU and sidecars untouched).kubectl kustomize deploy/overlays/integrationanddeploy/overlays/prodboth render thecoder-acp-copilotDeployment's main container withrequests.memory: 1Giandlimits.memory: 2.3Gi.docs/architecture/deployment.mddoes not document worker resource limits, so no doc change was needed.Note / possible follow-up
The separate
coder-acp-copilot-windowsworker (deploy/base/workers/coder-acp-copilot-windows.yaml) still has512Mi/1Gi. It was not part of this repro and is intentionally left unchanged here; if it exhibits the same OOM symptom it can be bumped in a follow-up.Relationship to #1067
#1067 tracks a narrower variant — heavy Rust
cargo buildstarving the event loop via the 1-core CPU limit (worker survives, throttled). #1249 (this PR's fix) is the general memory/OOM failure at the1Giceiling that affects every run regardless of workload. The local repro here resolves #1067's open "confirm CPU-throttle vs OOM" question in favor of OOM/memory, and this memory bump also partially satisfies #1067's memory-as-secondary-candidate remedy. CPU is deliberately left at1since it is not the bottleneck for the systemic failure.Local dev: mirror prod K8s limits in Docker Compose
So local repros surface the same OOM behavior we just root-caused (and so dev resource usage reflects prod), the Compose stack now carries prod-accurate resource limits.
Base
docker-compose.yml— added top-level v2cpus:+mem_limit:(andmem_reservation:mirroring prod requests where prod brackets one) to every service with a prod Deployment analog, sourced 1:1 fromdeploy/base/*.yaml/deploy/base/workers/*.yaml(overlays only patch image tags, so base = int and prod). Docker'sm/gsuffixes are binary, so they map 1:1 to K8sMi/Gi. We use the top-level keys (notdeploy.resources.limits, whichdocker compose upignores outside swarm — verified viadocker inspect).The copilot worker sits at
2.3g/1g, matching this PR's manifest change — so a local run reproduces the exact OOM ceiling instead of hiding it behind an unbounded container.Excluded (no prod Deployment analog — managed services in prod — or short-lived init/registration jobs):
mongodb,redis,azurite,lowkey-vault,azurite-init,devproxy-*-init,register-agents,register-version-post-processor.Dev-target overrides (
docker-compose.dev.yml)The base limits mirror the compiled prod runtime, but the dev build target runs memory-hungrier processes:
tsx+ in-containertsc --watch(api/scheduler/token-manager) and the Vite dev server + esbuild dependency-optimization (portal). I boot-tested the dev stack and measured actual usage, then addedmem_limitoverrides only where the dev watcher exceeds the prod cap:OOMKilled=true); Vite dep-optimization spikesThe base (prod-accurate) values were not loosened to accommodate dev — the override layer keeps
docker-compose.ymla faithful mirror of prod.Compose validation
docker compose -f docker-compose.yml configand... -f docker-compose.yml -f docker-compose.dev.yml configboth parse cleanly; rendered byte values match prod exactly (e.g. copilotmem_limit: 2469606195= 2.3 GiB).OOMKilled=false,Restarts=0— except portal at the initial 512m override, which wasOOMKilled=trueduring Vite's esbuild optimization burst, so its override was raised to 1g and re-verified stable (563 MiB, survives the burst).docker:dev:copilotstack boot-tested end-to-end: all applied cgroup limits match prod exactly (copilot 2.30 GiB / 1 cpu, dev overrides at 1 GiB, sidecars 512 MiB); copilot worker booted healthy (MongoDB connected, queue ensured) idle at ~542 MiB with full 2.3 GiB headroom; no OOM/restarts across the run.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com