Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Summary
The Worker presumed dead (… queue message redelivered while run was in 'processing' state) failure mode is still occurring in integration after the 2026-07-06 fixes (#1249, #1067, PR #1247). Both of those trackers are closed, and there is currently no open issue tracking the recurrence.
A sweep of the integration API (msscope-int, build 3f2a4fc, project Default Project d95231c0-884b-48fd-a950-b0e71d449989) found 16 failed runs (outcome: failed) whose run.error matches "Worker presumed dead". 12 pre-date the 07-06 fixes and are explained by #1249/#1067. 4 post-date them and are not covered by any fix or open issue.
Post-fix recurrences (not covered by #1249/#1067)
| Run ID | Worker | Model | Agent ver | Created (UTC) | Pod / instance |
|---|---|---|---|---|---|
75638258-791b-4c42-b06f-c37f10d4b599 |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 | …-qtsmt / 461d13be… |
60aa6bc9-940c-4f28-8584-065c37c1e480 |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 | …-qtsmt / 12cfe0d1… |
27cb2832-1a95-4da8-9450-89b7468e88af |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 | …-9hxmf / ad2a717a… |
3b56ccf8-c115-4e5e-ad22-25b703ea7c2f |
coder-acp-copilot | claude-opus-4.8 | copilot-1.0.65 | 2026-07-14 21:40 | …-5ztql / c6bec9da… |
All four show a consistent gap: last heartbeat 182s ago, threshold 120s. Example:
Worker presumed dead (instance=461d13be-25ae-4fed-b04e-14d1ab6abf1f pod=coder-acp-claude-code-69cbd5bbcd-qtsmt, last heartbeat 182s ago, threshold 120s); queue message redelivered while run was in 'processing' state
The three 07-10 claude-code runs are one batch (submission de0c0ff7-d0ed-4049-8ea0-aeb7d63b1327, scenario "Rayfin: New Project Setup").
Root-cause hypothesis
The coder-acp-claude-code worker never received the resource bump that fixed the copilot worker in #1247. Its main container still has the exact 1Gi memory + 1-core CPU limits that #1249 (OOM) and #1067 (CPU starvation) identified as the root cause of stalled heartbeats:
| main-container CPU | main-container memory | |
|---|---|---|
coder-acp-copilot (post #1247) |
req 250m / lim 1 |
req 1Gi / lim 2.3Gi |
coder-acp-claude-code (current) |
req 250m / lim 1 |
req 512Mi / lim 1Gi |
Source: deploy/base/workers/coder-acp-claude-code.yaml (lines ~40–46) vs deploy/base/workers/coder-acp-copilot.yaml (lines ~32–38). The claude-code worker is the "memory/CPU sibling" that #1249/#1067 called out but only patched for copilot.
The single 07-14 copilot claude-opus-4.8 recurrence (3b56ccf8…) is a separate, lower-confidence data point — copilot already has 2.3Gi, so this is either a heavier-model/heavier-task run exceeding even the raised ceiling, or a genuine transient (pod eviction / queue hiccup). Worth confirming from pod metrics/events before assuming the limit is still too low.
Proposed next steps
- Raise
coder-acp-claude-codemain-container limits to parity with copilot (memory: 2.3Gi,requests.memory: 1Gi), matching the fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247 fix — most likely resolves the 07-10 cluster. - Investigate the 07-14 copilot
opus-4.8run (3b56ccf8…) via pod events/metrics to determine OOM vs eviction vs transient. - Consider the open hardening items that reduce blast radius but don't fix root cause: Worker reliability under Azure Storage Queue intermittent failures #806 (queue reliability), Guard all run mutations behind heartbeat ownership (follow-up to PR #793/#794) #803/Self-terminate worker process on heartbeat ownership loss (alternative to #803) #804 (heartbeat ownership), Heartbeat silently hangs on frozen queue endpoint (no client-side timeout on updateMessage) #805 (heartbeat hang on frozen queue), Add observability for safeDeferMessage failures in queue redelivery path #1056 (redelivery observability).
- Retry the 4 affected runs after limits are raised.
Full match list (16 runs, integration / Default Project)
12 of these pre-date the 07-06 fixes (explained by #1249/#1067); listed for completeness.
| Run ID | Worker | Model | Agent ver | Created (UTC) |
|---|---|---|---|---|
3b56ccf8-c115-4e5e-ad22-25b703ea7c2f |
coder-acp-copilot | claude-opus-4.8 | copilot-1.0.65 | 2026-07-14 21:40 |
75638258-791b-4c42-b06f-c37f10d4b599 |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 |
60aa6bc9-940c-4f28-8584-065c37c1e480 |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 |
27cb2832-1a95-4da8-9450-89b7468e88af |
coder-acp-claude-code | claude-sonnet-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-07-10 13:03 |
d1916e3f-47d9-453b-a94b-14d565f96506 |
coder-acp-copilot | gpt-5.5 | copilot-1.0.55 | 2026-07-02 09:27 |
dfa1969d-ba33-4bcf-b3a5-8522b9ecd78b |
coder-acp-copilot | claude-haiku-4.5 | copilot-1.0.65 | 2026-06-30 16:26 |
1065b6bd-703b-40ff-82ce-7b7fdf683644 |
coder-acp-copilot | claude-haiku-4.5 | copilot-1.0.65 | 2026-06-30 09:24 |
7a3e09f4-7087-4da5-a8a6-9eb0686cdbea |
coder-acp-claude-code | claude-opus-4-6 | acp-0.29.0-sdk-0.2.111 | 2026-06-16 15:21 |
04edddcb-4622-4b7d-8993-973daf2439a2 |
coder-vscode-electron-driver-ext | claude-haiku-4.5 | vscode-electron-1.122.0 | 2026-06-15 16:28 |
a72aa868-883b-4682-9435-4c4b3c11c954 |
coder-vscode-electron-driver-ext | claude-haiku-4.5 | vscode-electron-1.122.0 | 2026-06-11 21:43 |
f77498a7-f9d8-4ad4-8fa7-3ff452f20351 |
coder-acp-copilot | gpt-5.3-codex | copilot-1.0.43 | 2026-05-26 19:05 |
b4121a45-ea03-4468-9ae0-cbfd284d15a6 |
coder-acp-copilot | gpt-5.4 | copilot-1.0.43 | 2026-05-26 19:05 |
6f1ca384-16bb-474e-bb34-8e81be71b159 |
coder-acp-copilot | gpt-5.4 | copilot-1.0.43 | 2026-05-26 19:05 |
b86a3373-e432-493d-a153-a09567c52cb3 |
coder-acp-copilot | gpt-5.4 | copilot-1.0.43 | 2026-05-26 17:19 |
822bde3b-a0e8-411b-b61e-89c03fc8a386 |
coder-acp-copilot | claude-opus-4.7 | copilot-1.0.43 | 2026-05-26 17:19 |
7e76f73d-25be-4364-a4f0-83925a34f1f6 |
coder-acp-copilot | claude-opus-4.7 | copilot-1.0.43 | 2026-05-26 17:19 |
Environment
- Environment: integration (
msscope-int, API build3f2a4fc837c8a6491cf0aa5fd6dd77380947dd14) - Project: Default Project (
d95231c0-884b-48fd-a950-b0e71d449989) — 63 totalfailedruns, 16 matching this signature. The other integration project ("Cedric's tests") has 0.
References
- Copilot worker OOM-killed at 1Gi memory limit → systemic "Worker presumed dead" failures (all runs) #1249 — Copilot worker OOM at 1Gi → "Worker presumed dead" (CLOSED, fixed by PR fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247)
- Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067 — Coder workers stall on heavy cargo builds, heartbeat starved by 1-core CPU (CLOSED)
- PR fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247 — raise copilot memory 1Gi→2.3Gi
- StuckRunReaper false-positives: healthy long-running runs reaped on stale heartbeat (integration) #1064 / PR fix(scheduler): stop StuckRunReaper from reaping healthy runs (#1064) #1065 — StuckRunReaper false-positives
- PR fix(shared): fail-fast when queue message is redelivered mid-run #793 / PR fix: stop runs getting permanently stuck in processing (redelivery token + reaper) #1052 — redelivery fail-fast + reaper (the path that emits this error)
Activity
cedricvidal commentedon Jul 29, 2026
Original author: Cedric Vidal (@cedricvidal)
Mitigation options — ideas to prevent "Worker presumed dead"
Root observation: the liveness heartbeat is an in-process
setIntervalrunning in the same Node event loop as the agent workload. When the workload melts down (OOM, CPU throttling, event-loop blocking), the mechanism meant to detect liveness dies with it. That makes failures both (a) hard to prevent and (b) hard to attribute. Ideas grouped by strategy, roughly in priority order.1. Stop the death — right-size resources (quickest, highest leverage)
coder-acp-claude-codeis still1Gi/1-core — the exact config Copilot worker OOM-killed at 1Gi memory limit → systemic "Worker presumed dead" failures (all runs) #1249/Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067 flagged. Size memory to observed p99 per-run RSS (~1.3–2.1 GiB), not a round number.cargo buildspike throttles the container to its quota and delays the heartbeat timer past 120s. This is the direct mechanism behind Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067. Requests still guarantee scheduling; dropping the limit eliminates throttle-induced false reaps.--max-old-space-size) so an overspend fails one run cleanly instead of OOM-killing the whole pod and stalling Node.2. Stop false reaping — decouple the heartbeat from the workload (structural)
worker_threador a lightweight sidecar container with its own CPU/mem budget, reporting on the main process's real liveness (PID alive + progress). CPU saturation in the agent then can't starve it.Running? process alive?) instead of failing at the first stale threshold.updateMessage(open issue Heartbeat silently hangs on frozen queue endpoint (no client-side timeout on updateMessage) #805) so a frozen queue endpoint can't silently hang the beat.3. Survive the death — make it non-fatal to the run (resilience)
4. See it coming & attribute it — observability
memory.current/memory.maxinside the worker; when nearing the limit, proactively fail with a clear "insufficient memory" error + requeue, instead of an ambiguous OOM → presumed-dead.lastState.terminated.reason == OOMKilledand pod restart count so "presumed dead" splits into OOM vs CPU-stall vs eviction vs transient (today they're indistinguishable).perf_hooks.monitorEventLoopDelay) and CPU throttle seconds; alert before the beat goes stale. (App Insights instrumentation already in flight as PR feat: add Application Insights instrumentation and worker observability metrics #1226.)Recommended sequence
Related open hardening issues to fold in: #803 / #804 (heartbeat ownership), #805 (heartbeat hang on frozen queue), #806 (queue reliability), #1056 (redelivery observability).