Skip to content

"Worker presumed dead" recurs after #1249/#1067 fixes — claude-code worker still on 1Gi/1-core (integration) #1316

Open

Description

Original author: Cedric Vidal (@cedricvidal)

Summary

The Worker presumed dead (… queue message redelivered while run was in 'processing' state) failure mode is still occurring in integration after the 2026-07-06 fixes (#1249, #1067, PR #1247). Both of those trackers are closed, and there is currently no open issue tracking the recurrence.

A sweep of the integration API (msscope-int, build 3f2a4fc, project Default Project d95231c0-884b-48fd-a950-b0e71d449989) found 16 failed runs (outcome: failed) whose run.error matches "Worker presumed dead". 12 pre-date the 07-06 fixes and are explained by #1249/#1067. 4 post-date them and are not covered by any fix or open issue.

Post-fix recurrences (not covered by #1249/#1067)

Run ID Worker Model Agent ver Created (UTC) Pod / instance
75638258-791b-4c42-b06f-c37f10d4b599 coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03 …-qtsmt / 461d13be…
60aa6bc9-940c-4f28-8584-065c37c1e480 coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03 …-qtsmt / 12cfe0d1…
27cb2832-1a95-4da8-9450-89b7468e88af coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03 …-9hxmf / ad2a717a…
3b56ccf8-c115-4e5e-ad22-25b703ea7c2f coder-acp-copilot claude-opus-4.8 copilot-1.0.65 2026-07-14 21:40 …-5ztql / c6bec9da…

All four show a consistent gap: last heartbeat 182s ago, threshold 120s. Example:

Worker presumed dead (instance=461d13be-25ae-4fed-b04e-14d1ab6abf1f pod=coder-acp-claude-code-69cbd5bbcd-qtsmt, last heartbeat 182s ago, threshold 120s); queue message redelivered while run was in 'processing' state

The three 07-10 claude-code runs are one batch (submission de0c0ff7-d0ed-4049-8ea0-aeb7d63b1327, scenario "Rayfin: New Project Setup").

Root-cause hypothesis

The coder-acp-claude-code worker never received the resource bump that fixed the copilot worker in #1247. Its main container still has the exact 1Gi memory + 1-core CPU limits that #1249 (OOM) and #1067 (CPU starvation) identified as the root cause of stalled heartbeats:

main-container CPU main-container memory
coder-acp-copilot (post #1247) req 250m / lim 1 req 1Gi / lim 2.3Gi
coder-acp-claude-code (current) req 250m / lim 1 req 512Mi / lim 1Gi

Source: deploy/base/workers/coder-acp-claude-code.yaml (lines ~40–46) vs deploy/base/workers/coder-acp-copilot.yaml (lines ~32–38). The claude-code worker is the "memory/CPU sibling" that #1249/#1067 called out but only patched for copilot.

The single 07-14 copilot claude-opus-4.8 recurrence (3b56ccf8…) is a separate, lower-confidence data point — copilot already has 2.3Gi, so this is either a heavier-model/heavier-task run exceeding even the raised ceiling, or a genuine transient (pod eviction / queue hiccup). Worth confirming from pod metrics/events before assuming the limit is still too low.

Proposed next steps

  1. Raise coder-acp-claude-code main-container limits to parity with copilot (memory: 2.3Gi, requests.memory: 1Gi), matching the fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247 fix — most likely resolves the 07-10 cluster.
  2. Investigate the 07-14 copilot opus-4.8 run (3b56ccf8…) via pod events/metrics to determine OOM vs eviction vs transient.
  3. Consider the open hardening items that reduce blast radius but don't fix root cause: Worker reliability under Azure Storage Queue intermittent failures #806 (queue reliability), Guard all run mutations behind heartbeat ownership (follow-up to PR #793/#794) #803/Self-terminate worker process on heartbeat ownership loss (alternative to #803) #804 (heartbeat ownership), Heartbeat silently hangs on frozen queue endpoint (no client-side timeout on updateMessage) #805 (heartbeat hang on frozen queue), Add observability for safeDeferMessage failures in queue redelivery path #1056 (redelivery observability).
  4. Retry the 4 affected runs after limits are raised.

Full match list (16 runs, integration / Default Project)

12 of these pre-date the 07-06 fixes (explained by #1249/#1067); listed for completeness.

Run ID Worker Model Agent ver Created (UTC)
3b56ccf8-c115-4e5e-ad22-25b703ea7c2f coder-acp-copilot claude-opus-4.8 copilot-1.0.65 2026-07-14 21:40
75638258-791b-4c42-b06f-c37f10d4b599 coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03
60aa6bc9-940c-4f28-8584-065c37c1e480 coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03
27cb2832-1a95-4da8-9450-89b7468e88af coder-acp-claude-code claude-sonnet-4-6 acp-0.29.0-sdk-0.2.111 2026-07-10 13:03
d1916e3f-47d9-453b-a94b-14d565f96506 coder-acp-copilot gpt-5.5 copilot-1.0.55 2026-07-02 09:27
dfa1969d-ba33-4bcf-b3a5-8522b9ecd78b coder-acp-copilot claude-haiku-4.5 copilot-1.0.65 2026-06-30 16:26
1065b6bd-703b-40ff-82ce-7b7fdf683644 coder-acp-copilot claude-haiku-4.5 copilot-1.0.65 2026-06-30 09:24
7a3e09f4-7087-4da5-a8a6-9eb0686cdbea coder-acp-claude-code claude-opus-4-6 acp-0.29.0-sdk-0.2.111 2026-06-16 15:21
04edddcb-4622-4b7d-8993-973daf2439a2 coder-vscode-electron-driver-ext claude-haiku-4.5 vscode-electron-1.122.0 2026-06-15 16:28
a72aa868-883b-4682-9435-4c4b3c11c954 coder-vscode-electron-driver-ext claude-haiku-4.5 vscode-electron-1.122.0 2026-06-11 21:43
f77498a7-f9d8-4ad4-8fa7-3ff452f20351 coder-acp-copilot gpt-5.3-codex copilot-1.0.43 2026-05-26 19:05
b4121a45-ea03-4468-9ae0-cbfd284d15a6 coder-acp-copilot gpt-5.4 copilot-1.0.43 2026-05-26 19:05
6f1ca384-16bb-474e-bb34-8e81be71b159 coder-acp-copilot gpt-5.4 copilot-1.0.43 2026-05-26 19:05
b86a3373-e432-493d-a153-a09567c52cb3 coder-acp-copilot gpt-5.4 copilot-1.0.43 2026-05-26 17:19
822bde3b-a0e8-411b-b61e-89c03fc8a386 coder-acp-copilot claude-opus-4.7 copilot-1.0.43 2026-05-26 17:19
7e76f73d-25be-4364-a4f0-83925a34f1f6 coder-acp-copilot claude-opus-4.7 copilot-1.0.43 2026-05-26 17:19

Environment

  • Environment: integration (msscope-int, API build 3f2a4fc837c8a6491cf0aa5fd6dd77380947dd14)
  • Project: Default Project (d95231c0-884b-48fd-a950-b0e71d449989) — 63 total failed runs, 16 matching this signature. The other integration project ("Cedric's tests") has 0.

References

Activity

  1. cedricvidal commented on Jul 29, 2026

    @cedricvidal
    CollaboratorAuthor

    Original author: Cedric Vidal (@cedricvidal)

    Mitigation options — ideas to prevent "Worker presumed dead"

    Root observation: the liveness heartbeat is an in-process setInterval running in the same Node event loop as the agent workload. When the workload melts down (OOM, CPU throttling, event-loop blocking), the mechanism meant to detect liveness dies with it. That makes failures both (a) hard to prevent and (b) hard to attribute. Ideas grouped by strategy, roughly in priority order.

    1. Stop the death — right-size resources (quickest, highest leverage)

    Note from the current data: prod copilot already has the 2.3Gi bump yet still shows presumed-dead on heavy Claude models (opus-4.8, sonnet-4.6). So resource right-sizing alone likely won't fully close it — pair it with §2/§4 to confirm OOM vs CPU-stall vs eviction.

    2. Stop false reaping — decouple the heartbeat from the workload (structural)

    • Move the heartbeat off the main event loop — a worker_thread or a lightweight sidecar container with its own CPU/mem budget, reporting on the main process's real liveness (PID alive + progress). CPU saturation in the agent then can't starve it.
    • Progress-based liveness, not just a wall clock — drive the beat from actual ACP stream events / tool-calls / log lines, with an absolute max-run ceiling as backstop, so a quiet 3-min build reads as "busy," not "dead."
    • Two-strike reaping — before declaring dead, do a secondary probe (pod still Running? process alive?) instead of failing at the first stale threshold.
    • Client-side timeout on updateMessage (open issue Heartbeat silently hangs on frozen queue endpoint (no client-side timeout on updateMessage) #805) so a frozen queue endpoint can't silently hang the beat.

    3. Survive the death — make it non-fatal to the run (resilience)

    • Auto-retry "presumed dead" runs (these are infra failures, not agent failures) with backoff, ideally escalating to a higher-resource worker tier.
    • Checkpoint / resume instead of fail — iteration snapshots already exist; a redelivered message could resume from the last snapshot rather than mark the run failed.
    • Dedicated high-resource node pool (taints/tolerations) for heavy scenarios (cargo builds, large repos, heavy models), isolating them from light runs.

    4. See it coming & attribute it — observability

    • Read cgroup memory.current / memory.max inside the worker; when nearing the limit, proactively fail with a clear "insufficient memory" error + requeue, instead of an ambiguous OOM → presumed-dead.
    • Attribute the cause — inspect K8s lastState.terminated.reason == OOMKilled and pod restart count so "presumed dead" splits into OOM vs CPU-stall vs eviction vs transient (today they're indistinguishable).
    • Track event-loop lag (perf_hooks.monitorEventLoopDelay) and CPU throttle seconds; alert before the beat goes stale. (App Insights instrumentation already in flight as PR feat: add Application Insights instrumentation and worker observability metrics #1226.)

    Recommended sequence

    1. Quick win: claude-code resource parity + drop CPU limits (fixes the concrete int 07-10 recurrence and the Coder workers stall during heavy cargo builds — heartbeat starved by 1-core CPU limit ("Worker presumed dead") #1067 class).
    2. Structural: decouple the heartbeat (sidecar / worker_thread + progress-based) — eliminates the whole false-reap category.
    3. Resilience: auto-retry + snapshot-resume so even a real OOM doesn't lose the run.
    4. Confirm the prod copilot residual: pull pod-level OOMKilled/eviction detail for the post-fix(copilot-worker): raise memory limit 1Gi->2.3Gi to stop OOM "worker presumed dead" failures #1247 copilot runs to decide whether 2.3Gi is still too low for opus-4.8/sonnet-4.6, or the cause has shifted.

    Related open hardening issues to fold in: #803 / #804 (heartbeat ownership), #805 (heartbeat hang on frozen queue), #806 (queue reliability), #1056 (redelivery observability).

  2. added
    type: bugIncorrect behavior or a regression that needs fixing.
    area: workerCoding agent worker runtime, task execution, and lifecycle.
    area: infrastructureCloud resources, hosting, networking, and shared infrastructure.
    on Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureCloud resources, hosting, networking, and shared infrastructure.area: workerCoding agent worker runtime, task execution, and lifecycle.type: bugIncorrect behavior or a regression that needs fixing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions