Skip to content

StuckRunReaper false-positives: healthy long-running runs reaped on stale heartbeat (integration) #1064

Closed

Description

Original author: Cedric Vidal (@cedricvidal)

Summary

In integration (https://msscope-int.azurewebsites.net), runs that are healthy and actively making progress are being marked done/failed by the StuckRunReaper with:

Reaped by scheduler — run stuck in 'processing' with stale/missing heartbeat (…); no worker recovered it. Use the retry endpoint to start a new attempt.

The worker is not actually dead — it keeps doing real work for many minutes after being reaped, then discards its result because the run is already terminal. This is a false-positive reap, not a genuine failure.

How it was observed

While re-running (force-retry) the 16 runs migrated from prod for growth-ecosystems/scope-doc#51, 8 of 8 runs that got picked up failed with the identical reaper error. Each was reaped ~190–228s after starting:

run        started   reaped    alive_secs
822bde3b   21:23:15  21:26:56  221
7e76f73d   21:23:15  21:26:56  221
e1ede0fd   21:23:12  21:26:56  224
b86a3373   21:23:18  21:26:56  218
f77498a7   21:23:45  21:26:56  191
b4121a45   21:27:47  21:30:56  189
6f1ca384   21:31:36  21:34:56  200
dea61104   21:36:08  21:39:56  228
afaa975c   21:39:20  21:42:56  216

Note the first 5 runs were all reaped at the exact same second (21:26:56), suggesting a systemic heartbeat-refresh failure under concurrent load rather than 5 independent worker deaths.

Smoking gun — worker was alive the whole time (run 822bde3b)

Full run.jsonl timeline (worker coder-acp-copilot, model claude-opus-4.7):

Time (PDT) Event
2:23:15 Run starts; agent authenticates; session created
2:23:33–2:25:38 Agent thinking + creating Rust project files
2:26:56 Scheduler REAPS run → marked done/failed (221s after start)
2:27:45 Post-processor atif handler runs; worker keeps going
2:28:57–2:31:46 More file edits + Check rust toolchain
2:32:09–2:37:15 Check compile — a ~5 minute cargo build
2:39:19 Worker finishes, tries to report → Run already terminal — discarding queue message

The worker kept building for ~12 minutes after the scheduler declared it dead; all that work was thrown away. There is a ~127s gap in streamed agent events (2:25:38 → 2:27:45) that lines up with the heartbeat going stale.

Mechanism (code references)

  • Workers extend queue-message visibility and write a per-run liveness timestamp to Redis every 15s: packages/shared/src/queue/visibility-heartbeat.ts (HEARTBEAT_INTERVAL_MS = 15_000).
  • The Redis liveness write is the onTick callback (packages/shared/src/queue/base-queue-processor.ts ~L256–281 → heartbeatStore.set(logRunId, new Date())). It only runs after queueClient.updateMessage() (Azure Queue visibility extension) succeeds — if the visibility extension throws, onTick is skipped and no liveness is recorded for that tick (visibility-heartbeat.ts ~L85–114, catch path).
  • StuckRunReaper fails any processing run whose Redis heartbeat is >120s stale (SCOPE_RUN_HEARTBEAT_STALE_MS, default 120000), once it has been stale across two consecutive sweeps: apps/scheduler/src/stuck-run-reaper.ts (threshold doc ~L53–54; reap message ~L261–300).

So if the 15s ticks don't land in Redis for ~2 minutes, a perfectly healthy, busy worker is declared dead.

Two candidate root causes (need pod stdout to disambiguate)

  1. Threshold too aggressive for this workload. These are heavy Rust microservice tasks. During a long single agent turn (large multi-file tool call) or a multi-minute cargo build, the heartbeat isn't refreshing within the 120s window. If the worker's event loop is blocked while handling a very large ACP stream/tool-call payload, the 15s setTimeout heartbeat is starved.
  2. Heartbeat writes failing in integration specifically. queueClient.updateMessage (Azurite/Storage in int) and/or the Redis heartbeatStore.set failing repeatedly, so onTick never records liveness. The 5-runs-reaped-at-the-same-second pattern points at a systemic refresh failure under concurrent load.

The published run.jsonl does not include the Visibility heartbeat tick #N / Visibility heartbeat tick failed console lines — those go to worker pod stdout. Confirming (1) vs (2) requires kubectl logs on the coder-acp-copilot-* pods in integration around the reap times above.

Suggested next steps

  • Pull coder-acp-copilot pod stdout in integration for one reaped run (e.g. 822bde3b, runId e6c8843a-0921-47bf-a53f-13b61bb2d1c3) and check for missing/failing Visibility heartbeat tick lines to confirm cause (1) vs (2).
  • If event-loop starvation during large tool calls / streamed responses is the issue, decouple the Redis liveness heartbeat from the queue-visibility tick and/or make it robust to a blocked main loop (e.g. dedicated timer, worker thread, or write liveness independent of updateMessage success).
  • Consider raising SCOPE_RUN_HEARTBEAT_STALE_MS (and the matching redelivery value) in integration if the threshold is simply too tight for these workloads — but only after confirming the heartbeat is genuinely being emitted.
  • When a reaped run's worker later completes, its result is discarded (Run already terminal — discarding queue message). Consider whether a still-alive worker should be able to reclaim/commit, to avoid wasting completed work.

Impact

  • Heavy/long-running benchmark runs cannot complete in integration — they are reaped mid-flight and report failed, wasting full agent executions (including multi-minute compiles).
  • Affects benchmark result fidelity and wastes worker capacity.

Environment

  • Integration: https://msscope-int.azurewebsites.net
  • Worker: coder-acp-copilot, version copilot-1.0.55-2026-06-10T19:34:03Z-83f508c5adaf03ca95f5b9c83ee1608700ea7ff8
  • Reaper threshold: SCOPE_RUN_HEARTBEAT_STALE_MS default 120000 (120s)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugIncorrect behavior or a regression that needs fixing.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions