Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Summary
In integration (https://msscope-int.azurewebsites.net), runs that are healthy and actively making progress are being marked done/failed by the StuckRunReaper with:
Reaped by scheduler — run stuck in 'processing' with stale/missing heartbeat (…); no worker recovered it. Use the retry endpoint to start a new attempt.
The worker is not actually dead — it keeps doing real work for many minutes after being reaped, then discards its result because the run is already terminal. This is a false-positive reap, not a genuine failure.
How it was observed
While re-running (force-retry) the 16 runs migrated from prod for growth-ecosystems/scope-doc#51, 8 of 8 runs that got picked up failed with the identical reaper error. Each was reaped ~190–228s after starting:
run started reaped alive_secs
822bde3b 21:23:15 21:26:56 221
7e76f73d 21:23:15 21:26:56 221
e1ede0fd 21:23:12 21:26:56 224
b86a3373 21:23:18 21:26:56 218
f77498a7 21:23:45 21:26:56 191
b4121a45 21:27:47 21:30:56 189
6f1ca384 21:31:36 21:34:56 200
dea61104 21:36:08 21:39:56 228
afaa975c 21:39:20 21:42:56 216
Note the first 5 runs were all reaped at the exact same second (21:26:56), suggesting a systemic heartbeat-refresh failure under concurrent load rather than 5 independent worker deaths.
Smoking gun — worker was alive the whole time (run 822bde3b)
Full run.jsonl timeline (worker coder-acp-copilot, model claude-opus-4.7):
| Time (PDT) | Event |
|---|---|
| 2:23:15 | Run starts; agent authenticates; session created |
| 2:23:33–2:25:38 | Agent thinking + creating Rust project files |
| 2:26:56 | Scheduler REAPS run → marked done/failed (221s after start) |
| 2:27:45 | Post-processor atif handler runs; worker keeps going |
| 2:28:57–2:31:46 | More file edits + Check rust toolchain |
| 2:32:09–2:37:15 | Check compile — a ~5 minute cargo build |
| 2:39:19 | Worker finishes, tries to report → Run already terminal — discarding queue message |
The worker kept building for ~12 minutes after the scheduler declared it dead; all that work was thrown away. There is a ~127s gap in streamed agent events (2:25:38 → 2:27:45) that lines up with the heartbeat going stale.
Mechanism (code references)
- Workers extend queue-message visibility and write a per-run liveness timestamp to Redis every 15s:
packages/shared/src/queue/visibility-heartbeat.ts(HEARTBEAT_INTERVAL_MS = 15_000). - The Redis liveness write is the
onTickcallback (packages/shared/src/queue/base-queue-processor.ts~L256–281 →heartbeatStore.set(logRunId, new Date())). It only runs afterqueueClient.updateMessage()(Azure Queue visibility extension) succeeds — if the visibility extension throws,onTickis skipped and no liveness is recorded for that tick (visibility-heartbeat.ts~L85–114, catch path). StuckRunReaperfails anyprocessingrun whose Redis heartbeat is >120s stale (SCOPE_RUN_HEARTBEAT_STALE_MS, default120000), once it has been stale across two consecutive sweeps:apps/scheduler/src/stuck-run-reaper.ts(threshold doc ~L53–54; reap message ~L261–300).
So if the 15s ticks don't land in Redis for ~2 minutes, a perfectly healthy, busy worker is declared dead.
Two candidate root causes (need pod stdout to disambiguate)
- Threshold too aggressive for this workload. These are heavy Rust microservice tasks. During a long single agent turn (large multi-file tool call) or a multi-minute
cargo build, the heartbeat isn't refreshing within the 120s window. If the worker's event loop is blocked while handling a very large ACP stream/tool-call payload, the 15ssetTimeoutheartbeat is starved. - Heartbeat writes failing in integration specifically.
queueClient.updateMessage(Azurite/Storage in int) and/or the RedisheartbeatStore.setfailing repeatedly, soonTicknever records liveness. The 5-runs-reaped-at-the-same-second pattern points at a systemic refresh failure under concurrent load.
The published run.jsonl does not include the Visibility heartbeat tick #N / Visibility heartbeat tick failed console lines — those go to worker pod stdout. Confirming (1) vs (2) requires kubectl logs on the coder-acp-copilot-* pods in integration around the reap times above.
Suggested next steps
- Pull
coder-acp-copilotpod stdout in integration for one reaped run (e.g.822bde3b, runIde6c8843a-0921-47bf-a53f-13b61bb2d1c3) and check for missing/failingVisibility heartbeat ticklines to confirm cause (1) vs (2). - If event-loop starvation during large tool calls / streamed responses is the issue, decouple the Redis liveness heartbeat from the queue-visibility tick and/or make it robust to a blocked main loop (e.g. dedicated timer, worker thread, or write liveness independent of
updateMessagesuccess). - Consider raising
SCOPE_RUN_HEARTBEAT_STALE_MS(and the matching redelivery value) in integration if the threshold is simply too tight for these workloads — but only after confirming the heartbeat is genuinely being emitted. - When a reaped run's worker later completes, its result is discarded (
Run already terminal — discarding queue message). Consider whether a still-alive worker should be able to reclaim/commit, to avoid wasting completed work.
Impact
- Heavy/long-running benchmark runs cannot complete in integration — they are reaped mid-flight and report
failed, wasting full agent executions (including multi-minute compiles). - Affects benchmark result fidelity and wastes worker capacity.
Environment
- Integration:
https://msscope-int.azurewebsites.net - Worker:
coder-acp-copilot, versioncopilot-1.0.55-2026-06-10T19:34:03Z-83f508c5adaf03ca95f5b9c83ee1608700ea7ff8 - Reaper threshold:
SCOPE_RUN_HEARTBEAT_STALE_MSdefault120000(120s)
Activity