Skip to content

fix: stop runs getting permanently stuck in processing (redelivery token + reaper) - #1052

Closed
Cedric Vidal (cedricvidal) wants to merge 4 commits into
mainfrom
cedricvidal-microsoft/fix-stuck-runs-duplicate-queue
Closed

Cedric Vidal (cedricvidal) wants to merge 4 commits into
mainfrom
cedricvidal-microsoft/fix-stuck-runs-duplicate-queue

Conversation

@cedricvidal

Copy link
Copy Markdown
Collaborator

Original author: Cedric Vidal (@cedricvidal)

Why

Benchmark runs could get permanently stuck in processing — sometimes for hours or days. Two gaps combined to cause it:

  1. The redelivery token was thrown away. Azure Storage Queue is at-least-once: when a duplicate message arrived for a run already in processing, the worker deleted the duplicate. That deletion discarded the only token that could have resurfaced the work. If the original worker then died hard (OOM, node eviction, SIGKILL) without writing a terminal state, the run sat in processing forever. (growth-ecosystems/scope-core#1294)
  2. There was no heartbeat-based backstop. Nothing swept for runs whose worker had died and whose heartbeat had gone stale, so a stuck run was never reaped. (growth-ecosystems/scope-doc#51)

This PR closes both gaps.

Approach

Two complementary layers — a primary fix on the hot path and an authoritative backstop.

1. Preserve the redelivery token (primary fix — growth-ecosystems/scope-core#1294)

When a worker dequeues a duplicate for a run already processing, it now consults the per-run Redis liveness heartbeat instead of blindly deleting:

  • Fresh heartbeat (owner alive) -> re-defer the duplicate (push its visibility out by SCOPE_RUN_REDELIVER_DEFER_MS) instead of deleting it. The message survives as the at-least-once recovery token. Each time it resurfaces, a fresh heartbeat re-defers it (cheap); a stale heartbeat fails the run.
  • Stale / missing heartbeat (owner presumed dead) -> mark the run failed atomically.

To make this safe, both terminal writes (success and error paths) are now gated on run.status: "processing", so a redelivered message can never clobber a run that was already finished, cancelled, retried, or reaped. The re-defer freezes the pop receipt via heartbeat.stop() first so a concurrent heartbeat tick can't rotate the receipt out from under it.

2. Stuck-run reaper (scheduler backstop — growth-ecosystems/scope-doc#51)

A new StuckRunReaper runs in the scheduler and periodically sweeps MongoDB for processing runs whose heartbeat has gone stale, failing them as status: "done" + outcome: "failed". This is the heartbeat-based reap that growth-ecosystems/scope-doc#51 called for. It is deliberately conservative:

  • Redis is non-fatal — if REDIS_HOST is unset or the store can't be constructed, the reaper self-disables and the dispatch loop keeps running.
  • Redis-outage guard — skips the sweep if Redis is unreachable (a missing heartbeat during an outage is meaningless), with a second ping to close the ping->die->mget window.
  • Two-strikes — a run must look stale in two consecutive sweeps before it is reaped.
  • Atomic claim — findOneAndUpdate gated on _id + run._id + run.status: "processing" + worker.instanceId, so it races cleanly against a recovering worker; whoever writes first wins and the loser no-ops.
  • CosmosDB-safe query — filters only on the indexed run.status equality; the startedAt cutoff is applied in-memory (no range index required).

Operational caveat: the reaper circuit breaker is all-or-nothing

SCOPE_REAPER_MAX_PER_SWEEP (default 30) is not a batch size — it is an all-or-nothing tripwire. If the number of runs stale across two sweeps exceeds the cap, the sweep fails none of them and returns. This is intentional: a large correlated spike is almost always one systemic cause (Redis throttled, CosmosDB 429 storm lagging heartbeats fleet-wide, network partition) rather than that many independent worker deaths, and in that case the workers are usually still alive — mass-failing them would destroy healthy runs.

The tradeoff: a genuine mass failure above the cap (e.g. a node pool dies with 40 real worker pods) is not auto-reaped. Those runs stay processing, every sweep skips them, and they remain stuck until an operator raises SCOPE_REAPER_MAX_PER_SWEEP or fails them manually. Queue redelivery remains the primary recovery path, so this is an acceptable safety-over-automation tradeoff, but it has two load-bearing implications:

  1. The circuit-breaker skip log must be alertable. It is a distinctive console.error from StuckRunReaper; something needs to page on it, otherwise a real mass death is silent.
  2. The runbook needs the manual step ("raise the cap or reap manually"), because nothing else will clear those runs.

Configuration

New env knobs (all documented in ENV_VARIABLES.md):

Var Default Purpose
SCOPE_RUN_REDELIVER_DEFER_MS SCOPE_RUN_HEARTBEAT_STALE_MS (120000) How far a live-owner duplicate is re-deferred
SCOPE_REAPER_ENABLED true Kill-switch for the reaper
SCOPE_REAPER_POLL_INTERVAL_MS 60000 Reaper sweep cadence
SCOPE_REAPER_MAX_PER_SWEEP 30 Circuit-breaker cap (see caveat above)

The reaper reuses SCOPE_RUN_HEARTBEAT_STALE_MS as its staleness threshold — keep the scheduler and workers on the same value so both recovery paths agree on "worker dead". deploy/base/scheduler.yaml adds the redis-secrets secretRef and the SCOPE_REAPER_* env.

Testing

  • Unit tests for the redelivery handler, the gated terminal writes, the reaper sweep logic, and the heartbeat-store ping().
  • Integration tests (redelivery.integration.test.ts, stuck-run-reaper.integration.test.ts) that self-provision Azurite / Redis / Mongo via testcontainers — no manual pnpm docker:up:infra. They skip locally when Docker is unavailable but fail in CI, and run behind a new gated integration-test-queue job in ci.yml.
  • The B3 integration scenario was refactored to exercise the real worker terminal-status guard; disabling the guard makes it fail, confirming the test is non-vacuous.

Docs updated: ENV_VARIABLES.md, docs/architecture/queue-scheduler.md.

Fixes growth-ecosystems/scope-core#1294
Fixes growth-ecosystems/scope-doc#51

Fixes growth-ecosystems/scope-doc#37 — runs get permanently stuck in
`processing` with "Duplicate queue message dropped — original worker still
heart-beating".

Root cause: recovery of a `processing` run whose worker dies hard relies
entirely on Azure queue at-least-once redelivery, but the redelivery handler
permanently DELETED a duplicate message when the original worker's heartbeat
was fresh. If that worker then died (OOM/SIGKILL, no graceful failure write),
the run was left `processing` with no queue message in existence — and the
scheduler only dispatches `pending` runs, so it never recovered.

Part 1 — stop destroying the recovery token (root cause):
- queue-processor fresh-heartbeat branch now RE-DEFERS the duplicate instead of
  deleting it: it freezes this consumer's pop receipt via heartbeat.stop(),
  then pushes the message's visibility out by SCOPE_RUN_REDELIVER_DEFER_MS
  (default = staleness threshold). Each resurfacing re-checks liveness: fresh
  beat re-defers (cheap), stale beat marks failed (unchanged). The terminal
  `done` guard still deletes the message once the original completes.
- New safeDeferMessage() helper swallows updateMessage failures so a stale pop
  receipt can never fall through to the error path and false-fail a live run.

Part 1b — status-gate the worker error-path write:
- The failure write is now gated on run.status:"processing" (mirroring the
  success-path write) so a worker that throws after a cancel/retry/reap no-ops
  instead of clobbering the terminal state and re-enqueuing post-processing.

Part 2 — scheduler-side StuckRunReaper (authoritative backstop):
- New apps/scheduler/src/stuck-run-reaper.ts, wired non-fatally into index.ts.
  Pings Redis first (skips the sweep if unreachable — never mass-fails healthy
  runs during a blip); finds candidates on the indexed run.status only and
  applies the startedAt cutoff in memory (CosmosDB-safe, no range index);
  requires a run to be stale in two consecutive sweeps; a circuit-breaker skips
  a sweep that would reap more than SCOPE_REAPER_MAX_PER_SWEEP runs; claims each
  run atomically gated on worker identity and leaves postProcessorStatus unset
  so the PostProcessorDispatcher enqueues PP. Distinct reaper error message for
  forensics. Redis is non-fatal: missing/unreachable Redis self-disables the
  reaper while dispatch keeps running; health probe stays Redis-independent.
- heartbeat-store: implement ping() in both Redis and in-memory stores.
- deploy/base/scheduler.yaml: add redis-secrets secretRef + SCOPE_REAPER_* env.

Docs: queue-scheduler.md (re-defer rationale + reaper section) and
ENV_VARIABLES.md (SCOPE_RUN_REDELIVER_DEFER_MS, SCOPE_REAPER_*).

Tests: 61 pass across shared queue + scheduler (re-defer, swallowed defer
failure, frozen pop receipt, two-strikes, Redis-down skip, read-failure
re-ping, circuit-breaker, worker-identity claim, no-op on ownership change).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds real-infra integration tests (Azurite queue + Redis + Mongo) that
exercise the production recovery paths end-to-end:

- redelivery.integration.test.ts (A1): drives the real
  CodingAgentQueueProcessor.handleRequest() fresh-heartbeat branch and
  proves the duplicate message is re-deferred (not deleted) and
  re-surfaces after the defer window with the same runId — i.e. the
  recovery token survives.
- stuck-run-reaper.integration.test.ts (A2/B3/B4): drives the real
  StuckRunReaper.sweep() and proves it reaps a stuck processing run with
  a stale/missing heartbeat after two strikes (sparing fresh-beat runs),
  that a late worker terminal write no-ops once reaped (Mongo status
  gate), and that a run completed between strikes is never reaped.

Both files match the existing vitest.integration.config.ts globs, so
they run under `pnpm test:integration` (require `pnpm docker:up:infra`).
Verified non-vacuous via mutation testing: reintroducing the bug
(delete instead of re-defer; disable reapOne) makes A1/A2/B3 fail.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
B3 previously asserted MongoDB's filter evaluation via a hand-written
collection.updateOne gated on run.status:"processing" — that tested the
database middleware, not our code. Deleting the production status gate
would not have failed the test.

Rewrite B3 to drive the real recovery path: reap a run for real, then
construct a real CodingAgentQueueProcessor against a live Azurite queue,
enqueue + receive the reaped run's message, and call the real
handleRequest. Assert the production terminal-status guard discards the
message (deletes it from Azurite, proven by no redelivery after the
visibility timeout expires) without re-running the agent or reviving the
run. A stub agent that throws if invoked keeps the test mutation-sensitive.

Mutation-verified: disabling the guard makes B3 fail fast (stub agent
runs and throws); restoring it makes B3 pass via the real discard path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The shared/scheduler stuck-run integration tests previously depended on a
developer running `pnpm docker:up:infra` first and were not wired into CI,
so they never ran automatically and could silently rot.

Make them boot their own throwaway MongoDB/Redis/Azurite containers via
testcontainers on Docker-assigned ports (per-package helpers, kept inside
each package's src so they stay build-safe under each rootDir). Tests skip
locally when Docker is absent but FAIL in CI so they can never pass
vacuously. Ryuk is disabled because its container startup is fragile on
some Docker setups; afterAll stops every container failure-safe instead.

Add a `test:integration:queue` root script and a gated `integration-test-queue`
CI job (runs on shared/scheduler changes), wired into the build-images gate
and the ci-summary required check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1857

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
1889 1889 0 0 0 0 0 1m50s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1859

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
1889 1889 0 0 0 0 0 1m45s

🎉 All tests passed!

Github Test Reporter

@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: Josh Duffney (@duffney)

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant