Repository navigation
fix: stop runs getting permanently stuck in processing (redelivery token + reaper) - #1052
Closed
Cedric Vidal (cedricvidal) wants to merge 4 commits into
Closed
Cedric Vidal (cedricvidal) wants to merge 4 commits into
Cedric Vidal (cedricvidal) wants to merge 4 commits into
Conversation
Fixes growth-ecosystems/scope-doc#37 — runs get permanently stuck in `processing` with "Duplicate queue message dropped — original worker still heart-beating". Root cause: recovery of a `processing` run whose worker dies hard relies entirely on Azure queue at-least-once redelivery, but the redelivery handler permanently DELETED a duplicate message when the original worker's heartbeat was fresh. If that worker then died (OOM/SIGKILL, no graceful failure write), the run was left `processing` with no queue message in existence — and the scheduler only dispatches `pending` runs, so it never recovered. Part 1 — stop destroying the recovery token (root cause): - queue-processor fresh-heartbeat branch now RE-DEFERS the duplicate instead of deleting it: it freezes this consumer's pop receipt via heartbeat.stop(), then pushes the message's visibility out by SCOPE_RUN_REDELIVER_DEFER_MS (default = staleness threshold). Each resurfacing re-checks liveness: fresh beat re-defers (cheap), stale beat marks failed (unchanged). The terminal `done` guard still deletes the message once the original completes. - New safeDeferMessage() helper swallows updateMessage failures so a stale pop receipt can never fall through to the error path and false-fail a live run. Part 1b — status-gate the worker error-path write: - The failure write is now gated on run.status:"processing" (mirroring the success-path write) so a worker that throws after a cancel/retry/reap no-ops instead of clobbering the terminal state and re-enqueuing post-processing. Part 2 — scheduler-side StuckRunReaper (authoritative backstop): - New apps/scheduler/src/stuck-run-reaper.ts, wired non-fatally into index.ts. Pings Redis first (skips the sweep if unreachable — never mass-fails healthy runs during a blip); finds candidates on the indexed run.status only and applies the startedAt cutoff in memory (CosmosDB-safe, no range index); requires a run to be stale in two consecutive sweeps; a circuit-breaker skips a sweep that would reap more than SCOPE_REAPER_MAX_PER_SWEEP runs; claims each run atomically gated on worker identity and leaves postProcessorStatus unset so the PostProcessorDispatcher enqueues PP. Distinct reaper error message for forensics. Redis is non-fatal: missing/unreachable Redis self-disables the reaper while dispatch keeps running; health probe stays Redis-independent. - heartbeat-store: implement ping() in both Redis and in-memory stores. - deploy/base/scheduler.yaml: add redis-secrets secretRef + SCOPE_REAPER_* env. Docs: queue-scheduler.md (re-defer rationale + reaper section) and ENV_VARIABLES.md (SCOPE_RUN_REDELIVER_DEFER_MS, SCOPE_REAPER_*). Tests: 61 pass across shared queue + scheduler (re-defer, swallowed defer failure, frozen pop receipt, two-strikes, Redis-down skip, read-failure re-ping, circuit-breaker, worker-identity claim, no-op on ownership change). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds real-infra integration tests (Azurite queue + Redis + Mongo) that exercise the production recovery paths end-to-end: - redelivery.integration.test.ts (A1): drives the real CodingAgentQueueProcessor.handleRequest() fresh-heartbeat branch and proves the duplicate message is re-deferred (not deleted) and re-surfaces after the defer window with the same runId — i.e. the recovery token survives. - stuck-run-reaper.integration.test.ts (A2/B3/B4): drives the real StuckRunReaper.sweep() and proves it reaps a stuck processing run with a stale/missing heartbeat after two strikes (sparing fresh-beat runs), that a late worker terminal write no-ops once reaped (Mongo status gate), and that a run completed between strikes is never reaped. Both files match the existing vitest.integration.config.ts globs, so they run under `pnpm test:integration` (require `pnpm docker:up:infra`). Verified non-vacuous via mutation testing: reintroducing the bug (delete instead of re-defer; disable reapOne) makes A1/A2/B3 fail. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
B3 previously asserted MongoDB's filter evaluation via a hand-written collection.updateOne gated on run.status:"processing" — that tested the database middleware, not our code. Deleting the production status gate would not have failed the test. Rewrite B3 to drive the real recovery path: reap a run for real, then construct a real CodingAgentQueueProcessor against a live Azurite queue, enqueue + receive the reaped run's message, and call the real handleRequest. Assert the production terminal-status guard discards the message (deletes it from Azurite, proven by no redelivery after the visibility timeout expires) without re-running the agent or reviving the run. A stub agent that throws if invoked keeps the test mutation-sensitive. Mutation-verified: disabling the guard makes B3 fail fast (stub agent runs and throws); restoring it makes B3 pass via the real discard path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The shared/scheduler stuck-run integration tests previously depended on a developer running `pnpm docker:up:infra` first and were not wired into CI, so they never ran automatically and could silently rot. Make them boot their own throwaway MongoDB/Redis/Azurite containers via testcontainers on Docker-assigned ports (per-package helpers, kept inside each package's src so they stay build-safe under each rootDir). Tests skip locally when Docker is absent but FAIL in CI so they can never pass vacuously. Ryuk is disabled because its container startup is fragile on some Docker setups; afterAll stops every container failure-safe instead. Add a `test:integration:queue` root script and a gated `integration-test-queue` CI job (runs on shared/scheduler changes), wired into the build-images gate and the ci-summary required check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #1857
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #1859
🎉 All tests passed! |
Collaborator
Author
|
Original author: Josh Duffney (@duffney) LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original author: Cedric Vidal (@cedricvidal)
Why
Benchmark runs could get permanently stuck in
processing— sometimes for hours or days. Two gaps combined to cause it:processing, the worker deleted the duplicate. That deletion discarded the only token that could have resurfaced the work. If the original worker then died hard (OOM, node eviction, SIGKILL) without writing a terminal state, the run sat inprocessingforever. (growth-ecosystems/scope-core#1294)This PR closes both gaps.
Approach
Two complementary layers — a primary fix on the hot path and an authoritative backstop.
1. Preserve the redelivery token (primary fix — growth-ecosystems/scope-core#1294)
When a worker dequeues a duplicate for a run already
processing, it now consults the per-run Redis liveness heartbeat instead of blindly deleting:SCOPE_RUN_REDELIVER_DEFER_MS) instead of deleting it. The message survives as the at-least-once recovery token. Each time it resurfaces, a fresh heartbeat re-defers it (cheap); a stale heartbeat fails the run.To make this safe, both terminal writes (success and error paths) are now gated on
run.status: "processing", so a redelivered message can never clobber a run that was already finished, cancelled, retried, or reaped. The re-defer freezes the pop receipt viaheartbeat.stop()first so a concurrent heartbeat tick can't rotate the receipt out from under it.2. Stuck-run reaper (scheduler backstop — growth-ecosystems/scope-doc#51)
A new
StuckRunReaperruns in the scheduler and periodically sweeps MongoDB forprocessingruns whose heartbeat has gone stale, failing them asstatus: "done"+outcome: "failed". This is the heartbeat-based reap that growth-ecosystems/scope-doc#51 called for. It is deliberately conservative:REDIS_HOSTis unset or the store can't be constructed, the reaper self-disables and the dispatch loop keeps running.findOneAndUpdategated on_id+run._id+run.status: "processing"+worker.instanceId, so it races cleanly against a recovering worker; whoever writes first wins and the loser no-ops.run.statusequality; thestartedAtcutoff is applied in-memory (no range index required).Operational caveat: the reaper circuit breaker is all-or-nothing
SCOPE_REAPER_MAX_PER_SWEEP(default 30) is not a batch size — it is an all-or-nothing tripwire. If the number of runs stale across two sweeps exceeds the cap, the sweep fails none of them and returns. This is intentional: a large correlated spike is almost always one systemic cause (Redis throttled, CosmosDB 429 storm lagging heartbeats fleet-wide, network partition) rather than that many independent worker deaths, and in that case the workers are usually still alive — mass-failing them would destroy healthy runs.The tradeoff: a genuine mass failure above the cap (e.g. a node pool dies with 40 real worker pods) is not auto-reaped. Those runs stay
processing, every sweep skips them, and they remain stuck until an operator raisesSCOPE_REAPER_MAX_PER_SWEEPor fails them manually. Queue redelivery remains the primary recovery path, so this is an acceptable safety-over-automation tradeoff, but it has two load-bearing implications:console.errorfromStuckRunReaper; something needs to page on it, otherwise a real mass death is silent.Configuration
New env knobs (all documented in
ENV_VARIABLES.md):SCOPE_RUN_REDELIVER_DEFER_MSSCOPE_RUN_HEARTBEAT_STALE_MS(120000)SCOPE_REAPER_ENABLEDtrueSCOPE_REAPER_POLL_INTERVAL_MS60000SCOPE_REAPER_MAX_PER_SWEEP30The reaper reuses
SCOPE_RUN_HEARTBEAT_STALE_MSas its staleness threshold — keep the scheduler and workers on the same value so both recovery paths agree on "worker dead".deploy/base/scheduler.yamladds theredis-secretssecretRef and theSCOPE_REAPER_*env.Testing
ping().redelivery.integration.test.ts,stuck-run-reaper.integration.test.ts) that self-provision Azurite / Redis / Mongo via testcontainers — no manualpnpm docker:up:infra. They skip locally when Docker is unavailable but fail in CI, and run behind a new gatedintegration-test-queuejob inci.yml.Docs updated:
ENV_VARIABLES.md,docs/architecture/queue-scheduler.md.Fixes growth-ecosystems/scope-core#1294
Fixes growth-ecosystems/scope-doc#51