Skip to content

fix(scheduler): stop StuckRunReaper from reaping healthy runs (#1064) - #1065

Closed
Cedric Vidal (cedricvidal) wants to merge 1 commit into
mainfrom
cedricvidal-microsoft/fix-stuck-run-reaper-false-positives
Closed

Cedric Vidal (cedricvidal) wants to merge 1 commit into
mainfrom
cedricvidal-microsoft/fix-stuck-run-reaper-false-positives

Conversation

@cedricvidal

Copy link
Copy Markdown
Collaborator

Original author: Cedric Vidal (@cedricvidal)

Why

In integration, the StuckRunReaper was failing healthy, actively-progressing runs with Reaped by scheduler — run stuck in 'processing' with stale/missing heartbeat. The reaping was systematic: runs whose heartbeats were visibly fresh on the run detail page got killed anyway.

Root cause

RedisHeartbeatStore.mget used a native multi-key MGET. Integration Redis runs the Azure Cache Enterprise clustering policy, where a multi-key MGET whose keys span hash slots fails with CROSSSLOT. The store's error handling swallowed that into an empty map, so the reaper's batch read returned zero beats for every processing candidate -> all looked heartbeat-less -> mass false reaps.

This also explains the two confusing symptoms:

  • Run detail / side panel use single-key get (always shard-routable) -> showed fresh beats.
  • Runs list view uses the same multi-key mget -> showed no heartbeats at all.

The existing guards (ping, two-strikes, circuit-breaker) could not catch it: ping is single-key so always succeeded, both consecutive sweeps read empty (persistent, not transient), and int had few concurrent runs.

Approach

  • Extract the cluster-safe read into a shared helper (clusterSafeMget): a pipeline of single-key GETs recombined by index, with per-key errors isolated to that one entry. Each command is single-key and therefore routable regardless of slot, so it is correct on standalone and clustered Redis alike. RedisHeartbeatStore.mget delegates to it, so the reaper sweep and the API runs-list enrichment share one tested implementation and cannot drift.
  • Defense-in-depth in the reaper: a single-key get() confirmation re-read immediately before failing a run. A fresh beat aborts the reap regardless of why the batch read missed it. Added per-sweep diagnostics and a loud "0 beats while Redis reachable" warning pointing at misconfig.
  • Liveness decoupling (separate robustness gap): the per-run liveness write now runs on a dedicated setInterval, independent of the queue visibility-extension loop, so a slow or failing updateMessage can no longer starve the heartbeat signal.

Notes for reviewers

  • The fix hinges on single-key vs multi-key commands, not on client-side shard awareness: the store uses a standalone ioredis client and the Azure Enterprise proxy routes each single-key GET to its owning shard. Pipelining is only a transport optimization (one round-trip).
  • The liveness decoupling is intentionally kept even though it was not the cause of the fresh-heartbeat reap; it closes a real starvation gap.

Tests

  • New cluster-safe-mget.test.ts (6) and heartbeat-store.test.ts (4) suites pinning the single-key-GET pipeline and asserting MGET is never used.
  • +2 reaper tests (confirmation read aborts reap when batch under-reports but single-key read is fresh; still reaps when truly dead).
  • +3 visibility-heartbeat tests (liveness fires despite updateMessage failures).
  • All queue + scheduler tests pass; shared, scheduler, api tsc clean.

Fixes: #1064

The reaper falsely failed healthy, actively-progressing runs in integration
with "stuck in 'processing' with stale/missing heartbeat".

Root cause: RedisHeartbeatStore.mget used a native multi-key MGET. Integration
Redis runs the Azure Cache Enterprise clustering policy, where a cross-slot
MGET fails with CROSSSLOT; the store swallowed that into an empty map, so every
processing run looked heartbeat-less and was reaped. Single-key GETs (run
detail / side panel) kept working, which is why heartbeats looked fresh there
while the reaper killed the run. The runs-list view was blank for the same
reason.

- Extract the canonical cluster-safe read into clusterSafeMget (pipeline of
  single-key GETs, recombined by index, per-key errors isolated). The store
  delegates to it, so the reaper sweep and the API runs-list enrichment share
  one tested implementation.
- Add a single-key confirmation re-read in the reaper before failing a run, so
  a fresh beat aborts the reap regardless of why the batch read missed it, plus
  per-sweep diagnostics.
- Decouple the liveness signal onto a dedicated setInterval so a slow/hung
  queue visibility extension can no longer starve heartbeats.

Tests: new cluster-safe-mget (6) and heartbeat-store (4) suites, +2 reaper
confirmation-read tests, +3 liveness-decoupling tests. shared/scheduler/api
tsc clean.

Fixes: #1064

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cedricvidal

Copy link
Copy Markdown
Collaborator Author

Original author: @github-actions[bot]

Test Results (Node.js 22)

test: Run #1867

Tests 📝 Passed ✅ Failed ❌ Skipped ⏭️ Pending ⏳ Other ❓ Flaky 🍂 Duration ⏱️
1904 1904 0 0 0 0 0 1m56s

🎉 All tests passed!

Github Test Reporter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StuckRunReaper false-positives: healthy long-running runs reaped on stale heartbeat (integration)

1 participant