Repository navigation
fix(scheduler): stop StuckRunReaper from reaping healthy runs (#1064) - #1065
Closed
Cedric Vidal (cedricvidal) wants to merge 1 commit into
Closed
Cedric Vidal (cedricvidal) wants to merge 1 commit into
Cedric Vidal (cedricvidal) wants to merge 1 commit into
Conversation
The reaper falsely failed healthy, actively-progressing runs in integration with "stuck in 'processing' with stale/missing heartbeat". Root cause: RedisHeartbeatStore.mget used a native multi-key MGET. Integration Redis runs the Azure Cache Enterprise clustering policy, where a cross-slot MGET fails with CROSSSLOT; the store swallowed that into an empty map, so every processing run looked heartbeat-less and was reaped. Single-key GETs (run detail / side panel) kept working, which is why heartbeats looked fresh there while the reaper killed the run. The runs-list view was blank for the same reason. - Extract the canonical cluster-safe read into clusterSafeMget (pipeline of single-key GETs, recombined by index, per-key errors isolated). The store delegates to it, so the reaper sweep and the API runs-list enrichment share one tested implementation. - Add a single-key confirmation re-read in the reaper before failing a run, so a fresh beat aborts the reap regardless of why the batch read missed it, plus per-sweep diagnostics. - Decouple the liveness signal onto a dedicated setInterval so a slow/hung queue visibility extension can no longer starve heartbeats. Tests: new cluster-safe-mget (6) and heartbeat-store (4) suites, +2 reaper confirmation-read tests, +3 liveness-decoupling tests. shared/scheduler/api tsc clean. Fixes: #1064 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #1867
🎉 All tests passed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original author: Cedric Vidal (@cedricvidal)
Why
In integration, the
StuckRunReaperwas failing healthy, actively-progressing runs withReaped by scheduler — run stuck in 'processing' with stale/missing heartbeat. The reaping was systematic: runs whose heartbeats were visibly fresh on the run detail page got killed anyway.Root cause
RedisHeartbeatStore.mgetused a native multi-keyMGET. Integration Redis runs the Azure Cache Enterprise clustering policy, where a multi-keyMGETwhose keys span hash slots fails withCROSSSLOT. The store's error handling swallowed that into an empty map, so the reaper's batch read returned zero beats for everyprocessingcandidate -> all looked heartbeat-less -> mass false reaps.This also explains the two confusing symptoms:
get(always shard-routable) -> showed fresh beats.mget-> showed no heartbeats at all.The existing guards (ping, two-strikes, circuit-breaker) could not catch it:
pingis single-key so always succeeded, both consecutive sweeps read empty (persistent, not transient), and int had few concurrent runs.Approach
clusterSafeMget): a pipeline of single-keyGETs recombined by index, with per-key errors isolated to that one entry. Each command is single-key and therefore routable regardless of slot, so it is correct on standalone and clustered Redis alike.RedisHeartbeatStore.mgetdelegates to it, so the reaper sweep and the API runs-list enrichment share one tested implementation and cannot drift.get()confirmation re-read immediately before failing a run. A fresh beat aborts the reap regardless of why the batch read missed it. Added per-sweep diagnostics and a loud "0 beats while Redis reachable" warning pointing at misconfig.setInterval, independent of the queue visibility-extension loop, so a slow or failingupdateMessagecan no longer starve the heartbeat signal.Notes for reviewers
GETto its owning shard. Pipelining is only a transport optimization (one round-trip).Tests
cluster-safe-mget.test.ts(6) andheartbeat-store.test.ts(4) suites pinning the single-key-GET pipeline and assertingMGETis never used.updateMessagefailures).shared,scheduler,apitsc clean.Fixes: #1064