Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Background
PR #856 added a per-run liveness heartbeat to fix #846 (false-positive failures on Azure Storage Queue redelivery). Heartbeats are currently written to MongoDB (run.lastHeartbeatAt) every HEARTBEAT_INTERVAL_MS (15s) for every actively-processing run.
Each tick is one $set on a nested field — roughly 10 RU on CosmosDB, so ~40 RU/min per active run. At low concurrency this is trivial; at hundreds of concurrent runs it becomes meaningful (~12k RU/min at 300 concurrent).
The data is intrinsically ephemeral (only meaningful within seconds of being written), making it a natural fit for Redis.
Goal
Move the per-run heartbeat to Redis. Keep run.worker.{instanceId,podName} in Mongo (it's persistent identity, written once per attempt — no recurring cost).
Design
Storage
run-heartbeat:<runId> → <iso-timestamp> TTL = 5 × HEARTBEAT_VISIBILITY_SECONDS (default 300s)
Single key per active run. No claim key — race protection comes from Mongo's findOneAndUpdate filter on run.worker.instanceId (already the source of truth).
TTL of 5× visibility (300s) is generous enough that the portal tooltip can still show "Last heartbeat: Xs ago" for a few minutes after a run ends, without bloating Redis. Configurable via SCOPE_RUN_HEARTBEAT_REDIS_TTL_MS.
Write path (worker)
| Event | Today (Mongo) | Proposed |
|---|---|---|
| Pickup (queued → processing) | $set run.worker, run.lastHeartbeatAt |
$set run.worker only; SET run-heartbeat:<id> with TTL |
| Heartbeat tick (every 15s) | $set run.lastHeartbeatAt |
SET run-heartbeat:<id> with TTL |
| Terminal status (done / failed) | $set run.finishedAt, status, etc. |
also $set run.lastHeartbeatAt (one-shot, for permanent record); DEL run-heartbeat:<id> |
Read path: redelivery handler (worker)
const beat = await redis.get(`run-heartbeat:${runId}`);
const beatMs = beat ? Date.parse(beat) : null;
const ageMs = beatMs ? Date.now() - beatMs : Infinity;
const isStale = beat
? ageMs > staleThresholdMs
// Missing-heartbeat guard: only stale if the run was picked up long ago.
// Prevents a Redis blip from killing healthy just-picked-up runs.
: (Date.now() - new Date(run.startedAt).getTime() > staleThresholdMs);Read path: API serializer
Batch enrichment of the runs payload for visible processing runs:
const processing = runs.filter(r => r.run?.status === "processing");
if (processing.length) {
const vals = await redis.mget(processing.map(r => `run-heartbeat:${r.run._id}`));
processing.forEach((r, i) => { if (vals[i]) r.run.lastHeartbeatAt = vals[i]; });
}Apply at GET /api/v1/requests, GET /api/v1/requests/:id, and the SSE update path. Portal types unchanged — the field still arrives at runtime, just not from Mongo.
Race safety
Two workers receive a redelivery for the same stale run:
- Both
GET run-heartbeat:<id>→ both miss. - Both
findOnethe run → both seerun.worker.instanceId = X(the original owner). - Both attempt
findOneAndUpdate({ _id, "run.worker.instanceId": X, ...staleness guards }, { $set: { "run.worker.instanceId": <self>, ... } }). - First wins; second's filter no longer matches → second drops the message.
Same atomicity guarantee as today, just gated on worker.instanceId instead of on lastHeartbeatAt < cutoff.
Files to touch
| Path | Change |
|---|---|
packages/shared/src/queue/base-queue-processor.ts |
onTick writes Redis instead of Mongo |
packages/shared/src/queue/queue-processor.ts |
pickup seeds Redis; terminal transitions delete Redis key + persist final beat to Mongo; redelivery handler reads Redis |
packages/shared/src/queue/queue-processor.test.ts |
rewrite redelivery tests to mock Redis |
packages/shared/src/types/types.ts |
keep lastHeartbeatAt (still set at terminal transition); document its lifecycle |
apps/api/src/routes/requests.ts (or serializer) |
Redis enrichment for processing runs |
apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap |
regenerate if shape changes |
docs/architecture/queue-scheduler.md |
update heartbeat section: Redis storage, missing-heartbeat guard, race protocol |
ENV_VARIABLES.md |
document SCOPE_RUN_HEARTBEAT_REDIS_TTL_MS |
Test plan
Unit (mock Redis client):
- Heartbeat tick writes Redis key with TTL.
- Redelivery + fresh Redis key → drop, no DB mutation.
- Redelivery + stale Redis key → fail run with worker-identity error.
- Redelivery + missing Redis key + recent
startedAt→ drop (missing-heartbeat guard). - Redelivery + missing Redis key + old
startedAt→ fail run. - Race: two workers see stale state; only one's
findOneAndUpdatesucceeds. - Terminal transition deletes the Redis key and persists final
lastHeartbeatAtto Mongo. SCOPE_RUN_HEARTBEAT_STALE_MSenv override still respected.
Integration:
- One real Redis (already in
pnpm docker:up:infra) + one worker: process a run, assert key exists during processing, gone after completion, final beat in Mongo.
Rollout
- Land this PR. New code tolerates an absent Mongo
lastHeartbeatAtvalue during the live window (it's only set by Redis enrichment + final-beat write). - Watch one prod cycle for: redelivery-drop log volume, mass-fail incidents, Redis ops/sec, CosmosDB RU savings.
- Optionally tune
HEARTBEAT_VISIBILITY_SECONDSdown (e.g. 30s) once Redis cost is essentially zero — tighter detection of dead workers.
Out of scope
- Replace at-least-once Azure Storage Queue redelivery with FIFO + dead-letter (would obviate most of this).
References
- PR fix(shared): use heartbeat to disambiguate spurious queue redelivery + pulsing status badges (#846) #856 — original Mongo-based heartbeat
- Issue Unexpected Run marked failed: queue message was redelivered while run was already in 'processing' state #846 — original bug fix
Activity
cedricvidal commentedon Jul 29, 2026
Original author: Cedric Vidal (@cedricvidal)
Folding into PR #856 instead.