Skip to content

Move per-run liveness heartbeat from Mongo to Redis #867

Closed

Description

Original author: Cedric Vidal (@cedricvidal)

Background

PR #856 added a per-run liveness heartbeat to fix #846 (false-positive failures on Azure Storage Queue redelivery). Heartbeats are currently written to MongoDB (run.lastHeartbeatAt) every HEARTBEAT_INTERVAL_MS (15s) for every actively-processing run.

Each tick is one $set on a nested field — roughly 10 RU on CosmosDB, so ~40 RU/min per active run. At low concurrency this is trivial; at hundreds of concurrent runs it becomes meaningful (~12k RU/min at 300 concurrent).

The data is intrinsically ephemeral (only meaningful within seconds of being written), making it a natural fit for Redis.

Goal

Move the per-run heartbeat to Redis. Keep run.worker.{instanceId,podName} in Mongo (it's persistent identity, written once per attempt — no recurring cost).

Design

Storage

run-heartbeat:<runId>  →  <iso-timestamp>   TTL = 5 × HEARTBEAT_VISIBILITY_SECONDS (default 300s)

Single key per active run. No claim key — race protection comes from Mongo's findOneAndUpdate filter on run.worker.instanceId (already the source of truth).

TTL of 5× visibility (300s) is generous enough that the portal tooltip can still show "Last heartbeat: Xs ago" for a few minutes after a run ends, without bloating Redis. Configurable via SCOPE_RUN_HEARTBEAT_REDIS_TTL_MS.

Write path (worker)

Event Today (Mongo) Proposed
Pickup (queued → processing) $set run.worker, run.lastHeartbeatAt $set run.worker only; SET run-heartbeat:<id> with TTL
Heartbeat tick (every 15s) $set run.lastHeartbeatAt SET run-heartbeat:<id> with TTL
Terminal status (done / failed) $set run.finishedAt, status, etc. also $set run.lastHeartbeatAt (one-shot, for permanent record); DEL run-heartbeat:<id>

Read path: redelivery handler (worker)

const beat = await redis.get(`run-heartbeat:${runId}`);
const beatMs = beat ? Date.parse(beat) : null;
const ageMs = beatMs ? Date.now() - beatMs : Infinity;

const isStale = beat
  ? ageMs > staleThresholdMs
  // Missing-heartbeat guard: only stale if the run was picked up long ago.
  // Prevents a Redis blip from killing healthy just-picked-up runs.
  : (Date.now() - new Date(run.startedAt).getTime() > staleThresholdMs);

Read path: API serializer

Batch enrichment of the runs payload for visible processing runs:

const processing = runs.filter(r => r.run?.status === "processing");
if (processing.length) {
  const vals = await redis.mget(processing.map(r => `run-heartbeat:${r.run._id}`));
  processing.forEach((r, i) => { if (vals[i]) r.run.lastHeartbeatAt = vals[i]; });
}

Apply at GET /api/v1/requests, GET /api/v1/requests/:id, and the SSE update path. Portal types unchanged — the field still arrives at runtime, just not from Mongo.

Race safety

Two workers receive a redelivery for the same stale run:

  1. Both GET run-heartbeat:<id> → both miss.
  2. Both findOne the run → both see run.worker.instanceId = X (the original owner).
  3. Both attempt findOneAndUpdate({ _id, "run.worker.instanceId": X, ...staleness guards }, { $set: { "run.worker.instanceId": <self>, ... } }).
  4. First wins; second's filter no longer matches → second drops the message.

Same atomicity guarantee as today, just gated on worker.instanceId instead of on lastHeartbeatAt < cutoff.

Files to touch

Path Change
packages/shared/src/queue/base-queue-processor.ts onTick writes Redis instead of Mongo
packages/shared/src/queue/queue-processor.ts pickup seeds Redis; terminal transitions delete Redis key + persist final beat to Mongo; redelivery handler reads Redis
packages/shared/src/queue/queue-processor.test.ts rewrite redelivery tests to mock Redis
packages/shared/src/types/types.ts keep lastHeartbeatAt (still set at terminal transition); document its lifecycle
apps/api/src/routes/requests.ts (or serializer) Redis enrichment for processing runs
apps/api/src/__snapshots__/openapi-snapshot.test.ts.snap regenerate if shape changes
docs/architecture/queue-scheduler.md update heartbeat section: Redis storage, missing-heartbeat guard, race protocol
ENV_VARIABLES.md document SCOPE_RUN_HEARTBEAT_REDIS_TTL_MS

Test plan

Unit (mock Redis client):

  1. Heartbeat tick writes Redis key with TTL.
  2. Redelivery + fresh Redis key → drop, no DB mutation.
  3. Redelivery + stale Redis key → fail run with worker-identity error.
  4. Redelivery + missing Redis key + recent startedAt → drop (missing-heartbeat guard).
  5. Redelivery + missing Redis key + old startedAt → fail run.
  6. Race: two workers see stale state; only one's findOneAndUpdate succeeds.
  7. Terminal transition deletes the Redis key and persists final lastHeartbeatAt to Mongo.
  8. SCOPE_RUN_HEARTBEAT_STALE_MS env override still respected.

Integration:

  • One real Redis (already in pnpm docker:up:infra) + one worker: process a run, assert key exists during processing, gone after completion, final beat in Mongo.

Rollout

  1. Land this PR. New code tolerates an absent Mongo lastHeartbeatAt value during the live window (it's only set by Redis enrichment + final-beat write).
  2. Watch one prod cycle for: redelivery-drop log volume, mass-fail incidents, Redis ops/sec, CosmosDB RU savings.
  3. Optionally tune HEARTBEAT_VISIBILITY_SECONDS down (e.g. 30s) once Redis cost is essentially zero — tighter detection of dead workers.

Out of scope

  • Replace at-least-once Azure Storage Queue redelivery with FIFO + dead-letter (would obviate most of this).

References

Activity

  1. cedricvidal commented on Jul 29, 2026

    @cedricvidal
    CollaboratorAuthor

    Original author: Cedric Vidal (@cedricvidal)

    Folding into PR #856 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions