Skip to content

fix(usage): page heavy rollup hours once over active workspaces - #10203

Merged
waleedlatif1 merged 5 commits into
stagingfrom
fix/activity-rollup-heavy-hour-paging
Nov 19, 2026
Merged

waleedlatif1 merged 5 commits into
stagingfrom
fix/activity-rollup-heavy-hour-paging

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This follows up #10201 with the remaining findings of its post-merge audit. Reads and the flags-off path are unchanged, and both flags still default off. Migration 0465 adds failed_passes (constant default) and corrections_at (nullable) to the one-row activity_rollup_state, and attempts (constant default) to the activity_rollup_dirty queue. Deployed code doesn't write attempts, so its inserts get the default.

  • A. Heavy hours are paged once, over active workspaces only.
    • Once a quarter-hour slice still times out, the rest of the hour, [slice start, hour end), is read once, a page at a time. A retry pages the whole hour once.
    • A page walks the next 1,000 workspaces that have any run, by a bounded loose index scan (one index descent each). Each is probed for a run in the range with a LATERAL LIMIT 1. Only workspaces with runs there are aggregated, 100 per statement.
    • Archived workspaces, and ones with no runs in the range, cost one descent and are never aggregated.
    • The pass deadline is checked before every aggregate statement, not only between pages, so at most one bounded statement can run past it.
  • B. Proven page plan. A page's runs are read through a VALUES list driving a LATERAL range scan per workspace on workflow_execution_logs_workspace_activity_level_idx, the way organizationWorkflowRuns reads. Cell recomputes and the sampler use the same path.
    • Plain EXPLAIN on a local 1M-run, 5k-workspace copy:
      • Walk: a loose index-only scan, plus an index-only LIMIT 1 probe per workspace on the activity index.
      • Aggregate: a VALUES scan driving index-only scans on the activity index.
      • The first version of the probe was an EXISTS. The planner turned it into a hashed started_at_idx range scan of the whole slice, which is why it is a LATERAL LIMIT 1.
    • Plans are below.
  • C. Running out of budget isn't "too heavy".
    • An hour whose paging is cut short by the pass deadline is left for the next pass: an attempts = 0 row with retry_at = now(), a warning-level log, and no backoff or error.
    • The next pass pages such a deferred hour from the start under its own 2-minute budget, with the lease extended to match, so an hour that simply needs more than one pass's time is not stuck forever.
    • It is not a retry: no attempt is counted and no backoff is committed. Only a hold for the paging window is written first, so a pass that dies mid-page does not restart it at once.
    • A deferred hour that runs out of its own budget too is heavy after all, and is backed off from then on.
    • Only a page that still times out, or a real retry that exhausts its own budget, backs an hour off.
  • D. The heartbeat means "maintained", not "every step succeeded".
    • maintained_at moves as soon as a pass at this version gets past its reset decision. A step that fails later loses no information: marks persist until a recompute removes them, and the late-end scan resumes from its cursor.
    • The renewal is one conditional UPDATE against the database clock, the clock writers use to gate their marks. If the heartbeat has already expired by that clock, the pass rebuilds instead of renewing it, whatever the worker's clock says.
    • So a persistently failing step no longer forces a rebuild every 6h. Gap rebuilds now happen only when no pass ran at all (cron or Trigger.dev down, or maintenance off).
    • Failures are counted in failed_passes (reset by the next pass that completes) and logged as Activity rollup pass failed with consecutiveFailures.
    • A pass that fails before its reset commits still leaves the heartbeat untouched, so the next pass rebuilds (unchanged).
  • E. Retry backoff is committed before the retry. A blocked hour's attempts and retry_at are bumped in their own statement before paging starts. A retry that crashes or loses its lease therefore waits out its backoff rather than looping.
  • F. Runbook: maxDuration is 10 minutes, not 5. The runbook below is updated end to end.
  • G. Correction queues can't silently stall reads (final-audit finding on D). With the heartbeat no longer tied to every step, one cell failing on every pass could have kept the dirty-mark and in-flight queues from draining while reads went on serving drifting cells. Three changes:
    1. Per-item isolation.
      • A dirty mark or in-flight recheck whose recompute fails is logged and backed off, and the queue moves on to the next item. A dirty mark pushes due_at out with backoff and counts attempts; an in-flight recheck backs off next_check_at the same way.
      • A failed in-flight recheck also inserts a dirty mark for its cell. Its runs may have ended early with an ended_at the late-end scan never sees, so the cell is read live until a recompute succeeds.
      • A step-level failure (the queue can't be read at all) still fails the pass.
    2. Marked cells are read live.
      • Each workspace-hour with a pending dirty mark is read from source in the read's own snapshot, through a per-cell LATERAL on the workspace activity index, for both the summary and the breakdown. Other workspaces' stored cells for the same hour are still served, so one mark costs one workspace-hour of live reading.
      • That covers marks that are due, backed off, or not yet processed, so a cell known to be stale is never served, whether or not its recompute works.
      • Why this over falling back the whole window: a single failing cell should cost one hour of live reading, not the rollup for every window. Live-reading the hour is exact by construction.
    3. Read gate.
      • corrections_at is set only when a pass gets through every due item of both correction queues. Item failures are isolated and don't count, but a queue cut short by the pass budget or truncated by its batch size does: that pass keeps the old timestamp. planRollupRead requires it within 6h, as it does fresh_through, and reads it in the same snapshot.
      • Why not a failed_passes threshold: that counter mixes in unrelated failures (housekeeping, backfill) and can't tell "the queues ran" from "the pass reached the end". A timestamp written only after the queues ran states exactly the property reads depend on. Any failure here degrades to the exact legacy read.
    • Cheap follow-on: once failed_passes reaches 3, a deferred hour counts as a retry, so a pass that keeps crashing while paging it backs it off.
  • H. Latent bug from feat(usage): serve organization activity from an hourly rollup #10200, fixed: the summary had been reading every hour live.
    • The straddle-hour test compared local(h) - h with local(last) - last, but last was an unparenthesized h + interval '1 hour' - interval '1 microsecond'. The subtraction became local(last) - h + 1h - 1µs, which marked every hour as a straddle.
    • Results stayed exact, which is why every equivalence check passed, but the summary never summed stored cells and so got none of the rollup's speedup. The breakdown was unaffected.
    • With the parentheses, the whole timezone matrix is still exact, and stored cells are now actually served. A new case corrupts a stored cell and checks that the summary reflects it; it goes red without the fix.

Plans (local, plain EXPLAIN)

=== walk (loose index scan, active check)
                                                                                            QUERY PLAN                                                                                             
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop Left Join
   CTE walk
     ->  Recursive Union
           ->  Limit
                 ->  Index Only Scan using workflow_execution_logs_workspace_id_cost_total_idx on workflow_execution_logs l_1
                       Index Cond: (workspace_id > 'ws2'::text)
           ->  WorkTable Scan on walk walk_1
                 Filter: ((id IS NOT NULL) AND (n < 1000))
   ->  CTE Scan on walk
         Filter: (id IS NOT NULL)
   ->  Limit
         ->  Index Only Scan using workflow_execution_logs_workspace_id_started_at_status_tota_idx on workflow_execution_logs l
               Index Cond: ((workspace_id = walk.id) AND (started_at >= '2026-11-05 10:15:00'::timestamp without time zone) AND (started_at < '2026-11-05 11:00:00'::timestamp without time zone))
(13 rows)

=== page aggregate (VALUES-driven LATERAL)
                                                                                                     QUERY PLAN                                                                                                     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 GroupAggregate
   Group Key: l.workspace_id, (COALESCE(l.workflow_id, ''::text)), l.trigger
   ->  Sort
         Sort Key: l.workspace_id, (COALESCE(l.workflow_id, ''::text)), l.trigger
         ->  Nested Loop
               ->  Values Scan on "*VALUES*"
               ->  Index Only Scan using workflow_execution_logs_workspace_id_started_at_status_tota_idx on workflow_execution_logs l
                     Index Cond: ((workspace_id = "*VALUES*".column1) AND (started_at >= '2026-11-05 10:15:00'::timestamp without time zone) AND (started_at < '2026-11-05 11:00:00'::timestamp without time zone))
(8 rows)

Test evidence

  • Integration test: organization-activity-rollup.integration.ts ran against a throwaway local Postgres 17 (all migrations through 0465): 32/32 pass. New in this round:

    • a permanently failing dirty cell is isolated and backed off, the other marks are still processed, its hour reads exact (live) meanwhile, and it is retried and cleared later;
    • reads fall back to legacy while no pass has run the correction queues within the bound, including after a pass whose correction step fails;
    • only the marked workspace's cell is read live; another workspace's stored cell in the same hour still serves;
    • a cell whose in-flight recheck keeps failing is marked dirty and reads exact, then recovers;
    • a pass whose budget cuts its queues short leaves corrections_at unchanged;
    • the summary sums stored cells (H);
    • a deferred hour counts as a retry once 3 passes in a row have failed. New cases:
    • C: an hour cut short by the pass budget is left for the next pass with attempts = 0, and that pass pages it.
    • D: a failing step keeps the heartbeat moving and counts failed_passes, which the next completed pass resets.
    • E: a retry that dies mid-read still has its backoff committed: attempts 2, retry_at 20 minutes out.
    • Deferred hours: a deferred hour paged again by a pass that dies counts no attempt, and keeps its hold for the paging window.
    • Clock skew: with the worker clock an hour behind, a heartbeat expired by the database clock still rebuilds.
    • The heavy-hour case now recovers through the real walk-and-page reader. Every read still asserts that the rollup served it, deep-equal to the uncached legacy SQL.
  • Red/green:

    Guard reverted Test Result
    A: page the rest of the hour once unit red (slices 2 and 3 read again)
    C: budget-cut hour not treated as heavy integration red
    D: heartbeat at the reset decision integration red (heartbeat 5h old)
    E: backoff committed before the retry integration red (attempts 1, retry_at not in the future)
    Deferred hour is not a retry integration red (attempt counted)
    Hold committed before paging a deferred hour integration red (no hold)
    G: per-mark isolation integration red (the pass throws, other marks stuck)
    G: mark backoff integration red (mark stays due)
    G: marked hours read live integration red (stale cell served)
    G: corrections gate integration red (served without corrections)
    G: marks scoped per workspace-hour integration red (other workspace's stored cell not served)
    G: failed in-flight recheck marks its cell dirty integration red (no mark)
    G: corrections_at only when both queues finished integration red (stamped after a budget-cut pass)
    G: corrections_at set only after both steps integration red
    H: parenthesized straddle test integration red (stored cells never served)
    Deferred hour counted once passes keep failing integration red
    Heartbeat renewed against the database clock integration red (no rebuild under a worker clock 1h behind)

    Every earlier guard in the code this touches was re-run and is still red:

    • the reset, sampler, late-end ties, marks gate and both gap guards;
    • heavy-hour backoff and no early retry;
    • retention marks, late-end scan, fresh-through, coverage, version and atomic coverage.
  • Earlier guards: all 13 from feat(usage): serve organization activity from an hourly rollup #10200 and every guard from fix(usage): harden activity rollup maintenance #10201 were re-run against this code and are still red. The straddle guard is now meaningful, because stored cells are actually summed.

  • Plans: shown above. Run against a locally seeded 1M-run, 5k-workspace copy, plain EXPLAIN.

  • Unit: cells.test.ts (6) covers summing slices, paging the rest of the hour once with a cursor, stopping before paging when the budget has run out, retries paging the whole hour, and no slicing on other errors.

  • Checks:

    • check:migrations origin/staging, drizzle-kit generate (no diff), check:schema-mock;
    • check:utils, check:test-patterns, check:comment-hygiene, check:file-names, check:explicit-any, check:sql-date-binding, check:boundaries.

    Full test, type-check and check:audits are left to CI.

Operator runbook (activity rollup: #10200, #10201 and this PR)

Nothing below happens on merge.

0. Pre-flight (needs approval; run against a READ REPLICA only)

SET statement_timeout = '60s';
SELECT version();
-- logs → workflow FK must be validated ON DELETE SET NULL (convalidated = t, confdeltype = 'n').
SELECT conname, convalidated, confdeltype FROM pg_constraint
WHERE conname = 'workflow_execution_logs_workflow_id_workflow_id_fk';
-- the late-end index from 0463 is valid
SELECT indexrelid::regclass, indisvalid, pg_size_pretty(pg_relation_size(indexrelid))
FROM pg_index WHERE indexrelid = 'workflow_execution_logs_late_end_idx'::regclass;
-- cardinality for the busiest recent day: runs vs rollup cells per hour
SELECT date_trunc('hour', started_at) AS hour, count(*) AS runs,
  count(DISTINCT (workspace_id, coalesce(workflow_id, ''), trigger)) AS cells
FROM workflow_execution_logs
WHERE started_at >= '<busy-day> 00:00' AND started_at < '<busy-day> 00:00'::timestamp + interval '1 day'
GROUP BY 1 ORDER BY 1;
SELECT min(started_at) FROM workflow_execution_logs;
SELECT reltuples::bigint AS approx_runs FROM pg_class WHERE relname = 'workflow_execution_logs';
SELECT count(*) FILTER (WHERE ended_at >= started_at + interval '2 hours') AS late, count(*) AS sampled
FROM workflow_execution_logs TABLESAMPLE SYSTEM (0.1);

1. Enable maintenance

  • Hosted: set "activity-rollup-maintenance": { "enabled": true } in AppConfig feature-flags, then deploy sim-<env>-fast.
  • Self-hosted: set ACTIVITY_ROLLUP_MAINTENANCE=true.
  • One activity-rollup run is enqueued every 2 minutes. The first pass resets (reset: "coverage") and starts backfilling newest to oldest.

2. Watch the backfill, its signals, and the thresholds

SELECT version, coverage_start, fresh_through, backfill_completed_at, last_end_scan,
  maintained_at, corrections_at, failed_passes, lease_owner, housekeeping_at, updated_at
FROM activity_rollup_state;
-- attempts = 0 marks an hour a pass ran out of budget paging: the next pass pages it, with no backoff.
SELECT bucket_start, attempts, retry_at FROM activity_rollup_blocked ORDER BY bucket_start;
SELECT count(*) AS dirty, count(*) FILTER (WHERE attempts > 0) AS failing, max(attempts)
FROM activity_rollup_dirty;
SELECT count(*) AS inflight, max(attempts) FROM activity_rollup_inflight;
SELECT pg_size_pretty(pg_total_relation_size('workflow_activity_hourly'));

Every pass logs Activity rollup pass finished, with:

  • reset, blockedHours, finalizedHours, backfilledHours;
  • lateEndMarks, dirtyRecomputed, inflightRechecked, samplerMismatches, collectedWorkspaces;
  • replicaLagSeconds, coverageStart, freshThrough, backfillComplete.
Signal Healthy Act when
activity-rollup task duration < 60s (budget 40s); up to ~7 min while retrying a heavy hour repeatedly hits maxDuration (10 min)
failed_passes / Activity rollup pass failed (consecutiveFailures) 0 3+ in a row: a step is failing every pass. Marks are kept and the heartbeat still moves, so nothing rebuilds; fix the step.
reset null (except the first pass, or after a gap/version bump) non-null with no reset you started: a maintenance gap > 6h or coverage cleared
freshThrough 3–4h behind now > 5h behind (reads fall back to legacy at 6h)
blockedHours / Activity rollup hour too heavy to read; backing off 0 any: those hours (and progress past them in their direction) wait; check replica load
maintained_at within the last few minutes older than 6h: no pass ran (the cron or Trigger.dev is down); the next pass rebuilds
Activity rollup hour read in slices / hour read by workspace / hour left for paging by the next pass rare on most hours
replicaLagSeconds < 1s null with Activity rollup skipped replica reads for lag on most passes
dirty count drains to ~0 each pass grows across passes. Marked hours are read live, so a backlog costs read time, not accuracy.
failing dirty marks / Activity rollup dirty cell failed to recompute; backing off, in-flight cell failed … 0 any sustained: that cell can't be recomputed (its hour keeps being read live); investigate
corrections_at within the last few minutes older than 6h: reads fall back to legacy; the correction step is failing every pass (see Activity rollup pass failed)
inflight count roughly paused/running runs older than 3h grows without bound
samplerMismatches 0 sustained non-zero: turn reads off and investigate
Failed to mark activity rollup cells for recomputing, Activity rollup sampler failed, workspace collection failed absent any occurrence
Backfill speed ~20k source rows/s replica lag/CPU climbs: lower ACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND

3. Enable reads: internal org, then the large org, then globally

Wait until backfill_completed_at is set (at minimum, until coverage_start is older than now − 93 days).

  1. Set "organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] } and deploy.
    • Compare the activity page (summary, every dimension and sort) with the flag off and on.
    • Expect small differences on old days after a retention delete: legacy serves cached settled days for up to 7 days (the segment cache's TTL), while the rollup reflects the delete as soon as its mark is processed and matches the uncached legacy SQL.
    • Organization activity rollup read failed must be absent. Leave it for a day.
  2. Add the large org's id. Watch p95 latency on both routes, replica CPU and cancellations, and Organization activity chat days read.
  3. Set "enabled": true.

4. Rollback and reset

  • Reads (instant, ~30s AppConfig cache): remove the org from orgIds, or disable organization-activity-rollup-reads. Always do this first.
  • Maintenance: disable activity-rollup-maintenance. Passes stop, and writers stop marking 6h after the last pass. When maintenance is re-enabled after more than 6h, the first pass rebuilds automatically (reset: "maintenance-gap"). Within 6h it simply resumes.
  • Force a rebuild (needs approval; primary): with reads off, run this single statement:
    UPDATE activity_rollup_state SET coverage_start = NULL;
    • Reads fall back at once, since they require both coverage ends.
    • The next pass (reset: "coverage") clears all progress and queues, truncates the cells and backfills again, under its lease.
    • It is safe to run while a pass holds the lease: that pass's next hour write rolls back.
    • No code-level alternative is needed. Bumping ACTIVITY_ROLLUP_VERSION takes the same path on deploy (reset: "version").
  • Unblock a heavy hour early (needs approval): UPDATE activity_rollup_blocked SET retry_at = now();
  • Removal: with both flags off the tables are inert. Dropping them is a later contract migration.

@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 19, 2026 9:45am UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

This PR bounds heavy-hour reads, improves retry recovery, and keeps activity totals accurate when corrections fail.

  • Heavy hours resume by workspace pages instead of starting over.
  • Activity reads keep pending correction hours current.
  • Activity summaries now sum stored cells for ordinary hours.

The five previous threads are addressed in the current code. No new actionable issues were found.

Reviews (5) · Last reviewed commit: "fix(usage): read only marked workspace-h..." · Reviewed by Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

This PR changes payment, billing, or quota logic. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic

…e per aggregate, renew the heartbeat by the database clock
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 11 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 11 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

… gate reads on drained queues, and fix the summary straddle test
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

…ight cells dirty, and stamp corrections only when both queues drain
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 27a73ac into staging Nov 19, 2026
83 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/activity-rollup-heavy-hour-paging branch November 19, 2026 17:12

This branch was previously deployed

1 inactive deployment
Preview — 18469526 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant