Skip to content

fix(usage): harden activity rollup maintenance - #10201

Merged
waleedlatif1 merged 3 commits into
stagingfrom
fix/activity-rollup-ops-hardening
Nov 19, 2026
Merged

waleedlatif1 merged 3 commits into
stagingfrom
fix/activity-rollup-ops-hardening

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This hardens the activity rollup maintenance from #10200 against the six operational findings of its post-merge audit. Reads and the flags-off path are unchanged: both flags still default off, so merging this changes nothing until maintenance is enabled. Migration 0464 adds a new activity_rollup_blocked table and one nullable maintained_at column on the one-row activity_rollup_state. Nothing deployed writes either.

  1. Resets rebuild for real, and no state can idle the rollup.
    • Any pass that finds either end of the coverage missing starts over under its lease (resetRollup): it clears coverage, the backfill completion, the late-end cursor and any blocked hour, empties the dirty and in-flight queues, truncates the cells and backfills again.
    • So the documented manual reset (coverage_start = NULL) rebuilds even after a completed backfill. Before, the completed backfill stayed set, the backfill never reran, and reads served nearly the whole window live.
    • A reset racing a pass can no longer leave fresh_through set with coverage_start NULL. Every hour write now moves progress only while both ends are set (otherwise the hour's transaction rolls back), and a pass that finds either end missing rebuilds.
    • A version bump goes through the same path.
  2. The sampler no longer reports false mismatches.
    • Stored cells' workflow ids are resolved the way reads resolve them, (SELECT f.id FROM workflow f WHERE f.id = nullif(h.workflow_id, '')), and re-summed. A workflow deleted after its cells were computed matches its nulled source rows.
    • Hours with a pending mark, or with an in-flight recheck, are not counted (ON CONFLICT DO NOTHING).
    • The sampled workspace is picked on the replica.
  3. A heavy hour can't wedge the pass.
    • If a quarter-hour slice still times out, that slice and the rest of the hour are read a page of 500 workspaces at a time, each page on the workspace activity index, within the pass's deadline.
    • If a page still times out, or the budget runs out, the hour is backed off in activity_rollup_blocked (one row per hour), with backoff from 10 minutes doubling to a cap of 6 hours and a loud Activity rollup hour too heavy to read; backing off error.
    • Finalize or backfill waits at that hour while every other step still runs. A heavy hour on each side keeps its own backoff, so the two can't keep resetting each other.
    • A retry skips the reads known to time out and pages through workspaces under its own budget: 2 minutes times its attempts, capped at 6, with the lease extended to cover it and the task's maxDuration raised to 10 minutes. The block is deleted in the same transaction that finally writes the hour. An hour that still cannot be paged within the cap stays blocked behind a loud error for an operator, rather than retrying forever.
    • Each housekeeping step (workspace GC, sampler) runs in its own error boundary.
  4. The late-end scan is bounded.
    • It consumes ends in ended_at order, 5,000 per statement, up to a horizon 5 minutes behind now (commits stamped earlier than that are visible by then). Each statement marks every end up to its batch's last one, including ties beyond the batch, and moves last_end_scan there in the same statement.
    • Catching up after a long pause is incremental and never one huge statement. A run's late end is now noticed about 5 minutes after it is stamped instead of on the next pass, which is harmless for hours that settled long ago.
  5. Dirty marks while maintenance is off: gated, plus a rebuild on resume.
    • Writers (retention, HITL reopen) mark only while a pass completed within the last 6h (activity_rollup_state.maintained_at). Only a completed pass, or the start of a new coverage, writes that column. Taking the lease doesn't, so a failing pass, or one idling on a newer version, does not keep marks flowing.
    • A pass that finds maintained_at more than 6h old (or missing) starts over. The reason is read from the stored state alone, so if a pass fails before its reset commits, the next pass rebuilds for the same reason.
    • A completed pass refreshes maintained_at only if the previous value was still within the gap when the pass ends. If it expired mid-pass, writers had already stopped marking, so it stays expired and the next pass rebuilds.
    • Why this design and not a bounded drain: marks are the only record of deletes and reopened runs, so a drain that falls behind would leave stale cells, and reads come back once fresh_through is fresh, possibly before the backlog drains. Resuming after a long gap also leaves the late-end cursor far behind.
    • A rebuild is the only exact way back. It needs no flag lookup on the hot write paths (the gate is an EXISTS on the one-row state table inside the existing savepoint), and it bounds the marks table to at most 6h of writes.
    • The cost is a full backfill after any gap over 6h (including a long Trigger.dev outage). A deployment that already ran feat(usage): serve organization activity from an hourly rollup #10200's maintenance also rebuilds once, since its state has no maintained_at. Reads fall back to legacy meanwhile, and recent windows return first because the backfill runs newest to oldest.
  6. Runbook: this documents that legacy serves cached settled days for up to 7 days, so a comparison after a retention delete can differ on old days. The rollup matches the uncached SQL (see the runbook below).

Simplify/cleanup notes:

  • Done: reused the existing backoff helper and folded the guarded progress update into updateLeasedState.
  • Skipped deliberately:
    • a single-scan late-end batch (the second, bounded scan is what makes ties exact);
    • a LEFT JOIN instead of the scalar workflow lookup (that lookup mirrors the read path exactly).

Test evidence

  • Integration test: organization-activity-rollup.integration.ts ran against a throwaway local Postgres 17 (all migrations through 0464): 23/23 pass in about 80s.

    • The full time-zone × window matrix is still deep-equal to the uncached legacy SQL, and asserts the rollup actually served each read.
    • New cases:
      • rebuilding after the coverage is cleared by hand, even with the backfill complete;
      • no sampler mismatch for a workflow deleted after its cells were computed;
      • a stale hour already marked dirty is left to its pending recompute;
      • every late end tied at a scan boundary is marked across batches of 3;
      • marking stops during a maintenance gap, and the next pass rebuilds;
      • a heavy hour is backed off while housekeeping still runs, is not retried before its backoff, and recovers;
      • a pass that fails before its reset commits leaves the next pass to rebuild;
      • a heartbeat that expires while a pass runs is not refreshed by it, so the next pass rebuilds.
    • The heavy-hour case uses a real failing read at every grain: a view gate that raises the 57014 statement timeout only inside 20s-bounded hour reads.
  • Red/green: each new guard reverted turns its case red:

    Guard reverted Red result
    Reset when either coverage end is null coverage case
    Read-time workflow resolution in the sampler 22 false mismatches
    Pending-mark exclusion 1
    Tie-inclusive scan boundary equivalence fails
    Marks gated on recent maintenance 131 marks written
    Maintenance-gap rebuild no reset
    Gap read from the stored heartbeat (not refreshed by the lease) no reset after a failed reset
    Heartbeat refreshed only while unexpired no reset after a mid-pass expiry
    Heavy-hour backoff the pass throws HeavyHourError
    No retry before the backoff attempts 2

    All 13 guard reverts from feat(usage): serve organization activity from an hourly rollup #10200 were re-run against this code and are still red.

  • Unit: cells.test.ts (6) covers summing slices, paging by workspace, the budget bound, retries starting at workspace pages, and no slicing on other errors. The retry case goes red when retries start from the span.

  • Existing tests: cleanup-logs, cleanup-logs-files, batch-delete and get-organization-activity pass.

  • Checks: check:migrations origin/staging, drizzle-kit generate (no diff), check:schema-mock, check:api-validation:strict, check:cron-parity, check:sql-date-binding, check:utils, check:file-names, check:test-patterns, check:explicit-any, check:boundaries, check:source-conventions, check:comment-hygiene. Full test, type-check and check:audits are left to CI.

Operator runbook (activity rollup, #10200 plus this PR)

Nothing below happens on merge.

0. Pre-flight (needs approval; run against a READ REPLICA only)

SET statement_timeout = '60s';
SELECT version();
-- logs → workflow FK must be validated ON DELETE SET NULL (convalidated = t, confdeltype = 'n').
SELECT conname, convalidated, confdeltype FROM pg_constraint
WHERE conname = 'workflow_execution_logs_workflow_id_workflow_id_fk';
-- the late-end index from 0463 is valid
SELECT indexrelid::regclass, indisvalid, pg_size_pretty(pg_relation_size(indexrelid))
FROM pg_index WHERE indexrelid = 'workflow_execution_logs_late_end_idx'::regclass;
-- cardinality for the busiest recent day: runs vs rollup cells per hour
SELECT date_trunc('hour', started_at) AS hour, count(*) AS runs,
  count(DISTINCT (workspace_id, coalesce(workflow_id, ''), trigger)) AS cells
FROM workflow_execution_logs
WHERE started_at >= '<busy-day> 00:00' AND started_at < '<busy-day> 00:00'::timestamp + interval '1 day'
GROUP BY 1 ORDER BY 1;
SELECT min(started_at) FROM workflow_execution_logs;
SELECT reltuples::bigint AS approx_runs FROM pg_class WHERE relname = 'workflow_execution_logs';
SELECT count(*) FILTER (WHERE ended_at >= started_at + interval '2 hours') AS late, count(*) AS sampled
FROM workflow_execution_logs TABLESAMPLE SYSTEM (0.1);

1. Enable maintenance

  • Hosted: set "activity-rollup-maintenance": { "enabled": true } in AppConfig feature-flags, then deploy sim-<env>-fast.
  • Self-hosted: set ACTIVITY_ROLLUP_MAINTENANCE=true.
  • One activity-rollup run is enqueued every 2 minutes. The first pass resets (reset: "coverage") and starts backfilling newest to oldest.

2. Watch the backfill, its signals, and the thresholds

SELECT version, coverage_start, fresh_through, backfill_completed_at, last_end_scan,
  maintained_at, lease_owner, housekeeping_at, updated_at
FROM activity_rollup_state;
SELECT bucket_start, attempts, retry_at FROM activity_rollup_blocked ORDER BY bucket_start;
SELECT count(*) AS dirty FROM activity_rollup_dirty;
SELECT count(*) AS inflight, max(attempts) FROM activity_rollup_inflight;
SELECT pg_size_pretty(pg_total_relation_size('workflow_activity_hourly'));

Every pass logs Activity rollup pass finished, with:

  • reset, blockedHours, finalizedHours, backfilledHours;
  • lateEndMarks, dirtyRecomputed, inflightRechecked, samplerMismatches, collectedWorkspaces;
  • replicaLagSeconds, coverageStart, freshThrough, backfillComplete.
Signal Healthy Act when
activity-rollup task duration < 60s (budget 40s) repeatedly hits maxDuration (5 min), or 3+ failed passes in a row
reset null (except the first pass, or after a gap/version bump) non-null with no reset you started: a maintenance gap > 6h or coverage cleared
freshThrough 3–4h behind now > 5h behind (reads fall back to legacy at 6h)
blockedHours / Activity rollup hour too heavy to read; backing off 0 any: those hours (and progress past them in their direction) wait; check replica load
maintained_at within the last few minutes older than 6h means the next pass rebuilds
Activity rollup hour read in slices / slice read by workspace rare on most hours
replicaLagSeconds < 1s null with Activity rollup skipped replica reads for lag on most passes
dirty count drains to ~0 each pass grows across passes
inflight count roughly paused/running runs older than 3h grows without bound
samplerMismatches 0 sustained non-zero: turn reads off and investigate
Failed to mark activity rollup cells for recomputing, Activity rollup sampler failed, workspace collection failed absent any occurrence
Backfill speed ~20k source rows/s replica lag/CPU climbs: lower ACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND

3. Enable reads: internal org, then the large org, then globally

Wait until backfill_completed_at is set (at minimum, until coverage_start is older than now − 93 days).

  1. Set "organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] } and deploy.
    • Compare the activity page (summary, every dimension and sort) with the flag off and on.
    • Expect small differences on old days after a retention delete: legacy serves cached settled days for up to 7 days (the segment cache's TTL), while the rollup reflects the delete as soon as its mark is processed and matches the uncached legacy SQL.
    • Organization activity rollup read failed must be absent. Leave it for a day.
  2. Add the large org's id. Watch p95 latency on both routes, replica CPU and cancellations, and Organization activity chat days read.
  3. Set "enabled": true.

4. Rollback and reset

  • Reads (instant, ~30s AppConfig cache): remove the org from orgIds, or disable organization-activity-rollup-reads. Always do this first.
  • Maintenance: disable activity-rollup-maintenance. Passes stop, and writers stop marking 6h after the last completed pass. When maintenance is re-enabled after more than 6h, the first pass rebuilds automatically (reset: "maintenance-gap"). Within 6h it simply resumes.
  • Force a rebuild (needs approval; primary): with reads off, run this single statement:
    UPDATE activity_rollup_state SET coverage_start = NULL;
    • Reads fall back at once, since they require both coverage ends.
    • The next pass (reset: "coverage") clears all progress and queues, truncates the cells and backfills again, under its lease.
    • It is safe to run while a pass holds the lease: that pass's next hour write rolls back.
    • No code-level alternative is needed. Bumping ACTIVITY_ROLLUP_VERSION takes the same path on deploy (reset: "version").
  • Unblock a heavy hour early (needs approval): UPDATE activity_rollup_blocked SET retry_at = now();
  • Removal: with both flags off the tables are inert. Dropping them is a later contract migration.

@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 19, 2026 7:08am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files

This PR changes payment, billing, or quota logic. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic

Comment thread packages/db/schema.ts Resolved
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge; no new blocking issues were found.

Summary

This PR hardens activity rollup maintenance so resets rebuild stored totals and heavy hours retry without losing progress.

  • Rollups rebuild after coverage clears or maintenance pauses.
  • Heavy hours page through workspaces and retry with backoff.
  • Late workflow ends are scanned in bounded batches.
  • The rollup sampler matches the workflow rules used by activity reads.

No new actionable issues found. The earlier findings are addressed in the current code.

Reviews (3) · Last reviewed commit: "fix(usage): keep a heartbeat that expire..." · Reviewed by Greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 13 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

This branch was previously deployed

1 inactive deployment
Preview — 00732029 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant