Skip to content

feat(usage): serve organization activity from an hourly rollup - #10200

Merged
waleedlatif1 merged 4 commits into
stagingfrom
feat/usage-activity-rollup
Nov 19, 2026
Merged

waleedlatif1 merged 4 commits into
stagingfrom
feat/usage-activity-rollup

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Organization activity (/api/organizations/{id}/usage/activity/summary and .../breakdown) scans every workflow run in the window on each cold request, which is slow for large organizations and gets cancelled on the replica. This adds an hourly rollup of workflow runs that reproduces today's results exactly, behind two flags that default off. Merging changes no reads, starts no maintenance, and the migrations only add tables and one small partial index.

Rollup (workflow_activity_hourly): counters per UTC hour × workspace × workflow × trigger (runs, completed, failed, cancelled, in-flight, duration sum/count). There's no organization column and no foreign keys. Reads join workspace for the current organization and workflow for the current name, so moving a workspace or deleting a workflow needs no write.

Maintenance (lib/billing/activity-rollup/): one serialized pass per 2-minute cron tick (activity-rollup Trigger.dev task, concurrencyLimit: 1, or inline without Trigger.dev), under a lease on activity_rollup_state. Each pass:

  1. Marks cells of runs that ended late (ended_at >= started_at + 2h), read through the new partial index from the last scan minus a 5-minute margin.
  2. Computes the hours that have settled (hour end + 3h) on the replica, and writes each hour together with the fresh_through advance in one primary transaction.
  3. Recomputes dirty cells (retention deletes, reopened runs, late ends) and in-flight cells (backoff from 10 min, capped at 6h) on the primary, under per-hour transaction advisory locks.
  4. Backfills newest to oldest on the replica, skipping empty hours, paced to ACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND (default 20k) with at least 200ms between hours. Replica steps run only when the replica shows the lease this pass just committed within 10s. The pass's own commit is the heartbeat, so a quiet primary never reads as lag, and a disconnected WAL receiver always does.
  5. Hourly: garbage-collects cells of deleted workspaces and samples random workspace-days against the source, marking any hour that disagrees.

Every primary statement the pass runs is bounded (statement_timeout 15s, lock_timeout 2s), and replica hour reads are bounded at 20s.

Writes are diffs: changed rows are upserted, rows that vanished are deleted, and untouched rows are left alone. An hour too heavy for one statement (statement timeout or replica recovery conflict) is read in 15-minute slices and summed.

Reads (organization-activity-rollup-queries.ts): the state and the cells are read in one repeatable-read snapshot on the replica.

  • Whole hours in [coverage_start, fresh_through) come from the rollup. The window edges and the recent tail are read live, per org workspace, through LATERAL scans on workflow_execution_logs_workspace_activity_level_idx.
  • For the summary, an hour that crosses a local midnight or a UTC-offset change (only in non-whole-hour zones) is also read live, batched into one statement.
  • The breakdown is one statement in today's grouped → named → ORDER BY …, label, id LIMIT/OFFSET shape. Its average is sum(duration)::numeric / sum(count)::numeric, the same numeric_div that avg(int) performs, so it is bit-identical.
  • Chats stay live, as today. The summary's chat days keep the segment cache under an activity-chat: namespace, and their timing is logged per request so the Phase 2 chat rollup can be measured.
  • Reads fall back to the unchanged legacy path when:
    • the flag is off;
    • the version differs;
    • there is no coverage yet;
    • fresh_through is more than 6h old;
    • the window reaches before an unfinished backfill;
    • the window is longer than the legacy summary's 999-day segment cap;
    • the rollup read throws (this is logged).
  • In the summary, chats are read only once the snapshot shows the rollup can serve, in parallel with the workflow days.

Retention and change capture:

  • chunkedBatchDelete gains an afterDelete hook that runs in the DELETE's own transaction. cleanup-logs uses it to mark the deleted runs' cells dirty.
  • The HITL path that moves a finished run back to pending (clearing ended_at) now marks its cell in the same transaction. That update moved into lib/workflows/executor/reopen-execution-log.ts unchanged.
  • Marking runs in a savepoint and never throws, so retention and resumes behave exactly as before even if the marks table were missing.
  • Marking writes nothing until the rollup has run once, so deployments that never enable it accumulate no marks.

Migrations (expand only):

  • 0462_workflow_activity_rollup: four new tables, their indexes, and table-level autovacuum settings. There's no change to an existing table and no trigger.
  • 0463_workflow_execution_logs_late_end_index: CREATE INDEX CONCURRENTLY of the partial late-end index on workflow_execution_logs, using the replay-safe pattern from 0446/0461. Applied and replayed against a local database.

Flags (via the add-feature-flag skill):

  • organization-activity-rollup-reads: org-targeted through AppConfig orgIds. The env fallback is ORGANIZATION_ACTIVITY_ROLLUP_READS.
  • activity-rollup-maintenance: global. The env fallback is ACTIVITY_ROLLUP_MAINTENANCE. The cron's due check and the task both check it.

The cron is added in helm/sim/values.yaml, docker/crontab and the self-hosting background-jobs docs (check:cron-parity). The route is added to the API validation allowlist.

Where this deviates from the design (and why)

  • No history_floor. The backfill finds the next populated hour below the coverage with max(started_at) < coverage_start (an index seek on started_at), skips empty hours in one step, and completes when none is left.
  • Dirty marks are keyed by (workspace_id, bucket_start), with no source/scope_id. The chat rollup is deferred (YAGNI, as in addendum K).
  • The settle lag S is a code constant (3h), not an env knob. It is bound to the late-end index predicate (2h) plus an hour of commit and replica margin, and that relationship is documented on both sides. The only env knob is the backfill row rate.
  • Marks wait until their hour is covered. A mark for an hour not yet finalized or backfilled is kept rather than deleted. Those hours are read on the replica, and a mark consumed before a stale replica write would lose the change. Marks below the coverage are dropped only once the backfill is complete.
  • Retention marking (addendum C). Marks are written in the delete's own transaction (the afterDelete hook, in a savepoint that logs and never throws). Retention deletes the same rows as before even if the marks table were missing mid-deploy. The insert is gated on the state row existing, so a deployment that never enables the rollup writes no marks; it still pays the hook's transaction and savepoint per 500-row batch.
  • HITL reopen (B.a). The update that moves a finished run back to pending moved verbatim into reopen-execution-log.ts, so it marks in the same transaction and is testable without the HITL manager.
  • Straddle hours. Beyond a crossed local midnight, an hour is also read live when the UTC offset changes inside it, which covers transitions not aligned to the hour (e.g. Lord Howe).
  • Coverage after the backfill. Once the backfill is complete, instants before the coverage are read live (an empty index probe) rather than assumed empty, so a late insert older than the coverage still counts.
  • Long summaries. Summary windows of 999 days or more go to legacy, which cuts a window into at most 1000 days (USAGE_WINDOW_MAX_DAYS, now exported for this).
  • Member breakdown is chat-only, so it is served by the legacy query unchanged.
  • Rollup read errors are logged and fall back to legacy.
  • workflow_activity_hourly_read_idx covers every counter (cancelled and in_flight added), so stored-cell reads are index-only.
  • Equivalence relies on the logs → workflow FK being valid with ON DELETE SET NULL (it is in the migrations). A stored cell's workflow is resolved at read time, so a deleted workflow reads as deleted:<workspace>, exactly as its nulled runs do. The pre-flight checks the FK.
  • Pre-existing legacy tie order. The breakdown's ORDER BY …, label, id is not total when one workflow id has runs in two workspaces (a moved workflow): those two rows tie, and both paths order them arbitrarily. Rows and counts are identical, but their relative order is plan-dependent in legacy too. Both paths keep it unchanged; the test seeds each workflow in one workspace.
  • Plans (plain EXPLAIN on a local 1M-run, 2k-workspace dataset).
    • Live edges and straddles are per-workspace index-only scans on the activity index.
    • Workflow resolution is an index-only lookup on workflow_pkey.
    • For stored cells the planner is cost-based. On that dataset it chose a bucket_start range plus a hash join with the org's workspaces; the per-workspace read_idx path is available as the deployment grows.
  • Signals are structured log lines (Activity rollup pass finished and the warnings in the runbook), not new CloudWatch metrics.
  • Deliberately not done.
    • Batching dirty and in-flight recomputes per hour: each cell is one short primary transaction, and the queues are small.
    • Table-level change capture through a new column on workflow_execution_logs: that is a hot-table change, and the late-end index plus in-flight rechecks already cover non-delete changes.

Test evidence

  • Integration test: organization-activity-rollup.integration.ts ran against a throwaway local Postgres 17 (all migrations applied, 0463 replayed): 15/15 passing in ~60s.
    • Data: a seeded PRNG generates 2.2k runs over 460 days. It covers every status, null/-1/0/random durations, :00:00.000000 and :59:59.999999 starts, the US, AU, Lord Howe, Chatham and Santiago DST changes, a foreign org, an archived workspace, and chats.
    • Matrix: UTC, LA, Kolkata, Kathmandu, Lord_Howe, Chatham and Santiago × custom 92d, trailing 7d/30d, a mid-hour period, a 120-day period (weeks), a 450-day period (months), workspace-scoped, and another org.
    • Assertions: every summary and every breakdown dimension × sort × page deep-equals the uncached legacy SQL.
  • Red/green: with each guard reverted, its case fails:
Guard reverted Failing case
Read-time workflow lookup workflow deleted after cells computed
Org join on current workspace org workspace moved to another org
Retention afterDelete marks retention delete
Late-end scan late end of an old run; late insert into a computed hour
Reopen mark reopened run without an end
In-flight recheck unfinished run that finishes without a late end
Pass lease two concurrent passes (both ran)
fresh_through bound on the rollup span hours after fresh_through served live
Coverage check window reaching before an unfinished backfill
Version check cells of another version
Coverage advanced in the write transaction write failure after the replica read (coverage claimed the lost hour)
Straddle hours summary matrix in Asia/Kolkata
  • Unit: cells.test.ts covers heavy-hour slicing on statement timeout and on replica recovery conflict, plus no slicing on other errors. It goes red with the slice fallback removed.
  • Existing tests: cleanup-logs, cleanup-logs-files, batch-delete, human-in-the-loop-manager, pause-persistence, feature-flags, get-organization-activity and use-cases pass.
  • Checks: check:migrations origin/staging, drizzle-kit generate (no diff), generate-schema-mock --check, check:api-validation:strict, check:cron-parity, check:sql-date-binding, check:utils, check:file-names, check:test-patterns, check:explicit-any, check:boundaries, check:source-conventions, check:comment-hygiene.
  • Full test, type-check and check:audits are left to CI.

Operator runbook

Nothing below happens on merge. Each step is a separate, deliberate operation.

0. Pre-flight (needs approval; run against a READ REPLICA only, with a statement timeout)

SET statement_timeout = '60s';

-- Postgres version (the straddle reads use the database's tzdata, as the legacy reads do).
SELECT version();

-- The logs → workflow FK must be validated ON DELETE SET NULL. A deleted workflow's runs then read as
-- 'deleted:<workspace>' from both paths. Expect convalidated = t, confdeltype = 'n'.
SELECT conname, convalidated, confdeltype FROM pg_constraint
WHERE conname = 'workflow_execution_logs_workflow_id_workflow_id_fk';

-- The 0463 index built and is valid.
SELECT indexrelid::regclass, indisvalid, pg_size_pretty(pg_relation_size(indexrelid))
FROM pg_index WHERE indexrelid = 'workflow_execution_logs_late_end_idx'::regclass;

-- Cardinality for one busy day (pick the busiest recent day): runs vs rollup cells per hour.
-- Rollup rows for that day ≈ sum(cells). Expect cells to be a small fraction of runs.
SELECT date_trunc('hour', started_at) AS hour, count(*) AS runs,
  count(DISTINCT (workspace_id, coalesce(workflow_id, ''), trigger)) AS cells
FROM workflow_execution_logs
WHERE started_at >= '<busy-day> 00:00' AND started_at < '<busy-day> 00:00'::timestamp + interval '1 day'
GROUP BY 1 ORDER BY 1;

-- History depth (index seek) and table size, to size the backfill.
SELECT min(started_at) FROM workflow_execution_logs;
SELECT reltuples::bigint AS approx_runs FROM pg_class WHERE relname = 'workflow_execution_logs';

-- Share of runs that end 2h+ after they start (what the late-end scan reads), sampled.
SELECT count(*) FILTER (WHERE ended_at >= started_at + interval '2 hours') AS late,
  count(*) AS sampled
FROM workflow_execution_logs TABLESAMPLE SYSTEM (0.1);

1. Enable maintenance

  • Hosted: in the AppConfig feature-flags document, set "activity-rollup-maintenance": { "enabled": true }, then start a sim-<env>-fast deployment.
  • Self-hosted: set ACTIVITY_ROLLUP_MAINTENANCE=true.
  • The cron then enqueues one activity-rollup task run every 2 minutes. The first pass:
    • creates the state row;
    • sets coverage_start = fresh_through = floor(now) - 3h;
    • starts backfilling backwards.

2. Watch the backfill, its signals, and the thresholds

Progress (primary or replica):

SELECT version, coverage_start, fresh_through, backfill_completed_at, last_end_scan,
  lease_owner, lease_expires_at, housekeeping_at, updated_at
FROM activity_rollup_state;
SELECT count(*) AS dirty FROM activity_rollup_dirty;
SELECT count(*) AS inflight, max(attempts) FROM activity_rollup_inflight;
SELECT pg_size_pretty(pg_total_relation_size('workflow_activity_hourly'));

Logs. Every pass writes Activity rollup pass finished with these fields:

  • finalizedHours, backfilledHours, lateEndMarks, dirtyRecomputed, inflightRechecked, samplerMismatches, collectedWorkspaces;
  • replicaLagSeconds, coverageStart, freshThrough, backfillComplete.

Thresholds:

Signal Healthy Act when
activity-rollup task duration < 60s (budget is 40s) repeatedly hits maxDuration (5 min), or fails 3+ passes in a row
freshThrough 3–4h behind now > 5h behind (reads fall back to legacy at 6h)
Backfill speed ~20k source rows/s while running replica lag or CPU climbs: lower ACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND, or turn maintenance off
replicaLagSeconds (time for the replica to show this pass's lease) < 1s null with Activity rollup skipped replica reads for lag on most passes
Activity rollup hour read in slices rare on most hours (raise the hour budget before reads go on)
dirty count drains to ~0 each pass grows across passes
inflight count roughly the paused/running runs older than 3h grows without bound
samplerMismatches 0 any sustained non-zero: turn reads off and investigate (a missed change path)
Failed to mark activity rollup cells for recomputing absent any occurrence
Primary write latency, lock waits on activity_rollup advisory locks unchanged any regression

3. Enable reads: internal org, then the large org, then globally

Wait until backfill_completed_at is set. At minimum, coverage_start must be older than now − 93 days, or every window reaching past it still falls back to legacy.

  1. Internal org. Set "organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] } and deploy sim-<env>-fast.
    • Open the activity page (summary and every breakdown dimension and sort) and compare it with what it showed with the flag off.
    • Watch request latency on both routes.
    • Organization activity rollup read failed must be absent.
    • Leave it for a day.
  2. The large org. Add the large org's id to orgIds. Watch p95 latency on both routes, replica CPU, and the absence of replica cancellations on these routes. Organization activity chat days read logs the chat-branch time per request, to decide on the Phase 2 chat rollup.
  3. Globally. Set "enabled": true, keep orgIds, and deploy.

4. Rollback

  • Reads (instant, within the ~30s AppConfig cache): remove the org from orgIds, or set "enabled": false, for organization-activity-rollup-reads. Every request goes back to the legacy path. Always do this first.
  • Maintenance: set "activity-rollup-maintenance": { "enabled": false }. The cron starts no passes and a queued pass exits immediately.
    • With reads still on, they keep using the rollup plus a growing live tail, and fall back to legacy on their own once fresh_through is 6h old.
    • Turning maintenance back on resumes from the stored state. It catches up hours and late changes, because the late-end scan resumes from last_end_scan.
  • Force legacy without a deploy (needs approval; primary): UPDATE activity_rollup_state SET coverage_start = NULL, fresh_through = NULL; Reads fall back at once. The next pass truncates the cells and rebuilds from scratch.
  • Code-level: bump ACTIVITY_ROLLUP_VERSION. Reads on the new version ignore the old cells, and the new pass resets and rebuilds.
  • Removal: the tables are inert when both flags are off. Dropping them and the 0463 index would be a later contract migration.

@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 19, 2026 5:27am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 33 files

This PR changes a database schema or migrates data. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/background/cleanup-logs.ts Resolved
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The changes since the previous review appear safe to merge.

Summary

Adds an opt-in hourly workflow activity rollup, with live reads for recent runs and a fallback to the existing queries. The follow-up replaces the replica's last-write age with a check for the current pass's lease.

  • Organization activity uses hourly counts for covered workflow history.
  • A maintenance pass builds and refreshes hourly workflow-activity counts.
  • Retention and reopened runs flag their hourly cells for refresh.
  • Operators can schedule the rollup and enable its reads and maintenance separately.

No new actionable issues found in the changes since the previous review.

Reviews (4) · Last reviewed commit: "fix(usage): measure replica lag with the..." · Reviewed by Greptile

Comment thread apps/sim/background/cleanup-logs.ts Resolved
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 34 files

Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/background/cleanup-logs.ts Resolved
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 34 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 34 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

This branch was previously deployed

1 inactive deployment
Preview — 42303bf8 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant