Repository navigation
feat(usage): serve organization activity from an hourly rollup - #10200
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
There was a problem hiding this comment.
All reported issues were addressed across 33 files
This PR changes a database schema or migrates data. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.
Turn on auto-fix | Re-trigger cubic
[Medium impact] The changes since the previous review appear safe to merge. SummaryAdds an opt-in hourly workflow activity rollup, with live reads for recent runs and a fallback to the existing queries. The follow-up replaces the replica's last-write age with a check for the current pass's lease.
No new actionable issues found in the changes since the previous review. Reviews (4) · Last reviewed commit: "fix(usage): measure replica lag with the..." · Reviewed by Greptile |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 34 files
Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.
Turn on auto-fix | Re-trigger cubic
…ats, bounded housekeeping
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Summary
Organization activity (
/api/organizations/{id}/usage/activity/summaryand.../breakdown) scans every workflow run in the window on each cold request, which is slow for large organizations and gets cancelled on the replica. This adds an hourly rollup of workflow runs that reproduces today's results exactly, behind two flags that default off. Merging changes no reads, starts no maintenance, and the migrations only add tables and one small partial index.Rollup (
workflow_activity_hourly): counters per UTC hour × workspace × workflow × trigger (runs, completed, failed, cancelled, in-flight, duration sum/count). There's no organization column and no foreign keys. Reads joinworkspacefor the current organization andworkflowfor the current name, so moving a workspace or deleting a workflow needs no write.Maintenance (
lib/billing/activity-rollup/): one serialized pass per 2-minute cron tick (activity-rollupTrigger.dev task,concurrencyLimit: 1, or inline without Trigger.dev), under a lease onactivity_rollup_state. Each pass:ended_at >= started_at + 2h), read through the new partial index from the last scan minus a 5-minute margin.fresh_throughadvance in one primary transaction.ACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND(default 20k) with at least 200ms between hours. Replica steps run only when the replica shows the lease this pass just committed within 10s. The pass's own commit is the heartbeat, so a quiet primary never reads as lag, and a disconnected WAL receiver always does.Every primary statement the pass runs is bounded (
statement_timeout15s,lock_timeout2s), and replica hour reads are bounded at 20s.Writes are diffs: changed rows are upserted, rows that vanished are deleted, and untouched rows are left alone. An hour too heavy for one statement (statement timeout or replica recovery conflict) is read in 15-minute slices and summed.
Reads (
organization-activity-rollup-queries.ts): the state and the cells are read in one repeatable-read snapshot on the replica.[coverage_start, fresh_through)come from the rollup. The window edges and the recent tail are read live, per org workspace, through LATERAL scans onworkflow_execution_logs_workspace_activity_level_idx.ORDER BY …, label, id LIMIT/OFFSETshape. Its average issum(duration)::numeric / sum(count)::numeric, the samenumeric_divthatavg(int)performs, so it is bit-identical.activity-chat:namespace, and their timing is logged per request so the Phase 2 chat rollup can be measured.fresh_throughis more than 6h old;Retention and change capture:
chunkedBatchDeletegains anafterDeletehook that runs in the DELETE's own transaction.cleanup-logsuses it to mark the deleted runs' cells dirty.pending(clearingended_at) now marks its cell in the same transaction. That update moved intolib/workflows/executor/reopen-execution-log.tsunchanged.Migrations (expand only):
0462_workflow_activity_rollup: four new tables, their indexes, and table-level autovacuum settings. There's no change to an existing table and no trigger.0463_workflow_execution_logs_late_end_index:CREATE INDEX CONCURRENTLYof the partial late-end index onworkflow_execution_logs, using the replay-safe pattern from 0446/0461. Applied and replayed against a local database.Flags (via the add-feature-flag skill):
organization-activity-rollup-reads: org-targeted through AppConfigorgIds. The env fallback isORGANIZATION_ACTIVITY_ROLLUP_READS.activity-rollup-maintenance: global. The env fallback isACTIVITY_ROLLUP_MAINTENANCE. The cron'sduecheck and the task both check it.The cron is added in
helm/sim/values.yaml,docker/crontaband the self-hosting background-jobs docs (check:cron-parity). The route is added to the API validation allowlist.Where this deviates from the design (and why)
history_floor. The backfill finds the next populated hour below the coverage withmax(started_at) < coverage_start(an index seek onstarted_at), skips empty hours in one step, and completes when none is left.(workspace_id, bucket_start), with nosource/scope_id. The chat rollup is deferred (YAGNI, as in addendum K).afterDeletehook, in a savepoint that logs and never throws). Retention deletes the same rows as before even if the marks table were missing mid-deploy. The insert is gated on the state row existing, so a deployment that never enables the rollup writes no marks; it still pays the hook's transaction and savepoint per 500-row batch.pendingmoved verbatim intoreopen-execution-log.ts, so it marks in the same transaction and is testable without the HITL manager.USAGE_WINDOW_MAX_DAYS, now exported for this).workflow_activity_hourly_read_idxcovers every counter (cancelledandin_flightadded), so stored-cell reads are index-only.ON DELETE SET NULL(it is in the migrations). A stored cell's workflow is resolved at read time, so a deleted workflow reads asdeleted:<workspace>, exactly as its nulled runs do. The pre-flight checks the FK.ORDER BY …, label, idis not total when one workflow id has runs in two workspaces (a moved workflow): those two rows tie, and both paths order them arbitrarily. Rows and counts are identical, but their relative order is plan-dependent in legacy too. Both paths keep it unchanged; the test seeds each workflow in one workspace.workflow_pkey.bucket_startrange plus a hash join with the org's workspaces; the per-workspaceread_idxpath is available as the deployment grows.Activity rollup pass finishedand the warnings in the runbook), not new CloudWatch metrics.workflow_execution_logs: that is a hot-table change, and the late-end index plus in-flight rechecks already cover non-delete changes.Test evidence
organization-activity-rollup.integration.tsran against a throwaway local Postgres 17 (all migrations applied, 0463 replayed): 15/15 passing in ~60s.:00:00.000000and:59:59.999999starts, the US, AU, Lord Howe, Chatham and Santiago DST changes, a foreign org, an archived workspace, and chats.afterDeletemarksfresh_throughbound on the rollup spanfresh_throughserved livecells.test.tscovers heavy-hour slicing on statement timeout and on replica recovery conflict, plus no slicing on other errors. It goes red with the slice fallback removed.cleanup-logs,cleanup-logs-files,batch-delete,human-in-the-loop-manager,pause-persistence,feature-flags,get-organization-activityanduse-casespass.check:migrations origin/staging,drizzle-kit generate(no diff),generate-schema-mock --check,check:api-validation:strict,check:cron-parity,check:sql-date-binding,check:utils,check:file-names,check:test-patterns,check:explicit-any,check:boundaries,check:source-conventions,check:comment-hygiene.test,type-checkandcheck:auditsare left to CI.Operator runbook
Nothing below happens on merge. Each step is a separate, deliberate operation.
0. Pre-flight (needs approval; run against a READ REPLICA only, with a statement timeout)
1. Enable maintenance
feature-flagsdocument, set"activity-rollup-maintenance": { "enabled": true }, then start asim-<env>-fastdeployment.ACTIVITY_ROLLUP_MAINTENANCE=true.activity-rolluptask run every 2 minutes. The first pass:coverage_start = fresh_through = floor(now) - 3h;2. Watch the backfill, its signals, and the thresholds
Progress (primary or replica):
Logs. Every pass writes
Activity rollup pass finishedwith these fields:finalizedHours,backfilledHours,lateEndMarks,dirtyRecomputed,inflightRechecked,samplerMismatches,collectedWorkspaces;replicaLagSeconds,coverageStart,freshThrough,backfillComplete.Thresholds:
activity-rolluptask durationmaxDuration(5 min), or fails 3+ passes in a rowfreshThroughACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND, or turn maintenance offreplicaLagSeconds(time for the replica to show this pass's lease)nullwithActivity rollup skipped replica reads for lagon most passesActivity rollup hour read in slicesdirtycountinflightcountsamplerMismatchesFailed to mark activity rollup cells for recomputingactivity_rollupadvisory locks3. Enable reads: internal org, then the large org, then globally
Wait until
backfill_completed_atis set. At minimum,coverage_startmust be older than now − 93 days, or every window reaching past it still falls back to legacy."organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] }and deploysim-<env>-fast.Organization activity rollup read failedmust be absent.orgIds. Watch p95 latency on both routes, replica CPU, and the absence of replica cancellations on these routes.Organization activity chat days readlogs the chat-branch time per request, to decide on the Phase 2 chat rollup."enabled": true, keeporgIds, and deploy.4. Rollback
orgIds, or set"enabled": false, fororganization-activity-rollup-reads. Every request goes back to the legacy path. Always do this first."activity-rollup-maintenance": { "enabled": false }. The cron starts no passes and a queued pass exits immediately.fresh_throughis 6h old.last_end_scan.UPDATE activity_rollup_state SET coverage_start = NULL, fresh_through = NULL;Reads fall back at once. The next pass truncates the cells and rebuilds from scratch.ACTIVITY_ROLLUP_VERSION. Reads on the new version ignore the old cells, and the new pass resets and rebuilds.