Repository navigation
fix(usage): harden activity rollup maintenance - #10201
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
There was a problem hiding this comment.
All reported issues were addressed across 12 files
This PR changes payment, billing, or quota logic. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.
Turn on auto-fix | Re-trigger cubic
[Medium impact] The reviewed changes appear safe to merge; no new blocking issues were found. SummaryThis PR hardens activity rollup maintenance so resets rebuild stored totals and heavy hours retry without losing progress.
No new actionable issues found. The earlier findings are addressed in the current code. Reviews (3) · Last reviewed commit: "fix(usage): keep a heartbeat that expire..." · Reviewed by Greptile |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 12 files
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
…y-hour retry budgets
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Summary
This hardens the activity rollup maintenance from #10200 against the six operational findings of its post-merge audit. Reads and the flags-off path are unchanged: both flags still default off, so merging this changes nothing until maintenance is enabled. Migration
0464adds a newactivity_rollup_blockedtable and one nullablemaintained_atcolumn on the one-rowactivity_rollup_state. Nothing deployed writes either.resetRollup): it clears coverage, the backfill completion, the late-end cursor and any blocked hour, empties the dirty and in-flight queues, truncates the cells and backfills again.coverage_start = NULL) rebuilds even after a completed backfill. Before, the completed backfill stayed set, the backfill never reran, and reads served nearly the whole window live.fresh_throughset withcoverage_startNULL. Every hour write now moves progress only while both ends are set (otherwise the hour's transaction rolls back), and a pass that finds either end missing rebuilds.(SELECT f.id FROM workflow f WHERE f.id = nullif(h.workflow_id, '')), and re-summed. A workflow deleted after its cells were computed matches its nulled source rows.ON CONFLICT DO NOTHING).activity_rollup_blocked(one row per hour), with backoff from 10 minutes doubling to a cap of 6 hours and a loudActivity rollup hour too heavy to read; backing offerror.maxDurationraised to 10 minutes. The block is deleted in the same transaction that finally writes the hour. An hour that still cannot be paged within the cap stays blocked behind a loud error for an operator, rather than retrying forever.ended_atorder, 5,000 per statement, up to a horizon 5 minutes behind now (commits stamped earlier than that are visible by then). Each statement marks every end up to its batch's last one, including ties beyond the batch, and moveslast_end_scanthere in the same statement.activity_rollup_state.maintained_at). Only a completed pass, or the start of a new coverage, writes that column. Taking the lease doesn't, so a failing pass, or one idling on a newer version, does not keep marks flowing.maintained_atmore than 6h old (or missing) starts over. The reason is read from the stored state alone, so if a pass fails before its reset commits, the next pass rebuilds for the same reason.maintained_atonly if the previous value was still within the gap when the pass ends. If it expired mid-pass, writers had already stopped marking, so it stays expired and the next pass rebuilds.fresh_throughis fresh, possibly before the backlog drains. Resuming after a long gap also leaves the late-end cursor far behind.EXISTSon the one-row state table inside the existing savepoint), and it bounds the marks table to at most 6h of writes.maintained_at. Reads fall back to legacy meanwhile, and recent windows return first because the backfill runs newest to oldest.Simplify/cleanup notes:
updateLeasedState.LEFT JOINinstead of the scalar workflow lookup (that lookup mirrors the read path exactly).Test evidence
Integration test:
organization-activity-rollup.integration.tsran against a throwaway local Postgres 17 (all migrations through 0464): 23/23 pass in about 80s.57014statement timeout only inside 20s-bounded hour reads.Red/green: each new guard reverted turns its case red:
HeavyHourErrorAll 13 guard reverts from feat(usage): serve organization activity from an hourly rollup #10200 were re-run against this code and are still red.
Unit:
cells.test.ts(6) covers summing slices, paging by workspace, the budget bound, retries starting at workspace pages, and no slicing on other errors. The retry case goes red when retries start from the span.Existing tests:
cleanup-logs,cleanup-logs-files,batch-deleteandget-organization-activitypass.Checks:
check:migrations origin/staging,drizzle-kit generate(no diff),check:schema-mock,check:api-validation:strict,check:cron-parity,check:sql-date-binding,check:utils,check:file-names,check:test-patterns,check:explicit-any,check:boundaries,check:source-conventions,check:comment-hygiene. Fulltest,type-checkandcheck:auditsare left to CI.Operator runbook (activity rollup, #10200 plus this PR)
Nothing below happens on merge.
0. Pre-flight (needs approval; run against a READ REPLICA only)
1. Enable maintenance
"activity-rollup-maintenance": { "enabled": true }in AppConfigfeature-flags, then deploysim-<env>-fast.ACTIVITY_ROLLUP_MAINTENANCE=true.activity-rolluprun is enqueued every 2 minutes. The first pass resets (reset: "coverage") and starts backfilling newest to oldest.2. Watch the backfill, its signals, and the thresholds
Every pass logs
Activity rollup pass finished, with:reset,blockedHours,finalizedHours,backfilledHours;lateEndMarks,dirtyRecomputed,inflightRechecked,samplerMismatches,collectedWorkspaces;replicaLagSeconds,coverageStart,freshThrough,backfillComplete.activity-rolluptask durationmaxDuration(5 min), or 3+ failed passes in a rowresetnull(except the first pass, or after a gap/version bump)freshThroughblockedHours/Activity rollup hour too heavy to read; backing offmaintained_atActivity rollup hour read in slices/slice read by workspacereplicaLagSecondsnullwithActivity rollup skipped replica reads for lagon most passesdirtycountinflightcountsamplerMismatchesFailed to mark activity rollup cells for recomputing,Activity rollup sampler failed,workspace collection failedACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND3. Enable reads: internal org, then the large org, then globally
Wait until
backfill_completed_atis set (at minimum, untilcoverage_startis older than now − 93 days)."organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] }and deploy.Organization activity rollup read failedmust be absent. Leave it for a day.Organization activity chat days read."enabled": true.4. Rollback and reset
orgIds, or disableorganization-activity-rollup-reads. Always do this first.activity-rollup-maintenance. Passes stop, and writers stop marking 6h after the last completed pass. When maintenance is re-enabled after more than 6h, the first pass rebuilds automatically (reset: "maintenance-gap"). Within 6h it simply resumes.reset: "coverage") clears all progress and queues, truncates the cells and backfills again, under its lease.ACTIVITY_ROLLUP_VERSIONtakes the same path on deploy (reset: "version").UPDATE activity_rollup_blocked SET retry_at = now();