Repository navigation
fix(usage): page heavy rollup hours once over active workspaces - #10203
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
[Medium impact] The PR appears safe to merge. SummaryThis PR bounds heavy-hour reads, improves retry recovery, and keeps activity totals accurate when corrections fail.
The five previous threads are addressed in the current code. No new actionable issues were found. Reviews (5) · Last reviewed commit: "fix(usage): read only marked workspace-h..." · Reviewed by Greptile |
There was a problem hiding this comment.
All reported issues were addressed across 10 files
This PR changes payment, billing, or quota logic. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.
Turn on auto-fix | Re-trigger cubic
…e per aggregate, renew the heartbeat by the database clock
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
… gate reads on drained queues, and fix the summary straddle test
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
…ight cells dirty, and stamp corrections only when both queues drain
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Turn on auto-fix | Re-trigger cubic
Summary
This follows up #10201 with the remaining findings of its post-merge audit. Reads and the flags-off path are unchanged, and both flags still default off. Migration
0465addsfailed_passes(constant default) andcorrections_at(nullable) to the one-rowactivity_rollup_state, andattempts(constant default) to theactivity_rollup_dirtyqueue. Deployed code doesn't writeattempts, so its inserts get the default.[slice start, hour end), is read once, a page at a time. A retry pages the whole hour once.LIMIT 1. Only workspaces with runs there are aggregated, 100 per statement.VALUESlist driving a LATERAL range scan per workspace onworkflow_execution_logs_workspace_activity_level_idx, the wayorganizationWorkflowRunsreads. Cell recomputes and the sampler use the same path.EXPLAINon a local 1M-run, 5k-workspace copy:LIMIT 1probe per workspace on the activity index.VALUESscan driving index-only scans on the activity index.EXISTS. The planner turned it into a hashedstarted_at_idxrange scan of the whole slice, which is why it is a LATERALLIMIT 1.attempts = 0row withretry_at = now(), a warning-level log, and no backoff or error.maintained_atmoves as soon as a pass at this version gets past its reset decision. A step that fails later loses no information: marks persist until a recompute removes them, and the late-end scan resumes from its cursor.UPDATEagainst the database clock, the clock writers use to gate their marks. If the heartbeat has already expired by that clock, the pass rebuilds instead of renewing it, whatever the worker's clock says.failed_passes(reset by the next pass that completes) and logged asActivity rollup pass failedwithconsecutiveFailures.attemptsandretry_atare bumped in their own statement before paging starts. A retry that crashes or loses its lease therefore waits out its backoff rather than looping.maxDurationis 10 minutes, not 5. The runbook below is updated end to end.due_atout with backoff and countsattempts; an in-flight recheck backs offnext_check_atthe same way.ended_atthe late-end scan never sees, so the cell is read live until a recompute succeeds.corrections_atis set only when a pass gets through every due item of both correction queues. Item failures are isolated and don't count, but a queue cut short by the pass budget or truncated by its batch size does: that pass keeps the old timestamp.planRollupReadrequires it within 6h, as it doesfresh_through, and reads it in the same snapshot.failed_passesthreshold: that counter mixes in unrelated failures (housekeeping, backfill) and can't tell "the queues ran" from "the pass reached the end". A timestamp written only after the queues ran states exactly the property reads depend on. Any failure here degrades to the exact legacy read.failed_passesreaches 3, a deferred hour counts as a retry, so a pass that keeps crashing while paging it backs it off.local(h) - hwithlocal(last) - last, butlastwas an unparenthesizedh + interval '1 hour' - interval '1 microsecond'. The subtraction becamelocal(last) - h + 1h - 1µs, which marked every hour as a straddle.Plans (local, plain EXPLAIN)
Test evidence
Integration test:
organization-activity-rollup.integration.tsran against a throwaway local Postgres 17 (all migrations through 0465): 32/32 pass. New in this round:corrections_atunchanged;attempts = 0, and that pass pages it.failed_passes, which the next completed pass resets.retry_at20 minutes out.Red/green:
retry_atnot in the future)corrections_atonly when both queues finishedcorrections_atset only after both stepsEvery earlier guard in the code this touches was re-run and is still red:
Earlier guards: all 13 from feat(usage): serve organization activity from an hourly rollup #10200 and every guard from fix(usage): harden activity rollup maintenance #10201 were re-run against this code and are still red. The straddle guard is now meaningful, because stored cells are actually summed.
Plans: shown above. Run against a locally seeded 1M-run, 5k-workspace copy, plain
EXPLAIN.Unit:
cells.test.ts(6) covers summing slices, paging the rest of the hour once with a cursor, stopping before paging when the budget has run out, retries paging the whole hour, and no slicing on other errors.Checks:
check:migrations origin/staging,drizzle-kit generate(no diff),check:schema-mock;check:utils,check:test-patterns,check:comment-hygiene,check:file-names,check:explicit-any,check:sql-date-binding,check:boundaries.Full
test,type-checkandcheck:auditsare left to CI.Operator runbook (activity rollup: #10200, #10201 and this PR)
Nothing below happens on merge.
0. Pre-flight (needs approval; run against a READ REPLICA only)
1. Enable maintenance
"activity-rollup-maintenance": { "enabled": true }in AppConfigfeature-flags, then deploysim-<env>-fast.ACTIVITY_ROLLUP_MAINTENANCE=true.activity-rolluprun is enqueued every 2 minutes. The first pass resets (reset: "coverage") and starts backfilling newest to oldest.2. Watch the backfill, its signals, and the thresholds
Every pass logs
Activity rollup pass finished, with:reset,blockedHours,finalizedHours,backfilledHours;lateEndMarks,dirtyRecomputed,inflightRechecked,samplerMismatches,collectedWorkspaces;replicaLagSeconds,coverageStart,freshThrough,backfillComplete.activity-rolluptask durationmaxDuration(10 min)failed_passes/Activity rollup pass failed(consecutiveFailures)resetnull(except the first pass, or after a gap/version bump)freshThroughblockedHours/Activity rollup hour too heavy to read; backing offmaintained_atActivity rollup hour read in slices/hour read by workspace/hour left for paging by the next passreplicaLagSecondsnullwithActivity rollup skipped replica reads for lagon most passesdirtycountfailingdirty marks /Activity rollup dirty cell failed to recompute; backing off,in-flight cell failed …corrections_atActivity rollup pass failed)inflightcountsamplerMismatchesFailed to mark activity rollup cells for recomputing,Activity rollup sampler failed,workspace collection failedACTIVITY_ROLLUP_BACKFILL_ROWS_PER_SECOND3. Enable reads: internal org, then the large org, then globally
Wait until
backfill_completed_atis set (at minimum, untilcoverage_startis older than now − 93 days)."organization-activity-rollup-reads": { "orgIds": ["<internal org id>"] }and deploy.Organization activity rollup read failedmust be absent. Leave it for a day.Organization activity chat days read."enabled": true.4. Rollback and reset
orgIds, or disableorganization-activity-rollup-reads. Always do this first.activity-rollup-maintenance. Passes stop, and writers stop marking 6h after the last pass. When maintenance is re-enabled after more than 6h, the first pass rebuilds automatically (reset: "maintenance-gap"). Within 6h it simply resumes.reset: "coverage") clears all progress and queues, truncates the cells and backfills again, under its lease.ACTIVITY_ROLLUP_VERSIONtakes the same path on deploy (reset: "version").UPDATE activity_rollup_blocked SET retry_at = now();