Prevent warm queries from reading partially published input - #627
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warm queries could return one group while another accepted group was still queued, and counter readout treated an inactive series as an incomplete pane. This adds store-owned admission receipts and publication checks across Remote Write, workers, maintenance outputs and the complete query DAG.
Before this PR, a fast worker could publish a partial population or advance replay retention past a slower worker; branch-local checks could also mix publication revisions. After this PR, overlapping unpublished input forces fallback, accepted slow-worker outputs remain publishable, and finite drain can prove a known series has no input in a retained window. For example, the two-group regression holds one worker and rejects warm readout until both groups publish.
Queue capacity is reserved before admission. Outputs carry their actual first/last consumed revisions, corrections are combined before acknowledging a receipt, and catalog generation is checked before state mutation. Startup installs the catalog before recovery. Accepted-sample and materialized-output throughput counters are connected to successful admission/publication boundaries.
Verification: seven production compatibility process tests pass, including the complete counter/fallback matrix, ERP, CMS/CS TopK and repeated dashboards; 322 precompute tests and six admission tests pass. The Remote Write suite also covers a real two-worker partial-publication race. Performance measurement is not applicable to this correctness change.
Stacked on #616. The revision fence is currently global, so unrelated continuous updates can cause conservative fallback. Absence proof requires finite drain; live event-time completeness is not inferred. Receipts are in memory, and durable series identity migration remains separate. This does not add general multi-input maintenance transforms.