feat: add maintained population and temporal aggregate candidates - #404
Merged
Merged
Conversation
* refactor(types): split ExactKind::MinMax into Min and Max `ExactKind::MinMax` was never a min/max pair: `function_rules` maps `AggIntent::Max` onto it and `AggIntent::Min` onto the separately added `ExactKind::Min`, so the name promised a two-sided accumulator that the variant never was. Consumers that match on the family had to carry the direction out-of-band (ASAPQuery-backend threads it through the `aggregationSubType` wire string) because the name gave no guarantee about which extremum the state holds. Rename the variant to `Max` in both `ExactKind` and `ExactParams`, so the two directions are symmetric variants and a stored minimum can no longer content-address onto a maximum. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(promql): restore a distinct-count idiom over mergeable cardinality Lowering `count(v)` to a row count (15e6cf8) matched PromQL, but it left no way to ask for a distinct count that reduces across series. The `Cardinality` intent survived only under `distinct_over_time(v[w])`, which is per-series: a consumer holding one mergeable cardinality state per source could answer the merged question and had no expression that asked it. Collapse `count(distinct_over_time(v[w]))` into a single `Aggregate{[Cardinality]}` reduced by the outer `by`. The nested reading -- count the series that have a distinct-count -- is not what anyone writes this expression for, and it discards the mergeability of the state underneath. The collapsed form needs its own reduction rule: `reduction_for` reads the surviving `TimeRange` as one row per series and answers `PerEntity`, which is right for a bare `distinct_over_time(v[w])` and wrong once an explicit aggregator wraps it. `windowed_reduce` keeps the outer aggregator's grouping, so an empty `by` reduces everything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(promql): preserve count over per-series cardinality --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Backend issues ProjectASAP/ASAPQuery-backend#701 and ProjectASAP/ASAPQuery-backend#702 require candidate DAGs that preserve aggregate populations and expression accuracy.
What
Add language-independent maintained-population rules and typed readouts, distinct minimum state, realized temporal average, exact TopK over temporal values, and guarded relative division.
A maintained population is the multiset of qualifying records represented by state retained across query evaluations and updated as members enter, change, leave or expire. A readout computes an aggregate from that state. For example, two current series with values
7and7are two members; updating the first to9changes the population to{9, 7}, not{7, 7, 9}. SQL table rows retain row multiplicity; PromQL current series retain only each series' latest live sample.How
Planner rules produce executable typed DAGs and accuracy certificates. The backend lowers and prices these candidates without inventing the logical rewrites. The new
MaintainedPopulationStrategyrule is specified in Shared maintained population rule, linked from the mapping overview and optimization catalog. It defines target/replacement DAGs, membership contracts, sharing identity, validation, costs and the compiler/runtime boundary.Before this PR
quantile(0.5, a),quantile(0.99, a),topk(1, a)andtopk(5, a)did not have a Planner-owned shared current-series state contract. The backend could not consume a typed candidate describing latest-value replacement, stale removal and lookback expiry.count(a)lowered to a distinct-value cardinality intent. Two live series with values7and7could therefore be represented as one distinct value instead of two series.min_over_time(a[5m])used the same exact-state kind as maximum.avg_over_time(a[5m])andtopk(5, sum_over_time(a[5m]))lacked the maintained candidate forms added here.quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m])lacked a checked, whole-expression relative-error candidate; two individual quantile certificates were insufficient.SELECT median(latency) FROM samplesandSELECT approx_percentile_cont(latency, 0.99) FROM samplescould not use the current-series rule to express shared table-row state. Table rows must not inherit PromQL's latest-series membership or five-minute lookback.After this PR
Typed candidates executable with companion backend PR #700:
quantile(0.5, a)andquantile(0.99, a)share one exact current-value population. Addingtopk(1, a)andtopk(5, a)shares the same population with a maximum-k cache of5; changing one series replaces its old value, and stale/expired series are removed.by(job)queries share within the same grouping; different groupings or selectors retain separate populations.sum(a),count(a)andavg(a)use typed current-population readouts. For two live series valued7and7,count(a)is2.min_over_time(a[5m])emits a distinct minimum accumulator;avg_over_time(a[5m])exposes maintained sum/count states with a typed finite-division guard (an overflowing sum falls back to the original average, while a zero average remains accelerated); explicitly exacttopk(5, sum_over_time(a[5m]))selects the five largest finalized per-series sums.quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]), Planner constructs a checked DDSketch division candidate with component accuracy below approximately 0.4975%. The backend checks the execution domain and falls back for cases such as a zero denominator.Shared SQL / PromQL rule and IR; SQL table-row deployment is not included yet:
These SQL queries now generate
MaintainPopulation(Rows) -> ReadPopulation(Quantile(q))candidates that share the table-row producer. Likewise,SELECT * FROM samples ORDER BY latency DESC LIMIT 1and the same query withLIMIT 5share a maximum-k population. SQL projections/aliases are preserved, and different filters/groupings do not share state.The table-row rule initially supports non-null Float64 value columns and single-measure aggregates. Backend #700 explicitly rejects deployment of this membership model until a row-update/deletion executor is available; existing SQL window-summary acceleration remains separate. These are candidate and correctness capabilities, not a claim of measured performance improvement.
Evidence
Companion backend PR ProjectASAP/ASAPQuery-backend#700 passes real Prometheus 3.5.0 comparison for the issue workloads, including moving windows and guarded fallback. Performance measurements and screenshots: not applicable.
Verification
cargo +1.98.0 test --workspacepassed, 1,097 tests.Latest generalization checks: all affected types/mapping/SQL package tests pass; six SQL frontend regressions cover shared quantiles, scalar readouts, maximum-k, group/filter separation and malformed-state rejection. Affected all-targets clippy passes with warnings denied.
Tests verify typed population sharing, row-count intent, minimum/maximum distinction, temporal-average realization, exact TopK inputs, executable DAG legality, and whole-expression relative-error composition.
Architectural decisions
The candidate declares runtime-checked division semantics. Rank-only KLL guarantees cannot establish relative value error. Exact temporal TopK adds support for explicit exact targets while preserving existing approximate heap candidate selection. Temporal average is a conditional physical candidate, not an unconditional logical sum/count rewrite: two finite
1e308samples must not turn a finite average into infinity.Limitations and follow-up
Requires matching backend lowering in PR 700. The backend deploys current-series populations; table-row state requires a row-update/deletion executor and is not silently mapped onto remote write. The initial table rule supports non-null Float64 values and single-measure aggregates. Existing SQL window summaries remain supported separately. Runtime must enforce finite operands, a nonzero divisor, and a normal finite quotient. These tests establish correctness, not a measured speedup.
Human review — do not complete with an agent