Skip to content

feat: add maintained population and temporal aggregate candidates - #404

Merged
zzylol merged 15 commits into
mainfrom
feat/current-series-rules
Sep 14, 2026
Merged

zzylol merged 15 commits into
mainfrom
feat/current-series-rules

Conversation

@zzylol

@zzylol zzylol commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

Backend issues ProjectASAP/ASAPQuery-backend#701 and ProjectASAP/ASAPQuery-backend#702 require candidate DAGs that preserve aggregate populations and expression accuracy.

What

Add language-independent maintained-population rules and typed readouts, distinct minimum state, realized temporal average, exact TopK over temporal values, and guarded relative division.

A maintained population is the multiset of qualifying records represented by state retained across query evaluations and updated as members enter, change, leave or expire. A readout computes an aggregate from that state. For example, two current series with values 7 and 7 are two members; updating the first to 9 changes the population to {9, 7}, not {7, 7, 9}. SQL table rows retain row multiplicity; PromQL current series retain only each series' latest live sample.

How

Planner rules produce executable typed DAGs and accuracy certificates. The backend lowers and prices these candidates without inventing the logical rewrites. The new MaintainedPopulationStrategy rule is specified in Shared maintained population rule, linked from the mapping overview and optimization catalog. It defines target/replacement DAGs, membership contracts, sharing identity, validation, costs and the compiler/runtime boundary.

Before this PR

  • quantile(0.5, a), quantile(0.99, a), topk(1, a) and topk(5, a) did not have a Planner-owned shared current-series state contract. The backend could not consume a typed candidate describing latest-value replacement, stale removal and lookback expiry.
  • count(a) lowered to a distinct-value cardinality intent. Two live series with values 7 and 7 could therefore be represented as one distinct value instead of two series.
  • min_over_time(a[5m]) used the same exact-state kind as maximum. avg_over_time(a[5m]) and topk(5, sum_over_time(a[5m])) lacked the maintained candidate forms added here.
  • quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]) lacked a checked, whole-expression relative-error candidate; two individual quantile certificates were insufficient.
  • SQL SELECT median(latency) FROM samples and SELECT approx_percentile_cont(latency, 0.99) FROM samples could not use the current-series rule to express shared table-row state. Table rows must not inherit PromQL's latest-series membership or five-minute lookback.

After this PR

Typed candidates executable with companion backend PR #700:

  • quantile(0.5, a) and quantile(0.99, a) share one exact current-value population. Adding topk(1, a) and topk(5, a) shares the same population with a maximum-k cache of 5; changing one series replaces its old value, and stale/expired series are removed. by(job) queries share within the same grouping; different groupings or selectors retain separate populations.
  • sum(a), count(a) and avg(a) use typed current-population readouts. For two live series valued 7 and 7, count(a) is 2.
  • min_over_time(a[5m]) emits a distinct minimum accumulator; avg_over_time(a[5m]) exposes maintained sum/count states with a typed finite-division guard (an overflowing sum falls back to the original average, while a zero average remains accelerated); explicitly exact topk(5, sum_over_time(a[5m])) selects the five largest finalized per-series sums.
  • For a 1% relative-error target on quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]), Planner constructs a checked DDSketch division candidate with component accuracy below approximately 0.4975%. The backend checks the execution domain and falls back for cases such as a zero denominator.

Shared SQL / PromQL rule and IR; SQL table-row deployment is not included yet:

SELECT median(latency) FROM samples;
SELECT approx_percentile_cont(latency, 0.99) FROM samples;

These SQL queries now generate MaintainPopulation(Rows) -> ReadPopulation(Quantile(q)) candidates that share the table-row producer. Likewise, SELECT * FROM samples ORDER BY latency DESC LIMIT 1 and the same query with LIMIT 5 share a maximum-k population. SQL projections/aliases are preserved, and different filters/groupings do not share state.

The table-row rule initially supports non-null Float64 value columns and single-measure aggregates. Backend #700 explicitly rejects deployment of this membership model until a row-update/deletion executor is available; existing SQL window-summary acceleration remains separate. These are candidate and correctness capabilities, not a claim of measured performance improvement.

Evidence

Companion backend PR ProjectASAP/ASAPQuery-backend#700 passes real Prometheus 3.5.0 comparison for the issue workloads, including moving windows and guarded fallback. Performance measurements and screenshots: not applicable.

Verification

  • Earlier issue-workload validation — unit/integration/doc tests: cargo +1.98.0 test --workspace passed, 1,097 tests.
  • End-to-end tests: companion backend compatibility suite passed 14 tests, with one existing Collector-schema test ignored.
  • Other checks: workspace/all-targets clippy passed with warnings denied; formatting passed.

Latest generalization checks: all affected types/mapping/SQL package tests pass; six SQL frontend regressions cover shared quantiles, scalar readouts, maximum-k, group/filter separation and malformed-state rejection. Affected all-targets clippy passes with warnings denied.

Tests verify typed population sharing, row-count intent, minimum/maximum distinction, temporal-average realization, exact TopK inputs, executable DAG legality, and whole-expression relative-error composition.

Architectural decisions

The candidate declares runtime-checked division semantics. Rank-only KLL guarantees cannot establish relative value error. Exact temporal TopK adds support for explicit exact targets while preserving existing approximate heap candidate selection. Temporal average is a conditional physical candidate, not an unconditional logical sum/count rewrite: two finite 1e308 samples must not turn a finite average into infinity.

Limitations and follow-up

Requires matching backend lowering in PR 700. The backend deploys current-series populations; table-row state requires a row-update/deletion executor and is not silently mapped onto remote write. The initial table rule supports non-null Float64 values and single-measure aggregates. Existing SQL window summaries remain supported separately. Runtime must enforce finite operands, a nonzero divisor, and a normal finite quotient. These tests establish correctness, not a measured speedup.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

@zzylol zzylol changed the title Add rules and typed IR for shared current-series quantiles and TopK Add typed current-series and temporal aggregate candidates Sep 12, 2026
@zzylol zzylol changed the title Add typed current-series and temporal aggregate candidates Add typed maintained-population and temporal aggregate candidates Sep 13, 2026
@zzylol zzylol changed the title Add typed maintained-population and temporal aggregate candidates feat: add maintained population and temporal aggregate candidates Sep 13, 2026
zzylol and others added 5 commits September 13, 2026 08:27
* refactor(types): split ExactKind::MinMax into Min and Max

`ExactKind::MinMax` was never a min/max pair: `function_rules` maps
`AggIntent::Max` onto it and `AggIntent::Min` onto the separately added
`ExactKind::Min`, so the name promised a two-sided accumulator that the
variant never was. Consumers that match on the family had to carry the
direction out-of-band (ASAPQuery-backend threads it through the
`aggregationSubType` wire string) because the name gave no guarantee
about which extremum the state holds.

Rename the variant to `Max` in both `ExactKind` and `ExactParams`, so
the two directions are symmetric variants and a stored minimum can no
longer content-address onto a maximum.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(promql): restore a distinct-count idiom over mergeable cardinality

Lowering `count(v)` to a row count (15e6cf8) matched PromQL, but it left
no way to ask for a distinct count that reduces across series. The
`Cardinality` intent survived only under `distinct_over_time(v[w])`,
which is per-series: a consumer holding one mergeable cardinality state
per source could answer the merged question and had no expression that
asked it.

Collapse `count(distinct_over_time(v[w]))` into a single
`Aggregate{[Cardinality]}` reduced by the outer `by`. The nested reading
-- count the series that have a distinct-count -- is not what anyone
writes this expression for, and it discards the mergeability of the
state underneath.

The collapsed form needs its own reduction rule: `reduction_for` reads
the surviving `TimeRange` as one row per series and answers `PerEntity`,
which is right for a bare `distinct_over_time(v[w])` and wrong once an
explicit aggregator wraps it. `windowed_reduce` keeps the outer
aggregator's grouping, so an empty `by` reduces everything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(promql): preserve count over per-series cardinality

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@zzylol

zzylol commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@milindsrivastava1997 FYI

@zzylol
zzylol merged commit 59aaf59 into main Sep 14, 2026
3 checks passed
@milindsrivastava1997

Copy link
Copy Markdown
Collaborator

@zzylol does this solve #410 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants