PromQL instant aggregates lose their evaluation horizon and lower directly over Scan

[milindsrivastava1997]

Problem

ASAPPlanner lowers PromQL instant aggregates over a bare Scan, with no TimeRange that represents the instant-query horizon. This makes the semantic time scope unavailable to summary planning and downstream materialization.

In ASAPQuery, the workload declares a 1 s instant lookback (also its scrape interval). The backend can select an exact Sum summary for sum(data), but the physical materializer rejects the resulting raw Scan input because it requires a temporal accumulator readout:

instantaneous sample selection is not a temporal accumulator readout

Reproduction

Pinned ASAPPlanner revision: ca7546de792d74aee8231e9a1100ca893d9e86d3.

printf '%s\n' \
  'promql> sum(data)' \
  'promql> sum by (job) (data)' \
  'promql> count(data)' \
  'promql> count by (job) (data)' \
  'promql> avg(data)' \
  'promql> avg by (job) (data)' \
  'promql> quantile(0.9, data)' \
  'promql> quantile by (job) (0.9, data)' \
| cargo run --locked -p asap-devtools --bin show_pre_asap_ir

Every query lowers as Aggregate { …, child: Scan { … } }:

Query family	Lowered intent	Child
sum, sum by (job)	Sum	Scan(data)
count, count by (job)	Cardinality { accuracy: Exact }	Scan(data)
avg, avg by (job)	Avg	Scan(data)
quantile, quantile by (job)	Quantile { q: 0.9, accuracy: Exact }	Scan(data)

Expected direction

For an instant query evaluated with a caller-supplied horizon, lower the selector side to:

Aggregate { … }
└── TimeRange { range: instant_horizon }
    └── Scan(data)

For the ASAPQuery workload above, instant_horizon = scrape interval = 1 s. More generally, this must be supplied by the caller’s instant-query context (lookback/scrape policy); the current lower_promql(query, accuracy) API has no such parameter and therefore cannot derive it safely from query text. This should not silently change the semantics of the AST-only API by inventing a global default.

Acceptance criteria

• A lowering/workload API can receive the explicit instant-query horizon.
• The eight instant aggregate variants above lower their input selector through TimeRange(horizon) → Scan.
• Explicit range selectors/functions (for example sum_over_time(data[5m])) preserve their explicit 5 m range rather than receiving an additional inferred range.
• Coverage pins both ungrouped and by (...) forms.

Related downstream symptom: ProjectASAP/ASAPQuery-backend#702.

[zzylol]

I am not sure the planner should need the scrape interval for the correctness of an instant PromQL query.

For an expression such as sum(data) evaluated at time T, PromQL first performs instant-vector selection: for each series, it selects the newest valid sample at or before T, subject to the configured lookback delta and staleness semantics. It then aggregates those selected samples across series. It does not aggregate all samples collected during one scrape interval.

In this workload, lookback = scrape interval = 1s, but that equality appears to be specific to the workload rather than a general PromQL rule. In a normal deployment, a scrape may be late or missed. If the scrape interval is 15s and the newest sample is 17s old, Prometheus may still select it when it is within the lookback period. Conversely, a time range can contain multiple samples for one series, but an instant selector must select only the latest valid one.

Therefore, lowering the selector simply as:

TimeRange(scrape_interval) -> Scan

may change the query semantics unless the downstream operator also implements latest-sample-per-series selection, including lookback and staleness handling.

Would it be more accurate for the API to receive an instant-selector context such as evaluation_time, lookback_delta, and the relevant staleness policy, and for the IR to represent LatestPerSeries (or equivalent) explicitly? The scrape/source-sample interval could remain an optional physical-planning input for choosing materialization windows and estimating cost, rather than a semantic requirement for lowering the PromQL AST.

Which parameters should be passed?

It may help to distinguish these parameters explicitly:

Parameter	Purpose
evaluation_time	The timestamp at which the instant query is evaluated.
lookback_delta	How far backward the selector may search for the latest valid sample.
scrape_interval	How frequently the source is expected to be sampled; primarily useful for optimization and capacity planning.
Query step/repeat interval	How frequently the query is evaluated.

lookback_delta versus an explicit range such as data[5m]

These represent different selector semantics:

Expression	Time parameter	Selection result
data	Implicit lookback_delta from the query context	At most one latest valid sample per series.
data[5m]	Explicit 5m range in the query	All samples for each series in the five-minute range.
avg(data)	Implicit lookback_delta	Average across the latest selected sample from each series.
avg_over_time(data[5m])	Explicit 5m range	Average over the samples in the five-minute range for each series.

Thus, the 5m in data[5m] does not already express lookback_delta. A bare instant selector such as data uses lookback to locate one latest valid sample per series, whereas a range selector returns all samples in its explicit range. Explicit range selectors should therefore preserve their range and should not receive an additional inferred lookback range.

This distinction may also be clearer if the IR represents the two operations separately, for example InstantSelector { evaluation_time, lookback_delta } and RangeSelector { evaluation_time, range }.

[zzylol]

Following up on the distinction above, I think the underlying fix should model instant-vector selection explicitly rather than treating it as an ordinary time range.

Proposed semantic shape

For:

sum(data)

the semantics are conceptually:

Aggregate(Sum)
└── latest valid sample per series as of T, within lookback L
    └── Scan(data)

TimeRange(scrape_interval) -> Scan is not sufficient because a range only identifies a set of records. It does not express the essential operation:

PARTITION BY series identity
ORDER BY sample timestamp DESC
LIMIT 1 per series

It may also miss a valid older sample after a delayed/missed scrape, or include multiple samples for one series.

Keeping Pre-ASAP IR compatible with SQL

I would avoid adding a PromQL-only InstantSelector to the shared Pre-ASAP IR. A language-neutral operator could instead be introduced (or recognized during normalization), for example:

LatestPerKey {
    keys: Vec<Column>,
    order_by: TimestampColumn,
    as_of: Expr,
    max_age: Option<Duration>,
    child: Box<Plan>,
}

The PromQL frontend could lower a bare selector to:

LatestPerKey {
    keys: complete_series_label_identity,
    order_by: sample_timestamp,
    as_of: evaluation_time,
    max_age: lookback_delta,
}

The same IR is useful for SQL forms such as ROW_NUMBER() OVER (PARTITION BY ... ORDER BY ts DESC) = 1, DISTINCT ON, arg_max, and as-of/latest-row queries. This keeps Pre-ASAP IR language-neutral and lets SQL and PromQL share rewrites, costing, indexes, and materialization strategies.

If adding a node is undesirable, the frontend could initially express this with generic relational operators:

Filter(row_number = 1)
└── Window(
      RowNumber,
      partition_by = series identity,
      order_by = timestamp DESC
    )
    └── Filter(timestamp <= T AND timestamp > T - lookback_delta)
        └── Scan(data)

A normalization rule could then recognize this shape and convert it into LatestPerKey for physical planning.

Staleness ordering

Prometheus staleness should be applied carefully. If the latest record is a stale marker, filtering stale records before latest-per-key selection could incorrectly expose an older value. The safer conceptual order is:

Aggregate(Sum)
└── Filter(latest record is not stale)
    └── LatestPerKey(...)
        └── Scan(data)

The PromQL frontend can supply the Prometheus-specific validity predicate while the shared IR operators remain generic.

API/context separation

The semantic query context should carry values such as:

QueryContext {
    evaluation_time,
    lookback_delta,
    staleness_policy,
    step,
}

The scrape/source sample interval should remain source metadata or a physical-planning hint. It is useful for estimating sample counts, choosing panes/windows, and costing, but should not determine the meaning of a bare PromQL selector.

This could be implemented in two phases so the AST-only API does not need to invent defaults:

1. Lower the query with unbound instant-selector context.
2. Bind evaluation_time, lookback_delta, and staleness policy when planning the concrete workload/request.

Materialization behavior

Correct acceleration of sum(data) requires state equivalent to the latest valid value of every series, not simply a sum of samples in a tumbling time window. For example, a latest-value materialization may need to track:

series key -> (latest timestamp, latest value, stale status)

and update grouped sums/counts when a series value is replaced, expires from lookback, or becomes stale. This matters when different targets are scraped at different times.

Until such a physical realization exists, the planner should choose an exact fallback rather than generating a semantically different temporal-accumulator read or producing a hard compile error.

Suggested acceptance criteria

• The query/workload context can bind evaluation_time, lookback_delta, and the relevant staleness behavior.
• Bare vector selectors represent latest-per-series selection explicitly (directly or through a normalized generic relational shape).
• Explicit range selectors such as data[5m] retain their explicit range and return all samples in that range; they do not receive an additional inferred lookback range.
• scrape_interval remains source/optimizer metadata rather than a semantic PromQL lowering parameter.
• If no semantically equivalent instant-summary realization exists, planning uses exact fallback instead of failing compilation.
• Tests cover delayed and missed scrapes, multiple samples per series inside the lookback, stale markers, and grouped/ungrouped aggregates.

In short, I think the semantic requirement is latest valid record per series as of the evaluation time, not all records within one scrape interval. Modeling that as a generic LatestPerKey operation should preserve SQL compatibility while making the PromQL behavior explicit.

[milindsrivastava1997]

Based on a call with @zzylol, the consensus is the following:

• For now, we stick to instant selectors being translated to "find a summary in [T-scrape_interval, T]. This is NOT PromQL semantics, but allows us to simplify things without worrying about late/stale data.
• Later, we can update this to be, find the latest summary in `[T-lookback_delta, T]. See https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness
• To exactly support PromQL semantics, we may need support for executing a query partially over materialized summaries, and partially over raw data.

[zzylol]

• To exactly support PromQL semantics, we may need support for executing a query partially over materialized summaries, and partially over raw data.

This is being supported in ASAPQuery backend.

[milindsrivastava1997]

This is being supported in ASAPQuery backend.

Sounds great. Let's first establish the correctness of the easy case, then we can do this.