Skip to content
25 changes: 25 additions & 0 deletions docs/l1-query-language.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,28 @@ A front end is expected to cleanly reject a shape it can't yet
represent — rather than silently mis-lowering it — since a
resolvable-later gap is one design choice away from becoming a
correctness bug if it's lowered wrong instead of rejected outright.

## Interface

There is no shared L1 trait — each front end exposes its own free
functions, differing in language-specific arguments (SQL takes a
catalog + dialect; PromQL doesn't) but converging on the same return
type once lowering finishes:

```rust
// PromQL
pub fn lower_promql(query: &str, accuracy: AccuracyTarget) -> Result<QueryExpr, PromqlError>;
pub fn lower_promql_batch(workload: &QueryWorkload) -> Vec<Result<QueryExpr, PromqlError>>;

// SQL
pub async fn lower_sql(query: &str, catalog: &SqlCatalog, accuracy: AccuracyTarget) -> Result<QueryExpr, SqlError>;
pub async fn lower_sql_dialect(query: &str, catalog: &SqlCatalog, dialect: SqlDialect, accuracy: AccuracyTarget) -> Result<QueryExpr, SqlError>;
pub async fn lower_sql_batch(workload: &QueryWorkload, catalog: &SqlCatalog) -> Vec<Result<QueryExpr, SqlError>>;
```

Both converge on the canonical L3 `QueryExpr` — the interface that
actually matters lives one step down, at the shared L2 → L3 step this
doc already describes (see [`l2-logical-plan.md`](./l2-logical-plan.md#interface)).
SQL's extra `catalog`/`dialect` parameters are exactly the schema-source
asymmetry covered above: SQL supplies its own resolved schema up front;
PromQL doesn't have one to supply, so it has no equivalent parameter.
62 changes: 62 additions & 0 deletions docs/l2-logical-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,65 @@ row): an already-cataloged source can carry a real one through
unchanged, while a usage-derived schema has no way to prove one, and so
never gets one. Why a unique key matters is covered in
[`l3-intent-algebra.md`](./l3-intent-algebra.md#unique-keys-and-cross-query-sharing).

## Interface

The one real extension point at this layer is the schema source a
front end's leaves resolve against:

```rust
pub trait SchemaCatalog {
fn columns_for(&self, source: &str) -> Option<Vec<Column>>;
}

pub struct Binder<C: SchemaCatalog = UsageDerivedCatalog> { .. }
impl<C: SchemaCatalog> Binder<C> {
pub fn bind(&self, tree: &QueryExpr) -> Schema;
}
```
Comment thread
zzylol marked this conversation as resolved.

The default `Binder` (`UsageDerivedCatalog`) implements this by always
returning `None` — a schema built purely from what the query happens to
reference. A catalog-backed language (or a future registry-backed one)
implements `columns_for` to return a real, closed column set instead;
nothing about `Binder` itself has to change for that swap.

For example, PromQL has no real catalog — a metric's label set is only
knowable from what the query itself references — so its front end binds
with the default:

```rust
// PromQL: `sum by (job) (http_requests_total)`, no catalog available.
let schema = Binder::default().bind(&tree);
// -> Schema { columns: [ts, value, job], time_index: Some(0), closed: false }
// ("job" was seeded because the query references it; anything the
// query never mentions is simply absent from this schema)
```

SQL has a real, declared catalog, so its front end supplies one instead:

```rust
struct SqlCatalog { /* wraps a table registry */ }
impl SchemaCatalog for SqlCatalog {
fn columns_for(&self, source: &str) -> Option<Vec<Column>> {
// "requests" -> its full declared column list, looked up from
// whatever table registry this deployment has, e.g.:
// Some(vec![Column::new("host", Utf8, false), Column::new("bytes", Int64, false)])
..
}
}
let schema = Binder::with_catalog(SqlCatalog { .. }).bind(&tree);
// -> Schema { columns: [host, bytes], time_index: None, closed: true }
// (the catalog's declared columns are used verbatim, regardless of
// which ones the query actually references)
```

The L2 → L3 step every front end shares:

```rust
pub fn convert_root(legacy: &QueryExpr, accuracy: &AccuracyTarget) -> Result<QueryExpr, ConvertError>;
```

Takes L2's per-language tree in, returns L3's canonical tree out —
binding, structural conversion, and canonicalization in one call, so no
front end can reach L3 through any other path.
253 changes: 253 additions & 0 deletions docs/l3-intent-algebra.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,3 +151,256 @@ key, sharing is unsound and each consumer must recompute independently
— which is why a source with no way to prove a key (an open,
usage-derived schema) can never participate in this kind of sharing,
while a source with a declared key from an external catalog can.

## Interface

The canonical `QueryExpr` is one Rust enum covering the whole relational
vocabulary, every variant:

```rust
pub enum QueryExpr {
// ── leaves ────────────────────────────────────────────────────────
Scan { source: Source, predicates: Vec<Predicate>, schema: Schema },
Ref { name: BindingName }, // a LetBinding reference
Scalar(f64), // a constant scalar literal
EvalTime, // the query evaluation time as a scalar

// ── scalar/vector bridges ────────────────────────────────────────
VectorFromScalar(Box<QueryExpr>), // promote a scalar to a label-less vector
ScalarFromVector(Box<QueryExpr>), // collapse a single-series vector to a scalar

// ── per-row transforms ───────────────────────────────────────────
Relabel { dst: String, value: L3Expr, child: Box<QueryExpr> },
InfoJoin { selector: Vec<InfoMatcher>, child: Box<QueryExpr> },
Sample { by: GroupKeys, kind: SampleKind, child: Box<QueryExpr> },

// ── core relational ──────────────────────────────────────────────
Filter { pred: Predicate, child: Box<QueryExpr> },
Project { cols: Vec<ProjectItem>, qualifier: Option<String>, child: Box<QueryExpr> },
Aggregate {
reduction: Reduction,
aggs: Vec<AggIntent>,
output_names: Vec<String>,
having: Option<Predicate>,
child: Box<QueryExpr>,
},

// ── windowing, dedup, set composition ────────────────────────────
Window { kind: WindowKind, size: Duration, slide: Option<Duration>, child: Box<QueryExpr> },
Distinct { cols: Vec<ColumnId>, child: Box<QueryExpr> },
Merge { children: Vec<QueryExpr> }, // exact, n-ary UNION ALL
Join { kind: JoinKind, pred: Predicate, left: Box<QueryExpr>, right: Box<QueryExpr> },
SetOp { kind: SetOpKind, all: bool, left: Box<QueryExpr>, right: Box<QueryExpr> },

// ── ordering / limiting ──────────────────────────────────────────
Sort { keys: Vec<SortKey>, partition_by: GroupKeys, child: Box<QueryExpr> },
Limit { n: usize, offset: usize, child: Box<QueryExpr> },

// ── sharing and temporal wrappers ────────────────────────────────
LetBinding { name: BindingName, expr: Box<QueryExpr>, child: Box<QueryExpr> },
Subquery { range: Duration, resolution: Option<Duration>, child: Box<QueryExpr> },
TimeRange { range: Duration, child: Box<QueryExpr> },
TimeShift { shift: TimeShift, child: Box<QueryExpr> },

// ── SQL analytic window functions ────────────────────────────────
WindowFunc {
func: WindowFuncKind,
args: Vec<L3Expr>,
partition_by: GroupKeys,
order_by: Vec<SortKey>,
output_name: String,
child: Box<QueryExpr>,
},

// ── arithmetic / comparison / boolean composition ────────────────
BinaryOp { op: BinaryOpKind, lhs: Box<QueryExpr>, rhs: Box<QueryExpr>, vector_match: Option<VectorMatch> },
}
```

`reduction` is a field *on* the `Aggregate` variant itself — not a
separate node in the tree, and not something any other variant carries.
It answers a question only `Aggregate` ever needs to ask: is this node
collapsing rows at all, and if so, by which (possibly empty) key set —
or does it have no grouping concept to begin with. Making that an
explicit field, rather than something a consumer infers from whether a
key list happens to be empty, is deliberate: the two cases are easy to
conflate (both can present as "empty keys") but require opposite
handling downstream — see [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md#interface)'s
`SummaryExecutor::find_candidates` for where that distinction is
actually load-bearing.

The two small types that field's shape turns on, in full — neither is a
tree node either; both are plain data reachable only through
`Aggregate.reduction`, and `GroupKeys` only exists at all when
`reduction` is `Reduce` (it's meaningless for `PerEntity`, which is
exactly why it isn't a sibling field instead):

```rust
pub enum Reduction {
Comment thread
zzylol marked this conversation as resolved.
Reduce(GroupKeys),
PerEntity,
}

pub struct GroupKeys { /* private fields */ }
Comment thread
zzylol marked this conversation as resolved.
impl GroupKeys {
pub fn by(keys: Vec<ColumnId>) -> Self; // keep these columns
pub fn without(keys: Vec<ColumnId>) -> Self; // exclude these columns
pub fn is_without(&self) -> bool;
pub fn keys(&self) -> &[ColumnId];
}
```

`AggIntent` — the full vocabulary:

```rust
pub enum AggIntent {
Comment thread
zzylol marked this conversation as resolved.
// data-model-agnostic reducers
Count { accuracy: AccuracyTarget },
Sum { col: Option<ColumnId> },
Min { col: Option<ColumnId> },
Max { col: Option<ColumnId> },
Avg { col: Option<ColumnId> },
StdDev { col: Option<ColumnId>, population: bool },
Variance { col: Option<ColumnId>, population: bool },
Quantile { col: Option<ColumnId>, q: f64, accuracy: AccuracyTarget },
TopK { k: usize, accuracy: AccuracyTarget },
Cardinality { col: Option<ColumnId>, accuracy: AccuracyTarget },

// counter-aware streaming derivatives
Rate,
Increase,

// further counter-derivative / range-vector functions
Changes, Delta, IDelta, Deriv, Resets,
PredictLinear { seconds: f64 },
DoubleExpSmoothing { smoothing: f64, trend: f64 },
LastOverTime, FirstOverTime, MadOverTime,
Comment thread
zzylol marked this conversation as resolved.
TsOfMinOverTime, TsOfMaxOverTime, TsOfFirstOverTime, TsOfLastOverTime,

// native-histogram accessors
HistogramCount, HistogramSum, HistogramAvg, HistogramStdDev, HistogramStdVar,
HistogramFraction { lower: f64, upper: f64 },
HistogramQuantile { q: f64 },

// per-sample transforms
Math(MathFunc),
TimeFn(TimeFunc),

// presence
Absent, AbsentOverTime, PresentOverTime,

// extended aggregation operators — grouped by *sample value* rather
// than reducing to a single number per group, so both need a schema
// shape no other reducer in this list uses:
//
// - `Group`: emits a constant `1` per group ("does this group have any
// members at all"), independent of the input values — the output
// carries no information about the samples beyond their presence.
// Kept as its own intent rather than folded into `Sum`/`Count`
// because the value has nothing to do with what's being summed or
// counted.
// - `CountValues { label }`: groups the input further by each
// distinct *sample value* (not just the usual grouping keys),
// counting how many samples land in each. Since the sample value
// itself becomes part of the output's identity, this is the one
// reducer whose output schema gains a new column (a synthesized
// label named `label`, holding the stringified value) rather than
// just a single retyped aggregate column.
Group,
CountValues { label: String },
Comment thread
zzylol marked this conversation as resolved.

// deployment-specific escape hatch (see "Design rules" above) — core
// treats this opaquely; the owning deployment defines and interprets
// `payload` itself, keyed by its own `ext_kind` tag
Extension { ext_kind: String, payload: serde_json::Value },
}
```

One example source expression per variant, PromQL unless noted (`v` stands in
for any instant/range vector selector):

| `AggIntent` | Example |
|---|---|
| `Count` | `count(up)` |
| `Sum` | `sum(rate(http_requests_total[5m]))`; SQL `SUM(bytes)` |
| `Min` | `min(cpu_temp)` |
| `Max` | `max(cpu_temp)` |
| `Avg` | `avg(cpu_usage)` |
| `StdDev` | `stddev(latency_ms)`; SQL `STDDEV(col)` |
| `Variance` | `stdvar(latency_ms)`; SQL `VARIANCE(col)` |
| `Quantile` | `quantile(0.99, latency_ms)`; SQL `approx_percentile_cont(col, 0.99)` |
| `TopK` | `topk(5, http_requests_total)` |
| `Cardinality` | SQL `COUNT(DISTINCT user_id)`; the PromQL analogue is `count(...)` over a metric whose underlying storage is a cardinality sketch, not a literal PromQL function call |
| `Rate` | `rate(http_requests_total[5m])` |
| `Increase` | `increase(http_requests_total[5m])` |
| `Changes` | `changes(v[5m])` |
| `Delta` | `delta(v[5m])` |
| `IDelta` | `idelta(v[5m])` |
| `Deriv` | `deriv(v[5m])` |
| `Resets` | `resets(v[5m])` |
| `PredictLinear` | `predict_linear(v[5m], 3600)` |
| `DoubleExpSmoothing` | `double_exponential_smoothing(v[5m], 0.5, 0.5)` |
| `HistogramCount` | `histogram_count(v)` |
| `HistogramSum` | `histogram_sum(v)` |
| `HistogramAvg` | `histogram_avg(v)` |
| `HistogramStdDev` | `histogram_stddev(v)` |
| `HistogramStdVar` | `histogram_stdvar(v)` |
| `HistogramFraction` | `histogram_fraction(0.1, 0.5, v)` |
| `HistogramQuantile` | `histogram_quantile(0.99, v)` |
| `Math` | `abs(v)`, `sqrt(v)`, `ceil(v)`, … (one `MathFunc` per PromQL math/trig builtin) |
| `TimeFn` | `hour()`, `day_of_week(v)`, … (one `TimeFunc` per PromQL calendar builtin) |
| `Absent` | `absent(v)` |
| `AbsentOverTime` | `absent_over_time(v[5m])` |
| `PresentOverTime` | `present_over_time(v[5m])` |
| `Group` | `group(v)` |
| `CountValues` | `count_values("version", v)` |
| `LastOverTime` | `last_over_time(v[5m])` |
| `FirstOverTime` | `first_over_time(v[5m])` |
| `MadOverTime` | `mad_over_time(v[5m])` |
| `TsOfMinOverTime` | `ts_of_min_over_time(v[5m])` |
| `TsOfMaxOverTime` | `ts_of_max_over_time(v[5m])` |
| `TsOfFirstOverTime` | `ts_of_first_over_time(v[5m])` |
| `TsOfLastOverTime` | `ts_of_last_over_time(v[5m])` |
| `Extension` | deployment-defined — e.g. a deployment-specific membership-test function no core language has a builtin for |

**Why the `*OverTime` reducers (`LastOverTime` … `TsOfLastOverTime`) each need
their own intent, rather than composing from `Sort`/`Limit` or another generic
node.** They all reduce a single series' *raw sample sequence inside a range
window* to one value — but nothing else in this vocabulary exposes that raw
sequence as rows a generic operator could sort or limit. `Sort`/`Limit`
operate on the relation `Aggregate` already produced (one row per series,
post-reduction); a `TimeRange` window's underlying samples are never
materialized as a queryable row set at this layer, only fed directly into
whichever `AggIntent` sits above it. So there is no composition available to
express "the timestamp of this window's minimum sample" from generic parts —
each of these is the only path that can reach into the window's raw stream
for its particular statistic, which is exactly the design rule 1 exception
(a genuinely different computational access pattern, not an ordinary
composition of existing operators).

And the schema every edge in the tree carries:

```rust
pub struct Schema {
pub columns: Vec<Column>,
pub time_index: Option<ColumnId>,
pub unique_keys: Vec<Vec<ColumnId>>,
pub closed: bool,
}
```
Comment thread
zzylol marked this conversation as resolved.

For example, `sum by (job) (http_requests_total)` binds to an `Aggregate`
whose input schema is `{ columns: [ts, value, job], time_index: Some(0),
unique_keys: [], closed: false }` (PromQL — open, since a metric's label
set is a superset the runtime may exceed) and whose *output* schema —
after `Reduction::by([job])` collapses every other row — is:

```rust
Schema {
columns: vec![Column::new("job", DataType::Utf8, true), Column::new("sum", DataType::Float64, false)],
time_index: None, // a cross-series reduction collapses the time axis
unique_keys: vec![vec![0]], // grouping by job makes job unique in the output
closed: true, // by(...) enumerates its columns exactly — this is
// where an open input schema freezes to closed
}
```
Loading
Loading