From 395b1ff721a44d5df2235fa41f147873b5d177c9 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 15:21:31 -0600 Subject: [PATCH 1/8] docs: add an Interface section to each L1-L5 doc Fixes #168. PR #157 rewrote design.md + the per-layer docs to be pure principle/ design-level, stripped of code-level detail on purpose. This adds the promised follow-up: a concrete Rust API surface per layer, verified against current `main` rather than carried over from an old draft. - L1: no shared trait exists -- each front end exposes its own free functions (lower_promql*, lower_sql*), converging on the same L3 QueryExpr return type. Kept thin rather than padded, since that's the honest current shape. - L2: SchemaCatalog (the pluggable schema-source trait) + Binder + convert_root. - L3: QueryExpr's shape (a representative slice, not the full ~24-variant enum), Reduction + GroupKeys in full (small, central to this doc's own design principles), a representative AggIntent slice, and Schema. - L4: SummaryExpr/L4Node, the binding entry points + Implementation, the CostModel extension point (all default-having methods shown), and SummaryExecutor in full -- the real, load-bearing serving-time interface just resolved via #163/#164/#165 this session. - L5: nothing is built yet, so this is explicitly speculative -- recovered the PhysicalPlanner/TopologyDescriptor/StageAllocator sketch an earlier draft of this doc had (before PR #157's final rewrite trimmed it), updated to current summary-bound-IR naming (L4Node, SummaryKind) and kept the same "target to design against, not an API to depend on" framing so it can't be mistaken for shipped code. Every signature was checked against the actual current source in this repo (crates/frontend-promql, crates/frontend-sql, crates/l2, crates/ir, crates/sketch, crates/plan) rather than assumed from memory. Co-Authored-By: Claude Sonnet 5 --- docs/l1-query-language.md | 25 +++++++++ docs/l2-logical-plan.md | 32 ++++++++++++ docs/l3-intent-algebra.md | 68 ++++++++++++++++++++++++ docs/l4-summary-bound-ir.md | 101 ++++++++++++++++++++++++++++++++++++ docs/l5-physical-plan.md | 57 ++++++++++++++++++++ 5 files changed, 283 insertions(+) diff --git a/docs/l1-query-language.md b/docs/l1-query-language.md index 50757d9a..5f434b0b 100644 --- a/docs/l1-query-language.md +++ b/docs/l1-query-language.md @@ -67,3 +67,28 @@ A front end is expected to cleanly reject a shape it can't yet represent — rather than silently mis-lowering it — since a resolvable-later gap is one design choice away from becoming a correctness bug if it's lowered wrong instead of rejected outright. + +## Interface + +There is no shared L1 trait — each front end exposes its own free +functions, differing in language-specific arguments (SQL takes a +catalog + dialect; PromQL doesn't) but converging on the same return +type once lowering finishes: + +```rust +// PromQL +pub fn lower_promql(query: &str, accuracy: AccuracyTarget) -> Result; +pub fn lower_promql_batch(workload: &QueryWorkload) -> Vec>; + +// SQL +pub async fn lower_sql(query: &str, catalog: &SqlCatalog, accuracy: AccuracyTarget) -> Result; +pub async fn lower_sql_dialect(query: &str, catalog: &SqlCatalog, dialect: SqlDialect, accuracy: AccuracyTarget) -> Result; +pub async fn lower_sql_batch(workload: &QueryWorkload, catalog: &SqlCatalog) -> Vec>; +``` + +Both converge on the canonical L3 `QueryExpr` — the interface that +actually matters lives one step down, at the shared L2 → L3 step this +doc already describes (see [`l2-logical-plan.md`](./l2-logical-plan.md#interface)). +SQL's extra `catalog`/`dialect` parameters are exactly the schema-source +asymmetry covered above: SQL supplies its own resolved schema up front; +PromQL doesn't have one to supply, so it has no equivalent parameter. diff --git a/docs/l2-logical-plan.md b/docs/l2-logical-plan.md index ba80657c..dbf16425 100644 --- a/docs/l2-logical-plan.md +++ b/docs/l2-logical-plan.md @@ -86,3 +86,35 @@ row): an already-cataloged source can carry a real one through unchanged, while a usage-derived schema has no way to prove one, and so never gets one. Why a unique key matters is covered in [`l3-intent-algebra.md`](./l3-intent-algebra.md#unique-keys-and-cross-query-sharing). + +## Interface + +The one real extension point at this layer is the schema source a +front end's leaves resolve against: + +```rust +pub trait SchemaCatalog { + fn columns_for(&self, source: &str) -> Option>; +} + +pub struct Binder { .. } +impl Binder { + pub fn bind(&self, tree: &QueryExpr) -> Schema; +} +``` + +The default `Binder` (`UsageDerivedCatalog`) implements this by always +returning `None` — a schema built purely from what the query happens to +reference. A catalog-backed language (or a future registry-backed one) +implements `columns_for` to return a real, closed column set instead; +nothing about `Binder` itself has to change for that swap. + +The L2 → L3 step every front end shares: + +```rust +pub fn convert_root(legacy: &QueryExpr, accuracy: &AccuracyTarget) -> Result; +``` + +Takes L2's per-language tree in, returns L3's canonical tree out — +binding, structural conversion, and canonicalization in one call, so no +front end can reach L3 through any other path. diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md index 5b69e5a4..700ff602 100644 --- a/docs/l3-intent-algebra.md +++ b/docs/l3-intent-algebra.md @@ -151,3 +151,71 @@ key, sharing is unsound and each consumer must recompute independently — which is why a source with no way to prove a key (an open, usage-derived schema) can never participate in this kind of sharing, while a source with a declared key from an external catalog can. + +## Interface + +The canonical `QueryExpr` is one Rust enum covering the whole relational +vocabulary — a representative slice, not the full list: + +```rust +pub enum QueryExpr { + Scan { source: Source, predicates: Vec, schema: Schema }, + Filter { pred: Predicate, child: Box }, + Project { cols: Vec, qualifier: Option, child: Box }, + Aggregate { + reduction: Reduction, + aggs: Vec, + output_names: Vec, + having: Option, + child: Box, + }, + Window { .. }, Distinct { .. }, Merge { .. }, Join { .. }, SetOp { .. }, + Sort { .. }, Limit { .. }, LetBinding { .. }, Ref { .. }, + Subquery { .. }, TimeRange { .. }, TimeShift { .. }, WindowFunc { .. }, + BinaryOp { .. }, + // .. plus scalar/vector bridges and a small number of + // language-specific extension points with no general equivalent yet +} +``` + +The two small types this doc's design principles turn on, in full: + +```rust +pub enum Reduction { + Reduce(GroupKeys), + PerEntity, +} + +pub struct GroupKeys { /* private fields */ } +impl GroupKeys { + pub fn by(keys: Vec) -> Self; // keep these columns + pub fn without(keys: Vec) -> Self; // exclude these columns + pub fn is_without(&self) -> bool; + pub fn keys(&self) -> &[ColumnId]; +} +``` + +`AggIntent` — again a representative slice: + +```rust +pub enum AggIntent { + Count { accuracy: AccuracyTarget }, + Sum { col: Option }, + Quantile { col: Option, q: f64, accuracy: AccuracyTarget }, + TopK { k: usize, accuracy: AccuracyTarget }, + Cardinality { col: Option, accuracy: AccuracyTarget }, + Rate, Increase, Changes, Delta, IDelta, Deriv, Resets, // counter-derivative family + // .. native-histogram accessors, per-sample transforms, Extension +} +``` + +And the schema every edge in the tree carries: + +```rust +pub struct Schema { + pub columns: Vec, + pub time_index: Option, + pub unique_keys: Vec>, + pub closed: bool, +} +``` diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 9935dde5..0b18fb1d 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -146,3 +146,104 @@ computation requires richer partial state than a single summary value can capture, or whose result is fundamentally non-mergeable across partitions, has no summary realization by design — it always stays as ordinary logical computation over raw data. + +## Interface + +The planning-time IR: + +```rust +pub struct L4Node { + pub expr: SummaryExpr, + pub schema: L4Schema, +} + +pub enum SummaryExpr { + Logical(Box), + SummaryAgg { + child: Rc, + sketch: SummaryKind, + params: SummaryParams, + col: ColumnRef, + reduction: Reduction, + }, + SummaryJoin { outer: Rc, inner: Rc, key: ColumnRef, sketch: SummaryKind, params: SummaryParams }, + SummarySubtract { left: Rc, right: Rc }, + SummaryDelete { sketch_input: Rc, key: ColumnRef }, + SummaryEstimate { sketch_input: Rc, query: SketchQuery }, + SummaryMerge { children: Vec> }, +} +``` + +Binding — turning a canonical `QueryExpr` into an `L4Node` tree: + +```rust +pub fn implement_tree(expr: &QueryExpr) -> Result, ImplementError>; +pub fn implement_tree_with(expr: &QueryExpr, cost_model: &dyn CostModel) -> Result, ImplementError>; +``` + +The per-intent decision binding drives, and its result type: + +```rust +pub enum Implementation { + Sketch { kind: SummaryKind, params: SummaryParams }, + ExactAccumulator { kind: SummaryKind, params: SummaryParams }, + PassThrough, +} + +pub fn implementation_for(intent: &AggIntent) -> Implementation; +``` + +The one pluggable extension point at this layer — a deployment supplies +its own `CostModel` to override which summary family is chosen and how +its parameters are sized, without touching the binding pass itself. +Every method past the first has a sensible default, so a deployment that +only wants to reorder candidates doesn't have to implement the rest: + +```rust +pub trait CostModel { + fn rank_candidates(&self, intent: &AggIntent, candidates: &[SummaryKind]) -> Vec; + + fn size_params(&self, kind: SummaryKind, intent: &AggIntent, eps: f64, delta: f64) -> SummaryParams { + /* default: this crate's built-in sizing formulas */ + } + fn realize_extension(&self, ext_kind: &str, payload: &serde_json::Value) -> Implementation { + /* default: PassThrough */ + } + fn readout_extension(&self, ext_kind: &str, payload: &serde_json::Value, col: &ColumnRef) -> SketchQuery { + /* default: panics -- only reachable if realize_extension was overridden without this */ + } +} +``` + +Serving-time — the interface a deployment implements to actually answer +a query against an already-bound `L4Node` tree: + +```rust +pub trait SummaryExecutor { + type Handle: Clone; + type State; + type Value; + type Error; + type GroupKey: Clone + Ord + Default; + + fn find_candidates( + &self, + sketch: &SummaryKind, + params: &SummaryParams, + col: &ColumnRef, + reduction: &Reduction, + child: &L4Node, + ) -> Result, Self::Error>; + + fn fetch_state(&self, handle: &Self::Handle) -> Result; + fn merge_states(&self, states: Vec) -> Result; + fn readout(&self, state: &Self::State, query: &SketchQuery) -> Result; + fn logical(&self, expr: &QueryExpr) -> Result; +} + +pub fn execute(node: &L4Node, exec: &E) -> Result, ExecError>; +``` + +`find_candidates`'s `reduction` parameter is exactly the `Reduction` +this doc's design section already explains — an implementer must branch +on `Reduce` vs. `PerEntity` there, not guess from an empty key list. diff --git a/docs/l5-physical-plan.md b/docs/l5-physical-plan.md index fd3cc1bd..53b7ed23 100644 --- a/docs/l5-physical-plan.md +++ b/docs/l5-physical-plan.md @@ -72,3 +72,60 @@ infrastructure that every deployment reads from, rather than something each deployment reinvents independently — though which summary families and parameter ranges are actually registered is itself deployment policy. + +## Interface + +Speculative — nothing here is built (see this doc's status note at the +top). Kept as a concrete target to design against, not as an API to +depend on: + +```rust +pub trait PhysicalPlanner { + type Topology: TopologyDescriptor; + type Output; + fn lower(&self, l4: Rc, t: &Self::Topology) -> Result; +} + +pub trait TopologyDescriptor { + fn stages(&self) -> &[StageDescriptor]; + fn edges(&self) -> &[StageEdge]; +} + +pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" + +pub struct Executor { + pub id: ExecutorId, + pub stage: StageId, + pub capabilities: ExecutorCaps, // memory budget, available summary backends, network neighbours + pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle +} + +// Stage-level allocation: given an L4 tree + a topology, decide which +// nodes land on which stage, subject to deployment constraints. +// Per-executor fan-out is the deployment model's own PhysicalPlanner, +// using the executor list from DeploymentConstraints::executors(). +pub struct StageAllocator; +impl StageAllocator { + pub fn allocate( + &self, l4: Rc, topology: &T, c: &DeploymentConstraints, + ) -> Result, PlanError>; +} +``` + +A deployment model implements `PhysicalPlanner`, delegating the +stage-level decision to `StageAllocator` and handling its own +per-executor fan-out on top: + +```rust +impl PhysicalPlanner for LifecyclePlanner { + type Topology = ThreeStage; + type Output = Vec<(ExecutorId, ExecutorPlan)>; + fn lower(&self, l4: Rc, t: &ThreeStage) -> Result { + let assignments = StageAllocator.allocate(l4, t, &self.constraints)?; + let executors: &[Executor] = self.constraints.executors(); + // deployment-specific post-processing (cost accounting, backend + // push preparation, ...) happens here. + Ok(/* ... */) + } +} +``` From 0b70dc7f3610a111db3fcfe7e3589d513fa66fc0 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 15:39:43 -0600 Subject: [PATCH 2/8] docs: address PR #169 review comments on the L3/L4/L5 interface sections - L3: clarify reduction/GroupKeys/Reduction aren't tree nodes (they're plain data reached only through Aggregate.reduction), and expand the AggIntent listing from a representative slice to the full vocabulary. - L4: explain what "Implementation" means (the per-intent binding decision's own result type); note the sketch/summary field-naming tension (sketch/sketch_input predate this doc's own "summary" terminology) and the Sketch/ExactAccumulator merge question as deliberate follow-ups, not silently resolved here -- see PR discussion. - L5: fold "what does edge mean", "StageId values are deployment-owned", and "interface only in ASAPController" into one clarification at the top of the Interface section, plus reword StageId's own comment so its example values don't read as a fixed vocabulary. Co-Authored-By: Claude Sonnet 5 --- docs/l3-intent-algebra.md | 60 ++++++++++++++++++++++++++++++++++--- docs/l4-summary-bound-ir.md | 32 +++++++++++++++++++- docs/l5-physical-plan.md | 19 ++++++++++-- 3 files changed, 103 insertions(+), 8 deletions(-) diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md index 700ff602..392ddeae 100644 --- a/docs/l3-intent-algebra.md +++ b/docs/l3-intent-algebra.md @@ -178,7 +178,23 @@ pub enum QueryExpr { } ``` -The two small types this doc's design principles turn on, in full: +`reduction` is a field *on* the `Aggregate` variant itself — not a +separate node in the tree, and not something any other variant carries. +It answers a question only `Aggregate` ever needs to ask: is this node +collapsing rows at all, and if so, by which (possibly empty) key set — +or does it have no grouping concept to begin with. Making that an +explicit field, rather than something a consumer infers from whether a +key list happens to be empty, is deliberate: the two cases are easy to +conflate (both can present as "empty keys") but require opposite +handling downstream — see [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md#interface)'s +`SummaryExecutor::find_candidates` for where that distinction is +actually load-bearing. + +The two small types that field's shape turns on, in full — neither is a +tree node either; both are plain data reachable only through +`Aggregate.reduction`, and `GroupKeys` only exists at all when +`reduction` is `Reduce` (it's meaningless for `PerEntity`, which is +exactly why it isn't a sibling field instead): ```rust pub enum Reduction { @@ -195,17 +211,53 @@ impl GroupKeys { } ``` -`AggIntent` — again a representative slice: +`AggIntent` — the full vocabulary: ```rust pub enum AggIntent { + // data-model-agnostic reducers Count { accuracy: AccuracyTarget }, Sum { col: Option }, + Min { col: Option }, + Max { col: Option }, + Avg { col: Option }, + StdDev { col: Option, population: bool }, + Variance { col: Option, population: bool }, Quantile { col: Option, q: f64, accuracy: AccuracyTarget }, TopK { k: usize, accuracy: AccuracyTarget }, Cardinality { col: Option, accuracy: AccuracyTarget }, - Rate, Increase, Changes, Delta, IDelta, Deriv, Resets, // counter-derivative family - // .. native-histogram accessors, per-sample transforms, Extension + + // counter-aware streaming derivatives + Rate, + Increase, + + // further counter-derivative / range-vector functions + Changes, Delta, IDelta, Deriv, Resets, + PredictLinear { seconds: f64 }, + DoubleExpSmoothing { smoothing: f64, trend: f64 }, + LastOverTime, FirstOverTime, MadOverTime, + TsOfMinOverTime, TsOfMaxOverTime, TsOfFirstOverTime, TsOfLastOverTime, + + // native-histogram accessors + HistogramCount, HistogramSum, HistogramAvg, HistogramStdDev, HistogramStdVar, + HistogramFraction { lower: f64, upper: f64 }, + HistogramQuantile { q: f64 }, + + // per-sample transforms + Math(MathFunc), + TimeFn(TimeFunc), + + // presence + Absent, AbsentOverTime, PresentOverTime, + + // extended aggregation operators + Group, + CountValues { label: String }, + + // deployment-specific escape hatch (see "Design rules" above) — core + // treats this opaquely; the owning deployment defines and interprets + // `payload` itself, keyed by its own `ext_kind` tag + Extension { ext_kind: String, payload: serde_json::Value }, } ``` diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 0b18fb1d..1d865214 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -174,6 +174,16 @@ pub enum SummaryExpr { } ``` +The `sketch`/`sketch_input` field names above predate this doc's own +"summary" umbrella term (see the top of this doc) — they're kept +verbatim here because they're the real, current identifiers, not +because the naming is settled. Renaming them to `summary`/`summary_input` +is a reasonable follow-up now that "sketch" is understood as one kind of +summary (alongside exact accumulators), not the whole vocabulary — not +done in this pass because it's a breaking rename of a type every +`SummaryExecutor` implementer pattern-matches on, worth landing as its +own deliberate change rather than a docs-driven side effect. + Binding — turning a canonical `QueryExpr` into an `L4Node` tree: ```rust @@ -181,7 +191,16 @@ pub fn implement_tree(expr: &QueryExpr) -> Result, ImplementError>; pub fn implement_tree_with(expr: &QueryExpr, cost_model: &dyn CostModel) -> Result, ImplementError>; ``` -The per-intent decision binding drives, and its result type: +**"Implementation" is the answer to one question: how is this one +intent actually realized?** It's the return type of the per-intent +binding decision (`implementation_for`) — for a given `AggIntent`, +exactly one of: build an approximate sketch (bounded error, sized to +the accuracy target), build an exact accumulator (zero error, still +mergeable — see "Choosing a summary for an intent" above), or don't +summarize at all and evaluate the intent directly over raw data +(`PassThrough`). It's the same three-way choice this doc's "shape of a +summary-bound plan" section already describes in prose; this is that +decision's concrete type: ```rust pub enum Implementation { @@ -193,6 +212,17 @@ pub enum Implementation { pub fn implementation_for(intent: &AggIntent) -> Implementation; ``` +`Sketch` and `ExactAccumulator` carry identical shapes +(`{kind, params}`) — the split exists because binding needs to know, +right here, whether the chosen `kind` requires a `SummaryEstimate` +readout afterward (approximate) or *is* the answer once built (exact — +no estimate step). Collapsing them into one `Summary { kind, params }` +variant is possible in principle, but only if that same fact becomes +recoverable another way — e.g. a `SummaryKind::is_exact()` — since +today the variant tag *is* how the binder knows which one it has. Worth +doing together with the `sketch`/`summary` rename above rather than as +two separate breaking changes to the same call sites. + The one pluggable extension point at this layer — a deployment supplies its own `CostModel` to override which summary family is chosen and how its parameters are sized, without touching the binding pass itself. diff --git a/docs/l5-physical-plan.md b/docs/l5-physical-plan.md index 53b7ed23..8bb999df 100644 --- a/docs/l5-physical-plan.md +++ b/docs/l5-physical-plan.md @@ -77,7 +77,13 @@ deployment policy. Speculative — nothing here is built (see this doc's status note at the top). Kept as a concrete target to design against, not as an API to -depend on: +depend on. Every concrete-looking name below (`StageId`'s example +values, `Executor`'s addressing, an actual `TopologyDescriptor`) is +**interface only, in this repo** — ASAPController defines the shape a +deployment implements against; it never defines, reserves, or ships any +of the actual values. A deployment names its own stages, addresses its +own executors, and describes its own topology; nothing here is a fixed +vocabulary to conform to: ```rust pub trait PhysicalPlanner { @@ -91,13 +97,20 @@ pub trait TopologyDescriptor { fn edges(&self) -> &[StageEdge]; } -pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" +// `StageId` is an opaque, deployment-chosen string — ASAPController +// neither defines nor reserves any particular value. "edge" / "gateway" +// / "backend" / "in-process" below are one deployment's illustrative +// choice (roughly: close to data ingestion / an intermediate +// aggregation tier / a centralized serving tier / no network hop at +// all), not a fixed enum — a different deployment can and should name +// its own stages differently. +pub struct StageId(pub String); pub struct Executor { pub id: ExecutorId, pub stage: StageId, pub capabilities: ExecutorCaps, // memory budget, available summary backends, network neighbours - pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle + pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle — deployment-defined } // Stage-level allocation: given an L4 tree + a topology, decide which From f94b6bec95512066eb261f5a8a1b7d38a47fc45b Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 15:57:05 -0600 Subject: [PATCH 3/8] docs(l4): update Interface section now that sketch/summary rename + Implementation merge landed Both were called out as pending follow-ups in this PR's review; #170 implemented them. Updates the code snippets and prose here to match the real, current shape instead of describing the old one as still-pending. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 43 ++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 25 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 1d865214..c01b015f 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -161,28 +161,24 @@ pub enum SummaryExpr { Logical(Box), SummaryAgg { child: Rc, - sketch: SummaryKind, + summary: SummaryKind, params: SummaryParams, col: ColumnRef, reduction: Reduction, }, - SummaryJoin { outer: Rc, inner: Rc, key: ColumnRef, sketch: SummaryKind, params: SummaryParams }, + SummaryJoin { outer: Rc, inner: Rc, key: ColumnRef, summary: SummaryKind, params: SummaryParams }, SummarySubtract { left: Rc, right: Rc }, - SummaryDelete { sketch_input: Rc, key: ColumnRef }, - SummaryEstimate { sketch_input: Rc, query: SketchQuery }, + SummaryDelete { summary_input: Rc, key: ColumnRef }, + SummaryEstimate { summary_input: Rc, query: SketchQuery }, SummaryMerge { children: Vec> }, } ``` -The `sketch`/`sketch_input` field names above predate this doc's own -"summary" umbrella term (see the top of this doc) — they're kept -verbatim here because they're the real, current identifiers, not -because the naming is settled. Renaming them to `summary`/`summary_input` -is a reasonable follow-up now that "sketch" is understood as one kind of -summary (alongside exact accumulators), not the whole vocabulary — not -done in this pass because it's a breaking rename of a type every -`SummaryExecutor` implementer pattern-matches on, worth landing as its -own deliberate change rather than a docs-driven side effect. +The `summary`/`summary_input` field names match this doc's own "summary" +umbrella term (see the top of this doc) directly. An earlier revision of +this type used `sketch`/`sketch_input`, predating that umbrella term — +renamed in #170, landed together with the `Implementation` variant merge +described below. Binding — turning a canonical `QueryExpr` into an `L4Node` tree: @@ -204,24 +200,21 @@ decision's concrete type: ```rust pub enum Implementation { - Sketch { kind: SummaryKind, params: SummaryParams }, - ExactAccumulator { kind: SummaryKind, params: SummaryParams }, + Summary { kind: SummaryKind, params: SummaryParams }, PassThrough, } pub fn implementation_for(intent: &AggIntent) -> Implementation; ``` -`Sketch` and `ExactAccumulator` carry identical shapes -(`{kind, params}`) — the split exists because binding needs to know, -right here, whether the chosen `kind` requires a `SummaryEstimate` +An earlier revision of this type split `Summary` into two variants, +`Sketch{kind,params}` and `ExactAccumulator{kind,params}`, carrying +identical shapes — the split existed only because binding needed to +know, right here, whether the chosen `kind` requires a `SummaryEstimate` readout afterward (approximate) or *is* the answer once built (exact — -no estimate step). Collapsing them into one `Summary { kind, params }` -variant is possible in principle, but only if that same fact becomes -recoverable another way — e.g. a `SummaryKind::is_exact()` — since -today the variant tag *is* how the binder knows which one it has. Worth -doing together with the `sketch`/`summary` rename above rather than as -two separate breaking changes to the same call sites. +no estimate step). #170 collapsed them into the single `Summary` +variant above, recovering that same fact from `SummaryKind::is_exact()` +instead of from the variant tag. The one pluggable extension point at this layer — a deployment supplies its own `CostModel` to override which summary family is chosen and how @@ -258,7 +251,7 @@ pub trait SummaryExecutor { fn find_candidates( &self, - sketch: &SummaryKind, + summary: &SummaryKind, params: &SummaryParams, col: &ColumnRef, reduction: &Reduction, From 36075142e7f9cf3037f3e078575940ef68ecb9e0 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 18:22:12 -0600 Subject: [PATCH 4/8] docs(l3): explain Group and CountValues in the AggIntent listing Review comment on #169: these two were listed with no explanation while every other category got at least a grouping comment. Both need one because they don't fit the "reduce to one number per group" shape every other reducer follows here. Co-Authored-By: Claude Sonnet 5 --- docs/l3-intent-algebra.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md index 392ddeae..fa3d467c 100644 --- a/docs/l3-intent-algebra.md +++ b/docs/l3-intent-algebra.md @@ -250,7 +250,23 @@ pub enum AggIntent { // presence Absent, AbsentOverTime, PresentOverTime, - // extended aggregation operators + // extended aggregation operators — grouped by *sample value* rather + // than reducing to a single number per group, so both need a schema + // shape no other reducer in this list uses: + // + // - `Group`: emits a constant `1` per group ("does this group have any + // members at all"), independent of the input values — the output + // carries no information about the samples beyond their presence. + // Kept as its own intent rather than folded into `Sum`/`Count` + // because the value has nothing to do with what's being summed or + // counted. + // - `CountValues { label }`: groups the input further by each + // distinct *sample value* (not just the usual grouping keys), + // counting how many samples land in each. Since the sample value + // itself becomes part of the output's identity, this is the one + // reducer whose output schema gains a new column (a synthesized + // label named `label`, holding the stringified value) rather than + // just a single retyped aggregate column. Group, CountValues { label: String }, From 18375eb09bce2ef8610909d17ab99486a0fe7079 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 18:22:52 -0600 Subject: [PATCH 5/8] docs(l5): actually explain what a StageEdge/edge is, not just who owns StageId values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier fix for this review comment conflated two different questions — "what does an edge mean" and "who owns the concrete StageId values" — and only answered the second. Adds the missing explanation: an edge is a declared connection in the topology's connectivity graph, separate from which stages merely exist. Co-Authored-By: Claude Sonnet 5 --- docs/l5-physical-plan.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/l5-physical-plan.md b/docs/l5-physical-plan.md index 8bb999df..e58235de 100644 --- a/docs/l5-physical-plan.md +++ b/docs/l5-physical-plan.md @@ -94,6 +94,15 @@ pub trait PhysicalPlanner { pub trait TopologyDescriptor { fn stages(&self) -> &[StageDescriptor]; + // A `StageEdge` names a pair of stages data is allowed to flow + // between (e.g. "edge → backend" if that deployment's edge tier + // ships summaries up to a backend tier) — the topology's connectivity + // graph, distinct from which stages merely *exist* (`stages()` above). + // `StageAllocator::allocate` only assigns a piece of the plan to move + // from one stage to another along an edge this list actually + // contains; a `TopologyDescriptor` with no edge between two stages is + // how a deployment declares those two stages can't exchange data + // directly. fn edges(&self) -> &[StageEdge]; } From e1b3be2b13efba176d23b962add7b86f4ac78223 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 18:36:48 -0600 Subject: [PATCH 6/8] =?UTF-8?q?docs:=20resolve=206=20more=20PR=20#169=20re?= =?UTF-8?q?view=20comments=20=E2=80=94=20full=20listings,=20examples,=20co?= =?UTF-8?q?nstraints?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - l3: expand QueryExpr from an abbreviated slice to the full, real variant list (25 variants, matching crates/ir/src/intent_algebra/query_expr.rs). - l3: add a source-expression example per AggIntent variant, sourced from the accurate PromQL/SQL mappings already in agg_intent.rs's own doc comments. - l3: explain why the *OverTime range reducers can't be composed from Sort/Limit or any other generic node (no node exposes a TimeRange window's raw sample sequence as queryable rows). - l3: add a worked Schema example (sum by (job) (...)), showing the input/output shape difference an aggregate produces. - l2: add a concrete usage example for SchemaCatalog/Binder (PromQL's usage-derived default vs. a SQL catalog-backed case). - l4: list the per-SummaryExpr-node validity constraints (subtractable/ deletable/mergeable catalog flags, state-vs-value child requirements), correcting along the way which nodes actually carry a `summary` field of their own vs. which infer their kind from a child's output type. Co-Authored-By: Claude Sonnet 5 --- docs/l2-logical-plan.md | 30 +++++++++ docs/l3-intent-algebra.md | 131 ++++++++++++++++++++++++++++++++++-- docs/l4-summary-bound-ir.md | 32 +++++++++ 3 files changed, 186 insertions(+), 7 deletions(-) diff --git a/docs/l2-logical-plan.md b/docs/l2-logical-plan.md index dbf16425..2f54e939 100644 --- a/docs/l2-logical-plan.md +++ b/docs/l2-logical-plan.md @@ -109,6 +109,36 @@ reference. A catalog-backed language (or a future registry-backed one) implements `columns_for` to return a real, closed column set instead; nothing about `Binder` itself has to change for that swap. +For example, PromQL has no real catalog — a metric's label set is only +knowable from what the query itself references — so its front end binds +with the default: + +```rust +// PromQL: `sum by (job) (http_requests_total)`, no catalog available. +let schema = Binder::default().bind(&tree); +// -> Schema { columns: [ts, value, job], time_index: Some(0), closed: false } +// ("job" was seeded because the query references it; anything the +// query never mentions is simply absent from this schema) +``` + +SQL has a real, declared catalog, so its front end supplies one instead: + +```rust +struct SqlCatalog { /* wraps a table registry */ } +impl SchemaCatalog for SqlCatalog { + fn columns_for(&self, source: &str) -> Option> { + // "requests" -> its full declared column list, looked up from + // whatever table registry this deployment has, e.g.: + // Some(vec![Column::new("host", Utf8, false), Column::new("bytes", Int64, false)]) + .. + } +} +let schema = Binder::with_catalog(SqlCatalog { .. }).bind(&tree); +// -> Schema { columns: [host, bytes], time_index: None, closed: true } +// (the catalog's declared columns are used verbatim, regardless of +// which ones the query actually references) +``` + The L2 → L3 step every front end shares: ```rust diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md index fa3d467c..20645386 100644 --- a/docs/l3-intent-algebra.md +++ b/docs/l3-intent-algebra.md @@ -155,11 +155,26 @@ while a source with a declared key from an external catalog can. ## Interface The canonical `QueryExpr` is one Rust enum covering the whole relational -vocabulary — a representative slice, not the full list: +vocabulary, every variant: ```rust pub enum QueryExpr { + // ── leaves ──────────────────────────────────────────────────────── Scan { source: Source, predicates: Vec, schema: Schema }, + Ref { name: BindingName }, // a LetBinding reference + Scalar(f64), // a constant scalar literal + EvalTime, // the query evaluation time as a scalar + + // ── scalar/vector bridges ──────────────────────────────────────── + VectorFromScalar(Box), // promote a scalar to a label-less vector + ScalarFromVector(Box), // collapse a single-series vector to a scalar + + // ── per-row transforms ─────────────────────────────────────────── + Relabel { dst: String, value: L3Expr, child: Box }, + InfoJoin { selector: Vec, child: Box }, + Sample { by: GroupKeys, kind: SampleKind, child: Box }, + + // ── core relational ────────────────────────────────────────────── Filter { pred: Predicate, child: Box }, Project { cols: Vec, qualifier: Option, child: Box }, Aggregate { @@ -169,12 +184,36 @@ pub enum QueryExpr { having: Option, child: Box, }, - Window { .. }, Distinct { .. }, Merge { .. }, Join { .. }, SetOp { .. }, - Sort { .. }, Limit { .. }, LetBinding { .. }, Ref { .. }, - Subquery { .. }, TimeRange { .. }, TimeShift { .. }, WindowFunc { .. }, - BinaryOp { .. }, - // .. plus scalar/vector bridges and a small number of - // language-specific extension points with no general equivalent yet + + // ── windowing, dedup, set composition ──────────────────────────── + Window { kind: WindowKind, size: Duration, slide: Option, child: Box }, + Distinct { cols: Vec, child: Box }, + Merge { children: Vec }, // exact, n-ary UNION ALL + Join { kind: JoinKind, pred: Predicate, left: Box, right: Box }, + SetOp { kind: SetOpKind, all: bool, left: Box, right: Box }, + + // ── ordering / limiting ────────────────────────────────────────── + Sort { keys: Vec, partition_by: GroupKeys, child: Box }, + Limit { n: usize, offset: usize, child: Box }, + + // ── sharing and temporal wrappers ──────────────────────────────── + LetBinding { name: BindingName, expr: Box, child: Box }, + Subquery { range: Duration, resolution: Option, child: Box }, + TimeRange { range: Duration, child: Box }, + TimeShift { shift: TimeShift, child: Box }, + + // ── SQL analytic window functions ──────────────────────────────── + WindowFunc { + func: WindowFuncKind, + args: Vec, + partition_by: GroupKeys, + order_by: Vec, + output_name: String, + child: Box, + }, + + // ── arithmetic / comparison / boolean composition ──────────────── + BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, vector_match: Option }, } ``` @@ -277,6 +316,68 @@ pub enum AggIntent { } ``` +One example source expression per variant, PromQL unless noted (`v` stands in +for any instant/range vector selector): + +| `AggIntent` | Example | +|---|---| +| `Count` | `count(up)` | +| `Sum` | `sum(rate(http_requests_total[5m]))`; SQL `SUM(bytes)` | +| `Min` | `min(cpu_temp)` | +| `Max` | `max(cpu_temp)` | +| `Avg` | `avg(cpu_usage)` | +| `StdDev` | `stddev(latency_ms)`; SQL `STDDEV(col)` | +| `Variance` | `stdvar(latency_ms)`; SQL `VARIANCE(col)` | +| `Quantile` | `quantile(0.99, latency_ms)`; SQL `approx_percentile_cont(col, 0.99)` | +| `TopK` | `topk(5, http_requests_total)` | +| `Cardinality` | SQL `COUNT(DISTINCT user_id)`; the PromQL analogue is `count(...)` over a metric whose underlying storage is a cardinality sketch, not a literal PromQL function call | +| `Rate` | `rate(http_requests_total[5m])` | +| `Increase` | `increase(http_requests_total[5m])` | +| `Changes` | `changes(v[5m])` | +| `Delta` | `delta(v[5m])` | +| `IDelta` | `idelta(v[5m])` | +| `Deriv` | `deriv(v[5m])` | +| `Resets` | `resets(v[5m])` | +| `PredictLinear` | `predict_linear(v[5m], 3600)` | +| `DoubleExpSmoothing` | `double_exponential_smoothing(v[5m], 0.5, 0.5)` | +| `HistogramCount` | `histogram_count(v)` | +| `HistogramSum` | `histogram_sum(v)` | +| `HistogramAvg` | `histogram_avg(v)` | +| `HistogramStdDev` | `histogram_stddev(v)` | +| `HistogramStdVar` | `histogram_stdvar(v)` | +| `HistogramFraction` | `histogram_fraction(0.1, 0.5, v)` | +| `HistogramQuantile` | `histogram_quantile(0.99, v)` | +| `Math` | `abs(v)`, `sqrt(v)`, `ceil(v)`, … (one `MathFunc` per PromQL math/trig builtin) | +| `TimeFn` | `hour()`, `day_of_week(v)`, … (one `TimeFunc` per PromQL calendar builtin) | +| `Absent` | `absent(v)` | +| `AbsentOverTime` | `absent_over_time(v[5m])` | +| `PresentOverTime` | `present_over_time(v[5m])` | +| `Group` | `group(v)` | +| `CountValues` | `count_values("version", v)` | +| `LastOverTime` | `last_over_time(v[5m])` | +| `FirstOverTime` | `first_over_time(v[5m])` | +| `MadOverTime` | `mad_over_time(v[5m])` | +| `TsOfMinOverTime` | `ts_of_min_over_time(v[5m])` | +| `TsOfMaxOverTime` | `ts_of_max_over_time(v[5m])` | +| `TsOfFirstOverTime` | `ts_of_first_over_time(v[5m])` | +| `TsOfLastOverTime` | `ts_of_last_over_time(v[5m])` | +| `Extension` | deployment-defined — e.g. a deployment-specific membership-test function no core language has a builtin for | + +**Why the `*OverTime` reducers (`LastOverTime` … `TsOfLastOverTime`) each need +their own intent, rather than composing from `Sort`/`Limit` or another generic +node.** They all reduce a single series' *raw sample sequence inside a range +window* to one value — but nothing else in this vocabulary exposes that raw +sequence as rows a generic operator could sort or limit. `Sort`/`Limit` +operate on the relation `Aggregate` already produced (one row per series, +post-reduction); a `TimeRange` window's underlying samples are never +materialized as a queryable row set at this layer, only fed directly into +whichever `AggIntent` sits above it. So there is no composition available to +express "the timestamp of this window's minimum sample" from generic parts — +each of these is the only path that can reach into the window's raw stream +for its particular statistic, which is exactly the design rule 1 exception +(a genuinely different computational access pattern, not an ordinary +composition of existing operators). + And the schema every edge in the tree carries: ```rust @@ -287,3 +388,19 @@ pub struct Schema { pub closed: bool, } ``` + +For example, `sum by (job) (http_requests_total)` binds to an `Aggregate` +whose input schema is `{ columns: [ts, value, job], time_index: Some(0), +unique_keys: [], closed: false }` (PromQL — open, since a metric's label +set is a superset the runtime may exceed) and whose *output* schema — +after `Reduction::by([job])` collapses every other row — is: + +```rust +Schema { + columns: vec![Column::new("job", DataType::Utf8, true), Column::new("sum", DataType::Float64, false)], + time_index: None, // a cross-series reduction collapses the time axis + unique_keys: vec![vec![0]], // grouping by job makes job unique in the output + closed: true, // by(...) enumerates its columns exactly — this is + // where an open input schema freezes to closed +} +``` diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index c01b015f..1898eb71 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -180,6 +180,38 @@ this type used `sketch`/`sketch_input`, predating that umbrella term — renamed in #170, landed together with the `Implementation` variant merge described below. +Per-node validity constraints — each is a summary-catalog fact (see +"Summary catalog" above) checked at planning time, before this shape ever +reaches serving-time execution: + +- **`Logical`** — none; wraps an arbitrary unrewritten `QueryExpr` subtree. +- **`SummaryAgg`** — none beyond what `col`'s intent already required + (accuracy target compatible with `summary`); this is the leaf every + other node's constraints are checked *against*. +- **`SummaryJoin`** — only emitted where a join-specific summary technique + applies (issue-tracked as a future `Bind*OnJoin` rule); no such rule + exists in the base design yet, so this variant has no live producer. +- **`SummarySubtract`** — note this variant carries no `summary` field of + its own; the kind is read off `left`/`right`'s own output type instead. + That shared kind's catalog entry must set `subtractable`, and `left` + and `right` must agree on `(kind, params)` (their output schemas must + carry the identical summary type for the operation to type-check). +- **`SummaryDelete`** — likewise no `summary` field of its own; the kind + is `summary_input`'s output type, and that kind's catalog entry must + set `deletable`. +- **`SummaryEstimate`** — `query`'s `SketchQuery` variant must be one + `summary_input`'s summary kind actually supports (e.g. a `Quantile` + query only makes sense against a quantile-sketch kind) — this isn't a + single boolean catalog flag like the two above, since which queries a + kind answers is closer to that kind's whole reason for existing. +- **`SummaryMerge`** — also carries no `summary` field; `children` must be + non-empty, every child must itself produce summary *state* (another + `SummaryAgg` or `SummaryMerge` — never a `SummaryEstimate` readout or a + `Logical` value, which have already collapsed to a plain value), and + every child must agree on `(kind, params)`, with that shared kind's + catalog entry setting `mergeable`. These are exactly the "Nested + composition" rules above, restated per field. + Binding — turning a canonical `QueryExpr` into an `L4Node` tree: ```rust From 94137a3e887f159396e6ad402bf4d57f261669b3 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 28 Jul 2026 18:43:53 -0600 Subject: [PATCH 7/8] docs(l4): turn the per-SummaryExpr-node constraints into a table + a diagram Prose bullet list was hard to scan against 7 variants. Table gives a one-glance summary; the mermaid diagram visualizes the rule that's hardest to hold in your head from prose alone (SummaryMerge only accepts state-producing children, never a value-producing one). Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 60 ++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 27 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 1898eb71..0f0c4da2 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -184,33 +184,39 @@ Per-node validity constraints — each is a summary-catalog fact (see "Summary catalog" above) checked at planning time, before this shape ever reaches serving-time execution: -- **`Logical`** — none; wraps an arbitrary unrewritten `QueryExpr` subtree. -- **`SummaryAgg`** — none beyond what `col`'s intent already required - (accuracy target compatible with `summary`); this is the leaf every - other node's constraints are checked *against*. -- **`SummaryJoin`** — only emitted where a join-specific summary technique - applies (issue-tracked as a future `Bind*OnJoin` rule); no such rule - exists in the base design yet, so this variant has no live producer. -- **`SummarySubtract`** — note this variant carries no `summary` field of - its own; the kind is read off `left`/`right`'s own output type instead. - That shared kind's catalog entry must set `subtractable`, and `left` - and `right` must agree on `(kind, params)` (their output schemas must - carry the identical summary type for the operation to type-check). -- **`SummaryDelete`** — likewise no `summary` field of its own; the kind - is `summary_input`'s output type, and that kind's catalog entry must - set `deletable`. -- **`SummaryEstimate`** — `query`'s `SketchQuery` variant must be one - `summary_input`'s summary kind actually supports (e.g. a `Quantile` - query only makes sense against a quantile-sketch kind) — this isn't a - single boolean catalog flag like the two above, since which queries a - kind answers is closer to that kind's whole reason for existing. -- **`SummaryMerge`** — also carries no `summary` field; `children` must be - non-empty, every child must itself produce summary *state* (another - `SummaryAgg` or `SummaryMerge` — never a `SummaryEstimate` readout or a - `Logical` value, which have already collapsed to a plain value), and - every child must agree on `(kind, params)`, with that shared kind's - catalog entry setting `mergeable`. These are exactly the "Nested - composition" rules above, restated per field. +| Node | `summary` field | Constraint | +|---|---|---| +| `Logical` | — | none — wraps an arbitrary unrewritten `QueryExpr` subtree | +| `SummaryAgg` | own | none beyond `col`'s intent already requiring an accuracy target compatible with `summary`; this is the leaf every other row's constraints are checked *against* | +| `SummaryJoin` | own | only emitted by a join-specific `Bind*OnJoin` rule — none exist yet in the base design, so this variant has no live producer | +| `SummarySubtract` | read from `left`/`right` | `left`/`right` agree on `(kind, params)`; that kind's catalog entry sets `subtractable` | +| `SummaryDelete` | read from `summary_input` | that kind's catalog entry sets `deletable` | +| `SummaryEstimate` | read from `summary_input` | `query`'s `SketchQuery` variant is one that kind actually supports (not a single boolean flag — closer to that kind's whole reason for existing) | +| `SummaryMerge` | read from `children`, which must all agree | `children` non-empty; every child produces summary *state*, never a value (see diagram below); that shared kind's catalog entry sets `mergeable` | + +The trickiest of these to hold in your head is `SummaryMerge`'s "children +must produce state, not a value" rule — everything here either produces +partial summary **state** (still mergeable, not yet read out) or a final +**value** (already collapsed, e.g. by a readout). Only a state-producing +node may feed a merge: + +```mermaid +flowchart LR + subgraph State["produce summary state"] + SA["SummaryAgg"] + SM["SummaryMerge"] + end + subgraph Value["produce a plain value"] + SE["SummaryEstimate"] + LG["Logical"] + end + SA -->|"valid merge child"| SM + SM -->|"valid merge child (nested)"| SM + SA --> SE + SM --> SE + SE -. "✗ already a value" .-> SM + LG -. "✗ already a value" .-> SM +``` Binding — turning a canonical `QueryExpr` into an `L4Node` tree: From 423c36a1cb53379b936212dd5c18855f14ca55e2 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Tue, 4 Aug 2026 08:56:57 -0400 Subject: [PATCH 8/8] docs(l4): a taxonomy and recipe for composing summaries (bespoke) (#174) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(l4): design proposals for #171 and #172 composition gaps Stacked on #169's Interface section. Adds two concrete proposals: - #171 (exact <-> summary composition, either nesting order): a new SummaryFold node for an outer exact fold over an inner realized summary, computed generically by the shared execute walk with no new SummaryExecutor method; and a relaxed SummaryAgg input-column rule letting an outer summary read directly off an exact accumulator's own state (zero added error), while explicitly refusing the approximate- child case since that's #172's gap, not this one's. - #172 (nested approximate-over-approximate error propagation): a SummaryKind::implied_accuracy inverse-sizing function, a new CostModel::size_params child_accuracy parameter (opt-in, default ignores it, matching this trait's existing default-body convention), and a paired accepts_nested_approx gate so an outer approximate summary over an approximate child's estimate refuses by default instead of silently double-approximating. Co-Authored-By: Claude Sonnet 5 * docs(l4): reframe #171/#172 proposal as a shared accuracy algebra, fold in #173 Replaces the earlier implementation-narrative version (which described specific match arms/functions in boundary.rs/bind.rs) with a design-level treatment: a formal (epsilon, delta) accuracy guarantee every SummaryKind already satisfies, a composition theorem (triangle inequality + union bound) for building a summary over another summary's output, and clean interfaces (Accuracy, ColumnRef::FromSummary, CostModel::size_params/ accepts_nested_approx, FoldOp/SummaryFold) derived from it. Shows #171 direction 2 (exact child) and #172 (approximate child) are the same composition formula at Accuracy::EXACT vs. Accuracy > 0, rather than two separate rules. Folds in #173: generalizes FromSummary/SummaryFold to N children for Hydra-style sketch-of-sketches, and adds a Bounded Accuracy variant (deterministic interval radius) for QTree-style range trees alongside the Probabilistic (epsilon, delta) variant. Co-Authored-By: Claude Sonnet 5 * docs(l4): separate the design from which issue it solves; norm-aware composition Two changes, both from review: - Restructure into "the design" (guarantee, composition theorem, three interfaces — no issue numbers threaded through the reasoning) followed by a single "Which issue this solves" section mapping each piece back to #171/#172/#173, instead of interleaving issue callouts throughout. - The composition theorem assumed a single scalar Lipschitz constant L under an implicit pointwise/L1 error model. That's wrong for this catalog's own CountSketch/CountSketchWithHeap, whose guarantee is stated against the L2 norm of the residual frequency vector, not L1 -- a materially different, not-directly-comparable quantity on skewed data. Accuracy and compose now carry an explicit ErrorNorm; compose returns None on a norm mismatch instead of silently assuming L=1. Folded Accuracy::Bounded (deterministic radius, for #173's QTree case) into the same enum/compose function rather than a separate type. Co-Authored-By: Claude Sonnet 5 * docs(l4): replace unified accuracy algebra with three grounded composition patterns Surveyed how real systems actually compose summaries (DGIM SICOMP'02, UnivMon SIGCOMM'16, Hydra VLDB'22, PromSketch VLDB'25) rather than inventing one generic accuracy algebra. Finding: there is no single law -- three structurally distinct patterns exist, only two have a governing theorem, and the pattern #171/#172 actually describe has none at all. - Pattern A (same statistic across partitions -- EH/DGIM windowing): real, DGIM Theorem 6/7 grounds it precisely, but composes the same statistic across a partition of the same data, not different statistics along a query's data-flow -- out of scope for #171/#172/#173, flagged as a separate future gap (sliding-window range-vector queries). - Pattern B (hash-collision routing -- Hydra's sketch of sketches): Theorem 2's asymmetric bound (multiplicative + additive-vs-global-total) doesn't fit the shared Accuracy shape -- Hydra and QTree each need their own SummaryKind fed directly from raw data, not a generalization of FromSummary/SummaryFold. This replaces the previous (incorrect) N-ary FromSummary framing for #173. - Pattern D (heterogeneous sequential nesting -- what #171/#172 actually ask about): no borrowed theorem exists for this shape in any of the four systems surveyed -- all four avoid it structurally. Split into D1 (inner exact, composes free -- #171 direction 2), D2 (outer exact fold over inner summary -- #171 direction 1, SummaryFold), and D3 (outer sketch over an inner statistic with no exact sub-linear form -- #172's actual remaining scope, narrower than "any nested approximation"). D3's compose()/accepts_nested_approx gate is now labeled honestly as this design's own first-principles proposal, not a literature result -- defaulting closed matches what every system surveyed actually does (none of them re-sketch another sketch's readout). Co-Authored-By: Claude Sonnet 5 * docs(l4): recipe (not formula) for compound summary bounds, state not readout Reframes the design around the actual structural move every surveyed system (DGIM, UnivMon, Hydra, PromSketch) makes: never compose past a readout -- build the outer structure directly over the inner's state. Replaces the "three patterns" framing with a four-type taxonomy (same- kind merge; inner-exact; heterogeneous-with-derivable-bound; heterogeneous- undecidable) and, for the interesting type-3 case, a four-step recipe for deriving a compound bound instead of a formula: (1) treat inner's state as the outer's input schema, (2) check the outer's own construction algorithm can actually run over that state, (3) if so, re-derive -- not reuse -- the outer's own concentration argument against the composed randomness, (4) accept the resulting bound is pair-specific, not universal (DGIM's (1+e)^2*Cf^2/k+Cf-1+e and Hydra's Gi(1+/-e)+eps*GS are worked examples of the same recipe, not the same formula). Drops the generic Accuracy::compose()/Sensitivity mechanism from the previous draft -- it implied a bound could be computed from kind- independent accuracy values alone, which the DGIM/Hydra worked examples show is false. CostModel::size_params_composed now takes the child's concrete (kind, params) directly and has no default body: the deployment supplies its own pair-specific derivation, or the composition is refused (accepts_composed_state defaults false). FromSummary(Rc) is now constrained to state-producing nodes only (SummaryAgg/SummaryMerge), reusing the same rule this doc's SummaryMerge table already enforces, rather than allowing reference to a readout. Co-Authored-By: Claude Sonnet 5 * docs(l4): restore readout-based composition as a distinct, legitimate mechanism Correction: the previous commit banned composing over a readout entirely, requiring FromSummary to reference only state-producing nodes. That over-corrected -- readout-based composition (mechanism 3a) is a real, general mechanism in its own right: any inner reporting a public (eps, delta) can be composed via a Lipschitz-style Sensitivity argument (triangle inequality + union bound), without needing access to the inner's raw state at all. It's strictly looser than state-based composition (3b, the DGIM/Hydra recipe) when 3b is available, but it's the only option when the inner's state genuinely isn't accessible (e.g. across a service boundary) or when 3b's construction-compatibility check fails. ColumnRef now has two variants -- FromReadout (value-producing node) and FromState (state-producing node) -- routing to two separate CostModel gates: accepts_readout_composition/sensitivity_for/compose_over_readout for 3a, accepts_state_composition/size_params_from_state for 3b. A deployment facing a novel (outer, inner) pair has a real choice between the two, not one closed gate. Co-Authored-By: Claude Sonnet 5 * docs(l4): drop the "Which issue this solves" mapping section Requested in review. The design (four compound types, mechanisms 3a/3b) stands on its own; the issue-by-issue mapping was redundant restatement. Co-Authored-By: Claude Sonnet 5 * docs(l4): tighten prose, cut redundant explanation No content change -- same taxonomy, same two mechanisms, same recipe, same interfaces. Trims repeated justification and meta-commentary about prior review rounds, shortens sentences, removes duplicate restatement of the DGIM/Hydra examples across steps. Co-Authored-By: Claude Sonnet 5 * docs(l4): make the 3b recipe steps readable -- one idea per step, examples in a table The four recipe steps each interleaved DGIM and Hydra inline, forcing readers to track two examples at once inside every sentence. Splits the steps (now plain, one idea each) from the worked examples (DGIM/EH, Hydra, and a KLL-over-HLL counter-example), laid out as a comparison table keyed to steps 2 and 3. Co-Authored-By: Claude Sonnet 5 * docs(l4): rename DGIM/EH to Exponential Histogram [Datar et al., SICOMP'02] Co-Authored-By: Claude Sonnet 5 --------- Co-authored-by: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 177 +++++++++++++++++++++++++++++++++++- 1 file changed, 176 insertions(+), 1 deletion(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 0f0c4da2..7b1683e2 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -117,7 +117,10 @@ already-computed value, at query time." Every design considered so far only builds summaries incrementally from raw samples, at ingest time; there's no established answer for building one from a query-time-derived scalar. A deployment that hits this shape should -treat it as unsupported rather than assume either answer. +treat it as unsupported rather than assume either answer. The design +proposal below narrows exactly which cases of this remain genuinely +open (see "Pattern D" below) — it is not fully open anymore, but it is +not fully resolved either. ### What's out of scope here @@ -308,3 +311,175 @@ pub fn execute(node: &L4Node, exec: &E) -> Result (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c +``` + +by the triangle inequality plus a union bound on the two failure events: + +```rust +pub enum ErrorNorm { L1, L2, Pointwise } + +pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } + +impl Accuracy { + /// `None` on a norm mismatch — refuse rather than apply an `L` that + /// was never derived for it. `Sensitivity` is always the + /// deployment's own claim (linear aggregates like Sum/Count have a + /// provable `L=1`; rank-based ones like Quantile/TopK only have + /// `L≈1` under an unproven local-density assumption). + pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + _ => None, + } + } +} +``` + +Computable for any pair given a justified `Sensitivity`, but looser than +3b, since it treats the inner as an opaque noisy scalar. + +#### 3b: composing over the inner's state (the recipe) + +Every sketch here is a randomized construction `state = Φ(input)`, and +its bound is proved by a specific argument over that construction — a +Markov bound over hash-collision mass for CMS, a compaction invariant for +KLL, a variance calculation over the max-order-statistic register for +HLL. There's no shortcut around re-examining that argument: + +1. **Feed the outer sketch the inner's state, not its answer.** Use the + inner's raw, pre-`SummaryEstimate` state — typed here as + `L4DataType::Sketch(kind, params)` — as the outer's input, instead of + a scalar readout. +2. **Check the outer's own build procedure can actually run on that + state.** Does feeding it the inner's state, instead of raw data, still + make sense? This can fail — see the worked examples below for a case + where it does and one where it clearly doesn't. If it fails, the pair + is type 4: refused. +3. **If it works, prove a new bound — don't reuse the outer's old one.** + The outer's published bound assumed clean raw input; it says nothing + about input that's itself another sketch's noisy state. The outer's + own proof technique has to be redone against this two-stage + construction. +4. **Expect a different bound for each new pair.** There's no fixed + shape this converges to — the two worked examples below land on two + different formulas. + +**Worked examples.** Exponential Histogram (a sketch built by concatenating other +sketches' state across time buckets) and Hydra (a sketch that routes +into other sketches by hash) both follow this recipe: + +| | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound | +|---|---|---| +| Exponential Histogram | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | +| Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. | +| *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) | + +Neither Exponential Histogram's nor Hydra's bound came from combining two pre-existing +formulas — both required redoing the outer's own proof for the two-layer +construction. A third, novel pair should not be expected to land on +either shape. + +#### Interface + +Two `ColumnRef` variants make which mechanism a query uses explicit at +the type level, reusing the state/value distinction `SummaryMerge` +already enforces (`SummaryAgg`/`SummaryMerge` produce state; +`SummaryEstimate`/`Logical` produce a value): + +```rust +pub enum ColumnRef { + Named(String), + Qualified { table: String, name: String }, + SampleValue, + FromReadout(Rc), // NEW — 3a; must be value-producing + FromState(Rc), // NEW — 3b; must be state-producing +} +``` + +Each routes to its own half of `CostModel`, both defaulting closed: + +```rust +pub trait CostModel { + /// 3a: has the deployment justified a Sensitivity for this pair? + fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + fn sensitivity_for(&self, outer: &AggIntent, child_kind: &SummaryKind) -> Option { + None + } + + /// 3b: has the deployment derived a bound for this pair? + fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + /// No default body — each pair gets its own derivation. Takes the + /// child's concrete (kind, params), not an abstracted accuracy + /// value, since the derivation depends on the specific pair. + fn size_params_from_state( + &self, + kind: SummaryKind, + intent: &AggIntent, + target_eps: f64, + target_delta: f64, + child: (&SummaryKind, &SummaryParams), + ) -> SummaryParams; +} +``` + +A deployment facing a new pair has a real choice: try 3b, fall back to +3a, or refuse. + +`Accuracy` (`implied_accuracy`, `is_exact`) stays a reporting type — what +a single `SummaryKind` guarantees on its own — used for type 2 and as +`compose_over_readout`'s input/output. It isn't a composition primitive +for 3b; 3b's bound comes from the recipe, not from a function of two +`Accuracy` values.