diff --git a/docs/l1-query-language.md b/docs/l1-query-language.md index 50757d9a..5f434b0b 100644 --- a/docs/l1-query-language.md +++ b/docs/l1-query-language.md @@ -67,3 +67,28 @@ A front end is expected to cleanly reject a shape it can't yet represent — rather than silently mis-lowering it — since a resolvable-later gap is one design choice away from becoming a correctness bug if it's lowered wrong instead of rejected outright. + +## Interface + +There is no shared L1 trait — each front end exposes its own free +functions, differing in language-specific arguments (SQL takes a +catalog + dialect; PromQL doesn't) but converging on the same return +type once lowering finishes: + +```rust +// PromQL +pub fn lower_promql(query: &str, accuracy: AccuracyTarget) -> Result; +pub fn lower_promql_batch(workload: &QueryWorkload) -> Vec>; + +// SQL +pub async fn lower_sql(query: &str, catalog: &SqlCatalog, accuracy: AccuracyTarget) -> Result; +pub async fn lower_sql_dialect(query: &str, catalog: &SqlCatalog, dialect: SqlDialect, accuracy: AccuracyTarget) -> Result; +pub async fn lower_sql_batch(workload: &QueryWorkload, catalog: &SqlCatalog) -> Vec>; +``` + +Both converge on the canonical L3 `QueryExpr` — the interface that +actually matters lives one step down, at the shared L2 → L3 step this +doc already describes (see [`l2-logical-plan.md`](./l2-logical-plan.md#interface)). +SQL's extra `catalog`/`dialect` parameters are exactly the schema-source +asymmetry covered above: SQL supplies its own resolved schema up front; +PromQL doesn't have one to supply, so it has no equivalent parameter. diff --git a/docs/l2-logical-plan.md b/docs/l2-logical-plan.md index ba80657c..2f54e939 100644 --- a/docs/l2-logical-plan.md +++ b/docs/l2-logical-plan.md @@ -86,3 +86,65 @@ row): an already-cataloged source can carry a real one through unchanged, while a usage-derived schema has no way to prove one, and so never gets one. Why a unique key matters is covered in [`l3-intent-algebra.md`](./l3-intent-algebra.md#unique-keys-and-cross-query-sharing). + +## Interface + +The one real extension point at this layer is the schema source a +front end's leaves resolve against: + +```rust +pub trait SchemaCatalog { + fn columns_for(&self, source: &str) -> Option>; +} + +pub struct Binder { .. } +impl Binder { + pub fn bind(&self, tree: &QueryExpr) -> Schema; +} +``` + +The default `Binder` (`UsageDerivedCatalog`) implements this by always +returning `None` — a schema built purely from what the query happens to +reference. A catalog-backed language (or a future registry-backed one) +implements `columns_for` to return a real, closed column set instead; +nothing about `Binder` itself has to change for that swap. + +For example, PromQL has no real catalog — a metric's label set is only +knowable from what the query itself references — so its front end binds +with the default: + +```rust +// PromQL: `sum by (job) (http_requests_total)`, no catalog available. +let schema = Binder::default().bind(&tree); +// -> Schema { columns: [ts, value, job], time_index: Some(0), closed: false } +// ("job" was seeded because the query references it; anything the +// query never mentions is simply absent from this schema) +``` + +SQL has a real, declared catalog, so its front end supplies one instead: + +```rust +struct SqlCatalog { /* wraps a table registry */ } +impl SchemaCatalog for SqlCatalog { + fn columns_for(&self, source: &str) -> Option> { + // "requests" -> its full declared column list, looked up from + // whatever table registry this deployment has, e.g.: + // Some(vec![Column::new("host", Utf8, false), Column::new("bytes", Int64, false)]) + .. + } +} +let schema = Binder::with_catalog(SqlCatalog { .. }).bind(&tree); +// -> Schema { columns: [host, bytes], time_index: None, closed: true } +// (the catalog's declared columns are used verbatim, regardless of +// which ones the query actually references) +``` + +The L2 → L3 step every front end shares: + +```rust +pub fn convert_root(legacy: &QueryExpr, accuracy: &AccuracyTarget) -> Result; +``` + +Takes L2's per-language tree in, returns L3's canonical tree out — +binding, structural conversion, and canonicalization in one call, so no +front end can reach L3 through any other path. diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md index 5b69e5a4..20645386 100644 --- a/docs/l3-intent-algebra.md +++ b/docs/l3-intent-algebra.md @@ -151,3 +151,256 @@ key, sharing is unsound and each consumer must recompute independently — which is why a source with no way to prove a key (an open, usage-derived schema) can never participate in this kind of sharing, while a source with a declared key from an external catalog can. + +## Interface + +The canonical `QueryExpr` is one Rust enum covering the whole relational +vocabulary, every variant: + +```rust +pub enum QueryExpr { + // ── leaves ──────────────────────────────────────────────────────── + Scan { source: Source, predicates: Vec, schema: Schema }, + Ref { name: BindingName }, // a LetBinding reference + Scalar(f64), // a constant scalar literal + EvalTime, // the query evaluation time as a scalar + + // ── scalar/vector bridges ──────────────────────────────────────── + VectorFromScalar(Box), // promote a scalar to a label-less vector + ScalarFromVector(Box), // collapse a single-series vector to a scalar + + // ── per-row transforms ─────────────────────────────────────────── + Relabel { dst: String, value: L3Expr, child: Box }, + InfoJoin { selector: Vec, child: Box }, + Sample { by: GroupKeys, kind: SampleKind, child: Box }, + + // ── core relational ────────────────────────────────────────────── + Filter { pred: Predicate, child: Box }, + Project { cols: Vec, qualifier: Option, child: Box }, + Aggregate { + reduction: Reduction, + aggs: Vec, + output_names: Vec, + having: Option, + child: Box, + }, + + // ── windowing, dedup, set composition ──────────────────────────── + Window { kind: WindowKind, size: Duration, slide: Option, child: Box }, + Distinct { cols: Vec, child: Box }, + Merge { children: Vec }, // exact, n-ary UNION ALL + Join { kind: JoinKind, pred: Predicate, left: Box, right: Box }, + SetOp { kind: SetOpKind, all: bool, left: Box, right: Box }, + + // ── ordering / limiting ────────────────────────────────────────── + Sort { keys: Vec, partition_by: GroupKeys, child: Box }, + Limit { n: usize, offset: usize, child: Box }, + + // ── sharing and temporal wrappers ──────────────────────────────── + LetBinding { name: BindingName, expr: Box, child: Box }, + Subquery { range: Duration, resolution: Option, child: Box }, + TimeRange { range: Duration, child: Box }, + TimeShift { shift: TimeShift, child: Box }, + + // ── SQL analytic window functions ──────────────────────────────── + WindowFunc { + func: WindowFuncKind, + args: Vec, + partition_by: GroupKeys, + order_by: Vec, + output_name: String, + child: Box, + }, + + // ── arithmetic / comparison / boolean composition ──────────────── + BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, vector_match: Option }, +} +``` + +`reduction` is a field *on* the `Aggregate` variant itself — not a +separate node in the tree, and not something any other variant carries. +It answers a question only `Aggregate` ever needs to ask: is this node +collapsing rows at all, and if so, by which (possibly empty) key set — +or does it have no grouping concept to begin with. Making that an +explicit field, rather than something a consumer infers from whether a +key list happens to be empty, is deliberate: the two cases are easy to +conflate (both can present as "empty keys") but require opposite +handling downstream — see [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md#interface)'s +`SummaryExecutor::find_candidates` for where that distinction is +actually load-bearing. + +The two small types that field's shape turns on, in full — neither is a +tree node either; both are plain data reachable only through +`Aggregate.reduction`, and `GroupKeys` only exists at all when +`reduction` is `Reduce` (it's meaningless for `PerEntity`, which is +exactly why it isn't a sibling field instead): + +```rust +pub enum Reduction { + Reduce(GroupKeys), + PerEntity, +} + +pub struct GroupKeys { /* private fields */ } +impl GroupKeys { + pub fn by(keys: Vec) -> Self; // keep these columns + pub fn without(keys: Vec) -> Self; // exclude these columns + pub fn is_without(&self) -> bool; + pub fn keys(&self) -> &[ColumnId]; +} +``` + +`AggIntent` — the full vocabulary: + +```rust +pub enum AggIntent { + // data-model-agnostic reducers + Count { accuracy: AccuracyTarget }, + Sum { col: Option }, + Min { col: Option }, + Max { col: Option }, + Avg { col: Option }, + StdDev { col: Option, population: bool }, + Variance { col: Option, population: bool }, + Quantile { col: Option, q: f64, accuracy: AccuracyTarget }, + TopK { k: usize, accuracy: AccuracyTarget }, + Cardinality { col: Option, accuracy: AccuracyTarget }, + + // counter-aware streaming derivatives + Rate, + Increase, + + // further counter-derivative / range-vector functions + Changes, Delta, IDelta, Deriv, Resets, + PredictLinear { seconds: f64 }, + DoubleExpSmoothing { smoothing: f64, trend: f64 }, + LastOverTime, FirstOverTime, MadOverTime, + TsOfMinOverTime, TsOfMaxOverTime, TsOfFirstOverTime, TsOfLastOverTime, + + // native-histogram accessors + HistogramCount, HistogramSum, HistogramAvg, HistogramStdDev, HistogramStdVar, + HistogramFraction { lower: f64, upper: f64 }, + HistogramQuantile { q: f64 }, + + // per-sample transforms + Math(MathFunc), + TimeFn(TimeFunc), + + // presence + Absent, AbsentOverTime, PresentOverTime, + + // extended aggregation operators — grouped by *sample value* rather + // than reducing to a single number per group, so both need a schema + // shape no other reducer in this list uses: + // + // - `Group`: emits a constant `1` per group ("does this group have any + // members at all"), independent of the input values — the output + // carries no information about the samples beyond their presence. + // Kept as its own intent rather than folded into `Sum`/`Count` + // because the value has nothing to do with what's being summed or + // counted. + // - `CountValues { label }`: groups the input further by each + // distinct *sample value* (not just the usual grouping keys), + // counting how many samples land in each. Since the sample value + // itself becomes part of the output's identity, this is the one + // reducer whose output schema gains a new column (a synthesized + // label named `label`, holding the stringified value) rather than + // just a single retyped aggregate column. + Group, + CountValues { label: String }, + + // deployment-specific escape hatch (see "Design rules" above) — core + // treats this opaquely; the owning deployment defines and interprets + // `payload` itself, keyed by its own `ext_kind` tag + Extension { ext_kind: String, payload: serde_json::Value }, +} +``` + +One example source expression per variant, PromQL unless noted (`v` stands in +for any instant/range vector selector): + +| `AggIntent` | Example | +|---|---| +| `Count` | `count(up)` | +| `Sum` | `sum(rate(http_requests_total[5m]))`; SQL `SUM(bytes)` | +| `Min` | `min(cpu_temp)` | +| `Max` | `max(cpu_temp)` | +| `Avg` | `avg(cpu_usage)` | +| `StdDev` | `stddev(latency_ms)`; SQL `STDDEV(col)` | +| `Variance` | `stdvar(latency_ms)`; SQL `VARIANCE(col)` | +| `Quantile` | `quantile(0.99, latency_ms)`; SQL `approx_percentile_cont(col, 0.99)` | +| `TopK` | `topk(5, http_requests_total)` | +| `Cardinality` | SQL `COUNT(DISTINCT user_id)`; the PromQL analogue is `count(...)` over a metric whose underlying storage is a cardinality sketch, not a literal PromQL function call | +| `Rate` | `rate(http_requests_total[5m])` | +| `Increase` | `increase(http_requests_total[5m])` | +| `Changes` | `changes(v[5m])` | +| `Delta` | `delta(v[5m])` | +| `IDelta` | `idelta(v[5m])` | +| `Deriv` | `deriv(v[5m])` | +| `Resets` | `resets(v[5m])` | +| `PredictLinear` | `predict_linear(v[5m], 3600)` | +| `DoubleExpSmoothing` | `double_exponential_smoothing(v[5m], 0.5, 0.5)` | +| `HistogramCount` | `histogram_count(v)` | +| `HistogramSum` | `histogram_sum(v)` | +| `HistogramAvg` | `histogram_avg(v)` | +| `HistogramStdDev` | `histogram_stddev(v)` | +| `HistogramStdVar` | `histogram_stdvar(v)` | +| `HistogramFraction` | `histogram_fraction(0.1, 0.5, v)` | +| `HistogramQuantile` | `histogram_quantile(0.99, v)` | +| `Math` | `abs(v)`, `sqrt(v)`, `ceil(v)`, … (one `MathFunc` per PromQL math/trig builtin) | +| `TimeFn` | `hour()`, `day_of_week(v)`, … (one `TimeFunc` per PromQL calendar builtin) | +| `Absent` | `absent(v)` | +| `AbsentOverTime` | `absent_over_time(v[5m])` | +| `PresentOverTime` | `present_over_time(v[5m])` | +| `Group` | `group(v)` | +| `CountValues` | `count_values("version", v)` | +| `LastOverTime` | `last_over_time(v[5m])` | +| `FirstOverTime` | `first_over_time(v[5m])` | +| `MadOverTime` | `mad_over_time(v[5m])` | +| `TsOfMinOverTime` | `ts_of_min_over_time(v[5m])` | +| `TsOfMaxOverTime` | `ts_of_max_over_time(v[5m])` | +| `TsOfFirstOverTime` | `ts_of_first_over_time(v[5m])` | +| `TsOfLastOverTime` | `ts_of_last_over_time(v[5m])` | +| `Extension` | deployment-defined — e.g. a deployment-specific membership-test function no core language has a builtin for | + +**Why the `*OverTime` reducers (`LastOverTime` … `TsOfLastOverTime`) each need +their own intent, rather than composing from `Sort`/`Limit` or another generic +node.** They all reduce a single series' *raw sample sequence inside a range +window* to one value — but nothing else in this vocabulary exposes that raw +sequence as rows a generic operator could sort or limit. `Sort`/`Limit` +operate on the relation `Aggregate` already produced (one row per series, +post-reduction); a `TimeRange` window's underlying samples are never +materialized as a queryable row set at this layer, only fed directly into +whichever `AggIntent` sits above it. So there is no composition available to +express "the timestamp of this window's minimum sample" from generic parts — +each of these is the only path that can reach into the window's raw stream +for its particular statistic, which is exactly the design rule 1 exception +(a genuinely different computational access pattern, not an ordinary +composition of existing operators). + +And the schema every edge in the tree carries: + +```rust +pub struct Schema { + pub columns: Vec, + pub time_index: Option, + pub unique_keys: Vec>, + pub closed: bool, +} +``` + +For example, `sum by (job) (http_requests_total)` binds to an `Aggregate` +whose input schema is `{ columns: [ts, value, job], time_index: Some(0), +unique_keys: [], closed: false }` (PromQL — open, since a metric's label +set is a superset the runtime may exceed) and whose *output* schema — +after `Reduction::by([job])` collapses every other row — is: + +```rust +Schema { + columns: vec![Column::new("job", DataType::Utf8, true), Column::new("sum", DataType::Float64, false)], + time_index: None, // a cross-series reduction collapses the time axis + unique_keys: vec![vec![0]], // grouping by job makes job unique in the output + closed: true, // by(...) enumerates its columns exactly — this is + // where an open input schema freezes to closed +} +``` diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 9935dde5..7b1683e2 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -117,7 +117,10 @@ already-computed value, at query time." Every design considered so far only builds summaries incrementally from raw samples, at ingest time; there's no established answer for building one from a query-time-derived scalar. A deployment that hits this shape should -treat it as unsupported rather than assume either answer. +treat it as unsupported rather than assume either answer. The design +proposal below narrows exactly which cases of this remain genuinely +open (see "Pattern D" below) — it is not fully open anymore, but it is +not fully resolved either. ### What's out of scope here @@ -146,3 +149,337 @@ computation requires richer partial state than a single summary value can capture, or whose result is fundamentally non-mergeable across partitions, has no summary realization by design — it always stays as ordinary logical computation over raw data. + +## Interface + +The planning-time IR: + +```rust +pub struct L4Node { + pub expr: SummaryExpr, + pub schema: L4Schema, +} + +pub enum SummaryExpr { + Logical(Box), + SummaryAgg { + child: Rc, + summary: SummaryKind, + params: SummaryParams, + col: ColumnRef, + reduction: Reduction, + }, + SummaryJoin { outer: Rc, inner: Rc, key: ColumnRef, summary: SummaryKind, params: SummaryParams }, + SummarySubtract { left: Rc, right: Rc }, + SummaryDelete { summary_input: Rc, key: ColumnRef }, + SummaryEstimate { summary_input: Rc, query: SketchQuery }, + SummaryMerge { children: Vec> }, +} +``` + +The `summary`/`summary_input` field names match this doc's own "summary" +umbrella term (see the top of this doc) directly. An earlier revision of +this type used `sketch`/`sketch_input`, predating that umbrella term — +renamed in #170, landed together with the `Implementation` variant merge +described below. + +Per-node validity constraints — each is a summary-catalog fact (see +"Summary catalog" above) checked at planning time, before this shape ever +reaches serving-time execution: + +| Node | `summary` field | Constraint | +|---|---|---| +| `Logical` | — | none — wraps an arbitrary unrewritten `QueryExpr` subtree | +| `SummaryAgg` | own | none beyond `col`'s intent already requiring an accuracy target compatible with `summary`; this is the leaf every other row's constraints are checked *against* | +| `SummaryJoin` | own | only emitted by a join-specific `Bind*OnJoin` rule — none exist yet in the base design, so this variant has no live producer | +| `SummarySubtract` | read from `left`/`right` | `left`/`right` agree on `(kind, params)`; that kind's catalog entry sets `subtractable` | +| `SummaryDelete` | read from `summary_input` | that kind's catalog entry sets `deletable` | +| `SummaryEstimate` | read from `summary_input` | `query`'s `SketchQuery` variant is one that kind actually supports (not a single boolean flag — closer to that kind's whole reason for existing) | +| `SummaryMerge` | read from `children`, which must all agree | `children` non-empty; every child produces summary *state*, never a value (see diagram below); that shared kind's catalog entry sets `mergeable` | + +The trickiest of these to hold in your head is `SummaryMerge`'s "children +must produce state, not a value" rule — everything here either produces +partial summary **state** (still mergeable, not yet read out) or a final +**value** (already collapsed, e.g. by a readout). Only a state-producing +node may feed a merge: + +```mermaid +flowchart LR + subgraph State["produce summary state"] + SA["SummaryAgg"] + SM["SummaryMerge"] + end + subgraph Value["produce a plain value"] + SE["SummaryEstimate"] + LG["Logical"] + end + SA -->|"valid merge child"| SM + SM -->|"valid merge child (nested)"| SM + SA --> SE + SM --> SE + SE -. "✗ already a value" .-> SM + LG -. "✗ already a value" .-> SM +``` + +Binding — turning a canonical `QueryExpr` into an `L4Node` tree: + +```rust +pub fn implement_tree(expr: &QueryExpr) -> Result, ImplementError>; +pub fn implement_tree_with(expr: &QueryExpr, cost_model: &dyn CostModel) -> Result, ImplementError>; +``` + +**"Implementation" is the answer to one question: how is this one +intent actually realized?** It's the return type of the per-intent +binding decision (`implementation_for`) — for a given `AggIntent`, +exactly one of: build an approximate sketch (bounded error, sized to +the accuracy target), build an exact accumulator (zero error, still +mergeable — see "Choosing a summary for an intent" above), or don't +summarize at all and evaluate the intent directly over raw data +(`PassThrough`). It's the same three-way choice this doc's "shape of a +summary-bound plan" section already describes in prose; this is that +decision's concrete type: + +```rust +pub enum Implementation { + Summary { kind: SummaryKind, params: SummaryParams }, + PassThrough, +} + +pub fn implementation_for(intent: &AggIntent) -> Implementation; +``` + +An earlier revision of this type split `Summary` into two variants, +`Sketch{kind,params}` and `ExactAccumulator{kind,params}`, carrying +identical shapes — the split existed only because binding needed to +know, right here, whether the chosen `kind` requires a `SummaryEstimate` +readout afterward (approximate) or *is* the answer once built (exact — +no estimate step). #170 collapsed them into the single `Summary` +variant above, recovering that same fact from `SummaryKind::is_exact()` +instead of from the variant tag. + +The one pluggable extension point at this layer — a deployment supplies +its own `CostModel` to override which summary family is chosen and how +its parameters are sized, without touching the binding pass itself. +Every method past the first has a sensible default, so a deployment that +only wants to reorder candidates doesn't have to implement the rest: + +```rust +pub trait CostModel { + fn rank_candidates(&self, intent: &AggIntent, candidates: &[SummaryKind]) -> Vec; + + fn size_params(&self, kind: SummaryKind, intent: &AggIntent, eps: f64, delta: f64) -> SummaryParams { + /* default: this crate's built-in sizing formulas */ + } + fn realize_extension(&self, ext_kind: &str, payload: &serde_json::Value) -> Implementation { + /* default: PassThrough */ + } + fn readout_extension(&self, ext_kind: &str, payload: &serde_json::Value, col: &ColumnRef) -> SketchQuery { + /* default: panics -- only reachable if realize_extension was overridden without this */ + } +} +``` + +Serving-time — the interface a deployment implements to actually answer +a query against an already-bound `L4Node` tree: + +```rust +pub trait SummaryExecutor { + type Handle: Clone; + type State; + type Value; + type Error; + type GroupKey: Clone + Ord + Default; + + fn find_candidates( + &self, + summary: &SummaryKind, + params: &SummaryParams, + col: &ColumnRef, + reduction: &Reduction, + child: &L4Node, + ) -> Result, Self::Error>; + + fn fetch_state(&self, handle: &Self::Handle) -> Result; + fn merge_states(&self, states: Vec) -> Result; + fn readout(&self, state: &Self::State, query: &SketchQuery) -> Result; + fn logical(&self, expr: &QueryExpr) -> Result; +} + +pub fn execute(node: &L4Node, exec: &E) -> Result, ExecError>; +``` + +`find_candidates`'s `reduction` parameter is exactly the `Reduction` +this doc's design section already explains — an implementer must branch +on `Reduce` vs. `PerEntity` there, not guess from an empty key list. + +### Design proposal: composing summaries — a taxonomy and a recipe, not a formula + +There's no single formula for the error bound of "summary built over +another summary." Exponential Histogram [Datar, Gionis, Indyk, Motwani, +SICOMP'02] and Hydra [VLDB'22] each prove a +bound for this shape, and the two bounds have different forms, because +they come from composing different concentration arguments, not from +adding two epsilons. What is general is a method. Four compound types, +by structural relationship: + +1. **Same kind, same config — merge.** Already `SummaryMerge`: exact at + the state level, zero added error, gated by the catalog's `mergeable` + flag. +2. **Heterogeneous, inner exact.** An exact inner (`Rate`, `Increase`, …) + has `Accuracy::EXACT` state, so composing over it adds nothing, + unconditionally. +3. **Heterogeneous, both approximate.** Splits into two mechanisms: + - **3a — over the inner's readout.** The outer consumes the inner's + already-collapsed estimate as an ordinary noisy input. Always + available — needs only the inner's published `(ε, δ)` — but only as + tight as a Lipschitz-style sensitivity argument, generally loose. + - **3b — over the inner's state.** The outer's construction runs + directly on the inner's raw, pre-readout representation. Only + available when the outer's construction can consume that + representation, but tight when it applies — Exponential Histogram + and Hydra are + worked examples. + 3b is strictly tighter when available; 3a is the fallback when the + inner's state isn't accessible (e.g. across a service boundary) or 3b + doesn't apply. +4. **Neither 3a nor 3b justified for the pair.** Refused by default. + +#### 3a: composing over the inner's readout + +An ordinary error-propagation argument. Let the inner produce `ṽ` +satisfying `Accuracy A_c`, and the outer be built over `ṽ` as if exact, +satisfying `A_o`. If the outer's true function `φ_outer` is `L`-Lipschitz +with respect to the norm `A_c` is stated in: + +``` +Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_inner(X))| > (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c +``` + +by the triangle inequality plus a union bound on the two failure events: + +```rust +pub enum ErrorNorm { L1, L2, Pointwise } + +pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } + +impl Accuracy { + /// `None` on a norm mismatch — refuse rather than apply an `L` that + /// was never derived for it. `Sensitivity` is always the + /// deployment's own claim (linear aggregates like Sum/Count have a + /// provable `L=1`; rank-based ones like Quantile/TopK only have + /// `L≈1` under an unproven local-density assumption). + pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + _ => None, + } + } +} +``` + +Computable for any pair given a justified `Sensitivity`, but looser than +3b, since it treats the inner as an opaque noisy scalar. + +#### 3b: composing over the inner's state (the recipe) + +Every sketch here is a randomized construction `state = Φ(input)`, and +its bound is proved by a specific argument over that construction — a +Markov bound over hash-collision mass for CMS, a compaction invariant for +KLL, a variance calculation over the max-order-statistic register for +HLL. There's no shortcut around re-examining that argument: + +1. **Feed the outer sketch the inner's state, not its answer.** Use the + inner's raw, pre-`SummaryEstimate` state — typed here as + `L4DataType::Sketch(kind, params)` — as the outer's input, instead of + a scalar readout. +2. **Check the outer's own build procedure can actually run on that + state.** Does feeding it the inner's state, instead of raw data, still + make sense? This can fail — see the worked examples below for a case + where it does and one where it clearly doesn't. If it fails, the pair + is type 4: refused. +3. **If it works, prove a new bound — don't reuse the outer's old one.** + The outer's published bound assumed clean raw input; it says nothing + about input that's itself another sketch's noisy state. The outer's + own proof technique has to be redone against this two-stage + construction. +4. **Expect a different bound for each new pair.** There's no fixed + shape this converges to — the two worked examples below land on two + different formulas. + +**Worked examples.** Exponential Histogram (a sketch built by concatenating other +sketches' state across time buckets) and Hydra (a sketch that routes +into other sketches by hash) both follow this recipe: + +| | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound | +|---|---|---| +| Exponential Histogram | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | +| Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. | +| *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) | + +Neither Exponential Histogram's nor Hydra's bound came from combining two pre-existing +formulas — both required redoing the outer's own proof for the two-layer +construction. A third, novel pair should not be expected to land on +either shape. + +#### Interface + +Two `ColumnRef` variants make which mechanism a query uses explicit at +the type level, reusing the state/value distinction `SummaryMerge` +already enforces (`SummaryAgg`/`SummaryMerge` produce state; +`SummaryEstimate`/`Logical` produce a value): + +```rust +pub enum ColumnRef { + Named(String), + Qualified { table: String, name: String }, + SampleValue, + FromReadout(Rc), // NEW — 3a; must be value-producing + FromState(Rc), // NEW — 3b; must be state-producing +} +``` + +Each routes to its own half of `CostModel`, both defaulting closed: + +```rust +pub trait CostModel { + /// 3a: has the deployment justified a Sensitivity for this pair? + fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + fn sensitivity_for(&self, outer: &AggIntent, child_kind: &SummaryKind) -> Option { + None + } + + /// 3b: has the deployment derived a bound for this pair? + fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + /// No default body — each pair gets its own derivation. Takes the + /// child's concrete (kind, params), not an abstracted accuracy + /// value, since the derivation depends on the specific pair. + fn size_params_from_state( + &self, + kind: SummaryKind, + intent: &AggIntent, + target_eps: f64, + target_delta: f64, + child: (&SummaryKind, &SummaryParams), + ) -> SummaryParams; +} +``` + +A deployment facing a new pair has a real choice: try 3b, fall back to +3a, or refuse. + +`Accuracy` (`implied_accuracy`, `is_exact`) stays a reporting type — what +a single `SummaryKind` guarantees on its own — used for type 2 and as +`compose_over_readout`'s input/output. It isn't a composition primitive +for 3b; 3b's bound comes from the recipe, not from a function of two +`Accuracy` values. diff --git a/docs/l5-physical-plan.md b/docs/l5-physical-plan.md index fd3cc1bd..e58235de 100644 --- a/docs/l5-physical-plan.md +++ b/docs/l5-physical-plan.md @@ -72,3 +72,82 @@ infrastructure that every deployment reads from, rather than something each deployment reinvents independently — though which summary families and parameter ranges are actually registered is itself deployment policy. + +## Interface + +Speculative — nothing here is built (see this doc's status note at the +top). Kept as a concrete target to design against, not as an API to +depend on. Every concrete-looking name below (`StageId`'s example +values, `Executor`'s addressing, an actual `TopologyDescriptor`) is +**interface only, in this repo** — ASAPController defines the shape a +deployment implements against; it never defines, reserves, or ships any +of the actual values. A deployment names its own stages, addresses its +own executors, and describes its own topology; nothing here is a fixed +vocabulary to conform to: + +```rust +pub trait PhysicalPlanner { + type Topology: TopologyDescriptor; + type Output; + fn lower(&self, l4: Rc, t: &Self::Topology) -> Result; +} + +pub trait TopologyDescriptor { + fn stages(&self) -> &[StageDescriptor]; + // A `StageEdge` names a pair of stages data is allowed to flow + // between (e.g. "edge → backend" if that deployment's edge tier + // ships summaries up to a backend tier) — the topology's connectivity + // graph, distinct from which stages merely *exist* (`stages()` above). + // `StageAllocator::allocate` only assigns a piece of the plan to move + // from one stage to another along an edge this list actually + // contains; a `TopologyDescriptor` with no edge between two stages is + // how a deployment declares those two stages can't exchange data + // directly. + fn edges(&self) -> &[StageEdge]; +} + +// `StageId` is an opaque, deployment-chosen string — ASAPController +// neither defines nor reserves any particular value. "edge" / "gateway" +// / "backend" / "in-process" below are one deployment's illustrative +// choice (roughly: close to data ingestion / an intermediate +// aggregation tier / a centralized serving tier / no network hop at +// all), not a fixed enum — a different deployment can and should name +// its own stages differently. +pub struct StageId(pub String); + +pub struct Executor { + pub id: ExecutorId, + pub stage: StageId, + pub capabilities: ExecutorCaps, // memory budget, available summary backends, network neighbours + pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle — deployment-defined +} + +// Stage-level allocation: given an L4 tree + a topology, decide which +// nodes land on which stage, subject to deployment constraints. +// Per-executor fan-out is the deployment model's own PhysicalPlanner, +// using the executor list from DeploymentConstraints::executors(). +pub struct StageAllocator; +impl StageAllocator { + pub fn allocate( + &self, l4: Rc, topology: &T, c: &DeploymentConstraints, + ) -> Result, PlanError>; +} +``` + +A deployment model implements `PhysicalPlanner`, delegating the +stage-level decision to `StageAllocator` and handling its own +per-executor fan-out on top: + +```rust +impl PhysicalPlanner for LifecyclePlanner { + type Topology = ThreeStage; + type Output = Vec<(ExecutorId, ExecutorPlan)>; + fn lower(&self, l4: Rc, t: &ThreeStage) -> Result { + let assignments = StageAllocator.allocate(l4, t, &self.constraints)?; + let executors: &[Executor] = self.constraints.executors(); + // deployment-specific post-processing (cost accounting, backend + // push preparation, ...) happens here. + Ok(/* ... */) + } +} +```