From d48c8c3bc9c79839806eb679131d423f6e7be997 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 16:33:54 -0600 Subject: [PATCH 01/10] docs(l4): design proposals for #171 and #172 composition gaps Stacked on #169's Interface section. Adds two concrete proposals: - #171 (exact <-> summary composition, either nesting order): a new SummaryFold node for an outer exact fold over an inner realized summary, computed generically by the shared execute walk with no new SummaryExecutor method; and a relaxed SummaryAgg input-column rule letting an outer summary read directly off an exact accumulator's own state (zero added error), while explicitly refusing the approximate- child case since that's #172's gap, not this one's. - #172 (nested approximate-over-approximate error propagation): a SummaryKind::implied_accuracy inverse-sizing function, a new CostModel::size_params child_accuracy parameter (opt-in, default ignores it, matching this trait's existing default-body convention), and a paired accepts_nested_approx gate so an outer approximate summary over an approximate child's estimate refuses by default instead of silently double-approximating. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 147 ++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 0f0c4da2..ba33fd80 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -308,3 +308,150 @@ pub fn execute(node: &L4Node, exec: &E) -> Result, + fold: FoldOp, // Min | Max | Avg | StdDev { population: bool } | Variance { population: bool } + by: Vec, // this node's own grouping — may be coarser than child's +}, +``` + +`bind.rs`'s `implement_tree_in_with` gains one new rule, checked *before* +falling back to `implementation_for_with`'s per-intent answer: for +`Min`/`Max`/`Avg`/`StdDev`/`Variance`, bind the child first (as it already +does for every `Aggregate`); if the child's bound `L4Node` is anything +other than `SummaryExpr::Logical` — i.e. it independently committed to +*some* summary realization — emit `SummaryFold` over it instead of +calling `accumulator()`/`PassThrough` for the outer intent. If the child +stayed `Logical`, nothing changes: today's `Implementation::{Summary, +PassThrough}` answer still applies, so this is additive, not a +replacement. + +`SummaryFold` needs no new deployment-supplied `SummaryExecutor` method: +folding already-materialized per-group scalars by min/max/avg/stddev/ +variance is ordinary, deployment-independent math, exactly the kind of +thing the doc's "same walk and merge logic for free" principle already +covers for `execute`'s existing node kinds — `execute` recurses into +`child` via the existing walk, then folds the resulting rows itself, +regrouping to `by` when it's coarser than whatever grouping the child +already produced. + +**Direction 2 fix — let an exact accumulator's output stand in for a +plain column.** Add one constraint to the per-node table: a `SummaryAgg` +may take another `L4Node` as its effective input column when that node's +own realization is an *exact* `SummaryAgg` (`kind.is_exact()` — no +`SummaryEstimate` needed, so "the partial state is the value" already +holds). `summarised_column` grows a second resolution path: when `child` +is itself a `SummaryAgg` with an exact kind, resolve `col` to that node's +own state field by name instead of requiring a plain L3 `Schema` column. +Composing this way adds exactly the outer sketch's own error and nothing +more — the exact child contributes zero approximation error of its own — +which is what makes this safe to bless outright, unlike the approximate- +child case below. + +This also answers, narrowly, the question this doc's "Nested composition" +section left open ("whether a summary aggregation can validly sit above +a *readout*"): sitting above an **exact accumulator's own state** is now +a yes, with zero added error. Sitting above an **approximate sketch's +estimate** is still a no — that's not a different opinion, it's the same +question [#172](https://github.com/ProjectASAP/ASAPController/issues/172) +is about, and it stays unsupported until that issue's error-composition +model exists. `implement_tree_in_with` should reject (fall back to +`PassThrough`) an attempt to resolve `col` against an *approximate* +child's estimate, rather than silently accepting it — see below. + +### Design proposal: nested approximate-over-approximate composition (#172) + +Once a deployment (or a future extension of the direction-2 fix above) +wants to sketch over another sketch's *approximate* readout rather than +an exact accumulator's state, [#172](https://github.com/ProjectASAP/ASAPController/issues/172)'s +gap applies: `CostModel::size_params` sizes the outer `(eps, delta)` as +if its input were exact, with no way to know the input already carries +error from the child. Concretely: + +1. **Recover the child's own resolved error bound.** Add + `SummaryKind::implied_accuracy(&SummaryParams) -> (f64, f64)` — the + inverse of `boundary::default_size_params`'s sizing formulas (e.g. + `Kll`: `eps ≈ 2/k`; `Hll`: `eps ≈ 1.04/√(2^p)`; `Cms`: `eps ≈ e/width`, + `delta ≈ e^{-depth}`). This is mechanical and symmetric with the + sizing math that already exists in `boundary.rs`. +2. **Thread it into sizing, opt-in.** Extend `CostModel::size_params` + with one new parameter: + + ```rust + fn size_params( + &self, + kind: SummaryKind, + intent: &AggIntent, + eps: f64, + delta: f64, + child_accuracy: Option<(f64, f64)>, // NEW — Some((eps, delta)) iff the + // designated input is itself an + // approximate summary's estimate + ) -> SummaryParams { + // default: ignores child_accuracy — byte-identical to today's formulas, + // same "sensible default" convention this trait already follows for + // every method past `rank_candidates`. + } + ``` + + A deployment that wants real composition can tighten its own target + before sizing (e.g. a conservative additive/union-bound `remaining_eps + = max(eps - child_eps, floor)`); core doesn't mandate a formula, the + same way it doesn't mandate `rank_candidates`' ordering policy. +3. **Gate it explicitly rather than silently double-approximating.** Add + `fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: SummaryKind) -> bool`, + defaulting to `false`. Binding an outer approximate summary over an + *approximate* child's estimate falls back to `Implementation::PassThrough` + unless a deployment overrides this to `true` — the same paired-method + pattern `realize_extension`/`readout_extension` already uses (one + method unlocks a shape, and overriding it is what states "I'm handling + the consequences," rather than the shape silently working half-right). + This directly answers the issue's own open question — refuse by + default, let a deployment that has actually thought about its combined + error budget opt in. + +This keeps the exact-child case (#171 direction 2) and the approximate- +child case (#172) as two different defaults: the former composes for +free because it adds no error; the latter is refused by default because +core has no basis for deciding what a deployment's combined accuracy +budget should be. From 97b9708faab7ec7c20bfc7fd7921da883cd3ff07 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 16:42:33 -0600 Subject: [PATCH 02/10] docs(l4): reframe #171/#172 proposal as a shared accuracy algebra, fold in #173 Replaces the earlier implementation-narrative version (which described specific match arms/functions in boundary.rs/bind.rs) with a design-level treatment: a formal (epsilon, delta) accuracy guarantee every SummaryKind already satisfies, a composition theorem (triangle inequality + union bound) for building a summary over another summary's output, and clean interfaces (Accuracy, ColumnRef::FromSummary, CostModel::size_params/ accepts_nested_approx, FoldOp/SummaryFold) derived from it. Shows #171 direction 2 (exact child) and #172 (approximate child) are the same composition formula at Accuracy::EXACT vs. Accuracy > 0, rather than two separate rules. Folds in #173: generalizes FromSummary/SummaryFold to N children for Hydra-style sketch-of-sketches, and adds a Bounded Accuracy variant (deterministic interval radius) for QTree-style range trees alongside the Probabilistic (epsilon, delta) variant. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 366 ++++++++++++++++++++++-------------- 1 file changed, 226 insertions(+), 140 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index ba33fd80..4f20ded1 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -309,149 +309,235 @@ pub fn execute(node: &L4Node, exec: &E) -> Result ε·φ(X) ] ≤ δ (G) +``` + +with `(ε, δ) = (0, 0)` for the exact accumulators (`Sum`/`Count`/`MinMax`/ +`Rate`/`Increase`) — precisely what `SummaryKind::is_exact()` records. Made +explicit as a type: + +```rust +pub struct Accuracy { + pub epsilon: f64, + pub delta: f64, +} + +impl Accuracy { + pub const EXACT: Accuracy = Accuracy { epsilon: 0.0, delta: 0.0 }; +} + +impl SummaryKind { + /// The (ε, δ) this kind, at these params, actually satisfies for (G) — + /// the inverse of `boundary::default_size_params`'s sizing formulas. + /// E.g. Kll: ε ≈ 2/k; Hll: ε ≈ 1.04/√(2^p); Cms: ε ≈ e/width, δ ≈ e^(−depth). + /// `Accuracy::EXACT` for every kind where `is_exact()` holds. + pub fn implied_accuracy(&self, params: &SummaryParams) -> Accuracy { .. } +} +``` + +This is the same abstraction Algebird's `Approximate[T]` monoid packages — +a confidence interval that composes algebraically *within one chain*. All +three issues are asking what happens at the boundary where a second +instance of (G) is layered on top of the *output* of a first, instead of +being built once over raw `X`. + +#### The composition theorem + +Let a child summary produce `ṽ = ρ_child(β_child(X))` satisfying (G) with +`Accuracy A_c = (ε_c, δ_c)` for `φ_child`. Let an outer summary be built +over `ṽ` as if it were exact input, satisfying (G) with `A_o = (ε_o, δ_o)` +for `φ_outer`. If `φ_outer` is `L`-Lipschitz in its input (linear +aggregates like `Sum`/`Count`/CMS-frequency have `L = 1` exactly; rank-based +aggregates like `Quantile`/`TopK` have `L = 1` under the standard +assumption that a relative perturbation of each element doesn't move its +rank by more than a proportional amount — an assumption, not a proof, which +is why the gate below defaults closed), then by the triangle inequality on +the two error terms and a union bound on their two failure events: + +``` +Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_child(X))| > (ε_o + L·ε_c)·φ_outer(φ_child(X)) ] ≤ δ_o + δ_c (G∘) +``` + +As an operator on `Accuracy`: ```rust +impl Accuracy { + pub fn compose(outer: Accuracy, child: Accuracy, lipschitz: f64) -> Accuracy { + Accuracy { + epsilon: outer.epsilon + lipschitz * child.epsilon, + delta: outer.delta + child.delta, + } + } +} +``` + +**#171 direction 2 and #172 are the same formula at two points on one +axis.** `#171`'s outer-summary-over-inner-exact-accumulator is exactly +`A_c = Accuracy::EXACT` — `compose` returns `A_o` unchanged, so composing +over an exact child is free and can be blessed unconditionally. `#172`'s +outer-summary-over-inner-approximate-sketch is the same formula with +`A_c.epsilon, A_c.delta > 0` — composing is no longer free, and the +combined bound must be checked against what the query actually asked for. + +#### Interface: letting a summary's input be another summary's output + +One new `ColumnRef` variant carries an `Accuracy` alongside the reference, +so every consumer of `col` can see what it's actually built on without +re-deriving it: + +```rust +pub enum ColumnRef { + Named(String), + Qualified { table: String, name: String }, + SampleValue, + FromSummary { node: Rc, accuracy: Accuracy }, // NEW +} +``` + +Constructing `FromSummary` is legal in exactly two cases: + +- `accuracy == Accuracy::EXACT` — always allowed (#171 direction 2). +- `accuracy.epsilon > 0 || accuracy.delta > 0` — allowed only when the + deployment's `CostModel` opts in (#172): + + ```rust + pub trait CostModel { + /// Does this deployment accept building `outer` over a child already + /// carrying `child_kind`'s own approximation error? Default `false` — + /// refuse rather than silently double-approximate. + fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + + /// Size `kind`'s params so the composed guarantee (G∘) holds against + /// `target` (what the query asked for), given the child's own + /// resolved `Accuracy` when the input is a `FromSummary` column. + /// Solves `target ≥ Accuracy::compose(result, child_accuracy, L)`, + /// i.e. `result.epsilon = target.epsilon − L·child_accuracy.epsilon`, + /// `result.delta = target.delta − child_accuracy.delta` — rejecting + /// if either goes non-positive (no achievable outer accuracy leaves + /// enough of the target budget for the child's contribution). + fn size_params( + &self, + kind: SummaryKind, + intent: &AggIntent, + target: Accuracy, + child_accuracy: Option, // None over raw/exact input + ) -> SummaryParams { .. } + } + ``` + + When `accepts_nested_approx` returns `false` (the default), binding + falls back to `Implementation::PassThrough` instead of constructing an + unsound `FromSummary` column — the same paired-method shape + `realize_extension`/`readout_extension` already uses one layer up. + +#### Interface: folding over a child, as a monoid homomorphism + +The other half of #171 (an outer *exact fold* over an inner realized +summary) isn't an accuracy question — the fold itself is exact — it's a +question of which of `Min`/`Max`/`Avg`/`StdDev`/`Variance` are valid to +apply directly to a child's already-*collapsed* value, versus which need +the child's pre-readout state. This is exactly Algebird's `Averaged`/ +moment-monoid distinction: `Avg`/`Variance`/`StdDev` are not associative +on the scalar output (the average of two averages isn't the true average +under unequal group sizes) — the monoid lives on the sufficient statistic +(`Σv`; `Σv, n`; `Σv, Σv², n`), not on the folded scalar itself. + +```rust +pub enum FoldOp { + Min, + Max, + Avg, + Variance { population: bool }, + StdDev { population: bool }, +} + +pub enum FoldInput { + /// (ℝ, min, +∞) / (ℝ, max, −∞) are commutative monoids on the value + /// itself — the child's already-read-out scalar estimate suffices. + Value, + /// No monoid exists on the scalar average/variance alone — folding + /// needs the child's pre-readout sufficient statistic instead. + SufficientStatistic, +} + +impl FoldOp { + pub fn required_input(&self) -> FoldInput { + match self { + FoldOp::Min | FoldOp::Max => FoldInput::Value, + FoldOp::Avg | FoldOp::Variance { .. } | FoldOp::StdDev { .. } => { + FoldInput::SufficientStatistic + } + } + } +} + SummaryExpr::SummaryFold { - child: Rc, - fold: FoldOp, // Min | Max | Avg | StdDev { population: bool } | Variance { population: bool } - by: Vec, // this node's own grouping — may be coarser than child's + child: Rc, // a SummaryEstimate/exact state for FoldInput::Value; + // a pre-readout SummaryAgg state for FoldInput::SufficientStatistic + fold: FoldOp, + by: Vec, // this node's own grouping — may be coarser than child's }, ``` -`bind.rs`'s `implement_tree_in_with` gains one new rule, checked *before* -falling back to `implementation_for_with`'s per-intent answer: for -`Min`/`Max`/`Avg`/`StdDev`/`Variance`, bind the child first (as it already -does for every `Aggregate`); if the child's bound `L4Node` is anything -other than `SummaryExpr::Logical` — i.e. it independently committed to -*some* summary realization — emit `SummaryFold` over it instead of -calling `accumulator()`/`PassThrough` for the outer intent. If the child -stayed `Logical`, nothing changes: today's `Implementation::{Summary, -PassThrough}` answer still applies, so this is additive, not a -replacement. - -`SummaryFold` needs no new deployment-supplied `SummaryExecutor` method: -folding already-materialized per-group scalars by min/max/avg/stddev/ -variance is ordinary, deployment-independent math, exactly the kind of -thing the doc's "same walk and merge logic for free" principle already -covers for `execute`'s existing node kinds — `execute` recurses into -`child` via the existing walk, then folds the resulting rows itself, -regrouping to `by` when it's coarser than whatever grouping the child -already produced. - -**Direction 2 fix — let an exact accumulator's output stand in for a -plain column.** Add one constraint to the per-node table: a `SummaryAgg` -may take another `L4Node` as its effective input column when that node's -own realization is an *exact* `SummaryAgg` (`kind.is_exact()` — no -`SummaryEstimate` needed, so "the partial state is the value" already -holds). `summarised_column` grows a second resolution path: when `child` -is itself a `SummaryAgg` with an exact kind, resolve `col` to that node's -own state field by name instead of requiring a plain L3 `Schema` column. -Composing this way adds exactly the outer sketch's own error and nothing -more — the exact child contributes zero approximation error of its own — -which is what makes this safe to bless outright, unlike the approximate- -child case below. - -This also answers, narrowly, the question this doc's "Nested composition" -section left open ("whether a summary aggregation can validly sit above -a *readout*"): sitting above an **exact accumulator's own state** is now -a yes, with zero added error. Sitting above an **approximate sketch's -estimate** is still a no — that's not a different opinion, it's the same -question [#172](https://github.com/ProjectASAP/ASAPController/issues/172) -is about, and it stays unsupported until that issue's error-composition -model exists. `implement_tree_in_with` should reject (fall back to -`PassThrough`) an attempt to resolve `col` against an *approximate* -child's estimate, rather than silently accepting it — see below. - -### Design proposal: nested approximate-over-approximate composition (#172) - -Once a deployment (or a future extension of the direction-2 fix above) -wants to sketch over another sketch's *approximate* readout rather than -an exact accumulator's state, [#172](https://github.com/ProjectASAP/ASAPController/issues/172)'s -gap applies: `CostModel::size_params` sizes the outer `(eps, delta)` as -if its input were exact, with no way to know the input already carries -error from the child. Concretely: - -1. **Recover the child's own resolved error bound.** Add - `SummaryKind::implied_accuracy(&SummaryParams) -> (f64, f64)` — the - inverse of `boundary::default_size_params`'s sizing formulas (e.g. - `Kll`: `eps ≈ 2/k`; `Hll`: `eps ≈ 1.04/√(2^p)`; `Cms`: `eps ≈ e/width`, - `delta ≈ e^{-depth}`). This is mechanical and symmetric with the - sizing math that already exists in `boundary.rs`. -2. **Thread it into sizing, opt-in.** Extend `CostModel::size_params` - with one new parameter: - - ```rust - fn size_params( - &self, - kind: SummaryKind, - intent: &AggIntent, - eps: f64, - delta: f64, - child_accuracy: Option<(f64, f64)>, // NEW — Some((eps, delta)) iff the - // designated input is itself an - // approximate summary's estimate - ) -> SummaryParams { - // default: ignores child_accuracy — byte-identical to today's formulas, - // same "sensible default" convention this trait already follows for - // every method past `rank_candidates`. - } - ``` - - A deployment that wants real composition can tighten its own target - before sizing (e.g. a conservative additive/union-bound `remaining_eps - = max(eps - child_eps, floor)`); core doesn't mandate a formula, the - same way it doesn't mandate `rank_candidates`' ordering policy. -3. **Gate it explicitly rather than silently double-approximating.** Add - `fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: SummaryKind) -> bool`, - defaulting to `false`. Binding an outer approximate summary over an - *approximate* child's estimate falls back to `Implementation::PassThrough` - unless a deployment overrides this to `true` — the same paired-method - pattern `realize_extension`/`readout_extension` already uses (one - method unlocks a shape, and overriding it is what states "I'm handling - the consequences," rather than the shape silently working half-right). - This directly answers the issue's own open question — refuse by - default, let a deployment that has actually thought about its combined - error budget opt in. - -This keeps the exact-child case (#171 direction 2) and the approximate- -child case (#172) as two different defaults: the former composes for -free because it adds no error; the latter is refused by default because -core has no basis for deciding what a deployment's combined accuracy -budget should be. +Because `fold` is always exact, `SummaryFold` carries no `Accuracy` of its +own — it's transparent to whatever accuracy the child already has. + +#### Generalizing to #173: N-ary composition and non-probabilistic accuracy + +Two changes make the same primitives serve #173 without inventing a +parallel mechanism when it lands: + +- **Hydra's sketch-of-sketches is `FromSummary`/`SummaryFold` with more + than one child.** Generalize both to `children: Vec<(Rc, Accuracy)>` + combined by some `combine` function (per-dimension sketches feeding one + cross-dimension estimate). The composition theorem generalizes the same + way a union bound always does: + + ``` + ε_total = ε_outer + Σ_i Lᵢ·εᵢ δ_total = δ_outer + Σ_i δᵢ + ``` + + — no new theorem, just summing (G∘)'s per-child term over the children + vector instead of a single child. + +- **QTree-style range trees need a non-probabilistic `Accuracy`.** `(G)` + assumes a point estimate with probabilistic error; a range-tree kind's + `rangeQuantileBounds`/`rangeSumBounds` return a *deterministic* interval + whose half-width shrinks with tree capacity, not a `(ε, δ)` pair. Model + it as a second `Accuracy` variant instead of forcing a probabilistic + shape onto it: + + ```rust + pub enum Accuracy { + Probabilistic { epsilon: f64, delta: f64 }, + Bounded { radius: f64 }, // e.g. a QTree node's interval half-width + } + ``` + + `compose` for two `Bounded` accuracies is `radius_total = radius_outer + + L·radius_child` — the same triangle-inequality shape as `(G∘)`, just + without a failure probability to union-bound. `FromSummary`/`SummaryFold` + don't need to know which variant they're carrying; only `compose` and + `size_params` do. From 2dbcbc01e9995ac41abdfad3e5d8ef9df3dee4d2 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 20:21:19 -0600 Subject: [PATCH 03/10] docs(l4): separate the design from which issue it solves; norm-aware composition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes, both from review: - Restructure into "the design" (guarantee, composition theorem, three interfaces — no issue numbers threaded through the reasoning) followed by a single "Which issue this solves" section mapping each piece back to #171/#172/#173, instead of interleaving issue callouts throughout. - The composition theorem assumed a single scalar Lipschitz constant L under an implicit pointwise/L1 error model. That's wrong for this catalog's own CountSketch/CountSketchWithHeap, whose guarantee is stated against the L2 norm of the residual frequency vector, not L1 -- a materially different, not-directly-comparable quantity on skewed data. Accuracy and compose now carry an explicit ErrorNorm; compose returns None on a norm mismatch instead of silently assuming L=1. Folded Accuracy::Bounded (deterministic radius, for #173's QTree case) into the same enum/compose function rather than a separate type. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 309 ++++++++++++++++++++++-------------- 1 file changed, 194 insertions(+), 115 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 4f20ded1..db639a6b 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -309,17 +309,16 @@ pub fn execute(node: &L4Node, exec: &E) -> Result ε·φ(X) ] ≤ δ (G) ``` with `(ε, δ) = (0, 0)` for the exact accumulators (`Sum`/`Count`/`MinMax`/ -`Rate`/`Increase`) — precisely what `SummaryKind::is_exact()` records. Made -explicit as a type: +`Rate`/`Increase`) — precisely what `SummaryKind::is_exact()` records. + +This is already two distinct shapes in the current catalog, not one, and +the type has to say which: (G) as written bounds a single output value +against a *norm* on the underlying value vector, and that norm isn't +uniform across kinds — CMS bounds per-item error against the *L1* norm of +the whole frequency vector (`‖X‖₁`); `CountSketch`/`CountSketchWithHeap` +bound it against the *L2* norm of the residual vector instead +(`boundary.rs`'s own sizing comment already flags this as an unresolved +refinement) — a tighter but not directly comparable quantity on skewed +data; KLL/HLL/DDSketch/Theta/Kmv have no underlying vector norm at all, a +plain pointwise bound. And separately, a structured kind like a QTree-style +range tree ([#173](https://github.com/ProjectASAP/ASAPController/issues/173)) +doesn't give a probabilistic `(ε, δ)` bound at all — it returns a +*deterministic* interval whose half-width shrinks with tree capacity. +`Accuracy` has to represent both shapes: ```rust -pub struct Accuracy { - pub epsilon: f64, - pub delta: f64, +pub enum ErrorNorm { + /// Per-item error relative to Σ|x| over the value vector — CMS. + L1, + /// Per-item error relative to the L2 norm of the (residual) value + /// vector — Count-Sketch. Tighter than L1 on skewed data, but not + /// directly convertible to it without an explicit, workload-dependent + /// constant. + L2, + /// A single-value guarantee with no underlying vector norm — KLL rank + /// error, HLL cardinality error, DDSketch, Theta, Kmv. + Pointwise, +} + +pub enum Accuracy { + /// (G)'s probabilistic bound, against a specific norm. + Probabilistic { epsilon: f64, delta: f64, norm: ErrorNorm }, + /// A deterministic interval half-width — e.g. a QTree node's bound — + /// with no failure probability to speak of. + Bounded { radius: f64 }, } impl Accuracy { - pub const EXACT: Accuracy = Accuracy { epsilon: 0.0, delta: 0.0 }; + pub const EXACT: Accuracy = Accuracy::Probabilistic { epsilon: 0.0, delta: 0.0, norm: ErrorNorm::Pointwise }; } impl SummaryKind { - /// The (ε, δ) this kind, at these params, actually satisfies for (G) — - /// the inverse of `boundary::default_size_params`'s sizing formulas. - /// E.g. Kll: ε ≈ 2/k; Hll: ε ≈ 1.04/√(2^p); Cms: ε ≈ e/width, δ ≈ e^(−depth). - /// `Accuracy::EXACT` for every kind where `is_exact()` holds. + /// The `Accuracy` this kind, at these params, actually satisfies for + /// (G) — the inverse of `boundary::default_size_params`'s sizing + /// formulas (e.g. Kll: ε ≈ 2/k; Hll: ε ≈ 1.04/√(2^p); Cms: ε ≈ e/width, + /// δ ≈ e^(−depth)). `Accuracy::EXACT` for every kind where + /// `is_exact()` holds. pub fn implied_accuracy(&self, params: &SummaryParams) -> Accuracy { .. } } ``` This is the same abstraction Algebird's `Approximate[T]` monoid packages — -a confidence interval that composes algebraically *within one chain*. All -three issues are asking what happens at the boundary where a second -instance of (G) is layered on top of the *output* of a first, instead of -being built once over raw `X`. +a confidence interval that composes algebraically *within one chain*. The +open question is what happens at the boundary where a second instance of +(G) is layered on top of the *output* of a first, instead of being built +once over raw `X`. #### The composition theorem -Let a child summary produce `ṽ = ρ_child(β_child(X))` satisfying (G) with -`Accuracy A_c = (ε_c, δ_c)` for `φ_child`. Let an outer summary be built -over `ṽ` as if it were exact input, satisfying (G) with `A_o = (ε_o, δ_o)` -for `φ_outer`. If `φ_outer` is `L`-Lipschitz in its input (linear -aggregates like `Sum`/`Count`/CMS-frequency have `L = 1` exactly; rank-based -aggregates like `Quantile`/`TopK` have `L = 1` under the standard -assumption that a relative perturbation of each element doesn't move its -rank by more than a proportional amount — an assumption, not a proof, which -is why the gate below defaults closed), then by the triangle inequality on -the two error terms and a union bound on their two failure events: - -``` -Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_child(X))| > (ε_o + L·ε_c)·φ_outer(φ_child(X)) ] ≤ δ_o + δ_c (G∘) -``` - -As an operator on `Accuracy`: +Let a child summary produce `ṽ = ρ_child(β_child(X))` with `Accuracy A_c` +for `φ_child`. Let an outer summary be built over `ṽ` as if it were exact +input, with `Accuracy A_o` for `φ_outer`, where `φ_outer` is `L`-Lipschitz +with respect to `A_c`'s own representation (its `norm`, if `Probabilistic`; +plain magnitude, if `Bounded`) — call that pairing a `Sensitivity`: ```rust +/// `φ_outer`'s sensitivity, valid only against inputs whose error is +/// already stated in `from`. `None`/mismatched `from` means "no proven +/// bound for this combination" — composition must refuse, not guess. +pub struct Sensitivity { + pub lipschitz: f64, + pub from: ErrorNorm, // ignored when composing two `Bounded` accuracies +} + impl Accuracy { - pub fn compose(outer: Accuracy, child: Accuracy, lipschitz: f64) -> Accuracy { - Accuracy { - epsilon: outer.epsilon + lipschitz * child.epsilon, - delta: outer.delta + child.delta, + pub fn compose(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + (Accuracy::Bounded { radius: r_o }, Accuracy::Bounded { radius: r_c }) => { + Some(Accuracy::Bounded { radius: r_o + sensitivity.lipschitz * r_c }) + } + _ => None, // norm mismatch, or a Probabilistic/Bounded pairing } } } ``` -**#171 direction 2 and #172 are the same formula at two points on one -axis.** `#171`'s outer-summary-over-inner-exact-accumulator is exactly -`A_c = Accuracy::EXACT` — `compose` returns `A_o` unchanged, so composing -over an exact child is free and can be blessed unconditionally. `#172`'s -outer-summary-over-inner-approximate-sketch is the same formula with -`A_c.epsilon, A_c.delta > 0` — composing is no longer free, and the -combined bound must be checked against what the query actually asked for. +For two `Probabilistic` accuracies, by the triangle inequality on the two +error terms and a union bound on their two failure events: + +``` +Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_child(X))| > (ε_o + L·ε_c)·φ_outer(φ_child(X)) ] ≤ δ_o + δ_c (G∘) +``` + +For two `Bounded` accuracies the same triangle-inequality shape holds on +the interval radius directly, with no failure probability to union-bound: +`radius_total = radius_outer + L·radius_child`. + +**Why `compose` refuses instead of defaulting to `L = 1`.** Linear +aggregates (`Sum`/`Count`/`Cms`-frequency) are `(L = 1, from: +Pointwise-or-L1)`-sensitive — the clean, dimension-free case most +compositions land in. Rank-based aggregates (`Quantile`/`TopK`) are +`(L = 1, from: Pointwise)`-sensitive only under the standard assumption +that a relative perturbation of each element doesn't move its rank by more +than a proportional amount — an assumption, not a proof. Neither +`Sensitivity` is valid against an `ErrorNorm::L2` child: composing over a +`CountSketch`/`CountSketchWithHeap` child needs its own, separately +justified `Sensitivity { from: L2, .. }` — for a linear outer this is a +real, derivable (if `√n`-dependent, by Cauchy–Schwarz) bound; for a +rank-based outer, sensitivity to an L2-bounded noise vector is +data-dependent (it depends on the local density of values near the query +rank, not just the aggregate norm bound), so no fixed `L` exists for it at +all — a deployment enabling that composition is asserting a +workload-specific bound, not one this crate can derive. #### Interface: letting a summary's input be another summary's output @@ -410,30 +467,37 @@ pub enum ColumnRef { Named(String), Qualified { table: String, name: String }, SampleValue, - FromSummary { node: Rc, accuracy: Accuracy }, // NEW + FromSummary { nodes: Vec>, accuracy: Accuracy }, // NEW } ``` +`nodes` is plural: a single already-realized child is the common case +(`nodes.len() == 1`), but nothing about `compose` or the guarantee above +is specific to one child — summing `(G∘)`'s per-child term over several +children is the same union bound, just applied more than once (`ε_total = +ε_outer + Σᵢ Lᵢ·εᵢ`, `δ_total = δ_outer + Σᵢ δᵢ`). + Constructing `FromSummary` is legal in exactly two cases: -- `accuracy == Accuracy::EXACT` — always allowed (#171 direction 2). -- `accuracy.epsilon > 0 || accuracy.delta > 0` — allowed only when the - deployment's `CostModel` opts in (#172): +- `accuracy == Accuracy::EXACT` — always allowed, unconditionally. +- Any non-`EXACT` `accuracy` — allowed only when the deployment's + `CostModel` opts in: ```rust pub trait CostModel { - /// Does this deployment accept building `outer` over a child already - /// carrying `child_kind`'s own approximation error? Default `false` — - /// refuse rather than silently double-approximate. + /// Does this deployment accept building `outer` over input already + /// carrying `child_kind`'s own approximation error? Default `false` + /// — refuse rather than silently double-approximate. fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { false } - /// Size `kind`'s params so the composed guarantee (G∘) holds against - /// `target` (what the query asked for), given the child's own - /// resolved `Accuracy` when the input is a `FromSummary` column. - /// Solves `target ≥ Accuracy::compose(result, child_accuracy, L)`, - /// i.e. `result.epsilon = target.epsilon − L·child_accuracy.epsilon`, + /// Size `kind`'s params so the composed guarantee holds against + /// `target` (what the query asked for), given the input's own + /// resolved `Accuracy` when it's a `FromSummary` column. Solves + /// `target == Accuracy::compose(result, child_accuracy, sensitivity)` + /// for `result` — e.g. for two `Probabilistic` accuracies, + /// `result.epsilon = target.epsilon − L·child_accuracy.epsilon`, /// `result.delta = target.delta − child_accuracy.delta` — rejecting /// if either goes non-positive (no achievable outer accuracy leaves /// enough of the target budget for the child's contribution). @@ -454,15 +518,16 @@ Constructing `FromSummary` is legal in exactly two cases: #### Interface: folding over a child, as a monoid homomorphism -The other half of #171 (an outer *exact fold* over an inner realized -summary) isn't an accuracy question — the fold itself is exact — it's a -question of which of `Min`/`Max`/`Avg`/`StdDev`/`Variance` are valid to -apply directly to a child's already-*collapsed* value, versus which need -the child's pre-readout state. This is exactly Algebird's `Averaged`/ -moment-monoid distinction: `Avg`/`Variance`/`StdDev` are not associative -on the scalar output (the average of two averages isn't the true average -under unequal group sizes) — the monoid lives on the sufficient statistic -(`Σv`; `Σv, n`; `Σv, Σv², n`), not on the folded scalar itself. +Not every composition is an accuracy question. Folding an *exact* function +(`Min`/`Max`/`Avg`/`StdDev`/`Variance`) over a child's already-produced +output is exact by construction — the open question there is which of +those fold operators are valid to apply directly to a child's +already-*collapsed* value, versus which need the child's pre-readout +state. This is exactly Algebird's `Averaged`/moment-monoid distinction: +`Avg`/`Variance`/`StdDev` are not associative on the scalar output (the +average of two averages isn't the true average under unequal group sizes) +— the monoid lives on the sufficient statistic (`Σv`; `Σv, n`; `Σv, Σv², +n`), not on the folded scalar itself. ```rust pub enum FoldOp { @@ -494,50 +559,64 @@ impl FoldOp { } SummaryExpr::SummaryFold { - child: Rc, // a SummaryEstimate/exact state for FoldInput::Value; - // a pre-readout SummaryAgg state for FoldInput::SufficientStatistic + children: Vec>, // per FoldOp::required_input: already-read-out + // values, or pre-readout sufficient-statistic state fold: FoldOp, - by: Vec, // this node's own grouping — may be coarser than child's + by: Vec, // this node's own grouping — may be coarser than any child's }, ``` Because `fold` is always exact, `SummaryFold` carries no `Accuracy` of its -own — it's transparent to whatever accuracy the child already has. - -#### Generalizing to #173: N-ary composition and non-probabilistic accuracy - -Two changes make the same primitives serve #173 without inventing a -parallel mechanism when it lands: - -- **Hydra's sketch-of-sketches is `FromSummary`/`SummaryFold` with more - than one child.** Generalize both to `children: Vec<(Rc, Accuracy)>` - combined by some `combine` function (per-dimension sketches feeding one - cross-dimension estimate). The composition theorem generalizes the same - way a union bound always does: - - ``` - ε_total = ε_outer + Σ_i Lᵢ·εᵢ δ_total = δ_outer + Σ_i δᵢ - ``` - - — no new theorem, just summing (G∘)'s per-child term over the children - vector instead of a single child. - -- **QTree-style range trees need a non-probabilistic `Accuracy`.** `(G)` - assumes a point estimate with probabilistic error; a range-tree kind's - `rangeQuantileBounds`/`rangeSumBounds` return a *deterministic* interval - whose half-width shrinks with tree capacity, not a `(ε, δ)` pair. Model - it as a second `Accuracy` variant instead of forcing a probabilistic - shape onto it: - - ```rust - pub enum Accuracy { - Probabilistic { epsilon: f64, delta: f64 }, - Bounded { radius: f64 }, // e.g. a QTree node's interval half-width - } - ``` - - `compose` for two `Bounded` accuracies is `radius_total = radius_outer + - L·radius_child` — the same triangle-inequality shape as `(G∘)`, just - without a failure probability to union-bound. `FromSummary`/`SummaryFold` - don't need to know which variant they're carrying; only `compose` and - `size_params` do. +own — it's transparent to whatever accuracy its children already have. + +### Which issue this solves + +The design above is issue-agnostic on purpose; here is what each piece +answers. + +**[#171](https://github.com/ProjectASAP/ASAPController/issues/171) — +composed exact ↔ summary aggregation, either nesting order.** + +- *Direction 1* (an outer exact fold, e.g. `max by (zone) + (quantile_over_time(0.99, m[5m]))`, over an inner independently-realized + summary) is answered by `SummaryFold`/`FoldOp`. `Min`/`Max` consume the + child's scalar estimate directly; `Avg`/`StdDev`/`Variance` consume the + child's pre-readout sufficient-statistic state instead, per + `required_input`. No new accumulator kind is needed — today's code path + (`accumulator(intent, SummaryKind::MinMax, ..)`, which requires a real, + independently-materialized `MinMax` instance) is what made this + direction look like it needed one. +- *Direction 2* (an outer summary, e.g. `TopK`, over an inner exact + accumulator like `Rate`) is the `FromSummary` interface at + `accuracy = Accuracy::EXACT`. `compose`'s `Probabilistic` arm with + `A_c = Accuracy::EXACT` returns `A_o` unchanged — composing over an + exact child is free, so this case is allowed unconditionally, with no + `CostModel` opt-in required. + +**[#172](https://github.com/ProjectASAP/ASAPController/issues/172) — +nested approximate-over-approximate composition has no error-propagation +model.** This is the `FromSummary` interface at a non-`EXACT` `Accuracy`: +`compose`'s `Probabilistic` arm with `A_c.epsilon > 0` (or `A_c.delta > +0`) is the formula this issue asked for — `ε_total = ε_outer + L·ε_child`, +`δ_total = δ_outer + δ_child` — gated by `CostModel::accepts_nested_approx` +(default `false`, answering the issue's own open question: refuse to bind +by default rather than silently double-approximate) and sized by +`size_params` receiving `child_accuracy` so the *combined* bound, not just +the outer layer's, is checked against the query's actual target. + +**[#173](https://github.com/ProjectASAP/ASAPController/issues/173) — +multi-dimensional/structured summaries (QTree, Hydra).** Two things fall +out of the design without extra machinery: + +- Hydra's "sketch of sketches" is `FromSummary`/`SummaryFold` with + `nodes`/`children` containing more than one entry — the union bound in + `compose`'s per-child term already generalizes to a sum over children, + so a cross-dimension combiner is the existing interface applied to a + vector instead of a single node, not a new mechanism. +- QTree-style range trees are the `Accuracy::Bounded` variant — a + deterministic interval radius rather than a probabilistic `(ε, δ)` pair. + `compose`'s `Bounded` arm gives the same triangle-inequality composition + without a failure probability to union-bound, so a range-tree kind + slots into `FromSummary`/`SummaryFold` the same way a probabilistic + sketch does; only `compose` and `size_params` need to know which variant + they're holding. From fb883032f55885d29415788c11b1a2917754e8e3 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:12:49 -0600 Subject: [PATCH 04/10] docs(l4): replace unified accuracy algebra with three grounded composition patterns Surveyed how real systems actually compose summaries (DGIM SICOMP'02, UnivMon SIGCOMM'16, Hydra VLDB'22, PromSketch VLDB'25) rather than inventing one generic accuracy algebra. Finding: there is no single law -- three structurally distinct patterns exist, only two have a governing theorem, and the pattern #171/#172 actually describe has none at all. - Pattern A (same statistic across partitions -- EH/DGIM windowing): real, DGIM Theorem 6/7 grounds it precisely, but composes the same statistic across a partition of the same data, not different statistics along a query's data-flow -- out of scope for #171/#172/#173, flagged as a separate future gap (sliding-window range-vector queries). - Pattern B (hash-collision routing -- Hydra's sketch of sketches): Theorem 2's asymmetric bound (multiplicative + additive-vs-global-total) doesn't fit the shared Accuracy shape -- Hydra and QTree each need their own SummaryKind fed directly from raw data, not a generalization of FromSummary/SummaryFold. This replaces the previous (incorrect) N-ary FromSummary framing for #173. - Pattern D (heterogeneous sequential nesting -- what #171/#172 actually ask about): no borrowed theorem exists for this shape in any of the four systems surveyed -- all four avoid it structurally. Split into D1 (inner exact, composes free -- #171 direction 2), D2 (outer exact fold over inner summary -- #171 direction 1, SummaryFold), and D3 (outer sketch over an inner statistic with no exact sub-linear form -- #172's actual remaining scope, narrower than "any nested approximation"). D3's compose()/accepts_nested_approx gate is now labeled honestly as this design's own first-principles proposal, not a literature result -- defaulting closed matches what every system surveyed actually does (none of them re-sketch another sketch's readout). Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 470 ++++++++++++++++-------------------- 1 file changed, 207 insertions(+), 263 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index db639a6b..6ff79993 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -117,7 +117,10 @@ already-computed value, at query time." Every design considered so far only builds summaries incrementally from raw samples, at ingest time; there's no established answer for building one from a query-time-derived scalar. A deployment that hits this shape should -treat it as unsupported rather than assume either answer. +treat it as unsupported rather than assume either answer. The design +proposal below narrows exactly which cases of this remain genuinely +open (see "Pattern D" below) — it is not fully open anymore, but it is +not fully resolved either. ### What's out of scope here @@ -309,314 +312,255 @@ pub fn execute(node: &L4Node, exec: &E) -> Result ε·φ(X) ] ≤ δ (G) +P1. f(Bᵢ) ≥ 0 +P2. f(Bᵢ) ≤ poly(|Bᵢ|) +P3. f(B1+B2) ≥ f(B1)+f(B2) (sub-additive) +P4. f(B1+B2) ≤ Cf·(f(B1)+f(B2)), Cf ≥ 1 (weakly super-additive) +P5. f admits a composable sketch ``` -with `(ε, δ) = (0, 0)` for the exact accumulators (`Sum`/`Count`/`MinMax`/ -`Rate`/`Increase`) — precisely what `SummaryKind::is_exact()` records. - -This is already two distinct shapes in the current catalog, not one, and -the type has to say which: (G) as written bounds a single output value -against a *norm* on the underlying value vector, and that norm isn't -uniform across kinds — CMS bounds per-item error against the *L1* norm of -the whole frequency vector (`‖X‖₁`); `CountSketch`/`CountSketchWithHeap` -bound it against the *L2* norm of the residual vector instead -(`boundary.rs`'s own sizing comment already flags this as an unresolved -refinement) — a tighter but not directly comparable quantity on skewed -data; KLL/HLL/DDSketch/Theta/Kmv have no underlying vector norm at all, a -plain pointwise bound. And separately, a structured kind like a QTree-style -range tree ([#173](https://github.com/ProjectASAP/ASAPController/issues/173)) -doesn't give a probabilistic `(ε, δ)` bound at all — it returns a -*deterministic* interval whose half-width shrinks with tree capacity. -`Accuracy` has to represent both shapes: +giving, when the per-bucket sketch itself carries relative error `ε̂`: -```rust -pub enum ErrorNorm { - /// Per-item error relative to Σ|x| over the value vector — CMS. - L1, - /// Per-item error relative to the L2 norm of the (residual) value - /// vector — Count-Sketch. Tighter than L1 on skewed data, but not - /// directly convertible to it without an explicit, workload-dependent - /// constant. - L2, - /// A single-value guarantee with no underlying vector norm — KLL rank - /// error, HLL cardinality error, DDSketch, Theta, Kmv. - Pointwise, -} +``` +Er ≤ (1+ε̂)²·Cf²/k + Cf − 1 + ε̂ +``` -pub enum Accuracy { - /// (G)'s probabilistic bound, against a specific norm. - Probabilistic { epsilon: f64, delta: f64, norm: ErrorNorm }, - /// A deterministic interval half-width — e.g. a QTree node's bound — - /// with no failure probability to speak of. - Bounded { radius: f64 }, -} +(`k` is the bucket-count knob). PromSketch's own EHKLL (`εEHKLL ≤ +2εEH+εKLL`) and EHUniv are both direct instances of this theorem for +specific `f` (rank-error, and `L2²` respectively) — EHUniv specifically +tunes `k = O(1/ε²)` so the `Cf²/k` term becomes asymptotically smaller +than `ε̂`, which is why its headline bound shows a single `ε` rather than +an explicit sum. -impl Accuracy { - pub const EXACT: Accuracy = Accuracy::Probabilistic { epsilon: 0.0, delta: 0.0, norm: ErrorNorm::Pointwise }; -} +This is real, general, and probably something ASAPController needs +eventually for windowed/range-vector PromQL queries — but it composes +the *same* statistic across a partition of *the same data*, which is not +what #171/#172/#173 ask about (composing *different* statistics along a +query's data-flow). It's flagged here and left out of scope; it belongs +in a separate issue, not this design. -impl SummaryKind { - /// The `Accuracy` this kind, at these params, actually satisfies for - /// (G) — the inverse of `boundary::default_size_params`'s sizing - /// formulas (e.g. Kll: ε ≈ 2/k; Hll: ε ≈ 1.04/√(2^p); Cms: ε ≈ e/width, - /// δ ≈ e^(−depth)). `Accuracy::EXACT` for every kind where - /// `is_exact()` holds. - pub fn implied_accuracy(&self, params: &SummaryParams) -> Accuracy { .. } -} -``` +#### Pattern B — hash-collision routing (#173's Hydra and QTree cases) -This is the same abstraction Algebird's `Approximate[T]` monoid packages — -a confidence interval that composes algebraically *within one chain*. The -open question is what happens at the boundary where a second instance of -(G) is layered on top of the *output* of a first, instead of being built -once over raw `X`. +Hydra's "sketch of sketches" hashes *subpopulations* to shared inner +sketch instances the same way CMS hashes *keys* to shared counters — raw +data flows directly into whichever inner instance a subpopulation routes +to; several subpopulations can collide into one. Theorem 2 gives the +combined bound: -#### The composition theorem +``` +Gi(1 − εUS) ≤ Ĝi ≤ Gi(1 + εUS) + ε·GS w.p. ≥ 1 − δ +``` -Let a child summary produce `ṽ = ρ_child(β_child(X))` with `Accuracy A_c` -for `φ_child`. Let an outer summary be built over `ṽ` as if it were exact -input, with `Accuracy A_o` for `φ_outer`, where `φ_outer` is `L`-Lipschitz -with respect to `A_c`'s own representation (its `norm`, if `Probabilistic`; -plain magnitude, if `Bounded`) — call that pairing a `Sensitivity`: +`εUS` is the inner universal sketch's own error; the additive term is +referenced against `GS`, the **global** total across the whole stream, +not against `Gi` itself — an asymmetric, non-self-referential shape that +doesn't fit the same-kind `(ε, δ)`-relative-to-self guarantee every other +`SummaryKind` in this catalog reports. That's precisely why this isn't a +composition of two independently-built `L4Node`s: it needs its own +dedicated `SummaryKind` (its own `w×r` grid, its own `implied_accuracy` +shaped like Theorem 2), fed directly from raw data, with one readout — +not a generic combination of already-built children. + +QTree-style range trees belong in this pattern too, for a different +reason: their accuracy isn't a probabilistic `(ε,δ)` bound at all — it's +a deterministic interval radius that shrinks with tree capacity. Same +conclusion: its own `SummaryKind`, its own accuracy representation +(`Accuracy::Bounded { radius }`, alongside the existing `Probabilistic` +shape every sketch and accumulator already reports), no composition +primitive needed. ```rust -/// `φ_outer`'s sensitivity, valid only against inputs whose error is -/// already stated in `from`. `None`/mismatched `from` means "no proven -/// bound for this combination" — composition must refuse, not guess. -pub struct Sensitivity { - pub lipschitz: f64, - pub from: ErrorNorm, // ignored when composing two `Bounded` accuracies +pub enum Accuracy { + Probabilistic { epsilon: f64, delta: f64, norm: ErrorNorm }, + Bounded { radius: f64 }, } -impl Accuracy { - pub fn compose(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { - match (outer, child) { - ( - Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, - Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, - ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { - epsilon: e_o + sensitivity.lipschitz * e_c, - delta: d_o + d_c, - norm, - }), - (Accuracy::Bounded { radius: r_o }, Accuracy::Bounded { radius: r_c }) => { - Some(Accuracy::Bounded { radius: r_o + sensitivity.lipschitz * r_c }) - } - _ => None, // norm mismatch, or a Probabilistic/Bounded pairing - } - } +pub enum ErrorNorm { + L1, // per-item error relative to Σ|x| — CMS + L2, // per-item error relative to the L2 norm of the (residual) value + // vector — Count-Sketch; tighter than L1 on skewed data, not + // directly convertible to it without a workload-dependent factor + Pointwise, // no underlying vector norm — KLL, HLL, DDSketch, Theta, Kmv } ``` -For two `Probabilistic` accuracies, by the triangle inequality on the two -error terms and a union bound on their two failure events: - -``` -Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_child(X))| > (ε_o + L·ε_c)·φ_outer(φ_child(X)) ] ≤ δ_o + δ_c (G∘) -``` - -For two `Bounded` accuracies the same triangle-inequality shape holds on -the interval radius directly, with no failure probability to union-bound: -`radius_total = radius_outer + L·radius_child`. - -**Why `compose` refuses instead of defaulting to `L = 1`.** Linear -aggregates (`Sum`/`Count`/`Cms`-frequency) are `(L = 1, from: -Pointwise-or-L1)`-sensitive — the clean, dimension-free case most -compositions land in. Rank-based aggregates (`Quantile`/`TopK`) are -`(L = 1, from: Pointwise)`-sensitive only under the standard assumption -that a relative perturbation of each element doesn't move its rank by more -than a proportional amount — an assumption, not a proof. Neither -`Sensitivity` is valid against an `ErrorNorm::L2` child: composing over a -`CountSketch`/`CountSketchWithHeap` child needs its own, separately -justified `Sensitivity { from: L2, .. }` — for a linear outer this is a -real, derivable (if `√n`-dependent, by Cauchy–Schwarz) bound; for a -rank-based outer, sensitivity to an L2-bounded noise vector is -data-dependent (it depends on the local density of values near the query -rank, not just the aggregate norm bound), so no fixed `L` exists for it at -all — a deployment enabling that composition is asserting a -workload-specific bound, not one this crate can derive. - -#### Interface: letting a summary's input be another summary's output - -One new `ColumnRef` variant carries an `Accuracy` alongside the reference, -so every consumer of `col` can see what it's actually built on without -re-deriving it: +`#173`'s answer, in full: Hydra and QTree are both new `SummaryKind` +catalog entries with bespoke accuracy math, not a generalization of the +mechanism built for #171/#172 below. + +#### Pattern D — heterogeneous sequential nesting (#171, #172) + +This is what the two issues actually describe: an outer `AggIntent` +built over an inner `AggIntent` of a *different* kind (`TopK` over +`Rate`; `Max` over `Quantile`; `TopK` over `Cardinality`). None of the +four papers surveyed instantiate this shape — each avoids it +structurally (Pattern A merges same-kind state; Pattern B routes raw +data into a shared inner instance; nothing here ever re-sketches another +sketch's already-collapsed readout). There is no borrowed theorem for +this pattern. It splits into three cases by what the inner side actually +is. + +**D1 — inner is exact (composes for free).** If the inner `AggIntent` +computes something exactly and cheaply from raw data (`Rate`, `Increase`, +`Delta`, `Deriv`, …), there's no need to reference its *readout* at all +— the outer summary can be built directly over the inner's exact, +mergeable **state**, which is definitionally `Accuracy::EXACT` +(`epsilon = delta = 0`). Composing over it adds zero error by +definition, so it's allowed unconditionally: ```rust pub enum ColumnRef { Named(String), Qualified { table: String, name: String }, SampleValue, - FromSummary { nodes: Vec>, accuracy: Accuracy }, // NEW + FromSummary { node: Rc, accuracy: Accuracy }, // NEW } ``` -`nodes` is plural: a single already-realized child is the common case -(`nodes.len() == 1`), but nothing about `compose` or the guarantee above -is specific to one child — summing `(G∘)`'s per-child term over several -children is the same union bound, just applied more than once (`ε_total = -ε_outer + Σᵢ Lᵢ·εᵢ`, `δ_total = δ_outer + Σᵢ δᵢ`). - -Constructing `FromSummary` is legal in exactly two cases: - -- `accuracy == Accuracy::EXACT` — always allowed, unconditionally. -- Any non-`EXACT` `accuracy` — allowed only when the deployment's - `CostModel` opts in: - - ```rust - pub trait CostModel { - /// Does this deployment accept building `outer` over input already - /// carrying `child_kind`'s own approximation error? Default `false` - /// — refuse rather than silently double-approximate. - fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { - false - } - - /// Size `kind`'s params so the composed guarantee holds against - /// `target` (what the query asked for), given the input's own - /// resolved `Accuracy` when it's a `FromSummary` column. Solves - /// `target == Accuracy::compose(result, child_accuracy, sensitivity)` - /// for `result` — e.g. for two `Probabilistic` accuracies, - /// `result.epsilon = target.epsilon − L·child_accuracy.epsilon`, - /// `result.delta = target.delta − child_accuracy.delta` — rejecting - /// if either goes non-positive (no achievable outer accuracy leaves - /// enough of the target budget for the child's contribution). - fn size_params( - &self, - kind: SummaryKind, - intent: &AggIntent, - target: Accuracy, - child_accuracy: Option, // None over raw/exact input - ) -> SummaryParams { .. } - } - ``` - - When `accepts_nested_approx` returns `false` (the default), binding - falls back to `Implementation::PassThrough` instead of constructing an - unsound `FromSummary` column — the same paired-method shape - `realize_extension`/`readout_extension` already uses one layer up. - -#### Interface: folding over a child, as a monoid homomorphism - -Not every composition is an accuracy question. Folding an *exact* function -(`Min`/`Max`/`Avg`/`StdDev`/`Variance`) over a child's already-produced -output is exact by construction — the open question there is which of -those fold operators are valid to apply directly to a child's -already-*collapsed* value, versus which need the child's pre-readout -state. This is exactly Algebird's `Averaged`/moment-monoid distinction: -`Avg`/`Variance`/`StdDev` are not associative on the scalar output (the -average of two averages isn't the true average under unequal group sizes) -— the monoid lives on the sufficient statistic (`Σv`; `Σv, n`; `Σv, Σv², -n`), not on the folded scalar itself. +`accuracy == Accuracy::EXACT` is always a legal `FromSummary` — this is +`#171` direction 2 in full. + +**D2 — outer is an exact fold over an inner realized summary.** `Min`/ +`Max`/`Avg`/`StdDev`/`Variance` folded over an inner *already-realized* +summary (`max by (zone) (quantile_over_time(0.99, m[5m]))`) is exact by +construction on the fold side — the open question is only which of these +fold operators can consume the inner's collapsed scalar directly versus +which need its pre-readout sufficient statistic (the same distinction as +Algebird's `Averaged`/moment monoids: `Avg`/`Variance`/`StdDev` aren't +associative on the scalar output, only on `(Σv, n)` / `(Σv, Σv², n)`): ```rust -pub enum FoldOp { - Min, - Max, - Avg, - Variance { population: bool }, - StdDev { population: bool }, -} +pub enum FoldOp { Min, Max, Avg, Variance { population: bool }, StdDev { population: bool } } pub enum FoldInput { - /// (ℝ, min, +∞) / (ℝ, max, −∞) are commutative monoids on the value - /// itself — the child's already-read-out scalar estimate suffices. - Value, - /// No monoid exists on the scalar average/variance alone — folding - /// needs the child's pre-readout sufficient statistic instead. - SufficientStatistic, + Value, // Min/Max: the inner's already-read-out scalar suffices + SufficientStatistic, // Avg/Variance/StdDev: needs the inner's pre-readout state } impl FoldOp { pub fn required_input(&self) -> FoldInput { match self { FoldOp::Min | FoldOp::Max => FoldInput::Value, - FoldOp::Avg | FoldOp::Variance { .. } | FoldOp::StdDev { .. } => { - FoldInput::SufficientStatistic - } + _ => FoldInput::SufficientStatistic, } } } -SummaryExpr::SummaryFold { - children: Vec>, // per FoldOp::required_input: already-read-out - // values, or pre-readout sufficient-statistic state - fold: FoldOp, - by: Vec, // this node's own grouping — may be coarser than any child's -}, +SummaryExpr::SummaryFold { child: Rc, fold: FoldOp, by: Vec }, +``` + +`SummaryFold` carries no `Accuracy` of its own — the fold is exact, so +it's transparent to whatever accuracy its child already has. This is +`#171` direction 1 in full. + +**D3 — outer approximate over an inner statistic that cannot be computed +exactly in sub-linear space (the genuinely open part of #172).** `Rate` +can always be computed inline from raw data (case D1), so it's never +truly stuck behind an approximate readout. But `Cardinality`/`Quantile`/ +`TopK` as an *inner* statistic have no exact sub-linear form to fall back +to — `topk(5, count(hll_metric) by (key))` genuinely has no choice but to +build the outer `TopK` over the inner HLL's approximate estimate. This is +the actual, narrowed scope of what remains unsolved: not "any nested +approximation," but specifically *outer sketch over an inner statistic +that is inherently sketch-only*. + +For this case only, the design keeps a composition hook — but labeled +honestly as a first-principles derivation this design is proposing, not +a result borrowed from any of the four systems surveyed (none of them +attempt this shape; Hydra's answer to "I need to combine an +inherently-approximate inner statistic with an outer one" is to build a +*single* bespoke `SummaryKind`, per Pattern B, not to compose two): + +```rust +pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } + +impl Accuracy { + /// `None` if `child.norm != sensitivity.from` — refuse rather than + /// silently apply an unproven `lipschitz` constant across a norm it + /// wasn't derived for. + pub fn compose(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + _ => None, + } + } +} + +pub trait CostModel { + /// Does this deployment accept building `outer` over an inner + /// statistic that is inherently sketch-only (Cardinality/Quantile/ + /// TopK as the inner kind)? Default `false` — no known system + /// attempts this composition; refuse rather than assume a Lipschitz + /// bound nobody has proven for this specific pairing. + fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + + fn size_params( + &self, + kind: SummaryKind, + intent: &AggIntent, + target: Accuracy, + child_accuracy: Option, + ) -> SummaryParams { .. } +} ``` -Because `fold` is always exact, `SummaryFold` carries no `Accuracy` of its -own — it's transparent to whatever accuracy its children already have. +`accepts_nested_approx` defaulting to `false` isn't a placeholder pending +proof — it's the design matching what every real system surveyed +actually does: none of them build an outer sketch over an inner sketch's +readout, ever. A deployment that overrides it to `true` is asserting a +`Sensitivity` it has independently justified for its specific +`(outer, child_kind)` pair, not relying on anything this design or the +literature behind it provides. ### Which issue this solves -The design above is issue-agnostic on purpose; here is what each piece -answers. - -**[#171](https://github.com/ProjectASAP/ASAPController/issues/171) — -composed exact ↔ summary aggregation, either nesting order.** - -- *Direction 1* (an outer exact fold, e.g. `max by (zone) - (quantile_over_time(0.99, m[5m]))`, over an inner independently-realized - summary) is answered by `SummaryFold`/`FoldOp`. `Min`/`Max` consume the - child's scalar estimate directly; `Avg`/`StdDev`/`Variance` consume the - child's pre-readout sufficient-statistic state instead, per - `required_input`. No new accumulator kind is needed — today's code path - (`accumulator(intent, SummaryKind::MinMax, ..)`, which requires a real, - independently-materialized `MinMax` instance) is what made this - direction look like it needed one. -- *Direction 2* (an outer summary, e.g. `TopK`, over an inner exact - accumulator like `Rate`) is the `FromSummary` interface at - `accuracy = Accuracy::EXACT`. `compose`'s `Probabilistic` arm with - `A_c = Accuracy::EXACT` returns `A_o` unchanged — composing over an - exact child is free, so this case is allowed unconditionally, with no - `CostModel` opt-in required. - -**[#172](https://github.com/ProjectASAP/ASAPController/issues/172) — -nested approximate-over-approximate composition has no error-propagation -model.** This is the `FromSummary` interface at a non-`EXACT` `Accuracy`: -`compose`'s `Probabilistic` arm with `A_c.epsilon > 0` (or `A_c.delta > -0`) is the formula this issue asked for — `ε_total = ε_outer + L·ε_child`, -`δ_total = δ_outer + δ_child` — gated by `CostModel::accepts_nested_approx` -(default `false`, answering the issue's own open question: refuse to bind -by default rather than silently double-approximate) and sized by -`size_params` receiving `child_accuracy` so the *combined* bound, not just -the outer layer's, is checked against the query's actual target. - -**[#173](https://github.com/ProjectASAP/ASAPController/issues/173) — -multi-dimensional/structured summaries (QTree, Hydra).** Two things fall -out of the design without extra machinery: - -- Hydra's "sketch of sketches" is `FromSummary`/`SummaryFold` with - `nodes`/`children` containing more than one entry — the union bound in - `compose`'s per-child term already generalizes to a sum over children, - so a cross-dimension combiner is the existing interface applied to a - vector instead of a single node, not a new mechanism. -- QTree-style range trees are the `Accuracy::Bounded` variant — a - deterministic interval radius rather than a probabilistic `(ε, δ)` pair. - `compose`'s `Bounded` arm gives the same triangle-inequality composition - without a failure probability to union-bound, so a range-tree kind - slots into `FromSummary`/`SummaryFold` the same way a probabilistic - sketch does; only `compose` and `size_params` need to know which variant - they're holding. +- **[#171](https://github.com/ProjectASAP/ASAPController/issues/171)** — + direction 1 (outer exact fold over inner summary) is Pattern D2, + `SummaryFold`/`FoldOp`. Direction 2 (outer summary over inner exact + accumulator) is Pattern D1, `FromSummary` at `Accuracy::EXACT`, + unconditional. +- **[#172](https://github.com/ProjectASAP/ASAPController/issues/172)** — + Pattern D3 exactly: narrowed from "any nested approximation" to "outer + sketch over an inner statistic with no exact sub-linear form." No + known system solves this generically; the design keeps a gated, + explicitly-unproven `compose`/`accepts_nested_approx` hook rather than + either forbidding the shape outright or pretending a borrowed theorem + covers it. +- **[#173](https://github.com/ProjectASAP/ASAPController/issues/173)** — + Pattern B. Hydra and QTree are each their own `SummaryKind` with + bespoke accuracy math (Theorem 2's shape; a deterministic + `Accuracy::Bounded` radius, respectively) fed directly from raw data — + not a generalization of `FromSummary`/`SummaryFold`. +- **Pattern A** (same-statistic windowing, DGIM-grounded) answers none of + the three issues directly — it's flagged as a real, separate gap + (sliding-window range-vector queries) worth its own future issue. From dfbffedf02be7a00fd5a6404a2510a5843d8a1aa Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:31:55 -0600 Subject: [PATCH 05/10] docs(l4): recipe (not formula) for compound summary bounds, state not readout Reframes the design around the actual structural move every surveyed system (DGIM, UnivMon, Hydra, PromSketch) makes: never compose past a readout -- build the outer structure directly over the inner's state. Replaces the "three patterns" framing with a four-type taxonomy (same- kind merge; inner-exact; heterogeneous-with-derivable-bound; heterogeneous- undecidable) and, for the interesting type-3 case, a four-step recipe for deriving a compound bound instead of a formula: (1) treat inner's state as the outer's input schema, (2) check the outer's own construction algorithm can actually run over that state, (3) if so, re-derive -- not reuse -- the outer's own concentration argument against the composed randomness, (4) accept the resulting bound is pair-specific, not universal (DGIM's (1+e)^2*Cf^2/k+Cf-1+e and Hydra's Gi(1+/-e)+eps*GS are worked examples of the same recipe, not the same formula). Drops the generic Accuracy::compose()/Sensitivity mechanism from the previous draft -- it implied a bound could be computed from kind- independent accuracy values alone, which the DGIM/Hydra worked examples show is false. CostModel::size_params_composed now takes the child's concrete (kind, params) directly and has no default body: the deployment supplies its own pair-specific derivation, or the composition is refused (accepts_composed_state defaults false). FromSummary(Rc) is now constrained to state-producing nodes only (SummaryAgg/SummaryMerge), reusing the same rule this doc's SummaryMerge table already enforces, rather than allowing reference to a readout. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 388 +++++++++++++++--------------------- 1 file changed, 163 insertions(+), 225 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 6ff79993..753d4587 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -312,255 +312,193 @@ pub fn execute(node: &L4Node, exec: &E) -> Result, accuracy: Accuracy }, // NEW -} -``` - -`accuracy == Accuracy::EXACT` is always a legal `FromSummary` — this is -`#171` direction 2 in full. - -**D2 — outer is an exact fold over an inner realized summary.** `Min`/ -`Max`/`Avg`/`StdDev`/`Variance` folded over an inner *already-realized* -summary (`max by (zone) (quantile_over_time(0.99, m[5m]))`) is exact by -construction on the fold side — the open question is only which of these -fold operators can consume the inner's collapsed scalar directly versus -which need its pre-readout sufficient statistic (the same distinction as -Algebird's `Averaged`/moment monoids: `Avg`/`Variance`/`StdDev` aren't -associative on the scalar output, only on `(Σv, n)` / `(Σv, Σv², n)`): - -```rust -pub enum FoldOp { Min, Max, Avg, Variance { population: bool }, StdDev { population: bool } } - -pub enum FoldInput { - Value, // Min/Max: the inner's already-read-out scalar suffices - SufficientStatistic, // Avg/Variance/StdDev: needs the inner's pre-readout state + FromSummary(Rc), // NEW — must be a state-producing node + // (SummaryAgg / SummaryMerge), never + // SummaryEstimate / Logical. Same rule + // as SummaryMerge's children, extended + // to this new consumer. } - -impl FoldOp { - pub fn required_input(&self) -> FoldInput { - match self { - FoldOp::Min | FoldOp::Max => FoldInput::Value, - _ => FoldInput::SufficientStatistic, - } - } -} - -SummaryExpr::SummaryFold { child: Rc, fold: FoldOp, by: Vec }, ``` -`SummaryFold` carries no `Accuracy` of its own — the fold is exact, so -it's transparent to whatever accuracy its child already has. This is -`#171` direction 1 in full. - -**D3 — outer approximate over an inner statistic that cannot be computed -exactly in sub-linear space (the genuinely open part of #172).** `Rate` -can always be computed inline from raw data (case D1), so it's never -truly stuck behind an approximate readout. But `Cardinality`/`Quantile`/ -`TopK` as an *inner* statistic have no exact sub-linear form to fall back -to — `topk(5, count(hll_metric) by (key))` genuinely has no choice but to -build the outer `TopK` over the inner HLL's approximate estimate. This is -the actual, narrowed scope of what remains unsolved: not "any nested -approximation," but specifically *outer sketch over an inner statistic -that is inherently sketch-only*. - -For this case only, the design keeps a composition hook — but labeled -honestly as a first-principles derivation this design is proposing, not -a result borrowed from any of the four systems surveyed (none of them -attempt this shape; Hydra's answer to "I need to combine an -inherently-approximate inner statistic with an outer one" is to build a -*single* bespoke `SummaryKind`, per Pattern B, not to compose two): +This mechanically forbids the "readout, then re-sketch" shape the old +draft of this design allowed for the type-3 case — the exact shape none +of the four systems surveyed ever uses. What it does *not* do is supply +a bound: per the recipe, the bound for a specific `(outer_kind, +inner_kind)` pair is the deployment's own derivation, done once per pair, +following steps 1–4 above. ```rust -pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } - -impl Accuracy { - /// `None` if `child.norm != sensitivity.from` — refuse rather than - /// silently apply an unproven `lipschitz` constant across a norm it - /// wasn't derived for. - pub fn compose(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { - match (outer, child) { - ( - Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, - Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, - ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { - epsilon: e_o + sensitivity.lipschitz * e_c, - delta: d_o + d_c, - norm, - }), - _ => None, - } - } -} - pub trait CostModel { - /// Does this deployment accept building `outer` over an inner - /// statistic that is inherently sketch-only (Cardinality/Quantile/ - /// TopK as the inner kind)? Default `false` — no known system - /// attempts this composition; refuse rather than assume a Lipschitz - /// bound nobody has proven for this specific pairing. - fn accepts_nested_approx(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + /// Has this deployment derived (following steps 1-4 above, or + /// equivalent) a bound for building `outer` over a `FromSummary` + /// input of `child_kind`? Default `false` — matches every system + /// surveyed: none of them build an outer sketch over an inner + /// sketch's state without first doing exactly this derivation for + /// their specific pair. + fn accepts_composed_state(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { false } - fn size_params( + /// Size `kind`'s params given that its input is `child`'s own + /// (kind, params) rather than raw data. Only reachable when + /// `accepts_composed_state` returned `true` for this pair — the + /// deployment's own derivation is what this method encodes; there + /// is no default formula to fall back to; two different pairs get + /// two different bodies, per the recipe's step 4. + fn size_params_composed( &self, kind: SummaryKind, intent: &AggIntent, - target: Accuracy, - child_accuracy: Option, - ) -> SummaryParams { .. } + target_eps: f64, + target_delta: f64, + child: (&SummaryKind, &SummaryParams), + ) -> SummaryParams; } ``` -`accepts_nested_approx` defaulting to `false` isn't a placeholder pending -proof — it's the design matching what every real system surveyed -actually does: none of them build an outer sketch over an inner sketch's -readout, ever. A deployment that overrides it to `true` is asserting a -`Sensitivity` it has independently justified for its specific -`(outer, child_kind)` pair, not relying on anything this design or the -literature behind it provides. +`size_params_composed` deliberately takes the child's concrete `(kind, +params)`, not an abstracted accuracy summary — the whole point of the +recipe is that the derivation is specific to *which two constructions* +are being composed, so the interface shouldn't pretend a +kind-independent accuracy value could carry enough information to size +against. + +`Accuracy` (`implied_accuracy`, `is_exact`) stays as a **reporting** +type — what a single, already-built `SummaryKind` guarantees on its own — +used for type 2 (`Accuracy::EXACT` gates `FromSummary` unconditionally) +and for a deployment to describe what it derived in step 4. It is not a +composition primitive; there is no generic `compose()` over it, because +step 4 established there's nothing generic to compute. ### Which issue this solves - **[#171](https://github.com/ProjectASAP/ASAPController/issues/171)** — - direction 1 (outer exact fold over inner summary) is Pattern D2, - `SummaryFold`/`FoldOp`. Direction 2 (outer summary over inner exact - accumulator) is Pattern D1, `FromSummary` at `Accuracy::EXACT`, - unconditional. + direction 2 (outer summary over inner exact accumulator, e.g. `TopK` + over `Rate`) is compound type 2: `FromSummary` over `Accuracy::EXACT` + state, unconditional. Direction 1 (outer exact fold over inner realized + summary) is a separate axis from this taxonomy entirely — the fold + itself is exact, so it's not a compound-*accuracy* question; it needs + its own `SummaryFold`/`FoldOp` node distinguishing which folds + (`Min`/`Max`) can consume the inner's already-read-out scalar versus + which (`Avg`/`Variance`/`StdDev`, per Algebird's `Averaged`/moment- + monoid distinction) need the inner's pre-readout sufficient statistic. - **[#172](https://github.com/ProjectASAP/ASAPController/issues/172)** — - Pattern D3 exactly: narrowed from "any nested approximation" to "outer - sketch over an inner statistic with no exact sub-linear form." No - known system solves this generically; the design keeps a gated, - explicitly-unproven `compose`/`accepts_nested_approx` hook rather than - either forbidding the shape outright or pretending a borrowed theorem - covers it. + compound type 3 when a deployment has actually done the steps 1-4 + derivation for its specific `(outer, inner)` pair (`accepts_composed_ + state` + `size_params_composed`); compound type 4, and therefore + refused, otherwise. This replaces the earlier draft's generic + `compose()`/`Sensitivity` mechanism, which implied a bound could be + computed from kind-independent accuracy values alone — the DGIM/Hydra + worked examples show that's false; the bound is pair-specific and + requires redoing the inner construction's own proof, not gluing two + numbers together. - **[#173](https://github.com/ProjectASAP/ASAPController/issues/173)** — - Pattern B. Hydra and QTree are each their own `SummaryKind` with - bespoke accuracy math (Theorem 2's shape; a deterministic - `Accuracy::Bounded` radius, respectively) fed directly from raw data — - not a generalization of `FromSummary`/`SummaryFold`. -- **Pattern A** (same-statistic windowing, DGIM-grounded) answers none of - the three issues directly — it's flagged as a real, separate gap - (sliding-window range-vector queries) worth its own future issue. + Hydra and QTree are each type-3 compositions *already worked out* by + their respective papers (Theorem 2; a deterministic range bound) — + concrete instances of the recipe above, not a fourth pattern. Each + still needs its own dedicated `SummaryKind` in the catalog (Hydra's + bound doesn't fit the reporting `Accuracy` shape any single-layer kind + uses, being referenced against a *global* total rather than the + queried subpopulation's own value) rather than being expressed as a + generic `FromSummary` composition — but the *reason* it needs one is + now precise: it's a type-3 pair someone has already derived, exactly + like DGIM/EH is, not a structurally different kind of problem. From 93d2795f055e19733ad304bf3a70d18e1a2adab3 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:38:10 -0600 Subject: [PATCH 06/10] docs(l4): restore readout-based composition as a distinct, legitimate mechanism Correction: the previous commit banned composing over a readout entirely, requiring FromSummary to reference only state-producing nodes. That over-corrected -- readout-based composition (mechanism 3a) is a real, general mechanism in its own right: any inner reporting a public (eps, delta) can be composed via a Lipschitz-style Sensitivity argument (triangle inequality + union bound), without needing access to the inner's raw state at all. It's strictly looser than state-based composition (3b, the DGIM/Hydra recipe) when 3b is available, but it's the only option when the inner's state genuinely isn't accessible (e.g. across a service boundary) or when 3b's construction-compatibility check fails. ColumnRef now has two variants -- FromReadout (value-producing node) and FromState (state-producing node) -- routing to two separate CostModel gates: accepts_readout_composition/sensitivity_for/compose_over_readout for 3a, accepts_state_composition/size_params_from_state for 3b. A deployment facing a novel (outer, inner) pair has a real choice between the two, not one closed gate. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 246 +++++++++++++++++++++++++----------- 1 file changed, 169 insertions(+), 77 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 753d4587..f9bc3cff 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -338,16 +338,89 @@ is." `Increase`, …) has `Accuracy::EXACT` state by construction — composing over it adds nothing, unconditionally. Not really a "compound bound" at all; it's a bound of zero. -3. **Heterogeneous, both approximate, inner's *state* is a valid input to - the outer's own construction.** The genuinely interesting case, and - the subject of the recipe below. `DGIM`/`EH` and `Hydra` are two - **worked examples** of this case, not separate unrelated patterns — - see the recipe. -4. **Heterogeneous, both approximate, no known way to run the outer's - construction over the inner's state.** Structurally undecidable with - what this design can check; stays refused. +3. **Heterogeneous, both approximate.** The genuinely interesting case — + and it splits into **two different mechanisms**, not one, depending on + *what the outer is actually built over*: + - **3a — over the inner's readout.** The outer consumes the inner's + already-collapsed `SummaryEstimate` value as if it were an ordinary + (noisy) input. Always *available* — it needs nothing more than the + inner's published `(ε, δ)` — but the bound it produces is only as + good as a Lipschitz-style sensitivity argument, and is generally + loose. + - **3b — over the inner's state.** The outer's own construction runs + directly on the inner's raw, pre-readout representation. Only + available when the outer's construction can actually consume that + representation's shape (see the recipe below) — but when it applies, + it's tight: `DGIM`/`EH` and `Hydra` are worked examples. + Neither subsumes the other: 3a is the fallback when a deployment only + has API-level access to the inner's estimate (e.g. it crosses a + service boundary and the raw sketch state isn't exposed), or when the + outer's construction genuinely can't consume the inner's state shape + (3b's step 2 fails). 3b is strictly better *when* it's available. +4. **Heterogeneous, both approximate, neither 3a nor 3b apply** — 3a + always applies in principle (any inner reporting an `(ε,δ)` can be + composed via *some* sensitivity argument), so type 4 really means "no + sensitivity argument has been justified for 3a, and 3b's step 2 + fails." Stays refused. + +#### Mechanism 3a: composing over the inner's readout + +This is an ordinary error-propagation argument, not specific to any of +the four systems surveyed (none of them need it, since raw data or state +is always available to them) — but it's the one every deployment +*always* has available, because it only needs the inner's already-public +`(ε, δ)`, not its internals. Let the inner produce `ṽ` satisfying +`Accuracy A_c`, and the outer be built over `ṽ` as if exact, satisfying +`A_o` under that assumption. If the outer's true function `φ_outer` is +`L`-Lipschitz with respect to the norm `A_c.norm` is stated in: -#### The recipe for type 3 +``` +Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_inner(X))| > (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c +``` + +by the triangle inequality on the two error terms and a union bound on +their failure events — the same derivation this design used earlier in +review, now correctly scoped to *only* the readout case rather than +presented as if it covered everything: + +```rust +pub enum ErrorNorm { L1, L2, Pointwise } // see Pattern-B discussion above for L1-vs-L2 + +pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } + +impl Accuracy { + /// `None` if `child.norm != sensitivity.from` — refuse rather than + /// apply an `L` that was never derived for that norm. `Sensitivity` + /// is always a claim the deployment is making about `φ_outer`, not + /// something this crate can derive on its own (linear aggregates + /// like Sum/Count have a provable `L=1`; rank-based ones like + /// Quantile/TopK only have `L≈1` under an unproven local-density + /// assumption — see the earlier discussion of why this must be + /// opt-in). + pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + _ => None, + } + } +} +``` + +This bound is real and computable for *any* `(outer, inner)` pair, +provided a `Sensitivity` is justified — but it's provably looser than 3b +when 3b is available, because it throws away the inner's actual +construction and treats it as an opaque noisy scalar. It's the right +tool when the inner's state genuinely isn't accessible, or as a fallback +when 3b's step 2 fails. + +#### Mechanism 3b: composing over the inner's state (the recipe) Every sketch in this catalog is built by some randomized construction `state = Φ(input)` (a hash-based projection, a compaction tree, …), and @@ -399,52 +472,63 @@ no shortcut that lets you skip re-examining that argument. The recipe is: land on either shape — the recipe produces *a* bound, derived the same way, not *the same* bound. -#### Interface consequence: state, not readout, is what a summary can be built on +#### Interface: two `ColumnRef` variants, one per mechanism Reusing the state/value distinction this doc's `SummaryMerge` table -already established (only a state-producing node — `SummaryAgg` or -`SummaryMerge` — may feed another state-consuming node; `SummaryEstimate` -and `Logical` have already collapsed to a value and may not): +already established (`SummaryAgg`/`SummaryMerge` produce state; +`SummaryEstimate`/`Logical` produce a value), the two mechanisms above +get two distinct `ColumnRef` variants, so which one a query is using is +explicit at the type level rather than inferred: ```rust pub enum ColumnRef { Named(String), Qualified { table: String, name: String }, SampleValue, - FromSummary(Rc), // NEW — must be a state-producing node - // (SummaryAgg / SummaryMerge), never - // SummaryEstimate / Logical. Same rule - // as SummaryMerge's children, extended - // to this new consumer. + FromReadout(Rc), // NEW — mechanism 3a. Must be a value- + // producing node (SummaryEstimate / + // Logical / an exact SummaryAgg with no + // estimate step). + FromState(Rc), // NEW — mechanism 3b. Must be a state- + // producing node (SummaryAgg / + // SummaryMerge). Same rule + // SummaryMerge's children already use, + // extended to this new consumer. } ``` -This mechanically forbids the "readout, then re-sketch" shape the old -draft of this design allowed for the type-3 case — the exact shape none -of the four systems surveyed ever uses. What it does *not* do is supply -a bound: per the recipe, the bound for a specific `(outer_kind, -inner_kind)` pair is the deployment's own derivation, done once per pair, -following steps 1–4 above. +Neither variant supplies a bound by itself — each routes to the matching +half of `CostModel`: ```rust pub trait CostModel { - /// Has this deployment derived (following steps 1-4 above, or - /// equivalent) a bound for building `outer` over a `FromSummary` - /// input of `child_kind`? Default `false` — matches every system - /// surveyed: none of them build an outer sketch over an inner - /// sketch's state without first doing exactly this derivation for - /// their specific pair. - fn accepts_composed_state(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + /// Mechanism 3a. Has this deployment justified a `Sensitivity` for + /// building `outer` over a `FromReadout` input of `child_kind`? + /// Default `false` — a Lipschitz-style claim about `outer`'s own + /// function is always the deployment's to justify, not something + /// this crate derives (see `Accuracy::compose_over_readout`). + fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { false } + fn sensitivity_for(&self, outer: &AggIntent, child_kind: &SummaryKind) -> Option { + None + } - /// Size `kind`'s params given that its input is `child`'s own - /// (kind, params) rather than raw data. Only reachable when - /// `accepts_composed_state` returned `true` for this pair — the - /// deployment's own derivation is what this method encodes; there - /// is no default formula to fall back to; two different pairs get - /// two different bodies, per the recipe's step 4. - fn size_params_composed( + /// Mechanism 3b. Has this deployment derived (following the recipe + /// above, or equivalent) a bound for building `outer` directly over + /// a `FromState` input of `child_kind`? Default `false` — matches + /// every system surveyed: none of them build an outer sketch over an + /// inner sketch's state without first doing exactly this derivation + /// for their specific pair. + fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + /// No default body: two different `(outer, inner)` pairs get two + /// different derivations, per the recipe's step 4. Takes the + /// child's concrete `(kind, params)` directly, not an abstracted + /// accuracy value — the whole point of 3b is that the derivation is + /// specific to *which two constructions* are being composed. + fn size_params_from_state( &self, kind: SummaryKind, intent: &AggIntent, @@ -455,50 +539,58 @@ pub trait CostModel { } ``` -`size_params_composed` deliberately takes the child's concrete `(kind, -params)`, not an abstracted accuracy summary — the whole point of the -recipe is that the derivation is specific to *which two constructions* -are being composed, so the interface shouldn't pretend a -kind-independent accuracy value could carry enough information to size -against. +A deployment facing a genuinely new `(outer, inner)` pair has a real +choice, not just a closed gate: try 3b first (tighter, needs the +recipe's step 2 to succeed), fall back to 3a (always computable given a +justified `Sensitivity`, but looser), or refuse (type 4, the safe +default either way). -`Accuracy` (`implied_accuracy`, `is_exact`) stays as a **reporting** -type — what a single, already-built `SummaryKind` guarantees on its own — -used for type 2 (`Accuracy::EXACT` gates `FromSummary` unconditionally) -and for a deployment to describe what it derived in step 4. It is not a -composition primitive; there is no generic `compose()` over it, because -step 4 established there's nothing generic to compute. +`Accuracy` (`implied_accuracy`, `is_exact`) stays a **reporting** type — +what a single, already-built `SummaryKind` guarantees on its own — used +for type 2 (`Accuracy::EXACT` makes both `FromReadout` and `FromState` +unconditional) and as the input/output of `compose_over_readout`. It is +not, by itself, a composition primitive for 3b — 3b's bound comes from +the recipe, not from any function of two `Accuracy` values. ### Which issue this solves - **[#171](https://github.com/ProjectASAP/ASAPController/issues/171)** — direction 2 (outer summary over inner exact accumulator, e.g. `TopK` - over `Rate`) is compound type 2: `FromSummary` over `Accuracy::EXACT` - state, unconditional. Direction 1 (outer exact fold over inner realized - summary) is a separate axis from this taxonomy entirely — the fold - itself is exact, so it's not a compound-*accuracy* question; it needs - its own `SummaryFold`/`FoldOp` node distinguishing which folds - (`Min`/`Max`) can consume the inner's already-read-out scalar versus - which (`Avg`/`Variance`/`StdDev`, per Algebird's `Averaged`/moment- - monoid distinction) need the inner's pre-readout sufficient statistic. + over `Rate`) is compound type 2: `FromState`/`FromReadout` over + `Accuracy::EXACT`, unconditional either way (state is preferred simply + because it's available and free — there's no accuracy reason to + prefer one over the other when the inner is exact). Direction 1 (outer + exact fold over inner realized summary) is a separate axis from this + taxonomy entirely — the fold itself is exact, so it's not a + compound-*accuracy* question; it needs its own `SummaryFold`/`FoldOp` + node distinguishing which folds (`Min`/`Max`) can consume the inner's + already-read-out scalar versus which (`Avg`/`Variance`/`StdDev`, per + Algebird's `Averaged`/moment-monoid distinction) need the inner's + pre-readout sufficient statistic. - **[#172](https://github.com/ProjectASAP/ASAPController/issues/172)** — - compound type 3 when a deployment has actually done the steps 1-4 - derivation for its specific `(outer, inner)` pair (`accepts_composed_ - state` + `size_params_composed`); compound type 4, and therefore - refused, otherwise. This replaces the earlier draft's generic - `compose()`/`Sensitivity` mechanism, which implied a bound could be - computed from kind-independent accuracy values alone — the DGIM/Hydra - worked examples show that's false; the bound is pair-specific and - requires redoing the inner construction's own proof, not gluing two - numbers together. + compound type 3, with a real choice between two mechanisms rather than + one closed gate: **3b** (`FromState`/`accepts_state_composition`/ + `size_params_from_state`) when a deployment has done the recipe's + derivation for its specific `(outer, inner)` pair and the outer's + construction can actually consume the inner's state shape — tighter, + but not always available. **3a** (`FromReadout`/ + `accepts_readout_composition`/`sensitivity_for`/ + `compose_over_readout`) as the always-available fallback, given a + justified `Sensitivity` — looser, but requires nothing more than the + inner's already-public `(ε, δ)`. Type 4 (refused, the default for + both) when neither has been justified for the pair in question. This + replaces the earlier draft, which only exposed 3b and mechanically + forbade 3a outright — 3a is a real, general, always-derivable (given a + sensitivity claim) mechanism in its own right, just a looser one; it + shouldn't have been designed out. - **[#173](https://github.com/ProjectASAP/ASAPController/issues/173)** — - Hydra and QTree are each type-3 compositions *already worked out* by - their respective papers (Theorem 2; a deterministic range bound) — - concrete instances of the recipe above, not a fourth pattern. Each - still needs its own dedicated `SummaryKind` in the catalog (Hydra's - bound doesn't fit the reporting `Accuracy` shape any single-layer kind - uses, being referenced against a *global* total rather than the - queried subpopulation's own value) rather than being expressed as a - generic `FromSummary` composition — but the *reason* it needs one is - now precise: it's a type-3 pair someone has already derived, exactly - like DGIM/EH is, not a structurally different kind of problem. + Hydra and QTree are each 3b compositions *already worked out* by their + respective papers (Theorem 2; a deterministic range bound) — concrete + instances of the recipe, not a fourth pattern. Each still needs its own + dedicated `SummaryKind` in the catalog (Hydra's bound doesn't fit the + reporting `Accuracy` shape any single-layer kind uses, being + referenced against a *global* total rather than the queried + subpopulation's own value) rather than being expressed as a generic + `FromState` composition — but the *reason* it needs one is now + precise: it's a 3b pair someone has already derived, exactly like + DGIM/EH is, not a structurally different kind of problem. From 2ecc4749719473ce5e3e896518228186a4acf06c Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:45:31 -0600 Subject: [PATCH 07/10] docs(l4): drop the "Which issue this solves" mapping section Requested in review. The design (four compound types, mechanisms 3a/3b) stands on its own; the issue-by-issue mapping was redundant restatement. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 43 ------------------------------------- 1 file changed, 43 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index f9bc3cff..f5758bf9 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -551,46 +551,3 @@ for type 2 (`Accuracy::EXACT` makes both `FromReadout` and `FromState` unconditional) and as the input/output of `compose_over_readout`. It is not, by itself, a composition primitive for 3b — 3b's bound comes from the recipe, not from any function of two `Accuracy` values. - -### Which issue this solves - -- **[#171](https://github.com/ProjectASAP/ASAPController/issues/171)** — - direction 2 (outer summary over inner exact accumulator, e.g. `TopK` - over `Rate`) is compound type 2: `FromState`/`FromReadout` over - `Accuracy::EXACT`, unconditional either way (state is preferred simply - because it's available and free — there's no accuracy reason to - prefer one over the other when the inner is exact). Direction 1 (outer - exact fold over inner realized summary) is a separate axis from this - taxonomy entirely — the fold itself is exact, so it's not a - compound-*accuracy* question; it needs its own `SummaryFold`/`FoldOp` - node distinguishing which folds (`Min`/`Max`) can consume the inner's - already-read-out scalar versus which (`Avg`/`Variance`/`StdDev`, per - Algebird's `Averaged`/moment-monoid distinction) need the inner's - pre-readout sufficient statistic. -- **[#172](https://github.com/ProjectASAP/ASAPController/issues/172)** — - compound type 3, with a real choice between two mechanisms rather than - one closed gate: **3b** (`FromState`/`accepts_state_composition`/ - `size_params_from_state`) when a deployment has done the recipe's - derivation for its specific `(outer, inner)` pair and the outer's - construction can actually consume the inner's state shape — tighter, - but not always available. **3a** (`FromReadout`/ - `accepts_readout_composition`/`sensitivity_for`/ - `compose_over_readout`) as the always-available fallback, given a - justified `Sensitivity` — looser, but requires nothing more than the - inner's already-public `(ε, δ)`. Type 4 (refused, the default for - both) when neither has been justified for the pair in question. This - replaces the earlier draft, which only exposed 3b and mechanically - forbade 3a outright — 3a is a real, general, always-derivable (given a - sensitivity claim) mechanism in its own right, just a looser one; it - shouldn't have been designed out. -- **[#173](https://github.com/ProjectASAP/ASAPController/issues/173)** — - Hydra and QTree are each 3b compositions *already worked out* by their - respective papers (Theorem 2; a deterministic range bound) — concrete - instances of the recipe, not a fourth pattern. Each still needs its own - dedicated `SummaryKind` in the catalog (Hydra's bound doesn't fit the - reporting `Accuracy` shape any single-layer kind uses, being - referenced against a *global* total rather than the queried - subpopulation's own value) rather than being expressed as a generic - `FromState` composition — but the *reason* it needs one is now - precise: it's a 3b pair someone has already derived, exactly like - DGIM/EH is, not a structurally different kind of problem. From 4442b3b3f348fe5be29240506ead1e947d55e6c3 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:48:08 -0600 Subject: [PATCH 08/10] docs(l4): tighten prose, cut redundant explanation No content change -- same taxonomy, same two mechanisms, same recipe, same interfaces. Trims repeated justification and meta-commentary about prior review rounds, shortens sentences, removes duplicate restatement of the DGIM/Hydra examples across steps. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 276 +++++++++++++----------------------- 1 file changed, 99 insertions(+), 177 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index f5758bf9..f2726ec8 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -312,91 +312,60 @@ pub fn execute(node: &L4Node, exec: &E) -> Result (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c ``` -by the triangle inequality on the two error terms and a union bound on -their failure events — the same derivation this design used earlier in -review, now correctly scoped to *only* the readout case rather than -presented as if it covered everything: +by the triangle inequality plus a union bound on the two failure events: ```rust -pub enum ErrorNorm { L1, L2, Pointwise } // see Pattern-B discussion above for L1-vs-L2 +pub enum ErrorNorm { L1, L2, Pointwise } pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } impl Accuracy { - /// `None` if `child.norm != sensitivity.from` — refuse rather than - /// apply an `L` that was never derived for that norm. `Sensitivity` - /// is always a claim the deployment is making about `φ_outer`, not - /// something this crate can derive on its own (linear aggregates - /// like Sum/Count have a provable `L=1`; rank-based ones like - /// Quantile/TopK only have `L≈1` under an unproven local-density - /// assumption — see the earlier discussion of why this must be - /// opt-in). + /// `None` on a norm mismatch — refuse rather than apply an `L` that + /// was never derived for it. `Sensitivity` is always the + /// deployment's own claim (linear aggregates like Sum/Count have a + /// provable `L=1`; rank-based ones like Quantile/TopK only have + /// `L≈1` under an unproven local-density assumption). pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { match (outer, child) { ( @@ -413,100 +382,64 @@ impl Accuracy { } ``` -This bound is real and computable for *any* `(outer, inner)` pair, -provided a `Sensitivity` is justified — but it's provably looser than 3b -when 3b is available, because it throws away the inner's actual -construction and treats it as an opaque noisy scalar. It's the right -tool when the inner's state genuinely isn't accessible, or as a fallback -when 3b's step 2 fails. - -#### Mechanism 3b: composing over the inner's state (the recipe) - -Every sketch in this catalog is built by some randomized construction -`state = Φ(input)` (a hash-based projection, a compaction tree, …), and -its published accuracy bound is proved by a *specific* concentration or -counting argument applied to that construction — CMS's is a Markov bound -over hash-collision mass; KLL's is a compaction-invariant argument; HLL's -is a variance calculation over the max-order-statistic register. There is -no shortcut that lets you skip re-examining that argument. The recipe is: - -1. **Treat the inner's state as the outer's input schema.** Not the - inner's readout — the inner's raw, pre-`SummaryEstimate` state, typed - the same way this doc already types it (`L4DataType::Sketch(kind, - params)`). This is a schema-level move, not a numerical one: it's the - difference between `outer built over Σ(inner)` and `outer built over - ρ(inner)`. -2. **Ask whether the outer's own construction algorithm can literally run - with the inner's state standing in for its usual raw input.** For - `DGIM`/`EH`, the outer (windowing) construction is *bucket - concatenation*, and any composable sketch's state supports that by - definition (property P5) — always yes. For `Hydra`, the outer - (hash-routing) construction just needs *something hashable to route*, - which any subpopulation identifier is — always yes, for its specific - shape. For an arbitrary `(outer, inner)` pair, this is not automatic — - e.g. running CMS's own hash-and-increment construction *again*, over - another CMS's counter array treated as a fresh multiset of - "(bucket-index, count)" items, is well-defined; running a rank-based - KLL construction over an HLL's register array is not obviously - meaningful, because KLL's construction needs orderable *items*, and an - HLL register isn't one. If this step fails, stop — you're in type 4. -3. **If step 2 succeeds, re-derive — don't reuse — the outer's own - concentration argument against the *composed* randomness.** The - composed construction is `Φ_outer ∘ Φ_inner`; the question is whether - the specific proof technique `Φ_outer`'s bound normally relies on - (independence assumptions, moment bounds, …) still holds when its - input distribution is "the output of `Φ_inner`" instead of "raw - samples." `DGIM` does exactly this in Theorem 7: the windowing - argument (Observation 1/2) is re-run assuming the per-bucket sketch - itself only supplies a `(1±ε̂)`-approximate `f`, not an exact one, and - the bound `(1+ε̂)²Cf²/k + Cf−1+ε̂` is what falls out. `Hydra` does the - same in Theorem 2: the routing argument (Markov on collision mass + - Chernoff over independent rows) is re-run treating each cell's - universal-sketch estimate as the noisy quantity, giving the - asymmetric `Gi(1±εUS) + ε·GS` shape. Neither bound was available by - composing two pre-existing formulas — both required redoing their - respective single-layer proof against the two-layer construction. -4. **The resulting bound's *shape* is pair-specific, not universal.** - `DGIM`'s is `(1+ε̂)²Cf²/k+Cf−1+ε̂`; `Hydra`'s is `Gi(1±εUS)+ε·GS`. - There is no reason to expect a third, novel `(outer, inner)` pair to - land on either shape — the recipe produces *a* bound, derived the same - way, not *the same* bound. - -#### Interface: two `ColumnRef` variants, one per mechanism - -Reusing the state/value distinction this doc's `SummaryMerge` table -already established (`SummaryAgg`/`SummaryMerge` produce state; -`SummaryEstimate`/`Logical` produce a value), the two mechanisms above -get two distinct `ColumnRef` variants, so which one a query is using is -explicit at the type level rather than inferred: +Computable for any pair given a justified `Sensitivity`, but looser than +3b, since it treats the inner as an opaque noisy scalar. + +#### 3b: composing over the inner's state (the recipe) + +Every sketch here is a randomized construction `state = Φ(input)`, and +its bound is proved by a specific argument over that construction — a +Markov bound over hash-collision mass for CMS, a compaction invariant for +KLL, a variance calculation over the max-order-statistic register for +HLL. There's no shortcut around re-examining that argument: + +1. **Treat the inner's state as the outer's input schema** — the raw, + pre-`SummaryEstimate` state (`L4DataType::Sketch(kind, params)`), not + the readout. The difference between `outer built over Σ(inner)` and + `outer built over ρ(inner)`. +2. **Check whether the outer's construction can run with the inner's + state standing in for raw input.** DGIM/EH's windowing construction + just concatenates buckets, which any composable sketch's state + supports (property P5) — always yes. Hydra's hash-routing just needs + something hashable — always yes for its shape. For an arbitrary pair + this isn't automatic: running CMS's hash-and-increment construction + over another CMS's counter array (treated as fresh `(bucket, count)` + items) is well-defined; running KLL's construction, which needs + orderable items, over an HLL's register array is not. If this fails, + the pair is type 4. +3. **If it succeeds, re-derive — don't reuse — the outer's concentration + argument against the composed construction `Φ_outer ∘ Φ_inner`.** + DGIM's Theorem 7 re-runs its windowing argument assuming the + per-bucket sketch is itself only `(1±ε̂)`-accurate, giving + `(1+ε̂)²Cf²/k+Cf−1+ε̂`. Hydra's Theorem 2 re-runs its Markov/Chernoff + routing argument treating each cell's estimate as noisy, giving the + asymmetric `Gi(1±εUS)+ε·GS`. Neither came from combining pre-existing + formulas. +4. **The result is pair-specific, not universal** — a third pair should + not be expected to land on either shape above. + +#### Interface + +Two `ColumnRef` variants make which mechanism a query uses explicit at +the type level, reusing the state/value distinction `SummaryMerge` +already enforces (`SummaryAgg`/`SummaryMerge` produce state; +`SummaryEstimate`/`Logical` produce a value): ```rust pub enum ColumnRef { Named(String), Qualified { table: String, name: String }, SampleValue, - FromReadout(Rc), // NEW — mechanism 3a. Must be a value- - // producing node (SummaryEstimate / - // Logical / an exact SummaryAgg with no - // estimate step). - FromState(Rc), // NEW — mechanism 3b. Must be a state- - // producing node (SummaryAgg / - // SummaryMerge). Same rule - // SummaryMerge's children already use, - // extended to this new consumer. + FromReadout(Rc), // NEW — 3a; must be value-producing + FromState(Rc), // NEW — 3b; must be state-producing } ``` -Neither variant supplies a bound by itself — each routes to the matching -half of `CostModel`: +Each routes to its own half of `CostModel`, both defaulting closed: ```rust pub trait CostModel { - /// Mechanism 3a. Has this deployment justified a `Sensitivity` for - /// building `outer` over a `FromReadout` input of `child_kind`? - /// Default `false` — a Lipschitz-style claim about `outer`'s own - /// function is always the deployment's to justify, not something - /// this crate derives (see `Accuracy::compose_over_readout`). + /// 3a: has the deployment justified a Sensitivity for this pair? fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { false } @@ -514,20 +447,13 @@ pub trait CostModel { None } - /// Mechanism 3b. Has this deployment derived (following the recipe - /// above, or equivalent) a bound for building `outer` directly over - /// a `FromState` input of `child_kind`? Default `false` — matches - /// every system surveyed: none of them build an outer sketch over an - /// inner sketch's state without first doing exactly this derivation - /// for their specific pair. + /// 3b: has the deployment derived a bound for this pair? fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { false } - /// No default body: two different `(outer, inner)` pairs get two - /// different derivations, per the recipe's step 4. Takes the - /// child's concrete `(kind, params)` directly, not an abstracted - /// accuracy value — the whole point of 3b is that the derivation is - /// specific to *which two constructions* are being composed. + /// No default body — each pair gets its own derivation. Takes the + /// child's concrete (kind, params), not an abstracted accuracy + /// value, since the derivation depends on the specific pair. fn size_params_from_state( &self, kind: SummaryKind, @@ -539,15 +465,11 @@ pub trait CostModel { } ``` -A deployment facing a genuinely new `(outer, inner)` pair has a real -choice, not just a closed gate: try 3b first (tighter, needs the -recipe's step 2 to succeed), fall back to 3a (always computable given a -justified `Sensitivity`, but looser), or refuse (type 4, the safe -default either way). - -`Accuracy` (`implied_accuracy`, `is_exact`) stays a **reporting** type — -what a single, already-built `SummaryKind` guarantees on its own — used -for type 2 (`Accuracy::EXACT` makes both `FromReadout` and `FromState` -unconditional) and as the input/output of `compose_over_readout`. It is -not, by itself, a composition primitive for 3b — 3b's bound comes from -the recipe, not from any function of two `Accuracy` values. +A deployment facing a new pair has a real choice: try 3b, fall back to +3a, or refuse. + +`Accuracy` (`implied_accuracy`, `is_exact`) stays a reporting type — what +a single `SummaryKind` guarantees on its own — used for type 2 and as +`compose_over_readout`'s input/output. It isn't a composition primitive +for 3b; 3b's bound comes from the recipe, not from a function of two +`Accuracy` values. From 0f884ac51bd5dd49f3bfa22eb2f935dc63381635 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:51:46 -0600 Subject: [PATCH 09/10] docs(l4): make the 3b recipe steps readable -- one idea per step, examples in a table The four recipe steps each interleaved DGIM and Hydra inline, forcing readers to track two examples at once inside every sentence. Splits the steps (now plain, one idea each) from the worked examples (DGIM/EH, Hydra, and a KLL-over-HLL counter-example), laid out as a comparison table keyed to steps 2 and 3. Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 56 +++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index f2726ec8..978b13c8 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -393,30 +393,38 @@ Markov bound over hash-collision mass for CMS, a compaction invariant for KLL, a variance calculation over the max-order-statistic register for HLL. There's no shortcut around re-examining that argument: -1. **Treat the inner's state as the outer's input schema** — the raw, - pre-`SummaryEstimate` state (`L4DataType::Sketch(kind, params)`), not - the readout. The difference between `outer built over Σ(inner)` and - `outer built over ρ(inner)`. -2. **Check whether the outer's construction can run with the inner's - state standing in for raw input.** DGIM/EH's windowing construction - just concatenates buckets, which any composable sketch's state - supports (property P5) — always yes. Hydra's hash-routing just needs - something hashable — always yes for its shape. For an arbitrary pair - this isn't automatic: running CMS's hash-and-increment construction - over another CMS's counter array (treated as fresh `(bucket, count)` - items) is well-defined; running KLL's construction, which needs - orderable items, over an HLL's register array is not. If this fails, - the pair is type 4. -3. **If it succeeds, re-derive — don't reuse — the outer's concentration - argument against the composed construction `Φ_outer ∘ Φ_inner`.** - DGIM's Theorem 7 re-runs its windowing argument assuming the - per-bucket sketch is itself only `(1±ε̂)`-accurate, giving - `(1+ε̂)²Cf²/k+Cf−1+ε̂`. Hydra's Theorem 2 re-runs its Markov/Chernoff - routing argument treating each cell's estimate as noisy, giving the - asymmetric `Gi(1±εUS)+ε·GS`. Neither came from combining pre-existing - formulas. -4. **The result is pair-specific, not universal** — a third pair should - not be expected to land on either shape above. +1. **Feed the outer sketch the inner's state, not its answer.** Use the + inner's raw, pre-`SummaryEstimate` state — typed here as + `L4DataType::Sketch(kind, params)` — as the outer's input, instead of + a scalar readout. +2. **Check the outer's own build procedure can actually run on that + state.** Does feeding it the inner's state, instead of raw data, still + make sense? This can fail — see the worked examples below for a case + where it does and one where it clearly doesn't. If it fails, the pair + is type 4: refused. +3. **If it works, prove a new bound — don't reuse the outer's old one.** + The outer's published bound assumed clean raw input; it says nothing + about input that's itself another sketch's noisy state. The outer's + own proof technique has to be redone against this two-stage + construction. +4. **Expect a different bound for each new pair.** There's no fixed + shape this converges to — the two worked examples below land on two + different formulas. + +**Worked examples.** DGIM/EH (a sketch built by concatenating other +sketches' state across time buckets) and Hydra (a sketch that routes +into other sketches by hash) both follow this recipe: + +| | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound | +|---|---|---| +| DGIM / EH | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | +| Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. | +| *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) | + +Neither DGIM's nor Hydra's bound came from combining two pre-existing +formulas — both required redoing the outer's own proof for the two-layer +construction. A third, novel pair should not be expected to land on +either shape. #### Interface From f5469c4528e6c2caccc6e627598dd5f25fddb449 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 29 Jul 2026 21:57:22 -0600 Subject: [PATCH 10/10] docs(l4): rename DGIM/EH to Exponential Histogram [Datar et al., SICOMP'02] Co-Authored-By: Claude Sonnet 5 --- docs/l4-summary-bound-ir.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 978b13c8..7b1683e2 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -315,7 +315,8 @@ on `Reduce` vs. `PerEntity` there, not guess from an empty key list. ### Design proposal: composing summaries — a taxonomy and a recipe, not a formula There's no single formula for the error bound of "summary built over -another summary." DGIM [SICOMP'02] and Hydra [VLDB'22] each prove a +another summary." Exponential Histogram [Datar, Gionis, Indyk, Motwani, +SICOMP'02] and Hydra [VLDB'22] each prove a bound for this shape, and the two bounds have different forms, because they come from composing different concentration arguments, not from adding two epsilons. What is general is a method. Four compound types, @@ -335,7 +336,8 @@ by structural relationship: - **3b — over the inner's state.** The outer's construction runs directly on the inner's raw, pre-readout representation. Only available when the outer's construction can consume that - representation, but tight when it applies — DGIM/EH and Hydra are + representation, but tight when it applies — Exponential Histogram + and Hydra are worked examples. 3b is strictly tighter when available; 3a is the fallback when the inner's state isn't accessible (e.g. across a service boundary) or 3b @@ -411,17 +413,17 @@ HLL. There's no shortcut around re-examining that argument: shape this converges to — the two worked examples below land on two different formulas. -**Worked examples.** DGIM/EH (a sketch built by concatenating other +**Worked examples.** Exponential Histogram (a sketch built by concatenating other sketches' state across time buckets) and Hydra (a sketch that routes into other sketches by hash) both follow this recipe: | | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound | |---|---|---| -| DGIM / EH | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | +| Exponential Histogram | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | | Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. | | *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) | -Neither DGIM's nor Hydra's bound came from combining two pre-existing +Neither Exponential Histogram's nor Hydra's bound came from combining two pre-existing formulas — both required redoing the outer's own proof for the two-layer construction. A third, novel pair should not be expected to land on either shape.