diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md index 0f0c4da2..7b1683e2 100644 --- a/docs/l4-summary-bound-ir.md +++ b/docs/l4-summary-bound-ir.md @@ -117,7 +117,10 @@ already-computed value, at query time." Every design considered so far only builds summaries incrementally from raw samples, at ingest time; there's no established answer for building one from a query-time-derived scalar. A deployment that hits this shape should -treat it as unsupported rather than assume either answer. +treat it as unsupported rather than assume either answer. The design +proposal below narrows exactly which cases of this remain genuinely +open (see "Pattern D" below) — it is not fully open anymore, but it is +not fully resolved either. ### What's out of scope here @@ -308,3 +311,175 @@ pub fn execute(node: &L4Node, exec: &E) -> Result (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c +``` + +by the triangle inequality plus a union bound on the two failure events: + +```rust +pub enum ErrorNorm { L1, L2, Pointwise } + +pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm } + +impl Accuracy { + /// `None` on a norm mismatch — refuse rather than apply an `L` that + /// was never derived for it. `Sensitivity` is always the + /// deployment's own claim (linear aggregates like Sum/Count have a + /// provable `L=1`; rank-based ones like Quantile/TopK only have + /// `L≈1` under an unproven local-density assumption). + pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option { + match (outer, child) { + ( + Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm }, + Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm }, + ) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic { + epsilon: e_o + sensitivity.lipschitz * e_c, + delta: d_o + d_c, + norm, + }), + _ => None, + } + } +} +``` + +Computable for any pair given a justified `Sensitivity`, but looser than +3b, since it treats the inner as an opaque noisy scalar. + +#### 3b: composing over the inner's state (the recipe) + +Every sketch here is a randomized construction `state = Φ(input)`, and +its bound is proved by a specific argument over that construction — a +Markov bound over hash-collision mass for CMS, a compaction invariant for +KLL, a variance calculation over the max-order-statistic register for +HLL. There's no shortcut around re-examining that argument: + +1. **Feed the outer sketch the inner's state, not its answer.** Use the + inner's raw, pre-`SummaryEstimate` state — typed here as + `L4DataType::Sketch(kind, params)` — as the outer's input, instead of + a scalar readout. +2. **Check the outer's own build procedure can actually run on that + state.** Does feeding it the inner's state, instead of raw data, still + make sense? This can fail — see the worked examples below for a case + where it does and one where it clearly doesn't. If it fails, the pair + is type 4: refused. +3. **If it works, prove a new bound — don't reuse the outer's old one.** + The outer's published bound assumed clean raw input; it says nothing + about input that's itself another sketch's noisy state. The outer's + own proof technique has to be redone against this two-stage + construction. +4. **Expect a different bound for each new pair.** There's no fixed + shape this converges to — the two worked examples below land on two + different formulas. + +**Worked examples.** Exponential Histogram (a sketch built by concatenating other +sketches' state across time buckets) and Hydra (a sketch that routes +into other sketches by hash) both follow this recipe: + +| | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound | +|---|---|---| +| Exponential Histogram | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. | +| Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. | +| *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) | + +Neither Exponential Histogram's nor Hydra's bound came from combining two pre-existing +formulas — both required redoing the outer's own proof for the two-layer +construction. A third, novel pair should not be expected to land on +either shape. + +#### Interface + +Two `ColumnRef` variants make which mechanism a query uses explicit at +the type level, reusing the state/value distinction `SummaryMerge` +already enforces (`SummaryAgg`/`SummaryMerge` produce state; +`SummaryEstimate`/`Logical` produce a value): + +```rust +pub enum ColumnRef { + Named(String), + Qualified { table: String, name: String }, + SampleValue, + FromReadout(Rc), // NEW — 3a; must be value-producing + FromState(Rc), // NEW — 3b; must be state-producing +} +``` + +Each routes to its own half of `CostModel`, both defaulting closed: + +```rust +pub trait CostModel { + /// 3a: has the deployment justified a Sensitivity for this pair? + fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + fn sensitivity_for(&self, outer: &AggIntent, child_kind: &SummaryKind) -> Option { + None + } + + /// 3b: has the deployment derived a bound for this pair? + fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool { + false + } + /// No default body — each pair gets its own derivation. Takes the + /// child's concrete (kind, params), not an abstracted accuracy + /// value, since the derivation depends on the specific pair. + fn size_params_from_state( + &self, + kind: SummaryKind, + intent: &AggIntent, + target_eps: f64, + target_delta: f64, + child: (&SummaryKind, &SummaryParams), + ) -> SummaryParams; +} +``` + +A deployment facing a new pair has a real choice: try 3b, fall back to +3a, or refuse. + +`Accuracy` (`implied_accuracy`, `is_exact`) stays a reporting type — what +a single `SummaryKind` guarantees on its own — used for type 2 and as +`compose_over_readout`'s input/output. It isn't a composition primitive +for 3b; 3b's bound comes from the recipe, not from a function of two +`Accuracy` values.