Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 176 additions & 1 deletion docs/l4-summary-bound-ir.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,10 @@ already-computed value, at query time." Every design considered so far
only builds summaries incrementally from raw samples, at ingest time;
there's no established answer for building one from a
query-time-derived scalar. A deployment that hits this shape should
treat it as unsupported rather than assume either answer.
treat it as unsupported rather than assume either answer. The design
proposal below narrows exactly which cases of this remain genuinely
open (see "Pattern D" below) — it is not fully open anymore, but it is
not fully resolved either.

### What's out of scope here

Expand Down Expand Up @@ -308,3 +311,175 @@ pub fn execute<E: SummaryExecutor>(node: &L4Node, exec: &E) -> Result<ExecOutcom
`find_candidates`'s `reduction` parameter is exactly the `Reduction`
this doc's design section already explains — an implementer must branch
on `Reduce` vs. `PerEntity` there, not guess from an empty key list.

### Design proposal: composing summaries — a taxonomy and a recipe, not a formula

There's no single formula for the error bound of "summary built over
another summary." Exponential Histogram [Datar, Gionis, Indyk, Motwani,
SICOMP'02] and Hydra [VLDB'22] each prove a
bound for this shape, and the two bounds have different forms, because
they come from composing different concentration arguments, not from
adding two epsilons. What is general is a method. Four compound types,
by structural relationship:

1. **Same kind, same config — merge.** Already `SummaryMerge`: exact at
the state level, zero added error, gated by the catalog's `mergeable`
flag.
2. **Heterogeneous, inner exact.** An exact inner (`Rate`, `Increase`, …)
has `Accuracy::EXACT` state, so composing over it adds nothing,
unconditionally.
3. **Heterogeneous, both approximate.** Splits into two mechanisms:
- **3a — over the inner's readout.** The outer consumes the inner's
already-collapsed estimate as an ordinary noisy input. Always
available — needs only the inner's published `(ε, δ)` — but only as
tight as a Lipschitz-style sensitivity argument, generally loose.
- **3b — over the inner's state.** The outer's construction runs
directly on the inner's raw, pre-readout representation. Only
available when the outer's construction can consume that
representation, but tight when it applies — Exponential Histogram
and Hydra are
worked examples.
3b is strictly tighter when available; 3a is the fallback when the
inner's state isn't accessible (e.g. across a service boundary) or 3b
doesn't apply.
4. **Neither 3a nor 3b justified for the pair.** Refused by default.

#### 3a: composing over the inner's readout

An ordinary error-propagation argument. Let the inner produce `ṽ`
satisfying `Accuracy A_c`, and the outer be built over `ṽ` as if exact,
satisfying `A_o`. If the outer's true function `φ_outer` is `L`-Lipschitz
with respect to the norm `A_c` is stated in:

```
Pr[ |ρ_outer(β_outer(ṽ)) − φ_outer(φ_inner(X))| > (ε_o + L·ε_c)·φ_outer(φ_inner(X)) ] ≤ δ_o + δ_c
```

by the triangle inequality plus a union bound on the two failure events:

```rust
pub enum ErrorNorm { L1, L2, Pointwise }

pub struct Sensitivity { pub lipschitz: f64, pub from: ErrorNorm }

impl Accuracy {
/// `None` on a norm mismatch — refuse rather than apply an `L` that
/// was never derived for it. `Sensitivity` is always the
/// deployment's own claim (linear aggregates like Sum/Count have a
/// provable `L=1`; rank-based ones like Quantile/TopK only have
/// `L≈1` under an unproven local-density assumption).
pub fn compose_over_readout(outer: Accuracy, child: Accuracy, sensitivity: Sensitivity) -> Option<Accuracy> {
match (outer, child) {
(
Accuracy::Probabilistic { epsilon: e_o, delta: d_o, norm },
Accuracy::Probabilistic { epsilon: e_c, delta: d_c, norm: child_norm },
) if child_norm == sensitivity.from => Some(Accuracy::Probabilistic {
epsilon: e_o + sensitivity.lipschitz * e_c,
delta: d_o + d_c,
norm,
}),
_ => None,
}
}
}
```

Computable for any pair given a justified `Sensitivity`, but looser than
3b, since it treats the inner as an opaque noisy scalar.

#### 3b: composing over the inner's state (the recipe)

Every sketch here is a randomized construction `state = Φ(input)`, and
its bound is proved by a specific argument over that construction — a
Markov bound over hash-collision mass for CMS, a compaction invariant for
KLL, a variance calculation over the max-order-statistic register for
HLL. There's no shortcut around re-examining that argument:

1. **Feed the outer sketch the inner's state, not its answer.** Use the
inner's raw, pre-`SummaryEstimate` state — typed here as
`L4DataType::Sketch(kind, params)` — as the outer's input, instead of
a scalar readout.
2. **Check the outer's own build procedure can actually run on that
state.** Does feeding it the inner's state, instead of raw data, still
make sense? This can fail — see the worked examples below for a case
where it does and one where it clearly doesn't. If it fails, the pair
is type 4: refused.
3. **If it works, prove a new bound — don't reuse the outer's old one.**
The outer's published bound assumed clean raw input; it says nothing
about input that's itself another sketch's noisy state. The outer's
own proof technique has to be redone against this two-stage
construction.
4. **Expect a different bound for each new pair.** There's no fixed
shape this converges to — the two worked examples below land on two
different formulas.

**Worked examples.** Exponential Histogram (a sketch built by concatenating other
sketches' state across time buckets) and Hydra (a sketch that routes
into other sketches by hash) both follow this recipe:

| | Step 2: does the outer's construction run on the inner's state? | Step 3: the re-derived bound |
|---|---|---|
| Exponential Histogram | Yes — the outer just concatenates buckets, and any composable sketch's state supports that (property P5). | `(1+ε̂)²Cf²/k + Cf − 1 + ε̂`, from re-running the windowing argument assuming the inner sketch is itself only `(1±ε̂)`-accurate. |
| Hydra | Yes — the outer just needs something hashable to route on, which any sub-population id is. | `Gi(1±εUS) + ε·GS`, from re-running the Markov/Chernoff routing argument treating each cell's inner estimate as noisy. |
| *Counter-example* | Not always — e.g. KLL's construction needs a stream of orderable items; an HLL's internal registers aren't that, so KLL can't run directly on HLL state. | — (type 4: refused) |

Neither Exponential Histogram's nor Hydra's bound came from combining two pre-existing
formulas — both required redoing the outer's own proof for the two-layer
construction. A third, novel pair should not be expected to land on
either shape.

#### Interface

Two `ColumnRef` variants make which mechanism a query uses explicit at
the type level, reusing the state/value distinction `SummaryMerge`
already enforces (`SummaryAgg`/`SummaryMerge` produce state;
`SummaryEstimate`/`Logical` produce a value):

```rust
pub enum ColumnRef {
Named(String),
Qualified { table: String, name: String },
SampleValue,
FromReadout(Rc<L4Node>), // NEW — 3a; must be value-producing
FromState(Rc<L4Node>), // NEW — 3b; must be state-producing
}
```

Each routes to its own half of `CostModel`, both defaulting closed:

```rust
pub trait CostModel {
/// 3a: has the deployment justified a Sensitivity for this pair?
fn accepts_readout_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool {
false
}
fn sensitivity_for(&self, outer: &AggIntent, child_kind: &SummaryKind) -> Option<Sensitivity> {
None
}

/// 3b: has the deployment derived a bound for this pair?
fn accepts_state_composition(&self, outer: &AggIntent, child_kind: &SummaryKind) -> bool {
false
}
/// No default body — each pair gets its own derivation. Takes the
/// child's concrete (kind, params), not an abstracted accuracy
/// value, since the derivation depends on the specific pair.
fn size_params_from_state(
&self,
kind: SummaryKind,
intent: &AggIntent,
target_eps: f64,
target_delta: f64,
child: (&SummaryKind, &SummaryParams),
) -> SummaryParams;
}
```

A deployment facing a new pair has a real choice: try 3b, fall back to
3a, or refuse.

`Accuracy` (`implied_accuracy`, `is_exact`) stays a reporting type — what
a single `SummaryKind` guarantees on its own — used for type 2 and as
`compose_over_readout`'s input/output. It isn't a composition primitive
for 3b; 3b's bound comes from the recipe, not from a function of two
`Accuracy` values.