From b78c362d70bee8b211e5007da87c65e1972fcd63 Mon Sep 17 00:00:00 2001 From: zzylol Date: Thu, 16 Jul 2026 12:26:36 -0600 Subject: [PATCH 1/3] docs(gos): unified isotropic-threshold design + per-family derivations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit design-gos-unified-edge-telemetry.md: new §11 "The 2026-07 redesign: no sub-window, insert-time detection for all 6 families" — the unified model (check-at-insert, sync-only, alerting moves to the backend), the wake-on-demand flush mechanism that keeps the existing OTLP/ SketchEnvelope pipeline (only the trigger changes, from purely-timer to timer-or-woken), retiring Gate 1 (subWindowShouldEmit/ackedCells) and CMS's local point-query read, a per-family cell/threshold/reset mechanism table, why the cold-start "threshold near zero" behavior is intentional, and three explicitly-scoped open items (anisotropic Activity_j redesign, DDSketch's unbounded bucket growth — sketchlib-go#72, independent of this work — and HLL's unverified small-cardinality regime). Old §11/§12 renumbered to §12/§13, with a cross-reference note where the old "acked snapshot" description is superseded. sampling-cdm-gos-derivations.md §8: fills in the previously-empty CDM staleness/threshold derivations for CMS (§8.2, L1/max-composition, T=εN/k — no √(dw), unlike CountSketch's L2/sum-composition), DDSketch range counts (§8.4, L1/sum-composition, T=εN/(kB) with B tracked O(1) per new bucket rather than assumed — an assumed constant would let the staleness bound be silently violated, not just loosened), KLL's concrete emit trigger (§8.6, R>=εN from the existing R/N bound), and HLL's register-change adapter (§8.7, OctoSketch's own max-merge treatment: |2^C'-2^C|>=2^tau on the linearized value, never reset — MAX-merge is idempotent). §11 summary table gets an "Isotropic threshold" column tying all 6 families together; §13 implementation notes updated to match (sub-window language replaced, CMS point-query retirement noted). --- docs/design-gos-unified-edge-telemetry.md | 871 ++++++++++ docs/sampling-cdm-gos-derivations.md | 1837 +++++++++++++++++++++ 2 files changed, 2708 insertions(+) create mode 100644 docs/design-gos-unified-edge-telemetry.md create mode 100644 docs/sampling-cdm-gos-derivations.md diff --git a/docs/design-gos-unified-edge-telemetry.md b/docs/design-gos-unified-edge-telemetry.md new file mode 100644 index 00000000..390eac25 --- /dev/null +++ b/docs/design-gos-unified-edge-telemetry.md @@ -0,0 +1,871 @@ +# GOS: A Unified Error / Threshold / Cost Framework for Distributed Edge Telemetry + +**One line.** One framework that unifies error-bounded sketching, coordinated +sampling, Geometric Monitoring (GM/AutoMon), and OctoSketch-style change +transmission into a single per-cell decision — reducing to **one atomic quantity** +`e_j = k·T_j` (per-cell backend error) — with a unified relative error bound, a +tunable memory/compute/communication objective, and a closed-form +water-filling solution whose worst-case communication matches the +Woodruff–Zhang lower bound `Θ̃(k/ε²)`. + +We call the construction **GOS** (Geometric-OctoSketch). + +--- + +## 1. Context and requirements + +Distributed data collection and transmission with a centralized analytics +backend. Edge telemetry must be **memory-, computation-, and +communication-efficient** under cloud economics, while the backend serves +**continuous, accurate, fresh** queries. + +- **Memory efficiency.** Cloud memory instances are priced above CPU instances. + With high-cardinality, high-frequency time-series metrics from many + distributed services, allocating *one sketch per series per window* puts heavy + memory pressure on the data-source nodes. Memory must be controlled across the + series×window space. +- **Computation efficiency.** Per-sample sketch update work at the edge must stay + within line-rate CPU budgets on shared instances. +- **Communication efficiency.** Cross-AZ egress is billed per byte; transmission + must be minimized. +- **Continuous & accurate queries.** Monitoring queries are *long-running* over + the stream, not the database "on-demand" model. We care about the + **freshness/timeliness** gap between when a sample is generated at the source + and when it can be queried accurately at the backend. + +These four axes are exactly the decision variables and constraints of the +optimization below. + +--- + +## 2. Positioning: what each prior line gives, and what it lacks + +All four lines are instances of "keep the local drift inside a **safe zone**, +communicate on violation." They differ in the *shape* of the safe zone and in +*what* they bound. (Attribution matters; see §10.) + +| Line | Bounds | Safe-zone shape | Continuous query? | Gap | +|---|---|---|---|---| +| **Error-bounded sketching** (Count-Sketch [Charikar+02], CMY [Cormode+08]) | one function, per window | — | window granularity only | intra-window staleness | +| **Geometric Monitoring** (Sharfman–Schuster–Keren [SIGMOD'06]; AutoMon [Sivan+ SIGMOD'22]) | one scalar `f(x̄)` | DC-quadric (ball / slab / general) | that **one function** only | not per-cell; not query-general | +| **OctoSketch** [Zhang+ NSDI'24] | **every counter cell** `|ΔC[j]| **Result.** At equal admitted work (`E[rows]=d·p` per item ⇒ same edge CPU), +> per-row sampling keeps the sampling error inside the median's high-probability +> envelope; whole-item sampling leaves it as an irreducible common-mode penalty. +> Per-row is the **strictly better estimator** — this, not the relocation, is the +> reason to push the decision into the SDK. + +**Effective `ε_sa` (composition).** Under per-row independence the sampling +*variance* folds into the **same** `median_r` step as the collision variance — +`Var[X_r] = F₂/w + (1−p)/p·S₂,r` under one median tail — so it composes **in +quadrature** with `ε_sk`: `|f̂(y)−f(y)| ≤ √(ε_sk²+ε_sa²)·‖f‖₂` w.p. `1−δ`, with +`ε_sa = Θ(√((1−p)/(p·w)))` (uniform `p`). This is exactly the random part of +Theorem 1 (§4). Whole-item admission instead leaves a **common-mode** term that +survives the median and adds to `ε_sk` **linearly** — strictly looser at equal +edge cost. + +**Threshold-allocation coupling (row-dependent floor).** The delta gate `T_j` for +cell `j=(r,c)` sits over a *row-`r`-subsampled* counter, so the +"don't-transmit-finer-than-you-sample" floor of §7 is **row-indexed** by that +row's admission rate: +``` +T_j ≥ T_j^floor = √( V_j · (1−p_i)/p_i ) (rate is per-site p_i, admission per-row) +``` +The GOS water-filling is unchanged in form; the ε-budget is split by §7 Layer B +(staleness `ε_st` peeled linearly, then `ε_sk²+ε_sa² = (ε_q−ε_st)²`), and this +floor closes the sampling↔threshold coupling. + +**Status.** The per-row *estimator* benefit above is **realized** in code via +`UpdateStringSampledPerRow`, so `ε_sa` sits inside the `1−δ` median guarantee. The +*location* move is realized on the **SDK-build path** — the OTLP SDK `CountSketch` +aggregator hosts the sampler (§3.1) — so admission happens at the source; the +collector-build path realizes the pre-deserialization saving at the edge +`otlpfilter` (whole-datapoint wire-thinning). CMS/DDSketch SDK-build hosting is +the remaining follow-up. + +--- + +## 4. Unified error bound + +The backend holds `Ĉ(t)`, perturbed from the ideal `C(t)` by two **random** +sources that the Count-Sketch median absorbs together, plus one **deterministic** +staleness term that adds on top: + +1. **Sketch + sampling (random, one median).** Per row `r`, the cell estimate + carries hash-collision variance `≈ F₂/w` *and* per-row sampling variance + `≈ (1−p_i)/p_i · S₂,r` (§3.2 — the `1/p_i` weight is unbiased; per-row + admission keeps the rows independent). Because both live in the **same** + `median_r` step, they compose in quadrature into one high-probability tail: + `|q̂ − q(f)| ≤ √(ε_sk² + ε_sa²)·‖f‖₂` w.p. `1−δ`, with `ε_sk=Θ(1/√w)`, + `ε_sa=Θ(√((1−p)/(p·w)))`, `δ=2^{−Θ(d)}`. (Sampling rate is per-site `p_i`; + admission is per-row-independent, i.e. `p_{i,r}=p_i` uniform across rows.) +2. **Staleness `{T_j}` (deterministic).** Site `i` withholds cell `j` until its + accumulated change reaches `T_j`, so `|Ĉ[j] − (C+η)[j]| ≤ k T_j =: e_j` **at + all times** — a worst-case bound, *not* a random variable, so it **adds + linearly** (it cannot be RMS-combined with the random tail). + +**Theorem 1 (unified relative error).** For any query `q` and any time `t`, w.p. +`≥ 1 − δ`: + +``` + ┌──── random (one median) ────┐ ┌──── staleness (linear) ────┐ +|q̂(t) − q(f)| ≤ √(ε_sk² + ε_sa²)·‖f‖₂ + k·Σ_j |r_{q,j}|·T_j +``` + +Dividing by `‖f‖₂` gives the **relative** budget: the random part (sketch ⊕ +sampling in quadrature) plus the deterministic staleness, linearly: + +> **√(ε_sk² + ε_sa²) + ε_st ≤ ε_q.** + +For a monitored non-linear `f_m` (`g_m = ∇f_m`, Hessian spectral bound `λ_m` from +AutoMon/ADCD), a Taylor + DC bound gives + +``` +|f̂_m − f_m| ≤ |g_m|ᵀ(ε_sk + ε_sa terms) + k·Σ_j|g_{m,j}|T_j + ½·λ_m·k²‖T‖₂² ≤ ε_m·|f_m|. +``` + +This holds **continuously**, not only at window boundaries — the OctoSketch +"online accuracy at any query time" property, here generalized to arbitrary +queries and to gradient-weighted function monitoring. + +### Freshness + +Cell `j` (activity `V_j = Σ_i V_{ij}`) reaches `T_j` after time `T_j / V_j`, so its +max staleness age is `Δ_j = T_j / V_j`. To guarantee query freshness `Δ*`: +`T_j ≤ V_j Δ*`. (Quiet cells have large `Δ_j` even at small `T_j`, so freshness +binds them → periodic heartbeat.) + +--- + +## 5. Cost models (per unit time) + +``` +Memory_edge = m·G·n·( 1 + 1{delta}[acked snapshot] + 1{aniso}[threshold vec] ) +Comp_sdk = c_rng·(Σ_r p_{i,r})·rate (geometric skip-sampling: O(1)/admit RNG) +Comp_coll = c_h·(Σ_r p_{i,r})·rate (hash + update — only admitted rows) + + c_s·Σ_j V_j/T_j (uploads — thresholds cut this) +Comm = b·Σ_j V_j/T_j (+ broadcast for geometric) + × (1 − ∏_r(1−p_{i,r})) (samples with no admitted row aren't sent) +Cost_coord = m·n·(1 + k·1{geo}) (running merge + per-edge refs) + + c_a·Σ_j V_j/T_j (incremental apply_delta) +``` + +The edge CPU is split across the two runtimes: the **SDK** pays only the +geometric-sampler RNG (`Comp_sdk`, `O(1)` per admit, no hashing), and the **agent +collector** pays the hashing/update (`Comp_coll`) *only for admitted rows* plus +the delta uploads. Lowering `p_{i,r}` cuts SDK RNG, collector hashing, and wire +volume together — one lever, three savings. + +**Structural insight.** `Comm`, the upload part of `Comp_edge`, and the apply part +of `Cost_coord` are all `∝ Σ_j V_j/T_j` → they collapse into one effective weight +`W = w_c·b + w_e·c_s + w_b·c_a`. Hence the *cost weights do not change the optimal +threshold shape* — the accuracy constraint pins `T_j`; the weights instead select +the **structural knobs** (uniform-vs-aniso, delta-vs-full, sampling `p`, per-edge +refs), which is where the memory/compute tradeoffs live. + +--- + +## 6. The optimization problem (P) + +``` +minimize w_m·Memory_edge + w_e·Comp_edge + w_c·Comm + w_b·Cost_coord +over d, w, G, {p_i}, {T_j}, structural flags +subject to + (query, ∀ q ∈ Q) √(ε_sk(w)² + ε_sa({p_i})²) + k·Σ_j|r_{q,j}|T_j/‖f‖ ≤ ε_q + (function, ∀ f_m) k·Σ_j|g_{m,j}|T_j + ½·λ_m·k²‖T‖² ≤ ε_m·|f_m| + (freshness) T_j ≤ V_j·Δ* + (confidence) d ≥ log₂(1/δ) + p_i ∈ (0,1], T_j ≥ 0, w ≥ 1, G ≥ 1 +``` + +`w_b` (coordinator) is the least-weighted axis by requirement. + +--- + +## 7. Solution: hierarchical decomposition + two water-fillings + +**Layer A — outer (small enumeration).** `d = ⌈log₂(1/δ)⌉`; choose `w` +(`ε_sk = c/√w`) and `G` (grouping) to trade the memory term `w_m·G·n` against the +accuracy the sketch must supply. + +**Layer B — budget split (staleness peeled linearly first).** Because staleness +is deterministic it comes off the top of `ε_q` **linearly** (Theorem 1), *then* +the remaining random budget is split in **quadrature**: + +1. choose `ε_st ∈ [0, ε_q − ε_sk]` — the edge-CPU↔communication knob (larger + `ε_st` ⇒ looser thresholds ⇒ less comm, but a smaller random budget ⇒ tighter + sampling ⇒ more CPU); pick it by the 1-D convex tradeoff of `w_e·Comp` vs + `w_c·Comm` (unique — both monotone); +2. the random budget after the linear peel is `ε_rand = ε_q − ε_st`; +3. split it in quadrature: `ε_sa² = ε_rand² − ε_sk²` (sketch ⊕ sampling), which + requires `ε_sk ≤ ε_rand`. + +So the composition is `√(ε_sk² + ε_sa²) + ε_st = ε_q`, matching §4 — **not** a +three-way quadrature. The code's `split_budget(ε_q, ε_sk, w_edge, w_comm)` +implements exactly this linear peel (`ε_st = t·(ε_q−ε_sk)`, +`ε_sa = √((ε_q−ε_st)²−ε_sk²)`, `t = w_comm/(w_edge+w_comm)`). + +**Layer C — two water-fillings (same KKT tool, two variables).** + +- **Sampling** (per-site). The per-key KKT water-filling + `p_i ∝ √(f_i/rate_i)` (binding `Σ_i f_i(1−p_i)/p_i ≤ V_sa(ε_sa)`) is the + general form, but it has been **retired for sketch sampling**: a sketch + point/L2 estimate's error is bounded by the sketch *norm*, not a single key's + `f(x)`, so the accuracy a per-edge `p_i` buys is only "keep this edge's L2 + contribution within ε." The **implemented** allocation is therefore the + whole-sketch ε-floor + ``` + p_i = 1 / (1 + ε²·rate_i), clamped to (0,1] + ``` + (`data_plane monitor::coordinator::allocate_p` → `epsilon_sample_floor`; see + derivations §5). The `√(f/rate)` split remains valid only when a key is + exact-counted *outside* the sketch — where sampling that one counter is + pointless anyway. +- **Thresholds** (per-cell; GOS). Effective weight `W` (§5), per-cell price + `c_j = k|g_j|` (function) or `k|r_{q,j}|` (query — take the binding constraint), + budget `B` (relative: `B = ε_m‖Ĉ‖²` for F₂): + + ``` + ┌ water-filling ┐ ┌──── clamps ────┐ + T_j = clamp( (B/Σ_ℓ√(c_ℓ V_ℓ))·√(V_j/c_j), T_j^floor, min(T_q, V_j·Δ*) ) + ``` + + with **sampling-coupling floor** `T_j^floor = √( V_j(1−p)/p )` ("don't transmit + finer than you sample"), **query cap** `T_q = ε_q‖Ĉ‖/(k·s_q)`, **freshness cap** + `V_j·Δ*`. Clamped cells release budget → box water-filling redistributes (a few + iterations). + +**Reading.** `T_j ∝ √(V_j/|g_j|)`: high-activity cells get larger thresholds (don't +chase high-frequency noise); cells the monitored function is sensitive to +(`|g_j|` large) get smaller thresholds (report early); everything capped by the +universal query cap and freshness. + +### Closed forms + +- **F₂, isotropic** (`g_j = 2Ĉ_j`, `λ=2`, uniform, relative): `T = ε‖Ĉ‖ / (2k√(dw))` + — adaptive: scales with the current norm. +- **F₂ threshold-alert version** (monitor `F₂ ≥ τ`, one-sided band): `T = (1/k)√((1−ε)τ/w)`. + The whole-sketch `F₂` readout here is the **mean-of-rows** estimator + `F̂₂ = ‖C‖²/d` (each row's `‖C_r‖²` is an unbiased `F₂` estimate; averaging the + `d` independent rows reduces its variance by `1/d`) — *not* the median-of-rows + point-query estimator of §3.2. This is deliberate: the geometric safe-zone is a + **ball** `‖C‖ ≤ √(d(1−ε)τ) ⇔ ‖C‖²/d ≤ (1−ε)τ`, so edge silence and coordinator + alert test the identical functional (`all-sites-safe ⟺ F̂₂ < (1−ε)τ`). The two + estimators serve different readouts — median for individual-key location + (robust tail), mean for the aggregate energy the ball bounds (variance + reduction) — and must not be conflated. +- **Linear `f` (sum/count/point)**: `λ = 0` → the curvature term vanishes → box + degenerates to a **slab** = the classic CMY slack countdown. + +### Unification of counter-based and F₂ + +Both are `f(x̄)` vs a band `[L,U]`, with the safe zone from the DC bound; the only +difference is the Hessian eigenvalue: `λ=0` → slab (counter-based), `λ=const·I` → +ball (F₂), general → ADCD quadric. One monitor, one edge check +`isLocallySafe(Δ, x₀, ∇f, λ, L, U)`; keep the scalar-countdown fast-path for the +linear case (no reference-vector broadcast needed). + +### Threshold band vs tracking band + +- **Threshold/alert**: one-sided band `[−∞, τ]` → fire on crossing. +- **Continuous ε-query**: moving band `[f(x₀)−ε, f(x₀)+ε]` → backend answers + `f(x̄)=f(x₀)±ε` at all times. Same monitor, different band — this is how + threshold monitoring and continuous approximate querying unify (Cormode– + Garofalakis continuous querying = GM with a tracking band). + +--- + +## 8. Optimality vs the Woodruff–Zhang lower bound + +**Rate vs total.** `Σ_j V_j/T_j` is an upload *rate*; the WZ `Θ̃(k/ε²)` is the +*total* communication to maintain one continuous `(1±ε)` `F₂` estimate. Compare +them over a fixed horizon of bounded total change: with `w ∝ 1/ε²` (the necessary +sketch width) each sketch is `Θ(1/ε²)` and `k` sites must each be represented, so +the total is `Θ̃(k/ε²)` — **matching the WZ STOC'12 tight lower bound** (bits vs +words absorbed in the `Θ̃`). The measured normalization in +[`gos-eval-results.md`](gos-eval-results.md) §3 uses the "one-round" unit `k·S` +for exactly this comparison. Consequences: + +- The `1/ε²` and the linear-in-`k` are **fundamental**; no protocol (GM, AutoMon, + OctoSketch, GOS) beats `k/ε²` adversarially. GOS's savings are **data-dependent** + (small `V_j/‖Ĉ‖` on stable streams). +- Report GOS's measured bytes as a **fraction of `k/ε²`** — a stronger baseline + than comparing to naive centralization. +- The relative-error caveat: relative bounds require the norm bounded below + (WZ tightness / OctoSketch `L1 > ε⁻¹k'τ`); relative error on a near-zero signal + is fundamentally not cheap. + +--- + +## 9. How it meets the four requirements + +| Requirement | Mechanism | +|---|---| +| **Memory efficiency** | `(w,G)` + `w_m`: grouping `G` avoids one sketch per series; `w=Θ(1/ε²)` is the WZ-minimum; delta/aniso flags dropped under high `w_m` (no snapshot / threshold vector) | +| **Computation efficiency** | `w_e`: sampling `p_i` (fewer updates) + threshold size (fewer uploads); both fold into one effective weight | +| **Communication efficiency** | `w_c`: per-cell water-filling thresholds + geometric silence; `Θ̃(k/ε²)` worst case, far less on stable data | +| **Continuous, accurate, fresh** | Theorem 1 bounds every query at **any** `t`, **relative**; freshness cap `T_j ≤ V_jΔ*` bounds staleness age; whole sketch queryable (OctoSketch), monitored `f_m` tighter (GM/AutoMon) | + +--- + +## 10. What is adopted vs contributed (attribution) + +- **Water-filling** — classic (information theory / convex optimization; optimal + power allocation across parallel channels). Already used in ASAP for sampling + (`p_i ∝ √(f_i/rate_i)`). *Adopted, not contributed.* +- **Per-cell change transmission with a threshold** — OctoSketch [Zhang+ NSDI'24]. + *Adopted.* +- **Function safe zone via DC decomposition of the Hessian (ADCD), gradient/ + Hessian bounds** — AutoMon [Sivan+ SIGMOD'22]; Geometric Monitoring [Sharfman+ + SIGMOD'06]. *Adopted.* +- **Coordinated sampling + geometric skip-sampling** — NitroSketch [Liu+ + SIGCOMM'19]; ASAP's own `AllocateSampleRates` (rate allocation) and + `sketchlib-go/common.GeometricSampler` (the `O(1)`-amortized skip sampler). + *Adopted.* +- **Lower bound `Θ̃(k/ε²)`** — Woodruff–Zhang [STOC'12]. *Yardstick.* +- **GOS (this doc)** — casts *per-cell threshold allocation* as water-filling with + **gradient-derived weights** (GM/AutoMon) and a **per-cell query cap** + (OctoSketch), coupled to **per-site sampling** through a shared ε-budget and a + granularity floor `T_j ≳ √(V_j(1−p)/p)`, inside one **tunable + memory/compute/communication objective**. *The synthesis is the contribution; + the optimization tools are off-the-shelf.* + +--- + +## 11. The 2026-07 redesign: no sub-window, insert-time detection for all 6 families + +Everything above (§1-10) establishes *what threshold to use*. This section is +about *when transmission actually happens* — a mechanism-level redesign that +applies uniformly across Sum, CMS, CountSketch, DDSketch, KLL, and HLL, +replacing the periodic sub-window tick with per-insert detection while +keeping the existing OTLP/`SketchEnvelope` wire and pipeline unchanged. + +### The unified model + +One mechanism for every family, isotropic case: **check at insert time +whether the accumulated-since-last-sync delta crosses the family's threshold +(§8 of `sampling-cdm-gos-derivations.md`); if it does, that cell's (or +scalar's) delta needs to reach the backend.** The sole purpose is data +synchronization — keeping the backend's reconstructed state accurate. Alerting +and any other query-time decision is made entirely at the backend against +that synced state; the edge no longer makes alerting decisions itself. This +retires the previously-separate Discipline B "continuous monitoring" path +(`monitor.Engine.Observe` → immediate `sendReportLocked` over +`monitor/grpcclient`) as a distinct alerting mechanism — Sum becomes just +another family running the same insert-time check, synced over the normal +`SketchEnvelope` pipeline like everything else. `Engine.Observe` today +conflates two things in one function: `obsCount++` (rate tracking, feeding +the coordinator's `SampleP` grant negotiation) and the +`value-baseline>=slack` alerting check (which calls `sendReportLocked`). Only +the *alerting* half retires; `obsCount`/rate-tracking must be preserved (it +feeds a genuinely separate concern — sampling-rate negotiation, not data +sync) — these two halves need to be split apart, not deleted together. + +### Wake-on-demand flush (keeps the existing OTLP pipeline) + +The naive read of "insert-time, no sub-window" is "bypass the pipeline +entirely, send out-of-band" — this was considered and rejected. The OTLP +export chain (SDK `PeriodicReader` → collector processor → exporter) stays +exactly as-is; only *when a flush cycle runs* changes, from purely +timer-driven to timer-**or**-woken: + +```go +for { + select { + case <-ticker.C: // slow fallback cadence, in case a wake is ever missed + flush() + case <-wakeCh: // fired the instant something crosses threshold + flush() + } +} +``` + +Insert path, non-blocking (never waits for the flush loop): + +```go +select { +case wakeCh <- struct{}{}: +default: // a wake is already pending; nothing to add +} +``` + +`flush()` keeps using the existing `SnapshotCache`/`ComputeDeltaAgainst` +machinery (full-frame fallback on cold start, the "never emit a delta larger +than a full frame" clamp) unchanged — the only thing that changed is the +trigger. This makes **Gate 1 (the old `subWindowShouldEmit` / +`L2DivergenceSinceEmit` divergence pre-check / `ackedCells`) redundant**: an +empty `dirtySet` at flush time IS "nothing was worth sending," computed once +per crossing instead of by a periodic O(dw) full-matrix scan. All of that +(`ackedCells`, `subWindowDivergence`, `subWindowShouldEmit`) is dead code +under this design and should be deleted, not kept as a fallback path. + +CMS's local point-query read (`ThresholdConfig.Functional: cms_point`) +retires alongside Discipline B's alerting: once cells reset in place at +insert time, a `min`-based local read is corrupted by any single recently- +reset row. All point/alert reads move to the backend's reconstructed copy. + +### Per-family cell / threshold / reset semantics + +See `sampling-cdm-gos-derivations.md` §8 for the derivations; summary of the +*mechanism* (not the math) each family needs: + +| Family | Detection unit | Reset on send? | Notes | +|---|---|---|---| +| CountSketch (isotropic) | matrix cell | zero it | `normSqAll` tracked incrementally (`+= 2·old·Δ+Δ²`), O(1) | +| CountMinSketch | matrix cell | zero it | same mechanism, $L_1$-scale threshold (§8.2) | +| DDSketch | bucket count | zero it | bucket count `B` tracked incrementally too (+1 on genuinely new bucket), no config constant needed (§8.4) | +| Sum | scalar | zero it (subtract reported amount) | degenerate 1-cell case | +| KLL | whole sketch (no per-cell structure) | full `Reset()` | trigger is `Count() >= εN`, not a per-cell check; already the existing disjoint-segment mechanism, just re-triggered by count instead of a timer | +| HLL | register | **never** — MAX-merge is idempotent, only clear a dirty flag | trigger is $\lvert 2^{C'}-2^{C}\rvert \ge 2^{\tau}$ on the linearized value, not raw register value (§8.7) | + +Backend reconstruction is unchanged for the additive families (Sum/CMS/CS/ +DDSketch): summing every fragment ever received for a cell — regardless of +how many times or when it was individually reset — telescopes to the true +cumulative value (`v_1+v_2+...+v_n+v_{residual}` = true total). This is why +resets can happen asynchronously, at different times per cell, without +breaking correctness — it only affects when transmission happens, never +what the backend eventually reconstructs. + +### Cold start is a feature, not a bug + +Every family's threshold scales with an accumulated quantity (`‖Ĉ‖`, `N`, +`R`) that starts near zero at window start, so the very first few inserts +cross threshold almost immediately. This is intentional: it gets the backend +a usable initial estimate as fast as possible, rather than waiting for data +to accumulate before syncing anything. No floor/minimum-threshold mechanism +is needed to suppress this. + +### Open items (not blocking) + +- **Anisotropic CountSketch's `Activity_j`** needs redefinition. The old + `Activity_j=|current-prev|` assumed a single, uniformly-timed `prev` + snapshot; under per-cell async reset, different cells' "since last touch" + windows are no longer comparable, and naively diffing against any + snapshot double-counts/under-counts around individual cell resets. A + per-cell EMA of `|Δ|` (`activityRate[r][c] = decay·activityRate[r][c] + + (1-decay)·|Δ|`, updated every insert) is the leading candidate — cheap, + reset-timing-independent — but it replaces the derivation's exact + `Activity_j=V_j` with a heuristic, and whether the §7 closed-form + water-filling solution still carries the same error guarantee under that + substitution has not been checked. The anisotropic water-filling solve + itself also still requires a periodic $O(dw)$ pass (unlike every other + family here) — per-cell detection at insert time only avoids the + "decode a serialized `prev`" cost, not the joint solve. +- **DDSketch's unbounded contiguous bucket-array growth** on outlier values + is a real memory-safety gap, independent of this redesign — tracked as + sketchlib-go#72. The dynamically-tracked `B` used in the threshold formula + above does not require fixing this; it is a separate, likely + higher-priority issue. +- **HLL's small-cardinality regime**: the register-change adapter's accuracy + proof (OctoSketch's Appendix B) is stated for "sufficiently large" + cardinality; behavior when most registers are still at 0 (early in a + window) has not been separately verified. A candidate mitigation (always + send a register's first-ever nonzero write unconditionally) is proposed + but unverified. + +--- + +## 12. Implementation notes (controller synthesizes, edge executes) + +Everything expensive is a **controller (backend) decision**; the edge only +executes a fixed per-cell comparison. + +- **Controller** (offline, per registered metric/query): runs ADCD (AD → Hessian + eigenvalue bounds → `∇f, λ`), estimates `{V_j}` from the workload, solves (P)'s + layers A–C, emits `(d, w, G, {p_i}, scalar GOS knobs, flags)` via OpAMP. This + slots into the existing controller multi-objective + (`controller-optimization-problem.md` SP-6: + `min w_bw·bw + w_cpu·cpu + w_mem·mem + …`) — GOS thresholds are new decision + variables there. **Note:** the controller ships *scalars* + (`ε_delta`, sites, aniso flag), **not** the full per-cell vector `{T_j}`; the + edge reconstructs `{T_j}` locally from those scalars plus its live sketch state + (see §7C, `sketches/gos_threshold.go`), so the `O(d·w)` vector never crosses the + wire. +- **Edge**: maintain sketch + acked snapshot; per flush, recompute `{T_j}` from + the pushed scalars + local `{V_j}`, upload cells with `|ΔC_j| ≥ T_j` as a sparse + delta; run one generic `isLocallySafe` for monitored functions. No AD, no + water-filling solve at the edge — only the closed-form threshold evaluation. + **Superseded for the isotropic case by §11**: there is no longer a + periodic "per flush" cadence or a separate acked-snapshot copy — the + per-cell comparison happens at insert time and the cell resets in place + on send. This paragraph's *acked-snapshot, periodic-flush* model still + describes anisotropic CountSketch's water-filling solve, which §11 flags + as an open item, not yet redesigned. +- **Backend**: `apply_delta` into a running merge (`O(#delta cells)`), keeping the + global sketch continuously queryable within the Theorem-1 envelope, surfaced in + the `accuracy: ε=…` response annotation. + +**Ties to existing code:** +- `ASAPQuery-backend/control_plane/src/epsilon_alloc.rs` — the ε-budget split. + Staleness is peeled **linearly** first (Theorem 1), then the remaining random + budget splits in quadrature: `√(ε_sk² + ε_sa²) + ε_st = ε_q` (see §7 Layer B). + This is **not** a three-way quadrature `ε² = ε_sk² + ε_sa² + ε_st²` — staleness + is deterministic and comes off the top linearly. +- `ASAPQuery-backend/data_plane/src/monitor/sampling_alloc.rs` + (`epsilon_sample_floor`) — the live whole-sketch sampling floor. (The Go + `monitor/sampling_alloc.go` `AllocateSampleRates` is the *retired* per-key + water-filling, kept only as a reference impl with no production caller.) +- `threshold_alloc` (`AllocateThresholds`, Go `sketches/gos_threshold.go` + Rust + `control_plane/src/threshold_alloc.rs`) — the per-cell threshold water-filling + (this doc's §7C). +- `data_plane/src/monitor/f2_coord.rs`, `asap-precompute-go/monitor/f2engine.go` + — generalize the F₂-specific ball to the `(∇f, λ)` DC safe zone; use the relative + radius `ε‖Ĉ‖/(2k√(dw))`. +- reuse `asap_sketchlib` `CountSketchDelta` + `compute_delta`/`apply_delta` + (byte-parity Go/Rust) for the sparse per-cell delta wire format. + +**Implementation status (as of this writing).** The pieces exist but the GOS +threshold control loop is **not yet wired end-to-end**: +- **Live today:** the scalar CDM loop (register → grant `(slack, sample_p)` → + countdown → report → alert) and the sampling grant path (`Grant.SampleP` → + `otlpfilter` Upsert + wrapper `WithSampleP`). +- **Implemented but unreachable from a production config:** the control plane + *derives and emits* the scalar GOS knobs (`gos_delta_epsilon`, `gos_sites`, + `gos_anisotropic` in `emit/agent.rs`), and the edge *consumes* + `PrecomputeConfig.GosDeltaEpsilon` (`applyGosMode` → `gosThresholdMatrix`), but + **no collector processor parses those YAML keys into the config**, so + `applyGosMode` is a production no-op and the per-cell delta-gating path is + exercised only by tests/eval. The sampling↔threshold coupling floor + `T_j ≥ √(V_j(1−p)/p)` (§3.2) is likewise implemented but inert — its only + production-shaped caller hardcodes `SampleP=1`. Closing this last hop + (a knob parser + threading the granted `p` into `GosParams`) is tracked work. + +--- + +## 13. Open problems / next steps + +1. **Anisotropic delta broadcast** — *partially done.* The **sparse-cell** + encoding of `ΔC_ref` on the coordinator→edge path is implemented and measured + (`CRefUpdate::Delta`; removes the `O(k)` broadcast amplification — see + gos-eval-results.md §2). Still **open:** the broadcast gate is currently + isotropic (ships every changed cell, `Δ ≠ 0`); giving it *anisotropic per-cell + thresholds* (the §7C water-filling, as already done on the edge→coordinator + upload path via `ComputeDeltaPerCell`) is the remaining work. +2. **Relative-error under small norm** — heartbeat / additive floor when `‖Ĉ‖` is + small (WZ / OctoSketch fundamental limit). +3. **Verified eigenvalue bounds** — AutoMon's numerical `λ` may miss the true + extreme → reserve an `ε_eig` slice of the budget or use interval bounds. +4. **Empirical validation** — measure achieved communication as a fraction of the + WZ `k/ε²`, sweep `(w_m, w_e, w_c)` to trace the Pareto surface. + +--- + +### References (attribution) + +- G. Cormode, S. Muthukrishnan, K. Yi. *Algorithms for Distributed Functional Monitoring.* SODA 2008 / ACM TALG 2011. +- M. Charikar, K. Chen, M. Farach-Colton. *Finding Frequent Items in Data Streams* (Count-Sketch). ICALP 2002. +- I. Sharfman, A. Schuster, D. Keren. *A Geometric Approach to Monitoring Threshold Functions over Distributed Data Streams.* SIGMOD 2006. +- H. Sivan, M. Gabel, A. Schuster. *AutoMon: Automatic Distributed Monitoring for Arbitrary Multivariate Functions.* SIGMOD 2022. +- Y. Zhang, P. Chen, Z. Liu. *OctoSketch: Enabling Real-Time, Continuous Network Monitoring over Multiple Cores.* NSDI 2024. +- D. Woodruff, Q. Zhang. *Tight Bounds for Distributed Functional Monitoring.* STOC 2012 (arXiv:1112.5153). +- Z. Liu, R. Ben-Basat, G. Einziger, Y. Kassner, V. Braverman, R. Friedman, + V. Sekar. *NitroSketch: Robust and General Sketch-Based Monitoring in Software + Switches.* SIGCOMM 2019. (Geometric skip-sampling: one RNG draw per admitted + update — `O(1)` amortized, "always line rate" — with inverse-probability + weighting; the SDK-side row-admission sampler here is this scheme.) diff --git a/docs/sampling-cdm-gos-derivations.md b/docs/sampling-cdm-gos-derivations.md new file mode 100644 index 00000000..400ca7f8 --- /dev/null +++ b/docs/sampling-cdm-gos-derivations.md @@ -0,0 +1,1837 @@ +# Sampling + Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) derivations for windowed sketch telemetry + +> Paper-facing derivation note. This document ties together +> [distributed-nitrosketch-coordinated-sampling.md](distributed-nitrosketch-coordinated-sampling.md), +> [continuous-monitoring-aggregation-taxonomy.md](continuous-monitoring-aggregation-taxonomy.md), +> [continuous-monitoring-tumbling-cost-analysis.md](continuous-monitoring-tumbling-cost-analysis.md), +> and [design-gos-unified-edge-telemetry.md](design-gos-unified-edge-telemetry.md). +> +> Scope: fixed tumbling-window epochs; open-window freshness inside one epoch; +> additive linear sketch states as the fully proved case; family-specific +> extensions for quantiles and cardinality sketches. +> +> Acronyms: **CDM** = **Continuous Distributed Monitoring**; **GOS** = +> **Geometric-OctoSketch**. + +## Paper positioning: bottleneck -> solution map + +The system bottleneck addressed by these derivations is +**resolution-coupled central ingestion**. In conventional observability +pipelines, higher temporal resolution, higher label cardinality, and lower +freshness latency all require more raw samples to traverse the central path: + +```text +collector/exporter -> remote write / queue / WAL -> backend ingest/index/storage +-> query scan +``` + +ASAPCollector changes the unit of work from raw-sample ingestion to +query-bounded sketch-state synchronization. Edge collectors still absorb the +high-resolution stream, but the backend receives bounded sketch summaries, +error-triggered deltas, and optional cold raw fallback instead of every raw +sample on the warm path. + +The problem-solution pairs are: + +| Existing-system bottleneck | Why it matters | ASAPCollector mechanism | Error/control consequence | +| --- | --- | --- | --- | +| Central ingest bottleneck | Cost scales with $\mathrm{series\_cardinality} \times \mathrm{sample\_frequency}$; raising resolution pushes more samples through write queues, indexing, storage, and query scan. | Maintain mergeable sketches at edge collectors and transmit sketch state/deltas. | Backend warm-path load scales with summary size and threshold crossings, not directly with raw sample rate. | +| Freshness vs cost bottleneck | Shorter scrape/export intervals improve open-window freshness but increase CPU, network, and backend ingest pressure; longer intervals reduce cost but make queries and alerts stale. | Use CDM/GOS residual thresholds $T_j$ for error-triggered synchronization. | Freshness becomes a bounded staleness term $\mathrm{Err}_q^{cdm}$ rather than an implicit consequence of a fixed reporting interval. | +| Query scan / post-ingest downsampling bottleneck | TSDB compression and downsampling help after raw data has already been ingested, and long-range queries still depend on stored sample layout. | Build query-ready sketches before/during ingestion. | Query cost is paid against sketch summaries, while the cold raw path remains available for unsupported or forensic queries. | +| Fixed-statistics / early-binding bottleneck | Histograms, summaries, and pre-aggregations commit early to bucket layouts, quantiles, windows, or rollups; changing the query later may be impossible or inaccurate. | Use a sketch-family adapter per query class: Sum/CMS/CountSketch for additive readouts, DDSketch/KLL for rank queries, HLL for cardinality. | The controller applies the correct sampling and staleness model for each sketch family instead of using one proof for all summaries. | +| Control-plane bottleneck | Existing cost controls such as dropping labels, filtering metrics, increasing intervals, or coarse downsampling often do not expose a query-level error budget. | Jointly tune sketch size, sampling probabilities $p_i$, and GOS thresholds $T_j$. | The system can allocate a query error budget across $\mathrm{Err}_q^{sk} + \mathrm{Err}_q^{sa} + \mathrm{Err}_q^{cdm}$. | +| Theory-to-system bottleneck | Prior sketching, approximate query processing, and CDM work each solve part of the problem, but not the end-to-end telemetry ingestion control loop. | Combine update sampling, mergeable sketches, and CDM/GOS synchronization inside the collector/backend architecture. | The paper claim is a system-level accuracy envelope, not only a faster sketch update or a lower communication protocol in isolation. | + +Compared with existing work, the paper's positioning is: + +| Existing work / system family | Solves | Leaves open | ASAPCollector angle | +| --- | --- | --- | --- | +| Prometheus / OpenTelemetry / remote-write pipelines | Standard metric collection, export, and backend ingestion. | Higher resolution and cardinality still increase central ingest work. | Move high-resolution absorption to edge sketches and synchronize bounded state. | +| Native histograms / exponential histograms / DDSketch-style distribution metrics | Compact, mergeable distribution summaries. | Mainly distribution-specific; does not give a general control plane for sampling plus staleness across sketch families. | Treat DDSketch as one adapter in a broader sketch taxonomy. | +| TSDB compression / Thanos-style downsampling | Long-range query acceleration and storage-layout optimization. | Mostly after-ingest optimization; raw samples still enter the central path first. | Summarize before or during ingestion, then query the warm sketch path. | +| BlinkDB / VerdictDB-style AQP | Approximate analytics over stored data with statistical error. | Data is already in the warehouse; not a continuous telemetry freshness and ingestion-control problem. | Provide telemetry-native edge ingestion, open-window freshness, and sketch-state synchronization. | +| NitroSketch | Sampling sketch updates to reduce sketch CPU. | Focuses on update work for sketches, not multi-sketch observability queries, backend freshness, or GOS/CDM synchronization. | Reuse the sampling idea but compose it with query-level variance budgets and delta suppression. | +| Continuous Distributed Monitoring / distributed functional monitoring | Communication-efficient tracking of distributed functions. | Mostly a theoretical monitoring model, not an observability pipeline with sketch-family adapters and cold fallback. | Use CDM as the proof model for bounded sketch residual synchronization. | +| Learned reconstruction systems such as Zoom2Net | Infer fine-grained telemetry from coarse measurements. | Error semantics depend on reconstruction/model behavior rather than preserving query-sufficient sketch state. | Maintain sketch statistics with explicit sketch, sampling, and staleness error terms. | + +The intended claim boundary is narrow: + +- ASAPCollector does not replace raw telemetry for every task; it provides a + warm approximate path plus cold raw fallback. +- ASAPCollector does not support arbitrary PromQL under one theorem; it supports + sketchable/decomposable query classes with family-specific adapters. +- The fully proved theorem target is fixed-query, fixed-time, additive-state + telemetry with independent Horvitz-Thompson update sampling and an + update-synchronous or explicitly overshoot-bounded residual protocol. +- The core claim is that the collector can reduce update work and network + traffic while keeping a query-level accuracy envelope. + +## 1. Scope and aggregation modes + +Continuous monitoring in ASAP is not one protocol. The aggregation axis decides +which proof model applies. + +| Mode | Query shape | Protocol | Coordinator? | Main guarantee | +| --- | --- | --- | --- | --- | +| A | per-series $\times$ window | local $\epsilon$-gated delta emission | no | open-window freshness for one local series/group | +| B | series $\times$ timestamp | mergeable sketch fan-in | no | sealed/instant merge; no monitoring suppression | +| C | series $\times$ window, fused spatial+temporal | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) over edge collectors | yes | global open-window tracking/alerting across $k$ sites | + +This note proves the composition used by modes A and C for the additive-state +case: + +1. sketch approximation, +2. update sampling, +3. stale backend state due to suppressed deltas. + +For sealed tumbling windows, only the first two sources matter. The merge of +mergeable summaries does not compound the base sketch error. + +## 2. Notation + +One fixed tumbling window is the universe of discourse. + +| Symbol | Meaning | +| --- | --- | +| $k$ | number of stable edge collectors/sites | +| $i$ | site index, $i \in \{1,\ldots,k\}$ | +| $u$ | update/sample index | +| $x_u$ | raw item/value | +| $f_i$ | local frequency vector at site $i$ | +| $f=\sum_i f_i$ | global frequency vector | +| $S_i$ | exact local sketch state for site $i$ | +| $S=\sum_i S_i$ | exact global additive sketch state | +| $\widehat S_i$ | sampled local sketch state | +| $\widehat S=\sum_i \widehat S_i$ | sampled global sketch state | +| $\widetilde S$ | backend's stale copy of $\widehat S$ | +| $j$ | sketch cell/bucket index | +| $a_j$ | linear readout coefficient for query $q(S)=\langle a,S\rangle$ | +| $p_i$ | sampling probability at site $i$ | +| $p_{i,r}$ | SDK/source-side sampling probability for row/counter-array $r$ at site/source $i$ | +| $C_{i,r}$ | candidate counter-update rate or source-to-agent cost weight for site/source $i$, row $r$ | +| $F_{i,r}$ | row-level protected mass or query sensitivity for site/source $i$, row $r$ | +| $T_j$ | cell/bucket delta threshold | +| $V_j$ | cell/bucket activity rate or expected change mass | +| $B$ | staleness budget for a query/function | +| $\epsilon_{sk}$ | base sketch error budget | +| $\epsilon_{sa}$ | sampling error budget | +| $\epsilon_{cdm}$ | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) staleness budget | +| $\delta$ | failure probability | + +For additive sketches, an update $u$ touches one or more sketch cells. Let +$j(u)$ be a touched cell in a one-cell-per-update sketch such as a DDSketch +bucket, or let $j=(r,c)$ be a row/cell pair for CMS/CountSketch. + +## 3. Generic error decomposition + +For a fixed query $q$ at a fixed time inside a window: + +```text +true answer q(f) +exact sketch q(S) +sampled sketch q(\widehat S) +backend answer q(\widetilde S) +``` + +The triangle inequality gives + +```math +|q(\widetilde S)-q(f)| +\le +|q(S)-q(f)| ++ |q(\widehat S)-q(S)| ++ |q(\widetilde S)-q(\widehat S)|. +``` + +We name the three terms: + +```math +\mathrm{Err}_{q} +\le +\mathrm{Err}^{sk}_{q} ++ \mathrm{Err}^{sa}_{q} ++ \mathrm{Err}^{cdm}_{q}. +``` + +For a sealed tumbling window, $\widetilde S=\widehat S$ after boundary emission, +so + +```math +\mathrm{Err}_{q,sealed} +\le +\mathrm{Err}^{sk}_{q} ++ \mathrm{Err}^{sa}_{q}. +``` + +For an open window, Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch +(GOS) controls the third term. + +This statement is pointwise: one fixed query and one fixed time. To claim a +finite workload $Q$ and $M$ possible query times, replace $\delta$ by +$\delta/(|Q|M)$ in each concentration bound and union bound. + +## 4. Additive-state sampling model + +### 4.1 Horvitz-Thompson update sampling + +In the SDK-side Nitro-style deployment, the sampled unit is not a raw +observation/packet. A raw measurement is first mapped to the sketch updates it +would perform, for example one candidate counter update per CountSketch/CMS row. +The SDK samples those candidate counter-array updates and sends only admitted, +inverse-probability weighted counter updates to the edge collector. The edge +collector then merges weighted sketch updates; it does not need the dropped raw +measurement. + +Thus, in this section, $u$ should be read as the sampled update unit. For +edge-admission baselines, $u$ may be a raw update. For the preferred +Nitro-style design, $u=(\text{raw item},\text{row/counter-array})$ is one +candidate sketch counter update. + +For sampled update unit $u$, site/source $i(u)$ admits it with probability +$p_{i(u)}$: + +```math +Z_u \sim \mathrm{Bernoulli}(p_{i(u)}). +``` + +Here $Z_u$ is the admission indicator: + +```math +Z_u = +\begin{cases} +1, & \text{if update }u\text{ is admitted and applied to the sketch},\\ +0, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +The inverse-probability weighted contribution of update $u$ is + +```math +Y_u = Z_u/p_{i(u)}. +``` + +Equivalently, + +```math +Y_u = +\begin{cases} +1/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +0, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +Thus $Y_u$ is the effective update weight written into an additive sketch +counter/cell. Without sampling, the update would contribute weight $1$; with +sampling, it contributes weight $1/p_{i(u)}$ only on admitted updates. + +Then + +```math +\mathbb{E}[Y_u]=1, +\qquad +\mathrm{Var}(Y_u)=\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +For a linear readout $q(S)=\langle a,S\rangle$, define $g_u$ as the contribution of update +$u$ to this readout if it were not sampled. For a one-bucket counter, +$g_u=a_{j(u)}$. For signed sketches, $g_u$ also includes the row sign. + +The sampled readout error is + +```math +X_q +=q(\widehat S)-q(S) +=\sum_u (Y_u-1)g_u. +``` + +It is unbiased: + +```math +\mathbb{E}[X_q]=0. +``` + +The variance follows from the variance of one update contribution. First, + +```math +\mathbb{E}[Y_u^2] += +p_{i(u)}\cdot \frac{1}{p_{i(u)}^2} += +\frac{1}{p_{i(u)}}. +``` + +Since $\mathbb{E}[Y_u]=1$, + +```math +\mathrm{Var}(Y_u) += +\mathbb{E}[Y_u^2]-\mathbb{E}[Y_u]^2 += +\frac{1}{p_{i(u)}}-1 += +\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +Subtracting a constant does not change variance, so + +```math +\mathrm{Var}(Y_u-1)=\mathrm{Var}(Y_u). +``` + +Multiplying by the deterministic readout contribution $g_u$ scales variance by +$g_u^2$: + +```math +\mathrm{Var}((Y_u-1)g_u) += +g_u^2\mathrm{Var}(Y_u) += +g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +If the sampling randomness is independent across updates, i.e. the admission +indicators $Z_u$ are independent, + +```math +\mathrm{Var}(X_q) += +\sum_u g_u^2 \frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +This assumption is only about the sampling coin flips, not about the input data +streams. The metric values or workloads at different sites may be correlated. +The variance identity needs the random admission decisions to be independent, or +at least uncorrelated. The high-probability Bernstein bound in Section 4.2 uses +the stronger independent-sampling assumption. + +This equation is the sampling-error budget for query $q$. The random variable +$X_q=q(\widehat S)-q(S)$ is the difference between answering $q$ from the sampled +sketch and answering $q$ from the same sketch without update sampling. Each term +in the sum says how much update $u$ contributes to that sampling error: + +- $g_u^2$ is the squared sensitivity of query $q$ to update $u$; +- $(1-p_{i(u)})/p_{i(u)}$ is the noise introduced by sampling at the site that + produced update $u$. + +Thus lowering $p_i$ saves update work at site $i$, but increases the variance of +queries whose relevant updates come from that site. This is the quantity the +controller constrains when it chooses sampling probabilities. + +The independence assumption is what removes covariance terms: + +```math +\mathrm{Var}\left(\sum_u A_u\right) += +\sum_u \mathrm{Var}(A_u) ++2\sum_{u0}\quad & \sum_j \frac{V_j}{T_j} \\ +\text{s.t.}\quad & \sum_j c_jT_j \le B , +\end{aligned} +``` + +where + +```math +c_j = k|a_j| +``` + +for a linear query. For a differentiable monitored functional, use + +```math +c_j = k|g_j|, +\qquad +g_j=\partial F/\partial S[j], +``` + +plus a separate curvature budget if the functional is nonlinear. + +The Lagrangian is + +```math +\mathcal{L} += +\sum_j \frac{V_j}{T_j} ++\lambda\left(\sum_j c_jT_j-B\right). +``` + +The first-order condition is + +```math +\frac{\partial \mathcal{L}}{\partial T_j} += +-\frac{V_j}{T_j^2}+\lambda c_j=0. +``` + +Thus + +```math +T_j=\sqrt{\frac{V_j}{\lambda c_j}}. +``` + +Solving for $\lambda$ from the active budget constraint: + +```math +\sum_j c_j\sqrt{\frac{V_j}{\lambda c_j}} +=B, +``` + +so + +```math +\frac{1}{\sqrt{\lambda}} += +\frac{B}{\sum_\ell \sqrt{c_\ell V_\ell}}. +``` + +The closed form is + +```math +\boxed{ +T_j += +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell \sqrt{c_\ell V_\ell}} +}. +``` + +With floors and caps: + +```math +T_j += +\mathrm{clamp}\left( +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell \sqrt{c_\ell V_\ell}}, +\; +T_j^{floor}, +\; +\min(T_j^{query}, V_j\Delta^*) +\right). +``` + +Clamped cells consume or release budget; recompute the water-filling expression +over the remaining free cells until no cell changes clamp status. + +## 8. Sketch-family instantiations + +### 8.1 Sum / Count + +State: + +```math +S_i=\sum_{u\in i} x_u +``` + +or $S_i=N_i$ for count. + +Base sketch error: + +```math +\mathrm{Err}^{sk}=0 +``` + +up to floating-point arithmetic. + +Sampling error for unit counts under uniform $p$: + +```math +|\widehat N-N| +\le +O\left( +\sqrt{\frac{N\log(1/\delta)}{p}} ++\frac{\log(1/\delta)}{p} +\right). +``` + +CDM staleness with scalar threshold $T$: + +```math +\mathrm{Err}^{cdm}\le kT. +``` + +For a threshold alert $N>\tau$, deterministic no-missed-crossing holds only in +the unsampled monotone setting. With sampling, use a high-probability margin. + +### 8.2 Count-Min Sketch point query + +State: $d \times w$ nonnegative counter matrix. + +Point query for key $x$: + +```math +\widehat f_{CMS}(x) += +\min_r S[r,h_r(x)]. +``` + +Base sketch error: + +```math +f(x) +\le +\widehat f_{CMS}(x) +\le +f(x)+\epsilon_{sk}N +``` + +with probability at least $1-\delta_{sk}$, for the standard choice of width/depth. + +Sampling error applies to every row counter. For row $r$, define + +```math +X_r(x) += +\widehat S[r,h_r(x)]-S[r,h_r(x)]. +``` + +Then $X_r(x)$ is unbiased with variance controlled by the updates landing in +that cell: + +```math +\mathrm{Var}(X_r(x)) += +\sum_{u:h_r(x_u)=h_r(x)} +\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +A conservative high-probability point statement can be made by union-bounding +the row-level sampling events with the usual CMS collision event: + +```math +\widetilde f_{CMS}(x) +\le +f(x) ++\epsilon_{sk}N ++m_{sa}(x,\delta) ++k\max_r T_{r,h_r(x)}. +``` + +Because CMS uses a $\min$, the usual no-underestimate property is not preserved +by two-sided sampling noise. This makes CMS a caveated adapter rather than the +clean first theorem target. For alerting, fire with a confidence margin: + +```math +\widetilde f_{CMS}(x)+m_{sa}(x,\delta)+m_{cdm}(x) +\ge +(1-\epsilon)\tau. +``` + +CMS is therefore useful for upper-bound-oriented point queries, but sampled CMS +alerts should be presented as probabilistic and margin-based. CountSketch, Sum, +and DDSketch range counts are cleaner first theorem targets for sampled GOS. + +#### CDM isotropic threshold ($L_1$, max-composition) + +CMS's CDM staleness composes via the same $\min$-is-1-Lipschitz argument as +CountSketch's median (§8.3), not via the $L_2$ sum-composition GOS uses for +$F_2$: only $d$ cells (one per row) matter for a given query $x$, and the +$\min$ estimator's staleness is bounded by the single worst (most stale) of +those $d$ cells, not their combined mass. So + +```math +\mathrm{Err}^{cdm}_{CMS}(x) +\le +k\max_r T_{r,h_r(x)}. +``` + +Solving for an isotropic (uniform) $T$ so this stays within a chosen fraction +of the base sketch error $\epsilon_{sk}N$ gives, directly (no water-filling, +no $\sqrt{dw}$ factor — the max-composition needs none): + +```math +\boxed{ +T=\frac{\epsilon N}{k} +} +``` + +where $N=\sum_j S[j]$ is the sketch's total (nonnegative) mass — the natural +$L_1$ scale for CMS, as opposed to CountSketch's $L_2$ scale $\|\widehat C\|$. +$d,w$ are fixed at construction (matrix dimensions), so unlike DDSketch's +bucket count (§8.4) there is no "assumed vs. actual" gap to worry about here. + +### 8.3 CountSketch point query + +State: $d \times w$ signed counter matrix. + +Row estimate: + +```math +E_r(x)=s_r(x)S[r,h_r(x)]. +``` + +Point estimate: + +```math +\widehat f_{CS}(x)=\mathrm{median}_{r=1}^d E_r(x). +``` + +Base sketch error: + +```math +|\widehat f_{CS}(x)-f(x)| +\le +\epsilon_{sk}\|f\|_2 +``` + +with probability at least $1-\delta_{sk}$. + +Sampling error is two-sided and unbiased at each row. Let + +```math +X_r(x) += +s_r(x)(\widehat S[r,h_r(x)]-S[r,h_r(x)]). +``` + +Then + +```math +\mathbb{E}[X_r(x)]=0, +``` + +and its variance is the sum of inverse-probability variances of the sampled +updates colliding into the queried cell. A median-of-rows bound follows by +combining row-level concentration with the usual CountSketch row amplification. +This row amplification statement assumes Nitro-style row/counter-array sampling +or otherwise independent row-level sampling noise. If the implementation uses +edge admission sampling, the true-key sampling noise is shared across rows; then +the sampling term must be bounded before the median step and should not be +credited with CountSketch row amplification. + +CDM staleness for row $r$ is + +```math +|s_r(x)(\widetilde S-\widehat S)[r,h_r(x)]| +\le +kT_{r,h_r(x)}. +``` + +Since the median is 1-Lipschitz in the $L_\infty$ perturbation across rows: + +```math +|\mathrm{median}_r(a_r+e_r)-\mathrm{median}_r(a_r)| +\le +\max_r |e_r|, +``` + +the point-query staleness satisfies + +```math +\mathrm{Err}^{cdm}_{CS}(x) +\le +k\max_r T_{r,h_r(x)}. +``` + +Thus a paper-safe fixed-time statement is + +```math +|\widetilde f_{CS}(x)-f(x)| +\le +\epsilon_{sk}\|f\|_2 ++m_{sa}(x,\delta) ++k\max_r T_{r,h_r(x)}. +``` + +CountSketch is the cleanest first target for sampled GOS: additive state, +two-sided estimator, and no CMS one-sided caveat. + +### 8.4 DDSketch value-range counts + +DDSketch maps a positive value $x$ to a logarithmic bucket $b(x)$. The bucket +mapping is chosen so values in one bucket have relative value error at most +$\alpha$. + +For a value range $[L,U]$, define a linear bucket readout + +```math +q_{[L,U]}(S) += +\sum_{b: v_b\in[L,U]} S[b], +``` + +where $v_b$ is the representative value of bucket $b$. + +This is an additive nonnegative aggregate. Sampling and CDM compose exactly as +in the generic additive-state theorem: + +```math +|q_{[L,U]}(\widetilde S)-q_{[L,U]}(S)| +\le +m_{sa}([L,U],\delta) ++k\sum_{b:v_b\in[L,U]}T_b. +``` + +This is the DDSketch query class that behaves like a scalar/bucket-count monitor. + +#### CDM isotropic threshold ($L_1$, sum-composition) + +Unlike CMS's point query, the range-count readout $q_{[L,U]}$ is a genuine +sum across buckets, so it composes the same way CountSketch's $F_2$ does +(§7's general water-filling), just with an $L_1$ (linear, not squared) +gradient: $q$ is linear in each $S[b]$, so $g_b=\partial q/\partial S[b]=1$ +for every bucket in range — a constant gradient, giving (isotropic, +$T_b=T$ for all $B$ buckets in range): + +```math +k\sum_{b}T_b=k\cdot B\cdot T\le\epsilon N +\quad\Longrightarrow\quad +\boxed{ +T=\frac{\epsilon N}{k\cdot B} +} +``` + +$B$ must be a genuine, always-current bound on the number of populated +buckets — not a fixed assumed constant. Unlike CMS/CountSketch's $d,w$ +(fixed at construction), DDSketch's bucket count is dynamic in the current +implementation (unbounded growth on demand — see sketchlib-go#72 for a +related, independent memory-safety concern with that growth). $B$ is, +however, cheap to track exactly: increment a counter by 1 the first time a +given bucket index is touched (an $O(1)$ event, far rarer than raw inserts +once the value distribution stabilizes), and recompute $T$ from the latest +$N,B$ on every insert ($O(1)$, just a division) — so $T$ and $B$ never +drift out of sync the way a periodically-refreshed quantity would. This +keeps the guarantee exact without requiring any structural cap on the +sketch itself. + +### 8.5 DDSketch quantiles + +DDSketch quantile queries are not linear readouts. They depend on prefix counts. +Sampling and CDM should therefore be expressed as rank error. + +Let the true bucket count be + +```math +n_b=\sum_{u=1}^N 1\{b(x_u)=b\}. +``` + +Let + +```math +N_{\le b}=\sum_{j\le b}n_j. +``` + +#### Implemented thinning view + +The current DDSketch implementation performs value-independent uniform admission +before the bucket update: if the item is not admitted, the entire update is +skipped. This subsection assumes a single uniform probability $p$ for all items. +Let + +```math +Z_u\sim\mathrm{Bernoulli}(p). +``` + +The sampled bucket count is + +```math +n_b^{sample} += +\sum_{u:b(x_u)=b}Z_u. +``` + +Quantile lookup over the sampled DDSketch uses the sampled total + +```math +M=\sum_u Z_u. +``` + +Uniform thinning preserves the distribution in expectation. With high +probability, the empirical CDF of the sampled stream is close to the true CDF. +For a finite set of $B$ nonempty buckets, a union bound over bucket prefixes +gives + +```math +\sup_b +\left| +\frac{N^{sample}_{\le b}}{M} +- +\frac{N_{\le b}}{N} +\right| +\le +\epsilon_{sa} +``` + +with + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right), +``` + +assuming $pN$ is not too small. + +#### Horvitz-Thompson analysis view + +Equivalently, for analysis one may assign every admitted item weight $1/p$: + +```math +Y_u=Z_u/p. +``` + +Then + +```math +\widehat n_b=\sum_{u:b(x_u)=b}Y_u +``` + +is an unbiased estimator of $n_b$, and + +```math +\mathrm{Var}(\widehat N_{\le b}) += +N_{\le b}\frac{1-p}{p}. +``` + +Bernstein plus a union bound over $B$ prefixes yields + +```math +\sup_b +|\widehat N_{\le b}-N_{\le b}| +\le +O\left( +\sqrt{\frac{N\log(B/\delta)}{p}} ++\frac{\log(B/\delta)}{p} +\right). +``` + +Dividing by $N$ gives the same rank scale: + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right). +``` + +This weighted view is useful for proof. The implemented unweighted sampled +quantile returns the same bucket as the uniformly weighted view, because every +admitted item has the same weight. Counts, however, need `sample_p` rescaling if +they are queried as counts. + +If the controller uses nonuniform per-site probabilities $p_i$, the unweighted +sampled DDSketch quantile is biased toward high-$p_i$ sites. A nonuniform +deployment needs weighted quantile semantics, a resampling correction, or a +separate proof for the chosen estimator. + +#### CDM rank staleness + +If bucket residuals satisfy $|\rho_i[b]|\le T_b$, then for any prefix: + +```math +|\widetilde N_{\le b}-\widehat N_{\le b}| +\le +k\sum_{j\le b}T_j. +``` + +Hence the CDM-induced rank error is + +```math +\epsilon_{cdm} += +\frac{k}{N} +\sup_b\sum_{j\le b}T_j. +``` + +If the backend's total count is also stale, allocate a small additional budget +for denominator error, or normalize by a lower bound on $N$. + +#### DDSketch quantile corollary + +Let $\widetilde x_q$ be the sampled and stale DDSketch estimate of the +$q$-quantile. With probability at least $1-\delta$, + +```math +\boxed{ +(1-\alpha)x_{q-\epsilon_{sa}-\epsilon_{cdm}} +\le +\widetilde x_q +\le +(1+\alpha)x_{q+\epsilon_{sa}+\epsilon_{cdm}} +}. +``` + +Thus DDSketch keeps its multiplicative value error $\alpha$, while sampling and +CDM/GOS add rank error. + +### 8.6 KLL quantiles + +KLL is mergeable, but it is not a subtractive additive counter array. It should +not be forced into the per-cell linear residual theorem. + +For a sealed window, mergeability preserves the KLL rank guarantee: + +```math +|\mathrm{rank}(\widehat x_q)-qN| +\le +\epsilon_{KLL}N +``` + +with the configured KLL failure probability. + +If updates are uniformly sampled before KLL insertion, the same thinning rank +term appears: + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(1/\delta)}{pN}} ++\frac{\log(1/\delta)}{pN} +\right). +``` + +For open-window freshness, use a segment model: unshipped KLL segments contain +$R$ samples. Then the stale-rank contribution is bounded by + +```math +\epsilon_{cdm} +\le +R/N. +``` + +The combined quantile rank error is + +```math +\epsilon_{rank} +\le +\epsilon_{KLL} ++\epsilon_{sa} ++R/N. +``` + +KLL is therefore a mergeable-summary adapter, not a GOS per-cell water-filling +instance. + +#### Emit trigger + +Since $\epsilon_{cdm}\le R/N$, holding rank staleness within a chosen budget +$\epsilon$ means emitting (a full disjoint segment, then resetting — KLL has +no subtractive delta, so there is nothing finer-grained to send) once the +unshipped sample count crosses: + +```math +\boxed{ +R\ge\epsilon N +} +``` + +$R$ is exactly the sketch's own item count since its last reset (already +tracked — `Count()` in the current implementation, which zeroes on +`Reset()`), and $N$ is the window's running total count — both $O(1)$ +incremental counters, checked at insert time, no periodic scan required. + +### 8.7 HyperLogLog distinct count + +HLL state is a vector of max registers, with merge defined by register-wise max. +It is mergeable but non-additive. + +Base error: + +```math +\epsilon_{HLL}\approx 1.04/\sqrt{m} +``` + +for $m$ registers. + +Nitro-style inverse-probability update sampling is not appropriate for HLL +register updates: the update is a max operation, not an additive counter +increment. Hash-threshold distinct sampling can be analyzed separately, but it +does not fit the additive sampling theorem above. + +For CDM/freshness, use a family-specific register-change adapter. The linear +per-cell bound $k\sum_j |a_j|T_j$ does not apply directly. + +#### Register-change adapter (OctoSketch Algorithm 1 variant for max-merge sketches) + +HLL's contribution to the cardinality estimate is on the harmonic-mean +$2^{-C}$ scale, not the raw register value $C$: a register going from +$C=5$ to $C=6$ doubles that register's linear-scale contribution, while +$C=30\to31$ is an even larger absolute jump on that same linear scale — a +threshold on the raw register value would not reflect this. OctoSketch's +own treatment of max-merge sketches (HyperLogLog, LogLog, Loglog-Beta) +gates on the *linearized* value instead: register $j$ goes from $C$ to +$C'$ ($C'\ge C$, MAX-merge is monotone non-decreasing); send iff + +```math +\boxed{ +|2^{C'}-2^{C}|\ge2^{\tau} +} +``` + +— send the raw value $C'$ itself, and (unlike every additive family above) +**never reset the register**: MAX-merge is idempotent, so re-sending an +unchanged or already-known value at the backend is harmless +(`max(x,x)=x`), which is exactly what makes HLL's CDM/GOS treatment +structurally different from every additive family in this section. $\tau$ +is naturally interpreted in "doublings" — $\tau=1$ means "don't bother +sending unless this register's contribution at least doubled since it was +last sent," $\tau=2$ means "at least 4x," etc. Because $C,C'$ are small +non-negative integers, $2^{C}$ is exact integer arithmetic (`1< v^*$ is equivalent to a bucket-prefix +count predicate up to the DDSketch value bucket error. The sampling and CDM +terms then enter as rank/count margins. + +## 11. Summary table + +| Family/query | Sampling term | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) term | Isotropic threshold (§8) | Safe paper claim | +| --- | --- | --- | --- | --- | +| Sum/count | scalar Bernoulli count variance | $kT$ | $T=\epsilon N/k$ ($B=1$ case of DDSketch) | exact base sketch plus sampling plus scalar staleness | +| CMS point | row-counter sampling variance | $k\max_r T_{r,h_r(x)}$ | $T=\epsilon N/k$ (§8.2, max-composition, no $\sqrt{dw}$) | caveated adapter: row union plus collision event; probabilistic alert margin | +| CountSketch point | row-counter sampling variance | $k\max_r T_{r,h_r(x)}$ | $T=\epsilon\lVert\widehat C\rVert/(2k\sqrt{dw})$ (§7, $L_2$/$F_2$, sum-composition) | clean additive/two-sided composition; best first theorem target | +| DDSketch range count | bucket-count sampling variance | $k\sum_{b\in \mathrm{range}}T_b$ | $T=\epsilon N/(kB)$ (§8.4, $L_1$ sum-composition; $B$ tracked $O(1)$ per new bucket) | additive bucket-count composition | +| DDSketch quantile | rank error $\epsilon_{sa}$ | prefix rank error $\epsilon_{cdm}$ | — (uses range-count $T$ above) | $(1\pm \alpha)$ value error around $q\pm \epsilon$ rank | +| KLL quantile | thinning rank error | unshipped segment rank $R/N$ | emit+reset when $R\ge\epsilon N$ (§8.6, count-based, no cell structure) | mergeable adapter, not per-cell GOS | +| HLL distinct | exclude Nitro-style additive sampling | register-change adapter | $\lvert2^{C'}-2^{C}\rvert\ge2^{\tau}$ (§8.7, no reset — MAX-merge idempotent) | mergeable but non-additive; family-specific proof | + +Anisotropic (gradient-weighted, non-uniform $T_j$ per cell) is only worked +out for CountSketch (§7's general water-filling). Extending it to the other +families is a TODO — the $L_1$-composed families (CMS/DDSketch/Sum) would +need their own $c_j=k|g_j|$ instantiation, and the async-per-cell-reset +transport (below) breaks the water-filling's "Activity since a common +reference point" assumption in a way that isn't resolved yet (see the +implementation notes). + +## 12. Paper theorem templates + +### Theorem 1: additive-state composition + +For a fixed linear query $q(S)=\langle a,S\rangle$ at a fixed time in a tumbling window, +assume: + +1. sampling randomness is independent across updates, i.e. the admission + indicators $Z_u$ are independent; +2. sampled updates use inverse-probability weights for additive counters; +3. each site/cell residual is bounded by $R_{ij}$; in the update-synchronous + ideal protocol $R_{ij}=T_j$, while periodic checks can use + $R_{ij}=T_j+U_{ij}\Delta_{check}$; +4. the base sketch error is at most $E_{sk}(q)$ with probability + $1-\delta_{sk}$. + +Then with probability at least $1-\delta_{sk}-\delta_{sa}$, + +```math +|q(\widetilde S)-q(f)| +\le +E_{sk}(q) ++m_{sa}(q,\delta_{sa}) ++\sum_j |a_j|\sum_i R_{ij}. +``` + +If only a variance statement is needed, pairwise uncorrelated admission +indicators are sufficient for the variance identity, but not for the Bernstein +margin above. If all sites use the update-synchronous bound $R_{ij}=T_j$, the +staleness term reduces to $k\sum_j |a_j|T_j$. + +### Theorem 2: Geometric-OctoSketch (GOS) threshold allocation + +Under the threshold-crossing cost model, given additive-state staleness budget +$B$ and cell activity $V_j$, the threshold vector minimizing $\sum_j V_j/T_j$ +subject to $\sum_j c_jT_j\le B$ is + +```math +T_j += +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell\sqrt{V_\ell c_\ell}}, +``` + +before floors and caps. The clamped solution is obtained by iterative +water-filling over unclamped cells. + +### Corollary: DDSketch sampled open-window quantile + +For DDSketch relative value parameter $\alpha$, uniform value-independent +admission probability $p$, $B$ nonempty buckets, and bucket residual thresholds +$T_b$, with probability at least $1-\delta$: + +```math +(1-\alpha)x_{q-\epsilon_{sa}-\epsilon_{cdm}} +\le +\widetilde x_q +\le +(1+\alpha)x_{q+\epsilon_{sa}+\epsilon_{cdm}}, +``` + +where + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right), +``` + +and + +```math +\epsilon_{cdm} += +\frac{k}{N}\sup_b\sum_{j\le b}T_j. +``` + +## 13. Implementation notes + +- The Go DDSketch wrapper performs value-independent admission before the + bucket update. This is equivalent to uniform thinning for quantile rank + analysis. Counts need `sample_p` rescaling if they are queried as counts. +- CountSketch sparse deltas currently require integral cell deltas. If sampling + creates fractional weighted cells, the implementation can fall back to full + frames. That preserves accuracy but weakens communication claims for the + sampled+delta combination unless a fractional delta wire is added. +- SDK-side Nitro-style sampling requires a wire representation for sampled + weighted counter updates from SDK/data source to agent collector. For signed + sketches, the payload must preserve row, column, sign, and weight + $1/p_{i,r}$. +- The strongest continuous $|\rho_i[j]|\le T_j$ proof assumes an update-synchronous + threshold check — i.e. the threshold is checked at insert time, not on a + periodic scan. As of the 2026-07 no-sub-window redesign this is now the + actual model for the isotropic threshold in all 6 families (checked and, + where applicable, reset at insert time; wire transmission still happens via + the existing OTLP/`SketchEnvelope` pipeline, triggered on-demand instead of + on a fixed `SubWindowInterval` — see `design-gos-unified-edge-telemetry.md` + for the transport/architecture side of this). The one place the + update-synchronous assumption is NOT yet realized is anisotropic CountSketch: + the per-cell $\{T_j\}$ water-filling solve is $O(dw)$ and cannot run on every + insert, so it still needs a periodic re-solve, and — unresolved as of this + writing — the $Activity_j$ input to that solve needs a redefinition that + doesn't depend on a single common "prev" snapshot, since cells can now reset + asynchronously at different times (naive `|current-prev|` against a stale or + per-cell-inconsistent reference double-counts or under-counts activity + around individual cell resets). TODO, not blocking the other 5 families. +- CMS's local point-query readout (`cms_point` functional) is retired as part + of the same redesign: once cells reset in place at insert time, a `min`-based + local read is no longer a valid point-in-time estimate (a single recently- + reset row poisons the whole `min`). Alerting/point-query reads move + entirely to the backend, which reconstructs exact state via additive + fragment-summation (see the transport doc) regardless of when any given + edge-side cell was last reset. +- For SDK-source or multi-unit sampling, the sampling randomness must make the + admission indicators independent per `(edge_id, agg_id, window)` or hash-based + per item. Shared sampler seeds can correlate admissions and invalidate + variance-addition by introducing covariance terms. + +## 14. Evaluation checklist + +The system claim is only credible if the evaluation shows that ASAPCollector +moves work off the central raw-sample path while preserving the advertised error +envelope for supported queries. At minimum, measure: + +- **SDK/source update work:** candidate sketch counter updates/sec, admitted + counter updates/sec, CPU, and memory as $p_{i,r}$ changes. +- **Source-to-agent network load:** bytes/sec and messages/sec for raw OTLP + export versus sampled weighted counter-update export. +- **Agent-to-backend network load:** bytes/sec and messages/sec for periodic + sketch export and GOS thresholded deltas. +- **Backend ingest load:** accepted samples/sec or summary frames/sec, WAL/queue + pressure, index/storage growth, and write amplification if applicable. +- **Freshness:** open-window staleness measured as both wall-clock lag and + query-space error $\mathrm{Err}_q^{cdm}$. +- **Accuracy envelope:** empirical query error decomposed into sketch error, + sampling error, and staleness error; report coverage of the claimed + high-probability bound. +- **Controller behavior:** selected $p_{i,r}$ and $T_j$ under hot/cold sources, + query-sensitive/query-insensitive rows or cells, and changing workloads. +- **Fallback boundary:** unsupported query classes and cold raw fallback cost, + so the paper does not imply arbitrary PromQL support. +- **Failure modes:** tick-based overshoot, delayed ACKs, retries, collector + restart, and correlated sampler seeds. + +Use CountSketch, Sum/count, and DDSketch range counts as the first theorem-backed +accuracy experiments. Treat CMS, DDSketch quantiles, KLL, and HLL as +family-specific adapter experiments with their own stated caveats. + +## Appendix A. Bernstein inequality step + +This appendix derives the concentration bound used in Section 4.2: + +```math +|X_q| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2G}{3p_{\min}}\log(2/\delta) +``` + +with probability at least $1-\delta$. + +Recall the sampled query error: + +```math +X_q += +\sum_u (Y_u-1)g_u. +``` + +Define the centered per-update random variable + +```math +A_u=(Y_u-1)g_u. +``` + +Then + +```math +X_q=\sum_u A_u. +``` + +### A.1 Zero mean + +Because $E[Y_u]=1$, + +```math +\mathbb{E}[A_u] += +\mathbb{E}[(Y_u-1)g_u] += +g_u(\mathbb{E}[Y_u]-1) +=0. +``` + +Thus $X_q$ is a sum of independent centered random variables, assuming the +sampling admission indicators are independent. + +### A.2 Variance parameter + +From Section 4.1: + +```math +\mathrm{Var}(A_u) += +g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +Therefore the total variance parameter is + +```math +\sigma_q^2 += +\sum_u \mathrm{Var}(A_u) += +\sum_u g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +### A.3 Uniform bound on one summand + +Bernstein's inequality also needs an almost-sure bound on $|A_u|$. + +Since + +```math +Y_u = +\begin{cases} +1/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +0, & \text{if update }u\text{ is skipped}, +\end{cases} +``` + +we have + +```math +Y_u-1 = +\begin{cases} +(1-p_{i(u)})/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +-1, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +Thus + +```math +|Y_u-1| +\le +\frac{1}{p_{i(u)}} +\le +\frac{1}{p_{\min}}. +``` + +If $|g_u| \le G$, then + +```math +|A_u| += +|(Y_u-1)g_u| +\le +\frac{G}{p_{\min}}. +``` + +Let + +```math +M=\frac{G}{p_{\min}}. +``` + +### A.4 Apply Bernstein + +One standard two-sided Bernstein bound for independent centered random variables +$A_u$ with $|A_u| \le M$ and variance sum $\sigma_q^2$ is + +```math +\Pr\left[ +\left|\sum_u A_u\right| +\ge +\sqrt{2\sigma_q^2 t} ++\frac{2M}{3}t +\right] +\le +2e^{-t}. +``` + +Set + +```math +t=\log(2/\delta). +``` + +Then $2e^{-t}=\delta$, so with probability at least $1-\delta$, + +```math +\left|\sum_u A_u\right| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2M}{3}\log(2/\delta). +``` + +Substituting $M=G/p_{\min}$ and $X_q=sum_u A_u$ gives + +```math +|X_q| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2G}{3p_{\min}}\log(2/\delta). +``` + +This is the Section 4.2 bound. + +## 15. References to cite + +- Prometheus remote write and storage documentation: central ingest, queue/WAL, + CPU, memory, and network costs for exporting high-resolution metrics. +- OpenTelemetry metrics data model: metric events, aggregation temporality, + histograms/exponential histograms, and the motivation for aggregation before + export. +- Prometheus histograms and summaries documentation: early binding of buckets, + quantiles, and aggregation constraints. +- Thanos compactor/downsampling documentation and TSDB compression papers: + post-ingest query/storage optimization rather than pre-ingest load reduction. +- BlinkDB and VerdictDB: approximate query processing over already-ingested + data. +- NitroSketch: update sampling for sketch update work. +- DDSketch: mergeable relative-error quantile sketch. +- Count-Min Sketch and CountSketch: base frequency guarantees. +- KLL: mergeable rank-error quantile sketch. +- HyperLogLog: base cardinality guarantee. +- Cormode et al. distributed functional monitoring / Continuous Distributed + Monitoring (CDM). +- Mergeable summaries: no error compounding under merge. +- Zoom2Net and related telemetry reconstruction work: learned fine-grained + reconstruction from coarse measurements, distinct from preserving + query-sufficient sketch state. From c0687058f918598a9d9b12200e5ef98c031bf212 Mon Sep 17 00:00:00 2001 From: zzylol Date: Thu, 16 Jul 2026 12:57:36 -0600 Subject: [PATCH 2/3] docs(gos): trim stale implementation-status cruft from the design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cut the "current vs target" sampling-rollout table (§3.1), the redundant §3.2 "Status." paragraph (which had drifted out of sync with §3.1's own later claim), and §12's "Implementation status as of this writing" block and half of "Ties to existing code" — several of which pointed at code already deleted (AllocateThresholds/gos_threshold matrix, gos_anisotropic, f2_coord.rs/f2engine.go from the F2 removal). Net: shorter, and no longer self-contradictory or pointing at dead files. Co-Authored-By: Claude Sonnet 5 --- docs/design-gos-unified-edge-telemetry.md | 90 ++++++----------------- 1 file changed, 22 insertions(+), 68 deletions(-) diff --git a/docs/design-gos-unified-edge-telemetry.md b/docs/design-gos-unified-edge-telemetry.md index 390eac25..44d16635 100644 --- a/docs/design-gos-unified-edge-telemetry.md +++ b/docs/design-gos-unified-edge-telemetry.md @@ -156,24 +156,12 @@ weighting is a linear correction). So it applies to: So the SDK runs row-admission for **DDSketch / CMS / CountSketch** and leaves **KLL / HLL** unsampled (they emit every update). -**Implementation status (current vs this target).** Today's edge runtime -implements a *weaker* form of this design, and closing the gap is tracked work: - -| | This design (target) | Current code | -|---|---|---| -| Algorithm | geometric skip-sampling | ✅ `sketchlib-go/common.GeometricSampler` | -| Weighting | `1/p` on admit | ✅ `CountSketchWrapper.UpdateString` (`count /= sampleP`) | -| Families | CMS/CS/DDSketch only | ✅ `applyGrantedSampleP` | -| **Granularity** | per-**row** admission (subset of `d` rows; hash only if ≥1 admitted) | ✅ **per-row** — `CountSketch.UpdateStringSampledPerRow` (sketchlib), wired in both the collector wrapper and the SDK aggregator | -| **Where** | SDK decides, then sends | ✅ **SDK-build path** — hosted in the OTLP SDK `CountSketch` aggregator (`opentelemetry-go-patch/.../aggregate/countsketch.go`, knob `sample_p`); collector-build path retains the wrapper fallback | - -The **granularity** matches the design (per-row admission, drop-before-hash, -`1/p` weight — mirroring `asap_sketchlib/.../nitro.rs`), so the sampling term is -the tight per-row estimator of §3.2 (median decorrelation, `ε_sa` in -quadrature). The **location** is now realized on the **SDK-build path**: the -OTLP metrics SDK's `CountSketch` aggregator hosts a per-series sampler and -routes every measurement through `UpdateStringSampledPerRow`, so admission -happens at the source. Two deployment modes and what each saves: +**Implemented as designed**, hosted on the SDK-build path (OTLP SDK +`CountSketch` aggregator, knob `sample_p`, routing through +`UpdateStringSampledPerRow`) with per-row admission, drop-before-hash, and +`1/p` weighting — mirroring `asap_sketchlib/.../nitro.rs`, giving the +sampling term the tight per-row estimator of §3.2 (median decorrelation, +`ε_sa` in quadrature). Two deployment modes exist, with different savings: - **SDK-build (sketch in the app).** The SDK builds the group sketch and ships one matrix per window. Sampling here is the source-side realization of the @@ -190,12 +178,11 @@ happens at the source. Two deployment modes and what each saves: on survivors. (Whole-item drop is the `R(x)=∅` fast path of the same geometric sampler.) -All three families are wired on the SDK-build path via a `sample_p` knob: -**CountSketch** and **CountMinSketch** host a per-series sampler and route -inserts through `UpdateStringSampledPerRow` / `InsertWithHashSampledPerRow` -(per-row admission, `1/p` applied in-place, wire stays exact — no downstream -rescale); **DDSketch** is the `d=1` whole-item case and uses the sketch's -built-in `WithSampleP` (raw counts, wire stamps `p`, consumer rescales `×1/p`). +**CountSketch** and **CountMinSketch** route inserts through +`UpdateStringSampledPerRow` / `InsertWithHashSampledPerRow` (per-row +admission, `1/p` applied in-place, wire stays exact — no downstream rescale); +**DDSketch** is the `d=1` whole-item case via the sketch's built-in +`WithSampleP` (raw counts, wire stamps `p`, consumer rescales `×1/p`). `KLL`/`HLL` stay unsampled by design (§ applicability table). ### 3.1.1 Single-location sampling via consistent (stateless) decisions @@ -381,14 +368,6 @@ The GOS water-filling is unchanged in form; the ε-budget is split by §7 Layer (staleness `ε_st` peeled linearly, then `ε_sk²+ε_sa² = (ε_q−ε_st)²`), and this floor closes the sampling↔threshold coupling. -**Status.** The per-row *estimator* benefit above is **realized** in code via -`UpdateStringSampledPerRow`, so `ε_sa` sits inside the `1−δ` median guarantee. The -*location* move is realized on the **SDK-build path** — the OTLP SDK `CountSketch` -aggregator hosts the sampler (§3.1) — so admission happens at the source; the -collector-build path realizes the pre-deserialization saving at the edge -`otlpfilter` (whole-datapoint wire-thinning). CMS/DDSketch SDK-build hosting is -the remaining follow-up. - --- ## 4. Unified error bound @@ -783,10 +762,11 @@ executes a fixed per-cell comparison. (`controller-optimization-problem.md` SP-6: `min w_bw·bw + w_cpu·cpu + w_mem·mem + …`) — GOS thresholds are new decision variables there. **Note:** the controller ships *scalars* - (`ε_delta`, sites, aniso flag), **not** the full per-cell vector `{T_j}`; the - edge reconstructs `{T_j}` locally from those scalars plus its live sketch state - (see §7C, `sketches/gos_threshold.go`), so the `O(d·w)` vector never crosses the - wire. + (`ε_delta`, sites), **not** a full per-cell vector `{T_j}`; the isotropic + case (§11, all 6 families) reconstructs its single scalar `T` from those + plus live sketch state, so no vector ever crosses the wire. The anisotropic + per-cell `{T_j}` water-filling described here is CountSketch-only and, per + §11's open items, not currently implemented. - **Edge**: maintain sketch + acked snapshot; per flush, recompute `{T_j}` from the pushed scalars + local `{V_j}`, upload cells with `|ΔC_j| ≥ T_j` as a sparse delta; run one generic `isLocallySafe` for monitored functions. No AD, no @@ -802,39 +782,13 @@ executes a fixed per-cell comparison. the `accuracy: ε=…` response annotation. **Ties to existing code:** -- `ASAPQuery-backend/control_plane/src/epsilon_alloc.rs` — the ε-budget split. - Staleness is peeled **linearly** first (Theorem 1), then the remaining random - budget splits in quadrature: `√(ε_sk² + ε_sa²) + ε_st = ε_q` (see §7 Layer B). - This is **not** a three-way quadrature `ε² = ε_sk² + ε_sa² + ε_st²` — staleness - is deterministic and comes off the top linearly. +- `ASAPQuery-backend/control_plane/src/epsilon_alloc.rs` — the ε-budget split + (linear staleness peel, then quadrature split: `√(ε_sk² + ε_sa²) + ε_st = ε_q`, + §7 Layer B — not a three-way quadrature). - `ASAPQuery-backend/data_plane/src/monitor/sampling_alloc.rs` - (`epsilon_sample_floor`) — the live whole-sketch sampling floor. (The Go - `monitor/sampling_alloc.go` `AllocateSampleRates` is the *retired* per-key - water-filling, kept only as a reference impl with no production caller.) -- `threshold_alloc` (`AllocateThresholds`, Go `sketches/gos_threshold.go` + Rust - `control_plane/src/threshold_alloc.rs`) — the per-cell threshold water-filling - (this doc's §7C). -- `data_plane/src/monitor/f2_coord.rs`, `asap-precompute-go/monitor/f2engine.go` - — generalize the F₂-specific ball to the `(∇f, λ)` DC safe zone; use the relative - radius `ε‖Ĉ‖/(2k√(dw))`. -- reuse `asap_sketchlib` `CountSketchDelta` + `compute_delta`/`apply_delta` - (byte-parity Go/Rust) for the sparse per-cell delta wire format. - -**Implementation status (as of this writing).** The pieces exist but the GOS -threshold control loop is **not yet wired end-to-end**: -- **Live today:** the scalar CDM loop (register → grant `(slack, sample_p)` → - countdown → report → alert) and the sampling grant path (`Grant.SampleP` → - `otlpfilter` Upsert + wrapper `WithSampleP`). -- **Implemented but unreachable from a production config:** the control plane - *derives and emits* the scalar GOS knobs (`gos_delta_epsilon`, `gos_sites`, - `gos_anisotropic` in `emit/agent.rs`), and the edge *consumes* - `PrecomputeConfig.GosDeltaEpsilon` (`applyGosMode` → `gosThresholdMatrix`), but - **no collector processor parses those YAML keys into the config**, so - `applyGosMode` is a production no-op and the per-cell delta-gating path is - exercised only by tests/eval. The sampling↔threshold coupling floor - `T_j ≥ √(V_j(1−p)/p)` (§3.2) is likewise implemented but inert — its only - production-shaped caller hardcodes `SampleP=1`. Closing this last hop - (a knob parser + threading the granted `p` into `GosParams`) is tracked work. + (`epsilon_sample_floor`) — the live whole-sketch sampling floor. +- `asap_sketchlib` `CountSketchDelta` + `compute_delta`/`apply_delta` + (byte-parity Go/Rust) — the sparse per-cell delta wire format. --- From d2025ed578595dc0e89aaa88f195df94fa291877 Mon Sep 17 00:00:00 2001 From: zzylol Date: Thu, 16 Jul 2026 13:08:42 -0600 Subject: [PATCH 3/3] docs(gos): integrate the insert-time mechanism into the doc's sections MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the former standalone "2026-07 redesign" section into the rest of the document instead of leaving it as a dated patch note bolted on at the end: - The unified insert-time model, wake-on-demand flush, per-family detection/reset table, and cold-start rationale move into §7 as "Layer D" (Layers A-C already establish *what* threshold each cell gets; Layer D is *when* a crossing is detected/shipped) - the same section that already derives the closed-form thresholds these checks use, instead of sitting in a separate, differently-scoped section. - The retirement specifics (Gate 1, CMS point-query, Discipline B alerting) move to §11 (Implementation notes), next to the existing "Ties to existing code" - this is where migration-from-current-code detail already lives, so it no longer needs a "superseded by §11" cross-reference note explaining itself. - The open items (Activity_j redefinition, DDSketch bucket growth, HLL small-cardinality) merge into the pre-existing Open problems list instead of duplicating it as a second list. Sections renumber 11->omitted, 12->11, 13->12; no content dropped, only relocated and de-dated. Net -9 lines despite the merge, from removing the now-unneeded cross-reference scaffolding. Co-Authored-By: Claude Sonnet 5 --- docs/design-gos-unified-edge-telemetry.md | 301 +++++++++++----------- 1 file changed, 146 insertions(+), 155 deletions(-) diff --git a/docs/design-gos-unified-edge-telemetry.md b/docs/design-gos-unified-edge-telemetry.md index 44d16635..9a522cd2 100644 --- a/docs/design-gos-unified-edge-telemetry.md +++ b/docs/design-gos-unified-edge-telemetry.md @@ -525,6 +525,81 @@ chase high-frequency noise); cells the monitored function is sensitive to (`|g_j|` large) get smaller thresholds (report early); everything capped by the universal query cap and freshness. +**Layer D — when: insert-time, no periodic scan.** Layers A–C fix *what* +threshold each cell gets; this layer is *when* a crossing is actually +detected and shipped. One mechanism for every family (Sum, CMS, CountSketch, +DDSketch, KLL, HLL), isotropic case: check at insert time whether the +accumulated-since-last-sync delta crosses the cell's `T_j` — or, for +KLL/HLL, the family's whole-sketch trigger; every family's exact formula is +in `sampling-cdm-gos-derivations.md` §8. If it crosses, that cell's (or +scalar's) delta needs to reach the backend. The sole purpose is data +synchronization — keeping the backend's reconstructed state accurate. +Alerting and any other query-time decision is made entirely at the backend +against that synced state; the edge does not make alerting decisions +itself (see Retirements in §11 for what this replaces in the current code). + +### Wake-on-demand flush + +The existing OTLP export chain (SDK `PeriodicReader` → collector processor +→ exporter) is unchanged; only *when a flush cycle runs* differs from a +purely timer-driven cadence — it is timer-**or**-woken: + +```go +for { + select { + case <-ticker.C: // slow fallback cadence, in case a wake is ever missed + flush() + case <-wakeCh: // fired the instant something crosses threshold + flush() + } +} +``` + +Insert path, non-blocking (never waits for the flush loop): + +```go +select { +case wakeCh <- struct{}{}: +default: // a wake is already pending; nothing to add +} +``` + +`flush()` keeps using the existing `SnapshotCache`/`ComputeDeltaAgainst` +machinery (full-frame fallback on cold start, the "never emit a delta +larger than a full frame" clamp) — only the trigger changed. A per-cell +`dirtySet`, populated as cells cross threshold, replaces the old periodic +whole-sketch divergence pre-check: an empty `dirtySet` at flush time IS +"nothing was worth sending," computed once per crossing instead of by a +periodic `O(dw)` full-matrix scan. + +### Per-family detection unit and reset semantics + +| Family | Detection unit | Reset on send? | Notes | +|---|---|---|---| +| CountSketch (isotropic) | matrix cell | zero it | `normSqAll` tracked incrementally (`+= 2·old·Δ+Δ²`), O(1) | +| CountMinSketch | matrix cell | zero it | same mechanism, $L_1$-scale threshold (derivations §8.2) | +| DDSketch | bucket count | zero it | bucket count `B` tracked incrementally too (+1 on genuinely new bucket), no config constant needed (derivations §8.4) | +| Sum | scalar | zero it (subtract reported amount) | degenerate 1-cell case | +| KLL | whole sketch (no per-cell structure) | full `Reset()` | trigger is `Count() >= εN`, not a per-cell check; already the existing disjoint-segment mechanism, just re-triggered by count instead of a timer | +| HLL | register | **never** — MAX-merge is idempotent, only clear a dirty flag | trigger is $\lvert 2^{C'}-2^{C}\rvert \ge 2^{\tau}$ on the linearized value, not raw register value (derivations §8.7) | + +Backend reconstruction is unchanged for the additive families (Sum/CMS/CS/ +DDSketch): summing every fragment ever received for a cell — regardless of +how many times or when it was individually reset — telescopes to the true +cumulative value (`v_1+v_2+...+v_n+v_{residual}` = true total). Resets can +therefore happen asynchronously, at different times per cell, without +breaking correctness — it only affects *when* transmission happens, never +*what* the backend eventually reconstructs. + +### Cold start is a feature, not a bug + +Every family's threshold scales with an accumulated quantity (`‖Ĉ‖`, `N`, +`R`) that starts near zero at window start, so the very first few inserts +cross threshold almost immediately. This is intentional: it gets the +backend a usable initial estimate as fast as possible, rather than waiting +for data to accumulate before syncing anything. No floor/minimum-threshold +mechanism is needed to suppress this. + ### Closed forms - **F₂, isotropic** (`g_j = 2Ĉ_j`, `λ=2`, uniform, relative): `T = ε‖Ĉ‖ / (2k√(dw))` @@ -617,170 +692,61 @@ for exactly this comparison. Consequences: --- -## 11. The 2026-07 redesign: no sub-window, insert-time detection for all 6 families - -Everything above (§1-10) establishes *what threshold to use*. This section is -about *when transmission actually happens* — a mechanism-level redesign that -applies uniformly across Sum, CMS, CountSketch, DDSketch, KLL, and HLL, -replacing the periodic sub-window tick with per-insert detection while -keeping the existing OTLP/`SketchEnvelope` wire and pipeline unchanged. - -### The unified model - -One mechanism for every family, isotropic case: **check at insert time -whether the accumulated-since-last-sync delta crosses the family's threshold -(§8 of `sampling-cdm-gos-derivations.md`); if it does, that cell's (or -scalar's) delta needs to reach the backend.** The sole purpose is data -synchronization — keeping the backend's reconstructed state accurate. Alerting -and any other query-time decision is made entirely at the backend against -that synced state; the edge no longer makes alerting decisions itself. This -retires the previously-separate Discipline B "continuous monitoring" path -(`monitor.Engine.Observe` → immediate `sendReportLocked` over -`monitor/grpcclient`) as a distinct alerting mechanism — Sum becomes just -another family running the same insert-time check, synced over the normal -`SketchEnvelope` pipeline like everything else. `Engine.Observe` today -conflates two things in one function: `obsCount++` (rate tracking, feeding -the coordinator's `SampleP` grant negotiation) and the -`value-baseline>=slack` alerting check (which calls `sendReportLocked`). Only -the *alerting* half retires; `obsCount`/rate-tracking must be preserved (it -feeds a genuinely separate concern — sampling-rate negotiation, not data -sync) — these two halves need to be split apart, not deleted together. - -### Wake-on-demand flush (keeps the existing OTLP pipeline) - -The naive read of "insert-time, no sub-window" is "bypass the pipeline -entirely, send out-of-band" — this was considered and rejected. The OTLP -export chain (SDK `PeriodicReader` → collector processor → exporter) stays -exactly as-is; only *when a flush cycle runs* changes, from purely -timer-driven to timer-**or**-woken: - -```go -for { - select { - case <-ticker.C: // slow fallback cadence, in case a wake is ever missed - flush() - case <-wakeCh: // fired the instant something crosses threshold - flush() - } -} -``` - -Insert path, non-blocking (never waits for the flush loop): - -```go -select { -case wakeCh <- struct{}{}: -default: // a wake is already pending; nothing to add -} -``` - -`flush()` keeps using the existing `SnapshotCache`/`ComputeDeltaAgainst` -machinery (full-frame fallback on cold start, the "never emit a delta larger -than a full frame" clamp) unchanged — the only thing that changed is the -trigger. This makes **Gate 1 (the old `subWindowShouldEmit` / -`L2DivergenceSinceEmit` divergence pre-check / `ackedCells`) redundant**: an -empty `dirtySet` at flush time IS "nothing was worth sending," computed once -per crossing instead of by a periodic O(dw) full-matrix scan. All of that -(`ackedCells`, `subWindowDivergence`, `subWindowShouldEmit`) is dead code -under this design and should be deleted, not kept as a fallback path. - -CMS's local point-query read (`ThresholdConfig.Functional: cms_point`) -retires alongside Discipline B's alerting: once cells reset in place at -insert time, a `min`-based local read is corrupted by any single recently- -reset row. All point/alert reads move to the backend's reconstructed copy. - -### Per-family cell / threshold / reset semantics - -See `sampling-cdm-gos-derivations.md` §8 for the derivations; summary of the -*mechanism* (not the math) each family needs: - -| Family | Detection unit | Reset on send? | Notes | -|---|---|---|---| -| CountSketch (isotropic) | matrix cell | zero it | `normSqAll` tracked incrementally (`+= 2·old·Δ+Δ²`), O(1) | -| CountMinSketch | matrix cell | zero it | same mechanism, $L_1$-scale threshold (§8.2) | -| DDSketch | bucket count | zero it | bucket count `B` tracked incrementally too (+1 on genuinely new bucket), no config constant needed (§8.4) | -| Sum | scalar | zero it (subtract reported amount) | degenerate 1-cell case | -| KLL | whole sketch (no per-cell structure) | full `Reset()` | trigger is `Count() >= εN`, not a per-cell check; already the existing disjoint-segment mechanism, just re-triggered by count instead of a timer | -| HLL | register | **never** — MAX-merge is idempotent, only clear a dirty flag | trigger is $\lvert 2^{C'}-2^{C}\rvert \ge 2^{\tau}$ on the linearized value, not raw register value (§8.7) | - -Backend reconstruction is unchanged for the additive families (Sum/CMS/CS/ -DDSketch): summing every fragment ever received for a cell — regardless of -how many times or when it was individually reset — telescopes to the true -cumulative value (`v_1+v_2+...+v_n+v_{residual}` = true total). This is why -resets can happen asynchronously, at different times per cell, without -breaking correctness — it only affects when transmission happens, never -what the backend eventually reconstructs. - -### Cold start is a feature, not a bug - -Every family's threshold scales with an accumulated quantity (`‖Ĉ‖`, `N`, -`R`) that starts near zero at window start, so the very first few inserts -cross threshold almost immediately. This is intentional: it gets the backend -a usable initial estimate as fast as possible, rather than waiting for data -to accumulate before syncing anything. No floor/minimum-threshold mechanism -is needed to suppress this. - -### Open items (not blocking) - -- **Anisotropic CountSketch's `Activity_j`** needs redefinition. The old - `Activity_j=|current-prev|` assumed a single, uniformly-timed `prev` - snapshot; under per-cell async reset, different cells' "since last touch" - windows are no longer comparable, and naively diffing against any - snapshot double-counts/under-counts around individual cell resets. A - per-cell EMA of `|Δ|` (`activityRate[r][c] = decay·activityRate[r][c] + - (1-decay)·|Δ|`, updated every insert) is the leading candidate — cheap, - reset-timing-independent — but it replaces the derivation's exact - `Activity_j=V_j` with a heuristic, and whether the §7 closed-form - water-filling solution still carries the same error guarantee under that - substitution has not been checked. The anisotropic water-filling solve - itself also still requires a periodic $O(dw)$ pass (unlike every other - family here) — per-cell detection at insert time only avoids the - "decode a serialized `prev`" cost, not the joint solve. -- **DDSketch's unbounded contiguous bucket-array growth** on outlier values - is a real memory-safety gap, independent of this redesign — tracked as - sketchlib-go#72. The dynamically-tracked `B` used in the threshold formula - above does not require fixing this; it is a separate, likely - higher-priority issue. -- **HLL's small-cardinality regime**: the register-change adapter's accuracy - proof (OctoSketch's Appendix B) is stated for "sufficiently large" - cardinality; behavior when most registers are still at 0 (early in a - window) has not been separately verified. A candidate mitigation (always - send a register's first-ever nonzero write unconditionally) is proposed - but unverified. - ---- - -## 12. Implementation notes (controller synthesizes, edge executes) +## 11. Implementation notes (controller synthesizes, edge executes) Everything expensive is a **controller (backend) decision**; the edge only executes a fixed per-cell comparison. - **Controller** (offline, per registered metric/query): runs ADCD (AD → Hessian eigenvalue bounds → `∇f, λ`), estimates `{V_j}` from the workload, solves (P)'s - layers A–C, emits `(d, w, G, {p_i}, scalar GOS knobs, flags)` via OpAMP. This + layers A–D, emits `(d, w, G, {p_i}, scalar GOS knobs, flags)` via OpAMP. This slots into the existing controller multi-objective (`controller-optimization-problem.md` SP-6: `min w_bw·bw + w_cpu·cpu + w_mem·mem + …`) — GOS thresholds are new decision variables there. **Note:** the controller ships *scalars* (`ε_delta`, sites), **not** a full per-cell vector `{T_j}`; the isotropic - case (§11, all 6 families) reconstructs its single scalar `T` from those - plus live sketch state, so no vector ever crosses the wire. The anisotropic - per-cell `{T_j}` water-filling described here is CountSketch-only and, per - §11's open items, not currently implemented. -- **Edge**: maintain sketch + acked snapshot; per flush, recompute `{T_j}` from - the pushed scalars + local `{V_j}`, upload cells with `|ΔC_j| ≥ T_j` as a sparse - delta; run one generic `isLocallySafe` for monitored functions. No AD, no - water-filling solve at the edge — only the closed-form threshold evaluation. - **Superseded for the isotropic case by §11**: there is no longer a - periodic "per flush" cadence or a separate acked-snapshot copy — the - per-cell comparison happens at insert time and the cell resets in place - on send. This paragraph's *acked-snapshot, periodic-flush* model still - describes anisotropic CountSketch's water-filling solve, which §11 flags - as an open item, not yet redesigned. + case (§7 Layer D, all 6 families) reconstructs its single scalar `T` from + those plus live sketch state, so no vector ever crosses the wire. The + anisotropic per-cell `{T_j}` water-filling described here is + CountSketch-only and, per Open problems below, not currently implemented. +- **Edge**: maintain sketch state; per insert, compare against `{T_j}` + (recomputed from the pushed scalars + local `{V_j}` — §7 Layer D) and mark + crossed cells dirty; a wake fires a flush of the current dirty set as a + sparse delta; run one generic `isLocallySafe` for monitored functions. No + AD, no water-filling solve at the edge for the isotropic case — only the + closed-form threshold evaluation, checked continuously rather than on a + periodic scan. The anisotropic CountSketch water-filling solve is the one + remaining case that still needs a periodic `O(dw)` re-solve and a separate + acked-snapshot copy (see Open problems). - **Backend**: `apply_delta` into a running merge (`O(#delta cells)`), keeping the global sketch continuously queryable within the Theorem-1 envelope, surfaced in the `accuracy: ε=…` response annotation. +**Retirements.** The insert-time model (§7 Layer D) replaces the following +in the current code — tracked here since it isn't deleted yet: + +- **Gate 1** (`subWindowShouldEmit` / `L2DivergenceSinceEmit` / `ackedCells`): + the old periodic whole-sketch divergence pre-check, run once per + `SubWindowInterval` tick regardless of whether anything had actually + crossed threshold. An empty per-cell `dirtySet` at flush time already + answers "nothing to send," computed once per crossing instead of by a + periodic `O(dw)` scan — dead code under Layer D, not kept as a fallback. +- **CMS's local point-query read** (`ThresholdConfig.Functional: cms_point`): + once cells reset in place at insert time, a `min`-based local read is + corrupted by any single recently-reset row. All point/alert reads move to + the backend's reconstructed copy. +- **Discipline B's alerting path** (`monitor.Engine.Observe` → + `sendReportLocked` over `monitor/grpcclient`): Sum becomes just another + family running the Layer D insert-time check, synced over the normal + `SketchEnvelope` pipeline like everything else. `Engine.Observe` today + conflates two things in one function: `obsCount++` (rate tracking, + feeding the coordinator's `SampleP` grant negotiation) and the + `value-baseline>=slack` alerting check (which calls `sendReportLocked`). + Only the *alerting* half retires; `obsCount`/rate-tracking must be + preserved (it feeds a genuinely separate concern — sampling-rate + negotiation, not data sync) — the two halves need to be split apart, not + deleted together. + **Ties to existing code:** - `ASAPQuery-backend/control_plane/src/epsilon_alloc.rs` — the ε-budget split (linear staleness peel, then quadrature split: `√(ε_sk² + ε_sa²) + ε_st = ε_q`, @@ -792,20 +758,45 @@ executes a fixed per-cell comparison. --- -## 13. Open problems / next steps - -1. **Anisotropic delta broadcast** — *partially done.* The **sparse-cell** +## 12. Open problems / next steps + +1. **Anisotropic CountSketch's `Activity_j`** needs redefinition. The old + `Activity_j=|current-prev|` assumed a single, uniformly-timed `prev` + snapshot; under per-cell async reset (§7 Layer D), different cells' + "since last touch" windows are no longer comparable, and naively diffing + against any snapshot double-counts/under-counts around individual cell + resets. A per-cell EMA of `|Δ|` (`activityRate[r][c] = + decay·activityRate[r][c] + (1-decay)·|Δ|`, updated every insert) is the + leading candidate — cheap, reset-timing-independent — but it replaces the + derivation's exact `Activity_j=V_j` with a heuristic, and whether the §7 + closed-form water-filling solution still carries the same error guarantee + under that substitution has not been checked. The anisotropic + water-filling solve itself also still requires a periodic `O(dw)` pass + (unlike every other family) — per-cell detection at insert time only + avoids the "decode a serialized `prev`" cost, not the joint solve. +2. **Anisotropic delta broadcast** — *partially done.* The **sparse-cell** encoding of `ΔC_ref` on the coordinator→edge path is implemented and measured (`CRefUpdate::Delta`; removes the `O(k)` broadcast amplification — see gos-eval-results.md §2). Still **open:** the broadcast gate is currently isotropic (ships every changed cell, `Δ ≠ 0`); giving it *anisotropic per-cell thresholds* (the §7C water-filling, as already done on the edge→coordinator upload path via `ComputeDeltaPerCell`) is the remaining work. -2. **Relative-error under small norm** — heartbeat / additive floor when `‖Ĉ‖` is +3. **DDSketch's unbounded contiguous bucket-array growth** on outlier values + is a real memory-safety gap, independent of Layer D — tracked as + sketchlib-go#72. The dynamically-tracked `B` used in the threshold formula + (derivations §8.4) does not require fixing this; it is a separate, likely + higher-priority issue. +4. **HLL's small-cardinality regime**: the register-change adapter's accuracy + proof (OctoSketch's Appendix B) is stated for "sufficiently large" + cardinality; behavior when most registers are still at 0 (early in a + window) has not been separately verified. A candidate mitigation (always + send a register's first-ever nonzero write unconditionally) is proposed + but unverified. +5. **Relative-error under small norm** — heartbeat / additive floor when `‖Ĉ‖` is small (WZ / OctoSketch fundamental limit). -3. **Verified eigenvalue bounds** — AutoMon's numerical `λ` may miss the true +6. **Verified eigenvalue bounds** — AutoMon's numerical `λ` may miss the true extreme → reserve an `ε_eig` slice of the budget or use interval bounds. -4. **Empirical validation** — measure achieved communication as a fraction of the +7. **Empirical validation** — measure achieved communication as a fraction of the WZ `k/ε²`, sweep `(w_m, w_e, w_c)` to trace the Pareto surface. ---