diff --git a/docs/design-gos-unified-edge-telemetry.md b/docs/design-gos-unified-edge-telemetry.md new file mode 100644 index 00000000..9a522cd2 --- /dev/null +++ b/docs/design-gos-unified-edge-telemetry.md @@ -0,0 +1,816 @@ +# GOS: A Unified Error / Threshold / Cost Framework for Distributed Edge Telemetry + +**One line.** One framework that unifies error-bounded sketching, coordinated +sampling, Geometric Monitoring (GM/AutoMon), and OctoSketch-style change +transmission into a single per-cell decision — reducing to **one atomic quantity** +`e_j = k·T_j` (per-cell backend error) — with a unified relative error bound, a +tunable memory/compute/communication objective, and a closed-form +water-filling solution whose worst-case communication matches the +Woodruff–Zhang lower bound `Θ̃(k/ε²)`. + +We call the construction **GOS** (Geometric-OctoSketch). + +--- + +## 1. Context and requirements + +Distributed data collection and transmission with a centralized analytics +backend. Edge telemetry must be **memory-, computation-, and +communication-efficient** under cloud economics, while the backend serves +**continuous, accurate, fresh** queries. + +- **Memory efficiency.** Cloud memory instances are priced above CPU instances. + With high-cardinality, high-frequency time-series metrics from many + distributed services, allocating *one sketch per series per window* puts heavy + memory pressure on the data-source nodes. Memory must be controlled across the + series×window space. +- **Computation efficiency.** Per-sample sketch update work at the edge must stay + within line-rate CPU budgets on shared instances. +- **Communication efficiency.** Cross-AZ egress is billed per byte; transmission + must be minimized. +- **Continuous & accurate queries.** Monitoring queries are *long-running* over + the stream, not the database "on-demand" model. We care about the + **freshness/timeliness** gap between when a sample is generated at the source + and when it can be queried accurately at the backend. + +These four axes are exactly the decision variables and constraints of the +optimization below. + +--- + +## 2. Positioning: what each prior line gives, and what it lacks + +All four lines are instances of "keep the local drift inside a **safe zone**, +communicate on violation." They differ in the *shape* of the safe zone and in +*what* they bound. (Attribution matters; see §10.) + +| Line | Bounds | Safe-zone shape | Continuous query? | Gap | +|---|---|---|---|---| +| **Error-bounded sketching** (Count-Sketch [Charikar+02], CMY [Cormode+08]) | one function, per window | — | window granularity only | intra-window staleness | +| **Geometric Monitoring** (Sharfman–Schuster–Keren [SIGMOD'06]; AutoMon [Sivan+ SIGMOD'22]) | one scalar `f(x̄)` | DC-quadric (ball / slab / general) | that **one function** only | not per-cell; not query-general | +| **OctoSketch** [Zhang+ NSDI'24] | **every counter cell** `|ΔC[j]| **Result.** At equal admitted work (`E[rows]=d·p` per item ⇒ same edge CPU), +> per-row sampling keeps the sampling error inside the median's high-probability +> envelope; whole-item sampling leaves it as an irreducible common-mode penalty. +> Per-row is the **strictly better estimator** — this, not the relocation, is the +> reason to push the decision into the SDK. + +**Effective `ε_sa` (composition).** Under per-row independence the sampling +*variance* folds into the **same** `median_r` step as the collision variance — +`Var[X_r] = F₂/w + (1−p)/p·S₂,r` under one median tail — so it composes **in +quadrature** with `ε_sk`: `|f̂(y)−f(y)| ≤ √(ε_sk²+ε_sa²)·‖f‖₂` w.p. `1−δ`, with +`ε_sa = Θ(√((1−p)/(p·w)))` (uniform `p`). This is exactly the random part of +Theorem 1 (§4). Whole-item admission instead leaves a **common-mode** term that +survives the median and adds to `ε_sk` **linearly** — strictly looser at equal +edge cost. + +**Threshold-allocation coupling (row-dependent floor).** The delta gate `T_j` for +cell `j=(r,c)` sits over a *row-`r`-subsampled* counter, so the +"don't-transmit-finer-than-you-sample" floor of §7 is **row-indexed** by that +row's admission rate: +``` +T_j ≥ T_j^floor = √( V_j · (1−p_i)/p_i ) (rate is per-site p_i, admission per-row) +``` +The GOS water-filling is unchanged in form; the ε-budget is split by §7 Layer B +(staleness `ε_st` peeled linearly, then `ε_sk²+ε_sa² = (ε_q−ε_st)²`), and this +floor closes the sampling↔threshold coupling. + +--- + +## 4. Unified error bound + +The backend holds `Ĉ(t)`, perturbed from the ideal `C(t)` by two **random** +sources that the Count-Sketch median absorbs together, plus one **deterministic** +staleness term that adds on top: + +1. **Sketch + sampling (random, one median).** Per row `r`, the cell estimate + carries hash-collision variance `≈ F₂/w` *and* per-row sampling variance + `≈ (1−p_i)/p_i · S₂,r` (§3.2 — the `1/p_i` weight is unbiased; per-row + admission keeps the rows independent). Because both live in the **same** + `median_r` step, they compose in quadrature into one high-probability tail: + `|q̂ − q(f)| ≤ √(ε_sk² + ε_sa²)·‖f‖₂` w.p. `1−δ`, with `ε_sk=Θ(1/√w)`, + `ε_sa=Θ(√((1−p)/(p·w)))`, `δ=2^{−Θ(d)}`. (Sampling rate is per-site `p_i`; + admission is per-row-independent, i.e. `p_{i,r}=p_i` uniform across rows.) +2. **Staleness `{T_j}` (deterministic).** Site `i` withholds cell `j` until its + accumulated change reaches `T_j`, so `|Ĉ[j] − (C+η)[j]| ≤ k T_j =: e_j` **at + all times** — a worst-case bound, *not* a random variable, so it **adds + linearly** (it cannot be RMS-combined with the random tail). + +**Theorem 1 (unified relative error).** For any query `q` and any time `t`, w.p. +`≥ 1 − δ`: + +``` + ┌──── random (one median) ────┐ ┌──── staleness (linear) ────┐ +|q̂(t) − q(f)| ≤ √(ε_sk² + ε_sa²)·‖f‖₂ + k·Σ_j |r_{q,j}|·T_j +``` + +Dividing by `‖f‖₂` gives the **relative** budget: the random part (sketch ⊕ +sampling in quadrature) plus the deterministic staleness, linearly: + +> **√(ε_sk² + ε_sa²) + ε_st ≤ ε_q.** + +For a monitored non-linear `f_m` (`g_m = ∇f_m`, Hessian spectral bound `λ_m` from +AutoMon/ADCD), a Taylor + DC bound gives + +``` +|f̂_m − f_m| ≤ |g_m|ᵀ(ε_sk + ε_sa terms) + k·Σ_j|g_{m,j}|T_j + ½·λ_m·k²‖T‖₂² ≤ ε_m·|f_m|. +``` + +This holds **continuously**, not only at window boundaries — the OctoSketch +"online accuracy at any query time" property, here generalized to arbitrary +queries and to gradient-weighted function monitoring. + +### Freshness + +Cell `j` (activity `V_j = Σ_i V_{ij}`) reaches `T_j` after time `T_j / V_j`, so its +max staleness age is `Δ_j = T_j / V_j`. To guarantee query freshness `Δ*`: +`T_j ≤ V_j Δ*`. (Quiet cells have large `Δ_j` even at small `T_j`, so freshness +binds them → periodic heartbeat.) + +--- + +## 5. Cost models (per unit time) + +``` +Memory_edge = m·G·n·( 1 + 1{delta}[acked snapshot] + 1{aniso}[threshold vec] ) +Comp_sdk = c_rng·(Σ_r p_{i,r})·rate (geometric skip-sampling: O(1)/admit RNG) +Comp_coll = c_h·(Σ_r p_{i,r})·rate (hash + update — only admitted rows) + + c_s·Σ_j V_j/T_j (uploads — thresholds cut this) +Comm = b·Σ_j V_j/T_j (+ broadcast for geometric) + × (1 − ∏_r(1−p_{i,r})) (samples with no admitted row aren't sent) +Cost_coord = m·n·(1 + k·1{geo}) (running merge + per-edge refs) + + c_a·Σ_j V_j/T_j (incremental apply_delta) +``` + +The edge CPU is split across the two runtimes: the **SDK** pays only the +geometric-sampler RNG (`Comp_sdk`, `O(1)` per admit, no hashing), and the **agent +collector** pays the hashing/update (`Comp_coll`) *only for admitted rows* plus +the delta uploads. Lowering `p_{i,r}` cuts SDK RNG, collector hashing, and wire +volume together — one lever, three savings. + +**Structural insight.** `Comm`, the upload part of `Comp_edge`, and the apply part +of `Cost_coord` are all `∝ Σ_j V_j/T_j` → they collapse into one effective weight +`W = w_c·b + w_e·c_s + w_b·c_a`. Hence the *cost weights do not change the optimal +threshold shape* — the accuracy constraint pins `T_j`; the weights instead select +the **structural knobs** (uniform-vs-aniso, delta-vs-full, sampling `p`, per-edge +refs), which is where the memory/compute tradeoffs live. + +--- + +## 6. The optimization problem (P) + +``` +minimize w_m·Memory_edge + w_e·Comp_edge + w_c·Comm + w_b·Cost_coord +over d, w, G, {p_i}, {T_j}, structural flags +subject to + (query, ∀ q ∈ Q) √(ε_sk(w)² + ε_sa({p_i})²) + k·Σ_j|r_{q,j}|T_j/‖f‖ ≤ ε_q + (function, ∀ f_m) k·Σ_j|g_{m,j}|T_j + ½·λ_m·k²‖T‖² ≤ ε_m·|f_m| + (freshness) T_j ≤ V_j·Δ* + (confidence) d ≥ log₂(1/δ) + p_i ∈ (0,1], T_j ≥ 0, w ≥ 1, G ≥ 1 +``` + +`w_b` (coordinator) is the least-weighted axis by requirement. + +--- + +## 7. Solution: hierarchical decomposition + two water-fillings + +**Layer A — outer (small enumeration).** `d = ⌈log₂(1/δ)⌉`; choose `w` +(`ε_sk = c/√w`) and `G` (grouping) to trade the memory term `w_m·G·n` against the +accuracy the sketch must supply. + +**Layer B — budget split (staleness peeled linearly first).** Because staleness +is deterministic it comes off the top of `ε_q` **linearly** (Theorem 1), *then* +the remaining random budget is split in **quadrature**: + +1. choose `ε_st ∈ [0, ε_q − ε_sk]` — the edge-CPU↔communication knob (larger + `ε_st` ⇒ looser thresholds ⇒ less comm, but a smaller random budget ⇒ tighter + sampling ⇒ more CPU); pick it by the 1-D convex tradeoff of `w_e·Comp` vs + `w_c·Comm` (unique — both monotone); +2. the random budget after the linear peel is `ε_rand = ε_q − ε_st`; +3. split it in quadrature: `ε_sa² = ε_rand² − ε_sk²` (sketch ⊕ sampling), which + requires `ε_sk ≤ ε_rand`. + +So the composition is `√(ε_sk² + ε_sa²) + ε_st = ε_q`, matching §4 — **not** a +three-way quadrature. The code's `split_budget(ε_q, ε_sk, w_edge, w_comm)` +implements exactly this linear peel (`ε_st = t·(ε_q−ε_sk)`, +`ε_sa = √((ε_q−ε_st)²−ε_sk²)`, `t = w_comm/(w_edge+w_comm)`). + +**Layer C — two water-fillings (same KKT tool, two variables).** + +- **Sampling** (per-site). The per-key KKT water-filling + `p_i ∝ √(f_i/rate_i)` (binding `Σ_i f_i(1−p_i)/p_i ≤ V_sa(ε_sa)`) is the + general form, but it has been **retired for sketch sampling**: a sketch + point/L2 estimate's error is bounded by the sketch *norm*, not a single key's + `f(x)`, so the accuracy a per-edge `p_i` buys is only "keep this edge's L2 + contribution within ε." The **implemented** allocation is therefore the + whole-sketch ε-floor + ``` + p_i = 1 / (1 + ε²·rate_i), clamped to (0,1] + ``` + (`data_plane monitor::coordinator::allocate_p` → `epsilon_sample_floor`; see + derivations §5). The `√(f/rate)` split remains valid only when a key is + exact-counted *outside* the sketch — where sampling that one counter is + pointless anyway. +- **Thresholds** (per-cell; GOS). Effective weight `W` (§5), per-cell price + `c_j = k|g_j|` (function) or `k|r_{q,j}|` (query — take the binding constraint), + budget `B` (relative: `B = ε_m‖Ĉ‖²` for F₂): + + ``` + ┌ water-filling ┐ ┌──── clamps ────┐ + T_j = clamp( (B/Σ_ℓ√(c_ℓ V_ℓ))·√(V_j/c_j), T_j^floor, min(T_q, V_j·Δ*) ) + ``` + + with **sampling-coupling floor** `T_j^floor = √( V_j(1−p)/p )` ("don't transmit + finer than you sample"), **query cap** `T_q = ε_q‖Ĉ‖/(k·s_q)`, **freshness cap** + `V_j·Δ*`. Clamped cells release budget → box water-filling redistributes (a few + iterations). + +**Reading.** `T_j ∝ √(V_j/|g_j|)`: high-activity cells get larger thresholds (don't +chase high-frequency noise); cells the monitored function is sensitive to +(`|g_j|` large) get smaller thresholds (report early); everything capped by the +universal query cap and freshness. + +**Layer D — when: insert-time, no periodic scan.** Layers A–C fix *what* +threshold each cell gets; this layer is *when* a crossing is actually +detected and shipped. One mechanism for every family (Sum, CMS, CountSketch, +DDSketch, KLL, HLL), isotropic case: check at insert time whether the +accumulated-since-last-sync delta crosses the cell's `T_j` — or, for +KLL/HLL, the family's whole-sketch trigger; every family's exact formula is +in `sampling-cdm-gos-derivations.md` §8. If it crosses, that cell's (or +scalar's) delta needs to reach the backend. The sole purpose is data +synchronization — keeping the backend's reconstructed state accurate. +Alerting and any other query-time decision is made entirely at the backend +against that synced state; the edge does not make alerting decisions +itself (see Retirements in §11 for what this replaces in the current code). + +### Wake-on-demand flush + +The existing OTLP export chain (SDK `PeriodicReader` → collector processor +→ exporter) is unchanged; only *when a flush cycle runs* differs from a +purely timer-driven cadence — it is timer-**or**-woken: + +```go +for { + select { + case <-ticker.C: // slow fallback cadence, in case a wake is ever missed + flush() + case <-wakeCh: // fired the instant something crosses threshold + flush() + } +} +``` + +Insert path, non-blocking (never waits for the flush loop): + +```go +select { +case wakeCh <- struct{}{}: +default: // a wake is already pending; nothing to add +} +``` + +`flush()` keeps using the existing `SnapshotCache`/`ComputeDeltaAgainst` +machinery (full-frame fallback on cold start, the "never emit a delta +larger than a full frame" clamp) — only the trigger changed. A per-cell +`dirtySet`, populated as cells cross threshold, replaces the old periodic +whole-sketch divergence pre-check: an empty `dirtySet` at flush time IS +"nothing was worth sending," computed once per crossing instead of by a +periodic `O(dw)` full-matrix scan. + +### Per-family detection unit and reset semantics + +| Family | Detection unit | Reset on send? | Notes | +|---|---|---|---| +| CountSketch (isotropic) | matrix cell | zero it | `normSqAll` tracked incrementally (`+= 2·old·Δ+Δ²`), O(1) | +| CountMinSketch | matrix cell | zero it | same mechanism, $L_1$-scale threshold (derivations §8.2) | +| DDSketch | bucket count | zero it | bucket count `B` tracked incrementally too (+1 on genuinely new bucket), no config constant needed (derivations §8.4) | +| Sum | scalar | zero it (subtract reported amount) | degenerate 1-cell case | +| KLL | whole sketch (no per-cell structure) | full `Reset()` | trigger is `Count() >= εN`, not a per-cell check; already the existing disjoint-segment mechanism, just re-triggered by count instead of a timer | +| HLL | register | **never** — MAX-merge is idempotent, only clear a dirty flag | trigger is $\lvert 2^{C'}-2^{C}\rvert \ge 2^{\tau}$ on the linearized value, not raw register value (derivations §8.7) | + +Backend reconstruction is unchanged for the additive families (Sum/CMS/CS/ +DDSketch): summing every fragment ever received for a cell — regardless of +how many times or when it was individually reset — telescopes to the true +cumulative value (`v_1+v_2+...+v_n+v_{residual}` = true total). Resets can +therefore happen asynchronously, at different times per cell, without +breaking correctness — it only affects *when* transmission happens, never +*what* the backend eventually reconstructs. + +### Cold start is a feature, not a bug + +Every family's threshold scales with an accumulated quantity (`‖Ĉ‖`, `N`, +`R`) that starts near zero at window start, so the very first few inserts +cross threshold almost immediately. This is intentional: it gets the +backend a usable initial estimate as fast as possible, rather than waiting +for data to accumulate before syncing anything. No floor/minimum-threshold +mechanism is needed to suppress this. + +### Closed forms + +- **F₂, isotropic** (`g_j = 2Ĉ_j`, `λ=2`, uniform, relative): `T = ε‖Ĉ‖ / (2k√(dw))` + — adaptive: scales with the current norm. +- **F₂ threshold-alert version** (monitor `F₂ ≥ τ`, one-sided band): `T = (1/k)√((1−ε)τ/w)`. + The whole-sketch `F₂` readout here is the **mean-of-rows** estimator + `F̂₂ = ‖C‖²/d` (each row's `‖C_r‖²` is an unbiased `F₂` estimate; averaging the + `d` independent rows reduces its variance by `1/d`) — *not* the median-of-rows + point-query estimator of §3.2. This is deliberate: the geometric safe-zone is a + **ball** `‖C‖ ≤ √(d(1−ε)τ) ⇔ ‖C‖²/d ≤ (1−ε)τ`, so edge silence and coordinator + alert test the identical functional (`all-sites-safe ⟺ F̂₂ < (1−ε)τ`). The two + estimators serve different readouts — median for individual-key location + (robust tail), mean for the aggregate energy the ball bounds (variance + reduction) — and must not be conflated. +- **Linear `f` (sum/count/point)**: `λ = 0` → the curvature term vanishes → box + degenerates to a **slab** = the classic CMY slack countdown. + +### Unification of counter-based and F₂ + +Both are `f(x̄)` vs a band `[L,U]`, with the safe zone from the DC bound; the only +difference is the Hessian eigenvalue: `λ=0` → slab (counter-based), `λ=const·I` → +ball (F₂), general → ADCD quadric. One monitor, one edge check +`isLocallySafe(Δ, x₀, ∇f, λ, L, U)`; keep the scalar-countdown fast-path for the +linear case (no reference-vector broadcast needed). + +### Threshold band vs tracking band + +- **Threshold/alert**: one-sided band `[−∞, τ]` → fire on crossing. +- **Continuous ε-query**: moving band `[f(x₀)−ε, f(x₀)+ε]` → backend answers + `f(x̄)=f(x₀)±ε` at all times. Same monitor, different band — this is how + threshold monitoring and continuous approximate querying unify (Cormode– + Garofalakis continuous querying = GM with a tracking band). + +--- + +## 8. Optimality vs the Woodruff–Zhang lower bound + +**Rate vs total.** `Σ_j V_j/T_j` is an upload *rate*; the WZ `Θ̃(k/ε²)` is the +*total* communication to maintain one continuous `(1±ε)` `F₂` estimate. Compare +them over a fixed horizon of bounded total change: with `w ∝ 1/ε²` (the necessary +sketch width) each sketch is `Θ(1/ε²)` and `k` sites must each be represented, so +the total is `Θ̃(k/ε²)` — **matching the WZ STOC'12 tight lower bound** (bits vs +words absorbed in the `Θ̃`). The measured normalization in +[`gos-eval-results.md`](gos-eval-results.md) §3 uses the "one-round" unit `k·S` +for exactly this comparison. Consequences: + +- The `1/ε²` and the linear-in-`k` are **fundamental**; no protocol (GM, AutoMon, + OctoSketch, GOS) beats `k/ε²` adversarially. GOS's savings are **data-dependent** + (small `V_j/‖Ĉ‖` on stable streams). +- Report GOS's measured bytes as a **fraction of `k/ε²`** — a stronger baseline + than comparing to naive centralization. +- The relative-error caveat: relative bounds require the norm bounded below + (WZ tightness / OctoSketch `L1 > ε⁻¹k'τ`); relative error on a near-zero signal + is fundamentally not cheap. + +--- + +## 9. How it meets the four requirements + +| Requirement | Mechanism | +|---|---| +| **Memory efficiency** | `(w,G)` + `w_m`: grouping `G` avoids one sketch per series; `w=Θ(1/ε²)` is the WZ-minimum; delta/aniso flags dropped under high `w_m` (no snapshot / threshold vector) | +| **Computation efficiency** | `w_e`: sampling `p_i` (fewer updates) + threshold size (fewer uploads); both fold into one effective weight | +| **Communication efficiency** | `w_c`: per-cell water-filling thresholds + geometric silence; `Θ̃(k/ε²)` worst case, far less on stable data | +| **Continuous, accurate, fresh** | Theorem 1 bounds every query at **any** `t`, **relative**; freshness cap `T_j ≤ V_jΔ*` bounds staleness age; whole sketch queryable (OctoSketch), monitored `f_m` tighter (GM/AutoMon) | + +--- + +## 10. What is adopted vs contributed (attribution) + +- **Water-filling** — classic (information theory / convex optimization; optimal + power allocation across parallel channels). Already used in ASAP for sampling + (`p_i ∝ √(f_i/rate_i)`). *Adopted, not contributed.* +- **Per-cell change transmission with a threshold** — OctoSketch [Zhang+ NSDI'24]. + *Adopted.* +- **Function safe zone via DC decomposition of the Hessian (ADCD), gradient/ + Hessian bounds** — AutoMon [Sivan+ SIGMOD'22]; Geometric Monitoring [Sharfman+ + SIGMOD'06]. *Adopted.* +- **Coordinated sampling + geometric skip-sampling** — NitroSketch [Liu+ + SIGCOMM'19]; ASAP's own `AllocateSampleRates` (rate allocation) and + `sketchlib-go/common.GeometricSampler` (the `O(1)`-amortized skip sampler). + *Adopted.* +- **Lower bound `Θ̃(k/ε²)`** — Woodruff–Zhang [STOC'12]. *Yardstick.* +- **GOS (this doc)** — casts *per-cell threshold allocation* as water-filling with + **gradient-derived weights** (GM/AutoMon) and a **per-cell query cap** + (OctoSketch), coupled to **per-site sampling** through a shared ε-budget and a + granularity floor `T_j ≳ √(V_j(1−p)/p)`, inside one **tunable + memory/compute/communication objective**. *The synthesis is the contribution; + the optimization tools are off-the-shelf.* + +--- + +## 11. Implementation notes (controller synthesizes, edge executes) + +Everything expensive is a **controller (backend) decision**; the edge only +executes a fixed per-cell comparison. + +- **Controller** (offline, per registered metric/query): runs ADCD (AD → Hessian + eigenvalue bounds → `∇f, λ`), estimates `{V_j}` from the workload, solves (P)'s + layers A–D, emits `(d, w, G, {p_i}, scalar GOS knobs, flags)` via OpAMP. This + slots into the existing controller multi-objective + (`controller-optimization-problem.md` SP-6: + `min w_bw·bw + w_cpu·cpu + w_mem·mem + …`) — GOS thresholds are new decision + variables there. **Note:** the controller ships *scalars* + (`ε_delta`, sites), **not** a full per-cell vector `{T_j}`; the isotropic + case (§7 Layer D, all 6 families) reconstructs its single scalar `T` from + those plus live sketch state, so no vector ever crosses the wire. The + anisotropic per-cell `{T_j}` water-filling described here is + CountSketch-only and, per Open problems below, not currently implemented. +- **Edge**: maintain sketch state; per insert, compare against `{T_j}` + (recomputed from the pushed scalars + local `{V_j}` — §7 Layer D) and mark + crossed cells dirty; a wake fires a flush of the current dirty set as a + sparse delta; run one generic `isLocallySafe` for monitored functions. No + AD, no water-filling solve at the edge for the isotropic case — only the + closed-form threshold evaluation, checked continuously rather than on a + periodic scan. The anisotropic CountSketch water-filling solve is the one + remaining case that still needs a periodic `O(dw)` re-solve and a separate + acked-snapshot copy (see Open problems). +- **Backend**: `apply_delta` into a running merge (`O(#delta cells)`), keeping the + global sketch continuously queryable within the Theorem-1 envelope, surfaced in + the `accuracy: ε=…` response annotation. + +**Retirements.** The insert-time model (§7 Layer D) replaces the following +in the current code — tracked here since it isn't deleted yet: + +- **Gate 1** (`subWindowShouldEmit` / `L2DivergenceSinceEmit` / `ackedCells`): + the old periodic whole-sketch divergence pre-check, run once per + `SubWindowInterval` tick regardless of whether anything had actually + crossed threshold. An empty per-cell `dirtySet` at flush time already + answers "nothing to send," computed once per crossing instead of by a + periodic `O(dw)` scan — dead code under Layer D, not kept as a fallback. +- **CMS's local point-query read** (`ThresholdConfig.Functional: cms_point`): + once cells reset in place at insert time, a `min`-based local read is + corrupted by any single recently-reset row. All point/alert reads move to + the backend's reconstructed copy. +- **Discipline B's alerting path** (`monitor.Engine.Observe` → + `sendReportLocked` over `monitor/grpcclient`): Sum becomes just another + family running the Layer D insert-time check, synced over the normal + `SketchEnvelope` pipeline like everything else. `Engine.Observe` today + conflates two things in one function: `obsCount++` (rate tracking, + feeding the coordinator's `SampleP` grant negotiation) and the + `value-baseline>=slack` alerting check (which calls `sendReportLocked`). + Only the *alerting* half retires; `obsCount`/rate-tracking must be + preserved (it feeds a genuinely separate concern — sampling-rate + negotiation, not data sync) — the two halves need to be split apart, not + deleted together. + +**Ties to existing code:** +- `ASAPQuery-backend/control_plane/src/epsilon_alloc.rs` — the ε-budget split + (linear staleness peel, then quadrature split: `√(ε_sk² + ε_sa²) + ε_st = ε_q`, + §7 Layer B — not a three-way quadrature). +- `ASAPQuery-backend/data_plane/src/monitor/sampling_alloc.rs` + (`epsilon_sample_floor`) — the live whole-sketch sampling floor. +- `asap_sketchlib` `CountSketchDelta` + `compute_delta`/`apply_delta` + (byte-parity Go/Rust) — the sparse per-cell delta wire format. + +--- + +## 12. Open problems / next steps + +1. **Anisotropic CountSketch's `Activity_j`** needs redefinition. The old + `Activity_j=|current-prev|` assumed a single, uniformly-timed `prev` + snapshot; under per-cell async reset (§7 Layer D), different cells' + "since last touch" windows are no longer comparable, and naively diffing + against any snapshot double-counts/under-counts around individual cell + resets. A per-cell EMA of `|Δ|` (`activityRate[r][c] = + decay·activityRate[r][c] + (1-decay)·|Δ|`, updated every insert) is the + leading candidate — cheap, reset-timing-independent — but it replaces the + derivation's exact `Activity_j=V_j` with a heuristic, and whether the §7 + closed-form water-filling solution still carries the same error guarantee + under that substitution has not been checked. The anisotropic + water-filling solve itself also still requires a periodic `O(dw)` pass + (unlike every other family) — per-cell detection at insert time only + avoids the "decode a serialized `prev`" cost, not the joint solve. +2. **Anisotropic delta broadcast** — *partially done.* The **sparse-cell** + encoding of `ΔC_ref` on the coordinator→edge path is implemented and measured + (`CRefUpdate::Delta`; removes the `O(k)` broadcast amplification — see + gos-eval-results.md §2). Still **open:** the broadcast gate is currently + isotropic (ships every changed cell, `Δ ≠ 0`); giving it *anisotropic per-cell + thresholds* (the §7C water-filling, as already done on the edge→coordinator + upload path via `ComputeDeltaPerCell`) is the remaining work. +3. **DDSketch's unbounded contiguous bucket-array growth** on outlier values + is a real memory-safety gap, independent of Layer D — tracked as + sketchlib-go#72. The dynamically-tracked `B` used in the threshold formula + (derivations §8.4) does not require fixing this; it is a separate, likely + higher-priority issue. +4. **HLL's small-cardinality regime**: the register-change adapter's accuracy + proof (OctoSketch's Appendix B) is stated for "sufficiently large" + cardinality; behavior when most registers are still at 0 (early in a + window) has not been separately verified. A candidate mitigation (always + send a register's first-ever nonzero write unconditionally) is proposed + but unverified. +5. **Relative-error under small norm** — heartbeat / additive floor when `‖Ĉ‖` is + small (WZ / OctoSketch fundamental limit). +6. **Verified eigenvalue bounds** — AutoMon's numerical `λ` may miss the true + extreme → reserve an `ε_eig` slice of the budget or use interval bounds. +7. **Empirical validation** — measure achieved communication as a fraction of the + WZ `k/ε²`, sweep `(w_m, w_e, w_c)` to trace the Pareto surface. + +--- + +### References (attribution) + +- G. Cormode, S. Muthukrishnan, K. Yi. *Algorithms for Distributed Functional Monitoring.* SODA 2008 / ACM TALG 2011. +- M. Charikar, K. Chen, M. Farach-Colton. *Finding Frequent Items in Data Streams* (Count-Sketch). ICALP 2002. +- I. Sharfman, A. Schuster, D. Keren. *A Geometric Approach to Monitoring Threshold Functions over Distributed Data Streams.* SIGMOD 2006. +- H. Sivan, M. Gabel, A. Schuster. *AutoMon: Automatic Distributed Monitoring for Arbitrary Multivariate Functions.* SIGMOD 2022. +- Y. Zhang, P. Chen, Z. Liu. *OctoSketch: Enabling Real-Time, Continuous Network Monitoring over Multiple Cores.* NSDI 2024. +- D. Woodruff, Q. Zhang. *Tight Bounds for Distributed Functional Monitoring.* STOC 2012 (arXiv:1112.5153). +- Z. Liu, R. Ben-Basat, G. Einziger, Y. Kassner, V. Braverman, R. Friedman, + V. Sekar. *NitroSketch: Robust and General Sketch-Based Monitoring in Software + Switches.* SIGCOMM 2019. (Geometric skip-sampling: one RNG draw per admitted + update — `O(1)` amortized, "always line rate" — with inverse-probability + weighting; the SDK-side row-admission sampler here is this scheme.) diff --git a/docs/sampling-cdm-gos-derivations.md b/docs/sampling-cdm-gos-derivations.md new file mode 100644 index 00000000..400ca7f8 --- /dev/null +++ b/docs/sampling-cdm-gos-derivations.md @@ -0,0 +1,1837 @@ +# Sampling + Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) derivations for windowed sketch telemetry + +> Paper-facing derivation note. This document ties together +> [distributed-nitrosketch-coordinated-sampling.md](distributed-nitrosketch-coordinated-sampling.md), +> [continuous-monitoring-aggregation-taxonomy.md](continuous-monitoring-aggregation-taxonomy.md), +> [continuous-monitoring-tumbling-cost-analysis.md](continuous-monitoring-tumbling-cost-analysis.md), +> and [design-gos-unified-edge-telemetry.md](design-gos-unified-edge-telemetry.md). +> +> Scope: fixed tumbling-window epochs; open-window freshness inside one epoch; +> additive linear sketch states as the fully proved case; family-specific +> extensions for quantiles and cardinality sketches. +> +> Acronyms: **CDM** = **Continuous Distributed Monitoring**; **GOS** = +> **Geometric-OctoSketch**. + +## Paper positioning: bottleneck -> solution map + +The system bottleneck addressed by these derivations is +**resolution-coupled central ingestion**. In conventional observability +pipelines, higher temporal resolution, higher label cardinality, and lower +freshness latency all require more raw samples to traverse the central path: + +```text +collector/exporter -> remote write / queue / WAL -> backend ingest/index/storage +-> query scan +``` + +ASAPCollector changes the unit of work from raw-sample ingestion to +query-bounded sketch-state synchronization. Edge collectors still absorb the +high-resolution stream, but the backend receives bounded sketch summaries, +error-triggered deltas, and optional cold raw fallback instead of every raw +sample on the warm path. + +The problem-solution pairs are: + +| Existing-system bottleneck | Why it matters | ASAPCollector mechanism | Error/control consequence | +| --- | --- | --- | --- | +| Central ingest bottleneck | Cost scales with $\mathrm{series\_cardinality} \times \mathrm{sample\_frequency}$; raising resolution pushes more samples through write queues, indexing, storage, and query scan. | Maintain mergeable sketches at edge collectors and transmit sketch state/deltas. | Backend warm-path load scales with summary size and threshold crossings, not directly with raw sample rate. | +| Freshness vs cost bottleneck | Shorter scrape/export intervals improve open-window freshness but increase CPU, network, and backend ingest pressure; longer intervals reduce cost but make queries and alerts stale. | Use CDM/GOS residual thresholds $T_j$ for error-triggered synchronization. | Freshness becomes a bounded staleness term $\mathrm{Err}_q^{cdm}$ rather than an implicit consequence of a fixed reporting interval. | +| Query scan / post-ingest downsampling bottleneck | TSDB compression and downsampling help after raw data has already been ingested, and long-range queries still depend on stored sample layout. | Build query-ready sketches before/during ingestion. | Query cost is paid against sketch summaries, while the cold raw path remains available for unsupported or forensic queries. | +| Fixed-statistics / early-binding bottleneck | Histograms, summaries, and pre-aggregations commit early to bucket layouts, quantiles, windows, or rollups; changing the query later may be impossible or inaccurate. | Use a sketch-family adapter per query class: Sum/CMS/CountSketch for additive readouts, DDSketch/KLL for rank queries, HLL for cardinality. | The controller applies the correct sampling and staleness model for each sketch family instead of using one proof for all summaries. | +| Control-plane bottleneck | Existing cost controls such as dropping labels, filtering metrics, increasing intervals, or coarse downsampling often do not expose a query-level error budget. | Jointly tune sketch size, sampling probabilities $p_i$, and GOS thresholds $T_j$. | The system can allocate a query error budget across $\mathrm{Err}_q^{sk} + \mathrm{Err}_q^{sa} + \mathrm{Err}_q^{cdm}$. | +| Theory-to-system bottleneck | Prior sketching, approximate query processing, and CDM work each solve part of the problem, but not the end-to-end telemetry ingestion control loop. | Combine update sampling, mergeable sketches, and CDM/GOS synchronization inside the collector/backend architecture. | The paper claim is a system-level accuracy envelope, not only a faster sketch update or a lower communication protocol in isolation. | + +Compared with existing work, the paper's positioning is: + +| Existing work / system family | Solves | Leaves open | ASAPCollector angle | +| --- | --- | --- | --- | +| Prometheus / OpenTelemetry / remote-write pipelines | Standard metric collection, export, and backend ingestion. | Higher resolution and cardinality still increase central ingest work. | Move high-resolution absorption to edge sketches and synchronize bounded state. | +| Native histograms / exponential histograms / DDSketch-style distribution metrics | Compact, mergeable distribution summaries. | Mainly distribution-specific; does not give a general control plane for sampling plus staleness across sketch families. | Treat DDSketch as one adapter in a broader sketch taxonomy. | +| TSDB compression / Thanos-style downsampling | Long-range query acceleration and storage-layout optimization. | Mostly after-ingest optimization; raw samples still enter the central path first. | Summarize before or during ingestion, then query the warm sketch path. | +| BlinkDB / VerdictDB-style AQP | Approximate analytics over stored data with statistical error. | Data is already in the warehouse; not a continuous telemetry freshness and ingestion-control problem. | Provide telemetry-native edge ingestion, open-window freshness, and sketch-state synchronization. | +| NitroSketch | Sampling sketch updates to reduce sketch CPU. | Focuses on update work for sketches, not multi-sketch observability queries, backend freshness, or GOS/CDM synchronization. | Reuse the sampling idea but compose it with query-level variance budgets and delta suppression. | +| Continuous Distributed Monitoring / distributed functional monitoring | Communication-efficient tracking of distributed functions. | Mostly a theoretical monitoring model, not an observability pipeline with sketch-family adapters and cold fallback. | Use CDM as the proof model for bounded sketch residual synchronization. | +| Learned reconstruction systems such as Zoom2Net | Infer fine-grained telemetry from coarse measurements. | Error semantics depend on reconstruction/model behavior rather than preserving query-sufficient sketch state. | Maintain sketch statistics with explicit sketch, sampling, and staleness error terms. | + +The intended claim boundary is narrow: + +- ASAPCollector does not replace raw telemetry for every task; it provides a + warm approximate path plus cold raw fallback. +- ASAPCollector does not support arbitrary PromQL under one theorem; it supports + sketchable/decomposable query classes with family-specific adapters. +- The fully proved theorem target is fixed-query, fixed-time, additive-state + telemetry with independent Horvitz-Thompson update sampling and an + update-synchronous or explicitly overshoot-bounded residual protocol. +- The core claim is that the collector can reduce update work and network + traffic while keeping a query-level accuracy envelope. + +## 1. Scope and aggregation modes + +Continuous monitoring in ASAP is not one protocol. The aggregation axis decides +which proof model applies. + +| Mode | Query shape | Protocol | Coordinator? | Main guarantee | +| --- | --- | --- | --- | --- | +| A | per-series $\times$ window | local $\epsilon$-gated delta emission | no | open-window freshness for one local series/group | +| B | series $\times$ timestamp | mergeable sketch fan-in | no | sealed/instant merge; no monitoring suppression | +| C | series $\times$ window, fused spatial+temporal | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) over edge collectors | yes | global open-window tracking/alerting across $k$ sites | + +This note proves the composition used by modes A and C for the additive-state +case: + +1. sketch approximation, +2. update sampling, +3. stale backend state due to suppressed deltas. + +For sealed tumbling windows, only the first two sources matter. The merge of +mergeable summaries does not compound the base sketch error. + +## 2. Notation + +One fixed tumbling window is the universe of discourse. + +| Symbol | Meaning | +| --- | --- | +| $k$ | number of stable edge collectors/sites | +| $i$ | site index, $i \in \{1,\ldots,k\}$ | +| $u$ | update/sample index | +| $x_u$ | raw item/value | +| $f_i$ | local frequency vector at site $i$ | +| $f=\sum_i f_i$ | global frequency vector | +| $S_i$ | exact local sketch state for site $i$ | +| $S=\sum_i S_i$ | exact global additive sketch state | +| $\widehat S_i$ | sampled local sketch state | +| $\widehat S=\sum_i \widehat S_i$ | sampled global sketch state | +| $\widetilde S$ | backend's stale copy of $\widehat S$ | +| $j$ | sketch cell/bucket index | +| $a_j$ | linear readout coefficient for query $q(S)=\langle a,S\rangle$ | +| $p_i$ | sampling probability at site $i$ | +| $p_{i,r}$ | SDK/source-side sampling probability for row/counter-array $r$ at site/source $i$ | +| $C_{i,r}$ | candidate counter-update rate or source-to-agent cost weight for site/source $i$, row $r$ | +| $F_{i,r}$ | row-level protected mass or query sensitivity for site/source $i$, row $r$ | +| $T_j$ | cell/bucket delta threshold | +| $V_j$ | cell/bucket activity rate or expected change mass | +| $B$ | staleness budget for a query/function | +| $\epsilon_{sk}$ | base sketch error budget | +| $\epsilon_{sa}$ | sampling error budget | +| $\epsilon_{cdm}$ | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) staleness budget | +| $\delta$ | failure probability | + +For additive sketches, an update $u$ touches one or more sketch cells. Let +$j(u)$ be a touched cell in a one-cell-per-update sketch such as a DDSketch +bucket, or let $j=(r,c)$ be a row/cell pair for CMS/CountSketch. + +## 3. Generic error decomposition + +For a fixed query $q$ at a fixed time inside a window: + +```text +true answer q(f) +exact sketch q(S) +sampled sketch q(\widehat S) +backend answer q(\widetilde S) +``` + +The triangle inequality gives + +```math +|q(\widetilde S)-q(f)| +\le +|q(S)-q(f)| ++ |q(\widehat S)-q(S)| ++ |q(\widetilde S)-q(\widehat S)|. +``` + +We name the three terms: + +```math +\mathrm{Err}_{q} +\le +\mathrm{Err}^{sk}_{q} ++ \mathrm{Err}^{sa}_{q} ++ \mathrm{Err}^{cdm}_{q}. +``` + +For a sealed tumbling window, $\widetilde S=\widehat S$ after boundary emission, +so + +```math +\mathrm{Err}_{q,sealed} +\le +\mathrm{Err}^{sk}_{q} ++ \mathrm{Err}^{sa}_{q}. +``` + +For an open window, Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch +(GOS) controls the third term. + +This statement is pointwise: one fixed query and one fixed time. To claim a +finite workload $Q$ and $M$ possible query times, replace $\delta$ by +$\delta/(|Q|M)$ in each concentration bound and union bound. + +## 4. Additive-state sampling model + +### 4.1 Horvitz-Thompson update sampling + +In the SDK-side Nitro-style deployment, the sampled unit is not a raw +observation/packet. A raw measurement is first mapped to the sketch updates it +would perform, for example one candidate counter update per CountSketch/CMS row. +The SDK samples those candidate counter-array updates and sends only admitted, +inverse-probability weighted counter updates to the edge collector. The edge +collector then merges weighted sketch updates; it does not need the dropped raw +measurement. + +Thus, in this section, $u$ should be read as the sampled update unit. For +edge-admission baselines, $u$ may be a raw update. For the preferred +Nitro-style design, $u=(\text{raw item},\text{row/counter-array})$ is one +candidate sketch counter update. + +For sampled update unit $u$, site/source $i(u)$ admits it with probability +$p_{i(u)}$: + +```math +Z_u \sim \mathrm{Bernoulli}(p_{i(u)}). +``` + +Here $Z_u$ is the admission indicator: + +```math +Z_u = +\begin{cases} +1, & \text{if update }u\text{ is admitted and applied to the sketch},\\ +0, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +The inverse-probability weighted contribution of update $u$ is + +```math +Y_u = Z_u/p_{i(u)}. +``` + +Equivalently, + +```math +Y_u = +\begin{cases} +1/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +0, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +Thus $Y_u$ is the effective update weight written into an additive sketch +counter/cell. Without sampling, the update would contribute weight $1$; with +sampling, it contributes weight $1/p_{i(u)}$ only on admitted updates. + +Then + +```math +\mathbb{E}[Y_u]=1, +\qquad +\mathrm{Var}(Y_u)=\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +For a linear readout $q(S)=\langle a,S\rangle$, define $g_u$ as the contribution of update +$u$ to this readout if it were not sampled. For a one-bucket counter, +$g_u=a_{j(u)}$. For signed sketches, $g_u$ also includes the row sign. + +The sampled readout error is + +```math +X_q +=q(\widehat S)-q(S) +=\sum_u (Y_u-1)g_u. +``` + +It is unbiased: + +```math +\mathbb{E}[X_q]=0. +``` + +The variance follows from the variance of one update contribution. First, + +```math +\mathbb{E}[Y_u^2] += +p_{i(u)}\cdot \frac{1}{p_{i(u)}^2} += +\frac{1}{p_{i(u)}}. +``` + +Since $\mathbb{E}[Y_u]=1$, + +```math +\mathrm{Var}(Y_u) += +\mathbb{E}[Y_u^2]-\mathbb{E}[Y_u]^2 += +\frac{1}{p_{i(u)}}-1 += +\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +Subtracting a constant does not change variance, so + +```math +\mathrm{Var}(Y_u-1)=\mathrm{Var}(Y_u). +``` + +Multiplying by the deterministic readout contribution $g_u$ scales variance by +$g_u^2$: + +```math +\mathrm{Var}((Y_u-1)g_u) += +g_u^2\mathrm{Var}(Y_u) += +g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +If the sampling randomness is independent across updates, i.e. the admission +indicators $Z_u$ are independent, + +```math +\mathrm{Var}(X_q) += +\sum_u g_u^2 \frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +This assumption is only about the sampling coin flips, not about the input data +streams. The metric values or workloads at different sites may be correlated. +The variance identity needs the random admission decisions to be independent, or +at least uncorrelated. The high-probability Bernstein bound in Section 4.2 uses +the stronger independent-sampling assumption. + +This equation is the sampling-error budget for query $q$. The random variable +$X_q=q(\widehat S)-q(S)$ is the difference between answering $q$ from the sampled +sketch and answering $q$ from the same sketch without update sampling. Each term +in the sum says how much update $u$ contributes to that sampling error: + +- $g_u^2$ is the squared sensitivity of query $q$ to update $u$; +- $(1-p_{i(u)})/p_{i(u)}$ is the noise introduced by sampling at the site that + produced update $u$. + +Thus lowering $p_i$ saves update work at site $i$, but increases the variance of +queries whose relevant updates come from that site. This is the quantity the +controller constrains when it chooses sampling probabilities. + +The independence assumption is what removes covariance terms: + +```math +\mathrm{Var}\left(\sum_u A_u\right) += +\sum_u \mathrm{Var}(A_u) ++2\sum_{u0}\quad & \sum_j \frac{V_j}{T_j} \\ +\text{s.t.}\quad & \sum_j c_jT_j \le B , +\end{aligned} +``` + +where + +```math +c_j = k|a_j| +``` + +for a linear query. For a differentiable monitored functional, use + +```math +c_j = k|g_j|, +\qquad +g_j=\partial F/\partial S[j], +``` + +plus a separate curvature budget if the functional is nonlinear. + +The Lagrangian is + +```math +\mathcal{L} += +\sum_j \frac{V_j}{T_j} ++\lambda\left(\sum_j c_jT_j-B\right). +``` + +The first-order condition is + +```math +\frac{\partial \mathcal{L}}{\partial T_j} += +-\frac{V_j}{T_j^2}+\lambda c_j=0. +``` + +Thus + +```math +T_j=\sqrt{\frac{V_j}{\lambda c_j}}. +``` + +Solving for $\lambda$ from the active budget constraint: + +```math +\sum_j c_j\sqrt{\frac{V_j}{\lambda c_j}} +=B, +``` + +so + +```math +\frac{1}{\sqrt{\lambda}} += +\frac{B}{\sum_\ell \sqrt{c_\ell V_\ell}}. +``` + +The closed form is + +```math +\boxed{ +T_j += +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell \sqrt{c_\ell V_\ell}} +}. +``` + +With floors and caps: + +```math +T_j += +\mathrm{clamp}\left( +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell \sqrt{c_\ell V_\ell}}, +\; +T_j^{floor}, +\; +\min(T_j^{query}, V_j\Delta^*) +\right). +``` + +Clamped cells consume or release budget; recompute the water-filling expression +over the remaining free cells until no cell changes clamp status. + +## 8. Sketch-family instantiations + +### 8.1 Sum / Count + +State: + +```math +S_i=\sum_{u\in i} x_u +``` + +or $S_i=N_i$ for count. + +Base sketch error: + +```math +\mathrm{Err}^{sk}=0 +``` + +up to floating-point arithmetic. + +Sampling error for unit counts under uniform $p$: + +```math +|\widehat N-N| +\le +O\left( +\sqrt{\frac{N\log(1/\delta)}{p}} ++\frac{\log(1/\delta)}{p} +\right). +``` + +CDM staleness with scalar threshold $T$: + +```math +\mathrm{Err}^{cdm}\le kT. +``` + +For a threshold alert $N>\tau$, deterministic no-missed-crossing holds only in +the unsampled monotone setting. With sampling, use a high-probability margin. + +### 8.2 Count-Min Sketch point query + +State: $d \times w$ nonnegative counter matrix. + +Point query for key $x$: + +```math +\widehat f_{CMS}(x) += +\min_r S[r,h_r(x)]. +``` + +Base sketch error: + +```math +f(x) +\le +\widehat f_{CMS}(x) +\le +f(x)+\epsilon_{sk}N +``` + +with probability at least $1-\delta_{sk}$, for the standard choice of width/depth. + +Sampling error applies to every row counter. For row $r$, define + +```math +X_r(x) += +\widehat S[r,h_r(x)]-S[r,h_r(x)]. +``` + +Then $X_r(x)$ is unbiased with variance controlled by the updates landing in +that cell: + +```math +\mathrm{Var}(X_r(x)) += +\sum_{u:h_r(x_u)=h_r(x)} +\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +A conservative high-probability point statement can be made by union-bounding +the row-level sampling events with the usual CMS collision event: + +```math +\widetilde f_{CMS}(x) +\le +f(x) ++\epsilon_{sk}N ++m_{sa}(x,\delta) ++k\max_r T_{r,h_r(x)}. +``` + +Because CMS uses a $\min$, the usual no-underestimate property is not preserved +by two-sided sampling noise. This makes CMS a caveated adapter rather than the +clean first theorem target. For alerting, fire with a confidence margin: + +```math +\widetilde f_{CMS}(x)+m_{sa}(x,\delta)+m_{cdm}(x) +\ge +(1-\epsilon)\tau. +``` + +CMS is therefore useful for upper-bound-oriented point queries, but sampled CMS +alerts should be presented as probabilistic and margin-based. CountSketch, Sum, +and DDSketch range counts are cleaner first theorem targets for sampled GOS. + +#### CDM isotropic threshold ($L_1$, max-composition) + +CMS's CDM staleness composes via the same $\min$-is-1-Lipschitz argument as +CountSketch's median (§8.3), not via the $L_2$ sum-composition GOS uses for +$F_2$: only $d$ cells (one per row) matter for a given query $x$, and the +$\min$ estimator's staleness is bounded by the single worst (most stale) of +those $d$ cells, not their combined mass. So + +```math +\mathrm{Err}^{cdm}_{CMS}(x) +\le +k\max_r T_{r,h_r(x)}. +``` + +Solving for an isotropic (uniform) $T$ so this stays within a chosen fraction +of the base sketch error $\epsilon_{sk}N$ gives, directly (no water-filling, +no $\sqrt{dw}$ factor — the max-composition needs none): + +```math +\boxed{ +T=\frac{\epsilon N}{k} +} +``` + +where $N=\sum_j S[j]$ is the sketch's total (nonnegative) mass — the natural +$L_1$ scale for CMS, as opposed to CountSketch's $L_2$ scale $\|\widehat C\|$. +$d,w$ are fixed at construction (matrix dimensions), so unlike DDSketch's +bucket count (§8.4) there is no "assumed vs. actual" gap to worry about here. + +### 8.3 CountSketch point query + +State: $d \times w$ signed counter matrix. + +Row estimate: + +```math +E_r(x)=s_r(x)S[r,h_r(x)]. +``` + +Point estimate: + +```math +\widehat f_{CS}(x)=\mathrm{median}_{r=1}^d E_r(x). +``` + +Base sketch error: + +```math +|\widehat f_{CS}(x)-f(x)| +\le +\epsilon_{sk}\|f\|_2 +``` + +with probability at least $1-\delta_{sk}$. + +Sampling error is two-sided and unbiased at each row. Let + +```math +X_r(x) += +s_r(x)(\widehat S[r,h_r(x)]-S[r,h_r(x)]). +``` + +Then + +```math +\mathbb{E}[X_r(x)]=0, +``` + +and its variance is the sum of inverse-probability variances of the sampled +updates colliding into the queried cell. A median-of-rows bound follows by +combining row-level concentration with the usual CountSketch row amplification. +This row amplification statement assumes Nitro-style row/counter-array sampling +or otherwise independent row-level sampling noise. If the implementation uses +edge admission sampling, the true-key sampling noise is shared across rows; then +the sampling term must be bounded before the median step and should not be +credited with CountSketch row amplification. + +CDM staleness for row $r$ is + +```math +|s_r(x)(\widetilde S-\widehat S)[r,h_r(x)]| +\le +kT_{r,h_r(x)}. +``` + +Since the median is 1-Lipschitz in the $L_\infty$ perturbation across rows: + +```math +|\mathrm{median}_r(a_r+e_r)-\mathrm{median}_r(a_r)| +\le +\max_r |e_r|, +``` + +the point-query staleness satisfies + +```math +\mathrm{Err}^{cdm}_{CS}(x) +\le +k\max_r T_{r,h_r(x)}. +``` + +Thus a paper-safe fixed-time statement is + +```math +|\widetilde f_{CS}(x)-f(x)| +\le +\epsilon_{sk}\|f\|_2 ++m_{sa}(x,\delta) ++k\max_r T_{r,h_r(x)}. +``` + +CountSketch is the cleanest first target for sampled GOS: additive state, +two-sided estimator, and no CMS one-sided caveat. + +### 8.4 DDSketch value-range counts + +DDSketch maps a positive value $x$ to a logarithmic bucket $b(x)$. The bucket +mapping is chosen so values in one bucket have relative value error at most +$\alpha$. + +For a value range $[L,U]$, define a linear bucket readout + +```math +q_{[L,U]}(S) += +\sum_{b: v_b\in[L,U]} S[b], +``` + +where $v_b$ is the representative value of bucket $b$. + +This is an additive nonnegative aggregate. Sampling and CDM compose exactly as +in the generic additive-state theorem: + +```math +|q_{[L,U]}(\widetilde S)-q_{[L,U]}(S)| +\le +m_{sa}([L,U],\delta) ++k\sum_{b:v_b\in[L,U]}T_b. +``` + +This is the DDSketch query class that behaves like a scalar/bucket-count monitor. + +#### CDM isotropic threshold ($L_1$, sum-composition) + +Unlike CMS's point query, the range-count readout $q_{[L,U]}$ is a genuine +sum across buckets, so it composes the same way CountSketch's $F_2$ does +(§7's general water-filling), just with an $L_1$ (linear, not squared) +gradient: $q$ is linear in each $S[b]$, so $g_b=\partial q/\partial S[b]=1$ +for every bucket in range — a constant gradient, giving (isotropic, +$T_b=T$ for all $B$ buckets in range): + +```math +k\sum_{b}T_b=k\cdot B\cdot T\le\epsilon N +\quad\Longrightarrow\quad +\boxed{ +T=\frac{\epsilon N}{k\cdot B} +} +``` + +$B$ must be a genuine, always-current bound on the number of populated +buckets — not a fixed assumed constant. Unlike CMS/CountSketch's $d,w$ +(fixed at construction), DDSketch's bucket count is dynamic in the current +implementation (unbounded growth on demand — see sketchlib-go#72 for a +related, independent memory-safety concern with that growth). $B$ is, +however, cheap to track exactly: increment a counter by 1 the first time a +given bucket index is touched (an $O(1)$ event, far rarer than raw inserts +once the value distribution stabilizes), and recompute $T$ from the latest +$N,B$ on every insert ($O(1)$, just a division) — so $T$ and $B$ never +drift out of sync the way a periodically-refreshed quantity would. This +keeps the guarantee exact without requiring any structural cap on the +sketch itself. + +### 8.5 DDSketch quantiles + +DDSketch quantile queries are not linear readouts. They depend on prefix counts. +Sampling and CDM should therefore be expressed as rank error. + +Let the true bucket count be + +```math +n_b=\sum_{u=1}^N 1\{b(x_u)=b\}. +``` + +Let + +```math +N_{\le b}=\sum_{j\le b}n_j. +``` + +#### Implemented thinning view + +The current DDSketch implementation performs value-independent uniform admission +before the bucket update: if the item is not admitted, the entire update is +skipped. This subsection assumes a single uniform probability $p$ for all items. +Let + +```math +Z_u\sim\mathrm{Bernoulli}(p). +``` + +The sampled bucket count is + +```math +n_b^{sample} += +\sum_{u:b(x_u)=b}Z_u. +``` + +Quantile lookup over the sampled DDSketch uses the sampled total + +```math +M=\sum_u Z_u. +``` + +Uniform thinning preserves the distribution in expectation. With high +probability, the empirical CDF of the sampled stream is close to the true CDF. +For a finite set of $B$ nonempty buckets, a union bound over bucket prefixes +gives + +```math +\sup_b +\left| +\frac{N^{sample}_{\le b}}{M} +- +\frac{N_{\le b}}{N} +\right| +\le +\epsilon_{sa} +``` + +with + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right), +``` + +assuming $pN$ is not too small. + +#### Horvitz-Thompson analysis view + +Equivalently, for analysis one may assign every admitted item weight $1/p$: + +```math +Y_u=Z_u/p. +``` + +Then + +```math +\widehat n_b=\sum_{u:b(x_u)=b}Y_u +``` + +is an unbiased estimator of $n_b$, and + +```math +\mathrm{Var}(\widehat N_{\le b}) += +N_{\le b}\frac{1-p}{p}. +``` + +Bernstein plus a union bound over $B$ prefixes yields + +```math +\sup_b +|\widehat N_{\le b}-N_{\le b}| +\le +O\left( +\sqrt{\frac{N\log(B/\delta)}{p}} ++\frac{\log(B/\delta)}{p} +\right). +``` + +Dividing by $N$ gives the same rank scale: + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right). +``` + +This weighted view is useful for proof. The implemented unweighted sampled +quantile returns the same bucket as the uniformly weighted view, because every +admitted item has the same weight. Counts, however, need `sample_p` rescaling if +they are queried as counts. + +If the controller uses nonuniform per-site probabilities $p_i$, the unweighted +sampled DDSketch quantile is biased toward high-$p_i$ sites. A nonuniform +deployment needs weighted quantile semantics, a resampling correction, or a +separate proof for the chosen estimator. + +#### CDM rank staleness + +If bucket residuals satisfy $|\rho_i[b]|\le T_b$, then for any prefix: + +```math +|\widetilde N_{\le b}-\widehat N_{\le b}| +\le +k\sum_{j\le b}T_j. +``` + +Hence the CDM-induced rank error is + +```math +\epsilon_{cdm} += +\frac{k}{N} +\sup_b\sum_{j\le b}T_j. +``` + +If the backend's total count is also stale, allocate a small additional budget +for denominator error, or normalize by a lower bound on $N$. + +#### DDSketch quantile corollary + +Let $\widetilde x_q$ be the sampled and stale DDSketch estimate of the +$q$-quantile. With probability at least $1-\delta$, + +```math +\boxed{ +(1-\alpha)x_{q-\epsilon_{sa}-\epsilon_{cdm}} +\le +\widetilde x_q +\le +(1+\alpha)x_{q+\epsilon_{sa}+\epsilon_{cdm}} +}. +``` + +Thus DDSketch keeps its multiplicative value error $\alpha$, while sampling and +CDM/GOS add rank error. + +### 8.6 KLL quantiles + +KLL is mergeable, but it is not a subtractive additive counter array. It should +not be forced into the per-cell linear residual theorem. + +For a sealed window, mergeability preserves the KLL rank guarantee: + +```math +|\mathrm{rank}(\widehat x_q)-qN| +\le +\epsilon_{KLL}N +``` + +with the configured KLL failure probability. + +If updates are uniformly sampled before KLL insertion, the same thinning rank +term appears: + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(1/\delta)}{pN}} ++\frac{\log(1/\delta)}{pN} +\right). +``` + +For open-window freshness, use a segment model: unshipped KLL segments contain +$R$ samples. Then the stale-rank contribution is bounded by + +```math +\epsilon_{cdm} +\le +R/N. +``` + +The combined quantile rank error is + +```math +\epsilon_{rank} +\le +\epsilon_{KLL} ++\epsilon_{sa} ++R/N. +``` + +KLL is therefore a mergeable-summary adapter, not a GOS per-cell water-filling +instance. + +#### Emit trigger + +Since $\epsilon_{cdm}\le R/N$, holding rank staleness within a chosen budget +$\epsilon$ means emitting (a full disjoint segment, then resetting — KLL has +no subtractive delta, so there is nothing finer-grained to send) once the +unshipped sample count crosses: + +```math +\boxed{ +R\ge\epsilon N +} +``` + +$R$ is exactly the sketch's own item count since its last reset (already +tracked — `Count()` in the current implementation, which zeroes on +`Reset()`), and $N$ is the window's running total count — both $O(1)$ +incremental counters, checked at insert time, no periodic scan required. + +### 8.7 HyperLogLog distinct count + +HLL state is a vector of max registers, with merge defined by register-wise max. +It is mergeable but non-additive. + +Base error: + +```math +\epsilon_{HLL}\approx 1.04/\sqrt{m} +``` + +for $m$ registers. + +Nitro-style inverse-probability update sampling is not appropriate for HLL +register updates: the update is a max operation, not an additive counter +increment. Hash-threshold distinct sampling can be analyzed separately, but it +does not fit the additive sampling theorem above. + +For CDM/freshness, use a family-specific register-change adapter. The linear +per-cell bound $k\sum_j |a_j|T_j$ does not apply directly. + +#### Register-change adapter (OctoSketch Algorithm 1 variant for max-merge sketches) + +HLL's contribution to the cardinality estimate is on the harmonic-mean +$2^{-C}$ scale, not the raw register value $C$: a register going from +$C=5$ to $C=6$ doubles that register's linear-scale contribution, while +$C=30\to31$ is an even larger absolute jump on that same linear scale — a +threshold on the raw register value would not reflect this. OctoSketch's +own treatment of max-merge sketches (HyperLogLog, LogLog, Loglog-Beta) +gates on the *linearized* value instead: register $j$ goes from $C$ to +$C'$ ($C'\ge C$, MAX-merge is monotone non-decreasing); send iff + +```math +\boxed{ +|2^{C'}-2^{C}|\ge2^{\tau} +} +``` + +— send the raw value $C'$ itself, and (unlike every additive family above) +**never reset the register**: MAX-merge is idempotent, so re-sending an +unchanged or already-known value at the backend is harmless +(`max(x,x)=x`), which is exactly what makes HLL's CDM/GOS treatment +structurally different from every additive family in this section. $\tau$ +is naturally interpreted in "doublings" — $\tau=1$ means "don't bother +sending unless this register's contribution at least doubled since it was +last sent," $\tau=2$ means "at least 4x," etc. Because $C,C'$ are small +non-negative integers, $2^{C}$ is exact integer arithmetic (`1< v^*$ is equivalent to a bucket-prefix +count predicate up to the DDSketch value bucket error. The sampling and CDM +terms then enter as rank/count margins. + +## 11. Summary table + +| Family/query | Sampling term | Continuous Distributed Monitoring (CDM) / Geometric-OctoSketch (GOS) term | Isotropic threshold (§8) | Safe paper claim | +| --- | --- | --- | --- | --- | +| Sum/count | scalar Bernoulli count variance | $kT$ | $T=\epsilon N/k$ ($B=1$ case of DDSketch) | exact base sketch plus sampling plus scalar staleness | +| CMS point | row-counter sampling variance | $k\max_r T_{r,h_r(x)}$ | $T=\epsilon N/k$ (§8.2, max-composition, no $\sqrt{dw}$) | caveated adapter: row union plus collision event; probabilistic alert margin | +| CountSketch point | row-counter sampling variance | $k\max_r T_{r,h_r(x)}$ | $T=\epsilon\lVert\widehat C\rVert/(2k\sqrt{dw})$ (§7, $L_2$/$F_2$, sum-composition) | clean additive/two-sided composition; best first theorem target | +| DDSketch range count | bucket-count sampling variance | $k\sum_{b\in \mathrm{range}}T_b$ | $T=\epsilon N/(kB)$ (§8.4, $L_1$ sum-composition; $B$ tracked $O(1)$ per new bucket) | additive bucket-count composition | +| DDSketch quantile | rank error $\epsilon_{sa}$ | prefix rank error $\epsilon_{cdm}$ | — (uses range-count $T$ above) | $(1\pm \alpha)$ value error around $q\pm \epsilon$ rank | +| KLL quantile | thinning rank error | unshipped segment rank $R/N$ | emit+reset when $R\ge\epsilon N$ (§8.6, count-based, no cell structure) | mergeable adapter, not per-cell GOS | +| HLL distinct | exclude Nitro-style additive sampling | register-change adapter | $\lvert2^{C'}-2^{C}\rvert\ge2^{\tau}$ (§8.7, no reset — MAX-merge idempotent) | mergeable but non-additive; family-specific proof | + +Anisotropic (gradient-weighted, non-uniform $T_j$ per cell) is only worked +out for CountSketch (§7's general water-filling). Extending it to the other +families is a TODO — the $L_1$-composed families (CMS/DDSketch/Sum) would +need their own $c_j=k|g_j|$ instantiation, and the async-per-cell-reset +transport (below) breaks the water-filling's "Activity since a common +reference point" assumption in a way that isn't resolved yet (see the +implementation notes). + +## 12. Paper theorem templates + +### Theorem 1: additive-state composition + +For a fixed linear query $q(S)=\langle a,S\rangle$ at a fixed time in a tumbling window, +assume: + +1. sampling randomness is independent across updates, i.e. the admission + indicators $Z_u$ are independent; +2. sampled updates use inverse-probability weights for additive counters; +3. each site/cell residual is bounded by $R_{ij}$; in the update-synchronous + ideal protocol $R_{ij}=T_j$, while periodic checks can use + $R_{ij}=T_j+U_{ij}\Delta_{check}$; +4. the base sketch error is at most $E_{sk}(q)$ with probability + $1-\delta_{sk}$. + +Then with probability at least $1-\delta_{sk}-\delta_{sa}$, + +```math +|q(\widetilde S)-q(f)| +\le +E_{sk}(q) ++m_{sa}(q,\delta_{sa}) ++\sum_j |a_j|\sum_i R_{ij}. +``` + +If only a variance statement is needed, pairwise uncorrelated admission +indicators are sufficient for the variance identity, but not for the Bernstein +margin above. If all sites use the update-synchronous bound $R_{ij}=T_j$, the +staleness term reduces to $k\sum_j |a_j|T_j$. + +### Theorem 2: Geometric-OctoSketch (GOS) threshold allocation + +Under the threshold-crossing cost model, given additive-state staleness budget +$B$ and cell activity $V_j$, the threshold vector minimizing $\sum_j V_j/T_j$ +subject to $\sum_j c_jT_j\le B$ is + +```math +T_j += +\frac{B\sqrt{V_j/c_j}} +{\sum_\ell\sqrt{V_\ell c_\ell}}, +``` + +before floors and caps. The clamped solution is obtained by iterative +water-filling over unclamped cells. + +### Corollary: DDSketch sampled open-window quantile + +For DDSketch relative value parameter $\alpha$, uniform value-independent +admission probability $p$, $B$ nonempty buckets, and bucket residual thresholds +$T_b$, with probability at least $1-\delta$: + +```math +(1-\alpha)x_{q-\epsilon_{sa}-\epsilon_{cdm}} +\le +\widetilde x_q +\le +(1+\alpha)x_{q+\epsilon_{sa}+\epsilon_{cdm}}, +``` + +where + +```math +\epsilon_{sa} += +O\left( +\sqrt{\frac{\log(B/\delta)}{pN}} ++\frac{\log(B/\delta)}{pN} +\right), +``` + +and + +```math +\epsilon_{cdm} += +\frac{k}{N}\sup_b\sum_{j\le b}T_j. +``` + +## 13. Implementation notes + +- The Go DDSketch wrapper performs value-independent admission before the + bucket update. This is equivalent to uniform thinning for quantile rank + analysis. Counts need `sample_p` rescaling if they are queried as counts. +- CountSketch sparse deltas currently require integral cell deltas. If sampling + creates fractional weighted cells, the implementation can fall back to full + frames. That preserves accuracy but weakens communication claims for the + sampled+delta combination unless a fractional delta wire is added. +- SDK-side Nitro-style sampling requires a wire representation for sampled + weighted counter updates from SDK/data source to agent collector. For signed + sketches, the payload must preserve row, column, sign, and weight + $1/p_{i,r}$. +- The strongest continuous $|\rho_i[j]|\le T_j$ proof assumes an update-synchronous + threshold check — i.e. the threshold is checked at insert time, not on a + periodic scan. As of the 2026-07 no-sub-window redesign this is now the + actual model for the isotropic threshold in all 6 families (checked and, + where applicable, reset at insert time; wire transmission still happens via + the existing OTLP/`SketchEnvelope` pipeline, triggered on-demand instead of + on a fixed `SubWindowInterval` — see `design-gos-unified-edge-telemetry.md` + for the transport/architecture side of this). The one place the + update-synchronous assumption is NOT yet realized is anisotropic CountSketch: + the per-cell $\{T_j\}$ water-filling solve is $O(dw)$ and cannot run on every + insert, so it still needs a periodic re-solve, and — unresolved as of this + writing — the $Activity_j$ input to that solve needs a redefinition that + doesn't depend on a single common "prev" snapshot, since cells can now reset + asynchronously at different times (naive `|current-prev|` against a stale or + per-cell-inconsistent reference double-counts or under-counts activity + around individual cell resets). TODO, not blocking the other 5 families. +- CMS's local point-query readout (`cms_point` functional) is retired as part + of the same redesign: once cells reset in place at insert time, a `min`-based + local read is no longer a valid point-in-time estimate (a single recently- + reset row poisons the whole `min`). Alerting/point-query reads move + entirely to the backend, which reconstructs exact state via additive + fragment-summation (see the transport doc) regardless of when any given + edge-side cell was last reset. +- For SDK-source or multi-unit sampling, the sampling randomness must make the + admission indicators independent per `(edge_id, agg_id, window)` or hash-based + per item. Shared sampler seeds can correlate admissions and invalidate + variance-addition by introducing covariance terms. + +## 14. Evaluation checklist + +The system claim is only credible if the evaluation shows that ASAPCollector +moves work off the central raw-sample path while preserving the advertised error +envelope for supported queries. At minimum, measure: + +- **SDK/source update work:** candidate sketch counter updates/sec, admitted + counter updates/sec, CPU, and memory as $p_{i,r}$ changes. +- **Source-to-agent network load:** bytes/sec and messages/sec for raw OTLP + export versus sampled weighted counter-update export. +- **Agent-to-backend network load:** bytes/sec and messages/sec for periodic + sketch export and GOS thresholded deltas. +- **Backend ingest load:** accepted samples/sec or summary frames/sec, WAL/queue + pressure, index/storage growth, and write amplification if applicable. +- **Freshness:** open-window staleness measured as both wall-clock lag and + query-space error $\mathrm{Err}_q^{cdm}$. +- **Accuracy envelope:** empirical query error decomposed into sketch error, + sampling error, and staleness error; report coverage of the claimed + high-probability bound. +- **Controller behavior:** selected $p_{i,r}$ and $T_j$ under hot/cold sources, + query-sensitive/query-insensitive rows or cells, and changing workloads. +- **Fallback boundary:** unsupported query classes and cold raw fallback cost, + so the paper does not imply arbitrary PromQL support. +- **Failure modes:** tick-based overshoot, delayed ACKs, retries, collector + restart, and correlated sampler seeds. + +Use CountSketch, Sum/count, and DDSketch range counts as the first theorem-backed +accuracy experiments. Treat CMS, DDSketch quantiles, KLL, and HLL as +family-specific adapter experiments with their own stated caveats. + +## Appendix A. Bernstein inequality step + +This appendix derives the concentration bound used in Section 4.2: + +```math +|X_q| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2G}{3p_{\min}}\log(2/\delta) +``` + +with probability at least $1-\delta$. + +Recall the sampled query error: + +```math +X_q += +\sum_u (Y_u-1)g_u. +``` + +Define the centered per-update random variable + +```math +A_u=(Y_u-1)g_u. +``` + +Then + +```math +X_q=\sum_u A_u. +``` + +### A.1 Zero mean + +Because $E[Y_u]=1$, + +```math +\mathbb{E}[A_u] += +\mathbb{E}[(Y_u-1)g_u] += +g_u(\mathbb{E}[Y_u]-1) +=0. +``` + +Thus $X_q$ is a sum of independent centered random variables, assuming the +sampling admission indicators are independent. + +### A.2 Variance parameter + +From Section 4.1: + +```math +\mathrm{Var}(A_u) += +g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +Therefore the total variance parameter is + +```math +\sigma_q^2 += +\sum_u \mathrm{Var}(A_u) += +\sum_u g_u^2\frac{1-p_{i(u)}}{p_{i(u)}}. +``` + +### A.3 Uniform bound on one summand + +Bernstein's inequality also needs an almost-sure bound on $|A_u|$. + +Since + +```math +Y_u = +\begin{cases} +1/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +0, & \text{if update }u\text{ is skipped}, +\end{cases} +``` + +we have + +```math +Y_u-1 = +\begin{cases} +(1-p_{i(u)})/p_{i(u)}, & \text{if update }u\text{ is admitted},\\ +-1, & \text{if update }u\text{ is skipped}. +\end{cases} +``` + +Thus + +```math +|Y_u-1| +\le +\frac{1}{p_{i(u)}} +\le +\frac{1}{p_{\min}}. +``` + +If $|g_u| \le G$, then + +```math +|A_u| += +|(Y_u-1)g_u| +\le +\frac{G}{p_{\min}}. +``` + +Let + +```math +M=\frac{G}{p_{\min}}. +``` + +### A.4 Apply Bernstein + +One standard two-sided Bernstein bound for independent centered random variables +$A_u$ with $|A_u| \le M$ and variance sum $\sigma_q^2$ is + +```math +\Pr\left[ +\left|\sum_u A_u\right| +\ge +\sqrt{2\sigma_q^2 t} ++\frac{2M}{3}t +\right] +\le +2e^{-t}. +``` + +Set + +```math +t=\log(2/\delta). +``` + +Then $2e^{-t}=\delta$, so with probability at least $1-\delta$, + +```math +\left|\sum_u A_u\right| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2M}{3}\log(2/\delta). +``` + +Substituting $M=G/p_{\min}$ and $X_q=sum_u A_u$ gives + +```math +|X_q| +\le +\sqrt{2\sigma_q^2\log(2/\delta)} ++\frac{2G}{3p_{\min}}\log(2/\delta). +``` + +This is the Section 4.2 bound. + +## 15. References to cite + +- Prometheus remote write and storage documentation: central ingest, queue/WAL, + CPU, memory, and network costs for exporting high-resolution metrics. +- OpenTelemetry metrics data model: metric events, aggregation temporality, + histograms/exponential histograms, and the motivation for aggregation before + export. +- Prometheus histograms and summaries documentation: early binding of buckets, + quantiles, and aggregation constraints. +- Thanos compactor/downsampling documentation and TSDB compression papers: + post-ingest query/storage optimization rather than pre-ingest load reduction. +- BlinkDB and VerdictDB: approximate query processing over already-ingested + data. +- NitroSketch: update sampling for sketch update work. +- DDSketch: mergeable relative-error quantile sketch. +- Count-Min Sketch and CountSketch: base frequency guarantees. +- KLL: mergeable rank-error quantile sketch. +- HyperLogLog: base cardinality guarantee. +- Cormode et al. distributed functional monitoring / Continuous Distributed + Monitoring (CDM). +- Mergeable summaries: no error compounding under merge. +- Zoom2Net and related telemetry reconstruction work: learned fine-grained + reconstruction from coarse measurements, distinct from preserving + query-sufficient sketch state.