From b93ce7ffd257f0cc4d77aa0129404f0f8dc49dea Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 09:46:13 -0600
Subject: [PATCH 01/12] =?UTF-8?q?docs(design):=20holistic=20edge=E2=86=92b?=
 =?UTF-8?q?ackend=20compression=20(cold=20chunk=20+=20warm=20sketch)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Design (for review, not code) for a custom lossless cold chunk format and a
unified offset/frame-of-reference scheme that also shrinks warm sketch state.

- Cold chunk: best-of-N {Gorilla-XOR, INT_FOR_DELTA, INT_FOR_DOD}, all lossless,
  with a decimal-exactness guard; write path stores to S3 WITHOUT decode; a
  decode-on-read Thanos StoreAPI decodes to XOR at query time so a stock PromQL
  engine queries the compact S3 form.
- Warm sketch: per-family offset/FOR/delta/sparse (SUM/KLL use the offset;
  DDSketch index-FOR; HLL sparse; CMS narrow-counter), plus offset×delta-
  transmission with the offset bound to the Full-snapshot epoch.
- offset-drift → Full re-base, unifying warm (re-emit Full) and cold (cut chunk,
  new base); per-series adaptive Full cadence + heartbeat; counters use delta.

Informed by an offline benchmark on the Chimp/Serf real datasets: INT FOR+delta
beats Gorilla ~4.8x on fixed-decimal series (the majority); Gorilla wins on true
high-precision floats — hence best-of-N. Decided out of scope: no zstd, no lossy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 197 ++++++++++++++++++
 1 file changed, 197 insertions(+)
 create mode 100644 docs/design/holistic-edge-backend-compression.md

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
new file mode 100644
index 000000000..bc48230e3
--- /dev/null
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -0,0 +1,197 @@
+# ASAP holistic edge→backend compression design
+
+Status: DRAFT for review. Informed by the offline micro-benchmark in this dir
+(`./compressbench -dir data_serf` on the Chimp/Serf real datasets) and the
+Gorilla / VictoriaMetrics / Serf literature.
+
+## 0. Goals & principles
+
+One pass over raw data at the edge feeds BOTH the cold (raw archive) and the
+warm (sketch/aggregation) processors (existing parse-once framework). We add a
+shared per-series **offset / frame-of-reference** so the numbers each consumer
+actually stores are small, then bit-pack — minimizing **bits, bandwidth,
+memory, CPU**.
+
+Unifying principle (the "offset / common-bits" idea, generalized):
+> Find the common base in each structure's natural integer representation,
+> subtract it, store small residuals bit-packed, exploit sparsity, and
+> re-base the frame when it drifts.
+
+Hard requirements:
+- **Cold = lossless** (raw archive). Warm sketches keep their existing
+  approximation guarantees; the encoding adds no extra error.
+- **Backend ingests edge-compressed chunks WITHOUT decode+reinsert.** Decode is
+  pushed to the (rare) READ path: a custom Thanos StoreAPI decodes chunks to
+  XOR `AggrChunk`s at query time, so a stock PromQL engine queries them while
+  S3 holds the compact custom format. Writes are far more frequent than cold
+  reads, so this is where decode belongs.
+
+Benchmark headline (real Chimp/Serf datasets, lossless, block-avg 1000/chunk):
+- Fixed-decimal series (11/12 datasets — temps, stocks, sensors, pressure,
+  GPS, dust, wind, grid): **VM-style integer FOR+delta beats Gorilla ~4.8×
+  avg (up to ~10×)**. This *is* the offset idea, on the integer-scaled values.
+- Genuinely high-precision float (float32-derived, 15 sig digits): VM can't
+  stay decimal-exact → falls back to bit-pattern (worse); **Gorilla-XOR wins**.
+- ⇒ The codec must be a per-block **best-of-N including Gorilla-XOR**, not
+  "VM replaces Gorilla".
+- Decode CPU: VM ~41 ns/sample vs Gorilla ~71 — VM decode is *faster*, good
+  for the decode-on-read path.
+
+---
+
+## 1. Cold raw chunk format
+
+### 1.1 Part (one S3 object per tenant / 2h-block / shard)
+```
+[part header]   magic "ASAPCC1" | u8 version | i64 block_start_ms | i64 block_end_ms | uvarint series_count
+[chunks]        per-series chunks, concatenated
+[index]         sorted by series: { labels(symbol refs), u64 chunk_off, u32 chunk_len, i64 min_ts, i64 max_ts }
+[symbol table]  deduped label strings (the index references offsets here)
+[footer]        u64 index_off | u64 index_len | u64 symtab_off | u32 crc32c
+```
+The index + symbol table let the StoreAPI answer `Series(matchers, mint, maxt)`
+without scanning chunk bodies.
+
+### 1.2 Per-series chunk
+```
+[chunk header]
+  u8  codec_tag          # see 1.3
+  uvarint n_samples
+  ts:  i64 t0_delta(block-relative) then delta-of-delta varints   # timestamps
+  # value codec params (codec-specific):
+  tag INT_FOR_DELTA / INT_FOR_DOD:
+      i8       scale_exp           # decimal exponent e: int_v = round(v * 10^-e); v = int_v * 10^e
+      zigzag-varint base           # the FOR reference (frame base; see drift, 1.4)
+      zigzag-varint first_residual
+  tag GORILLA_XOR:   (no extra params; standard XOR stream)
+[chunk body]
+  bit-packed residual stream (INT_*), or XOR stream (GORILLA)
+```
+
+### 1.3 codec_tag
+```
+0 = GORILLA_XOR        lossless float64 (fallback for high-precision floats)
+1 = INT_FOR_DELTA      scale→int64, FOR(base), delta, bit-pack          (gauges)
+2 = INT_FOR_DOD        scale→int64, FOR(base), delta-of-delta, bit-pack (counters / timestamps)
+All three are LOSSLESS. No lossy (Serf) and no zstd-wrapping — kept deliberately
+simple: the INT_* FOR+delta already captures the win on fixed-decimal data, and
+Gorilla-XOR is the lossless fallback for true high-precision floats.
+```
+
+### 1.4 Encoder: per-series-per-block best-of-N with exactness check
+```
+fn encode_chunk(values, opts):
+    cands = []
+    # INT path — ONLY if it round-trips EXACTLY (the lib/decimal precision trap)
+    (ok, e, ints) = try_scale_to_int64(values)        # find decimal exp e s.t. round-trip exact
+    if ok:
+        cands += encode_int(ints, FOR_DELTA)            # tag 1
+        cands += encode_int(ints, FOR_DOD)              # tag 2
+    # Gorilla is always valid + lossless
+    cands += encode_gorilla_xor(values)                 # tag 0
+    return argmin(cands, key=byte_len)                  # smallest — all candidates lossless
+```
+`try_scale_to_int64` is the load-bearing guard: never ship INT_* unless decode
+reproduces the original float64 bit-exactly (else a naive VM-decimal silently
+introduces ~1e-12 error — confirmed on Motor-temp in the benchmark).
+
+`base` re-bases on **drift** within a block: if a residual would overflow the
+chosen bit-width, cut the chunk early and start a new chunk with a fresh base
+(the cold analogue of "offset drift → new Full"; see §3).
+
+### 1.5 Backend write path (NO decode)
+On ingest, validate the part header/crc, store the object to S3, and register
+its series→part entries in the manifest/index. No decode, no re-encode.
+
+### 1.6 Decode-on-read StoreAPI (the compat shim)
+Implements the Thanos `storepb.StoreServer`:
+```
+Series(req {matchers, min_t, max_t}) -> stream of SeriesResponse:
+    for part in manifest.parts_overlapping(min_t, max_t):
+        for series in part.index.matching(req.matchers):
+            for chunk in series.chunks_overlapping(min_t, max_t):
+                samples = decode(chunk)                 # by codec_tag; INT_* adds base + scale
+                emit AggrChunk{ raw: xor_encode(samples) }   # hand PromQL a standard XOR chunk
+```
+Thanos-query unions this with the >=2h store-gateway path, exactly as the
+current gorilla-merger StoreAPI does — we extend that StoreAPI's `Series()`.
+
+---
+
+## 2. Warm sketch encoding
+
+Same offset idea, applied at each sketch's natural representation. The offset
+is a property of the **Full-snapshot epoch** (see §3): a Full carries the
+re-based offset/scale in its header; all Deltas until the next Full encode
+residuals in that frame; backend reconstructs absolute values at merge/query.
+
+| family | offset on raw values? | encoding |
+|---|---|---|
+| SUM / COUNT | yes (linear) | ship `Σresidual` (narrow int, varint) + N + epoch offset; backend `Σresidual + (ΣN)·offset`. Counter sums: ship per-window increment (delta), no fixed offset. |
+| KLL (quantile) | yes (shift-equivariant: `q(X−c)=q(X)−c`) | store sampled values as `(v−offset)` fixed-point (i16/i32 vs f64 ⇒ ~½ size); within a level the sorted samples delta-encode; backend adds offset to the quantile result. |
+| DDSketch (current family) | NO (log-scale; `v−c`→0 breaks relative error) | FOR+delta on the **bucket-index** array (`min_index` + Δindex varints) + varint counts. |
+| HLL (cardinality) | NO (hash) | **sparse mode** (low card: sorted non-zero registers, delta+varint = HLL++) + 6-bit dense packing. |
+| CMS / CountSketch (freq/topk) | NO (hash) | narrow counters + per-row FOR + bit-pack; cross-window delta of the matrix; topk heap shipped as k entries. |
+
+Cross-cutting warm levers:
+- **Delta transmission** (already in place: ProtoFull/ProtoDelta + the
+  delta-stitching carry-in): ship only changed sketch state between Fulls.
+- **Offset rides in the Full header only**; Deltas carry residuals only.
+- All Deltas in a Full-epoch share one offset frame ⇒ they remain mergeable
+  (merging `(v−off₁)` and `(v−off₂)` residual-KLLs would be garbage).
+
+---
+
+## 3. Offset drift → re-base (Full) — unifies warm & cold
+
+A frame stays valid only while residuals fit it. Re-base the frame (warm: emit
+a new **Full sketch**; cold: cut the chunk and start a new **base**) when:
+- **Correctness**: a residual would overflow the chosen narrow width → MUST
+  re-base.
+- **Efficiency** (optional): residuals now need K more bits than a re-base
+  would → re-base to reclaim ratio (K threshold weighed vs Full cost).
+
+Plus a **max-interval heartbeat Full** even without drift, because a Full is a
+self-contained base needed for: (a) durability/recovery — without a recent Full,
+a lost Delta makes the chain undecodable (cold S3 + restart-recovery depend on a
+recent base); (b) a newly-joining consumer/query window needs a base.
+
+Rule: **emit Full when (drift) OR (heartbeat elapsed)**.
+
+Consequences:
+- Full cadence becomes **per-series adaptive**: stable gauges almost never
+  re-base (nearly all Delta); volatile series re-base often (but volatile data
+  is inherently less compressible — cost lands where it should).
+- **Counters are the exception**: use delta-of-value (base = previous sample,
+  auto-re-bases every sample, never "drifts") — so drift→Full is the gauge/FOR
+  story; counters just delta.
+
+Minimal change to today's pipeline: the agent already emits periodic Fulls;
+add **drift** as a second Full trigger; backend delta-stitching already treats
+a Full as the carry-in base, so a re-base is just a new base.
+
+---
+
+## 4. Shared per-series offset from parse-once
+
+Compute the per-series reference once in the single parse pass (running
+min/last-value + the decimal scale exponent), reuse for BOTH the cold chunk and
+the warm sketch. If values are quantized once to fixed-point, both gorilla-cold
+and KLL store the quantized form ⇒ consistent + no recompute. CPU note: offset
+does NOT reduce sketch-build cost (hashing/compaction is fixed); the CPU wins
+come from parse-once (done) + decode-on-read (cold decode off the ingest hot
+path) + VM's cheaper decode.
+
+---
+
+## 5. Open decisions
+1. Drift thresholds: correctness (overflow) is forced; the efficiency K-bit
+   threshold + heartbeat Full interval need tuning (per-shape defaults).
+2. Which warm sketch families get the offset/FOR re-encoding first (SUM+KLL are
+   the clean wins; DDSketch index-FOR + HLL sparse depend on what the current
+   sketchlib-go / asap-sketchlib serialization already does — audit in progress
+   to size the gain).
+
+Explicitly OUT of scope (decided): no zstd-wrapped variants, and no lossy/Serf
+option — cold stays purely lossless with the {Gorilla-XOR, INT_FOR_DELTA,
+INT_FOR_DOD} best-of-N.

From 3df0e06edf01dad76d7d3cf564762ecf7ab4c90a Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 09:53:01 -0600
Subject: [PATCH 02/12] =?UTF-8?q?docs(design):=20fold=20in=20measured=20sk?=
 =?UTF-8?q?etchlib=20serialization=20audit=20(=C2=A72.1)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Backend stores the wire bytes opaquely (no re-serialize / no part recompression),
so wire = sketch_db storage = disk parts — one serialize change wins on all three.
Measured (N=5000): HLL dense 16,532 B flat → sparse 5–50× (P1, also cuts warm
memory since stored uncompressed); KLL raw-f64 2,157 B → (v−offset) fixed-point
~2× (P2). DDSketch (556 B, already FOR+varint), CMS/CountSketch (~12.5 KB, already
zigzag-varint), SUM (OTLP-framing-bound) already well-encoded — skip. Supersedes
the earlier DDSketch-index-FOR / narrow-CMS rows. §5.2 resolved.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 43 +++++++++++++++++--
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index bc48230e3..113014b65 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -140,6 +140,41 @@ Cross-cutting warm levers:
 - All Deltas in a Full-epoch share one offset frame ⇒ they remain mergeable
   (merging `(v−off₁)` and `(v−off₂)` residual-KLLs would be garbage).
 
+### 2.1 Audit of the current serialization (measured)
+
+The backend does NOT re-serialize: `asap_sketchlib` and `sketchlib-go` share one
+cross-language wire format, and the backend stores the wire bytes **opaquely**
+(`SketchSampleState{bytes, encoding}`; flushed parts write `sketch_bytes`
+verbatim, no part-level recompression). So **wire format = sketch_db storage =
+disk-part bytes** — optimizing `sketchlib-go`'s `Serialize*` wins on bandwidth,
+warm memory, AND cold disk at once.
+
+Measured (N=5000/window; harness `/mydata/sketch-audit`):
+
+| family | current encoding | bytes | headroom | verdict |
+|---|---|---|---|---|
+| HLL p=14 | dense 1 byte/register × 16384 (flat, any cardinality) | 16,532 | sparse full-state (delta-idx) → 5–50× for low card; 6-bit dense pack 1.34× | **P1** |
+| KLL k=200 | raw f64 items array | 2,157 | `(v−offset)` fixed-point f64→~4 B → ~2× | **P2** |
+| DDSketch α=.01 | dense varint counts keyed by FOR offset base, zigzag | 556 | already FOR+varint; sparse would be *larger* (85% occupancy) | skip |
+| CMS / CountSketch 3×4096 | sint64 zigzag-varint (~1 B/cell) | ~12.5 KB | per-row FOR ~0 gain | skip |
+| SUM/COUNT | OTLP Sum dp, raw f64/group | ~8 B/grp | OTLP framing dominates; residual marginal | skip |
+
+So the warm scope narrows to **two changes**:
+- **P1 — HLL sparse full-state serialize** (HLL++ style: sorted non-zero
+  registers, delta+varint; fall back to 6-bit-packed dense above the crossover
+  ~6k nonzero regs). The lib already has a sparse *delta* path (`hll/delta.go`),
+  just not for full-state. Biggest lever — and since the 16 KB dense state is
+  stored **uncompressed** per instance, this also cuts **warm SketchStore
+  memory** 5–50× for low-cardinality series (not just wire, which gzip masks).
+- **P2 — KLL value-offset/quantization** (`(v−offset)` fixed-point, ~2×) — the
+  offset idea, measured.
+
+DDSketch (already FOR+varint), CMS/CountSketch (already zigzag-varint, off the
+legacy float64 matrix), and SUM/COUNT (OTLP-framing-bound) are already
+well-encoded — do NOT touch. This **supersedes** the "FOR+delta on DDSketch
+indices" / "narrow CMS counters" rows in the table above, which the audit shows
+are redundant.
+
 ---
 
 ## 3. Offset drift → re-base (Full) — unifies warm & cold
@@ -187,10 +222,10 @@ path) + VM's cheaper decode.
 ## 5. Open decisions
 1. Drift thresholds: correctness (overflow) is forced; the efficiency K-bit
    threshold + heartbeat Full interval need tuning (per-shape defaults).
-2. Which warm sketch families get the offset/FOR re-encoding first (SUM+KLL are
-   the clean wins; DDSketch index-FOR + HLL sparse depend on what the current
-   sketchlib-go / asap-sketchlib serialization already does — audit in progress
-   to size the gain).
+2. ~~Which warm sketch families get the offset/FOR re-encoding first~~
+   **RESOLVED by the §2.1 audit**: P1 = HLL sparse full-state (5–50×, + cuts
+   warm memory), P2 = KLL value-offset (~2×). DDSketch / CMS / CountSketch /
+   SUM are already well-encoded — skip.
 
 Explicitly OUT of scope (decided): no zstd-wrapped variants, and no lossy/Serf
 option — cold stays purely lossless with the {Gorilla-XOR, INT_FOR_DELTA,

From 7374bdee3f14a382fc5d5626a8899ff724626af4 Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:13:20 -0600
Subject: [PATCH 03/12] =?UTF-8?q?docs(design):=20add=20=C2=A71.7=20grouped?=
 =?UTF-8?q?=20layout=20from=20cross-series=20benchmark?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Measured (/mydata/xseries-bench, real cluster groups + synthetic): shared
timestamp column is the cross-series win — ts is 36-52% of per-series bytes
(values FOR-compress well), so sharing it across a k-series group saves -43%
on the real aggregate, correlation-independent (the Heracles result). The
cross-series VALUE base is net-NEGATIVE (-3.5%): predictor is noise/signal
ratio not correlation, per-series delta already took the common bits, plus a
float-domain trap. Adopt shared-ts grouped layout; cross-series value base is
opt-in only for near-replica groups.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index 113014b65..df155df14 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -116,6 +116,34 @@ Series(req {matchers, min_t, max_t}) -> stream of SeriesResponse:
 Thanos-query unions this with the >=2h store-gateway path, exactly as the
 current gorilla-merger StoreAPI does — we extend that StoreAPI's `Series()`.
 
+### 1.7 Grouped layout — shared timestamp column (cross-series)
+
+Measured (`/mydata/xseries-bench`, real cluster groups @2h/15s k=50 + synthetic).
+Two cross-series levers; only one pays.
+
+- **Shared timestamp column — ADOPT.** Same-metric series in a part share ONE
+  timestamp column instead of every chunk carrying its own. Because INT_FOR
+  values compress so well, timestamps are **36–52% of per-series bytes** on real
+  groups; sharing the column across a k-series group removes ≈ ts_frac·(k−1)/k →
+  **−43% on the real cluster aggregate**, correlation-INdependent, low-risk.
+  (The Heracles VLDB'21 result, confirmed here.) Part layout becomes:
+  per-metric group = `{ one shared ts column (delta-of-delta) }` + `{ per-series
+  value chunks: codec_tag + residuals, NO ts }`. The decode-on-read StoreAPI
+  zips the shared ts with each series' values. Warm sketch parts get the same
+  win (same-window sketch series share the window-end column).
+- **Cross-series value base — do NOT adopt by default.** Per-timestamp base
+  `b(t)=min` + per-series value residuals. Measured **net-NEGATIVE (−3.5% vs
+  shared-ts aggregate)**: it only wins for near-identical-replica series, and
+  the predictor is the **noise/signal ratio, NOT correlation** (even ρ=0.99
+  lost +2.6% when noisy) — per-series delta-of-delta already extracted the
+  per-series common bits, so a noisy cross-series base just adds entropy. Trap:
+  must be done in the integer domain (naive float subtraction breaks decimal
+  representability, ~2× bloat). Optional per-group cost-based opt-in only.
+
+Takeaway: the remaining cross-series "common bits" worth taking are the
+**timestamps** (shared column, −43%), not the values (already extracted
+per-series).
+
 ---
 
 ## 2. Warm sketch encoding

From d15480ed757694c3c4ca450986861dfff4ddbe8e Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:18:58 -0600
Subject: [PATCH 04/12] =?UTF-8?q?docs(design):=20pin=20v1=20drift/heartbea?=
 =?UTF-8?q?t=20policy=20(=C2=A73.1),=20resolve=20=C2=A75.1?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

v1 is zero-tuning: drift = hard/overflow only (re-base when a residual exceeds
the frame's chosen integer width — a correctness bound, not a knob); heartbeat
= fixed N windows (= existing Full cadence; cold's chunk bound already serves).
Counters use delta + heartbeat. The soft/efficiency-K drift (the only tunable)
is deferred to a later optimization, added only if long-lived frames degrade
without overflowing. §5.1 marked decided.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 28 +++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index df155df14..6d437033a 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -233,6 +233,28 @@ Minimal change to today's pipeline: the agent already emits periodic Fulls;
 add **drift** as a second Full trigger; backend delta-stitching already treats
 a Full as the carry-in base, so a re-base is just a new base.
 
+### 3.1 v1 policy (concrete, zero-tuning)
+
+Ship the simplest correct version first; defer the only tunable knob.
+
+- **Drift trigger = hard/overflow ONLY.** Per frame, pick the residual integer
+  width from the Full's observed range (i16 if it fits, else i32, else i64);
+  re-base (warm: emit Full; cold: cut chunk + new base) the moment a residual
+  would exceed that width. This is a correctness bound, **not a tunable**.
+- **Heartbeat = fixed interval.** Emit a Full at least every **N windows** even
+  without drift. Default `N` = the existing agent Full cadence (today's
+  ProtoFull period); for cold, the natural chunk bound (a time-block / ≤~120-
+  sample chunk) already serves as the heartbeat. Bounds crash-loss and the
+  query base-lookback to ≤ one heartbeat.
+- **Counters:** delta-of-value (no drift); the heartbeat Full still applies
+  (recovery / new-consumer base).
+
+**DEFERRED — soft/efficiency drift (the tunable `K`):** re-base when residuals
+waste `> K` bits vs a fresh frame. Skipped in v1 — it's a second-order
+optimization and the only thing that would need per-shape tuning. Add it ONLY
+if observation shows long-lived frames whose residuals widen (compression
+silently degrading) without ever overflowing. Until then **v1 needs no tuning**.
+
 ---
 
 ## 4. Shared per-series offset from parse-once
@@ -248,8 +270,10 @@ path) + VM's cheaper decode.
 ---
 
 ## 5. Open decisions
-1. Drift thresholds: correctness (overflow) is forced; the efficiency K-bit
-   threshold + heartbeat Full interval need tuning (per-shape defaults).
+1. ~~Drift / heartbeat thresholds~~ **DECIDED (v1 — see §3.1)**: drift =
+   hard/overflow only (correctness bound, no tunable); heartbeat = fixed `N`
+   windows (= existing Full cadence). Soft/efficiency-`K` drift DEFERRED as a
+   later optimization — v1 needs no tuning.
 2. ~~Which warm sketch families get the offset/FOR re-encoding first~~
    **RESOLVED by the §2.1 audit**: P1 = HLL sparse full-state (5–50×, + cuts
    warm memory), P2 = KLL value-offset (~2×). DDSketch / CMS / CountSketch /

From 48de15c82744592a1ef5bf1dbb9f27fa56e6accd Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:35:50 -0600
Subject: [PATCH 05/12] =?UTF-8?q?docs(design):=20add=20=C2=A76=20Sampling?=
 =?UTF-8?q?=20=C3=97=20compression=20composition?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Sampling-enhanced sketches compose with the compression scheme on orthogonal,
complementary axes: compression cuts bytes, sampling cuts ingest CPU/update-rate
(the axis FOR/offset can't touch — it's what reduces hashing/compaction cost).
Warm-only (cold raw stays lossless). Composition rule: store the RAW SAMPLED
integer state + a global p per frame, rescale ×1/p at QUERY (not store), so
varint/FOR/sparse keep working on small integers. Per-family: HLL hash-threshold
sampling synergizes strongly with sparse encoding (thins registers, extends the
sparse regime ~1/p×); DDSketch/CMS store smaller sampled counts; KLL weighted
insert is orthogonal. Error budgets add (per-family bounds, KLL pN≳k²); p is a
per-metric control-plane knob alongside tier/sketch-type.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 56 +++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index 6d437033a..2314947e1 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -282,3 +282,59 @@ path) + VM's cheaper decode.
 Explicitly OUT of scope (decided): no zstd-wrapped variants, and no lossy/Serf
 option — cold stays purely lossless with the {Gorilla-XOR, INT_FOR_DELTA,
 INT_FOR_DOD} best-of-N.
+
+---
+
+## 6. Sampling × compression composition
+
+Sampling-enhanced sketches (inverse-probability bucket/counter updates,
+hash-threshold key sampling, weighted KLL insertion — each with its own derived
+error bound) compose with the compression scheme above. They sit on ORTHOGONAL
+cost axes and are COMPLEMENTARY:
+- **Compression** (FOR / delta / sparse / shared-ts) cuts BYTES — wire,
+  sketch_db memory, disk parts.
+- **Sampling** cuts INGEST CPU + update rate — each item triggers a sketch
+  update only with probability `p`. This is the axis compression structurally
+  CANNOT touch: offset/FOR don't reduce the hashing/compaction build cost;
+  sampling does. (§4's note "offset doesn't reduce sketch-build CPU" — sampling
+  is what closes that gap.)
+
+**Scope**: sampling applies to the WARM sketch path only. The cold raw archive
+is NOT sampled (it is the lossless backup; sampling would lose data).
+
+### 6.1 The composition rule
+> Store the RAW SAMPLED integer state + one global `p` per frame; apply the
+> `×1/p` rescale at QUERY, not at store.
+
+Storing the `1/p`-rescaled (inflated, often fractional) state would break
+varint/FOR and bloat. Storing the raw sampled accumulation — e.g. DDSketch
+`m_b = Σ Z_i` (≈ `p·n_b`, a SMALLER integer) + `p`, rescaled `m_b/p` at query —
+keeps counts as small integers, so FOR/delta/varint (and the common-bits idea)
+keep working. It is also numerically cleaner (integer accumulation, no per-update
+fraction). `p` rides in the Full-epoch frame header alongside the offset.
+
+### 6.2 Per-family interaction
+| family | sampling (CPU↓) | compression | interaction |
+|---|---|---|---|
+| HLL | hash-threshold (also thins registers) | sparse full-state (P1) | **strong synergy**: sampling zeroes more registers → sparser → sparse encoding wins more AND stays sparse up to ~`1/p`× higher true cardinality before the dense crossover. Query `n̂/p`. |
+| DDSketch | bucket-update (writes→`p`) | index FOR+varint (done) | orthogonal; bucket SET ≈ unchanged so index encoding unchanged; store sampled counts (smaller int) + `p` → count varint smaller. |
+| KLL | weighted insert (inserts→`p`) | value-offset/quantize (P2) | orthogonal; state still `k` items → offset-encode them; weight = global `(1/p)·2^h` (no per-item cost). |
+| CMS / CountSketch (Nitro) | sampled counter update (writes→`pd` or fixed `s`) | zigzag-varint (done) | synergy: store sampled small-int counts + `p` → varint smaller; writes cut to `pd`. |
+
+### 6.3 Cross-cutting
+- **Delta transmission**: sampling → fewer updates/window → fewer changed cells
+  → smaller delta frames. Merging sampled windows is benign — sampling error
+  `ε_s ∝ 1/√(pN)` shrinks as more windows merge (larger N).
+- **Full-epoch frame**: `p` is a frame-level constant in the Full header (like
+  the offset); deltas inherit it.
+
+### 6.4 Cautions
+- **Error budgets ADD**: `ε_total = sketch error + sampling ε_s (+ KLL value-
+  offset quantization)`, and must fit the metric's accuracy SLA. Family bounds
+  (derived separately): DDSketch `ε_s = O(√(log(B/δ)/pN))`; KLL `ε_total ≈
+  ε_k + ε_s` with the design balance `pN ≳ k²`; HLL `RSE ≈ √((1−p)/(pn) +
+  1.04²/m)`; Nitro adds variance `((1−p)/p)·Σ a_t²`.
+- **`p` is a per-metric control-plane knob**: chosen per metric from the
+  expected N (rate/cardinality) + accuracy SLA. Fits the existing
+  controller-driven model exactly — the controller already annotates each
+  metric's tier + sketch type; it adds `p` the same way.

From 83486ac65ed48d85e25ba9c5070c5404c9d4331e Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:39:25 -0600
Subject: [PATCH 06/12] =?UTF-8?q?docs(design):=20correct=20=C2=A74=20?=
 =?UTF-8?q?=E2=80=94=20shared=20parse-once=20stats,=20NOT=20a=20shared=20o?=
 =?UTF-8?q?ffset=20constant?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The cold (VM/INT_FOR) and warm (KLL/SUM) paths share the parse-once computation
— the per-series decimal SCALE (must be identical so both land in the same
integer domain) + value stats — but NOT a single offset constant: cold re-bases
per block (tightest per-block width) while warm re-bases per Full-epoch (stable
base so Deltas stay mergeable); those cadences conflict. Scope is the raw-value-
offset families only (cold-INT / KLL / SUM); DDSketch (index-FOR), HLL (hash),
CMS (counters) have no shared value offset. Kind matches per shape (gauge→FOR,
counter→delta), not as a number. Also note sampling (§6) is what closes the
sketch-build CPU gap the offset can't.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 34 +++++++++++++++----
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index 2314947e1..c996a494e 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -257,13 +257,33 @@ silently degrading) without ever overflowing. Until then **v1 needs no tuning**.
 
 ---
 
-## 4. Shared per-series offset from parse-once
-
-Compute the per-series reference once in the single parse pass (running
-min/last-value + the decimal scale exponent), reuse for BOTH the cold chunk and
-the warm sketch. If values are quantized once to fixed-point, both gorilla-cold
-and KLL store the quantized form ⇒ consistent + no recompute. CPU note: offset
-does NOT reduce sketch-build cost (hashing/compaction is fixed); the CPU wins
+## 4. Shared per-series value stats from parse-once (NOT a shared offset constant)
+
+What is shared between the cold path and the warm sketches is the **parse-once
+computation**, not a single offset constant:
+
+- **Shared (compute once, both consume):** the per-series **decimal scale
+  exponent** — which MUST be identical (one series has one natural precision, so
+  cold INT_FOR and KLL fixed-point land in the same integer domain and stay
+  mutually consistent) — plus the per-series value stats (running min/range).
+- **NOT shared — the actual FOR base differs by cadence.** Cold re-bases the
+  base **per block** (each chunk takes its own min/first for the tightest
+  per-block residual width); warm sketches re-base **per Full-epoch** (the base
+  must stay stable so a run of Deltas remains mergeable, §3). These cadences
+  conflict — forcing one shared base would hurt whichever tier it's wrong for —
+  so the actual subtracted constant generally differs even though both derive
+  from the same parse-once stats.
+- **Scope — only the raw-value-offset families:** cold INT_* values ↔ KLL ↔ SUM
+  (all subtract a reference from the same raw values). DDSketch (FOR on bucket
+  *indices*), HLL (hash), and CMS/CountSketch (counters) have **no shared
+  raw-value offset** — their compression uses their own structure (§2.1).
+- **Kind matches per shape, not as a constant:** VM uses min-FOR for *gauges*
+  (same kind as KLL's value-offset) but delta-of-delta for *counters* (base =
+  first value), which aligns with the warm side's SUM-as-delta — so the
+  correspondence is gauge→FOR / counter→delta, not one shared number.
+
+CPU note: the offset does NOT reduce sketch-build cost (hashing/compaction is
+fixed) — **sampling (§6) is what closes that CPU gap**. Compression's CPU wins
 come from parse-once (done) + decode-on-read (cold decode off the ingest hot
 path) + VM's cheaper decode.
 

From e6c52a0429047af9c8282614d7bf4158d0678e0f Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:42:07 -0600
Subject: [PATCH 07/12] =?UTF-8?q?docs(design):=20add=20=C2=A77=20sampling?=
 =?UTF-8?q?=20algorithms=20&=20error-bound=20derivations?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Formal backing for §6's cited bounds: the four sampling-enhanced sketches with
their algorithms + unbiasedness/variance/concentration proofs and final boxed
bounds — DDSketch (inverse-prob bucket; x_q∈(1±α)x_{q±ε_s}), KLL (weighted
insert; ε_k+ε_s, pN≳k²), HLL (hash-threshold, not additive; RSE≈√((1-p)/(pn)+
1.04²/m); why per-occurrence is biased), Nitro/CMS/CountSketch (additive; var
(1-p)/p·Σa²; CM loses no-underestimate). Design rules by update type. §6.5 (the
Go benchmark proving these empirically vs sketchlib-go) lands separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 99 +++++++++++++++++++
 1 file changed, 99 insertions(+)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index c996a494e..e6055ecf8 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -358,3 +358,102 @@ fraction). `p` rides in the Full-epoch frame header alongside the offset.
   expected N (rate/cardinality) + accuracy SLA. Fits the existing
   controller-driven model exactly — the controller already annotates each
   metric's tier + sketch type; it adds `p` the same way.
+
+> §6.5 (the offline Go sampling benchmark proving these benefits empirically —
+> CPU/update reduction vs measured error vs the bounds below) is added once the
+> benchmark lands; §7 is its formal backing.
+
+---
+
+## 7. Sampling-enhanced sketches: algorithms & error-bound derivations
+
+Formal backing for the bounds cited in §6. The rule: **sampling must respect
+each sketch's algebraic structure.**
+
+| Sketch | Core update | Valid sampling |
+|---|---|---|
+| DDSketch | additive bucket count | inverse-probability bucket update |
+| KLL | weighted samples + randomized compaction | inverse-probability item weight, then normal compaction |
+| HLL | max register | hash-threshold element sampling, then rescale |
+| CMS / CountSketch / Nitro | additive counters | inverse-probability counter update |
+
+Let $0<p\le 1$, and $Y_i=Z_i/p$ with $Z_i\sim\mathrm{Bernoulli}(p)$, so
+$\mathbb{E}[Y_i]=1$ and $\mathrm{Var}(Y_i)=\mathbb{E}[Y_i^2]-1=\frac{1-p}{p}$.
+
+### 7.1 DDSketch — inverse-probability bucket update
+Bucket $b(x)=\lceil\log x/\log\gamma\rceil$, $\gamma=\frac{1+\alpha}{1-\alpha}$;
+representative relative error $\alpha$. Update: for $x_i$, add $Y_i$ to
+$\widehat C[b_i]$ (i.e. $+1/p$ w.p. $p$). Sampled count
+$\widehat n_b=\sum_{i:b(x_i)=b}Y_i$.
+- **Unbiased:** $\mathbb{E}[\widehat n_b]=\sum_{i}\mathbb{E}[Y_i]=n_b$.
+- **Prefix concentration** (quantiles use prefix counts $N_{\le b}$):
+  $\mathrm{Var}(\widehat N_{\le b})=N_{\le b}\frac{1-p}{p}$. Bernstein + union over
+  $B$ buckets ⇒ w.p. $1-\delta$,
+  $\sup_b|\widehat N_{\le b}-N_{\le b}|=O\!\big(\sqrt{N\log(B/\delta)/p}+\log(B/\delta)/p\big)$.
+  Rank error $\epsilon_s=O\!\big(\sqrt{\log(B/\delta)/(pN)}+\log(B/\delta)/(pN)\big)$.
+- **Quantile bound:** $\;\tilde x_q\in(1\pm\alpha)\,x_{q\pm\epsilon_s}\;$ w.p.
+  $1-\delta$. The $\alpha$ now applies to $x_{q\pm\epsilon_s}$, not $x_q$ (the
+  pure relative-value guarantee is only preserved when $x_{q\pm\epsilon_s}$ is
+  near $x_q$). Cost: bucket writes/item $1\to p$.
+
+### 7.2 KLL — weighted update sampling
+Original rank error $\epsilon_k=O(\frac1k\sqrt{\log(1/\delta)})$. Update: w.p.
+$p$ insert $x_i$ with weight $1/p$ (a virtual level-$(-1)$ compactor; at level
+$h$ the weight is $2^h/p$); compact normally.
+- **Weighted CDF unbiased:** $\widehat F_s(t)=\frac1N\sum_i\frac{Z_i}{p}\mathbf 1\{x_i\le t\}$,
+  $\mathbb{E}[\widehat F_s(t)]=F(t)$.
+- **Concentration:** $\mathrm{Var}(\widehat F_s(t))=\frac{N_t}{N^2}\frac{1-p}{p}\le\frac{1-p}{pN}$;
+  DKW/union over breakpoints ⇒ $\sup_t|\widehat F_s-F|\le\epsilon_s=O(\sqrt{\log(1/\delta)/(pN)})$.
+- **Combined:** $\sup_t|\widehat F_{KLL}-F|\le\underbrace{|\widehat F_{KLL}-\widehat F_s|}_{\le\,\epsilon_k}+\underbrace{|\widehat F_s-F|}_{\le\,\epsilon_s}$, so
+  $\;\tilde x_q\in[x_{q-(\epsilon_k+\epsilon_s)},\,x_{q+(\epsilon_k+\epsilon_s)}]$.
+  (Martingale view: sampling adds one more zero-mean term to KLL's compaction-error
+  variance budget.)
+- **Design balance:** set $\epsilon_s\approx\epsilon_k$ with
+  $\epsilon_s\approx 1/\sqrt{pN}$, $\epsilon_k\approx 1/k$ ⇒ **$pN\approx k^2$** —
+  the effective number of sampled updates should be $\gtrsim k^2$ or sampling
+  dominates. Cost: insertions/item $1\to p$; memory $O(k\log(pN))$.
+
+### 7.3 HLL — hash-threshold sampling
+RSE $\approx 1.04/\sqrt m$. The update is $R[j]\leftarrow\max(R[j],\rho(x))$ —
+**a max, not additive**, so inverse-probability register updates are INVALID.
+Algorithm: keep a distinct key iff $u(h(x))<p$ (a stable hash ⇒ the same key
+always gets the same decision); update normally; estimate $\widehat n=\widehat n_s/p$.
+- **Why not per-occurrence:** a freq-$f_x$ key would be kept w.p. $1-(1-p)^{f_x}$
+  — frequency-dependent, hence biased for distinct counting. Hash-threshold gives
+  every distinct key the same $\Pr[\text{kept}]=p$.
+- **Unbiased:** $n_s=\sum_{x\in D}I_x$, $I_x\sim\mathrm{Bernoulli}(p)$ ⇒
+  $\mathbb{E}[n_s/p]=n$.
+- **Concentration:** $n_s\sim\mathrm{Binomial}(n,p)$; Chernoff ⇒
+  $\epsilon_s=\sqrt{3\log(2/\delta)/(pn)}$.
+- **Combined RSE:** $\frac{|\widehat n-n|}{n}\lesssim\epsilon_s+\epsilon_h$, i.e.
+  $\;\mathrm{RSE}\approx\sqrt{\frac{1-p}{pn}+\frac{1.04^2}{m}}$ (sampling term +
+  HLL term). Cost: register writes/item $\to p$.
+
+### 7.4 Nitro / additive counters (CMS, CountSketch)
+Update $C_{r,h_r(x)}\mathrel{+}=a_r(x)$ ($a=w$ for Count-Min; $a=s_r(x)w$,
+$s_r\in\{\pm1\}$, for CountSketch). Sampled: add $\frac{Z}{p}a$.
+- **Unbiased counters:** $\mathbb{E}[\widehat C_{r,c}]=C_{r,c}$ (the core
+  NitroSketch argument).
+- **Variance:** for a counter with updates $a_1..a_T$,
+  $\mathrm{Var}(\widehat C-C)=\frac{1-p}{p}\sum_t a_t^2$; Bernstein ⇒
+  $|\widehat C-C|=O\!\big(\sqrt{\frac{1-p}{p}(\sum a_t^2)\log(1/\delta)}+\frac{a_{\max}}{p}\log(1/\delta)\big)$.
+- **CountSketch:** median over $d$ rows; total error = hash-collision + sampling;
+  per-row variance gains $\frac{1-p}{p}\sum_{i:h_r(x_i)=h_r(x)}w_i^2$; the median
+  still drops failure prob exponentially in $d$.
+- **Count-Min:** the deterministic no-underestimate property is LOST (self-updates
+  may be skipped); analyze as a biased-up collision estimator + zero-mean sampling
+  noise.
+- **Fixed-$s$-row variant:** select exactly $s$ of $d$ rows per item, scale by
+  $d/s$; unbiased with a deterministic write count $s$ (set $p=s/d$). Cost:
+  counter writes/item $d\to pd$ (or fixed $s$).
+
+### 7.5 Design rules
+- **Additive update ⇒ inverse-probability update sampling** (DDSketch, CMS,
+  CountSketch, Nitro).
+- **Max update ⇒ threshold sampling + final rescale** (HLL).
+- **Compaction update ⇒ weighted sampling + weighted rank analysis** (KLL).
+
+All four are unbiased; the added error is the $\epsilon_s$ / variance term above,
+which §6.4 folds into the per-metric accuracy budget (and §6's composition rule —
+store the raw sampled integer state + global `p`, rescale at query — keeps these
+estimators compressible).

From 4ef760fb2a0687bc6aa1832cdab3c488918eac8c Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:46:03 -0600
Subject: [PATCH 08/12] =?UTF-8?q?docs(design):=20=C2=A76.5=20early-benefit?=
 =?UTF-8?q?s=20results=20from=20the=20offline=20Go=20sampling=20benchmark?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Measured vs real sketchlib-go (harness /mydata/sampling-bench): all §7 bounds
held. p=0.1 → ~10× fewer updates + ~6× wall-clock per family. KLL pN≳k² confirmed
as a hard boundary; HLL hash-threshold is unbiased to p=0.01 (n≥1e5) while
per-occurrence blows up (0.66→3.8) — and the threshold hash MUST be independent
of HLL's register hash (else 15–84% bias). Sampled serialization never bloats
(composition rule holds). Caveats: HLL m is compile-time const; sparser-HLL win
only below register saturation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 35 +++++++++++++++++--
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index e6055ecf8..729fc5f62 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -359,9 +359,38 @@ fraction). `p` rides in the Full-epoch frame header alongside the offset.
   controller-driven model exactly — the controller already annotates each
   metric's tier + sketch type; it adds `p` the same way.
 
-> §6.5 (the offline Go sampling benchmark proving these benefits empirically —
-> CPU/update reduction vs measured error vs the bounds below) is added once the
-> benchmark lands; §7 is its formal backing.
+### 6.5 Early benefits — offline Go benchmark (pre-integration)
+
+Measured against the REAL sketchlib-go sketches (harness `/mydata/sampling-bench`,
+`go run .`, ~16s; UNSAMPLED vs SAMPLED-at-`p` vs EXACT ground truth, with the
+§6.1 composition rule applied — raw sampled state stored, `×1/p` at query). **All
+§7 bounds held empirically** across the `p`/`N`/`k`/`m` sweeps.
+
+| family | benefit @ `p=0.1` | accuracy | safe-`p` |
+|---|---|---|---|
+| DDSketch (α=1%) | 10× fewer bucket writes (1e6→1e5), ~6× wall-clock (239→34 ms) | q99 rank err ~0.005 (within ε_s); value relErr ≈ α | `p`≈0.05–0.1 (q99 suffers first at tiny `p`) |
+| KLL | 10× fewer inserts, ~6× wall-clock | rank err tracks ε_s **iff `pN≳k²`**; below it q99 jumps (N=1e6,k=400,p=0.1 → pN<k² → relErr 0.225) | **`pN ≳ k²`** |
+| HLL (m=16384) | up to ~100× fewer updates (`p=0.01`) | UNBIASED: relErr 0.017 vs RSE bound 0.013 @ n=1e6 | `p` down to 0.01 for n≥1e5; **hash-threshold only** |
+| CMS / CountSketch | 10× fewer updates | heavy-hitter relErr ~1–2% @ N≥1e5; per-key err ~√(f(1−p)/p) | `p`≈0.1 |
+
+Findings:
+- **CPU benefit is real and ~linear in `p`** (updates ∝ `p`; ~6× wall-clock at
+  `p=0.1`) — exactly the build-cost axis compression can't touch.
+- **`pN≳k²` for KLL is a hard regime boundary**, confirmed empirically (not just
+  asymptotic): cross below it and q99 error jumps.
+- **HLL must use hash-threshold, NEVER per-occurrence** (per-occurrence relErr
+  blew up 0.66→3.8 by over-counting high-multiplicity keys). Implementation
+  gotcha: the threshold hash MUST be **independent of HLL's canonical register
+  hash** — using the same hash correlated the kept set with the register layout
+  and gave 15–84% bias until a separate seed was used.
+- **Serialization composes (§6.1 rule):** the sampled proto state is never larger
+  than unsampled — DDSketch sparser buckets, KLL fewer retained items, HLL/CMS
+  smaller below register saturation. Sampling doesn't bloat the wire.
+
+Caveats (documented in the harness README): HLL precision `m` is a compile-time
+const in sketchlib-go (the `m`-sweep is analytical in the RSE column); the HLL
+"sparser→smaller" win only holds below register saturation (distinct ≪ `m`) — at
+n=1e6 registers saturate and sampled/unsampled sizes converge.
 
 ---
 

From fa0e6d64e4338f44a4b3fd5031a658c8a8e01345 Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 10:59:55 -0600
Subject: [PATCH 09/12] =?UTF-8?q?docs(design):=20=C2=A76.6=20geometric=20s?=
 =?UTF-8?q?ampling=20(NitroSketch)=20note?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Geometric skip-ahead is a statistically-identical, cheaper implementation of
per-update Bernoulli(p) (gaps ~Geometric(p) ⟺ iid keep w.p. p): same §7 bounds /
composition rule, amortizes sampling RNG to ~O(p)/item. Applies to the per-update
families (DDSketch/KLL/CMS/CountSketch/Nitro), NOT HLL (hash-threshold per key,
already RNG-free). Orthogonal to shared-ts and the offset — it touches neither
timestamps nor the value base.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index 729fc5f62..3e9e55ed4 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -392,6 +392,30 @@ const in sketchlib-go (the `m`-sweep is analytical in the RSE column); the HLL
 "sparser→smaller" win only holds below register saturation (distinct ≪ `m`) — at
 n=1e6 registers saturate and sampled/unsampled sizes converge.
 
+### 6.6 Geometric sampling (NitroSketch) — cheap equivalent implementation
+Per-update Bernoulli(p) (a coin per update) wastes RNG. NitroSketch's
+**geometric sampling** instead draws, after each kept update, a `Geometric(p)`
+number of updates to SKIP, and jumps ahead. This is **statistically identical**
+to per-update Bernoulli(p) (gaps `~Geometric(p)` ⟺ each update kept iid w.p. `p`),
+so §7's bounds, unbiasedness, and the §6.1 composition rule are all unchanged —
+it only amortizes the sampling RNG to ~`O(p)` draws per item (and drops the
+per-item branch), on top of the "fewer sketch updates" win.
+- **Applies to the per-update/per-item families:** DDSketch (bucket updates),
+  KLL (per-item insert), CMS / CountSketch / Nitro (counter updates).
+- **NOT HLL:** HLL samples by hash-threshold on the distinct KEY
+  (`u(h(x))<p`) — a value-determined decision, not a stream-position coin — so
+  geometric skip-ahead doesn't apply; and its hash-compare is already O(1) with
+  no RNG, so it needs no such optimization.
+- **Orthogonal to shared-ts (§1.7) and the offset (§4)** — it changes only HOW
+  the kept set is generated and WHICH counter updates land, never timestamps or
+  the value base: (a) cold raw isn't sampled at all; (b) warm "shared-ts" is the
+  window-end column, fixed by the window cadence which sampling doesn't change;
+  (c) the offset comes from the per-series value magnitude, which a uniform
+  random `p`-subset preserves.
+- **Impl note:** over a multi-series interleaved stream a single global geometric
+  counter gives each series ~`p` in expectation (with per-series variance);
+  per-series warm sketches typically keep a per-series counter.
+
 ---
 
 ## 7. Sampling-enhanced sketches: algorithms & error-bound derivations

From 43889552163a28b6890346574950438dde4478bf Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 11:05:27 -0600
Subject: [PATCH 10/12] =?UTF-8?q?docs(design):=20=C2=A78=20implementation?=
 =?UTF-8?q?=20plan=20/=20PR=20sequence?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

4 phases, ordered by isolation+ROI+dependency+risk. Parallel start set PR1
(HLL-sparse) ∥ PR2 (KLL-offset) ∥ PR5 (cold INT codec lib); critical path
PR5→PR6(StoreAPI)→PR7(shared-ts). Sampling (PR3 gated no-op + PR4 controller p)
in phase 2; offset unification + warm drift→Full in phase 4. Format-changing PRs
use additive proto fields + dual-read for staggered rollout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md      | 47 +++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index 3e9e55ed4..e35a8715c 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -510,3 +510,50 @@ All four are unbiased; the added error is the $\epsilon_s$ / variance term above
 which §6.4 folds into the per-metric accuracy budget (and §6's composition rule —
 store the raw sampled integer state + global `p`, rescale at query — keeps these
 estimators compressible).
+
+---
+
+## 8. Implementation plan / PR sequence
+
+Ordered by isolation + ROI + dependency + risk. Wire/format-changing PRs ship a
+new (proto-additive) field + dual-read so agents and backend can deploy at
+different times; each is validated on-cluster via the auto build-and-load deploy
+(marquee + memory + the metric's accuracy SLA). `∥` = parallelizable.
+
+**Phase 1 — warm sketch serialization (isolated, audit-proven; relieves warm memory)**
+- **PR1 — HLL sparse full-state** (§2.1 P1): sketchlib-go encode + asap_sketchlib
+  decode, additive proto field, dual-read. ROI 5–50× + cuts warm SketchStore
+  memory. No deps.
+- **PR2 — KLL value-offset serialize** (§2.1 P2): `(v−offset)` fixed-point +
+  scale/offset in the sketch header. ROI ~2×. `∥ PR1` (separate sketch + proto
+  message).
+
+**Phase 2 — sampling (CPU; builds on the sketch format + composition rule)**
+- **PR3 — sampling layer in sketchlib-go** (§6/§7): per-family inverse-prob /
+  hash-threshold / weighted + geometric (§6.6) + composition rule (header stores
+  raw sampled int + `p`; backend ×1/p at query). Gated by per-sketch `p`,
+  default `p=1` (no-op). 
+- **PR4 — controller per-metric `p` knob** (§6.4): control-plane annotates +
+  pushes `p` (like tier/sketch-type); agent applies. Activates PR3. Deps PR3.
+
+**Phase 3 — cold custom codec + read path**
+- **PR5 — cold INT best-of-N codec library** (§1.2–1.4): `{Gorilla-XOR,
+  INT_FOR_DELTA, INT_FOR_DOD}` + decimal-exactness guard + chunk header +
+  overflow chunk-cut. Standalone lib + lossless round-trip tests; no deploy. ROI
+  4.8×. No deps — `∥ Phase 1/2`.
+- **PR6 — decode-on-read StoreAPI** (§1.5–1.6): gorilla-merger stores custom
+  chunks (write = no decode) + decodes → XOR `AggrChunk` at query; coexists with
+  existing gorilla blocks. Deps PR5.
+- **PR7 — shared-ts grouped layout** (§1.7): same-metric series share one ts
+  column per part. ROI −43% cross-series. Deps PR5/PR6.
+
+**Phase 4 — offset unification + warm delta-with-offset**
+- **PR8 — parse-once shared scale/stats (§4) + warm drift→Full + Delta-frame
+  epoch-offset (§3.1 v1)**: ties KLL-offset (PR2) into delta-transmission with an
+  epoch-bound offset; adds drift→Full (hard-overflow + fixed heartbeat,
+  zero-tuning). Deps PR2.
+
+Parallel start set (no deps): **PR1 ∥ PR2 ∥ PR5**. Critical path: PR5→PR6→PR7.
+Rationale: warm-serialize first (isolated, proven, relieves the live warm-memory
+concern); sampling next (the CPU axis, gated no-op so it merges safely); the cold
+codec migration (biggest, riskiest) in phase 3; offset unification last.

From 9cc4b37ec28026513dac2212a9f340edc4f90948 Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 12:07:26 -0600
Subject: [PATCH 11/12] =?UTF-8?q?docs(design):=20correct=20cold-codec=20he?=
 =?UTF-8?q?adline=20to=20no-zstd=20~2.2-3.6=C3=97=20(PR=20#434=20measured)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The 4.8× came from VictoriaMetrics lib/encoding WITH zstd-wrapping; since we
decided no-zstd (§5), the realized win is the no-zstd FOR+delta codec measured
in the intchunk PR: ~2.2-3.6× on fixed-decimal. A varint-residual refinement is
being added to push toward ~3×+ without zstd. §0 + §8 updated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 docs/design/holistic-edge-backend-compression.md | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index e35a8715c..d70511232 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -28,8 +28,14 @@ Hard requirements:
 
 Benchmark headline (real Chimp/Serf datasets, lossless, block-avg 1000/chunk):
 - Fixed-decimal series (11/12 datasets — temps, stocks, sensors, pressure,
-  GPS, dust, wind, grid): **VM-style integer FOR+delta beats Gorilla ~4.8×
-  avg (up to ~10×)**. This *is* the offset idea, on the integer-scaled values.
+  GPS, dust, wind, grid): integer FOR+delta beats Gorilla. This *is* the offset
+  idea, on the integer-scaled values. **NOTE on the magnitude:** the
+  VictoriaMetrics `lib/encoding` number (~4.8× avg) included zstd-wrapping on
+  some series; since we decided **no zstd** (§5), the realized win is the
+  **no-zstd FOR+delta codec measured in PR #434 (`asap-gorilla-go/intchunk`):
+  ~2.2–3.6× on fixed-decimal** (City-temp 2.8×, Dew-point 2.9×, Stocks 2.7×,
+  Wind 3.6×). A `varint`-residual refinement (vs fixed-width packing) is being
+  added to push toward ~3×+ without zstd.
 - Genuinely high-precision float (float32-derived, 15 sig digits): VM can't
   stay decimal-exact → falls back to bit-pattern (worse); **Gorilla-XOR wins**.
 - ⇒ The codec must be a per-block **best-of-N including Gorilla-XOR**, not
@@ -540,7 +546,8 @@ different times; each is validated on-cluster via the auto build-and-load deploy
 - **PR5 — cold INT best-of-N codec library** (§1.2–1.4): `{Gorilla-XOR,
   INT_FOR_DELTA, INT_FOR_DOD}` + decimal-exactness guard + chunk header +
   overflow chunk-cut. Standalone lib + lossless round-trip tests; no deploy. ROI
-  4.8×. No deps — `∥ Phase 1/2`.
+  ~2.2–3.6× no-zstd (PR #434, merged-pending; varint-residual refinement to
+  reach ~3×+). No deps — `∥ Phase 1/2`.
 - **PR6 — decode-on-read StoreAPI** (§1.5–1.6): gorilla-merger stores custom
   chunks (write = no decode) + decodes → XOR `AggrChunk` at query; coexists with
   existing gorilla blocks. Deps PR5.

From 6a3250086c979b518a98b456a02582bc85e120d7 Mon Sep 17 00:00:00 2001
From: zz_y <zeyingz@umd.edu>
Date: Mon, 25 May 2026 12:14:17 -0600
Subject: [PATCH 12/12] docs(design): finalize cold-codec ROI to ~2.33x
 (no-zstd best-of-N, PR #434)

Varint-residual sub-codecs landed in the intchunk best-of-N; on the Chimp/Serf
fixed-decimal class the aggregate moved 2.22x -> 2.33x vs Gorilla (up to ~3.6x
per series). Bottom line: no-zstd cold compression caps ~2.3x; the 4.8x needs
zstd (excluded). Gorilla-XOR stays the lossless fallback for true floats.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---
 .../holistic-edge-backend-compression.md       | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md
index d70511232..3a78c69e7 100644
--- a/docs/design/holistic-edge-backend-compression.md
+++ b/docs/design/holistic-edge-backend-compression.md
@@ -32,10 +32,14 @@ Benchmark headline (real Chimp/Serf datasets, lossless, block-avg 1000/chunk):
   idea, on the integer-scaled values. **NOTE on the magnitude:** the
   VictoriaMetrics `lib/encoding` number (~4.8× avg) included zstd-wrapping on
   some series; since we decided **no zstd** (§5), the realized win is the
-  **no-zstd FOR+delta codec measured in PR #434 (`asap-gorilla-go/intchunk`):
-  ~2.2–3.6× on fixed-decimal** (City-temp 2.8×, Dew-point 2.9×, Stocks 2.7×,
-  Wind 3.6×). A `varint`-residual refinement (vs fixed-width packing) is being
-  added to push toward ~3×+ without zstd.
+  **no-zstd best-of-N codec measured in PR #434 (`asap-gorilla-go/intchunk`):
+  ~2.33× aggregate on fixed-decimal** (up to ~3.6× per series — Wind 3.6×,
+  Dew-point 2.9×, City-temp 2.8×, Stocks 2.7×). The codec tries fixed-width AND
+  zigzag-varint residuals (×{delta, delta-of-delta}) plus Gorilla and keeps the
+  smallest; varint helps skewed blocks but only nudged the aggregate (2.22→2.33×).
+  **Bottom line: no-zstd cold compression caps ~2.3×; the 4.8× genuinely needs
+  zstd, which we excluded.** Gorilla-XOR remains the lossless fallback for true
+  high-precision floats.
 - Genuinely high-precision float (float32-derived, 15 sig digits): VM can't
   stay decimal-exact → falls back to bit-pattern (worse); **Gorilla-XOR wins**.
 - ⇒ The codec must be a per-block **best-of-N including Gorilla-XOR**, not
@@ -545,9 +549,9 @@ different times; each is validated on-cluster via the auto build-and-load deploy
 **Phase 3 — cold custom codec + read path**
 - **PR5 — cold INT best-of-N codec library** (§1.2–1.4): `{Gorilla-XOR,
   INT_FOR_DELTA, INT_FOR_DOD}` + decimal-exactness guard + chunk header +
-  overflow chunk-cut. Standalone lib + lossless round-trip tests; no deploy. ROI
-  ~2.2–3.6× no-zstd (PR #434, merged-pending; varint-residual refinement to
-  reach ~3×+). No deps — `∥ Phase 1/2`.
+  overflow chunk-cut + zigzag-varint residual sub-codecs in the best-of-N.
+  Standalone lib + lossless round-trip tests; no deploy. ROI ~2.33× aggregate
+  no-zstd (PR #434). No deps — `∥ Phase 1/2`.
 - **PR6 — decode-on-read StoreAPI** (§1.5–1.6): gorilla-merger stores custom
   chunks (write = no decode) + decodes → XOR `AggrChunk` at query; coexists with
   existing gorilla blocks. Deps PR5.