feat: CMS/CS bandwidth optimisations — full-sketch Opt-1/2/3, delta packed-array encoding, adaptive transmission

[zzylol]

Problem

CMS and CS full-sketch payloads were larger than raw samples before this PR. Even delta transmission made it worse — delta cells used fixed 64-bit floats, so a sparse delta could cost more than a full sketch:

Sketch	Raw B/win	Full B/win	Delta B/win	Raw/Full	Raw/Delta
CMS (5×2048)	40 000	246 003	150 014	0.16×	0.27×
CS (ε=0.01)	40 000	655 554	87 507	0.06×	0.46×
HLL (16k regs)	40 000	16 532	8 543	2.42×	4.68×
DD (α=0.01)	40 000	882	—	45×	—
KLL (k=256)	40 000	3 074	3 074	13×	13×

Workload: deltaaccbench — 20 windows × 5 000 inserts, Zipf s=1.10, threshold=1.0. Raw baseline = 5 000 × 8 B uint64 hashes = 40 000 B/window.

Root causes and full analysis: docs/cms-cs-delta-transmission-optimizations.md

---

What was implemented

Six independent optimisations, applied in three layers:

Layer 1 — Full-sketch payload (sketchlib-go#45, DataCollector)

Opt	Change	Where
Opt-1	SerializePortableFO() — omit sum_counts/sum2_counts from CMS full payload; receiver reconstructs Sum = Sum2 = Count for unweighted streams (3× fewer arrays)	sketchlib-go CountMinSketch/portable.go
Opt-2	counts_int sint64 (packed zigzag varint) replaces counts_float double for both CMS and CS; Zipf cells average 1–2 B instead of 8 B	sketchlib-go CountMinSketch/portable.go, CountSketch/portable.go
Opt-3	CS epsilon: 0.01 → 0.02 — halves the error bound relative to raw while reducing the matrix 4× (16 384 → 4 096 columns)	9 YAML configs in cmd/countsketchcol/

Layer 2 — Delta-cell encoding (sketchlib-go@019e21e, DataCollector)

Opt	Change	Where
Opt-5	Replace repeated CountMinCell / repeated CountSketchCell proto messages with three parallel packed arrays (cell_rows uint32, cell_cols uint32, d_counts sint64); drops d_sum/d_sum2 from CMS delta cells (same reconstruction as Opt-1). Per-cell cost: 35 B → 4 B for CMS, 17 B → 4 B for CS. Backward-compat: old cells_legacy field retained for reading old producers.	sketchlib-go proto/countminsketch/, proto/countsketch/, */delta_codec.go, */delta.go

Layer 3 — Adaptive transmission (DataCollector)

Fix	Change	Where
Opt-6	Both CMS and CS processors compute both full and delta payloads, then transmit whichever is smaller. Eliminates any scenario where delta costs more than full, regardless of workload fill rate.	processor/countminsketchprocessor/processor.go, processor/countsketchprocessor/processor.go

Other changes

Change	Where
Opt-4	CS TopK: SpaceSaving tracker replaces TopKHeap.UpdateCS; delta sends hh_keys (key strings only, no stale counts); downstream queries merged matrix for accurate globally-merged estimates. O(k) per insert → O(log k).
Fix cloneCMS	Used DeserializeCountMinSketchFromBytes (gob) on proto bytes → nil clone → nil-pointer panic on 2nd window
Fix HLL API	InsertValue → Insert, EstimateCardinality → Estimate (sketchlib-go API rename)
Dep bump	sketchlib-go pseudoversion → 019e21e25cce + local replace directives in all 22 go.mod files

---

Results

deltaaccbench — before vs after (20 windows × 5 000 inserts, Zipf s=1.10, threshold=1.0)

Sketch	Payload	Before	After	Reduction	Opt
CMS	full	246 003 B	10 950 B	22.5×	Opt-1+2: drop sum/sum2, sint64 varint
CMS	delta	150 014 B	17 755 B	8.5×	Opt-5: packed arrays, drop d_sum/d_sum2
CMS	wire (adaptive)	150 014 B	10 950 B	13.7×	Opt-6: full is smaller at 91% fill rate
CS	full	655 554 B	82 526 B	7.9×	Opt-2+3: sint64 varint + 4× fewer cols
CS	delta	87 507 B	22 500 B	3.9×	Opt-5: packed arrays
CS	wire (adaptive)	87 507 B	22 500 B	3.9×	Opt-6: delta wins over full

CMS fill rate note: At 5 000 inserts/window into 5×2048 the fill rate is ~91% — nearly every cell changes. Even at 4 B/cell (packed), 4 500 cells × 4 B ≈ 18 KB exceeds the 11 KB FO full. Opt-6 (adaptive) ensures the smaller option is always sent. In the production deployment (500 series × 100 inserts/window, ~22% fill rate), the CMS delta is sparse and beats full.

All sketches after optimisation:

Accuracy measured by deltaaccbench: mean and max relative query error vs ground truth over 20 windows. Accuracy is a property of the sketch algorithm and dimensions, not of the transmission encoding.

Sketch	Mode	Delta	Raw B/win	Full B/win	Delta B/win	Raw/Full	Raw/Delta	Mean Rel Err	Max Rel Err
CMS (5×2048)	batch	off	40 000	10 521	—	3.80×	—	0.16%	18.75%
CMS (5×2048)	batch	on (adaptive)	40 000	10 521	20 106	3.80×	1.99×	0.16%	18.75%
CMS (5×2048)	window	on (adaptive)	40 000	10 950	17 755	3.65×	2.25×	4.70%	150.00%
CS (ε=0.02)	batch	on	40 000	82 161	30 001	0.49×	1.33×	0.006%	7.14%
CS (ε=0.02)	window	on	40 000	82 526	22 500	0.48×	1.78×	0.091%	40.00%
HLL (16k regs)	window	on	40 000	16 532	8 543	2.42×	4.68×	0.73%	1.34%
DD (α=0.01)	window	off	40 000	882	—	45×	—	0.29%	0.82%
KLL (k=256)	batch	off	40 000	2 959	—	13.5×	—	0.26%	1.21%

CMS window mode shows high max relative error (150%) because the cumulative sketch is queried across 20 windows — most keys in later windows are rare and their absolute count is small, so even a 1-count estimation error produces a large relative error. The mean (4.70%) is dominated by the heavy hitters where CMS is accurate.

CPU & memory overhead — delta=on vs delta=off (deltaaccbench)

Sketch	Mode	Wall overhead	CPU user overhead	Heap delta
CMS	batch	+125%	+137%	+25%
CMS	window	+53%	+31%	+20%
CS	batch	+39%	+29%	−27%
CS	window	+37%	+37%	−15%
HLL	window	+72%	+85%	−5%

Real-OTel deployment — bench_4modes.sh (500 series × 10 sps, 60 s run)

OTel SDK → Agent Collector (:4317) → Backend nopcol (:4319). SDK→Agent bandwidth is identical across all modes (~270–287 KB/s) — the optimisation is entirely in the agent→backend hop.

Note: OTel SDK Float64Gauge uses LastValue aggregation — only the most recent value per series is exported each interval. "raw-batched(10s)" forwards ten 1-second exports at once; it does not recover intra-interval samples, hence only ~10% savings.

Bandwidth

Sketch	Mode	Agent→Backend KB/s	Reduction vs raw
CMS	raw-unbatched(1s)	23.1	baseline
CMS	raw-batched(10s)	21.0	1.1×
CMS	cms-full	2.6	9×
CMS	cms-delta	1.5	16×
CS	raw-unbatched(1s)	23.1	baseline
CS	raw-batched(10s)	21.0	1.1×
CS	cs-full	0.86	27×
CS	cs-delta	0.86	27×

CPU & memory

Sketch	Mode	Agent CPU avg	Agent Mem avg MB	Backend Mem avg MB
CMS	raw-unbatched(1s)	3.5%	191	33
CMS	raw-batched(10s)	3.2%	195	43
CMS	cms-full	7.4%	418	67
CMS	cms-delta	11.3%	752 ⚠️	44
CS	raw-unbatched(1s)	2.9%	43	35
CS	raw-batched(10s)	2.5%	50	42
CS	cs-full	2.6%	37	30
CS	cs-delta	2.6%	37	30

Key takeaways:

• CS (full or delta) achieves 27× bandwidth reduction at only 37 MB agent memory — best overall
• CMS delta achieves 16× reduction but costs 752 MB agent memory for 500 series (per-series snapshot storage for 10 s windows)
• CS memory efficiency is largely due to Opt-3 (ε=0.02 reduces matrix 4×)
• Raw batching saves ~10% — confirms sketch-based aggregation is the only meaningful approach

---

CS TopK: Space Saving vs prior heap

Property	Before (heap UpdateCS)	After (Weighted Space Saving)
Existing key update	Always +1 — ignores weight	+= w — correct for any weight
New key admission	Only if CS estimate > min; otherwise silently dropped	Always — evicts min, new entry starts at min + w
Find cost	O(k) linear scan	O(1) hash map
Total per insert	O(k)	O(log k)
Counts forwarded	Stale upstream-local estimates	None — downstream queries merged matrix
Formal guarantee	None	Any key with total weight > W/k is tracked

---

Test plan

• go build ./cmd/deltaaccbench passes (sketchlib-go@019e21e)
• go run ./cmd/deltaaccbench --windows=20 --inserts=5000 pre-opt baseline reproduced
• go run ./cmd/deltaaccbench --windows=20 --inserts=5000 post-opt retest (2026-03-30)
• sketchlib-go go test ./sketches/CountSketch/... ./sketches/CountMinSketch/... ./sketches/SpaceSaving/... pass
• cmd/countminsketchcol/dist binary builds clean
• cmd/countsketchcol/dist binary builds clean
• bench_4modes.sh --sketch cms --duration 60s completes (4 modes × CMS)
• bench_4modes.sh --sketch cs --duration 60s completes (4 modes × CS)

🤖 Generated with Claude Code