Skip to content

feat(delta): per-window delta producer for CMS/CountSketch/HLL (delta-against-empty) - #457

Merged
zzylol merged 1 commit into
mainfrom
feat/cms-cs-hll-delta-producer
May 26, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/cms-cs-hll-delta-producer

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

What

Extends the per-window delta-against-empty producer (landed for DDSketch in #456) to CountMinSketch, CountSketch, and HLL on both edge runtimes (asap-precompute-go and asap-precompute-rs). KLL stays full-only; DDSketch is unchanged.

Each window now emits a delta computed against an empty base, so the wire payload carries that window's own per-window state with no cross-window subtraction — the prerequisite for correct window-scoped backend aggregation.

How

asap-precompute-go (sketches/{cms,countsketch,hll}.go):

  • Each wrapper implements DeltaAgainstEmptyBase(), returning the encoded empty-sketch snapshot the SnapshotCache caches as the outbound base after each window-close emit. The next window's ComputeDeltaAgainst (already wired to the sketchlib-go ComputeDelta for each family) then diffs against the empty base.
  • CMS msgpack mode (which cannot carry deltas) returns a nil base to keep legacy always-refresh. SnapshotCache.ComputeDelta now treats a nil/empty empty-base as "fall through to legacy refresh", so a family can opt in conditionally.

asap-precompute-rs (src/sketches/{cms,countsketch,hll}.rs):

  • Each wrapper implements delta_against_empty_base() and a real compute_delta_against that decodes the prior envelope and diffs via the asap_sketchlib compute_delta for that family (CMS/CountSketch cell deltas, HLL register delta), with a full-snapshot fallback on an empty/undecodable prior.
  • apply_delta now dispatches on payload shape (full-state envelope vs raw delta proto via apply_delta_bytes), mirroring the DDSketch wrapper. Stale "always-full" caveats in sketches/mod.rs / precompute.rs doc comments updated.

The SnapshotCache dispatch was already sketch-agnostic (any sketch implementing the empty-base interface opts in; others fall back to legacy always-refresh), so no dispatch logic changed — only the per-family opt-in and the conditional nil-base fall-through.

Tests

Mirroring DDSketch's tests, for each of CMS / CountSketch / HLL:

  • applying the emitted delta to an empty base reconstructs the window's state (matrix / registers / estimate);
  • two consecutive windows each emit their own state (no cross-window subtraction). HLL's disjoint-window test pins that the empty-base reset defeats register-MAX leakage (delta-baseline-contract.md §1.5 / §2.3).

Go: sketches/{cms,countsketch,hll}_test.go. Rust: src/snapshot_cache.rs. Go go build/go vet/go test ./sketches/... and Rust cargo build/cargo test are green (Rust built against the asap_sketchlib branch carrying #59).

Byte-parity note: Go↔Rust per-window delta-frame byte parity is not directly asserted in this PR (no in-tree golden delta fixtures), but the underlying delta codec is the byte-parity twin verified by asap_sketchlib#59's Go-golden envelope probes; each Rust compute_delta is documented as byte-identical to the sketchlib-go proto.Marshal output for the same inputs.

Dependencies

🤖 Generated with Claude Code

DDSketch already opts in to per-window delta-against-empty (#456). This
extends the same opt-in to CountMinSketch, CountSketch, and HLL on both
edge runtimes; KLL stays full-only and DDSketch is unchanged.

asap-precompute-go (sketches/{cms,countsketch,hll}.go): each wrapper now
implements DeltaAgainstEmptyBase(), returning the encoded empty-sketch
snapshot the SnapshotCache caches as the outbound base after each
window-close emit. The next window's ComputeDeltaAgainst (already wired
to the sketchlib-go ComputeDelta for each family) then diffs against an
empty base, so each window's delta carries that window's own per-window
state — no cross-window subtraction. CMS msgpack mode (which cannot
carry deltas) returns a nil base to keep legacy always-refresh; the
SnapshotCache now treats a nil/empty empty-base as "fall through to
legacy refresh".

asap-precompute-rs (src/sketches/{cms,countsketch,hll}.rs): each wrapper
implements delta_against_empty_base() and a real compute_delta_against
that decodes the prior envelope and diffs via the asap_sketchlib
compute_delta for that family (CMS/CountSketch cell deltas, HLL register
delta), with a full-snapshot fallback on empty/undecodable prior. Each
apply_delta now dispatches on payload shape (full-state envelope vs raw
delta proto) via apply_delta_bytes, mirroring the DDSketch wrapper. The
stale sketches/mod.rs "always-full" caveat is updated.

Tests (mirroring DDSketch's): for each family, applying the emitted
delta to an empty base reconstructs the window's state, and consecutive
windows emit their own state (no cross-window subtraction) — Go in
sketches/*_test.go, Rust in snapshot_cache.rs. HLL's disjoint-window
test pins that the empty-base reset defeats register-MAX leakage.

Depends on asap_sketchlib#59 (CMS/CountSketch/HLL compute_delta /
apply_delta_bytes; the underlying codec is the byte-parity twin of the
sketchlib-go delta codec). The backend base rotation (#346) is already
sketch-agnostic, so no backend change is needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit aed3688 into main May 26, 2026
@zzylol
zzylol deleted the feat/cms-cs-hll-delta-producer branch May 26, 2026 16:30
zzylol added a commit that referenced this pull request May 26, 2026
Rebased onto #457 (which extended the per-window empty-base delta producer to
CMS/CountSketch/HLL). Adds the remaining piece: the delta-vs-full bandwidth
invariant — a delta frame is never larger than the equivalent full frame at the
same emit cadence.

- min(full, delta) clamp in ComputeDeltaAgainst for all four delta-capable
  families (DDSketch, CMS, CountSketch, HLL): after computing the sparse delta,
  emit the full frame instead whenever the delta is not strictly smaller. The
  sketch full frames are themselves compact (DDSketch packs a contiguous
  positional count array; HLL uses a sparse-packed register encoding), so a
  naive indexed/per-register delta can exceed the full frame on dense (DDSketch)
  or low-cardinality (HLL) data — the clamp bounds it. Delta transmission stays
  OFF by default; this only changes what an enabled delta frame looks like.
- docs: rename the design-alternative codenames to descriptive
  per-window-reset (PWR) / cumulative-epoch (CE).

Tests: DDSketch per-window tests use sparse data for the delta path and a dense
case asserting the clamp (payload ≤ full); HLL tests assert the clamp invariant
in both directions and that per-window reconstruction never leaks the prior
window. go test ./... green; edge processor builds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 26, 2026
…459)

Rebased onto #457 (which extended the per-window empty-base delta producer to
CMS/CountSketch/HLL). Adds the remaining piece: the delta-vs-full bandwidth
invariant — a delta frame is never larger than the equivalent full frame at the
same emit cadence.

- min(full, delta) clamp in ComputeDeltaAgainst for all four delta-capable
  families (DDSketch, CMS, CountSketch, HLL): after computing the sparse delta,
  emit the full frame instead whenever the delta is not strictly smaller. The
  sketch full frames are themselves compact (DDSketch packs a contiguous
  positional count array; HLL uses a sparse-packed register encoding), so a
  naive indexed/per-register delta can exceed the full frame on dense (DDSketch)
  or low-cardinality (HLL) data — the clamp bounds it. Delta transmission stays
  OFF by default; this only changes what an enabled delta frame looks like.
- docs: rename the design-alternative codenames to descriptive
  per-window-reset (PWR) / cumulative-epoch (CE).

Tests: DDSketch per-window tests use sparse data for the delta path and a dense
case asserting the clamp (payload ≤ full); HLL tests assert the clamp invariant
in both directions and that per-window reconstruction never leaks the prior
window. go test ./... green; edge processor builds.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant