feat(delta): per-window delta producer for CMS/CountSketch/HLL (delta-against-empty) - #457
Merged
Merged
Conversation
DDSketch already opts in to per-window delta-against-empty (#456). This extends the same opt-in to CountMinSketch, CountSketch, and HLL on both edge runtimes; KLL stays full-only and DDSketch is unchanged. asap-precompute-go (sketches/{cms,countsketch,hll}.go): each wrapper now implements DeltaAgainstEmptyBase(), returning the encoded empty-sketch snapshot the SnapshotCache caches as the outbound base after each window-close emit. The next window's ComputeDeltaAgainst (already wired to the sketchlib-go ComputeDelta for each family) then diffs against an empty base, so each window's delta carries that window's own per-window state — no cross-window subtraction. CMS msgpack mode (which cannot carry deltas) returns a nil base to keep legacy always-refresh; the SnapshotCache now treats a nil/empty empty-base as "fall through to legacy refresh". asap-precompute-rs (src/sketches/{cms,countsketch,hll}.rs): each wrapper implements delta_against_empty_base() and a real compute_delta_against that decodes the prior envelope and diffs via the asap_sketchlib compute_delta for that family (CMS/CountSketch cell deltas, HLL register delta), with a full-snapshot fallback on empty/undecodable prior. Each apply_delta now dispatches on payload shape (full-state envelope vs raw delta proto) via apply_delta_bytes, mirroring the DDSketch wrapper. The stale sketches/mod.rs "always-full" caveat is updated. Tests (mirroring DDSketch's): for each family, applying the emitted delta to an empty base reconstructs the window's state, and consecutive windows emit their own state (no cross-window subtraction) — Go in sketches/*_test.go, Rust in snapshot_cache.rs. HLL's disjoint-window test pins that the empty-base reset defeats register-MAX leakage. Depends on asap_sketchlib#59 (CMS/CountSketch/HLL compute_delta / apply_delta_bytes; the underlying codec is the byte-parity twin of the sketchlib-go delta codec). The backend base rotation (#346) is already sketch-agnostic, so no backend change is needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 26, 2026
Rebased onto #457 (which extended the per-window empty-base delta producer to CMS/CountSketch/HLL). Adds the remaining piece: the delta-vs-full bandwidth invariant — a delta frame is never larger than the equivalent full frame at the same emit cadence. - min(full, delta) clamp in ComputeDeltaAgainst for all four delta-capable families (DDSketch, CMS, CountSketch, HLL): after computing the sparse delta, emit the full frame instead whenever the delta is not strictly smaller. The sketch full frames are themselves compact (DDSketch packs a contiguous positional count array; HLL uses a sparse-packed register encoding), so a naive indexed/per-register delta can exceed the full frame on dense (DDSketch) or low-cardinality (HLL) data — the clamp bounds it. Delta transmission stays OFF by default; this only changes what an enabled delta frame looks like. - docs: rename the design-alternative codenames to descriptive per-window-reset (PWR) / cumulative-epoch (CE). Tests: DDSketch per-window tests use sparse data for the delta path and a dense case asserting the clamp (payload ≤ full); HLL tests assert the clamp invariant in both directions and that per-window reconstruction never leaks the prior window. go test ./... green; edge processor builds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 26, 2026
…459) Rebased onto #457 (which extended the per-window empty-base delta producer to CMS/CountSketch/HLL). Adds the remaining piece: the delta-vs-full bandwidth invariant — a delta frame is never larger than the equivalent full frame at the same emit cadence. - min(full, delta) clamp in ComputeDeltaAgainst for all four delta-capable families (DDSketch, CMS, CountSketch, HLL): after computing the sparse delta, emit the full frame instead whenever the delta is not strictly smaller. The sketch full frames are themselves compact (DDSketch packs a contiguous positional count array; HLL uses a sparse-packed register encoding), so a naive indexed/per-register delta can exceed the full frame on dense (DDSketch) or low-cardinality (HLL) data — the clamp bounds it. Delta transmission stays OFF by default; this only changes what an enabled delta frame looks like. - docs: rename the design-alternative codenames to descriptive per-window-reset (PWR) / cumulative-epoch (CE). Tests: DDSketch per-window tests use sparse data for the delta path and a dense case asserting the clamp (payload ≤ full); HLL tests assert the clamp invariant in both directions and that per-window reconstruction never leaks the prior window. go test ./... green; edge processor builds. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the per-window delta-against-empty producer (landed for DDSketch in #456) to CountMinSketch, CountSketch, and HLL on both edge runtimes (
asap-precompute-goandasap-precompute-rs). KLL stays full-only; DDSketch is unchanged.Each window now emits a delta computed against an empty base, so the wire payload carries that window's own per-window state with no cross-window subtraction — the prerequisite for correct window-scoped backend aggregation.
How
asap-precompute-go (
sketches/{cms,countsketch,hll}.go):DeltaAgainstEmptyBase(), returning the encoded empty-sketch snapshot theSnapshotCachecaches as the outbound base after each window-close emit. The next window'sComputeDeltaAgainst(already wired to the sketchlib-goComputeDeltafor each family) then diffs against the empty base.SnapshotCache.ComputeDeltanow treats a nil/empty empty-base as "fall through to legacy refresh", so a family can opt in conditionally.asap-precompute-rs (
src/sketches/{cms,countsketch,hll}.rs):delta_against_empty_base()and a realcompute_delta_againstthat decodes the prior envelope and diffs via theasap_sketchlibcompute_deltafor that family (CMS/CountSketch cell deltas, HLL register delta), with a full-snapshot fallback on an empty/undecodable prior.apply_deltanow dispatches on payload shape (full-state envelope vs raw delta proto viaapply_delta_bytes), mirroring the DDSketch wrapper. Stale "always-full" caveats insketches/mod.rs/precompute.rsdoc comments updated.The
SnapshotCachedispatch was already sketch-agnostic (any sketch implementing the empty-base interface opts in; others fall back to legacy always-refresh), so no dispatch logic changed — only the per-family opt-in and the conditional nil-base fall-through.Tests
Mirroring DDSketch's tests, for each of CMS / CountSketch / HLL:
Go:
sketches/{cms,countsketch,hll}_test.go. Rust:src/snapshot_cache.rs. Gogo build/go vet/go test ./sketches/...and Rustcargo build/cargo testare green (Rust built against the asap_sketchlib branch carrying #59).Byte-parity note: Go↔Rust per-window delta-frame byte parity is not directly asserted in this PR (no in-tree golden delta fixtures), but the underlying delta codec is the byte-parity twin verified by asap_sketchlib#59's Go-golden envelope probes; each Rust
compute_deltais documented as byte-identical to thesketchlib-goproto.Marshaloutput for the same inputs.Dependencies
compute_delta/apply_delta_bytes). Theasap-precompute-rspath-dep keeps the normal relative path; build against an asap_sketchlib carrying feat: unify sketch processor config interface #58 (DDSketch) + fix(series): send full attrs until collector confirms series ID (multi-hop fix) #59.🤖 Generated with Claude Code