feat(delta): per-window DDSketch delta producer (delta-against-empty) - #456
Merged
Merged
Conversation
…-against-empty) Implements the PRODUCER half of "Option A" delta enablement (delta-baseline-contract.md §3) for DDSketch only — the first rollout family. The edge tumbling window already resets per-series sketch state every window (window.go:rotateLocked / window.rs); that is unchanged. When delta transmission is ON, the edge now emits each window's own DDSketch state as a delta computed against an EMPTY base (no cross-window subtraction). After each window-close emit, the SnapshotCache resets the cached outbound base to the empty-sketch snapshot, so the next window's ComputeDelta diffs against empty = that window's full per-window sketch encoded as a delta. This is what makes the backend's (future) per-window base rotation correct. Changes: - snapshot_cache.go / snapshot_cache.rs: after a window-close emit, if the sketch opts in to Option-A (DDSketch only, via the new emptyBaseDeltaSketch type-assertion in Go / Sketch::delta_against_empty_base default-None trait method in Rust), reset the cached base to the empty-sketch snapshot. The non-delta (full) path and the other families (CMS / CountSketch / HLL / KLL) are unchanged — they keep legacy always-refresh. - sketches/ddsketch.go: add DeltaAgainstEmptyBase() returning an empty DDSketch envelope. - sketches/ddsketch.rs: implement the previously-stubbed compute_delta_against (it returned is_full=true and never emitted a delta) using asap_sketchlib's DDSketch bucket-delta compute; add delta_against_empty_base() and teach apply_delta to dispatch full-envelope vs DDSketchDelta-proto payloads. The emitted DDSketch delta frame is byte-identical between Go and Rust for identical window state (verified: 345-byte frame for input 1..=50, α=0.01) — closes part of #243. Tests: - Go (sketches/ddsketch_test.go) + Rust (snapshot_cache.rs): delta-against-empty round-trips to the window's full state, quantiles within relative-ε; two consecutive windows each emit their OWN state (no cross-window subtraction). NOTE: this depends on a new DdSketch::compute_delta / apply_delta_bytes + DDSketchDelta proto message added to asap_sketchlib (separate branch); the commit here keeps the normal relative dep paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
changed the title
Comment-only edit removing the internal codename from the per-window-delta producer comments across the Go and Rust precompute sources. No logic, API, or test changes; the delta-baseline-contract.md §3 references are preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 26, 2026
Merged
zzylol
added a commit
that referenced
this pull request
May 26, 2026
…457) DDSketch already opts in to per-window delta-against-empty (#456). This extends the same opt-in to CountMinSketch, CountSketch, and HLL on both edge runtimes; KLL stays full-only and DDSketch is unchanged. asap-precompute-go (sketches/{cms,countsketch,hll}.go): each wrapper now implements DeltaAgainstEmptyBase(), returning the encoded empty-sketch snapshot the SnapshotCache caches as the outbound base after each window-close emit. The next window's ComputeDeltaAgainst (already wired to the sketchlib-go ComputeDelta for each family) then diffs against an empty base, so each window's delta carries that window's own per-window state — no cross-window subtraction. CMS msgpack mode (which cannot carry deltas) returns a nil base to keep legacy always-refresh; the SnapshotCache now treats a nil/empty empty-base as "fall through to legacy refresh". asap-precompute-rs (src/sketches/{cms,countsketch,hll}.rs): each wrapper implements delta_against_empty_base() and a real compute_delta_against that decodes the prior envelope and diffs via the asap_sketchlib compute_delta for that family (CMS/CountSketch cell deltas, HLL register delta), with a full-snapshot fallback on empty/undecodable prior. Each apply_delta now dispatches on payload shape (full-state envelope vs raw delta proto) via apply_delta_bytes, mirroring the DDSketch wrapper. The stale sketches/mod.rs "always-full" caveat is updated. Tests (mirroring DDSketch's): for each family, applying the emitted delta to an empty base reconstructs the window's state, and consecutive windows emit their own state (no cross-window subtraction) — Go in sketches/*_test.go, Rust in snapshot_cache.rs. HLL's disjoint-window test pins that the empty-base reset defeats register-MAX leakage. Depends on asap_sketchlib#59 (CMS/CountSketch/HLL compute_delta / apply_delta_bytes; the underlying codec is the byte-parity twin of the sketchlib-go delta codec). The backend base rotation (#346) is already sketch-agnostic, so no backend change is needed. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delta enablement, Phase 1 (producer side), DDSketch-first — per the delta-baseline contract (
docs/delta-baseline-contract.md§3).The edge tumbling window already resets per-series sketch state each window. This makes the delta match: after each window close, the
SnapshotCacheoutbound base is reset to empty, so each window's delta is computed against an empty base = that window's own state encoded as a delta (no cross-window subtraction). The backend (Phase 2) then rotates its per-series base at the window boundary, giving a clean per-window total. Self-healing: a dropped delta corrupts only one window.asap-precompute-go/snapshot_cache.go+sketches/ddsketch.go: opt-inDeltaAgainstEmptyBase(); reset base to empty after window-close emit. CMS/CountSketch/HLL/KLL unchanged (legacy always-refresh).asap-precompute-rs/{snapshot_cache,precompute,sketches/ddsketch}.rs: mirror viaSketch::delta_against_empty_base()(defaultNone); closed the Rust delta-emit gap (compute_delta_againstnow emits a real bucket-delta;apply_deltadispatches full vsDDSketchDelta).Depends on
asap_sketchlib'sDdSketch::compute_delta(merged). Backend per-window base rotation (Phase 2) is a separate ASAPQuery-backend PR.🤖 Generated with Claude Code