Skip to content

Edge delta producer: guarantee delta ≤ full (min(full,delta) clamp) + decode-free / decoded-base caching - #459

Merged
zzylol merged 1 commit into
mainfrom
delta-invariant-producer
May 26, 2026
Merged

zzylol merged 1 commit into
mainfrom
delta-invariant-producer

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

What

Makes the edge sketch producer (asap-precompute-go) never emit a delta frame larger than the equivalent full frame at the same emit cadence — the delta-vs-full bandwidth invariant — and removes redundant per-window work on the emit path. Delta transmission stays OFF by default; this only changes what a delta frame looks like when enabled.

Why

A delta is only a win when it's smaller. But the sketch full frames are themselves compact (DDSketch packs a contiguous positional count array; HLL uses a sparse-packed register encoding), so a naive sparse/indexed delta can be larger than the full frame for dense (DDSketch) or low-cardinality (HLL) data. Measured on the SERF/Chimp/Elf real-world datasets, HLL single-emit deltas ran ~2.5× the full frame. The producer now bounds this.

Changes

  • min(full, delta) clamp
    • DDSketch — new DeltaFromEmpty computes the per-window delta against a fresh empty base (decode-free — no deserialize of the cached base) and returns the full frame whenever the delta isn't strictly smaller.
    • HLLComputeDeltaAgainst returns full when the register-update delta isn't smaller than the sparse-packed full frame (corrects the prior "register deltas are never larger" comment, false at low cardinality).
  • Decode-free / decoded-base caching
    • DDSketch per-window empty-base path no longer re-deserializes the (empty) cached base each window.
    • HLL/CMS/CountSketch ("always-refresh") cache a value-copy of just the count/register arrays the diff reads (clone-on-store), so the prior window's snapshot is never re-deserialized. Byte-parity tested against the serialize+decode path.
  • docs — renamed the design-alternative codenames to descriptive per-window-reset (PWR) / cumulative-epoch (CE).

Testing

  • Per-family clamp invariant (delta ≤ full, both clamp directions)
  • Decoded-base byte-parity (HLL/CMS/CountSketch): decode-free path == serialize+decode path, bit-for-bit
  • Decode-free round-trips + dense-vs-sparse DDSketch cases
  • go test ./... green; edge processor builds

🤖 Generated with Claude Code

Rebased onto #457 (which extended the per-window empty-base delta producer to
CMS/CountSketch/HLL). Adds the remaining piece: the delta-vs-full bandwidth
invariant — a delta frame is never larger than the equivalent full frame at the
same emit cadence.

- min(full, delta) clamp in ComputeDeltaAgainst for all four delta-capable
  families (DDSketch, CMS, CountSketch, HLL): after computing the sparse delta,
  emit the full frame instead whenever the delta is not strictly smaller. The
  sketch full frames are themselves compact (DDSketch packs a contiguous
  positional count array; HLL uses a sparse-packed register encoding), so a
  naive indexed/per-register delta can exceed the full frame on dense (DDSketch)
  or low-cardinality (HLL) data — the clamp bounds it. Delta transmission stays
  OFF by default; this only changes what an enabled delta frame looks like.
- docs: rename the design-alternative codenames to descriptive
  per-window-reset (PWR) / cumulative-epoch (CE).

Tests: DDSketch per-window tests use sparse data for the delta path and a dense
case asserting the clamp (payload ≤ full); HLL tests assert the clamp invariant
in both directions and that per-window reconstruction never leaks the prior
window. go test ./... green; edge processor builds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the delta-invariant-producer branch from 4be4ee9 to 1c39355 Compare May 26, 2026 19:32
@zzylol
zzylol merged commit cc37bab into main May 26, 2026
@zzylol
zzylol deleted the delta-invariant-producer branch May 26, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant